ABSTRACT OBJECTIVE To determine the impact of the physical and social surroundings of the neighborhood, which are presented as facilitators or barriers for the social participation of Brazilian older adults. METHODS The study was conducted in a probabilistic representative sample of the Brazilian population aged 50 years and older and who lived in urban areas (n = 7,935). The response variable was social participation, which was defined from two questions about activities performed with other persons: visited friends or relatives in their homes in the last 12 months (yes, no); went out with other persons to public places, such as restaurant, movies, club, park, in the last 12 months (yes, no). The explanatory variables included fear of falling because of defects in sidewalks, concern about the difficulty to get on a bus, subway, or train, difficulty to cross streets, and perception of violence in the neighborhood. Potential confounding variables included age, marital status, education level, self-rated health, living in an asphalted or paved street, time living in the municipality, and socioeconomic position score. Prevalence ratios and respective confidence intervals were estimated using Poisson regression. RESULT Difficulty to cross streets presented an independent association with restricted social participation (PR = 0.95; 95%CI 0.93–0.98) among both women (PR = 0.96; 95%CI 0.92–0.99) and men (PR = 0.94; 95%CI 0.90–0.99). Concern about the difficulty to get on a bus, subway, or train was associated with the outcome only among men (PR = 0.95; 95%CI 0.91–0.99). The fear of falling because of defects in sidewalks and the perception of violence in the neighborhood were not associated with social participation. CONCLUSIONS Urban characteristics that hinder the crossing of streets and accessibility to public transport can be inferred as important barriers for the social participation of Brazilian older adults.