Background: Neighborhood attributes are modifiable determinants of physical activity (PA) and sedentary behavior (SB). We tested whether the objectively-assessed built and social environment was associated with PA and SB in Swiss youth and whether sex, age and the socioeconomic position (Swiss-SEP) modified such associations. Methods: We combined data of 1742 youth (ages 4 to 17) from seven studies conducted within Switzerland between 2005–2010. All youth provided accelerometer data and a home address, which was linked to objective environmental data and the Swiss-SEP-index. Associations between neighborhood attributes and PA were analyzed by multivariable multilevel regression analyses. Results: The extent of green areas and building density was positively associated with PA in the total sample (p < 0.05). Factors representing centrally located areas, and more schoolchildren living nearby tended to increase PA in secondary schoolchildren, boys and those from lower-ranked socioeconomic areas. In primary schoolchildren, the extent of green areas was positively associated with PA (p = 0.05). Associations between neighborhood attributes and PA were more pronounced in youth from low socioeconomic areas. Conclusions: The results indicate that some associations between neighborhood attributes and PA differ by age, sex and socioeconomic area. This should be taken into account when planning interventions to increase childhood PA.