ABSTRACT OBJECTIVE To describe the prevalence of the practice of physical activity (PA) among older Brazilian adults and associated factors. In addition, potential effect modifiers of the association between PA and age were investigated. METHODS We have analyzed data from 8,736 participants (92.8%) aged 50 and older from the Brazilian Longitudinal Study of Aging (ELSI-Brazil). Physical activity was measured using the short version of the International Physical Activity Questionnaire. The outcome variable was defined as at least 150 minutes of weekly activities in all domains. The exploratory variables were age, sex, education, ethnicity, marital status, number of chronic diseases and medical appointments, and knowledge about or participation in public programs that encourage physical activity. Logistic regression and estimates of predicted probabilities were performed. RESULTS The prevalence of recommended levels of physical activity was 67.0% (95%CI 64.3–69.5). Physical activity was associated with age [odds ratio (OR) = 0.97; 95%CI 0.96–0.98], higher educational level (OR = 1.27; 95%CI 1.11–1.45 for 4–7 years and OR = 1.52; 95%CI 1.28–1.81 for eight years or more), participants who were married/ in a long term relationship (OR = 1.22; 95%CI 1.08–1.38), and those who reported knowledge about (OR = 1.34; 95%CI 1.16–1.54) or participation in (OR = 1.78; 95%CI 1.34–2.36) a program aimed at the practice of physical activity. Women and those with lower educational level (p value for interaction < 0.05) reported lower physical activity levels. CONCLUSIONS In addition to the association with marital status and health promotion programs, there were significant sex and educational level inequalities in physical activity decline later in life. These findings help the identification of groups more vulnerable to decreased physical activity levels with aging, as well as the planning of health promotion strategies, especially in older groups.