Abstract Background Despite knowing better how to screen older adults, understanding how frailty progression might be modified is unclear. We explored effects of modifiable and non-modifiable factors on changes in frailty in community-dwelling adults aged 50+ years who participated in the Canadian Multicentre Osteoporosis Study (CaMos). Methods Rates of change in frailty over 10 years were examined using the 30-item CaMos Frailty Index (CFI). Incident and prevalent low-trauma fractures were categorized by fracture site into hip, clinical vertebral and non-hip-non-vertebral fractures. Multivariable generalized estimating equation models accounted for the time of frailty assessment (baseline, 5 and 10 years), sex, age, body mass index (BMI, kg/m2), physical activity, bone mineral density, antiresorptive therapy, health-related quality of life (HRQL), cognitive status, and other factors for frailty or fractures. Multiple imputation and scenario analyses addressed bias due to attrition or missing data. Results The cohort included 5566 women (mean ± standard deviation: 66.8 ± 9.3 years) and 2187 men (66.3 ± 9.5 years) with the mean baseline CFI scores of 0.15 ± 0.11 and 0.12 ± 0.10, respectively. Incident fractures and obesity most strongly predicted frailty progression in multivariable analyses. The impact of fractures differed between the sexes. With each incident hip fracture, the adjusted mean CFI accelerated per 5 years by 0.07 in women (95% confidence interval [CI]: 0.03 to 0.11) and by 0.12 in men (95% CI: 0.08 to 0.16). An incident vertebral fracture increased frailty in women (0.05, 95% CI: 0.02 to 0.08) but not in men (0.01, 95% CI: -0.07 to 0.09). Irrespective of sex and prevalent fractures, baseline obesity was associated with faster frailty progression: a 5-year increase in the adjusted mean CFI ranged from 0.01 in overweight (BMI: 25.0 to 29.9 kg/m2) to 0.10 in obese individuals (BMI: ≥ 40 kg/m2). Greater physical activity and better HRQL decreased frailty over time. The results remained robust in scenario analyses. Conclusions Older women and men with new vertebral fractures, hip fractures or obesity represent high-risk groups that should be considered for frailty interventions.