Objectives To describe prevalence and predictors of poor sexual well-being for men and women over 5 years following treatment for colorectal cancer.Design Prospective longitudinal study, from presurgery to 5 years postsurgery, with eight assessment points. Logistic regression models predicted sexual well-being from presurgery to 24 months and 24 months to 60 months; time-adjusted then fully adjusted models were constructed at each stage.Setting Twenty-nine hospitals in the UK.Participants Patients with Dukes’ stage A–C, treated with curative intent, aged ≥18 years and able to complete questionnaires were eligible.Outcome measures The dependent variable was the Quality of Life in Adult Cancer Survivors sexual function score. Independent variables included sociodemographic, clinical and psychosocial characteristics.Results Seven hundred and ninety participants provided a sexual well-being score for at least one time point. Thirty-seven per cent of men and 14% of women reported poor sexual well-being at 5 years. Baseline predictors for men at 24 months included having a stoma (OR 1.5, 95% CI 1.02 to 2.20) and high levels of depression (OR 2.69/2.01, 95% CI 1.68 to 4.32/1.12 to 3.61); men with high self-efficacy (OR confident 0.33/0.48, 95% CI 0.18 to 0.61/0.24 to 1.00; very confident 0.25/0.42, 95% CI 0.13 to 0.49/0.19 to 0.94) and social support (OR 0.52/0.56, 95% CI 0.33 to 0.81/0.35 to 0.91) were less likely to report poor sexual well-being. Predictors at 60 months included having a stoma (OR 2.30/2.67, 95% CI 1.22 to 4.34/1.11 to 6.40) and high levels of depression (OR 5.61/2.58, 95% CI 2.58 to 12.21/0.81 to 8.25); men with high self-efficacy (very confident 0.14, 95% CI 0.047 to 0.44), full social support (OR 0.26; 95% CI 0.13 to 0.53) and higher quality of life (OR 0.97, 95% CI 0.95 to 0.98) were less likely to report poor sexual well-being. It was not possible to construct models for women due to low numbers reporting poor sexual well-being.Conclusions Several psychosocial variables were identified as predictors of poor sexual well-being among men. Interventions targeting low self-efficacy may be helpful. More research is needed to understand women’s sexual well-being.