Abstract Background To our knowledge, no population-based studies have examined whether family or friend caregivers of men and women differ in their experience of distress over time. Thus, we aimed to describe, on a population-level and longitudinally, how older men and women care-receivers differed in their health and care needs, compare their caregivers’ distress trajectories, and identify factors that contribute to the observed differences. Methods This is a population-based, retrospective cohort study using routinely collected data. We examined longitudinally 485,407 community-dwelling Ontario residents, aged over 50 years, who have received at least one Residential Assessment Instrument-Home Care (RAI-HC) assessment between April 2008 and June 2015. Descriptive analyses were performed on the demographic characteristics, health profiles, and care needs of men and women. We also compared their caregivers’ baseline and one-year change in distress status. Logistic regressions were performed to examine if the effect of gender on caregiver distress is reduced after controlling for care-receiver’s health and functional status as well as their caregiver’s kinship and co-residence status. Results Men (39.5% of our cohort) were frailer, required more care, were mostly cared for by their spouses (52%), and mostly lived with their caregiver (66%). In contrast, women (60.5%) were more likely cared for by their child/child-in-law (60%), less likely to live with caregivers (47%), and received less care. Caregivers of men were more likely to be distressed at baseline (27.7% versus 20.4% of women caregivers) and remain distressed (74.6% versus 69.5%) or become distressed (19.3% versus 14.3%) throughout the year. In logistic regression modelling, the effect of care-receiver’s gender on caregiver distress is reduced from an unadjusted odds ratio of 1.49 (95% CI: 1.47–1.51) to 1.17 (95% CI: 1.15–1.19) when care-receiver’s health and caregiving factors are controlled for. Conclusion Older men and women differed in health and care needs. Caregivers, especially those caring for men, were often distressed and remained so through time. These results highlight the need for policies that account for the differential care needs and caregiver profiles of men and women in order to offer targetted and appropriate support.