In humans, a first-degree family history of dementia (FH) is a well-documented risk factor for Alzheimer’s disease (AD); however, the influence of FH on cognition across the lifespan is poorly understood. To address this issue, we developed an internet-based paired-associates learning (PAL) task and tested 59,571 participants between the ages of 18–85. FH was associated with lower PAL performance in both sexes under 65 years old. Modifiers of this effect of FH on PAL performance included age, sex, education, and diabetes. The Apolipoprotein E ε4 allele was also associated with lower PAL scores in FH positive individuals. Here we show, FH is associated with reduced PAL performance four decades before the typical onset of AD; additionally, several heritable and non-heritable modifiers of this effect were identified.