Abstract Background The role of the innate immune system in Alzheimer’s disease (AD) and neurodegenerative disease susceptibility has recently been highlighted in genetic studies. However, we do not know whether risk for inflammatory disease predisposes unaffected individuals to late-life cognitive deficits or AD-related neuropathology. We investigated whether genetic risk scores for seven immune diseases and central nervous system traits were related to cognitive decline (nmax = 1601), classical AD neuropathology (nmax = 985), or microglial density (nmax = 184). Methods Longitudinal cognitive decline, postmortem amyloid and tau neuropathology, microglial density, and gene module expression from bulk brain tissue were all measured in participants from two large cohorts (the Rush Religious Orders Study and Memory and Aging Project; ROS/MAP) of elderly subjects (mean age at entry 78 +/− 8.7 years). We analyzed data primarily using robust regression methods. Neuropathologists were blind to clinical data. Results The AD genetic risk scores, including and excluding APOE effects, were strongly associated with cognitive decline in all domains (min P uncor = 3.2 × 10− 29). Multiple sclerosis (MS), Parkinson’s disease, and schizophrenia risk did not influence cognitive decline in older age, but the rheumatoid arthritis (RA) risk score alone was significantly associated with microglial density after correction (t146 = − 3.88, P uncor = 1.6 × 10− 4). Post-hoc tests found significant effects of the RA genetic risk score in multiple regions and stages of microglial activation (min P uncor = 1.5 × 10− 6). However, these associations were driven by only one or two variants, rather than cumulative polygenicity. Further, individual MS (P one-sided < 8.4 × 10− 4) and RA (P one-sided = 3 × 10− 4) variants associated with higher microglial density were also associated with increased expression of brain immune gene modules. Conclusions Our results demonstrate that global risk of inflammatory disease does not strongly influence aging-related cognitive decline but that susceptibility variants that influence peripheral immune function also alter microglial density and immune gene expression in the aging brain, opening a new perspective on the control of microglial and immune responses within the central nervous system. Further study on the molecular mechanisms of peripheral immune disease risk influencing glial cell activation will be required to identify key regulators of these pathways.