Specific brain structures (gray matter regions and white matter tracts) play a dominant role in determining cognitive decline and explain the heterogeneity in cognitive aging. Identification of these structures is crucial for screening of older adults at risk of cognitive decline. Using deep learning models augmented with a model-interpretation technique on data from 1432 Mayo Clinic Study of Aging participants, we identified a subset of brain structures that were most predictive of individualized cognitive trajectories and indicative of cognitively resilient vs. vulnerable individuals. Specifically, these structures explained why some participants were resilient to the deleterious effects of elevated brain amyloid and poor vascular health. Of these, medial temporal lobe and fornix, reflective of age and pathology-related degeneration, and corpus callosum, reflective of inter-hemispheric disconnection, accounted for 60% of the heterogeneity explained by the most predictive structures. Our results are valuable for identifying cognitively vulnerable individuals and for developing interventions for cognitive decline.