Background: This study evaluated the effect of exercise training on body temperature and clarified the relationship between body temperature and body composition in the elderly. Methods: In this retrospective cohort study, a total of 91 elderly participants performed aerobic and anaerobic exercise training twice a week for 2 years. Non-contact infrared thermometer and bioelectrical impedance analysis were performed at baseline and at 2 years. Results: Mean age of study participants was 81.0 years. The participants were divided into two groups by baseline body temperature of 36.3 °C; lower body temperature group (n = 67) and normal body temperature group (n = 24). Body temperature rose significantly after exercise training in the lower body temperature group (36.04 ± 0.11 °C to 36.30 ± 0.13 °C, p p = 0.39). A positive correlation was observed between the amount of change in body temperature and baseline body temperature (r = −0.68, p p = 0.02). Conclusions: Exercise training raised body temperature in the elderly, especially those with lower baseline body temperature.