Abstract Background Observational studies have linked childhood obesity with elevated risk of colorectal cancer; however, it is unclear if this association is causal or independent from the effects of obesity in adulthood on colorectal cancer risk. Methods We conducted Mendelian randomization (MR) analyses to investigate potential causal relationships between self-perceived body size (thinner, plumper, or about average) in early life (age 10) and measured body mass index in adulthood (mean age 56.5) with risk of colorectal cancer. The total and independent effects of body size exposures were estimated using univariable and multivariable MR, respectively. Summary data were obtained from a genome-wide association study of 453,169 participants in UK Biobank for body size and from a genome-wide association study meta-analysis of three colorectal cancer consortia of 125,478 participants. Results Genetically predicted early life body size was estimated to increase odds of colorectal cancer (odds ratio [OR] per category change: 1.12, 95% confidence interval [CI]: 0.98–1.27), with stronger results for colon cancer (OR: 1.16, 95% CI: 1.00–1.35), and distal colon cancer (OR: 1.25, 95% CI: 1.04–1.51). After accounting for adult body size using multivariable MR, effect estimates for early life body size were attenuated towards the null for colorectal cancer (OR: 0.97, 95% CI: 0.77–1.22) and colon cancer (OR: 0.97, 95% CI: 0.76–1.25), while the estimate for distal colon cancer was of similar magnitude but more imprecise (OR: 1.27, 95% CI: 0.90–1.77). Genetically predicted adult life body size was estimated to increase odds of colorectal (OR: 1.27, 95% CI: 1.03, 1.57), colon (OR: 1.32, 95% CI: 1.05, 1.67), and proximal colon (OR: 1.57, 95% CI: 1.21, 2.05). Conclusions Our findings suggest that the positive association between early life body size and colorectal cancer risk is likely due to large body size retainment into adulthood.