Abstract Background Little is known about the inter-relationship among fruit and vegetable intake, gut microbiota and metabolites, and type 2 diabetes (T2D) in human prospective cohort study. The aim of the present study was to investigate the prospective association of fruit and vegetable intake with human gut microbiota and to examine the relationship between fruit and vegetable-related gut microbiota and their related metabolites with type 2 diabetes (T2D) risk. Methods This study included 1879 middle-age elderly Chinese adults from Guangzhou Nutrition and Health Study (GNHS). Baseline dietary information was collected using a validated food frequency questionnaire (2008–2013). Fecal samples were collected at follow-up (2015–2019) and analyzed for 16S rRNA sequencing and targeted fecal metabolomics. Blood samples were collected and analyzed for glucose, insulin, and glycated hemoglobin. We used multivariable linear regression and logistic regression models to investigate the prospective associations of fruit and vegetable intake with gut microbiota and the association of the identified gut microbiota (fruit/vegetable-microbiota index) and their related fecal metabolites with T2D risk, respectively. Replications were performed in an independent cohort involving 6626 participants. Results In the GNHS, dietary fruit intake, but not vegetable, was prospectively associated with gut microbiota diversity and composition. The fruit-microbiota index (FMI, created from 31 identified microbial features) was positively associated with fruit intake (p < 0.001) and inversely associated with T2D risk (odds ratio (OR) 0.83, 95%CI 0.71–0.97). The FMI-fruit association (p = 0.003) and the FMI-T2D association (OR 0.90, 95%CI 0.84–0.97) were both successfully replicated in the independent cohort. The FMI-positive associated metabolite sebacic acid was inversely associated with T2D risk (OR 0.67, 95%CI 0.51–0.86). The FMI-negative associated metabolites cholic acid (OR 1.35, 95%CI 1.13–1.62), 3-dehydrocholic acid (OR 1.30, 95%CI 1.09–1.54), oleylcarnitine (OR 1.77, 95%CI 1.45–2.20), linoleylcarnitine (OR 1.66, 95%CI 1.37–2.05), palmitoylcarnitine (OR 1.62, 95%CI 1.33–2.02), and 2-hydroglutaric acid (OR 1.47, 95%CI 1.25–1.72) were positively associated with T2D risk. Conclusions Higher fruit intake-associated gut microbiota and metabolic alteration were associated with a lower risk of T2D, supporting the public dietary recommendation of adopting high fruit intake for the T2D prevention.