Communities of arbuscular mycorrhizal fungi (AMF) in plant roots improve host plant growth. In this study, AMF communities in the roots of the sago palm (Metroxylon sagu Rottb.) were investigated in mineral soil (MS) and shallow peat soil (SPS) in Sarawak, Malaysia. MS exhibited lower moisture content (MS, 38.1; SPS, 79.8%), higher pH (H2O) (MS, 4.6; SPS, 4.1), higher soil bulk density (MS, 1.03; SPS, 0.20 g cm−3), and higher nitrogen content (MS, 16.9; SPS, 2.7 kg m−3) than SPS at the same soil depth, while the phosphorus (P) content (Bray II) (MS, 1.6; SPS, 1.9 g P2O5 m−3) was similar. The AMF colonization rate was significantly lower in SPS (39.2 ± 12.5%) than in MS (73.2 ± 4.6%). The higher number of AMF operational taxonomic units (OTUs) was detected by amplicon sequencing of the partial small-subunit rRNA gene (MS, 78; SPS, 50). A neighbor-joining tree of obtained OTUs revealed that they belonged to Acaulosporaceae, Ambisporaceae, Claroideoglomeraceae, Gigasporaceae, and Glomeraceae. The lower abundance and diversity of AMF in SPS are possibly caused by abiotic factors, including soil physicochemical properties. Glomus and Acaulospora species detected in SPS might have strong tolerance against acidity and high soil moisture content.