Abstract Background Grain shape is a critical agronomic trait affecting grain yield and quality. Exploration and functional characterization of grain shape-related genes will facilitate rice breeding for higher quality and yield. Results Here, we characterized a recessive mutant named Oat-like rice for its unique grain shape which highly resembles oat grains. The Oat-like rice displayed abnormal floral organs, an open hull formed by remarkably elongated leafy lemmas and paleae, occasionally formed conjugated twin brown rice, an aberrant grain shape and a low seed setting rate. By map-based cloning, we discovered that Oat-like rice harbors a novel allele of OsMADS1 gene (OsMADS1 Olr), which has a spontaneous point mutation that causes the substitution of an amino acid that is highly conserved in the MADS-box domain of the MADS-box family. Further linkage analysis indicated that the point mutation in the OsMADS1 Olr is associated with Oat-like rice phenotype, and expression analysis of the OsMADS1 by qRT-PCR and GUS staining also indicated that it is highly expressed in flower organs as well as in the early stages of grain development. Furthermore, OsMADS1 Olr-overexpressing plants showed similar phenotypes of Oat-like rice in grain shape, possibly due to the dominant negative effect. And OsMADS1-RNAi plants also displayed grain phenotypes like Oat-like rice. These results suggested that OsMADS1 Olr is responsible for the Oat-like rice phenotype including aberrant grain shape. Moreover, the expression levels of representative genes related to grain shape regulation were apparently altered in Oat-like rice, OsMADS1 Olr-overexpressing and OsMADS1-RNAi transgenic plants. Finally, compared with Oat-like rice, OsMADS1 Olr-overexpressing and OsMADS1-RNAi plants, mild phenotype of seed-specific OsMADS1-RNAi transgenic plants indicated that OsMADS1 may has has a direct regulation role in grain development and the grain phenotypes of Oat-like rice, OsMADS1 Olr-overexpressing and OsMADS1-RNAi plants are majorly caused by the abnormal lemma and palea development. Conclusions Altogether, our results showed that grain shape and a low seed setting rate of the notable ‘Oat-like rice’ are caused by a spontaneous point mutation in the novel allele OsMADS1 Olr. Furthermore, our findings suggested that OsMADS1 mediates grain shape possibly by affecting the expression of representative genes related to grain shape regulation. Thus, this study not only revealed that OsMADS1 plays a vital role in regulating grain shape of rice but also highlighted the importance and value of OsMADS1 to improve the quality and yield of rice by molecular breeding.