Grain filling can directly influence rice yield. However, there is limited information on the growth relationship among grains at different positions on the entire panicle during grain filling. In this study, field experiments were conducted in 2014−2015 to compare the growth dynamics of grains at various positions for two rice cultivars (Nongle 1 and Guifeng 2). The results showed that a high similarity and a slow−fast−slow trend of dry-matter accumulation occurred in all primary branches. However, the maximum grain growth rates of the top primary branches were 86% and 44% higher than basal primary branches of Nongle 1 and Guifeng 2, respectively. Similarly, the maximum final grain weights were 32% and 18% greater in the top primary branches than in the basal primary branches of Nongle 1 and Guifeng 2, respectively. In contrast, the active grain filling duration was 1.5 and 1.3 times longer in the basal primary branches than the top primary branches of Nongle 1 and Guifeng 2, respectively. The time to reach the maximum rate of grain growth of the basal primary branches for Nongle 1 and Guifeng 2 was 2.2 and 2.5 times longer than those of the top primary branches, respectively. Based on cluster analysis of growth characteristics of all primary branches, Group I (superior primary-branches) was considered to be the fastest for grain filling and greatest for dry matter weight, followed by Group II (medium primary-branches). The poorest growth occurred in Group III (inferior primary-branches). Therefore, the yield of poor-filling grains at the basal panicle could be achieved primarily by improving the growth of Group III.