Background/Objectives: Neutropenia is a common adverse effect of oral valganciclovir (VGCV) treatment in infants with congenital cytomegalovirus infection (CCMVI), with an estimated prevalence of 20%. However, its clinical course and associated factors, including the influence of VGCV dosage, remain inadequately characterized. Methods: We conducted a single-center retrospective cohort study of infants treated with VGCV for symptomatic congenital CMV infection (CCMVI) at the Kobe University Hospital between 1 April 2009 and 31 March 2017. Detailed descriptive analyses of neutropenia were performed, and factors associated with its onset were explored using univariable logistic regression analyses. Results: A total of 31 patients were included, and neutropenia occurred in 35% of them during the 6-week treatment period. Its occurrence was observed throughout the treatment course, with no substantial difference in incidence between the 16 mg/kg/day and 32 mg/kg/day dosing groups. Neutropenia was more likely to occur in infants with shorter gestational age. Conclusions: Neutropenia occurred in 35% of patients during 6 weeks of VGCV treatment, irrespective of dosage, and was more common in those with shorter gestational age.