Background: Poor sleep quality is associated with poor quality of life and may even lead to mental illnesses. Several studies have indicated the association between exposure to air pollution and sleep quality. However, the evidence is very limited in China, especially in rural areas. Methods: Participants in this study were obtained from the Henan Rural Cohort established during 2015–2017. Sleep quality was evaluated using the Pittsburgh Sleep Quality Index (PSQI) in the baseline survey. Poor sleep quality was defined by the global score of PSQI > 5. Participants’ exposures to PM2.5, PM10 (particulate matter with aerodynamic diameters ≤2.5 μm and 10 μm, respectively) and NO2 (nitrogen dioxide) during the three years before the baseline survey were estimated using a satellite-based prediction. The associations between long-term exposure to air pollutants and sleep quality were examined using both the linear regression and logistic regression models. Results: The IQRs (interquartile range) of mean levels of participants’ exposures to PM2.5, PM10 and NO2 were 3.3 µg/m3, 8.8 µg/m3, and 4.8 µg/m3, respectively. After adjusted for potential confounders, the global score of PSQI (and 95%CI, 95% confidence intervals) increased by 0.16 (0.04, 0.27), 0.09 (−0.01, 0.19) and 0.14 (0.03, 0.24), associated with per IQR increase in PM2.5, PM10 and NO2, respectively. The odds ratios (and 95%CI) of poor sleep quality associated with per IQR increase in PM2.5, PM10 and NO2 were 1.15 (1.03, 1.29), 1.11 (1.02, 1.21) and 1.14 (1.03, 1.25), respectively. Conclusions: Long-term exposures to PM2.5, PM10 and NO2 were associated with poor sleep quality in rural China. Improvement of air quality may help to improve sleep quality among rural population of China.