Background: Identification of early modifiable factors is crucial to delay or prevent the development of cognitive impairment and reduce the social and economic burden. Objective: This study aimed to examine the longitudinal associations of childhood neighborhood quality (CNQ) with the risk of later-life cognitive dysfunction and the role of body mass index (BMI) in this association. Methods: A total of 8,289 community-dwelling middle-aged and elderly population from wave 2011, wave 2013, and wave 2015 of the China Health and Retirement Longitudinal Study (CHARLS) were included. Cognitive function and CNQ were measured by standardized questionnaires. Multilevel linear regression models were used to estimate the associations of CNQ and cognitive function. The interactions of BMI with CNQ in the progress of cognitive function were also estimated. Results: The participants with higher CNQ had a significantly low risk of cognitive impairment than those with lower CNQ score (β = 0.067, 95% CI: 0.031, 0.103), and the results remained similar (β = 0.039, 95% CI: 0.004, 0.075) after controlling other confounding variables. Furthermore, there was an interaction between BMI with CNQ score (P < 0.001) for the risk of cognitive impairment. In BMI-stratified analysis, we found that the association of CNQ and cognitive function was not statistically significant in overweight or obese population (β = 0.019, 95% CI: -0.032, 0.070), but was statistically significant in people with lower BMI (β = 0.059, 95% CI: 0.010, 0.107). Conclusions: Higher CNQ score is significantly associated with the lower risk of cognitive dysfunction in adulthood. BMI may moderate the associations of CNQ with the risk of cognitive function.