Japanese chestnut (Castanea crenata Siebold et Zucc.) is the only fruit tree species to have been domesticated in Japan, and it has been a part of human life since ancient times. Gene flow from cultivars to natural stands is common in Castanea species, which hinders a correct interpretation of genetic diversity. To clarify the genetic relationships and breeding history of chestnut, we analyzed its population structure, direction(s) of gene flow, and demography using 4 cultivar populations and 41 wild populations. The hierarchical population structure analysis revealed that the populations were roughly divided into Kyushu, southwest, and northeast clusters at K = 3, and cultivar clusters arose at K = 4. Bi-directional gene flow between cultivars and wild individuals was detected in Kanto region, whereas flow in only one direction — from cultivars to wild populations — was detected in Tanba region. The most likely scenario estimated by approximate Bayesian computation was that the Kyushu cluster diverged first, and then the southwest, northeast, and cultivar clusters separated simultaneously. Assuming a generation time of 20 years, the first divergence was around 50,000 years ago, which was before the Last Glacial Maximum (LGM, 21,000–18,000 years ago). The other clusters diverged around 20,000 years ago, during the LGM, which indicates a long breeding history for cultivars. The results obtained in this study provide new insight into the history of chestnut and would be useful for both breeding and conservation of Japanese chestnut.