Different austenitizing temperatures were used to obtain medium-carbon low-alloy (MCLA) steels with different initial states of tempering, including lath martensite microstructures and matrix carbon contents. The effects of these factors on the tempered carbides and impact toughness of the MCLA steels were investigated via optical microscopy (OM), scanning electron microscopy (SEM), electron backscatter diffraction (EBSD), X-ray diffraction (XRD) and transmission electron microscopy (TEM). The results showed that the matrix carbon content and lath martensite substructure sizes could be effectively controlled by the austenitizing temperature. The morphologies of the tempered carbides precipitated at the boundaries were controlled by both the matrix carbon content and the lath martensite substructure sizes. Higher matrix carbon content and boundary densities increased the nucleation rate and promoted the formation of spherical carbides. The spheroidization of carbides was conducive to improving the impact toughness. However, the decrease in the density of high-angle grain boundaries and the appearance of long strip-shaped carbides drastically reduced the impact toughness.