The diffusion-controlled growth mode is widely used to narrow the size distribution of colloidal quantum dots. However, this growth mode always suffers from size broadening at the later growth stage. By monitoring the growth process of CdS colloidal quantum dots, we show the size broadening is a result of different growth rates of CdS colloidal quantum dots (CQDs) with different morphologies. Monomer concentration-dependent growth experiments demonstrate the different growth rates are caused by the different ligand permeabilities of CdS CQDs. The cubic ones have lower ligand permeability but higher saturated surface reaction rate than the noncubic ones, leading to unexpected narrower size distribution under higher monomer concentration. More efficient narrowing can be obtained by the addition of chloride ions, which can increase the ligand permeability of all CdS CQDs, as well as the opposite discrepancies in ligand permeability and surface reaction between cubic and noncubic CdS CQDs. The photoluminescence (PL) full width at half maximum (FWHM) of CdS CQDs can be narrowed down to below 80 meV for PL peaks from 430 to 500 nm. Given the inevitable usage of the ligands in the solution synthesis of colloidal nanocrystals, the influence of morphology difference on growth rate should be common. Our results can provide an alternative solution to realize size focusing for the synthesis of colloidal nanocrystals.