Incomplete atomic bonding near grain boundaries forms trapping states and creates grain boundary potential energy barriers (E_b), which could influence thermoelectric (TE) properties. Here, we modulate the heights of E_b for light- and heavy-doped hafnium (Hf) in NbFeSb samples at different ball milling (BM) times. With the help of trapping-state model, we find in light-doped samples that E_b increases from 10 meV for the BM-3 h sample to 105 meV for the BM-12 h sample, leading to a reduction in mobility and change in carrier scattering mechanism. Further, we observe that the mobility of heavy-doped samples is unaffected by E_b, which assists to obtain a higher power factor. The stronger phonon scattering also helps to reach lower lattice thermal conductivity. Synergistically, a figure of merit of 0.85 at 973 K is achieved for the Nb_0.86Hf_0.14FeSb BM-3 h sample. This study offers insights into the role of E_b and could provide a promising method to optimize TE performance.