Nanoparticle assembly offers a versatile tool for constructing new structural materials with emergent or collective properties beyond individual nanoparticles. The achievement of desired properties and functions of these assembly materials often require delicate control over the interactions between nanoparticle building blocks. As of now, tremendous efforts have been devoted to manipulating the interparticle interactions by functionalizing the surface of nanoparticles with different ligands (e.g., small molecules, DNAs, proteins, and polymers). Among others, polymers are particularly attractive, owing to their tailorable molecular structures, rich functionalities, tunable responsiveness, superior biodegradability and biocompatibility, and easy mass production at low cost, etc. In this review, we present a summary of recent advances in engineering interparticle interactions between nanoparticles, especially inorganic nanoparticles with different sizes, shapes, and compositions, by tailoring the structurally defined polymers grafted or absorbed on the surface of nanoparticles. Discussions are focused on various interactions (i.e., steric repulsion, Coulombic interaction, hydrophobic interaction, hydrogen bonding, chemical reaction-induced recognitive interaction, and entropic effect) dominating the assembly of polymer-modified nanoparticles. Furthermore, the effect of external fields (e.g., light field, electric field, etc.) on the interactions between polymer-modified nanoparticles is presented.