The toughening of epoxy resin has always been one of the key topics in the researches about high-performance matrix used for advanced composites. Up to now, single-property enhancement is usually accompanied by some compromises of other properties. This work constructed a novel in-situ interpenetrating network (IPN) structure composed of epoxy resin and vinyl ester resin (VER), along with modified octavinyl polyhedral oligomeric silsesquioxane (EPOSS) as the toughening core. In the present resin system, the VER component was firstly introduced to form IPN structures without phase separation to bring the improvement of overall strength and plastic deformation capacity. Moreover, the EPOSS could realize nano-scale dispersion in the resin matrix through the chemical reactions, which acted as the trigger of plastic shear deformation toughening under particle debonding. The side chains' physical entanglement of EPOSS played the key role in maintaining a balance between toughness and mechanical strength. With only 2 wt% of EPOSS, the optimal increase in impact toughness reached 139.6%, in addition to the 17.9% improvement of tensile strength and 9.0% enhancement of Young's modulus.