In this study, a novel organic-inorganic hybrid flame retardant 10-(1,4-dicarboxylic acid magnesium salt)-9,10-dihydro-9-oxa-10-phosphaphenanthrene-10-oxide (DMMH) was synthesized via neutralization and addition reaction of maleic acid, magnesium hydroxide and 9,10-dihydro-9-oxa-10-phosphaphenanthrene-10-oxide (DOPO), and subsequently used in an intrinsic flame retardant epoxy resin. The chemical structure and morphology of DMMH were analyzed by Fourier transform infrared spectroscopy (FTIR), X-ray photoelectron spectroscopy (XPS) and scanning electron microscopy (SEM). Further, the prepared DMMH was combined with ammonium polyphosphate (APP) to form an intumescent flame retardant system. The thermal stability and flame retardance were evaluated by thermogravimetric analysis (TG), UL-94 vertical burning test, limiting oxygen index (LOI) and cone calorimetry. It was observed that the addition of 1.7% DMMH and 5.3% APP led EP-7 to acquire UL-94 V-0 rating, with the limiting oxygen index of 26.0%. As compared with pure EP, the peak heat release rate, total heat release, smoke production rate and total smoke production of the sample was noted to decrease by 54.5%, 35.1%, 43.6% and 38.1%, respectively. In addition, the introduction of DMMH did not negatively impact the mechanical properties of the epoxy resin.