Quantifying the effect of external forcings like seismicity or rain on slope destabilization is a long-standing and challenging issue. To investigate the respective roles of these forcings, we analyze an unprecedented 10-year long catalog of rockfalls occurring in the crater of the Piton de la Fournaise volcano (La Reunion Island), using statistical tools originally developed for earthquakes. Our analysis reveals the predominant effect of low amplitude repetitive seismicity in the triggering of rockfalls located at a few kilometers from the source, due to progressive damaging of the slope. Moreover, we show that the efficiency and time-delay of this dynamic triggering is controlled by the stability state of the slope, i.e. its closeness to the failure, as observed with lab-experiments on metastable granular slopes. Our results show the need to account for long-term swarm-type seismic activity that can affect the stability of geological structures like slopes and faults, but also buildings.