Antibody-mediated immunity plays a crucial role in protection against SARS-CoV-2 infection. We isolated a panel of neutralizing anti-receptor-binding domain (RBD) antibodies elicited upon natural infection and vaccination and showed that they recognize an immunogenic patch on the internal surface of the core RBD, which faces inwards and is hidden in the “down” state. These antibodies broadly neutralize wild type (Wuhan-Hu-1) SARS-CoV-2, Beta and Delta variants and some are effective against other sarbecoviruses. We observed a continuum of partially overlapping antibody epitopes from lower to upper part of the inner face of the RBD and some antibodies extend towards the receptor-binding motif. The majority of antibodies are substantially compromised by three mutational hotspots (S371L/F, S373P and S375F) in the lower part of the Omicron BA.1, BA.2 and BA.4/5 RBD. By contrast, antibody IY-2A induces a partial unfolding of this variable region and interacts with a conserved conformational epitope to tolerate all antigenic variations and neutralize diverse sarbecoviruses as well. This finding establishes that antibody recognition is not limited to the normal surface structures on the RBD. In conclusion, the delineation of functionally and structurally conserved RBD epitopes highlights potential vaccine and therapeutic candidates for COVID-19.