Botulinum neurotoxin E (BoNT/E) is one of the major causes of human botulism and paradoxically also a promising therapeutic agent. Here we determined the co-crystal structures of the receptor-binding domain of BoNT/E (H_CE) in complex with its neuronal receptor synaptic vesicle glycoprotein 2A (SV2A) and a nanobody that serves as a ganglioside surrogate. These structures reveal that the protein-protein interactions between H_CE and SV2 provide the crucial location and specificity information for H_CE to recognize SV2A and SV2B, but not the closely related SV2C. At the same time, H_CE exploits a separated sialic acid-binding pocket to mediate recognition of an N-glycan of SV2. Structure-based mutagenesis and functional studies demonstrate that both the protein-protein and protein-glycan associations are essential for SV2A-mediated cell entry of BoNT/E and for its potent neurotoxicity. Our studies establish the structural basis to understand the receptor-specificity of BoNT/E and to engineer BoNT/E variants for new clinical applications.