The hypothalamus–pituitary–adrenal axis is the main physiological stress response system and regulating the release of cortisol. The two corticoid receptors encoded by the genes NR3C1 and NR3C2 are the main players in regulating the physiological response to cortisol. This biological system has been linked to neurocognitive processes and memory, yet the mechanisms remain largely unclear. In two independent general population studies (SHIP, total sample size > 5500), we aim to diseantangle the effects of genetic variation, gene expression and cortisol on verbal memory and memory associated brain volume. Especially for NR3C1 results exhibited a consistent pattern of direct an interactive effects. All three biological layers, genetic variation (rs56149945), gene expression for NR3C1 and cortisol levels, were directly associated with verbal memory. Interactions between these components showed significant effects on verbal memory as well as hippocampal volume. For NR3C2 such a complex association pattern could not be observed. Our analyses revealed that different components of the stress response system are acting together on different aspects of cognition. Complex phenotypes, such as cognition and memory function are regulated by a complex interplay between different genetic and epigenetic features. We promote the glucocorticoid receptor NR3C1 as a main target to focus in the context of verbal memory and provided a mechanistic concept of the interaction between various biological layers spanning NR3C1 function and its effects on memory. Especially the NR3C1 transcript seemed to be a key element in this complex system.