During wakefulness, the VTA represents the valence of experiences and mediates affective response to the outside world. Recent work revealed that two major VTA populations – dopamine and GABA neurons – are highly active during REM sleep and less active during NREM sleep. Using long-term cell type and brain state-specific recordings, machine learning, and optogenetics, we examined the role that the sleep-activity of these neurons plays in subsequent awake behavior. We found that VTA activity during NREM (but not REM) sleep correlated with exploratory features of the next day’s behavior. Disrupting natural VTA activity during NREM (but not REM) sleep reduced future tendency to explore and increased preferences for familiarity and goal-directed actions, with no direct effect on learning or memory. Our data suggest that, during deep sleep, VTA neurons engage in offline processing, consolidating not memories but affective responses to remembered environments, shaping the way that animals respond to future experiences.