Acute caffeine intake can attenuate homeostatic sleep pressure and worsen sleep quality. Caffeine intake—particularly in high doses and close to bedtime—may also affect circadian-regulated rapid eye movement (REM) sleep promotion, an important determinant of subjective sleep quality. However, it is not known whether such changes persist under chronic caffeine consumption during daytime. Twenty male caffeine consumers (26.4 ± 4 years old, habitual caffeine intake 478.1 ± 102.8 mg/day) participated in a double-blind crossover study. Each volunteer completed a caffeine (3 × 150 mg caffeine daily for 10 days), a withdrawal (3 × 150 mg caffeine for 8 days then placebo), and a placebo condition. After 10 days of controlled intake and a fixed sleep-wake cycle, we recorded electroencephalography for 8 h starting 5 h after habitual bedtime (i.e., start on average at 04:22 h which is around the peak of circadian REM sleep promotion). A 60-min evening nap preceded each sleep episode and reduced high sleep pressure levels. While total sleep time and sleep architecture did not significantly differ between the three conditions, REM sleep latency was longer after daily caffeine intake compared with both placebo and withdrawal. Moreover, the accumulation of REM sleep proportion was delayed, and volunteers reported more difficulties with awakening after sleep and feeling more tired upon wake-up in the caffeine condition compared with placebo. Our data indicate that besides acute intake, also regular daytime caffeine intake affects REM sleep regulation in men, such that it delays circadian REM sleep promotion when compared with placebo. Moreover, the observed caffeine-induced deterioration in the quality of awakening may suggest a potential motive to reinstate caffeine intake after sleep.