This paper strives to recognize activities in the dark, as well as in the day. We first establish that state-of-the-art activity recognizers are effective during the day, but not trustworthy in the dark. The main causes are the limited availability of labeled dark videos to learn from, as well as the distribution shift towards the lower color contrast at test-time. To compensate for the lack of labeled dark videos, we introduce a pseudo-supervised learning scheme, which utilizes easy to obtain unlabeled and task-irrelevant dark videos to improve an activity recognizer in low light. As the lower color contrast results in visual information loss, we further propose to incorporate the complementary activity information within audio, which is invariant to illumination. Since the usefulness of audio and visual features differs depending on the amount of illumination, we introduce our ‘darkness-adaptive’ audio-visual recognizer. Experiments on EPIC-Kitchens, Kinetics-Sound, and Charades demonstrate our proposals are superior to image enhancement, domain adaptation and alternative audio-visual fusion methods, and can even improve robustness to local darkness caused by occlusions. Project page: https://xiaobai1217.github.io/Day2Dark/.

本文致力于识别黑暗和白天的活动。我们首先确定最先进的活动识别器在白天有效，但在黑暗中不值得信赖。主要原因是可供学习的标记暗视频数量有限，以及在测试时分布向较低的颜色对比度转变。为了弥补标记暗视频的不足，我们引入了一种伪监督学习方案，该方案利用易于获取的未标记和与任务无关的暗视频来提高弱光下的活动识别器。由于较低的颜色对比度会导致视觉信息丢失，我们进一步建议将互补的活动信息合并到音频中，这与照明无关。由于音频和视频功能的有用性因照明量而异，因此我们推出了我们的“暗度自适应”视听识别器。在 EPIC-Kitchens、Kinetics-Sound 和 Charades 上的实验表明，我们的建议优于图像增强、域自适应和替代视听融合方法，甚至可以提高对遮挡引起的局部黑暗的鲁棒性。项目页面：https://xiaobai1217.github.io/Day2Dark/。