Deep learning-based video denoising methods have achieved great performance improvements in recent years. However, the expensive computational cost arising from sophisticated network design has severely limited their applications in real-world scenarios. To address this practical weakness, we propose a multiscale spatio-temporal memory network for fast video denoising, named MSTMN, aiming at striking an improved trade-off between cost and performance. To develop an efficient and effective algorithm for video denoising, we exploit a multiscale representation based on the Gaussian-Laplacian pyramid decomposition so that the reference frame can be restored in a coarse-to-fine manner. Guided by a model-based optimization approach, we design an effective variance estimation module, an alignment error estimation module and an adaptive fusion module for each scale of the pyramid representation. For the fusion module, we employ a reconstruction recurrence strategy to incorporate local temporal information. Moreover, we propose a memory enhancement module to exploit the global spatio-temporal information. Meanwhile, the similarity computation of the spatio-temporal memory network enables the proposed network to adaptively search the valuable information at the patch level, which avoids computationally expensive motion estimation and compensation operations. Experimental results on real-world raw video datasets have demonstrated that the proposed lightweight network outperforms current state-of-the-art fast video denoising algorithms such as FastDVDnet, EMVD, and ReMoNet with fewer computational costs.

中文翻译：

---

用于轻量级视频去噪的多尺度时空记忆网络


近年来，基于深度学习的视频降噪方法取得了巨大的性能提升。然而，复杂的网络设计产生的昂贵计算成本严重限制了它们在实际场景中的应用。为了解决这一实际弱点，我们提出了一种用于快速视频去噪的多尺度时空记忆网络，名为 MSTMN，旨在改进成本和性能之间的权衡。为了开发一种高效的视频去噪算法，我们利用了基于高斯-拉普拉斯金字塔分解的多尺度表示，以便可以以粗到细的方式恢复参考帧。在基于模型的优化方法的指导下，我们为金字塔表示的每个尺度设计了一个有效的方差估计模块、一个对齐误差估计模块和一个自适应融合模块。对于融合模块，我们采用重建递归策略来整合局部时间信息。此外，我们提出了一个内存增强模块来利用全局时空信息。同时，时空记忆网络的相似性计算使所提出的网络能够在补丁级别自适应地搜索有价值的信息，从而避免了计算成本高昂的运动估计和补偿操作。在真实世界原始视频数据集上的实验结果表明，所提出的轻量级网络优于当前最先进的快速视频去噪算法，如 FastDVDnet、EMVD 和 ReMoNet，而且计算成本更低。