Spectral imaging acquires more analyzable and distinguishable information than RGB imaging and has become an emerging technique powering Internet of Things (IoT). Thus, generating spectral images from existing RGB cameras is instrumental to high level vision tasks. In this paper, an IoT-oriented spectral reconstruction model with parallel fusion of Convolutional Neural Network (CNN) and Transformer (PFCT) is proposed to efficiently recover hyperspectral images (HSI) from RGB counterparts, facilitating low-cost and non-hardware-specific spectral image acquisition. Recent works mainly utilize CNNs to extract local features by stacking more layers, ignoring the latent correlations of global features. In our PFCT network, we take advantage of lightweight CNNs to efficiently perceive local details, and exploit Transformer blocks to fully capture the global context. Based on the architecture, a parallel fusion module is further designed to deeply interact and fuse the features obtained by CNN and Transformer in both directions. The final output spectral image is generated with same spatial sizes of the original image based on the deeply fused features. Consequently, the proposed PFCT network achieved high performance on four benchmark datasets compared to several state-of-the-art networks with a relatively small number of parameters.

中文翻译：

---

基于CNN和Transformer并行融合的物联网谱重构


光谱成像比 RGB 成像获取更多可分析和可区分的信息，已成为支持物联网 (IoT) 的新兴技术。因此，从现有 RGB 相机生成光谱图像对于高级视觉任务很有帮助。本文提出了一种面向物联网的光谱重建模型，并行融合卷积神经网络（CNN）和变换器（PFCT），以有效地从 RGB 对应物中恢复高光谱图像（HSI），从而促进低成本和非硬件特定的光谱图像采集。最近的工作主要利用 CNN 通过堆叠更多层来提取局部特征，忽略全局特征的潜在相关性。在我们的 PFCT 网络中，我们利用轻量级 CNN 来有效感知局部细节，并利用 Transformer 块来充分捕获全局上下文。在此架构的基础上，进一步设计了并行融合模块，对CNN和Transformer获得的特征进行双向深度交互和融合。基于深度融合的特征，生成具有与原始图像相同的空间尺寸的最终输出光谱图像。因此，与参数数量相对较少的几个最先进的网络相比，所提出的 PFCT 网络在四个基准数据集上实现了高性能。