Accurate crop yield prediction is of utmost importance for informed decision-making in agriculture, aiding farmers, industry stakeholders, and policymakers in optimizing agricultural practices. However, this task is complex and depends on multiple factors, such as environmental conditions, soil properties, and management practices. Leveraging Remote Sensing (RS) technologies, multi-modal data from diverse global data sources can be collected to enhance predictive model accuracy. However, combining heterogeneous RS data poses a fusion challenge, like identifying the specific contribution of each modality in the predictive task. In this paper, we present a novel multi-modal learning approach to predict crop yield for different crops (soybean, wheat, rapeseed) and regions (Argentina, Uruguay, and Germany). Our multi-modal input data includes multi-spectral optical images from Sentinel-2 satellites and weather data as dynamic features during the crop growing season, complemented by static features like soil properties and topographic information. To effectively fuse the multi-modal data, we introduce a Multi-modal Gated Fusion (MMGF) model, comprising dedicated modality-encoders and a Gated Unit (GU) module. The modality-encoders handle the heterogeneity of data sources with varying temporal resolutions by learning a modality-specific representation. These representations are adaptively fused via a weighted sum. The fusion weights are computed for each sample by the GU using a concatenation of the multi-modal representations. The MMGF model is trained at sub-field level with 10 m resolution pixels. Our evaluations show that the MMGF outperforms conventional models on the same task, achieving the best results by incorporating all the data sources, unlike the usual fusion results in the literature. For Argentina, the MMGF model achieves an R2 value of 0.68 at sub-field yield prediction, while at the field level evaluation (comparing field averages), it reaches around 0.80 across different countries. The GU module learned different weights based on the country and crop-type, aligning with the variable significance of each data source to the prediction task. This novel method has proven its effectiveness in enhancing the accuracy of the challenging sub-field crop yield prediction. Our investigation indicates that the gated fusion approach promises a significant advancement in the field of agriculture and precision farming.

中文翻译：

---

多模态遥感数据自适应融合


准确的作物产量预测对于农业的明智决策至关重要，可帮助农民、行业利益相关者和政策制定者优化农业实践。然而，这项任务很复杂，取决于多种因素，例如环境条件、土壤特性和管理实践。利用遥感 （RS） 技术，可以收集来自不同全球数据源的多模态数据，以提高预测模型的准确性。然而，结合异构 RS 数据带来了融合挑战，例如确定预测任务中每种模态的具体贡献。在本文中，我们提出了一种新的多模态学习方法来预测不同作物（大豆、小麦、油菜籽）和地区（阿根廷、乌拉圭和德国）的作物产量。我们的多模态输入数据包括来自 Sentinel-2 卫星的多光谱光学图像和天气数据，作为作物生长季节的动态特征，并辅以土壤特性和地形信息等静态特征。为了有效地融合多模态数据，我们引入了一个多模态门控融合 （MMGF） 模型，包括专用模态编码器和一个门控单元 （GU） 模块。模态编码器通过学习特定于模态的表示来处理具有不同时间分辨率的数据源的异构性。这些表示通过加权和自适应融合。GU 使用多模态表示的串联计算每个样本的融合权重。MMGF 模型在子场级别以 10 m 分辨率像素进行训练。 我们的评估表明，MMGF 在同一任务上优于传统模型，通过整合所有数据源获得最佳结果，这与文献中通常的融合结果不同。对于阿根廷，MMGF 模型在子田产量预测中实现了 0.68 的 R2 值，而在田地层面的评估（比较田间平均值）中，不同国家的 R2 值达到 0.80 左右。GU 模块根据国家和作物类型学习了不同的权重，这与每个数据源对预测任务的变量显著性保持一致。这种新方法已经证明了它在提高具有挑战性的子田作物产量预测的准确性方面的有效性。我们的研究表明，门控融合方法有望在农业和精准农业领域取得重大进步。