The application of multimodal deep learning for emergency response and recovery, specifically in disaster social media analysis, is of utmost importance. It is worth noting that in real-world scenarios, sudden disaster events may differ from the training data, which may require the multimodal network to predict them as unknown classes instead of misclassifying them to known ones. Previous studies have primarily focused on model accuracy in a closed environment and may not be able to directly detect unknown classes. Thus, we propose a novel multimodal model for categorizing social media related to disasters in an open-world environment. Our methodology entails utilizing pre-trained unimodal models as encoders for each modality and performing information fusion with a cross-attention module to obtain the joint representation. For open-world detection, we use a multitask classifier that encompasses both a closed-world and an open-world classifier. The closed-world classifier is trained on the original data to classify known classes, whereas the open-world classifier is used to determine whether the input belongs to a known class. Furthermore, we propose a sample generation strategy that models the distribution of unknown samples using known data, which allows the open-world classifier to identify unknown samples. Our experiments were conducted on two public datasets, namely CrisisMMD and MHII. According to the experimental results, the proposed method outperforms other baselines and approaches in crisis information classification.

将多模态深度学习应用于应急响应和恢复，特别是在灾难社交媒体分析中，这一点至关重要。值得注意的是，在实际场景中，突发灾害事件可能与训练数据不同，这可能需要多模态网络将其预测为未知类别，而不是将它们错误分类为已知类别。以前的研究主要集中在封闭环境中的模型准确性，可能无法直接检测未知类别。因此，我们提出了一种新的多模态模型，用于在开放世界环境中对与灾难相关的社交媒体进行分类。我们的方法需要利用预先训练的单峰模型作为每种模态的编码器，并与交叉注意力模块进行信息融合以获得联合表示。对于开放世界检测，我们使用一个多任务分类器，它同时包含一个封闭世界和一个开放世界分类器。封闭世界分类器在原始数据上进行训练以对已知类进行分类，而开放世界分类器用于确定输入是否属于已知类。此外，我们提出了一种样本生成策略，该策略使用已知数据对未知样本的分布进行建模，从而允许开放世界分类器识别未知样本。我们的实验是在两个公共数据集上进行的，即 CrisisMMD 和 MHII。根据实验结果，所提出的方法在危机信息分类方面优于其他基线和方法。