Live streaming commerce thrives with a rich tapestry of multimodal information that intertwines with various entities, including the anchor, the commodities, and the live streaming environment. Despite the wealth of data at hand, the synthesis and analysis of this information to predict sales remains a significant challenge. This study introduces a framework for multi-entity multimodal fusion, which is characterized by the effective synthesis of multimodal data and its prioritization of entity-level fusion, thereby providing a comprehensive feature representation for improving predictive performance. In addressing the multimodal data associated with a diverse range of products, our framework improves the Transformer architecture to initially capture the intra-product modal features and subsequently integrate the inter-product features. Data experiments are conducted on a real-world dataset from Taobao Live. The framework outperforms both traditional machine learning methods and state-of-the-art multimodal fusion methods, which affirms its value as a robust decision-support tool for sales prediction, enabling more accurate pre-event predictions and strategic planning. We also examine the impact of different types of information in accurate sales prediction. It is found that harnessing a comprehensive suite of data leads to optimal performance across all evaluation metrics. Commodity-related data is primary factor in determining the prediction accuracy, followed by video data and streaming room-related data, providing insights regarding the resource allocation for collecting and analyzing multimodal data from live streaming platforms.

中文翻译：

---

MEMF：直播商务中销售预测的多实体多模态融合框架


直播商务凭借丰富的多模式信息蓬勃发展，这些信息与主播、商品和直播环境等各种实体交织在一起。尽管手头有大量数据，但综合和分析这些信息来预测销售仍然是一个重大挑战。本研究引入了一种多实体多模态融合框架，其特点是有效合成多模态数据并优先考虑实体级融合，从而提供全面的特征表示以提高预测性能。在处理与各种产品相关的多模态数据时，我们的框架改进了 Transformer 架构，以首先捕获产品内模态特征，然后集成产品间特征。数据实验是在淘宝直播的真实数据集上进行的。该框架优于传统的机器学习方法和最先进的多模态融合方法，这证实了其作为销售预测的强大决策支持工具的价值，从而实现更准确的事前预测和战略规划。我们还研究了不同类型的信息对准确销售预测的影响。研究发现，利用一套全面的数据可以在所有评估指标上实现最佳性能。商品相关数据是决定预测准确性的主要因素，其次是视频数据和直播间相关数据，为从直播平台收集和分析多模态数据的资源分配提供见解。