Information on firms' sustainability often partly resides in unstructured data published, for instance, in annual reports, news, and transcripts of earnings calls. In recent years, researchers and practitioners have started to extract information from these data sources using a broad range of natural language processing (NLP) methods. While there is much to be gained from these endeavors, studies that employ these methods rarely reflect upon the validity and quality of the chosen method—that is, how adequately NLP captures the sustainability information from text. This practice is problematic, as different NLP techniques lead to different results regarding the extraction of information. Hence, the choice of method may affect the outcome of the application and thus the inferences that users draw from their results. In this study, we examine how different types of NLP methods influence the validity and quality of extracted information. In particular, we compare four primary methods, namely (1) dictionary-based techniques, (2) topic modeling approaches, (3) word embeddings, and (4) large language models such as BERT and ChatGPT, and evaluate them on 75,000 manually labeled sentences from 10-K annual reports that serve as the ground truth. Our results show that dictionaries have a large variation in quality, topic models outperform other approaches that do not rely on large language models, and large language models show the strongest performance. In large language models, individual fine-tuning remains crucial. One-shot approaches (i.e., ChatGPT) have lately surpassed earlier approaches when using well-designed prompts and the most recent models.

中文翻译：

---

选择文本分析工具对企业报告中的可持续发展信息进行分类


有关公司可持续性的信息通常部分存在于发布的非结构化数据中，例如年度报告、新闻和财报电话会议记录中。近年来，研究人员和从业者开始使用广泛的自然语言处理（NLP）方法从这些数据源中提取信息。虽然从这些努力中可以获得很多收获，但采用这些方法的研究很少反思所选方法的有效性和质量，即 NLP 如何充分地从文本中捕获可持续性信息。这种做法是有问题的，因为不同的 NLP 技术会导致信息提取的不同结果。因此，方法的选择可能会影响应用程序的结果，从而影响用户从结果中得出的推论。在本研究中，我们研究了不同类型的 NLP 方法如何影响提取信息的有效性和质量。特别是，我们比较了四种主要方法，即 (1) 基于字典的技术、(2) 主题建模方法、(3) 词嵌入和 (4) 大型语言模型（例如 BERT 和 ChatGPT），并在 75,000 个手动模型上对它们进行评估10-K 年度报告中标记的句子作为基本事实。我们的结果表明，词典的质量差异很大，主题模型优于其他不依赖大型语言模型的方法，并且大型语言模型表现出最强的性能。在大型语言模型中，个体微调仍然至关重要。在使用精心设计的提示和最新模型时，一次性方法（即 ChatGPT）最近已经超越了早期的方法。