Recent advances in large language models are enabling the computational intelligent analysis of psychology in natural language. Here, the Fill-Mask Association Test (FMAT) is introduced as a novel and integrative method leveraging Masked Language Models to study and measure psychology from a propositional perspective at the societal level. The FMAT uses Bidirectional Encoder Representations from Transformers (BERT) models to compute semantic probabilities of option words filling in the masked blank of a designed query (i.e., a clozelike contextualized sentence). The current research presents 15 studies that establish the reliability and validity of the FMAT in predicting factual associations (Studies 1A-1C), measuring attitudes/biases (Studies 2A-2D), capturing social stereotypes (Studies 3A-3D), and retrospectively delineating lay perceptions of sociocultural changes over time (Studies 4A-4D). Empirically, the FMAT replicated seminal findings previously obtained with human participants (e.g., the Implicit Association Test) and other big-data text-analytic methods (e.g., word frequency analysis, the Word Embedding Association Test), demonstrating robustness across 12 BERT model variants and diverse training text corpora. Theoretically, the current findings substantiate the propositional (vs. associative) perspective on how semantic associations are represented in natural language. Methodologically, the FMAT allows for more fine-grained language-based psychological measurement, with an R package developed to streamline its workflow for use on broader research questions. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

中文翻译：

---

填充掩模关联测试 (FMAT)：测量自然语言中的命题。


大型语言模型的最新进展使得自然语言心理学的计算智能分析成为可能。在这里，引入填充掩模关联测试（FMAT）作为一种新颖的综合方法，利用掩模语言模型从社会层面的命题角度研究和测量心理学。 FMAT 使用来自 Transformers (BERT) 模型的双向编码器表示来计算填充设计查询的屏蔽空白的选项词的语义概率（即，完形填空的上下文化句子）。当前的研究提出了 15 项研究，这些研究确立了 FMAT 在预测事实关联（研究 1A-1C）、测量态度/偏见（研究 2A-2D）、捕捉社会刻板印象（研究 3A-3D）以及回顾性描述方面的可靠性和有效性。外行人对社会文化随时间变化的看法（研究 4A-4D）。根据经验，FMAT 复制了之前在人类参与者中获得的开创性发现（例如，隐式关联测试）和其他大数据文本分析方法（例如，词频分析、词嵌入关联测试），证明了 12 个 BERT 模型变体的稳健性和多样化的训练文本语料库。从理论上讲，当前的研究结果证实了关于如何在自然语言中表示语义关联的命题（与关联）观点。从方法上来说，FMAT 允许进行更细粒度的基于语言的心理测量，并开发了一个 R 包来简化其工作流程，以用于更广泛的研究问题。 （PsycInfo 数据库记录 (c) 2024 APA，保留所有权利）。