ABSTRACT

The empirical identification of frames drawing on automated text analysis has been discussed intensely with regard to the validity of measurements. Adding to an evolving discussion on automated frame identification, we systematically contrast different machine-learning approaches with a manually coded gold standard to shed light on the implications of using one or the other: (1) topic modeling, (2) keyword-assisted topic modeling (keyATM), and (3) supervised machine learning as three popular and/or promising approaches. Manual coding is based on the Policy Frames codebook, providing an established base that allows future research to dovetail our contribution. Analysing a large dataset of 12 Austrian newspapers’ EU coverage over 11 years (2009–2019), we contribute to addressing the methodological challenges that have emerged for social scientists interested in employing automated tools for frame analysis. While results confirm the superiority of supervised machine-learning, the semi-supervised approach (keyATM) seems unfit for frame analysis, whereas the topic model covers the middle ground. Results are extensively discussed regarding their implications for the validity of approaches.

摘要

关于测量的有效性，人们对基于自动文本分析的框架的经验识别进行了激烈的讨论。除了关于自动框架识别的不断发展的讨论之外，我们系统地将不同的机器学习方法与手动编码的黄金标准进行比较，以阐明使用其中一种或另一种的含义：(1) 主题建模，(2) 关键字辅助主题建模（keyATM），（3）监督机器学习作为三种流行和/或有前途的方法。手动编码基于政策框架密码本，提供了一个既定的基础，使未来的研究能够与我们的贡献相吻合。通过分析 12 家奥地利报纸在 11 年来（2009-2019 年）欧盟报道的大型数据集，我们有助于解决对使用自动化工具进行框架分析感兴趣的社会科学家所面临的方法论挑战。虽然结果证实了监督机器学习的优越性，但半监督方法（keyATM）似乎不适合框架分析，而主题模型则涵盖了中间立场。人们广泛讨论了结果对方法有效性的影响。