ABSTRACT

Incivility in online discussions has become an important issue in political communication research. Instruments and tools for the automated analysis of uncivil content, however, are rare, especially for non-English user-generated text. In this study, we present a) an extensive dictionary (DIKI - Diktionär für Inzivilität, English: Dictionary for Incivility) to detect incivility in German-language online discussions, and b) a semi-automated two-step-approach that combines manual content analysis with automated keyword collection using a pre-trained word embedding model. We show that DIKI clearly outperforms comparable dictionaries that have been used as alternative instruments to measure incivility (e.g., the LIWC) as well as basic machine learning approaches to text classification. Further, we provide evidence that pre-trained word embeddings can fruitfully be employed in the explorative phase of creating dictionaries. Still, the manual evaluation of DIKI confirms that detecting complex and context-dependent forms of incivility remains challenging and constant update would be needed to maintain performance. Finally, the detailed documentation of the developing and evaluation process of DIKI may serve as a guideline for further research. We therefore provide DIKI as a freely available instrument that also will be applicable in a web interface for drag-and-drop data analysis (diki.limitedminds.org).

摘要

网络讨论中的不礼貌行为已成为政治传播研究中的一个重要问题。然而，用于自动分析不文明内容的仪器和工具很少，特别是对于非英语用户生成的文本。在这项研究中，我们提出了 a) 一本内容广泛的词典 ( DIKI - Diktionär für Inzivilität，英语：不礼貌词典）检测德语在线讨论中的不礼貌行为，b）半自动两步方法，使用预先训练的词嵌入模型将手动内容分析与自动关键词收集相结合。我们表明，DIKI 明显优于用作衡量不文明行为替代工具的同类词典（例如 LIWC）以及文本分类的基本机器学习方法。此外，我们提供的证据表明，预先训练的词嵌入可以在创建词典的探索阶段卓有成效地使用。尽管如此，DIKI 的手动评估证实，检测复杂且依赖于上下文的不文明行为仍然具有挑战性，需要不断更新才能保持性能。最后，DIKI开发和评估过程的详细记录可以作为进一步研究的指南。因此，我们提供 DIKI 作为免费工具，该工具也适用于拖放数据分析的 Web 界面 (diki.limitedminds.org)。