ABSTRACT

In this work, we evaluate different instruction strategies to improve the quality of crowdcoding for the concept of civility. We test the effectiveness of training, codebooks, and their combination through 2 × 2 experiments conducted on two different populations – students and Amazon Mechanical Turk workers. In addition, we perform simulations to evaluate the trade-off between cost and performance associated with different instructional strategies and the number of human coders. We find that training improves crowdcoding quality, while codebooks do not. We further show that relying on several human coders and applying majority rule to their assessments significantly improves performance.

摘要

在这项工作中，我们评估了不同的教学策略，以提高文明概念的众包编码质量。我们通过在两个不同人群（学生和 Amazon Mechanical Turk 工人）上进行的 2 × 2 实验来测试培训、密码本及其组合的有效性。此外，我们进行模拟以评估与不同教学策略和人类编码员数量相关的成本和性能之间的权衡。我们发现训练提高了众包编码的质量，而码本则没有。我们进一步表明，依靠几个人工编码器并将多数规则应用于他们的评估显着提高了性能。