Online labor markets and the humans that power them serve a critical role in the advancement of artificial intelligence and supervised machine learning via the creation of useful training datasets. The use of human effort in online labor markets is not enough, however, as a key factor is understanding the possible interventions that market operators can leverage to incentivize human effort among their labor force. We propose that platforms could implement mechanisms such as rewards or punishments at individual or group levels to incentivize real-effort and output. We apply our interventions using a collaborative image tagging experiment—a folksonomy—and the results provide interesting insights and nonobvious consequences. On average, interventions applied at the group level outperformed interventions applied at the individual level. Punishing the group provided the most controversial incentive strategy and provided a nonobvious significant improvement in effort. Rewarding or sanctioning an individual had similar effects on average, with both treatments leading to significant increases in effort post-intervention. In contrast to predictions, sanctioning appears to have significantly motivated those that were punished. Overall, the interventions applied in our real-effort collaborative image tagging experiment had a significant impact on behavior, which provides guidance for online labor market operators and the use of incentives in the creation of labeled machine learning training datasets.

中文翻译：

---

在线劳动力市场中的真实努力激励：对个人和团体的惩罚和奖励

在线劳动力市场和为其提供动力的人类通过创建有用的训练数据集，在人工智能和监督机器学习的进步中发挥着关键作用。然而，在在线劳动力市场中仅仅利用人力是不够的，因为一个关键因素是了解市场运营商可以利用哪些可能的干预措施来激励劳动力中的人力。我们建议平台可以在个人或团体层面实施奖励或惩罚等机制，以激励真实的努力和产出。我们使用协作图像标记实验（一种民俗分类学）来应用我们的干预措施，结果提供了有趣的见解和不明显的后果。平均而言，团体层面的干预措施优于个人层面的干预措施。惩罚该群体提供了最具争议的激励策略，并在努力方面提供了非明显的显着改进。平均而言，奖励或制裁个人会产生类似的效果，两种治疗方法都会导致干预后的努力显着增加。与预测相反，制裁似乎极大地激励了那些受到惩罚的人。总体而言，在我们的真实努力协作图像标记实验中应用的干预措施对行为产生了重大影响，这为在线劳动力市场运营商以及在创建标记机器学习训练数据集时使用激励措施提供了指导。