ABSTRACT

We present the Conversation Analysis Modeling Schema (CAMS), a novel dialogue labeling schema that combines the Conversation Analysis concept of Adjacency Pairs, with Dialogue Acts. The aim is to capture both the semantic and syntactic structure of dialogue, in a format that is independent of the domain or topic, and which facilitates the computational modeling of dialogue. A labeling task undertaken by novice annotators is used to evaluate its efficacy on a selection of task-oriented and non-task-oriented dialogs, and to measure inter-annotator agreement. To deepen the “human-factors” analysis we also record and examine users’ self-reported confidence scores and average utterance annotation times. Inter-annotator agreement is shown to be higher for task-oriented dialogs than non-task-oriented, though the structure of the dialogue itself has a more significant impact. We further examine the assumptions around expected agreement for two weighted agreement coefficients, Alpha and Beta, and show that annotators assign labels using similar probability distributions, small variations can result in large differences in agreement values between biased and unbiased measures.

摘要

我们提出了对话分析建模模式 (CAMS)，这是一种新颖的对话标记模式，它将邻接对的对话分析概念与对话行为相结合。目的是以独立于领域或主题的格式捕获对话的语义和句法结构，并有助于对话的计算建模。由新手注释者承担的标记任务用于评估其在选择面向任务和非面向任务的对话中的功效，并测量注释者之间的一致性。为了加深“人为因素”分析，我们还记录和检查用户自我报告的置信度分数和平均话语注释时间。面向任务的对话显示出比非面向任务的对话更高的注释者间一致性，尽管对话的结构本身具有更显着的影响。我们进一步检查了有关两个加权一致性系数 Alpha 和 Beta 的预期一致性的假设，并表明注释者使用相似的概率分布分配标签，小的变化可能导致有偏和无偏测量之间的一致性值存在很大差异。