KorNAT：韩国社会价值观和常识的法学硕士一致性基准,arXiv - CS - Computation and Language

当前位置： X-MOL 学术 › arXiv.cs.CL › 论文详情

Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)

KorNAT：韩国社会价值观和常识的法学硕士一致性基准
arXiv - CS - Computation and Language Pub Date : 2024-02-21 , DOI: arxiv-2402.13605
Jiyoung Lee, Minwoo Kim, Seungho Kim, Junghwan Kim, Seunghyun Won, Hwaran Lee, Edward Choi

为了在特定国家有效部署大型语言模型（LLM），他们必须了解该国家的文化和基础知识。为此，我们引入国家一致性，从社会价值一致性和常识一致性两个方面衡量LLM与目标国家之间的一致性。社会价值一致性评估模型对特定国家社会价值观的理解程度，而常识一致性则检查模型捕获与国家相关的基本知识的程度。我们构建了 KorNAT，这是衡量国家与韩国一致性的第一个基准。对于社会价值数据集，我们从一项涉及 6,174 名韩国参与者的大规模调查中获得了真实标签。对于常识数据集，我们根据韩国教科书和GED参考资料构建了样本。KorNAT 包含分别针对社会价值和常识的 4K 和 6K 多项选择题。我们的数据集创建过程经过精心设计，基于统计抽样理论，并通过多轮人工审核进行完善。七个法学硕士的实验结果表明，只有少数模型符合我们的参考分数，表明有进一步增强的潜力。KorNAT 在通过了专门评估数据集质量的政府附属组织进行的评估后获得了政府批准。我们数据集的样本和详细评估协议可以在 \url{https://selectstar.ai/ko/papers-national-alignment#} 中找到

"点击查看英文标题和摘要"

KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge

For Large Language Models (LLMs) to be effectively deployed in a specific country, they must possess an understanding of the nation's culture and basic knowledge. To this end, we introduce National Alignment, which measures an alignment between an LLM and a targeted country from two aspects: social value alignment and common knowledge alignment. Social value alignment evaluates how well the model understands nation-specific social values, while common knowledge alignment examines how well the model captures basic knowledge related to the nation. We constructed KorNAT, the first benchmark that measures national alignment with South Korea. For the social value dataset, we obtained ground truth labels from a large-scale survey involving 6,174 unique Korean participants. For the common knowledge dataset, we constructed samples based on Korean textbooks and GED reference materials. KorNAT contains 4K and 6K multiple-choice questions for social value and common knowledge, respectively. Our dataset creation process is meticulously designed and based on statistical sampling theory and was refined through multiple rounds of human review. The experiment results of seven LLMs reveal that only a few models met our reference score, indicating a potential for further enhancement. KorNAT has received government approval after passing an assessment conducted by a government-affiliated organization dedicated to evaluating dataset quality. Samples and detailed evaluation protocols of our dataset can be found in \url{https://selectstar.ai/ko/papers-national-alignment#}

更新日期：2024-02-22

点击分享查看原文

点击收藏

阅读更多本刊新发论文