Large language models (LLMs) exhibit expert-level performance in tasks across a wide range of different domains. Ethical issues raised by LLMs and the need to align future versions makes it important to know how state of the art models reason about moral and legal issues. In this paper, we employ the methods of experimental psychology to probe into this question. We replicate eight studies from the experimental literature with instances of Google's Gemini Pro, Anthropic's Claude 2.1, OpenAI's GPT-4, and Meta's Llama 2 Chat 70b. We find that alignment with human responses shifts from one experiment to another, and that models differ amongst themselves as to their overall alignment, with GPT-4 taking a clear lead over all other models we tested. Nonetheless, even when LLM-generated responses are highly correlated to human responses, there are still systematic differences, with a tendency for models to exaggerate effects that are present among humans, in part by reducing variance. This recommends caution with regards to proposals of replacing human participants with current state-of-the-art LLMs in psychological research and highlights the need for further research about the distinctive aspects of machine psychology.

中文翻译：

---

探索LLMs道德和法律推理的心理学


大型语言模型 (LLMs) 在跨广泛不同领域的任务中表现出专家级的性能。 LLMs 提出的道德问题以及协调未来版本的需要使得了解最先进的模型如何推理道德和法律问题变得非常重要。本文运用实验心理学的方法来探讨这个问题。我们使用 Google 的 Gemini Pro、Anthropic 的 Claude 2.1、OpenAI 的 GPT-4 和 Meta 的 Llama 2 Chat 70b 的实例复制了实验文献中的八项研究。我们发现，与人类反应的一致性从一个实验转移到另一个实验，并且模型之间的整体一致性有所不同，GPT-4 明显领先于我们测试的所有其他模型。尽管如此，即使LLM生成的反应与人类反应高度相关，仍然存在系统性差异，模型倾向于夸大人类中存在的影响，部分是通过减少方差来实现的。这建议谨慎对待用心理学研究中当前最先进的LLMs取代人类参与者的提议，并强调需要对机器心理学的独特方面进行进一步研究。