OBJECTIVE To explore how the commercially available large language model (LLM) GPT-4 compares to endocrinologists when addressing medical questions when there is uncertainty regarding the best answer. RESEARCH DESIGN AND METHODS This study compared responses from GPT-4 to responses from 31 endocrinologists using hypothetical clinical vignettes focused on diabetes, specifically examining the prescription of metformin versus alternative treatments. The primary outcome was the choice between metformin and other treatments. RESULTS With a simple prompt, GPT-4 chose metformin in 12% (95% CI 7.9–17%) of responses, compared with 31% (95% CI 23–39%) of endocrinologist responses. After modifying the prompt to encourage metformin use, the selection of metformin by GPT-4 increased to 25% (95% CI 22–28%). GPT-4 rarely selected metformin in patients with impaired kidney function, or a history of gastrointestinal distress (2.9% of responses, 95% CI 1.4–5.5%). In contrast, endocrinologists often prescribed metformin even in patients with a history of gastrointestinal distress (21% of responses, 95% CI 12–36%). GPT-4 responses showed low variability on repeated runs except at intermediate levels of kidney function. CONCLUSIONS In clinical scenarios with no single right answer, GPT-4’s responses were reasonable, but differed from endocrinologists’ responses in clinically important ways. Value judgments are needed to determine when these differences should be addressed by adjusting the model. We recommend against reliance on LLM output until it is shown to align not just with clinical guidelines but also with patient and clinician preferences, or it demonstrates improvement in clinical outcomes over standard of care.

中文翻译：

---

大语言模型 GPT-4 与临床不确定条件下内分泌科医生对初始选择抗糖尿病药物的反应进行比较


目的 探讨在最佳答案存在不确定性的情况下，商业化大语言模型 ( LLM ) GPT-4 与内分泌学家在解决医学问题时的比较。研究设计和方法 本研究使用针对糖尿病的假设临床案例，比较了 GPT-4 的反应与 31 名内分泌科医生的反应，特别检查了二甲双胍处方与替代治疗的比较。主要结果是二甲双胍和其他治疗之间的选择。结果 通过简单的提示，GPT-4 在 12%（95% CI 7.9-17%）的答复中选择二甲双胍，而内分泌科医生的答复中选择二甲双胍的比例为 31%（95% CI 23-39%）。修改提示以鼓励使用二甲双胍后，GPT-4 对二甲双胍的选择增加至 25%（95% CI 22-28%）。 GPT-4 很少在肾功能受损或有胃肠道不适病史的患者中选择二甲双胍（2.9% 的缓解，95% CI 1.4–5.5%）。相比之下，内分泌科医生经常给有胃肠道不适病史的患者开二甲双胍（21% 的缓解，95% CI 12-36%）。除肾功能处于中等水平外，GPT-4 反应在重复运行中显示出较低的变异性。结论 在没有单一正确答案的临床情况下，GPT-4 的反应是合理的，但在临床重要方面与内分泌科医生的反应不同。需要进行价值判断来确定何时应通过调整模型来解决这些差异。我们建议不要依赖LLM输出，除非它被证明不仅符合临床指南，而且符合患者和临床医生的偏好，或者它证明临床结果比护理标准有所改善。