Development and evaluation of a large language model of ophthalmology in Chinese,British Journal of Ophthalmology

当前位置： X-MOL 学术 › Br. J. Ophthalmol. › 论文详情

Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)

Development and evaluation of a large language model of ophthalmology in Chinese
British Journal of Ophthalmology ( IF 3.7 ) Pub Date : 2024-10-01 , DOI: 10.1136/bjo-2023-324526
Ce Zheng _{1,

2} , Hongfei Ye _{1,

2} , Jinming Guo ₃ , Junrui Yang ₄ , Ping Fei ₁ , Yuanzhi Yuan ₅ , Danqing Huang ₆ , Yuqiang Huang ₃ , Jie Peng ₇ , Xiaoling Xie ₃ , Meng Xie ₁ , Peiquan Zhao ₁ , Li Chen ₈ , Mingzhi Zhang ₉

Affiliation

Ophthalmology, Xinhua Hospital Affiliated to Shanghai Jiaotong University School of Medicine, Shanghai, China.
Institute of Hospital Development Strategy, China Hospital Development Institute, Shanghai Jiao Tong University, Shanghai, China.
Joint Shantou International Eye Center of Shantou University and The Chinese University of Hong Kong, Shantou, Guangdong, China.
Ophthalmology, The 74th Army Group Hospital, Guangzhou, Guangdong, China.
Ophthalmology, Zhongshan Hospital Fudan University, Shanghai, China.
Discipline Inspection & Supervision Office, Xinhua Hospital Affiliated to Shanghai Jiaotong University School of Medicine, Shanghai, China.
Opthalmology, Xinhua Hospital Affiliated to Shanghai Jiaotong University School of Medicine, Shanghai, China.
Ophthalmology, Xinhua Hospital Affiliated to Shanghai Jiaotong University School of Medicine, Shanghai, China
Joint Shantou International Eye Center of Shantou University and The Chinese University of Hong Kong, Shantou, Guangdong, China

Background Large language models (LLMs), such as ChatGPT, have considerable implications for various medical applications. However, ChatGPT’s training primarily draws from English-centric internet data and is not tailored explicitly to the medical domain. Thus, an ophthalmic LLM in Chinese is clinically essential for both healthcare providers and patients in mainland China. Methods We developed an LLM of ophthalmology (MOPH) using Chinese corpora and evaluated its performance in three clinical scenarios: ophthalmic board exams in Chinese, answering evidence-based medicine-oriented ophthalmic questions and diagnostic accuracy for clinical vignettes. Additionally, we compared MOPH’s performance to that of human doctors. Results In the ophthalmic exam, MOPH’s average score closely aligned with the mean score of trainees (64.7 (range 62–68) vs 66.2 (range 50–92), p=0.817), but achieving a score above 60 in all seven mock exams. In answering ophthalmic questions, MOPH demonstrated an adherence of 83.3% (25/30) of responses following Chinese guidelines (Likert scale 4–5). Only 6.7% (2/30, Likert scale 1–2) and 10% (3/30, Likert scale 3) of responses were rated as ‘poor or very poor’ or ‘potentially misinterpretable inaccuracies’ by reviewers. In diagnostic accuracy, although the rate of correct diagnosis by ophthalmologists was superior to that by MOPH (96.1% vs 81.1%, p>0.05), the difference was not statistically significant. Conclusion This study demonstrated the promising performance of MOPH, a Chinese-specific ophthalmic LLM, in diverse clinical scenarios. MOPH has potential real-world applications in Chinese-language ophthalmology settings. Data are available upon reasonable request. Data are available on reasonable request.

更新日期：2024-09-20

点击分享查看原文

点击收藏

阅读更多本刊新发论文本刊介绍/投稿指南