Formative verbal feedback during live surgery is essential for adjusting trainee behavior and accelerating skill acquisition. Despite its importance, understanding optimal feedback is challenging due to the difficulty of capturing and categorizing feedback at scale. We propose a Human-AI Collaborative Refinement Process that uses unsupervised machine learning (Topic Modeling) with human refinement to discover feedback categories from surgical transcripts. Our discovered categories are rated highly for clinical clarity and are relevant to practice, including topics like “Handling and Positioning of (tissue)” and “(Tissue) Layer Depth Assessment and Correction [during tissue dissection].” These AI-generated topics significantly enhance predictions of trainee behavioral change, providing insights beyond traditional manual categorization. For example, feedback on “Handling Bleeding” is linked to improved behavioral change. This work demonstrates the potential of AI to analyze surgical feedback at scale, informing better training guidelines and paving the way for automated feedback and cueing systems in surgery.

现场手术期间的形成性口头反馈对于调整受训者的行为和加速技能习得至关重要。尽管它很重要，但由于难以大规模捕获和分类反馈，因此了解最佳反馈具有挑战性。我们提出了一种 Human-AI Collaborative Refinement Process，该过程使用无监督机器学习（主题建模）和人工细化来发现手术成绩单中的反馈类别。我们发现的类别在临床清晰度方面受到高度评价，并且与实践相关，包括“（组织）的处理和定位”和“（组织）层深度评估和校正 [在组织解剖期间] ”等主题。这些 AI 生成的主题显著增强了对受训者行为变化的预测，提供了超越传统手动分类的见解。例如，对 “Handling Bleeding” 的反馈与改善的行为改变有关。这项工作展示了人工智能在大规模分析手术反馈方面的潜力，为更好的培训指南提供信息，并为手术中的自动反馈和提示系统铺平了道路。