The prediction of chemical reaction pathways has been accelerated by the development of novel machine learning architectures based on the deep learning paradigm. In this context, deep neural networks initially designed for language translation have been used to accurately predict a wide range of chemical reactions. Among models suited for the task of language translation, the recently introduced molecular transformer reached impressive performance in terms of forward-synthesis and retrosynthesis predictions. In this study, we first present an analysis of the performance of transformer models for product, reactant, and reagent prediction tasks under different scenarios of data availability and data augmentation. We find that the impact of data augmentation depends on the prediction task and on the metric used to evaluate the model performance. Second, we probe the contribution of different combinations of input formats, tokenization schemes, and embedding strategies to model performance. We find that less stable input settings generally lead to better performance. Lastly, we validate the superiority of round-trip accuracy over simpler evaluation metrics, such as top-k accuracy, using a committee of human experts and show a strong agreement for predictions that pass the round-trip test. This demonstrates the usefulness of more elaborate metrics in complex predictive scenarios and highlights the limitations of direct comparisons to a predefined database, which may include a limited number of chemical reaction pathways.

中文翻译：

---

化学反应的变压器性能：不同预测和评估场景的分析

基于深度学习范式的新型机器学习架构的发展加速了化学反应途径的预测。在这种情况下，最初为语言翻译设计的深度神经网络已被用于准确预测范围广泛的化学反应。在适合语言翻译任务的模型中，最近推出的分子转换器在正向合成和逆向合成预测方面取得了令人印象深刻的表现。在这项研究中，我们首先分析了在数据可用性和数据增强的不同场景下，用于产品、反应物和试剂预测任务的变压器模型的性能。我们发现数据增强的影响取决于预测任务和用于评估模型性能的指标。其次，我们探讨了输入格式、标记化方案和嵌入策略的不同组合对模型性能的贡献。我们发现不太稳定的输入设置通常会带来更好的性能。最后，我们验证了往返精度优于更简单的评估指标，例如 top-k准确性，使用人类专家委员会，并对通过往返测试的预测表现出强烈的一致性。这证明了在复杂的预测场景中更精细的指标的有用性，并突出了与预定义数据库直接比较的局限性，其中可能包括有限数量的化学反应途径。