In the drug discovery process, where experiments can be costly and time-consuming, computational models that predict drug-target interactions are valuable tools to accelerate the development of new therapeutic agents. Estimating the uncertainty inherent in these neural network predictions provides valuable information that facilitates optimal decision-making when risk assessment is crucial. However, such models can be poorly calibrated, which results in unreliable uncertainty estimates that do not reflect the true predictive uncertainty. In this study, we compare different metrics, including accuracy and calibration scores, used for model hyperparameter tuning to investigate which model selection strategy achieves well-calibrated models. Furthermore, we propose to use a computationally efficient Bayesian uncertainty estimation method named HMC Bayesian Last Layer (HBLL), which generates Hamiltonian Monte Carlo (HMC) trajectories to obtain samples for the parameters of a Bayesian logistic regression fitted to the hidden layer of the baseline neural network. We report that this approach improves model calibration and achieves the performance of common uncertainty quantification methods by combining the benefits of uncertainty estimation and probability calibration methods. Finally, we show that combining post hoc calibration method with well-performing uncertainty quantification approaches can boost model accuracy and calibration. In this work we provide a comprehensive probability calibration study using neural networks for drug-target interaction predictions. We report a significant effect of the hyperparameter selection strategy, as well as uncertainty estimation and probability calibration approaches on the reliability of uncertainty estimates, which is crucial for an efficient drug discovery process.

中文翻译：

---

在药物发现中实现明智的决策：使用基于神经网络的结构-活性模型的综合校准研究


在药物发现过程中，实验可能既昂贵又耗时，预测药物-靶点相互作用的计算模型是加速新治疗药物开发的宝贵工具。估计这些神经网络预测中固有的不确定性可提供有价值的信息，从而在风险评估至关重要时促进最佳决策。然而，此类模型可能校准不佳，这会导致不可靠的不确定性估计，无法反映真正的预测不确定性。在这项研究中，我们比较了用于模型超参数调整的不同指标，包括准确性和校准分数，以研究哪种模型选择策略实现了校准良好的模型。此外，我们建议使用一种名为 HMC 贝叶斯最后一层 （HBLL） 的计算效率高的贝叶斯不确定性估计方法，该方法生成哈密顿蒙特卡洛 （HMC） 轨迹，以获得拟合到基线神经网络隐藏层的贝叶斯逻辑回归参数的样本。我们报告说，这种方法通过结合不确定性估计和概率校准方法的优点，改进了模型校准并实现了常见不确定性量化方法的性能。最后，我们表明，将事后校准方法与性能良好的不确定性量化方法相结合可以提高模型的准确性和校准。在这项工作中，我们提供了一个全面的概率校准研究，使用神经网络进行药物-靶点相互作用预测。 我们报告了超参数选择策略以及不确定性估计和概率校准方法对不确定性估计可靠性的显着影响，这对于高效的药物发现过程至关重要。