Decision makers, such as doctors, judges, and managers, make consequential choices based on predictions of unknown outcomes. Do these decision makers make systematic prediction mistakes based on the available information? If so, in what ways are their predictions systematically biased? In this article, I characterize conditions under which systematic prediction mistakes can be identified in empirical settings such as hiring, medical diagnosis, and pretrial release. I derive a statistical test for whether the decision maker makes systematic prediction mistakes under these assumptions and provide methods for estimating the ways the decision maker’s predictions are systematically biased. I analyze the pretrial release decisions of judges in New York City, estimating that at least 20% of judges make systematic prediction mistakes about misconduct risk given defendant characteristics. Motivated by this analysis, I estimate the effects of replacing judges with algorithmic decision rules and find that replacing judges with algorithms where systematic prediction mistakes occur dominates the status quo.

中文翻译：

---

识别观测数据中的预测错误


决策者，例如医生、法官和管理者，根据对未知结果的预测做出相应的选择。这些决策者是否根据现有信息犯了系统性的预测错误？如果是这样，他们的预测在哪些方面存在系统性偏差？在本文中，我描述了在招聘、医疗诊断和审前释放等经验环境中可以识别系统预测错误的条件。我对决策者在这些假设下是否犯系统性预测错误进行了统计检验，并提供了估计决策者的预测存在系统性偏差的方法。我分析了纽约市法官的审前释放决定，估计至少 20% 的法官在考虑到被告特征的不当行为风险方面犯了系统性预测错误。受此分析的启发，我估计了用算法决策规则取代法官的效果，发现用发生系统性预测错误的算法取代法官占主导地位。