We discuss a relatively new meta-scientific research design: many-analyst studies that attempt to assess the replicability and credibility of research based on large-scale observational data. In these studies, a large number of analysts try to answer the same research question using the same data. The key idea is the greater the variation in results, the greater the uncertainty in answering the research question and, accordingly, the lower the credibility of any individual research finding. Compared to individual replications, the large crowd of analysts allows for a more systematic investigation of uncertainty and its sources. However, many-analyst studies are also resource-intensive, and there are some doubts about their potential to provide credible assessments. We identify three issues that any many-analyst study must address: 1) identifying the source of variation in the results; 2) providing an incentive structure similar to that of standard research; and 3) conducting a proper meta-analysis of the results. We argue that some recent many-analyst studies have failed to address these issues satisfactorily and have therefore provided an overly pessimistic assessment of the credibility of science. We also provide some concrete guidance on how future many-analyst studies could provide a more constructive assessment.

中文翻译：

---

通过多位分析师的研究对科学的可信度进行更可信的评估


我们讨论一种相对较新的元科学研究设计：许多分析师的研究，试图评估基于大规模观测数据的研究的可重复性和可信度。在这些研究中，大量分析师试图使用相同的数据来回答相同的研究问题。关键思想是结果的差异越大，回答研究问题的不确定性就越大，相应地，任何单个研究发现的可信度就越低。与个人重复相比，大量分析师可以对不确定性及其来源进行更系统的调查。然而，许多分析师的研究也是资源密集型的，人们对其提供可信评估的潜力存在一些怀疑。我们确定了任何多位分析师研究都必须解决的三个问题：1）确定结果变异的来源； 2）提供与标准研究类似的激励结构； 3) 对结果进行适当的荟萃分析。我们认为，最近的一些多位分析师的研究未能令人满意地解决这些问题，因此对科学的可信度做出了过于悲观的评估。我们还就未来的多位分析师研究如何提供更具建设性的评估提供了一些具体指导。