In “Cause, Effect, and the Structure of the Social World” (2023), Megan Stevenson makes a claim that randomized controlled trials (RCTs) have not had a significant effect in criminal justice settings. She then draws the conclusion that the gold standard for research designs, RCTs, are inherently incapable of doing so, demonstrating that the social world they intervene on is too complex, but also too resilient, to respond to the types of interventions that are evaluable by RCT. She calls the insistence that RCTs can work an “engineer’s” view of the world, which she discards as a myth. The argument then conflates RCTs with other methods of generating and sustaining change in organizations and systems, and closes suggesting RCTs should be discarded for less rigorous but more sweeping means of social reform. This article proceeds as follows: It characterizes Stevenson’s argument, which she asserts is empirical, as a de facto meta-analysis of criminal justice RCTs executed as a heuristic and presented in a narrative format. It argues that if a formal meta-analysis would be rendered invalid by violating established protocols, then a heuristic analysis that commits the same errors would be invalid as well. The analysis then presents the prohibitions on pooling studies with heterogeneous designs, interventions, outcomes, and metrics for the purpose of meta-analysis. It demonstrates that Stevenson pools a wide range of heterogenous studies, rendering her empirical meta-analytic claims problematic. It is true that many criminal justice RCTs have produced null or lackluster results—which also constitute an important outcome—and attempts to replicate significant findings have often been unsuccessful. This is not unique to criminal justice: psychology was recently in crisis when it was determined few of its most prominent studies could be replicated. However, less rigorous methods of reform do not solve this problem. Instead, more comprehensive research designs such as hybrid implementation/effectiveness trials can reveal aspects of our social world that impact external validity and generalizability. Findings from these studies can help illuminate the conditions that impact outcomes and sustainably modify highly resilient human behaviors. These methods arise from techniques in medicine and public health, which Stevenson brackets off as fundamentally different from criminal justice. This type of thinking may be the actual myth that prevents progress.

梅根·史蒂文森 (Megan Stevenson) 在《社会世界的因果和结构》(Cause, Effect, and the Structure of the Social World)（2023 年）中声称，随机对照试验 (RCT) 在刑事司法环境中并未产生显着影响。然后，她得出的结论是，研究设计的黄金标准 RCT 本质上无法做到这一点，这表明它们干预的社会世界太复杂，但也太有弹性，无法对可评估的干预类型做出反应。随机对照试验。她称坚持随机对照试验可以发挥作用是“工程师”的世界观，她将其视为神话而抛弃。然后，该论点将随机对照试验与在组织和系统中产生和维持变革的其他方法混为一谈，最后建议应放弃随机对照试验，以采用不那么严格但更广泛的社会改革手段。本文的内容如下：它描述了史蒂文森的论点，她声称这是经验性的，作为对刑事司法随机对照试验的事实上的元分析，以启发式方式执行并以叙述形式呈现。它认为，如果正式的荟萃分析因违反既定协议而无效，那么犯同样错误的启发式分析也将无效。然后，该分析提出了出于荟萃分析目的而禁止采用异质设计、干预措施、结果和指标的汇总研究。它表明史蒂文森汇集了广泛的异质研究，使她的经验荟萃分析主张存在问题。确实，许多刑事司法随机对照试验产生了无效或乏善可陈的结果——这也是一个重要的结果——并且复制重要发现的尝试往往不成功。 这并不是刑事司法所独有的：心理学最近陷入了危机，因为人们确定其最著名的研究几乎没有可复制的。然而，不太严格的改革方法并不能解决这个问题。相反，更全面的研究设计，例如混合实施/有效性试验，可以揭示影响外部有效性和普遍性的社会世界的各个方面。这些研究的结果可以帮助阐明影响结果的条件并可持续地改变高度弹性的人类行为。这些方法源于医学和公共卫生领域的技术，史蒂文森将其视为与刑事司法根本不同。这种想法可能是阻碍进步的真正神话。