The term "ecological validity" (EV) has traditionally referred to test scores' ability to predict real-world functioning. However, a test's similarity to real-world tasks is sometimes mistaken for evidence of its ability to predict daily life, sometimes bypassing rigorous validation research. The goal of this systematic review was to examine the type and quality of evidence used to support claims of EV of novel face-valid tests of executive functions (EF). MEDLINE and PsychINFO databases were searched using the following terms: ecologic* AND neuropsychol* AND (executive function* OR executive dysfunction OR executive abilit*). Thirty-two articles that explicitly stated that their results demonstrated EV of a novel face-valid test of EF were identified. Results showed that only 60% of studies based their claims about EV on test scores' ability to predict functional outcomes, with the remaining 40% relying on other evidence (e.g., correlations with other measures, participant feedback, group differences). Among the studies that did base their conclusions on test scores' ability to predict outcomes (n = 19), an overwhelming majority relied on behavioral rating scales, utilized small sample sizes and participant-to-variable ratios, and failed to control for covariates and multiple comparisons. Poor scientific rigor was particularly pronounced in studies of "naturalistic" tests. The present systematic review reveals significant conceptual, methodological, and statistical flaws among an overwhelming majority of studies that claim to have found support for the EV of a novel face-valid test of EF. We call upon authors, reviewers, and editors to safeguard the scientific rigor of research in this area. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

中文翻译：

---

新型执行功能测试的生态有效性研究的陷阱：系统回顾和行动呼吁。


“生态有效性”（EV）一词传统上指的是测试分数预测现实世界功能的能力。然而，测试与现实世界任务的相似性有时会被误认为是其预测日常生活能力的证据，有时会绕过严格的验证研究。本系统评价的目的是检查用于支持新型执行功能（EF）面部有效测试的 EV 主张的证据的类型和质量。使用以下术语搜索 MEDLINE 和 PsychINFO 数据库：生态* AND 神经心理学* AND（执行功能* 或执行功能障碍或执行能力*）。已确定 32 篇文章明确指出其结果证明了 EF 的新型面部有效测试的 EV。结果显示，只有 60% 的研究将其关于 EV 的主张基于测试分数预测功能结果的能力，其余 40% 则依赖于其他证据（例如与其他测量的相关性、参与者反馈、群体差异）。在确实根据测试分数预测结果的能力得出结论的研究中（n = 19），绝大多数依赖于行为评分量表，利用小样本量和参与者与变量的比率，并且未能控制协变量和多重比较。科学严谨性差在“自然主义”测试的研究中尤其明显。目前的系统评价揭示了绝大多数研究中存在重大的概念、方法和统计缺陷，这些研究声称已经找到了对 EF 的新型面部有效测试的 EV 的支持。我们呼吁作者、审稿人和编辑维护该领域研究的科学严谨性。 （PsycInfo 数据库记录 (c) 2024 APA，保留所有权利）。