More and more scholars are expressing doubt about whether questionnaire-based and other human-rater-based forms of behavior measurement are trustworthy, even though many of these measures meet psychometric best practice standards. I identify a lack of behavioral counterfactuals as common yet avoidable underlying problem and the existence of behavioral counterfactuals as an overlooked validity criterion. When behavioral counterfactuals exist, variation in item responses indicates variation in the presence, magnitude, or temporal unfolding of behaviors. By contrast, responses to non-counterfactual items and measures represent an indefinite mix of behavioral variation and variation in raters’ evaluation of the social significance of behaviors. I offer a typology of behaviorally non-counterfactual item formulations and conduct a large-scale review that identifies non-counterfactual item formulations as a severe and widespread problem that has intensified in recent decades. Such non-counterfactual measurement undermines the correctness of research findings and the clarity of action recommendations for managers. Using the stylized example of helpful and harmful leadership, I illustrate how non-counterfactual measures can gain erroneous empirical support and provide flawed as well as opaque “information” about effective leadership behavior. To reinvigorate research, I provide recommendations for ensuring behavioral counterfactuals, for example, through better questionnaires and coding schemes, experimentation, and technology-based measurement.

越来越多的学者对基于问卷和其他基于人类评估者的行为测量形式是否值得信赖表示怀疑，尽管其中许多测量符合心理测量最佳实践标准。我认为缺乏行为反事实是常见但可以避免的根本问题，行为反事实的存在是一个被忽视的有效性标准。当行为反事实存在时，项目反应的变化表明行为的存在、程度或时间展开的变化。相比之下，对非反事实项目和测量的反应代表了行为变异和评估者对行为社会意义的评估变异的不确定组合。我提供了行为非反事实项目表述的类型学，并进行了大规模审查，将非反事实项目表述确定为近几十年来加剧的严重而普遍的问题。这种非反事实的衡量破坏了研究结果的正确性和管理者行动建议的清晰度。我使用有益和有害的领导力的典型例子，说明非反事实措施如何获得错误的实证支持，并提供有关有效领导行为的有缺陷且不透明的“信息”。为了重振研究，我提供了确保行为反事实的建议，例如通过更好的问卷和编码方案、实验和基于技术的测量。