Marketers use online advertising platforms to compare user responses to different ad content. But platforms’ experimentation tools deliver different ads to distinct and undetectably optimized mixes of users that vary across ads, even during the test. Because exposure to ads in the test is non-random, the estimated comparisons confound the effect of the ad content with the effect of algorithmic targeting. This means experimenters may not be learning what they think they are learning from ad A-B tests. The authors document these “divergent delivery” patterns during an online experiment for the first time. They explain how algorithmic targeting, user heterogeneity, and data aggregation conspire to confound the magnitude, and even the sign, of ad A-B test results. Analytically, the paper extends the potential outcomes model of causal inference to treat random assignment of ads and user exposure to ads as separate experimental design elements. Managerially, the authors explain why platforms lack incentives to allow experimenters to untangle the effects of ad content from proprietary algorithmic selection of users when running A-B tests. Given that experimenters have diverse reasons for comparing user responses to ads, the authors offer tailored prescriptive guidance to experimenters based on their specific goals.

中文翻译：

---

EXPRESS：AB 测试出错的地方：不同的交付方式如何影响在线实验无法（并且可以）告诉您客户对广告的反应


营销人员使用在线广告平台来比较用户对不同广告内容的反应。但平台的实验工具会向不同且难以察觉的优化用户组合提供不同的广告，即使在测试期间，这些用户组合也会因广告而异。由于测试中广告的曝光是非随机的，因此估计的比较混淆了广告内容的效果与算法定位的效果。这意味着实验者可能并没有学到他们认为从广告 AB 测试中学到的东西。作者首次在在线实验中记录了这些“不同的交付”模式。他们解释了算法定位、用户异质性和数据聚合如何共同混淆广告 AB 测试结果的大小甚至符号。从分析上来说，本文扩展了因果推理的潜在结果模型，将广告的随机分配和用户对广告的接触视为单独的实验设计元素。从管理上来说，作者解释了为什么平台缺乏激励措施来允许实验者在运行 AB 测试时从用户的专有算法选择中理清广告内容的影响。鉴于实验者出于不同的原因比较用户对广告的反应，作者根据实验者的具体目标为他们提供了量身定制的规范性指导。