The Role of Randomized Control Trials in Online Demand Generation: Exploratory Evidence from Facebook
Proceedings of the International Conference on Information Systems (ICIS) 2021
30 Pages Posted: 22 Mar 2021 Last revised: 27 Apr 2022
Date Written: September 27, 2021
Observing the organically evolving Facebook advertising marketplace, this paper sets out to study how the adoption of randomized control trials (RCTs) for policy evaluation impacts firms’ behavior and performance on the platform. Findings indicate strong positive effects from the adoption of RCTs: After or in coincidence with RCT adoption, firms increase their investment in demand generation, at stable or higher levels of performance, as measured by customer purchases achieved per advertising dollar spent. A human expert provided to firms by the platform is also able to induce increases in investment but smaller ones, and, importantly, performance gains appear to exclusively materialize after the adoption of RCTs for policy evaluation, and not through human expertise alone. By means of the unbiased causal estimates obtained through RCTs, firms may be able to uniquely reduce their uncertainty about the environment and identify effective policies to exploit it with certainty.
Keywords: Experimentation, firm learning, online advertising, randomized control trials, exploration-exploitation
JEL Classification: M31, M37
Suggested Citation: Suggested Citation