Experiments are not always the panacea
A/B tests are powerful, but not always feasible. When experiments are slow, unethical, or impossible, causal methods can provide faster, cheaper, and more informative answers - if used correctly.
Limits of naive experimentation
A/B tests assume you can randomize at the relevant unit and wait for a signal. In many business contexts, this isn't possible: rollout costs are high, interference invalidates randomization, or business constraints prohibit experiments. Forcing experiments can be wasteful or harmful.
Modern causal techniques as a complement
Quasi-experimental designs, instrumental variables, and synthetic controls provide credible estimates when randomization fails. They trade stronger assumptions for feasibility and speed. The right choice depends on the context; causal methods broaden your toolkit.
Decision Guide
Ask yourself: can you randomize at scale without interference? If not, consider RDD, DID, or instrumental variables. Use experiments when possible and causal methods when necessary. Combine them: small experiments to validate structural assumptions and observational analysis for broader inferences.
Towards smarter evaluations
Expect more hybrid designs: randomized pilots to validate assumptions, paired with observational causal analysis for extended inference. Tooling will make these solutions easier to implement and verify.
Use the right tool
A/B tests are not a hammer for every nail. Learn to choose between experimentation and causal analysis - and how to use them together - for better and faster product decisions.
Sources
- Quasi-experimental Designs in Practice by Applied Research Group (2020)