Holdouts: the test that shows what your marketing caused
A campaign can look great and still change nothing. A holdout tells you which.
A retention email goes out and renewals look strong. Did the email cause that, or would those customers have renewed anyway? Platform reports cannot tell you. A holdout can.
How it works
Before a campaign, randomly set aside a slice of the audience and send them nothing. Afterwards, compare their results with everyone who received it. The difference is the incremental lift: what the marketing actually caused.
Try it: drag the holdout size and rerun the test
Why results are often inconclusive
On one media client, we ran holdouts on retention email, SMS and push, and many came back inconclusive. Usually that is not a failure. It means the holdout was too small, the window too short, or the metric too noisy to detect a modest effect. The simulator above shows it: with a tiny holdout, the same real lift often looks like noise.
Making them work
- Size the holdout for the lift you expect to detect, before launch.
- Measure the outcome you care about, such as renewals or revenue, not opens.
- Run long enough to cover the full purchase or renewal cycle.
- Report a range with confidence intervals, not a single number.
Working on something like this? I help teams fix tracking, build CDPs and make marketing data ready for AI.
Start a conversation