What is a holdout group?
Also known as: Control group
Used in a sentence: Before we call the renewal nudge a win, let's check whether the holdout group renewed at nearly the same rate.
How it works
When a campaign launches, the targeting system randomly assigns a share of eligible users to the holdout. They meet every targeting rule but are never shown the message. Everyone else sees it as normal.
After a set period, the team compares the conversion rate of exposed users with that of the holdout. The difference is the incremental lift. Because assignment is random, both groups are alike in every way except exposure.
Assignment must be sticky, so a user stays in the same group for the whole test. A holdout can apply to a single campaign or, as a global holdout, to all campaigns, measuring the combined effect of the whole engagement programme.
Example
An investment app adds a bottom sheet on its SIP screens reminding users to top up. The team holds back a random slice of eligible users who never see it. After a few weeks, they compare new SIP collections between exposed users and the holdout. The gap, not the raw total, is reported as the campaign’s impact, and the holdout is kept running to check the lift lasts.
Holdout group vs A/B test control
| Holdout group | A/B test control | |
|---|---|---|
| What they see | Nothing from the campaign | The current version |
| Question answered | Does the campaign help at all? | Which version works better? |
| Typical duration | Can run long-term | Ends when a winner is clear |
A/B tests choose between versions. A holdout checks whether showing anything beats showing nothing.
Best practices
- Randomise assignment and keep it sticky for the full test period.
- Apply the same targeting rules to both groups, so the only difference is exposure.
- Decide the success metric and test length before launch, not after seeing results.
- Keep a small global holdout to measure the combined effect of all campaigns.
How to do this with Digia Engage
Digia Engage targets campaigns using segments and cohorts that already exist in CleverTap, MoEngage or WebEngage. To keep a holdout, a team creates a random sample as its own cohort in the CEP and excludes it from the campaign’s audience, then compares the two groups on the outcome metric. Campaigns change from the dashboard without an app release. The Dezerv case study shows a team A/B testing onboarding nudges and copy variants this way.
Frequently asked questions
- How big should a holdout group be?
- There is no fixed percentage. A larger holdout gives more statistical power and clearer results sooner, but more users miss a campaign that may help them. Base the size on your traffic and the smallest lift worth detecting. High-traffic apps can keep the holdout small; smaller apps need a bigger share or a longer test.
- Why do campaigns look better without a holdout?
- Without a holdout, every conversion from a user who saw the campaign gets credited to it, including users who would have converted anyway. Campaigns usually target high-intent users, so that baseline is often large. A holdout removes it and shows only the extra conversions the campaign actually caused.
Related terms
Read more
Part of the In-app engagement glossary.