Control

What is a Control in A/B Testing?

In A/B testing, a control is the original version of a webpage, feature, or user experience that remains unchanged during the experiment. It’s the benchmark, i.e., what you already have live today. Users in the control group continue to see this unchanged experience, while others are randomly shown one or more variations.

The purpose of the control is to give you a stable baseline to compare against. Without it, you can’t tell whether a new version is truly better or just different.

What is the Purpose of a Control Group?

The control group makes it possible to measure real impact. When test groups are compared to a control running in the same time period, you can isolate the effect of the change.

Here’s why it matters:

• Eliminates guesswork: You’re not comparing before-and-after data, you’re comparing side by side, under the same conditions.
• Accounts for external factors: Like seasonality, traffic spikes, or competitor activity, since these usually affect both groups.
• Controls for unobserved variables: Like user behavior trends you didn’t measure or anticipate.
• Reduces false positives: This helps ensure any change in performance is caused by the variant, not random noise.

Without a control, you’re just hoping the variant worked. With one, you know.

How Control Groups Work in A/B Tests

In a standard A/B test:

1. Users are randomly assigned to either the control or one of the variant groups.
2. The control group sees the original experience, unchanged.
3. Data is collected on the same metrics across both groups.
4. At the end of the test, you compare performance—usually through statistical analysis—to see if the variant performed better, worse, or the same.

Randomization is critical. It ensures both groups are similar in makeup, so differences in performance can be attributed to the change being tested.

Control vs Variant: What’s the Difference?

Control = Original experience
Variant = New version or modification being tested

The variant introduces a change you want to evaluate. The control shows you what would happen if you changed nothing. You compare both to understand the actual effect.

Control vs Holdout Group: Key Differences

A control group is active during the test and is used to measure short-term performance differences.

A holdout group is used for long-term measurement. It’s a segment of users withheld from all tests or feature rollouts. While the rest of your audience sees the change, the holdout group doesn’t so you can assess long-term impact on retention, revenue, or behavior.

Some companies even use “uber holdout” groups that never receive any tests or new features, just to track baseline performance over time.

In short:

• Control = “What happens right now if we change nothing?”
• Holdout = “What happens in the long run if we never apply the change?”

“Without a CONTROL GROUP you don’t know if the change you made in the VARIANT GROUP made the difference—or if it was simply by chance.A control is your baseline for the same time period in which you run your experiment. It shows what happens without the change being present.

Now what about holdout groups? Well, they are a little different. They measure the long-term impact. Like, if you want to see how a new feature affects retention or revenue over time.

The goal of both a control anda holdout group is the same. To understand the TRUE impact of a variation.”

Paul Randall, Associate Director of Strategy at Speero

Best Practices for Running a Control Group

1. Use random assignment to reduce bias and ensure both groups are statistically similar
2. Don’t modify the control during the test; it must stay unchanged
3. Keep sample sizes balanced unless you have a strong reason for unequal splits
4. Use the control as your performance baseline, especially when calculating lift
5. Monitor guardrail metrics, like sample ratio mismatch or latency, to validate the test is running properly
6. Run A/A tests first to validate the test setup and confirm randomization
7. Be mindful of opportunity cost: users in the control miss out on potential improvements
8. Avoid contamination between control and variant groups, especially in shared environments or multi-page experiences
9. Document control behavior clearly, so test analysis is grounded in the right baseline expectations