Covariates

What Are Covariates?

In A/B testing and online experiments, covariates are user-level variables — such as age, location, device type, or past behavior — that may influence the outcome you’re measuring. They’re independent of the treatment assignment and are typically observed before or during the experiment.

Covariates are not the intervention itself. They’re part of the context that helps you understand how users behave and why your results might vary, even before applying a change.

While covariates and confounding variables are related, they’re not interchangeable. Confounding variables bias your results if not accounted for. Covariates, when used properly, help control for bias and reduce unexplained variance.

Why Do Covariates Matter in A/B Testing?

Adding covariates to your analysis doesn’t change the treatment or the control — it sharpens your lens on the impact.

They:

• Reduce variance in your outcome metrics.
• Increase test sensitivity and statistical power.
• Help you detect real effects faster.
• Improve the precision of effect size estimates.
• Support better decisions with the same amount of data.

For example, if you’re testing a new homepage and account age is correlated with conversion, adjusting for account age can help you isolate the effect of the homepage change itself, rather than mixing it with user maturity.

How Are Covariates Used?

There are two key phases where covariates play a role:

Before the test:

• Use stratification or matching to ensure balanced distribution of covariates across groups.
• Choose covariates carefully — they must be independent of treatment assignment and correlated with the outcome.

During analysis:

• Apply regression techniques like ANCOVA, Lasso, or Ridge regression.
• Use control-variates methods such as CUPED to reduce noise.
• Weight data to adjust for imbalances.
• Run A/A tests to validate whether covariate-adjusted models remain unbiased.

Tools like Lasso regression help trim irrelevant covariates to avoid adding noise or computational overhead.

What Are the Benefits of Covariates?

• Faster learning: Smaller sample sizes needed to reach significance.
• Lower costs: Shorter test durations when variance is reduced.
• Increased confidence: More trustworthy estimates of treatment effects.
• Better targeting: Covariates help explain why a test worked for some users and not others.

In one example, adjusting for pre-test user engagement helped a team get a more accurate estimate of a website redesign’s impact on conversions.

What Can Go Wrong?

Covariates are powerful, but easy to misuse.

Watch out for:

• Covariate imbalance: Differences in covariate distributions between groups can bias results.
• Overfitting: Including too many irrelevant covariates adds noise.
• Dependent variables: Never use a covariate that’s influenced by the treatment.
• Computational complexity: Covariate-adjusted models are harder to implement and debug.
• Bias from bad data: If your covariates are based on flawed data, they’ll hurt more than help.

Always validate your setup with A/A tests and avoid relying on covariates as a shortcut for proper randomization.

“Covariate selection is important to approach deliberately, emphasizing sound practical and analytical justification. By proactively addressing outcome variability and relevant externalities upfront, you can strengthen the credibility of any causal relationships suggested by the results.

In addition to theory-guided covariates, methods such as LASSO regression can be leveraged to trim down the feature space analytically. Running null simulations and bootstrapped results also lend credence to the overall integrity of the model and confidence in the results.”

Stephen Hei, Forward Deployed Engineer, Peregrine