Sample Size

What is Sample Size in A/B Testing?

Sample size refers to the number of participants, users, sessions, or pageviews allocated to each variant in an experiment. It determines whether your test can reliably detect a true difference between your control and treatment groups.

A properly chosen sample size gives your experiment enough power to detect a meaningful effect, if one exists. Too small, and you risk misleading or inconclusive results. Too large, and you might waste time and resources, or find significance where none matters.

What Influences Required Sample Size?

Several key factors drive how large your sample should be:

• Statistical Power: The probability of detecting a true effect. Higher power = lower chance of a false negative. Usually set to 80% or 90%.
• Minimum Detectable Effect (MDE): The smallest effect size that you care about. Smaller MDEs require larger samples to detect.
• Significance Level (Alpha): Typically set at 0.05. This is your acceptable false positive rate, and it influences the required sample size.
• Variance: The spread of your data. More variability means a larger sample is needed to detect the same effect.
• Baseline Conversion Rate: Your starting point. It helps determine what change is statistically and practically meaningful.
• Number of Variants: A/B/n tests require more total users than simple A/B tests, since each additional variant needs enough observations.

Why Does Sample Size Matter?

A good sample size does more than just make your p-values valid:

• Reduces volatility in results, especially for small businesses
• Ensures underpowered tests don’t get mistaken as “no effect”
• Prevents misleading conclusions from tiny or inflated samples
• Makes results more precise and trustworthy

How to Calculate Sample Size

There’s no one-size-fits-all formula, but common methods include:

• Online Calculators: Tools like Convert’s Free A/B Test Statistical Calculator or calculators from platforms are widely used.
• Standard Formula: Requires inputs like desired alpha, power, baseline conversion rate, and MDE.
• Simulations: For complex experiments or multiple testing adjustments.
• Rules of Thumb: For conversion rate metrics, aim for 100+ conversions per variant. When unsure, adding more users is generally safer than using too few.

Mistakes to Avoid

• Testing with too few users = high noise, low signal
• Using too many = wasting time and finding trivial “wins”
• Peeking before reaching the required sample = inflated Type I error
• Forgetting that unequal group sizes hurt estimate precision
• Using unreliable or outdated variance estimates when planning

“I can’t stress this enough, sample size is super important in A/B testing. To make the best decisions, you must ensure that each test variation has a large enough sample size—that is, enough users or observations to collect solid statistical data for your analysis.

Why does this matter? Well, think about it: if you only ask two friends for restaurant recommendations and assume their favorite is the best choice in town, you might miss out on some amazing places. The same is true in testing. You can’t say which variation won with a small sample size. This uncertainty can lead to decisions that might not be in your best interest in the long run. So, having a good sample size isn’t just a technical detail; it’s key to making informed choices that can significantly impact your business.”

Veronika Morozová, Solution Consultant at Contentsquare