What is MDE and how do I choose the right number?

MDE, or minimum detectable effect, is the smallest improvement you want the test to be able to catch. If your baseline is 3% and you set an MDE of 10%, the calculator sizes your test to detect a lift to 3.3% or higher. Pick an MDE that reflects the smallest improvement actually worth shipping. Too small and you will be waiting months; too large and you will miss real but modest gains. For most teams, 5-10% relative is a reasonable starting point.

When should I use Bayesian vs. Frequentist vs. Sequential?

Use Frequentist when you want a classical test with a fixed sample size. Commit before you start and do not check early. Use Sequential when you need to monitor a live test and might want to stop early without blowing up your false positive rate. Use Bayesian when you want to talk about results as a probability rather than a p-value; it tends to land better with people outside of data and analytics.

How do I calculate sample size for a conversion-rate test?

Sample size depends on your baseline conversion rate, minimum detectable effect (MDE), statistical power (typically 80%), and confidence level (typically 95%). This calculator uses the two-proportion z-test formula, with Bonferroni or Sidak corrections when you are testing multiple variants.

What is statistical significance?

Statistical significance tells you how unlikely your observed difference would be if nothing actually changed. A 95% confidence level maps to a 5% significance threshold. Higher confidence means a bigger required sample. Significance tells you the result is probably real; it does not tell you it is meaningful.

Should I use one-tailed or two-tailed tests?

Two-tailed, almost always. Two-tailed tests catch both positive and negative effects. One-tailed only makes sense if you are certain about the direction of change and genuinely do not care about detecting regressions, which is rare. If your variant underperforms, you want to know.

What is Sample Ratio Mismatch?

SRM checks whether the number of visitors in each variant matches your planned traffic split. If it does not, your randomization may be broken. SRM can surface bugs in your SDK, consent filtering that only affects some variants, bot traffic, or targeting mistakes. If SRM is flagged, do not trust the result regardless of the p-value.

What are Type I and Type II errors?

A Type I error is a false positive: shipping something that is not actually better. A Type II error is a false negative: missing a real lift. Confidence controls Type I errors. Stricter thresholds mean fewer false positives, but larger samples. Power controls Type II errors. Higher power means fewer missed effects, also at the cost of a larger sample.

What's a good conversion rate for my website?

Average conversion rates vary by industry, but typically range from 2-5%. E-commerce sites often average 2-3%, while B2B sites often sit around 2-5%. The good rate for your site depends on your industry, traffic quality, and business model, which is why A/B testing beats benchmarking.

.deferred-embed-shell .deferred-embed-shell__mask{display:none!important} .deferred-embed-shell .deferred-embed-shell__content{position:static!important;width:auto!important;height:auto!important;margin:0!important;clip:auto!important;clip-path:none!important;overflow:visible!important;visibility:visible!important;opacity:1!important;pointer-events:auto!important;user-select:auto!important}

The A/B testing calculator for better launch decisions

Calculate sample size, statistical significance, and test duration in one place.

Three decisions to make before you start

Before you enter any inputs into the stat sig calculator, you’ll need to determine which test mode fits your situation, which metric you’re optimizing, and which statistical method your team works with.

Which mode?

Test Planning: You haven’t launched yet. Enter your baseline rate and target effect to find out how many visitors you need and how long to run.
Test Analysis: Your test is done. Enter your observed results to check whether the difference between variants is statistically significant.

Which metric?

Conversion rate: Best for binary outcomes (e.g., signups, purchases, clicks). Reaches significance faster because converted/not-converted has lower variance than revenue.
Revenue per visitor: Use this when revenue impact matters more than CVR alone. Combines two sources of variance (CVR and AOV), so you’ll need a bigger sample.
Products per visitor: For ecommerce tests where items per order is the primary outcome.

Which method?

Frequentist: Set a fixed sample size upfront and don’t check early. This is the standard approach, and it’s the most defensible if you need to justify your results.
Sequential: Check results as they come in and stop early if the evidence is clear. Good when high traffic makes a fixed-horizon test expensive.
Bayesian: Express results as a probability — “87% chance variant B is better.” Easier to explain to stakeholders who don’t live in stats.

Full methodology detail for each method is in the Statistical methodology section below.

What your result means

Required sample: Visitors needed per variant before your result is valid. Don’t stop before you hit this number, even if you see significance early. Stopping at first significance is one of the most reliable ways to ship a false winner.
Test duration: Estimated days to reach the required sample at your current traffic and allocation. If it’s longer than six to eight weeks, think about raising your MDE, narrowing your test scope, or increasing the traffic allocation.
MDE curve: Shows how the minimum detectable effect shrinks as your sample grows. Useful when you can’t commit to waiting for the full sample; it tells you what lift size you’ll be able to detect at any point in the run.
Before you act: run the SRM check, avoid stopping at first significance, and make sure you’re optimizing for the right metric.

What this calculator answers

How many visitors each variant needs before an A/B test is ready.
Which minimum detectable effects are realistic for your traffic and run time.
Whether p-values, confidence intervals, power, and expected loss support a launch decision.
Whether observed traffic split suggests Sample Ratio Mismatch or a broken randomization path.
How conversion rate, Average Order Value, Revenue Per Visitor, and Products Per Visitor moved.

Core concepts

Sample size depends on baseline performance, MDE, confidence, statistical power, test direction, and the number of variants. Statistical significance checks whether an observed difference is unlikely under the no-change assumption. Power helps prevent false negatives, while confidence helps control false positives.

Revenue Per Visitor is total revenue divided by total visitors. It combines conversion rate and Average Order Value while keeping the visitor as the randomization unit, making it the preferred revenue metric for most e-commerce experiments. AOV is still useful as a guardrail when order size changes matter.

SRM stands for Sample Ratio Mismatch. It checks whether the observed visitor counts for each variant match the planned allocation closely enough. A mismatch can point to randomization bugs, bot traffic, consent filtering, targeting mistakes, or analytics collection issues.

Start your 15-day free trial now.

No credit card needed
Access to premium features

First Name *

Business Email * (Input a valid email address)

I want to receive the Convert Newsletter

By submitting this form, you agree to have read and acknowledged our: terms of use, privacy notice and

You can always change your preferences later.