Bayesian A/B test calculator
Skip the p-values. Get the probability your variant actually wins.
A probability and risk view of your conversion test
Bayesian testing asks how likely the variant is to beat control, then pairs that probability with the downside of being wrong and a plausible range for the true effect.
- Probability you can communicate: Probability to win answers the stakeholder question directly: given the observed conversion data and model, how likely is the variant to outperform control?
- Risk you can compare: Expected loss quantifies the average conversion-rate downside of choosing the variant when control is actually better.
- Uncertainty you can see: The credible interval shows a plausible posterior range for the true conversion-rate effect. More data generally narrows that range.
How to use the Bayesian calculator
This view gives you Bayesian probabilities instead of a frequentist significance verdict. Configure the mode, conversion metric, and decision rule before you run the numbers.
1. Which mode?
- Test Planning: Use the fixed-horizon MDE sample as a stability reference before launch. It is a planning floor, not a Bayesian shipping threshold.
- Test Analysis: Enter live or completed control and variant data to estimate probability to win, expected loss, and the credible interval.
2. Which metric?
- Conversion rate: This Bayesian model is for binary outcomes such as a signup, click, purchase, or activation where each visitor converts or does not.
3. Decision rule
- Set it before launch: Choose acceptable probability, expected-loss, effect-size, and data-quality criteria before reading the result. Probability alone is not a shipping rule.
Revenue and products are available in the main calculator with frequentist or sequential analysis; this Bayesian view remains conversion-rate focused.
What your Bayesian results mean
Use probability, expected loss, and the credible interval together. No single number is enough for a launch decision.
- Probability to win: The posterior probability that the variant conversion rate is above control. Compare it with expected loss, effect size, and your prespecified decision threshold.
- Expected loss: The average conversion-rate downside of choosing the variant when it underperforms control across the posterior draws.
- Credible interval: A plausible posterior range for the true conversion-rate effect. If it still spans zero, the direction remains uncertain.
Decision discipline still matters
Bayesian output is easier to read, but it is not a license to ignore data quality.
- Check SRM before trusting any result.
- Do not ship on probability alone; consider expected loss, interval width, effect size, and business stakes together.
- Small samples can produce unstable probabilities. Use the planning sample as a stability reference and do not act on the first favorable read.
Bayesian conversion methodology
Bayesian calculations here are intentionally scoped to binary conversion-rate tests.
- Beta posteriors: Control and variant conversion rates are modeled with fixed Beta(1,1) priors updated by observed successes and failures. Use this when the primary outcome is converted or did not convert.
- Chance to win: The calculator uses 10,000 Monte Carlo draws from the Beta posteriors to estimate the probability that the variant conversion rate is higher than control. Use this when stakeholders need probability language instead of a p-value.
- Expected loss: For each posterior draw where the variant underperforms control, the calculator records the loss and averages it across all draws. Use this to avoid shipping a variant that is merely probably better but still too risky.
The Bayesian outputs that matter
These fields translate uncertainty into a product decision.
- Chance to win: The posterior probability that the variant conversion rate is above control.
- Expected loss: The expected penalty if you choose the wrong variant.
- Credible interval: A plausible posterior range for the true effect.
Related calculators
Common questions about Bayesian A/B testing
Bayesian output is more readable, but the same data-quality rules still apply.
- What is the difference between Bayesian and frequentist A/B testing? Frequentist testing evaluates how incompatible the data is with a null hypothesis under a prespecified error rate. Bayesian testing estimates a posterior probability that the variant beats control and adds expected loss and a credible interval.
- Do I need to set a sample size before a Bayesian test? Bayesian output can be computed as data arrives, but small-sample estimates can be unstable. This calculator uses the fixed-horizon MDE sample as a practical stability reference; do not treat the first favorable probability as a shipping signal.
- Who should use Bayesian testing? Use it when probability and downside risk fit the decision better than a p-value, especially when stakeholders need a direct probability statement and the cost of a wrong choice varies by test.
- What is a good win probability before I ship? There is no universal cutoff. Prespecify a probability threshold and an acceptable expected-loss limit based on the business stakes, then require a clean SRM check and review the credible interval and observed effect before acting.
- What prior does this calculator use? It uses a fixed Beta(1,1) prior for each conversion rate. Priors are not configurable in this calculator, which keeps the analysis rule consistent across the full test.
- What result means I am ready to ship? A decision should satisfy your prespecified probability and expected-loss thresholds, show a business-relevant effect with acceptable uncertainty, and pass the always-on SRM check. The calculator does not turn one probability into an automatic shipping recommendation.
- Can I use Bayesian testing here for revenue metrics? No. This page models binary conversion outcomes with Beta posteriors. Use the main calculator for revenue per visitor, average order value context, or products per visitor.