Hi, In order to better understand what you'd like to get out of our website, we track the URLs you visit in a session. This information is only used to customize future communications with you. Do you consent to allow this tracking?
I Agree I Disagree. Disable Tracking

Convert’s PRE A/B Testing Analysis Tool

Free Sample Size Calculator & A/B/N Test Duration Calculator

Go Straight to Calculator...

How long does your experiment need to run so that you're confident of the result?

Do you have the traffic to support the number of variants you’ve planned?

These are questions every serious optimizer asks.

We’ve created a tool that will answer them for you.

With the Convert Sample Size & A/B/N Test Duration calculator you can:

Never call another early winner. Never fall for improbable hypotheses.

This sample duration calculator keeps your testing rooted in real possibilities.

Number of Visitors
Number of Days


Any Questions?

A/B testing is the favorite tactic of a marketer looking to optimize budget.

But at its core, the concepts are all pure statistics.

Don’t worry - we are not setting you up for a maths lesson.

Let us assume you’re running an A/B test on two landing pages selling the same product. The first one (we’ll call it A, or the control) does not have 3D product images. The second one (we’ll call it B, or the variation) has them.

The conversion rate in terms of “Add to Cart” is 7% for A and 9% for B. So should you just add 3D images to your product pages across the site?

No! Because all the visitors to your website have not seen page B and you can’t make assumptions about their preferences simply from observing the behavior of a much smaller sample size. Right? (PS: Don’t make assumptions in marketing or optimization…. being data driven is the way to go).

How to solve this little problem?

P-value comes to the rescue.

P-value will give you the probability that you have seen a 2 percentage point increase in the “Add to Cart” KPI for your variation (Page B) simply because of luck or other random factors.

The smaller the p-value, the greater the chance that adding the 3D images meaningfully contributed to the uplift in conversions and would likely apply to all the visitors coming to your website.

P-values do not operate alone.

They need to be held in check by something called a significance level or threshold. Think of it as the promise data scientists make to themselves to not fall in love with their own hypotheses. Say you are really impressed by those 3D images. To the point that even if the p-value tells you that there is a big chance that the increase in conversions means nothing, you still go ahead and roll them out.

Not a sound decision!

This is why it is crucial for A/B test results to be statistically significant. To rule out biases and make sure your budget is spent on the best bet!

The rule of thumb is to choose a probability level up to which your test’s p-value can go, before you have to admit that the variation is no good.

This level is generally taken to be 5%. When you have a 5% significance threshold it means that if you randomly pick visitors from the people coming to your site, only in 5% of the cases would you see a 2% point increase in “Add to Carts” because of luck or noise.

95% of the time we can conclude with reasonable confidence that the 3D images have enhanced the shopping experience in some way, leading to the KPI improvement.

Despite best intentions, sometimes being 95% certain of your A/B test results is just not feasible in terms of the resources involved.

This is why Justin Christianson of Conversion Fanatics has put together a video with some working hacks to “save” a test, or at least learn from it, if 95% confidence looks like a pipe dream. Watch it here.

“Sample size” is the total number of visitors you need to serve your control and challenger(s) to, before you reach meaningful conclusions that are statistically significant.

A good sample size is the basis of a robust A/B test - especially if you are working with minor changes.

Without going into jargon, the more people see your variations and control, the quicker you can determine the impact of your hypothesis, and smaller are the chances that the visitors are being swayed by random factors and not the logic of the changes you’ve made.

Having the right traffic volume is a key factor driving optimization success. Especially if you are conducting A/B tests.

You can use a tool like our sample size calculator and test duration calculator to gauge how long (and with how many visitors) you have to experiment to reach results you are confident of.

However, if the planning phase shows that you can’t have the desired number of variants you want in the test, or if you see a test stretching out for a year before meaningful conclusions can be drawn, you can leverage other optimization methods.

Here is a full list of alternative testing options, if you don’t have enough traffic on-site.

Great question! Each testing method has its pros and cons.

We’ve compiled an exhaustive blog post that will walk you through the differences. You can read it here.


All Set With Your Test Planning? Try a Free
15-Day Trial of Convert Experiences.

No credit card required. Easy to use interface. Transparent winner selection. Let’s keep it simple!

Always working to improve outcomes.
© 2020, Convert Inc. All rights reserved Terms of use Privacy Notice

15-Day, Full-Access Free Trial of Convert Experiences.

Watch Icon
Quick form submission.
Credit Card Icon
No credit card needed.
Headphones Icon
Support available.
Your email ID is used to send you the registration and login details for the app, (up to 7) suggestions to use the application, and 1:1 emails from buyer journey experts who help you choose the right plans.
You also give Convert consent to send you top CRO content in a monthly newsletter*
You can always change your preferences later.

We've Improved Conversions for
5000+ Leading Brands

Check Icon
You're Almost Done.
Managing Marketing Team
Managing Tech Team
Hypothesizing Experiments
Coding and QA of Experiments
Convert is committed to protecting your privacy and sees GDPR, CCPA and other Privacy Laws as an opportunity to strengthen our commitment even further. We don’t collect & process users’ personal information beyond what is required for the functioning of our services, and this will never change.

You're successfully signed up for the 15-day
Convert Experiences free trial.

You'll be re-directed to the app in a few seconds.
You will also receive a welcome message to your registered email with your login access.