Hi, In order to better understand what you'd like to get out of our website, we track the URLs you visit in a session. This information is only used to customize future communications with you. Do you consent to allow this tracking?
I Agree I Disagree. Disable Tracking

# Convert’s PRE A/B Testing Analysis Tool

## Free Sample Size Calculator & A/B/N Test Duration Calculator

How long does your experiment need to run so that you're confident of the result?

Do you have the traffic to support the number of variants you’ve planned?

These are questions every serious optimizer asks.

We’ve created a tool that will answer them for you.

With the Convert Sample Size & A/B/N Test Duration calculator you can:

• Get started with basic details like the daily visitors to the page you want to test on, desired conversion percentage for the variant, and the statistical confidence you’re looking to achieve.
• Get results to plan for - in short, you will see the number of days an experiment has to run plus the sample size that must be tested on to eliminate chances of false positives and other errors.
• Run A/B/N tests, split tests and multivariate tests with a good idea of the resources needed.

Never call another early winner. Never fall for improbable hypotheses.

This sample duration calculator keeps your testing rooted in real possibilities.

## Any Questions?

A/B testing is the favorite tactic of a marketer looking to optimize budget.

But at its core, the concepts are all pure statistics.

Don’t worry - we are not setting you up for a maths lesson.

Let us assume you’re running an A/B test on two landing pages selling the same product. The first one (we’ll call it A, or the control) does not have 3D product images. The second one (we’ll call it B, or the variation) has them.

The conversion rate in terms of “Add to Cart” is 7% for A and 9% for B. So should you just add 3D images to your product pages across the site?

No! Because all the visitors to your website have not seen page B and you can’t make assumptions about their preferences simply from observing the behavior of a much smaller sample size. Right? (PS: Don’t make assumptions in marketing or optimization…. being data driven is the way to go).

How to solve this little problem?

P-value comes to the rescue.

P-value will give you the probability that you have seen a 2 percentage point increase in the “Add to Cart” KPI for your variation (Page B) simply because of luck or other random factors.

The smaller the p-value, the greater the chance that adding the 3D images meaningfully contributed to the uplift in conversions and would likely apply to all the visitors coming to your website.

P-values do not operate alone.

They need to be held in check by something called a significance level or threshold. Think of it as the promise data scientists make to themselves to not fall in love with their own hypotheses. Say you are really impressed by those 3D images. To the point that even if the p-value tells you that there is a big chance that the increase in conversions means nothing, you still go ahead and roll them out.

Not a sound decision!

This is why it is crucial for A/B test results to be statistically significant. To rule out biases and make sure your budget is spent on the best bet!

The rule of thumb is to choose a probability level up to which your test’s p-value can go, before you have to admit that the variation is no good.

This level is generally taken to be 5%. When you have a 5% significance threshold it means that if you randomly pick visitors from the people coming to your site, only in 5% of the cases would you see a 2% point increase in “Add to Carts” because of luck or noise.

95% of the time we can conclude with reasonable confidence that the 3D images have enhanced the shopping experience in some way, leading to the KPI improvement.

Despite best intentions, sometimes being 95% certain of your A/B test results is just not feasible in terms of the resources involved.

This is why Justin Christianson of Conversion Fanatics has put together a video with some working hacks to “save” a test, or at least learn from it, if 95% confidence looks like a pipe dream. Watch it here.

“Sample size” is the total number of visitors you need to serve your control and challenger(s) to, before you reach meaningful conclusions that are statistically significant.

A good sample size is the basis of a robust A/B test - especially if you are working with minor changes.

Without going into jargon, the more people see your variations and control, the quicker you can determine the impact of your hypothesis, and smaller are the chances that the visitors are being swayed by random factors and not the logic of the changes you’ve made.

Having the right traffic volume is a key factor driving optimization success. Especially if you are conducting A/B tests.

You can use a tool like our sample size calculator and test duration calculator to gauge how long (and with how many visitors) you have to experiment to reach results you are confident of.

However, if the planning phase shows that you can’t have the desired number of variants you want in the test, or if you see a test stretching out for a year before meaningful conclusions can be drawn, you can leverage other optimization methods.

Here is a full list of alternative testing options, if you don’t have enough traffic on-site.

Great question! Each testing method has its pros and cons.

We’ve compiled an exhaustive blog post that will walk you through the differences. You can read it here.

## [VIDEO] How to Come Up with Hypotheses That Solve (Real) Conversion Roadblocks

Let data lead the way! Easier said than done, right? Just getting “winning” tests is not enough. The journey starts when you identify the right conversion roadblocks, the true points of friction that keep your visitors from becoming customers and advocates. Only insights derived from quality data can point...

## [VIDEO] Optimization Goal Setting: Top Things Testers Should Keep in Mind When Creating a CRO Gameplan

Conversion Rate Optimization (CRO) is a lot more than a buzzword. Done right, done patiently, it actually brings results. But what is the “right” way of going about testing and experimentation. In this video Justin Christianson, co-founder and president of Conversion Fanatics (a Convert Premium Agency Partner) walks you...

## [VIDEO] How to Come Up with Hypotheses That Solve (Real) Conversion Roadblocks

Omnichannel started out as a buzzword, but fast forward to 2019 and it is essential to the growth and stability of most businesses in the market. Omnichannel is the diversified presence of a brand across multiple channels of customer interaction and acquisition. It is quite beautiful – Inbound meeting...

## All Set With Your Test Planning? Try a Free 15-Day Trial of Convert Experiences.

No credit card required. Easy to use interface. Transparent winner selection. Let’s keep it simple!

##### Always working to improve outcomes.

15-Day, Full-Access Free Trial of Convert Experiences.

Quick form submission.
No credit card needed.
Support available.
Your email ID is used to send you the registration and login details for the app, (up to 7) suggestions to use the application, and 1:1 emails from buyer journey experts who help you choose the right plans.
You also give Convert consent to send you top CRO content in a monthly newsletter*
You can always change your preferences later.

We've Improved Conversions for