Sample Size Calculator
Convert’s A/B test duration calculator also offers a sample size calculator. The sample size calculator offers insight into how much traffic you need using the Z-test. Providing data-points such as the conversion rate of the goal you are testing and the number of variations, you can calculate the amount of traffic needed to obtain confident A/B testing results.
Examples Using Sample Size Calculator
If the existing conversion rate is 3% and the expected improvement is 20% while testing two variations at a confidence level of 95%, you would need a sample size of 42,034 to get confident results. At 2,000 daily visitors to this test group, it would take 22 days according to our duration calculator.
Changing variable will give you different results:
- Increased confidence (97.5%) takes 50,117 visitors
- A 1% conversion rate of the goal means a testing of 128,700 visitors.
- Adding 2 variations, to make the total 4, gives us 127,400 tested visitors.
As you can see the sample size depends a lot on the numbers but using the A/B testing tool will provide real results at the end of the experiment. Some other things to consider beyond sample size and confidence:
- Run the experiments for a minimum one buying cycle (Run the experiment no less than 5 days – visitors buy within 5 days after visiting the website).
- Don’t run experiments more than once a month. Each month typically sees 10% of users cookies get deleted, making this group appear as a new visitor, polluting results.
- Consider setting 98% or 99% confidence in your testing tool (in Convert Experiences you can do this in the Experiment Summary)
At Convert Experiences you can always ask your account manager to help you configure your experiments on all managed, agency and enterprise accounts. Experience our service by contacting us or setup a free 15-day trial here.