A/B testing is the favorite tactic of a marketer looking to optimize budget.
But at its core, the concepts are all pure statistics.
Don’t worry - we are not setting you up for a maths lesson.
Let us assume you’re running an A/B test on two landing pages selling the same product. The first one (we’ll call it A, or the control) does not have 3D product images. The second one (we’ll call it B, or the variation) has them.
The conversion rate in terms to “Add to Cart” is 7% for A and 9% for B. So should you just add 3D images to your product pages across the site?
No! Because all the visitors to your website have not seen page B and you can’t make assumptions about their preferences simply from observing the behavior of a much smaller sample size. Right? (PS: Don’t make assumptions in marketing or optimization…. being data driven is the way to go).
How to solve this little problem?
P-value comes to the rescue.
P-value will give you the probability that you have seen a 2 percentage point increase in the “Add to Cart” KPI for your variation (Page B) simply because of luck or other random factors.
The smaller the p-value, the greater the chance that adding the 3D images meaningfully contributed to the uplift in conversions and would likely apply to all the visitors coming to your website.
You rarely need to calculate the p-value for your tests. This is done back-end by your A/B testing engine.