Using Meta-Analysis to A/B Test a New Layout or Theme Across Multiple Products or Websites

Karim Naufal
By
April 28, 2021 ·

If you’re an eCommerce brand, sooner or later, you’re likely going to come across this scenario: Experimenting with a new layout or website theme that applies across different products or websites. To find out if the new element works, you could A/B test each product or website individually…

That should give you some ideas, but what if you wanted to mathematically determine the overall macro-effect of the new theme or layout? It turns out, there is a way, it’s a technique called Meta-Analysis.

What is a Meta-Analysis?

Merriam-Webster defines Meta-Analysis as:

A quantitative statistical analysis of several separate but similar experiments or studies in order to test the pooled data for statistical significance.

So how does this apply to testing a layout or theme change?

In the following paragraphs, I’ll walk you through the best possible way to do it.

AB testing layouts

The best way to test one theme or layout change across multiple products is to run individual A/B tests for each of your products/sites and then reason in terms of the average effect size.

Let’s see how this works in practice:

  1. To make the following calculations valid, one should plan for the same sample size for each A/B test (with 1 variant per test) and ensure that all products tested are independent (one does not influence the other).
    plan for the same sample size for each A/B test
  2. For the pre-test analysis, find the product that gets the minimum weekly visitors and then choose a sample size per product that is slightly inferior to that. That way, we can ensure that they all can meet this criterion.
  3. Then, run a normal pre-test analysis with that amount of weekly visitors and choose the MDE that makes the most sense and that we know has a high likelihood of being achieved:
    run a normal pre-test analysis with that amount of weekly visitors and choose the MDE
    Pre-test analysis hypothetical example for a 3-product test
    Pre-test analysis in Convert’s A/B Testing Significance Calculator
    Pre-test analysis in Convert’s A/B Testing Significance Calculator
  4. Run the test and configure the traffic allocation to 100% for the product that gets the least amount of visitors, and a proportional allocation for the other products, so that they all end up testing the same traffic over time.
    traffic allocations across tests
  5. Wait for each test to accumulate the predetermined number of samples (as calculated in our pre-test analysis) for the chosen MDE.
  6. When this is done, run the Meta-Analysis. To do so, compute all individual Z values per test to get the overall Z using the formula below (where k is the total number of products/sites):
Formula extracted from “Statistical Methods in Online A/B Testing”
Formula extracted from “Statistical Methods in Online A/B Testing” by Georgi Z. Georgiev, see “Chapter 13 – Miscellaneous Topics”, “Meta-analysis of A/B test results”.

Using this method, we can get the overall p-value and see if the observed effect was significant or not.

Another way would be to deal with the average conversion rate and consider the total sample sizes across all tests. Then, get the overall Z value from it.

Definitely an advanced use case, but don’t get discouraged! It’s completely doable, following the steps I laid out above.

Have you tried this out or willing to try it? Reach out to me on LinkedIn and let me know what you think of this method and how it worked for you.

Tool Features Ecommerce
Tool Features Ecommerce
Mobile reading? Scan this QR code and take this blog with you, wherever you go.
Originally published April 28, 2021 - Updated November 06, 2024
Written By
Karim Naufal
Karim Naufal
Karim Naufal
Accomplished technologist with a background in EECS.
Edited By
Carmen Apostu
Carmen Apostu
Carmen Apostu
Head of Content at Convert

Start Your 15-Day Free Trial Right Now.
No Credit Card Required

You can always change your preferences later.
You're Almost Done.
Convert is committed to protecting your privacy.

Important. Please Read.

  • Check your inbox for the password to Convert’s trial account.
  • Log in using the link provided in that email.

This sign up flow is built for maximum security. You’re worth it!