Today’s Earth Day. We’re Planting 5 Trees for Every Trial Sign Up on Our Site. Optimize for a Better Tomorrow.
5 Ways Digital Marketers Use Call Tracking to Grow Their Business

Using Meta-Analysis to A/B Test a New Layout or Theme Across Multiple Products or Websites

April 28, 2021 –

If you’re an eCommerce brand, sooner or later, you’re likely going to come across this scenario: Experimenting with a new layout or website theme that applies across different products or websites. To find out if the new element works, you could A/B test each product or website individually…

That should give you some ideas, but what if you wanted to mathematically determine the overall macro-effect of the new theme or layout? It turns out, there is a way, it’s a technique called Meta-Analysis.

What is a Meta-Analysis?

Merriam-Webster defines Meta-Analysis as:

A quantitative statistical analysis of several separate but similar experiments or studies in order to test the pooled data for statistical significance.

So how does this apply to testing a layout or theme change?

In the following paragraphs, I’ll walk you through the best possible way to do it.

AB testing layouts

The best way to test one theme or layout change across multiple products is to run individual A/B tests for each of your products/sites and then reason in terms of the average effect size.

Let’s see how this works in practice:

  1. To make the following calculations valid, one should plan for the same sample size for each A/B test (with 1 variant per test) and ensure that all products tested are independent (one does not influence the other).
  2. For the pre-test analysis, find the product that gets the minimum weekly visitors and then choose a sample size per product that is slightly inferior to that. That way, we can ensure that they all can meet this criterion.
  3. Then, run a normal pre-test analysis with that amount of weekly visitors and choose the MDE that makes the most sense and that we know has a high likelihood of being achieved:
    Pre-test analysis hypothetical example for a 3-product test
  4. Run the test and configure the traffic allocation to 100% for the product that gets the least amount of visitors, and a proportional allocation for the other products, so that they all end up testing the same traffic over time.
  5. Wait for each test to accumulate the predetermined number of samples (as calculated in our pre-test analysis) for the chosen MDE.
  6. When this is done, run the Meta-Analysis. To do so, compute all individual Z values per test to get the overall Z using the formula below (where k is the total number of products/sites):
Formula extracted from “Statistical Methods in Online A/B Testing” by Georgi Z. Georgiev, see “Chapter 13 – Miscellaneous Topics”, “Meta-analysis of A/B test results”.

Using this method, we can get the overall p-value and see if the observed effect was significant or not.

Another way would be to deal with the average conversion rate and consider the total sample sizes across all tests. Then, get the overall Z value from it.

Definitely an advanced use case, but don’t get discouraged! It’s completely doable, following the steps I laid out above.

Have you tried this out or willing to try it? Reach out to me on LinkedIn and let me know what you think of this method and how it worked for you.

Start 15-Day Free Trial Of Convert Experiences.

Quick form submission
Quick submission.
No credit card needed
No credit card needed.
You can always change your preferences later.
Check Icon
You're Almost Done.
Managing Marketing Team
Managing Tech Team
Hypothesizing Experiments
Coding and QA of Experiments
Convert is committed to protecting your privacy.

You've successfully submitted your details to start your Convert Experiences 15-day free trial.

We are setting up your account for you. You'll receive a welcome message to your registered email with your login access. Please explore the app on desktop for an optimal experience.