Blog /A/B Testing

The Complete Guide to Multivariate Testing in 2021

04th Nov 2020 –

Multivariate testing has been proven to increase conversions for websites with high traffic. Finding the right combination of elements on a page works. Or high traffic websites won’t be doing it.

But should you run multivariate tests?

There are benefits and downsides to running multivariate tests. In this article, we will take you through:

Let’s dive in!

What Is Multivariate Testing?

Multivariate testing (MVT testing) is a technique for testing multiple variables on a web page in different combinations.

For example, you want to test two different headlines, two images and two button colors on the page. Your MVT test will look like this:

  • Variation 1 = Headline 1 + Image 1 + Color 1
  • Variation 2 = Headline 1 + Image 1 + Color 2
  • Variation 3 = Headline 1 + Image 2 + Color 1
  • Variation 4 = Headline 1 + Image 2 + Color 2
  • Variation 5 = Headline 2 + Image 1 + Color 1
  • Variation 6 = Headline 2 + Image 1 + Color 2
  • Variation 7 = Headline 2 + Image 2 + Color 1
  • Variation 8 = Headline 2 + Image 2 + Color 2

In the above MVT test, you’re testing different elements (headlines, color and image) at once in different combinations to find the best one.

The Difference Between A/B Testing and Multivariate Testing

It may seem that MVT testing is a more advanced form of A/B testing to the untrained eye. But both types of testing are very different.

The difference between A/B testing and multivariate testing lies in the number of elements being tested at once.

A/B testing usually tests one element. For example, you’re testing adding an image to your web page. Your experiment will have:

  • Control = No image
  • Variation 1 = Image

You’re only testing the image on your website. And you can have as many variations as possible in A/B testing as in A/B/N testing. But it still won’t be an MVT test, as you’re only testing one element.

An outstanding example of this comes from iProspect, a Convert Partner Agency. The agency ran a test that featured pricing page information. Their experiment featured:

  • A control with no pricing information,
  • Variation 1 with a low starting price,
  • Variation 2 with a high starting price.

While their test has two variations and a control, it is still an A/B test. This is because it is testing one element: adding pricing information.

Multivariate testing on the hand involves combining multiple variables in different combinations and testing them at once.

When Should You Use A/B/N Testing vs Multivariate Testing?

When you want to measure the effect of a significant change, use A/B testing. Multivariate testing is used for measuring the gradual effect of multiple small changes at once.

For example, you want to test if adding a headline to your landing page will improve conversions. You can do an A/B test like Split Base, a Convert Partner Agency. They ran an A/B test to determine if adding a benefit-driven headline will increase conversions.

PS: It did! By 27%.

For multivariate testing, you are testing granular changes in combination. Like our example above, we are testing different headlines, images, and colors, all combined in different variations.

The Benefits of Multivariate Testing

Multivariate testing has an undeniable impact on conversion rates and revenue. Think about it. If it didn’t, nobody would talk about using it!

One benefit of MVT testing is that it allows you to find out how slight changes on your website affect conversion rates. Slight changes like button color can have a tremendous impact on your conversion rate.

Multivariate testing enables you to measure the interaction effect between these small changes. You can measure the compounded effect of different small changes and see how that affects your conversation rate.

How Do You Run a Successful Multivariate Test?

  1. In Convert Experiences, select “New Experience”, then “Multivariate”, and name your experience:

    Multivariate test setup steps

  2. Make the variation

    After the URL of the page you use for the test loads, the first variation is ready to be edited. You can edit the copy, add new visuals, you name it. In the example below, we’ve:

    – Clicked on an element to change (it’s highlighted in orange)
    – Selected an action in the menu i.e. changing the image source

    Multivariate test setup Convert Experiences make variation

  3. Name the variation

    This is pretty self-explanatory. You’ll want to pick something that will stand out to you in reporting later on.

  4. Define the goals and audiences

    Go to the Experiment Summary, and set up your Audience and Experiment Goals.

There you have it! Your experience is up and running.

For all the steps to set up a solid multivariate test, check our Support resource here.

What Are the Downsides of Multivariate Testing?

There are reasons only high-traffic websites with big budgets run multivariate tests. Let’s talk about the downsides of running an MVT test.

1. It requires a large sample size

If you lack a large sample size, it will take a very long time for your MVT test to reach statistical significance. Because of the different combinations, you will need to send traffic to multiple variations. This makes it impossible for most websites to run MVT tests as they lack traffic.

2. MVT tests take time

Even with enough sample size, multivariate testing takes time to run. Multiple combinations equal multiple variations that will time to reach statistical significance.

This time factor makes it difficult for even high-traffic sites to justify running a multivariate test. Would you wait 3 months for your MVT test to tell you that Variation A with its combination is the best with a 7% increase in conversion rate? Or would you rather run 3-6 A/B tests that add a whopping 25% increase in conversion rate?

3. It’s expensive to run MVT tests

Each experiment you run has a cost. MVT tests can get very expensive quickly. Besides the time spent monitoring multiple variations, you still lose money on poor combinations. Because you have to let your variations run for long, you will lose time and resources spent on making poor combinations.

Multivariate Testing Example in Convert Experiences

In this example, a customer is experimenting with multiple combinations of the scroll area and jump link on their website. This MVT test has multiple combinations of the two elements they’re testing.

multivariate test example Convert Experiences

Now that you understand what multivariate testing is and how to successfully set up a multivariate test, let’s break down a few different platforms you can choose from. Many of the top A/B testing tools on the market offer multivariate testing and A/B testing in one software. So which one is best suited to your needs?

If the multivariate testing method is a deciding factor for you, you’re in the right place. The list below gives you the 9 top players for multivariate testing on the market.

What Are the Best A/B and Multivariate Testing Tools Available?

Here’s a look at the multivariate testing tools preferred by CRO experts, along with the multivariate testing methods they offer:

  • Convert Experiences – Full Factorial
  • AB Tasty – Full Factorial
  • Google Optimize & Optimize 360 – Hybrid approach that’s neither Full Factorial nor Fractional
  • Adobe Target – Full Factorial and Taguchi
  • Kameleoon – Full Factorial and Fractional Factorial
  • Optimizely – Full Factorial, Partial and Taguchi
  • Sitespect – Full Factorial and Fraction Factorial
  • VWO – Full Factorial
  • Webtrends Optimize – Full Factorial and Fractional Factorial

Still not sure what multivariate testing tool to pick? Check this complete breakdown of the best Multivariate Testing tools.


There are scenarios where it makes sense to run a multivariate test. You may have a high traffic website, but you still need to decide if multivariate testing is right for your website. Evaluate the cost, traffic and need for a multivariate before you run one.

Originally published November 04, 2020 - Updated September 24, 2021

Start 15-Day Free Trial Of Convert Experiences.

Quick form submission
Quick submission.
No credit card needed
No credit card needed.
You can always change your preferences later.
Check Icon
You're Almost Done.
Managing Marketing Team
Managing Tech Team
Hypothesizing Experiments
Coding and QA of Experiments
Convert is committed to protecting your privacy.

You've successfully submitted your details to start your Convert Experiences 15-day free trial.

We are setting up your account for you. You'll receive a welcome message to your registered email with your login access. Please explore the app on desktop for an optimal experience.