The Complete Guide to Multivariate Testing in 2021
November 4, 2020 –
Multivariate testing has been proven to increase conversions for websites with high traffic. Finding the right combination of elements on a page works. Or high traffic websites won’t be doing it.
But should you run multivariate tests?
There are benefits and downsides to running multivariate tests. In this article, we will take you through:
- what multivariate testing is,
- its benefits,
- its downsides,
- how to effectively run an MVT test,
- an example of an MVT test powered by our tool.
Let’s dive in!
What Is Multivariate Testing?
Multivariate testing (MVT testing) is a technique for testing multiple variables on a web page in different combinations.
For example, you want to test two different headlines, two images and two button colors on the page. Your MVT test will look like this:
- Variation 1 = Headline 1 + Image 1 + Color 1
- Variation 2 = Headline 1 + Image 1 + Color 2
- Variation 3 = Headline 1 + Image 2 + Color 1
- Variation 4 = Headline 1 + Image 2 + Color 2
- Variation 5 = Headline 2 + Image 1 + Color 1
- Variation 6 = Headline 2 + Image 1 + Color 2
- Variation 7 = Headline 2 + Image 2 + Color 1
- Variation 8 = Headline 2 + Image 2 + Color 2
In the above MVT test, you’re testing different elements (headlines, color and image) at once in different combinations to find the best one.
The Difference Between A/B Testing and Multivariate Testing
It may seem that MVT testing is a more advanced form of A/B testing to the untrained eye. But both types of testing are very different.
The difference between A/B testing and multivariate testing lies in the number of elements being tested at once.
A/B testing usually tests one element. For example, you’re testing adding an image to your web page. Your experiment will have:
- Control = No image
- Variation 1 = Image
You’re only testing the image on your website. And you can have as many variations as possible in A/B testing as in A/B/N testing. But it still won’t be an MVT test, as you’re only testing one element.
- A control with no pricing information,
- Variation 1 with a low starting price,
- Variation 2 with a high starting price.
While their test has two variations and a control, it is still an A/B test. This is because it is testing one element: adding pricing information.
Multivariate testing on the hand involves combining multiple variables in different combinations and testing them at once.
When Should You Use A/B/N Testing vs Multivariate Testing?
When you want to measure the effect of a significant change, use A/B testing. Multivariate testing is used for measuring the gradual effect of multiple small changes at once.
For example, you want to test if adding a headline to your landing page will improve conversions. You can do an A/B test like Split Base, a Convert Partner Agency. They ran an A/B test to determine if adding a benefit-driven headline will increase conversions.
PS: It did! By 27%.
For multivariate testing, you are testing granular changes in combination. Like our example above, we are testing different headlines, images, and colors, all combined in different variations.
The Benefits of Multivariate Testing
Multivariate testing has an undeniable impact on conversion rates and revenue. Think about it. If it didn’t, nobody would talk about using it!
One benefit of MVT testing is that it allows you to find out how slight changes on your website affect conversion rates. Slight changes like button color can have a tremendous impact on your conversion rate.
Multivariate testing enables you to measure the interaction effect between these small changes. You can measure the compounded effect of different small changes and see how that affects your conversation rate.
How Do You Run a Successful Multivariate Test?
- In Convert Experiences, select “New Experience”, then “Multivariate”, and name your experience:
- Make the variation
After the URL of the page you use for the test loads, the first variation is ready to be edited. You can edit the copy, add new visuals, you name it. In the example below, we’ve:
– Clicked on an element to change (it’s highlighted in orange)
– Selected an action in the menu i.e. changing the image source
- Name the variation
This is pretty self-explanatory. You’ll want to pick something that will stand out to you in reporting later on.
- Define the goals and audiences
Go to the Experiment Summary, and set up your Audience and Experiment Goals.
There you have it! Your experience is up and running.
For all the steps to set up a solid multivariate test, check our Support resource here.
What Are the Downsides of Multivariate Testing?
There are reasons only high traffic websites with big budgets run multivariate tests. Let’s talk about the downsides of running an MVT test.
1. It requires a large sample size
If you lack a large sample size, it will take a very long time for your MVT test to reach statistical significance. Because of the different combinations, you will need to send traffic to multiple variations. This makes it impossible for most websites to run MVT tests as they lack the traffic.
2. MVT tests take time
Even with enough sample size, multivariate testing takes time to run. Multiple combinations equal multiple variations that will time to reach statistical significance.
This time factor makes it difficult for even high traffic sites to justify running a multivariate test. Would you wait 3 months for your MVT test to tell you that Variation A with its combination is the best with a 7% increase in conversion rate? Or would you rather run 3-6 A/B tests that add a whopping 25% increase in conversion rate?
3. It’s expensive to run MVT tests
Each experiment you run has a cost. MVT tests can get very expensive quickly. Besides the time spent monitoring multiple variations, you still lose money on poor combinations. Because you have to let your variations run for long, you will lose time and resources spent on making poor combinations.
Multivariate Testing Examples
In this example, a customer is experimenting with multiple combinations of the scroll area and jump link on their website. This MVT test has multiple combinations of the two elements they’re testing.
What are some elements in this example that stand out to you? Let us know in the comments.
What Are the Best A/B and Multivariate Testing Tools Available?
There are scenarios where it makes sense to run a multivariate test. You may have a high traffic website, but you still need to decide if multivariate testing is right for your website. Evaluate the cost, traffic and need for a multivariate before you run one.