A/B Testing vs. Multivariate Testing vs. Multipage Testing
January 21, 2019 –
When it comes to running CRO experiments, you can choose among four types of tests: A/B tests, split tests, multivariate tests, and multipage tests.
Each of these tests has its own strengths and limitations.
And unless you choose the right test for your experiment, making conclusive data-backed CRO decisions can get difficult.
So let’s understand what these different optimization testing techniques are and how they compare with each other … and how you can be sure of choosing the right test each time you run an experiment.
What Is A/B Testing
A/B testing is a way to test changes to one or more elements on a page.
With A/B testing, you can add an element to a page or remove it or change it, and see how the addition, elimination, or change impacts the conversion rate of the page.
These elements could be any graphic or text items from the page — for example, its CTA or headline copy or the hero image.
In A/B testing, you create one or more exact copies of the original page with the only non-identical elements being the elements to test.
And when you test these copies (called the challengers) against the original one (called the control), you can see how the alterations impact the conversion rate.
Split URL Testing
There is another form of testing that is often clubbed with A/B testing, but is fundamentally different from A/B tests.
They are known as Split URL tests or split tests.
In split testing the traffic a page receives is split down the middle and directed to two completely different versions of the same page in order to identify the version that performs better against the specified goal.
Generally these versions are hosted at different URLs but the end goal is to boost conversions for a particular action, like say form fills or adds to cart.
Split URL testing it used to test totally different designs or versions of a page.
Here the goal is to find the one that performs the best.
So, if you want to test two radically different ideas, then you should go with split URL testing. For instance, if you’re looking to test two homepage designs, then this would be the right technique to use.
An A/B test example
SplitBase conducted its research and hypothesized that adding a benefit-driven headline to the landing page (which had no headline originally) could help rake in more sales.
SplitBase decided to run an A/B test and test a page with a headline against the original page. SplitBase also decided to test two versions of the headline.
So its experiment using a simple A/B test ended up with three versions:
- Version A (the control without a headline)
- Version B (the challenger with a headline)
- Version C (the challenger with a variation of the headline)
Traffic was split equally in all the three versions (~ 33% each) to see which version got the most sales.
As you can see, an A/B testing experiment can very well become an A/B/C/D…n testing experiment.
A/B testing use cases
A/B testing has a very specific use case:
Testing very similar designs (except for the change in the test elements).
This is the most common use case of A/B tests where you experiment with just one element and see how the change impacts the conversion rate.
If you hypothesize that a certain element can make a positive impact on the conversion rate, you should go with this testing technique. For instance, you can use this optimization testing technique to see the impact of a change to your homepage’s CTA.
The pros of A/B testing
- A/B tests are simple to implement: Whether you test two very similar or two entirely dissimilar experiences, setting up an A/B test is quite straightforward. They usually aren’t too time or resource-intensive.
- A/B tests need lesser traffic (as compared to multivariate) tests to yield definitive results: If you’ve a website where you get enough traffic to send each variant about 10k visitors, you should be able to use A/B tests to find statistically significant results to make design, UI, or copy changes to your website. This is especially true when you run A/B tests with just the control and a challenger (or, say, two challengers) as you can still send good volumes of test traffic to each.
- A/B tests don’t need too long to run. Usually, A/B tests can yield statistically significant results in 2-3 weeks and don’t need to run for months and months on end.
The limitations of A/B testing
- A/B tests usually offer limited insights: A/B tests of similar experiences (where just one element is varied) usually run two versions. Which means, the most that a marketer can learn from the experiment is which of the three versions (1 original + 2 challengers) did the best. That’s all — insights about just three possibilities. If you want to test more variants (for example, if you want to test 5 versions of the CTA button), you’ll need enough traffic to produce statistically significant decisions even after splitting the traffic for that many variations.
- A/B tests are mostly only suitable for making high-level website decisions: Because A/B tests are great at giving objective answers as to which version is the best, they work great at taking high-level decisions. So you can decide upon the approach you want to take with your website or landing page’s design or UI or copy with A/B tests. BUT if you want to finetune the more granular elements of design, UI, or copy, then you’ll need to likely run several A/B tests, which can be time consuming and labor extensive.
Convert is a powerful CRO solution that you can use to power all your A/B testing experiments and discover the most effective CTA buttons, headline copies, hero images, color schemes, logo placements and more to use on your website.
Convert allows for both conventional A/B/n tests and split URL tests through its Visual editor and the more robust code editor.
What is Multivariate Testing (MVT)
Multivariate testing or MVT is a way to test a GROUP OF ELEMENTS on a page.
In multivariate testing, you add, delete, or change a group of elements on a page thereby creating multiple versions with multiple tweaks, and see how the addition, elimination, or change to the group of elements impacts the conversion rate.
A group of elements in multivariate testing MUST comprise at least two elements. And each element must have two variants. Which means in multivariate testing, you’ll have at least four versions to test.
Here’s the breakdown of a simple multivariate test:
Group of elements to test: Element 1, Element 2.
Variants of Element 1:
Element 1 Variant A
Element 1 Variant B
Variants of Element 2
Element 2 Variant A
Element 2 variant B
The maths of multivariate testing:
The number of versions to test = The number of elements in a group * The number of variants of each element
Multivariate testing use cases
Companies usually use multivariate testing as an optimization technique to get more granular with their testing. Let’s get clear right off the bat. Multivariate testing is akin to “running several A/B/N tests at the same time”. This is why the experimentation cycle is prolonged.
But thanks to the fact that you can make changes to clusters of elements for each variant, you can go from identifying a larger problem/conversion roadblock (which you do with A/B testing) to fine tuning the visitor experience for a particular page.
A multivariate test example/case study
Let’s create a hypothetical multivariate testing extension of the case study example from above.
Now, because this eCommerce store has already used an A/B test to find a winning version of its flagship product page, it can now follow up with a round of multivariate testing to experiment with different combinations of the elements on the winning design to further improve its conversion rate.
So, for example, in its multivariate test, it could change the color of the CTA button, the hero image, and the CTA button copy and run it against the original to find a “more” winning combination.
The pros of multivariate testing
- Multivariate tests are great at making micro-conversion decisions: With multivariate tests, you can experiment with very many combinations of design, UI, and text changes. Which means you can take all the guesswork out of the process of “making tweaks” to your website for better conversion rates. You can actually validate even your most minute choices.
- Multivariate tests give deeper insights about how the elements play together: Because multivariate testing lets you test combinations of elements, you can see how your different combinations impact your users’ choices (rather than just testing stand-alone elements as in the case with A/B tests).
The limitations of multivariate testing
- Multivariate tests need a lot of traffic: If you look at the maths of multivariate testing, you’ll see that the number of versions to test gets huge depending on the number of elements to test and their variations. For example, if you’d like to use a multivariate test and test the different combinations of a CTA button (in three colors: one control and two challengers and three copies: one control and two challengers), then you’ll need to create 3*3, i.e., nine versions. Having enough traffic to split among so many versions is a challenge even for high-traffic websites.
- Multivariate tests can get difficult to plan: Planning a meaningful multivariate test is tricky because of the sheer number of combinations (of the different elements on your page) you can test. Choosing between combinations like a headline, a hero image, and the hero area CTA or the headline, USP, and the header can take a lot of effort. Also, testing too many elements in a multivariate test can skew the results.
- Multivariate testing isn’t suitable for taking definitive high-level decisions: Because multi-variate testing tests combinations of elements, you can’t always conclusively tell why a certain combination won. Which means taking a radical CRO decision is not always possible with multivariate testing. This reduces multivariate tests to powering just the more granular CRO decisions like testing placements/graphics/copy of the elements in the version you already know to work well.
Multivariate testing tools
With Convert, you can easily set up multivariate tests on your website and see how the different combinations of the elements impact your conversion rate.
Multivariate Testing vs. Split Testing: Which Should You Use?
Check out the multivariate testing vs. split base testing (or A/B testing) comparison table down below to see when you should use either of the two techniques.
What Is Multipage Testing & Funnel Testing
Multipage testing is a form of experimentation where you test changes to particular elements across multiple pages.
There are two ways to go about this.
You either take all the pages of your sales funnel and create new versions of each. This new set of pages make your “challenger” sales funnel and you then run it against the original sales funnel (or the “control”). This is called “Funnel Testing”.
Or you can test how the presence/absence of recurring element(s), like security badges, can impact conversions across an entire funnel. This is the classic or conventional multipage testing.
Multipage testing and Funnel testing use cases
With multipage testing, you are at the liberty to gauge the impact of changes made on elements across a full buyer’s journey.
Businesses that choose to invest in funnel testing yield tons of high level insights.
For example, you can use funnel testing to find the most effective:
- You can run funnel tests to test different voices and tones on your pages — for instance, if you’re an anti-virus service provider, you could test a fear-inspiring copy in your sales funnel against your neutral or trust-inspiring copy — and see what messaging resonates the best with your visitors and gets the most sales.
- Design philosophies. You can experiment with different design theories and see which one works the best.
- Sales/support strategies. You can design a challenger sales funnel version where you offer live chat support on a few select pages of your funnel and see how the live chat enabled funnel compares to the regular sales funnel (without live chat support). Likewise, you can test a host of sales/support strategies and see how you can assist more conversions across your sales funnel.
A multipage test example/case study
Again, let’s build a hypothetical extension to the same case study example from above.
Now, because this particular website is an online store, its most basic sales funnel is:
The product page > The add to cart page > The checkout page
So, to run a multipage test, this store would need to create a variation for each of these pages and create a challenger funnel such as:
The product page version A > The add to cart page version A > The checkout page version A
In its challenger funnel, the e-retailer might decide to test how the addition of the legend “Trusted by 10,000 Professionals” translates into conversions for each stage of the buyer’s journey. Or, it might choose to show very different pages (similar to split testing with two totally unique pages).
If they are feeling ambitious though, they can go in for funnel testing and actually play around with copy, a cart abandonment strategy and more.
You get the idea, right?
The pros of multipage testing
- Multipage funnel testing helps plug leaks in the sales funnel: If your Google Analytics data shows major drop offs on certain pages of your sales funnel, you can set up a multipage test to find what can reduce them and get more conversions.
- Multipage tests can give deeper insights into user behavior: With multipage testing, you can identify the different elements of your control and challenger funnels that your users engage with the most. You can use such insights to take even more informed design, copy, and user experience decisions.
The cons of multipage testing
- Multipage experiments can last for months. Because B2B has long buying journeys (often stretched over months), B2B multipage tests can run for months on end.
- Gaining insights from a multipage test’s results can be tricky: If your challenger version experiments with too many variables (for example, if you experiment with both the design and copy in a challenger), you might not be able to tell if the design caused more conversions or the copy. Creating a bunch of challengers (for example, one for testing the design and one for testing the copy) will need you to split your traffic in three. Unless you get thousands of visitors to your website, you might not have enough traffic to get statistically significant results when you run too many such variations.
Multipage testing tools
With Convert, you can easily set up multipage tests or funnel experiments and see how different changes impact conversions across pages.
A/B testing vs. Multivariate testing vs. Multipage testing
What it is:
In A/B testing, you test two (or more) very similar versions of the same page/ad/asset to find the best-performing variant.
When to use:
At its core A/B testing is the method optimizers use to identify the biggest roadblocks to conversion. You have to create a hypothesis that pinpoints the (purported) reason for low conversions and tests how changing/tweaking an element can mitigate the issue.
Through A/B testing you can take high level design and copy decisions.
For example, a high bounce rate from a home page is generally indicative of the fact that visitors do not get the “value” the product or solution offers. A/B/N tests with different versions of a value proposition headline are common.
Ease of Gaining Insights:
A/B tests need about 10K visitors to each variant and around a 1000 conversions to yield insights. But they are relatively quicker to conduct, when compared to multivariate tests.
What it is:
In multivariate testing, you test different combinations of elements on a page (that you already know to perform well).
When to use:
Multivariate tests can be used to make more granular decisions. For example, if you’ve a high-performing homepage that you’d like to further optimize, you can use multivariate tests to experiment with a group of elements on your homepage.
Think of it as plugging the smaller leaks after the deluge has been stemmed by A/B testing.
Ease of Gaining Insights:
Multivariate tests are extremely complicated owing to the inclusion of several elements over several variants. So both the traffic needs and the time to reach a conclusive outcome are compounded.
What it is:
In multipage testing, you test different buying journeys to find the most optimal one. Multipage testing is similar to running a bunch of A/B/N and multivariate tests together on multiple pages across the sales funnel.
When to use:
Multipage tests or funnel tests can be used to make high-level decisions across the entire sales journey.
For example, you can test a personalized sales journey against a regular one.
Ease of Gaining Insights:
Multipage tests with limited versions (say, 2) can yield significant results relatively quickly.
Comparing A/B testing, Multivariate and Multipage Testing
An experiment can give definitive results only when it uses the right test format.
As a synopsis, this is what you should keep in mind:
- A/B/N Tests – You generally test one significant element that has been identified as a conversion roadblock in your hypothesis across one, two or more versions (variants).
- Split Tests – Split Tests focus on hosting two completely different pages and experiences for visitors on two separate URLs while evaluating them for the same goal of conversions. Through split tests you can test two completely different set-ups – like hosting platforms and themes against each other.
- Multivariate Tests – MVTs are conducted on winners of A/B tests. In Multivariate testing you make changes to clusters of elements of a page, each unique or tweaked cluster becoming a variant or version of its own. It’s difficult to attribute higher conversions for a MVT variant to a particular element, but optimizers can test granular modifications to copy and design.
- Multipage Tests – Running a multipage test is similar to running multiple A/B/N tests together on a bunch of pages of the sales funnel. Alternatively, you can test entirely different sales funnel experiences (for example, sending half of the traffic to the control version Page 1> Page 2> Page 3 and sending the other half to the challenger version Page 1 variant> Page 2 variant > Page 3 variant with the challenger being quite dissimilar to the control.)
Choosing the right test for an experiment largely depends on getting the hypothesis right. Here are some CRO tools to help you with writing a clear hypothesis each time you run an experiment.
Also, when you’re ready to start with A/B, multivariate, or multipage tests for your experiments, check out Convert. Convert Experiences can handle all the optimization experiments that you can think of with ease for high traffic websites, even in data privacy sensitive industries.