Blog /A/B Testing

Most Popular Multivariate Testing Tools in 2021

08th Sep 2021 –

So you’re currently running A/B tests and looking to scale out your testing program.

You have good traffic, you’re seeing great results and now you want to run far more tests and get that data asap.

But how can you start testing faster? Or better still, how can you run more tests and get more information during the same time period?

That’s where multivariate testing comes in. It allows you to test multiple elements all at once, to see which particular version works best.

In this guide, we’re going to break down just what MVT is, how it works, what you need in place to use it and break down a selection of CRO tools that offer MVT functions.

Everything from their price and features, to how they do business so you can make the right choice when deciding which multivariate tool to get.

So let’s dive into it…

What Is Multivariate Testing?

The best way to understand multivariate testing is to compare it to other testing methods and see how it differs.

So let’s look at an A/B and an A/B/n test and how they compare, and the subtle differences in multivariate testing.

With an A/B test, we test a single variation page vs the original control design. Maybe we have an idea of how to improve the headline, so we create that variation and then see which performs best.

Your A/B test will look like this:

Headline 1 (Control) vs Headline 2 (The variation idea)

Another test we could run is an A/B/n test.

This is where we can test multiple variations of that one element at the same time, to find the overall winner.

For example, we could have a heap of headline ideas and are not sure which will work. But because we have the traffic to run a few test variations and still get accurate results, we decide to test multiple headline ideas to see which performs best.

Headline 1 (Control) vs Headline 2 vs Headline 3

Once we find the winning headline variation, we might then test a bunch of hero shot images, and then maybe a bunch of CTA angles.

As you can guess, this all takes time, right?

In theory, we could run just an A/B test where we change a bunch of things all in one go and see if that improves the metrics.

Here you can see our old home page design and our new one that gets more lift. But what changes caused the improvement?

We tend not to do this with most A/B testing unless we’re looking at making a radical change i.e throwing a hail mary to try and find a new winner because all other test ideas are stagnating or are not making any impact.

So why don’t we recommend doing radical changes like this for normal campaigns and what does it have to do with multivariate testing?

We A/B test one element at a time because if we make huge changes and the new page wins or fails, we’re not sure which of those changes caused that lift.

Multivariate testing or MVT allows us to test multiple things on the same page all at once, while also tracking each of those changes and how they affect the result.

Not only can we test multiple variations of elements, but we can even run different combinations of them until we find the ultimate winner.

Let’s say that we wanted to improve our home page and we have a few ideas of what to test to see the most lift:

  • The headline
  • The hero shot image
  • The CTA.

With a multivariate test, we could not only test every headline idea, hero shot image option, and CTA angle, but we can also test each combination of those variations to find the winning combination:

Headline 1 + Hero image 1 + CTA 1 vs

Headline 1 + Hero image 2 + CTA 1 vs

Headline 1 + Hero image 1 + CTA 2 vs 

Headline 2 + Hero image 1 + CTA 1 vs

Headline 2 + Hero image 2 + CTA 1 vs

Headline 2 + Hero image 2 + CTA 2

Multivariate testing can help you to find out the best combination of elements (often much faster than standard sequential A/B tests), while also understanding what caused those changes.

When to Use Multivariate Testing?

There are a few times that you might want to run an MVT test instead of an A/B test.

1. When You Have High Traffic and Want to Ramp Up Your Testing Program

Traffic is the major crux of multivariate testing. You need a high volume of traffic to be able to test each change and trust your test results.

But if you have the traffic and you want to scale up your testing to get more data and make more decisions, then MVT will allow you to run multiple ideas all at once.

Mmmm data!

2. When You Want to Increase Lift and Have a Few Ideas but Don’t Want to Wait to Test Them One by One

Some people think that you should only use MVT to refine current web pages with minor changes, due to the time investment, but it’s not true. Sure, you can use it for that, but you can also use it for broader ‘what if’ tests of larger elements.

(Maybe you’re not sure which element will provide the best lift and so you want to try throwing some smartly hypothesized ideas at the wall to see which works.)

If you have the volume of traffic to run an MVT, then it’s a no-brainer to not try and find broad page combinations far faster. Just be sure to measure the effects of that traffic AFTER the test from each variation also, as this will give you further insight into your audience and why it worked.

For example, on Facebook, you can set up basic multivariate test ads, by adding in variations of a thumbnail image, headline and body copy, etc.

The test will run and tell you which variation of that ad gets the highest CTR.

Now if your goal was just clicks, then this ad would win, right?

But if all that traffic then bounces off the page you send them to, then all you’ve found is a variation that causes that audience to click, not one that gets them to stay and convert.

It’s the same when running an MVT test on a page, especially if you’re using it as a broader idea gathering method. Check what happens on the page but also what happens after they click.

3. When Refining a Previous Test to Get Incremental Lift

You can also use MVT to measure and improve small refinements. In fact, a lot of agencies like to use them to eke out even more lift on current winning campaigns and test small changes.

Let’s say that you’ve already A/B tested a page multiple times to find a winning control, but you want to try and get as much lift as possible, even if it’s just 0.5%.

You can run MVT tests on that page to test small changes such as color variations of CTAs, etc. (Because at 100,000 visitors a month, every 0.5% helps is noticeable!)

4. When Searching for the Maximum Possible Lift From a Combination of Ideas (Interaction Effects)

Sometimes, a consecutive A/B test can give us winners on paper, but when we combine them, they are not always the highest performing option for our page.

Here’s an example:

  • You run a headline test and pick a winner and implement.
  • Then you run a hero shot image test on that same page with the winning headline variation and implement.
  • Then you run a CTA test on the same page with the previous winning elements and implement.

In theory, this should be the highest performing version of your page but weird things can happen because of interaction or lack thereof. Simply put, it’s possible for a losing element in one test to sometimes outperform your sequential A/B test winner when combined with another element variation.

How?

Well, maybe a losing headline from the first test provides a higher backend conversion rate when combined with the winners from the image and CTA test.

This can happen but you would never know this if you just ran A/B tests. Instead, you would have removed it from the options before you ever tested the other elements. This is why multivariate testing is so powerful. It allows you to find the highest overall page performance, even if it means that it’s taking a less-performing element from one part of the test.

Better still, it can even tell you which element helped the most…

5. When Searching for the Most Important Element on the Page for Your Audience, for Future Tests and Pages

MVT will not only show the winning combination of your different variations and elements, but it will also show you the element that provides the most lift.

Maybe your hero shot gets more lift, but you also see that out of all your elements on your variation page, it contributes most to your end result?

This will then give you some insight into where to test first for even further improvements, so that you can focus on hero shot improvements first, etc.

6. When Applying Machine Learning to Your Testing Process

Due to the nature of MVT requiring and tracking large amounts of data and interactions, it can actually be used as a great source to feed machine learning testing programs.

ML requires large data to work best, but once it gets it, it can actually start to see patterns and predict results and behaviors with less traffic over time. In fact, sites like Amazon are doing exactly this…

Multivariate Testing Examples

Multivariate Testing Example #1

Amazon used MVT and machine learning to adjust the layout of their landing pages, resulting in a 21% increase in purchase rate in just 7 days!

Source

Multivariate Testing Example #2

Booking.com uses multivariate testing to test out new unique features to specific audiences and then measures their performance before rolling them out sitewide.

Multivariate Testing Example #3

Microsoft saw a 40% increase in conversion rate for their SMB website, after running an MVT test on multiple page elements (8×4!)

Multivariate Testing Example #4

It’s not just the juggernauts who can use MVT either though. HawkHost saw a 204% increase in sales when they ran an MVT test on their homepage (testing their headline, subhead, and hero shot).

The Pros & Cons of Multivariate Testing

If multivariate testing is so powerful, why aren’t we all using it?

Well, there are a few pros and cons to it…

You see, the beauty of an A/B test is that you can test one idea fast, with a relatively small audience size. This means that most businesses can run a test even if they have fairly low traffic, and then see a result within 14-30 days.

But with a multivariate test, you need a much larger audience to run with, because you’re testing multiple things at once. (Especially so if you’re doing multipage testing or testing multiple combinations of variations.)

Let’s say that we wanted to test a headline on our home page:

  • Headline Control
  • Headline Variation

We’re just testing the single variation, so that’s 2 options total.

Now as a simple example, let’s say that we want to get 1,000 visitors to each version of the page (the variant and the control) so that we can be sure that the test results are trustworthy.

(Traffic needed can vary based on the page and its conversion rate, but let’s simplify here just so you get the idea.)

That means that we would need 2,000 visitors to run the test and trust the results, with 1000 visitors going to each page.

What about if we wanted to test:

  • 3 headlines
  • 3 hero images, and
  • 3 CTA’s?

That’s not 9 variants total,but actually 27 combinations of 9 variants (3x3x3).

This means that if we wanted 1000 visitors per variation, then our traffic requirement would be 27,000 visitors instead to run the test and trust the results (based on our oversimplification of 1,000 visitors being enough per page.)

In reality, you would probably need around 10,000+ visitors per variation to get accurate data, as traffic alone is not enough, you also need to see conversions.

This is why a lot of businesses don’t run MVT tests, simply because they don’t have the traffic to run them effectively.  In fact, most agencies recommend having around 100,000 monthly visitors before you start running MVT campaigns unless the page has a very high conversion rate already. And even then, they still recommend only running maybe 6 variations max, so as to be able to run them faster.

But if you do have that kind of traffic?

Well then MVT can allow you to perform the equivalent of far more tests in almost the exact same time frame…

What if we take that example from above and we allow each test to run for a full 30-day sales cycle to get an accurate measuring window?

Assuming we have the traffic, we could run 27 test combinations all at once on that one page over maybe 30-60 days as a single multivariate test. Not bad when you compare that to running 27 consecutive A/B tests on that page, over a 27-month time frame!

(Again, I’m simplifying here and assuming you run for a full sales cycle on each test.)

But as you can see, multivariate testing allows you to get insights and ideas, implement them faster and lower uptime on consecutive losing tests.

Pretty good, right!?

Obviously, there are some other drawbacks to MVT testing other than traffic requirements.

Because of the large number of variations and combinations you can run, it does greatly increase your QA time pre- and post-test.

You need to make sure that every variation works and also makes sense. (Not showing a headline and image that don’t match etc), and so it does increase your setup time and cost dramatically.

Also, be sure to check on effects.

Just because you see a lift in conversion on that particular page, track it forward to your ultimate goals. The last thing you want is more clicks but poor user experience or a high bounce rate (or more clicks but no sales).

The Pros of Multivariate Testing

  • Assuming you have the traffic, multivariate testing allows you to test a lot of variations on a single page, very fast.
  • MVT can help you to scale out your testing program to find more data on what works and what doesn’t.
  • It can help you to find winning combinations quicker than consecutive A/B tests.
  • It can help you to find winning combinations that you might have missed when running consecutive tests. (Imagine if one headline was 2nd performing and you cut it out, but paired with the 3rd image variation, it somehow doubled lift? With a consecutive test you would have missed this).
  • Some A/B testing software will allow you to see which particular element helped provide the most lift (giving you a deeper understanding of your audience).
  • It can also be used to help find minor lifts on high-performance pages. (Imagine you have a high-performing page already and you’re shotgunning new ideas to see minor lifts, such as color changes.)

The Cons of Multivariate Testing

  • Need a high volume of traffic to run, especially if you want stat sig so that you can trust your results and not see false positives or negatives (type 1 and type 2 errors), and stat power to measure smaller % lift.
  • Traffic requirements grow quickly the more variations and combinations that you test, so keep that in mind, and consider removing additional elements if they may not provide much lift. (ie test the headline before worrying about testing footer colors, etc. You can always test those after if you really want to.)
  • It’s not just traffic that affects sample size either. The conversion rate of the current page and variables, all affect the volume of the sample size required to hit statistical significance. Better conversions can lower the traffic needed, lower conversions increase that requirement.
  • The margin of error when testing more variables also increases the volume of traffic required before you can trust the results.
  • Need to run intensive QA tests to make sure they all work in advance.
  • Need to also run QA to make sure that the variations make sense together and don’t have opposing or conflicting combinations. (CTAs that don’t match offers, headlines for a beach holiday while advertising a ski trip, or even conflicting or blending font and background colors.)
  • If you don’t have a high traffic source, you can still run an MVT test, it just might take a LOT longer until you hit stat sig and can end the test. (Sometimes multiple months or longer). In fact, it can sometimes be quicker and more efficient to just run consecutive A/B tests.
  • You may lose some audience insights, simply by testing multiple things at once. (Testing a single element or angle can help you understand your audience better sometimes.)

Multivariate Testing Statistics: The Math Behind MVT

So let’s nerd out a little bit…

When it comes to multivariate testing, we use a different statistical analysis method to standard A/B testing or split testing.

I don’t want to get into the math of it all too much, but here’s a simplification:

  • With A/B testing, we use a t-test (or z-test) to measure the difference between 2 or more sets of results.
  • With multivariate testing, another method is used to find the test results called ANOVA or ‘Analysis of Variance’. You don’t need to know all the details, but here’s the core thing. The main difference is that ANOVA also measures the connection and impact between variables. This is one of the main strengths of MVT because it allows us to see how combinations of elements impact each other. 

With me so far?

Well not only do MVT tests use different testing models, but some tools even offer different testing features which can also affect your results.

Depending on which tool you use, there can be 3 types of multivariate testing methods offered when running an MVT campaign. Each of them affects the math on how your test will work.

We call these:

  • Full Factorial,
  • Fraction Factorial, and
  • Taguchi.

So what do they mean?

Full Factorial

Almost every MVT tool will offer Full Factorial testing as the basis for its main testing model and it’s pretty much what you want to use all the time.

This method refers to the ‘full’ distribution of traffic in your test.

Simply put, it’s where you test every combination at once, dividing the traffic to the test equally between each variant.

With an A/B test, you split the traffic 50:50. But with an MVT, if you had 10 variants, then this would mean that only 10% of the traffic goes to each variant.

Because of the number of variants and the traffic required to reach statistical significance with each variation, full factorial does require a very large traffic source, but it is the most accurate method.

However, some A/B testing platforms will also offer something called ‘Fraction Factorial’ testing as an alternative.

Fraction Factorial

This method attempts to bypass some of the larger traffic requirements by testing just a smaller sample of variations and then estimating end conversion rates based on current test results.

Its goal is not in the accuracy of predicting trustworthy results for scaling out like normal i.e if we scale this out to X audience will it perform the same.

Instead, it’s designed to help you see where the early traction is starting to happen so that you can get insight before you test deeper or with another method.

Let’s say that you have 10 variations you want to test.

Fractional factorial will actually test 5 of them instead and then try to infer the conversion rate on those other 5 tests, based on the results so far.

As you can guess, this is not as accurate and can lead to errors and false positives, and is actually a source of argument among optimizers and marketers. (With almost all recommending full factorial as the main testing method.)

Taguchi

Finally, you have the Taguchi method of multivariate testing. Originally designed as a method of measuring quality control against assumptions of acceptable losses in R+D for product manufacturing.

Most tools won’t offer this because of its complexity to use.

When to Stop a Multivariate Test?

There are 3 main things that we need to consider when running any test before we should stop it:

  • For the test to have run long enough to have collected a large enough sample size to each variant, and for the test to have collected enough conversions to be relevant.
  • For the test to have run either 1 or even 2 full sales cycles (to balance out traffic and shopping periods over 30-60 days.)
  • And finally, for the results to be statistically significant i.e if we take this test out in the wild and scale it up, then we can be 95% sure it will continue to perform the same way.

We recommend that you do not stop any test before this point. Otherwise, the data will be insufficient to work with. We might however stop and then re-start a test if there are issues with performance such as one variant’s payment forms not working etc, but ideally, we try not to even peek when it’s running.

(Some agencies do recommend stopping early if there is a clear winner, or even stopping specific variations if their conversion rate drops below 10%, and then allowing the test to keep running while filtering more traffic to the higher performers. Obviously, this can affect the test results greatly.)

How to Set Up a Multivariate Test?

The process of setting up a multivariate test is very similar to an A/B test.

Step #1: Decide on Your End Goal

First, you need to form a hypothesis for your page, its goal, its current results, and how you think it can be improved.

You can use our hypothesis generator to help you clarify what you want to achieve, and set some concrete goals.

Step #2: Figure Out What to Test

Then, decide on which elements to test and what variations you think could improve your current results.

For example, you may think that testing the ‘core elements’ above the fold will increase conversions so you want to test the headline, the hero shot, and the CTA button text.

Step #3: Calculate Testing Timeframe

Before you run the test or create any assets, we recommend that you perform a pre-test sample size calculation, so you can estimate the time frame and traffic requirements for this test to run. This will help give you some insight into how many variations you can feasibly run with your recurrent traffic and still get fast results. (Or even let you know if you need to test fewer options).

In the example above, you have 3 elements: the headline, the hero shot, and the CTA button text.

If you want to test a single variation for each that would be 2x2x2=8 combinations. Knowing this, you can start to figure out how much traffic you need.

Input your current weekly page traffic and conversion rate into the calculator (along with any power and confidence settings).

Then, set the number of pages that you want to test. (8 in this example)

multivariate testing pre-test sample size calculation

This will then give you a time frame for how long you need to run to be able to detect the minimum lift.

Step #4: Build the Test

Next, head across to your tool and set up the different versions and combinations for your multivariate test.

Start by selecting the multivariate test option.

multivariate testing Convert Experiences

Give it a name and then add the URL of the page you want to test (so the tool can pull it in for the WYSIWYG edits.)

multivariate testing Convert Experiences add URL

In this example, we’re going to be working on improving our home page conversion rate.

Once the page loads up in the tool, you need to set up your test elements.

Here at Convert, we use 2 naming conversions for the parts to edit in multivariate tests:

  • The Section, which refers to the element on the page that you want to test such as a headline or CTA, and
  • The Variation, which is your edit of that particular element.
multivariate testing Convert Experiences variation

I recommend that you pick one section at a time to edit and add each variation.

Give the section a name such as ‘Headline’, then go ahead and edit and add the variations for that section. (Be sure to save each variation as you work on it.)

multivariate testing Convert Experiences save variation

Then move onto the next ‘section’ or element that you want to test. In this example, it would be setting up another background hero shot, and then setting up a CTA button text variation.

multiple variables

Step #5: Make Sure It All Works

Once you’ve set up each variation, run a QA test on each new treatment to make sure that they all work and that each element fits together sensibly. (No competing CTAs or mismatched information/images, or double up on font color and buttons.)

Click on ‘save and continue’ and a new page will load. Then, go ahead and click on the “Reports” tab.

From here you will see page variation and information on them. Click on the ‘eye icon’ next to each page to see a preview of it and test the links work.

multivariate testing software Convert Experiences

From here you can run a QA test and use that same URL for mobile tests.

Step #6: Push It Live

As long as each variation is all working, you can now start the test and let it run until you have enough data.

(Notice that there’s an option to push the test live right from the report page where you opened the QA tests.)

Best Multivariate Testing Tools to Mature Your Optimization Program

So now that you understand how multivariate testing works, let’s break down the website optimization platforms on the market that offer multivariate testing. We’ll cover their features along with their pricing if they offer free trials and a few other details.

One quick thing though before we do that.

Remember that the tool you use is ONLY as good as the person setting it up. If cash flow is tight, then invest in someone to help internally first (or reach out to an external agency) and just use a cheaper tool for now.

It’s all very well having an amazing MVT tool, but it’s pointless if the person can’t set it up correctly, fast enough, or understand the data!

#teamovertools

So with that out of the way, here’s a list of the best conversion rate optimization tools offering multivariate testing in 2021:

  1. Convert Experiences
  2. AB Tasty
  3. Google Optimize 360
  4. Adobe Target
  5. Kameleoon
  6. Optimizely
  7. Sitespect
  8. VWO
  9. Webtrends Optimize

#1: Convert Experiences

multivariate testing tools Convert Experiences
Convert Experiences

G2 Rating: 4.7/5.0 (46 Reviews)

Pricing: Starting as low as $699 per month, with $199 for every 100k visitors after that.

Do they offer a free plan or trial? Yes, 14 days free trial with no credit card needed.

Cost per 100,000 visitors: $199.

Pros

  • Fully privacy compliant: no personal data is ever stored
  • Accurate reporting because of use of 1st party cookies
  • Fast and flicker-free
  • Run unlimited tests
  • Condition stacking with 40+ filters to target granular audience segments
  • Flexible JavaScript insertion at project, experiment & variation levels
  • Custom JS condition goal triggering
  • Integrates with 100+ 3rd party tools (e.g., Shopify, WordPress, Mixpanel, Hotjar)
  • 4x faster, human customer support via phone, email & chat.

Cons

  • Basic post segmentation on the Kickstart plan

Multivariate Testing Features: Full factorial. We also offer direct support and advice when running these tests 😉

Client Side, Server Side or Both? Convert works as a client-side WYSIWYG editor, but it can also be set up on server side and run custom JS.

Do they serve Enterprise? Yes. We are the tool of choice for companies like Sony, Jabra, Unicef, and others.

Do they offer Customer support? Yes, right from the start of the trial. What type? Live chat, blog, and knowledge base with more to come.

Do they care about Privacy? We are fully GDPR compliant. We even stopped using tools for our internal work from other companies that don’t meet GDPR, that’s how much it means to us.

Do they care about the World? Yes. It’s in our DNA. We plant trees, run community programs, champion diversity from the initial application, donate to charities, and much more. Heck, we’re 15x carbon negative.

TrustRadius Review:

See what Convert Experiences looks like in action.

Experience privacy-compliant, flicker-free, limitless testing. Try Convert Experiences for free for 15 days.

#2: AB Tasty

G2 Rating: 4.5/5.0 (76 Reviews)

Pricing: Starts at $1900/mo for 400k monthly tested visitors for the Essentials plan. The Growth plan costs $3800/mo while the Elite plan starts at $5700/mo.

Do they offer a free plan or trial? No. You can request a demo call to see their new features.

Cost per 100,000 visitors: Approx $475.

Pros:

  • AI and ML built into the optimization
  • Run unlimited experiments
  • Clean and easy-to-use user interface
  • Easy to set up and preview tests
  • Dynamic Widgets
  • Multiple integrations
  • Extensive analytics reports
  • Wide range of targeting options available, along with personalization
  • Reliable customer support
  • Pricing is mid-range

Cons:

  • No marketing automation to give insights about audience performance (especially for an active experiment) and will sometimes require customer support to see.
  • Statistical Significance calculator is a bit basic in its UX
  • Google Analytics integration is complex, it requires coding
  • No free trial

Multivariate Testing Features: Full Factorial

Client Side, Server Side or Both? Both

Frequentist or Bayesian? Bayesian.

Does it support Full Stack? Yes.

Core Web Vitals Ready? Yes. This tool loads fast.

Has ML or AI? Yes.

Do they serve Enterprise? Yes. They are the tool of choice for companies like Disney, L’Oreal, Kalviyo, and others.

Do they offer Customer support? Yes. They have a knowledge base and live chat.

Do they care about Privacy? Yes. User IP addresses are used to create an ID code and then immediately deleted before moving into the tool. ID codes are anonymized and then deleted after 13 months.

Do they care about the World? They donate directly to NGOs, they work with social action groups, they recycle and sponsor beehives.

TrustRadius Review:

What this multivariate testing tool looks like in action:

#3: Google Optimize & Optimize 360

G2 Rating: 4.3/5.0 (27 Reviews)

Pricing: There are 2 versions of this tool. Optimize is free, but for advanced features, you have to opt for Google Optimize 360.

The pricing for Optimize 360 is custom (although rumored to be around $150,000 per year). To get your monthly price, you’ll have to fill a form to contact their sales team.

Do they offer a free plan? Yes.

Cost per 100,000 visitors: N/A

Pros:

  • User-friendly — little technical knowledge required
  • Great for first-time A/B testers
  • Quick integration with Google Analytics
  • Has a super detailed audience segmentation, thanks to Google’s huge data resources

Cons:

  • Limited to 5 tests at a time on the free tier
  • No drag-and-drop editor functionality
  • Cannot upload your images directly
  • Cannot test apps, only browsers
  • Not suited for complicated tests
  • UI/UX and reports aren’t as visually appealing as most other tools
  • Flicker happens sometimes.

Multivariate Testing Features: Apparently Optimize uses a hybrid approach that is neither full factorial or fractional. According to their support base

“Our models allow us to use a hybrid approach, so you don’t have to make this tradeoff. We serve all combinations of a test, so you can learn about interactions and best combinations. But, we also model the fact that some variants show up across combinations — and so we can learn about variants within a factor, not just combinations”

Client Side, Server Side or Both? Both.

Frequentist or Bayesian? Google Optimize uses Bayesian methods rather than Frequentist methods, also known as Null Hypothesis Significance Testing (NHST).

Does it support Full Stack? Yes.

Core Web Vitals Ready? Yes. This tool loads fast, although it can flicker which might cause CLS issues.

Has ML or AI? No.

Do they serve Enterprise? Yes, but only via Optimize 360.

Do they offer Customer support? They have a resource hub with tips, video tutorials, help community, and more.

Do they care about privacy? They comply with applicable data protection laws in the countries where their products are used.

They also work with third parties and data protection authorities to keep users’ data safe.

Do they care about the World? Google has been carbon neutral since 2007 and plans to be carbon-free by 2030. Apart from that, they’re known for numerous philanthropic gestures with their $1 billion commitment.

TrustRadius Review:

What this multivariate testing tool looks like in action:

#4: Adobe Target

G2 Rating: 4.0/5.0 (29 Reviews)

Pricing: Only available after an 11 part form, an email, and a call. Starting rate is $10,000 a month.

Do they offer a free plan or trial? No.

Cost per 100,000 visitors: N/A

Pros

  • The interface is easy to use
  • Provides accurate real-time data reports
  • Integrates well with Adobe Analytics and is designed as an upsell offer
  • Website personalization tools for both beginners and experts
  • Walks you through the setup and testing process
  • Has advanced AI to continually test and improve campaigns and personalization

Cons

  • Optimizing on a large scale can be slow
  • Requires very high volume of traffic to work best
  • The form-based editor requires a learning curve
  • High price point
  • No trial option
  • Works best when already using Adobe Analytics
  • Can get great insights when connected to other paid Adobe Marketing Cloud tools

Multivariate Testing Features: Full Factorial and Taguchi (as a method of partial factorial).

Client Side, Server Side or Both? Both.

Do they serve Enterprise? Yes. Almost exclusively due to the cost.

Do they offer Customer support? Yes, in the form of a knowledge base and video training programs.

Do they care about Privacy? Yes, but ironically they require your email and phone number to learn more about this. They are however doing a lot of work in 1st party data for non-intrusive tracking.

Do they care about the World? Very much so. They champion diversity, work towards 100% renewable energy buildings, lowering emissions, running community action programs, and much more.

TrustRadius Review:

What this multivariate testing tool looks like in action:

#5: Kameleoon

G2 Rating: 4.7/5.0 (24 Reviews)

Pricing: Customized according to your requirements. You get your unique price by contacting the sales team.

Do they offer a free plan? Yes.

Cost per 100,000 visitors: N/A

Pros

  • Easy to set up click tracking
  • Smooth integration with many other tools
  • Has an easy-to-use WYSIWYG editor for non-developers
  • Advanced anti-flicker technology
  • Knowledgeable and helpful support team
  • Accurate and detailed planning and execution of tests

Cons

  • WYSIWYG editor loads slowly
  • The reporting dashboard could use a bit more personalization
  • Need developer-level skill to implement some complex scenarios
  • Cannot archive tests
  • Old browsers like IE7 are excluded from the tests

Multivariate Testing Features: Full Factorial and Fractional Factorial (via allocations in the test setup).

Client Side, Server Side or Both? Both. You can make edits on the WYSIWYG editor as well as use Kameleoon’s in-built code editor to edit CSS and JS.

Do they serve Enterprise? Yes. Lexus, Unity, Providence, Toyota, and Rakuten use Kameleoon.

Do they offer Customer support? Yes. You can even get a dedicated account manager to assist you with complicated projects.

Do they care about Privacy? They are HIPAA, GDPR and CCPA compliant. They also have a flexible consent management feature to adapt your privacy policy for each A/B test.

Do they care about the World? I couldn’t find evidence of any charity they sponsor or environmental causes they support. Although, it’s interesting to know they have two live chameleons in their Germany and France offices.

TrustRadius Review:

What this multivariate testing tool looks like in action:

#6: Optimizely

G2 Rating: 4.3/5.0 (109 Reviews)

Pricing: They’re using a custom pricing model. But Splitbase predicts they cost at least $36,000 per year.

Do they offer a free plan? No. They stopped the free plan in 2018.

Cost per 100,000 visitors: N/A

Pros:

  • Run unlimited experiments
  • Clean and easy-to-use user interface
  • The widget feature is fun to use
  • Wide range of advanced targeting options available
  • Reliable customer support

Cons:

  • Doesn’t give automatic insights about audience performance (especially for an active experiment)
  • Google Analytics integration is complex, requires coding
  • Optimizely snippet usually increases page loading time

Multivariate Testing Features: Full Factorial, Partial and Taguchi.

Client Side, Server Side or Both? Both. Optimizely offers client-side experimentation through a Javascript snippet and server-side experimentation through developer SDKs.

Frequentist or Bayesian? Optimizely’s Stats Engine uses sequential experimentation, not the fixed-horizon experiments that you would see in other platforms.

Does it support Full Stack? Yes.

Core Web Vitals Ready? Yes, this tool loads fast.

Has ML or AI? Yes.

Do they serve Enterprise? Due to pricing, it mostly caters to an enterprise level. Brands like Microsoft, IBM, HP, eBay, Yamaha, Pizza Hut, and Atlassian use it.

Do they offer Customer support? Yes. They have a bank of resources to help users get unstuck and phone numbers to call for help 24/7.

Do they care about Privacy? They take into consideration old and new privacy laws and integrate that into their products so that you, the user, don’t have to worry about compliance.

Do they care about the World? Most new hires are sent to volunteer in the community on their second day.

TrustRadius Review:

What this multivariate testing tool looks like in action:

Turned off by Optimizely pricing? Meet Convert Experiences! It’s the best of both worlds – it has all your favorite A/B testing features with 4x faster support that will save you up to USD 100k on a contract.

#7: SiteSpect

G2 Rating: 4.4/5.0 (50 Reviews)

Pricing: You’ll have to contact SiteSpect for a customized price according to your needs.

Do they offer a free plan? Yes, they offer a free trial.

Cost per 100,000 visitors: N/A

Pros:

  • Supports all markup languages (HTML, WML, XML, and JSON), style sheets, and scripting languages
  • No javascript tag means no content refresh or flicker
  • Versatile enough to test almost any scenario
  • Non-intrusive testing
  • Integrates with analytics tools
  • No need to modify the production version of your code

Cons:

  • Technical knowledge required to implement tests
  • Adds an extra hop that could slow down your site a bit
  • The reporting interface could be better

Multivariate Testing Features: Full Factorial and Fraction Factorial (The tool recommends moving to fractional if testing more than 128 combinations).

Client Side, Server Side or Both? Both.

Frequentist or Bayesian? SiteSpect uses a two-tailed t-test when comparing Variations against Controls to determine the point at which confidence intervals do not overlap and thus indicate significance. SiteSpect also computes the z-score and uses it in reports.

Does it support Full Stack? Yes.

Core Web Vitals Ready? Yes. This tool is incredibly fast.

Has ML or AI? Yes.

Do they serve Enterprise? Yes. Staples, PetSmart, AmericanGirl, and Urban Outfitters use SiteSpect.

Do they offer Customer support? Customer support is available via phone call and email for free and paid versions. There are also pre-recorded webinars available in their knowledge base.

Do they care about Privacy? They do. They are PCI DSS 3.2 certified; GDPR, CCPA, and privacy shield compliant, and HIPAA ready.

Do they care about the World? SiteSpect is known to sponsor some charity projects since 2014.

TrustRadius Review:

What this multivariate testing tool looks like in action:

#8: VWO Testing

G2 Rating: 4.2/5.0 (239 Reviews)

Pricing: Starts at $99/mo. Anything more requires customized pricing according to your needs. And you’ll have to contact the VWO sales team for yours.

Do they offer a free plan? Yes, you can get started with a 30-day free trial.

Cost per 100,000 visitors: ~$500

Pros:

  • Flexible customization options to adapt your test to a lot of scenarios
  • Easy to plan and execute tests with little coding knowledge
  • Dedicated support team to guide you through any challenges
  • Has the ability to track long-term goals
  • Ability to group tests together
  • Useful recordings for observing users and troubleshooting

Cons:

  • There’s a learning curve to understand the full functionality of VWO
  • Pricing plans change frequently
  • They only test client side

Multivariate Testing Features:  Full Factorial (although you can set up traffic percentages per page variation).

Client Side, Server Side or Both? Only client side.


Do they serve Enterprise?
Yes. Their enterprise clients include Ubisoft, AMD, Penn Foster, Optimeria, and IMB Bank.

Do they offer Customer support?  Yes, you can call them when you need help. Or consult their resource page.

Do they care about Privacy? Yes. They comply with the major privacy laws and regulations but they store some visitor data on their servers.

Do they care about the World? Recently, the chairman of Wingify (the brand behind VWO) Paras Chopra, tweeted their support for a couple of COVID relief initiatives including setting up a 10-bed COVID-care facility in Delhi.

TrustRadius Review:

What this multivariate testing tool looks like in action:

#9: Webtrends Optimize

G2 Rating: 4.9/5.0 (107 Reviews)

Pricing: Starts at $212 per month but can be lower or higher depending on what you require. They have a very flexible pricing structure, but you have to speak to the sales teams to confirm your price.

Do they offer a free plan? Yes, for less than 1,500 sessions per month.

Cost per 100,000 visitors: N/A

Pros:

  • User-friendly, suitable for non-technical users with a WYSIWYG editor
  • Supports team collaboration
  • Gives data-driven insights
  • The support team shows genuine interest in your success
  • Smooth integration with your website and existing tools
  • Clear and straightforward reports

Cons:

  • The UI can be difficult to use for new users
  • Takes time to set up even simple tests
  • The cost of the full service is on the high side

Multivariate Testing Features: Full Factorial and Fractional Factorial.

Client Side, Server Side or Both? Both. They have a proxy-less visual editor for client-side rendering and server-side testing capabilities too.


Do they serve Enterprise?
Microsoft, London North Eastern Railway, Marks and Spencer, and Virgin Wines use Optimize.

Do they offer Customer support? Yes. You get access to tech support, training, and your own dedicated CRO consultant.

Do they care about Privacy? They are fully GDPR compliant.

Do they care about the World? In April 2021, Webtrends Optimize challenged themselves to 100 Push-ups a day in a GoFundMe event to raise funds for Cancer Research UK. And they did something similar in 2020 called Xmas Charity Donations. So, they care.

Trust Radius Review:

Although there are reviews on Trust Radius for this tool, the latest was December 2019, and before that 2018. Because a tool can change and so can the service, I’ve pulled their latest review from G2.

What this multivariate testing tool looks like in action:

Conclusion

So there you have it.

Our list of the best A/B testing tools offering multivariate testing in 2021. We also show which method they use to test, their features, pricing, and reviews.

Now you can compare the best multivariate tools across these different parameters and pick one that would work best for your testing needs.

Originally published September 08, 2021 - Updated September 14, 2021

Start 15-Day Free Trial Of Convert Experiences.

Quick form submission
Quick submission.
No credit card needed
No credit card needed.
You can always change your preferences later.
Check Icon
You're Almost Done.
Managing Marketing Team
Managing Tech Team
Hypothesizing Experiments
Coding and QA of Experiments
Convert is committed to protecting your privacy.

You've successfully submitted your details to start your Convert Experiences 15-day free trial.

We are setting up your account for you. You'll receive a welcome message to your registered email with your login access. Please explore the app on desktop for an optimal experience.