Blog /A/B Testing

Top A/B Testing Tools That Are Good for the Next 5 Years (Vetted by Privacy, Features, Maturity & Price)

18th Aug 2021 –

So you’re looking to buy an A/B testing tool. Maybe it’s your first tool or maybe you’re in the market for a new one, but the key point is this:

You’re looking to get a tool that will serve your needs NOW, while also being able to grow with your needs and experimentation goals as they change over time so you won’t need to upgrade for a while after this.

That’s what we’re going to cover for you today.

The future trends and ideas that we’re seeing emerge in the CRO industry, what they mean, and which tools we think meet this demand (while also meeting your current testing needs).

This way, you can streamline your decision-making and find the best A/B testing tool for your investment.

So let’s dive in…

What Are A/B Testing Tools?

A/B testing tools:

  • Allow you to split test 2 (or more) elements on your website, to see which converts best.
  • Use a form of statistics (usually Frequentist or Bayesian) to measure the actions that your visitors take during that test and then provide the results. You can run that test to get a statistical significance and level of confidence that if you push that test result live, you can expect similar results from the rest of your audience.
  • Usually, come in two varieties. Either open source which allow you to keep your data in-house and use right away, or a closed source that usually asks you to contact them before you can get started.

How Can You Set Up an A/B Test?

We cover how to set up and run A/B tests in far more detail here, but here’s a quick overview:

What is the future of CRO?

As we look at it, there are 4 major trends in the CRO industry right now:

1. Using CRO as a more holistic tool for larger data analysis,
2. Concern over how testing might affect organic traffic and page experience,
3. User privacy and how this relates to testing and tracking,
4. Machine Learning in A/B testing — what is it and is it worth it?

1. A/B testing for Decision Making

Traditionally, CRO and A/B testing have been all about website optimization:

How do we get the traffic on our web pages to take more of the action that we want?

Nowadays, however, it feels like CRO and the entire landscape are evolving to a more holistic and strategic view. (Even the name CRO doesn’t quite fit with how it’s changing.)

Rather than just A/B testing software or thought process to measure conversions, marketing teams are now looking at it from the perspective of measuring and understanding all audience interactions to make larger business decisions.

CRO as a practice isn’t drastically changing, but its place in the organization has shifted.

What was once the domain of a sole team, or in some sad cases, a sole individual, is now an operating system by which data-driven teams across a company make decisions.

I’ve seen the shift from decentralized models to centralized teams, where all CRO efforts are owned and guarded by a core team of experts and specialists. Turns out, this doesn’t scale super well.

Now this model, in many mature organizations, is turned into a “center of excellence” model. In this way of operating, the CRO team simply enables and supports the rest of the organization’s experimentation and optimization efforts through education, process management, resource allotment, and product management (typically it’s at this stage that company’s invest in custom tooling or data pipelines to enable more sophisticated experimentation and greater scale).

Additionally, “CRO” as once practiced, is essentially called “web strategy” or “website product management” now.

Which makes a ton of sense, because growth product managers on the product side are doing the exact same work and calling it something different. This blending of the marketing and product side has yet to fully eclipse the previous silos, but when it does, it’s going to enable much more interesting full-funnel experimentation.

The tools that will help to enable this are the ones that balance server and client-side experimentation, have robust integrations, and have advanced features like experiment swim lanes as well as a mix of marketer-centric features (like your typical WYSIWYG editor) and product-centric features like feature flags.

Alex Birkett, BeOmniscient

Let’s say you run some tests on a few web pages and you see that your audience is resonating with a particular angle and it’s converting more.

In the past, we would have focused on that simply as a better way to convert.

But now we might start to look at this deeper. Not only can we pass this information on to other areas of the company, but we can research this angle more.

  • Is this a change in the industry that’s occurring?
  • Is this becoming more important to our audience and something we should focus on?
  • Why is it important to them?

Learning about this could help us not only convert better but improve conversions front to back. This can then improve everything from ad results to home page offers, upsells, average sales and repeat sales, etc.

Not only that, but it can help us look at new product offerings, and even change company-wide positioning or pricing.

(Maybe moving from a single price model to a recurring model to improve retention while also giving more breathing room for profitable acquisition.)

How Does CXO Affect Your A/B Testing Tool Choices?

Ideally, we’re looking at quantitative testing tools (like Convert Experiences) that can integrate with other tools you might use to help you fulfill these needs.

These could be QA tools so that you can learn more about the reasons why this is important to your audience, customer success tools, or even communication tools so that you can share the information internally.

Convert has 100+ integrations with other tools in your tech stack.

2. A/B Testing That Minimizes Impact on Page Experience

Google recently updated their Page Experience ranking algorithm, by adding in their Core Web Vitals update.

We cover this in detail here (along with some best practices on how to make sure your tests don’t mess up your score), but here’s a quick overview.

The CWV update is all about improving the user experience on the page:

  • How fast it loads
  • How quickly the user can understand what the page is about (LCP)
  • How soon they can interact with the page
  • How quickly the page responds after that interaction request (TBT)
  • How much the page elements shift as the page loads or is used (LCS)

How Does Page Experience Affect Your A/B Testing Tool Choices?

If you’re looking specifically at how your testing solutions may affect your Core Web Vitals score, then you need to look at:

  • Tool Speed: How fast does the testing platform work?
  • Flicker Rate: Does the test cause flickering elements?
  • How does it test the element: Does it hide, move, or cause layout shifts?

It may seem like a small thing to care about now, but it’s clear that Google is making big changes that are all focused on page and user experience.

You really don’t want your testing tool to start affecting your traffic or customer experience.

3. A/B Testing That Respects Privacy

In the last few years, there’s been a large push towards improving user privacy online.

Everything from how we handle users’ data and consent, to how we track and what we track, and now we’re seeing specific devices and browsers remove user information or limit tracking time frames.

This is a good thing for the user and something we should respect as site or app owners. (Here at Convert, we don’t even work with other 3rd party tools unless they meet GDPR – even if it’s not a client-facing tool).

However, these bylaws and features only affect client-side tools and not server-side.

What’s the Difference Between Client-Side and Server-Side Tools?

Client-side tools are where the tool is on your website via a Javascript code in your header. It tests and tracks by making changes in the user’s web browser to show different page designs etc.

Server-side tools are instead installed on your web server.

They each have pros and cons: Client-side tools are usually faster and easier to set up new campaigns, while server-side tools allow you to track things you might not usually track with a standard tool.

The main difference when it comes to privacy is how these new laws and tools are implemented.

You see, pop-up blockers and things like this can stop client-side tools from working until the user agrees to it. The browser or device sees the cookie event and stops it from triggering and reporting back.

(Users without blockers in place will hopefully see a GDPR consent form asking them if they agree to be tracked and share particular user data. Some sites, however, will track as standard and ask them to opt-out instead).

The thing is, server-side tools are not stopped by any of this. They don’t rely on the web browser cookies to send information as it’s tracking all your site events and sending them to your server instead. This means that server-side tools can still track events and user ids, then feed that information into the testing platform, 3rd party analytics, or ad platforms.

You could in theory track away to your heart’s content and never tell anyone. (While risking a potential lawsuit if they get access to your server data.)

But just because you can track all that information, should you?

A/B testing can be the best way to mitigate the risks of damaging the user experience in the future. If you think about it, testing is all about proving a hypothesis that one version is better than the others. So you really don’t need to store anything more than the number of users seeing these versions and the goals you’re tracking. Just that… two simple numbers, variationID and GoalID, and the number of times they are seen. You should set up your A/B testing in this clean way, nothing more.

You lose the trust of your users if you try to slice and dice your data in search of a winner. That’s not what A/B testing is for, that’s analytics — don’t mix it up.

Mitigate the risk of hurting your users and get an occasional win for the business. It will win you trust, save you money and so make you the best version of what you can be for your users.

If you keep it clean, you won’t be impacted by privacy laws. You don’t store any personal data and don’t track users. Your tracking usage… that is long-term the future of this niche in CRO.

Dennis van der Heijden, Convert.com

How Does Privacy Affect Your A/B Testing Tool Choices?

Regardless of if you use client-side tools for the convenience of set up or server-side tools for more in-depth tracking, you should still as a responsible business meet standard and upcoming privacy changes:

  • Respect Do Not Track browsers
  • Ask for consent
  • Track events instead of user data
  • Anonymise IDs, etc.

With that in mind, be sure to look at your experimentation platform’s privacy goals and what they allow you to do with it:

  • Do they meet privacy requirements as put down by the GDPR and similar laws?
  • Do they store data safely so your audience doesn’t get leaked and you don’t get fined?
  • Do they allow you to test and respect users’ privacy while doing so? I.e. can you set those settings in your test campaigns, even if it’s a server-side tool where technically you could get away with it?
  • Do they allow you to adjust elements of your test for different location laws?
  • Finally, you might want to check if your tool is client-side only, or if it can be set up server-side also.

But remember: Just because you can track all that information with a server-side tool, it doesn’t mean that you should. Use a tool that cares about your audience’s privacy!

(We removed a lot of our in-depth tracking details from our tool because of this. We can still track what’s needed for tests, but we don’t need to do it while knowing too much about the user).

A/B Testing Powered by Data & Machine Learning

The final trend over the last few years has been around integrating Machine Learning into CRO testing:

CRO is beginning to change and will very apparently change over the next 5 years, in two fundamental ways. CRO is becoming more empirical and data-sophisticated. (…)

In many ways, this is just a reflection of what the FAANG (Facebook, Amazon, Apple, Netflix and Google) companies have known and practiced for the last decade. Having realized the imperative of testing everything across the business and running as many experiments as technically feasible, FAANG companies have hired scientists to develop what are quite literally fundamental advancements to the field of statistics. These advancements allow them to run more experiments, run experiments faster, and measure increasingly complex outcomes (e.g. Bing measuring “did a user successfully find what they were searching for?“, or Netflix measuring how different approaches to loading videos on slow connections impacts user frustration).

The areas where CRO was previously thought of as more of an “art form” (guiding program strategy, roadmap prioritization) will increasingly rely on science and machines that can make more optimal, and empirical decisions about how to prioritize potential treatments to test and what audiences to test on.

Ryan Lucht, Senior Growth Strategist, Cro Metrics

What Is Machine Learning?

Machine learning is a type of artificial intelligence, but in simple terms, it’s a pattern recognition machine. If you give it enough data points (and it does need a lot), then it’s able to find connections in that data that humans might miss, and then predict potential future actions.

For example, Netflix uses machine learning to understand more about each user so that it can recommend shows that it thinks you might like.

This helps Netflix to improve their customer experience, lower churn, and even learn what genres of content they can invest in.

How Does Machine Learning Affect CRO?

Machine Learning’s real benefit is for companies that have a lot of traffic and conversions already and want to run tests at scale and gather as much data as possible.

It allows them to set up Multivariate campaigns, let the tool learn from actions taken, and optimize tests live. Even more interesting, it can learn from ‘almost’ conversions so you can start to gather a lot of data about those very warm leads who are close to converting.

Amazon is a great example of using machine learning for real-time personalization.

They can adjust front page recommendations, upsells, pricing, and even landing pages, all based on previous user data of people who are similar to you (same interests), or by tracking your past interactions and predicting potential future actions. This personalization alone helps them to improve their sales by 35%.

How Do Machine Learning and AI Affect Your A/B Testing Tool Choices?

The first thing to understand is that Machine Learning tools are not for everyone.

ML works best when it can be given more data. Unless you’re doing around 100,000 visitors a month, or 1,000 conversions at the minimum, then it’s not for you. (You might be able to get away with it if you had bulk historical data perhaps, but your tests will be stunted if you’re not hitting that 1000 conversion per month threshold.)

Like any testing process, machine learning is only as good as the data that you are feeding it. If something is broken in the test, or you’re feeding it the wrong or not enough data, then it will struggle to give you the answers you’re looking for.

You still need to analyze why you are getting the results back. It’s not a robot you can plug in and ‘do CRO’ for you.

What if you wanted it to help you scale out and run multiple tests at once, or to start deep learning about large data sets?

A CRO tool with Machine Learning can help you do this much faster than you could on your own.

Top 5 A/B Testing Tools That Are Good for the Next 5 Years (Based On Current CRO Trends)

With each of those trends in mind, here are the top 5 future-proofed A/B testing tools we recommend.

We’ve added in their features and pricing, along with if they meet these 4 emerging trends.

Here they are:

Each of these testing solutions either:

  • Integrate with additional tools to analyze data holistically,
  • Load lightning fast to help with Core Web Vitals,
  • Care about privacy,
  • Have client or client- and server-side installation options,
  • Have Machine Learning features.

#1: Convert Experiences

G2 Rating: 4.7/5.0 (46 Reviews)

Pricing: Starting as low as $699 per month, with $199 for every 100k visitors after that.

Do they offer a free plan or trial? Yes, 14 days free trial with no credit card needed.

Cost per 100,000 visitors: $199.

Pros:

  • Fully privacy compliant: no personal data is ever stored
  • Fast and flicker-free
  • Feature-rich, run unlimited tests
  • Integrates with 100+ 3rd party tools (e.g., Shopify, WordPress, Mixpanel, Hotjar)
  • Reliable, fast customer support.

Cons:

  • Basic post segmentation on the Kickstart plan

Client Side, Server Side or Both? Convert works as a client-side WYSIWYG editor, but it can also be set up on the server side and run custom JS.

Frequentist or Bayesian? Convert Experiences runs experiments using the Frequentists Two-Tailed Z Tests.

Does it support Full Stack? Currently being implemented. Join the waitlist here.

Core Web Vitals Ready? Yes. The Convert script loads incredibly fast, is flicker-free, and has minimal effect on CLS.

Has ML or AI? No.

Do they serve Enterprise? Yes. We are the tool of choice for companies like Sony, Jabra, Unicef, and others.

Do they offer Customer support? Yes, right from the start of the trial. What type? Live chat, blog, and knowledge base with more to come.

Do they care about Privacy? We are fully GDPR compliant. We even stopped using tools for our internal work from other companies that don’t meet GDPR, that’s how much it means to us.

Do they care about the World? Yes. It’s in our DNA. We plant trees, run community programs, champion diversity from the initial application, donate to charities, and much more. Heck, we’re 15x carbon negative.

Trust Radius Review:

See what Convert Experiences looks like in action.

Experience privacy-compliant, flicker-free, limitless testing. Try Convert Experiences for free for 15 days.

#2: Google Optimize/ Optimize 360

G2 Rating: 4.3/5.0 (27 Reviews)

Pricing: There are 2 versions of this tool. Optimize is free, but for advanced features, you have to opt for Google Optimize 360.

The pricing for Optimize 360 is custom (although rumored to be around $150,000 per year). To get your monthly price, you’ll have to fill a form to contact their sales team.

Do they offer a free plan? Yes.

Cost per 100,000 visitors: N/A

Pros:

  • User-friendly — little technical knowledge required
  • Great for first-time A/B testers
  • Quick integration with Google Analytics
  • Has a super detailed audience segmentation, thanks to Google’s huge data resources

Cons:

  • Limited to 5 tests at a time on the free tier
  • No drag-and-drop editor functionality
  • Cannot upload your images directly
  • Cannot test apps, only browsers
  • Not suited for complicated tests
  • UI/UX and reports aren’t as visually appealing as most other tools
  • Flicker happens sometimes.

Client Side, Server Side or Both? Both.

Frequentist or Bayesian? Google Optimize uses Bayesian methods rather than Frequentist methods, also known as Null Hypothesis Significance Testing (NHST)

Does it support Full Stack? Yes.

Core Web Vitals Ready? Yes. This tool loads fast, although it can flicker which might cause CLS issues.

Has ML or AI? No.

Do they serve Enterprise? Yes, but only via Optimize 360.

Do they offer Customer support? They have a resource hub with tips, video tutorials, help community, and more.

Do they care about Privacy? They comply with applicable data protection laws in the countries where their products are used.

They also work with third parties and data protection authorities to keep users’ data safe.

Do they care about the World? Google has been carbon neutral since 2007 and plans to be carbon-free by 2030. Apart from that, they’re known for numerous philanthropic gestures with their $1 billion commitment.

Trust Radius Review:

What this A/B testing tool looks like in action:

Need a tool that ranks better than Google Optimize for several critical factors like “Quality of Support”, “Ease of Doing Business” and “A/B Testing”? Here’s how Convert Experiences stacks up against Google Optimize & Optimize 360.

#3: AB Tasty

G2 Rating: 4.5/5.0 (76 Reviews)

Pricing: Starts at $1900/mo for 400k monthly tested visitors for the Essentials plan. The Growth plan costs $3800/mo while the Elite plan starts at $5700/mo.

Do they offer a free plan or trial? No. You can request a demo call to see their new features.

Cost per 100,000 visitors: Approx $475.

Pros:

  • AI and ML built into the optimization
  • Run unlimited experiments
  • Clean and easy-to-use user interface
  • Easy to set up and preview tests
  • Dynamic Widgets
  • Multiple integrations
  • Extensive analytics reports
  • Wide range of targeting options available, along with personalization
  • Reliable customer support
  • Pricing is mid-range

Cons:

  • No automation to give insights about audience performance (especially for an active experiment) and will sometimes require customer support to see
  • Statistical Significance calculator is a bit basic in its UX
  • Google Analytics integration is complex, it requires coding
  • No free trial

Client Side, Server Side or Both? Both

Frequentist or Bayesian? Bayesian.

Does it support Full Stack? Yes.

Core Web Vitals Ready? Yes. This tool loads fast.

Has ML or AI? Yes.

Do they serve Enterprise? Yes. They are the tool of choice for companies like Disney, L’Oreal, Kalviyo, and others.

Do they offer Customer support? Yes. They have a knowledge base and live chat.

Do they care about Privacy? Yes. User IP addresses are used to create an ID code and then immediately deleted before moving into the tool. ID codes are anonymized and then deleted after 13 months.

Do they care about the World? They donate directly to NGOs, they work with social action groups, they recycle and sponsor beehives.

Trust Radius Review:

What this split testing tool looks like in action:

Need an AB Tasty alternative that (actually) aces A/B testing? Take a look at the alternative that beats AB Tasty pricing.

#4: Optimizely

G2 Rating: 4.3/5.0 (109 Reviews)

Pricing: They’re using a custom pricing model. But Splitbase predicts they cost at least $36,000 per year.

Do they offer a free plan? No. They stopped the free plan in 2018.

Cost per 100,000 visitors: N/A

Pros:

  • Run unlimited experiments
  • Clean and easy-to-use user interface
  • The widget feature is fun to use
  • Wide range of targeting options available
  • Reliable customer support

Cons:

  • Doesn’t give automatic insights about audience performance (especially for an active experiment)
  • Google Analytics integration is complex, requires coding
  • Optimizely snippet usually increases page loading time

Client Side, Server Side or Both? Both. Optimizely offers client-side experimentation through a Javascript snippet and server-side experimentation through developer SDKs.

Frequentist or Bayesian? Optimizely’s Stats Engine uses sequential experimentation, not the fixed-horizon experiments that you would see in other platforms.

Does it support Full Stack? Yes.

Core Web Vitals Ready? Yes, this tool loads fast.

Has ML or AI? Yes.

Do they serve Enterprise? Due to pricing, it mostly caters to an enterprise level. Brands like Microsoft, IBM, HP, eBay, Yamaha, Pizza Hut, and Atlassian use it.

Do they offer Customer support? Yes. They have a bank of resources to help users get unstuck and phone numbers to call for help 24/7.

Do they care about Privacy? They take into consideration old and new privacy laws and integrate that into their products so that you, the user, don’t have to worry about compliance.

Do they care about the World? Most new hires are sent to volunteer in the community on their second day.

Trust Radius Review:

What this A/B testing tool looks like in action:

Turned off by Optimizely pricing? Meet Convert Experiences! It’s the best of both worlds – it has all your favorite A/B testing features with 4x faster support that will save you up to USD 100k on a contract.

#5: SiteSpect

G2 Rating: 4.4/5.0 (50 Reviews)

Pricing: You’ll have to contact SiteSpect for a customized price according to your needs.

Do they offer a free plan? Yes, they offer a free trial.

Cost per 100,000 visitors: N/A

Pros:

  • Supports all markup languages (HTML, WML, XML, and JSON), style sheets, and scripting languages
  • No javascript tag means no content refresh or flicker
  • Versatile enough to test almost any scenario
  • Non-intrusive testing
  • Integrates with analytics tools
  • No need to modify the production version of your code

Cons:

  • Technical knowledge required to implement tests
  • Adds an extra hop that could slow down your site a bit
  • The reporting interface could be better

Client Side, Server Side or Both? Both.

Frequentist or Bayesian? SiteSpect uses a two-tailed t-test when comparing Variations against Controls to determine the point at which confidence intervals do not overlap and thus indicate significance. SiteSpect also computes the z-score and uses it in reports.

Does it support Full Stack? Yes.

Core Web Vitals Ready? Yes. This tool is incredibly fast.

Has ML or AI? Yes.

Do they serve Enterprise? Yes. Staples, PetSmart, AmericanGirl, and Urban Outfitters use SiteSpect.

Do they offer Customer support? Customer support is available via phone call and email for free and paid versions. There are also pre-recorded webinars available in their knowledge base.

Do they care about Privacy? They do. They are PCI DSS 3.2 certified; GDPR, CCPA, and privacy shield compliant, and HIPAA ready.

Do they care about the World? SiteSpect is known to sponsor some charity projects since 2014.

Trust Radius Review:

What this A/B testing tool looks like in action:

Conclusion

So there you have it. The 4 emerging trends at the forefront of CRO right now, how they might affect your tool buying decision, and the 5 tools that we think either meet those requirements or are working towards it.

If you’re looking to get an A/B testing platform and want a future-proof choice, you can’t go far wrong with this list.

Originally published August 18, 2021 - Updated August 24, 2021

Start 15-Day Free Trial Of Convert Experiences.

Quick form submission
Quick submission.
No credit card needed
No credit card needed.
You can always change your preferences later.
Check Icon
You're Almost Done.
Managing Marketing Team
Managing Tech Team
Hypothesizing Experiments
Coding and QA of Experiments
Convert is committed to protecting your privacy.

You've successfully submitted your details to start your Convert Experiences 15-day free trial.

We are setting up your account for you. You'll receive a welcome message to your registered email with your login access. Please explore the app on desktop for an optimal experience.