Top 10 A/B Testing Tools That Are Good for the Next 5 Years (Vetted by Features, Privacy, Maturity & Price)
18th Aug 2021 –
So you’re looking to buy an A/B testing tool. Maybe it’s your first venture into experimentation or perhaps you’re in the market for a new one, but the key point is this:
The tool you choose should serve your needs NOW, while also being able to grow with your needs and experimentation goals over time, so you won’t have to upgrade for a while.
That’s what we’re going to cover in this guide. The best A/B testing platforms that meet today’s testing needs and features, while also looking ahead to future trends and ideas that we’re seeing emerge in the industry.
We break down each tool along with these current trends, what they mean, and which tools we think meet this demand (while also meeting your current testing needs).
This way, you can streamline your decision-making and find the best A/B testing solution for your investment.
So let’s dive in…
- Top 10 A/B Testing Tools That Are Good for the Next 5 Years (Based On Current Conversion Rate Optimization Trends)
- Learn More About A/B Testing Tools
- How do you do an A/B test on a website?
- Top Trends in A/B Testing & Why You Should Care When Choosing A/B Testing Tools
- What does the future of CRO hold?
- Conclusion
Top 10 A/B Testing Tools That Are Good for the Next 5 Years (Based On Current Conversion Rate Optimization Trends)
What are the best A/B testing tools?
Here are the top 10 future-proofed A/B testing tools we recommend:
- Convert Experiences
- Google Optimize
- AB Tasty
- Optimizely
- SiteSpect
- Adobe Target
- VWO Testing
- Kameleoon
- OmniConvert
- Dynamic Yield
We’ve added in their features, pricing, usability, independent reviews, and if they meet the 4 emerging trends in CRO.
You can skip to the end of the post to learn more about these trends and why they’re important, but to summarize, each of these testing solutions either:
- Integrates with additional tools to analyze data holistically,
- Loads lightning fast to help with Core Web Vitals and not affect SEO traffic when testing,
- Cares about user privacy,
- Has client- or server-side installation options,
- Has Machine Learning features for additional testing and scale.
Now let’s jump straight to the A/B testing tools comparison.
#1: Convert Experiences
Convert Experiences is perhaps best considered as a mid-level testing platform, but with one of the lowest price points. It’s for companies and teams who are looking to move from running their first experiments and are now looking to scale up their testing program and processes.
Consistently recommended by agencies (even when they have a suite of competitors’ tools to use), Convert Experiences also has a fantastic support team for chat, email, and live calls.
Our goal is to create the best platform that does the job you need and to keep improving from there.
We’re also highly focused on privacy and data protection, not only meeting current GDPR and other legislation but actively improving the platform in advance to meet new privacy laws.
We’ve recently built a new Shopify integration that tracks revenue per variation, regardless of which Shopify plan you are on. (Something that most tools limit to premium Shopify plans.)
G2 Rating: 4.7/5.0 (46 Reviews)
Pricing: Starting as low as $559/mo (annual), with $199 for every 100k visitors after that.
Do they offer a free plan or trial? Yes, 14 days free trial with no credit card needed.
Cost per 100,000 visitors: $199.
Pros:
- Fully privacy compliant: no personal data is ever stored
- Fast and flicker-free
- Feature-rich, run unlimited tests
- Integrates with 90+ 3rd party tools (e.g., Shopify, WordPress, Mixpanel, Hotjar, Google Analytics)
- Reliable, fast customer support.
Cons:
- Basic post segmentation on the Optimal and the Lite plan
Client Side, Server Side or Both? Convert works as a client-side WYSIWYG editor, but it can also be set up on the server side and run custom JS.
Frequentist or Bayesian approach? Convert Experiences runs experiments using the Frequentist Two-Tailed Z Tests.
Does it support Full Stack? Currently being implemented. Join the waitlist here.
Core Web Vitals Ready? Yes. The Convert script loads incredibly fast, is flicker-free, and has minimal effect on CLS.
Has ML or AI? No.
Do they serve Enterprise? Yes. We are the tool of choice for companies like Sony, Jabra, Unicef, and others.
Do they offer Customer support? Yes, right from the start of the trial. What type? Live messaging, blog, and knowledge base with more to come.
Do they care about Privacy? We are fully GDPR compliant. We even stopped using tools for our internal work from other companies that don’t meet GDPR, that’s how much it means to us.
Do they care about the World? Yes. It’s in our DNA. We plant trees, run community programs, champion diversity from the initial application, donate to charities, and much more. Heck, we’re 15x carbon negative.
A/B testing software review from TrustRadius:
See what Convert Experiences looks like in action.
Experience privacy-compliant, flicker-free, limitless testing. Try Convert Experiences for free for 15 days.
#2: Google Optimize / Optimize 360
Google has 2 testing platforms: Optimize, which is free to use with some limitations on testing, the number of tests run and features, and then Optimize 360 where all features are available but the price jumps up drastically.
For those totally new to testing, the free version of Optimize is a great place to start, and then from there, you can move on to some mid-range options once you hit the limitations on the tool or want to scale further.
G2 Rating: 4.3/5.0 (27 Reviews)
Pricing: There are 2 versions of this. Optimize is free, but for advanced features, you have to opt for Google Optimize 360.
The pricing for Optimize 360 is custom (although rumored to be around $150,000 per year). To get your price per month, you’ll have to fill a form to contact their sales team.
Do they offer a free plan? Yes.
Cost per 100,000 visitors: N/A
Pros:
- User-friendly — little technical knowledge required
- Great for first-time A/B testers
- Quick integration with Google Analytics
- Has a super detailed audience segmentation, thanks to Google’s huge data resources
Cons:
- Limited to 5 tests at a time on the free tier
- No drag-and-drop editor functionality
- Cannot upload your images directly
- Cannot test mobile apps, only browsers
- Not suited for complicated tests
- UI/UX and reports aren’t as visually appealing as most other tools
- Flicker happens sometimes.
Client Side, Server Side or Both? Both.
Frequentist or Bayesian? Google Optimize uses Bayesian methods rather than Frequentist methods, also known as Null Hypothesis Significance Testing (NHST)
Does it support Full Stack? Yes.
Core Web Vitals Ready? Yes. This tool loads fast, although it can flicker which might cause CLS issues.
Has ML or AI? No.
Do they serve Enterprise? Yes, but only via Optimize 360.
Do they offer Customer support? They have a resource hub with tips, video tutorials, help community, and more.
Do they care about Privacy? They comply with applicable data protection laws in the countries where their products are used.
They also work with third parties and data protection authorities to keep users’ data safe.
Do they care about the World? Google has been carbon neutral since 2007 and plans to be carbon-free by 2030. Apart from that, they’re known for numerous philanthropic gestures with their $1 billion commitment.
A/B testing software review from TrustRadius:
What this A/B testing solution looks like in action:
Need a platform that ranks better than Google Optimize for several critical factors like “Quality of Support”, “Ease of Doing Business” and “A/B Testing”? Here’s how Convert Experiences stacks up against Google Optimize & Optimize 360.
#3: AB Tasty
A/B Tasty is another mid-level tool. It features a clean interface and simple setup that makes it easy for companies looking to move past basic experimentation to start scaling their testing programs.
The simplicity extends into its machine learning functions to help sites with large traffic run more sophisticated personalization and data tracking.
The metrics reporting function could do with being updated as it is quite simple and uses bar graphs instead of plotting data points, but overall a great solution.
G2 Rating: 4.5/5.0 (76 Reviews)
Pricing: AB Tasty’s plan(s) are gated. You need to fill out a custom form to request a demo, and better understand their pricing model.
Do they offer a free plan or trial? No. You can request a demo call to see their new features.
Pros:
- AI and ML built into the optimization
- Run unlimited experiments
- Clean and easy-to-use user interface
- Easy to set up and preview tests
- Dynamic Widgets
- Multiple integrations
- Extensive analytics reports
- Wide range of targeting options available, along with personalization
- Reliable customer support
- Pricing is mid-range
Cons (Based on G2C reviews):
- No free trial. You have to fill out a custom form (demo) to get to understand their pricing models
- Personalizations and tests tend to overlap or collide
- Data dashboards can be more user friendly
- The data coming from the platform is often difficult to understand
- It is hard for non-technical users to set up Google Analytics tracking
Client Side, Server Side or Both? Both
Frequentist or Bayesian? Bayesian approach.
Does it support Full Stack? Yes.
Core Web Vitals Ready? Yes. This tool loads fast.
Has ML or AI? Yes.
Do they serve Enterprise? Yes. They are the tool of choice for companies like Disney, L’Oreal, Kalviyo, and others.
Do they offer Customer support? Yes. They have a knowledge base and live chat.
Do they care about Privacy? Yes. User IP addresses are used to create an ID code and then immediately deleted before moving into the tool. ID codes are anonymized and then deleted after 13 months.
Do they care about the World? They donate directly to NGOs, they work with social action groups, they recycle and sponsor beehives.
A/B testing software review from TrustRadius:
What this testing tool looks like in action:
#4: Optimizely
Optimizely is very Enterprise focused. Their goal seems to be delivering solutions that make data-driven decisions easier in high traffic environments while also offering some machine learning elements.
G2 Rating: 4.3/5.0 (109 Reviews)
Pricing: They’re using a custom pricing model. But Splitbase predicts they cost at least $36,000 per year.
Do they offer a free plan? No. They stopped the free plan in 2018.
Cost per 100,000 visitors: N/A
Pros:
- Run unlimited experiments
- Clean and easy-to-use user interface
- The widget feature is fun to use
- Wide range of targeting options available
- Reliable customer support
Cons:
- Doesn’t give automatic insights about audience performance (especially for an active experiment)
- Google Analytics integration is complex, requires coding
- Optimizely snippet usually increases page loading time
- Expensive option
Client Side, Server Side or Both? Both. Optimizely offers client-side experimentation through a Javascript snippet and server-side experimentation through developer SDKs.
Frequentist or Bayesian? Optimizely’s Stats Engine uses sequential experimentation, not the fixed-horizon experiments that you would see in other platforms.
Does it support Full Stack? Yes.
Core Web Vitals Ready? Yes, this tool loads fast.
Has ML or AI? Yes.
Do they serve Enterprise? Due to pricing, it mostly caters to an enterprise level. Brands like Microsoft, IBM, HP, eBay, Yamaha, Pizza Hut, and Atlassian use it.
Do they offer Customer support? Yes. They have a bank of resources to help users get unstuck and phone numbers to call for help 24/7.
Do they care about Privacy? They take into consideration old and new privacy laws and integrate that into their products so that you, the user, don’t have to worry about compliance.
Do they care about the World? Most new hires are sent to volunteer in the community on their second day.
A/B testing software review from TrustRadius:
What this A/B testing tool looks like in action:
Looking for Optimizely alternatives? Meet Convert Experiences! It’s the best of both worlds – it has all your favorite A/B testing features with 4x faster support that will save you up to USD 100k on a contract.
#5: SiteSpect
SiteSpect is an Enterprise-focused testing tool with currently the fastest load time on the market. (But only slightly faster than some mid-level tools on this list.)
Not for beginners, this platform is for teams who have found limitations with the current platform and are moving further into data analysis, personalization, and machine learning for high traffic sites, but this is reflected by their price point.
Like most testing tools, it offers a client- or server-side setup, allowing you to set up split tests that edit the HTML before it loads on the page. This is great for those who are security conscious and who want to be able to track all interactions on site.
G2 Rating: 4.4/5.0 (50 Reviews)
Pricing: You’ll have to contact SiteSpect for a customized price according to your needs.
Do they offer a free plan? Yes, they offer a free trial.
Cost per 100,000 visitors: N/A
Pros:
- Supports all markup languages (HTML, WML, XML, and JSON), style sheets, and scripting languages
- No JavaScript tag means no content refresh or flicker
- Versatile enough to test almost any scenario
- Non-intrusive testing
- Integrates with analytics tools
- No need to modify the production version of your code
Cons:
- Technical knowledge required to implement tests
- Adds an extra hop that could slow down your site a bit
- The reporting interface could be better
Client Side, Server Side or Both? Both.
Frequentist or Bayesian? SiteSpect uses a two-tailed t-test when comparing test variations against controls to determine the point at which confidence intervals do not overlap and thus indicate significance. SiteSpect also computes the z-score and uses it in reports.
Does it support Full Stack? Yes.
Core Web Vitals Ready? Yes. This tool is incredibly fast.
Has ML or AI? Yes.
Do they serve Enterprise? Yes. Staples, PetSmart, AmericanGirl, and Urban Outfitters use SiteSpect.
Do they offer Customer support? Customer support is available via phone call and email for free and paid versions. There are also pre-recorded webinars available in their knowledge base.
Do they care about Privacy? They do. They are PCI DSS 3.2 certified; GDPR, CCPA, and privacy shield compliant, and HIPAA ready.
Do they care about the World? SiteSpect is known to sponsor some charity projects since 2014.
A/B testing software review from TrustRadius:
What this A/B testing tool looks like in action:
#6: Adobe Target
Adobe Target is an incredibly powerful Enterprise testing tool but it does come with some limitations, in that it’s only available to customers of their analytics tool.
That being said, the integration between the 2 allows for a fantastic workflow switching between personalization and segment targeting (hence the name), especially when you factor in its machine learning capabilities to constantly learn more about your audience and targeting.
G2 Rating: 4.0/5.0 (29 Reviews)
Pricing: Only available after an 11-part form, an email, and a call. Starting rate is $10,000 a month.
Do they offer a free plan or trial? No.
Cost per 100,000 visitors: N/A
Pros
- The interface is easy to use
- Provides accurate real-time data reports
- Integrates well with Adobe Analytics and is designed as an upsell offer
- Website personalization tools for both beginners and experts
- Walks you through the setup and testing process
- Has advanced AI to continually test and improve campaigns and personalization
Cons
- Optimizing on a large scale can be slow
- Requires very high volume of traffic to work best
- The form-based editor requires a learning curve
- High price point
- No trial option
- Works best when already using Adobe Analytics
- Can get great insights when connected to other paid Adobe Marketing Cloud tools
Client Side, Server Side or Both? Both.
Frequentist or Bayesian? Frequentist approach.
Does it support Full Stack? Yes.
Core Web Vitals Ready? Yes. This tool loads fast.
Has ML or AI? Yes
Do they serve Enterprise? Yes. Almost exclusively due to the cost and requirement to be an analytics customer also.
Do they offer Customer support? Yes, in the form of a knowledge base and video training programs.
Do they care about Privacy? Yes, they are fully GDPR compliant.
Do they care about the World? Very much so. They champion diversity, work towards 100% renewable energy buildings, lowering emissions, running community action programs, and much more.
What this A/B testing tool looks like in action:
#7: VWO Testing
VWO is another entry/mid-level tool. It’s great for those either looking to start out in the A/B testing space or looking to start scaling their testing further.
Although it’s an A/B testing platform, it has a focus on marketing features also and A/B testing or assisting certain digital marketing campaigns, and has an intuitive UI design.
They offer the major A/B testing features that you might expect, along with the ability to run heatmaps, click-through recordings, on-page surveys, full-funnel tracking, full-stack tracking, and cart abandonment marketing features.
Another clever feature is their “Deploy” option, which allows you to make changes on the fly and push them live, separate from a test (although I highly recommend testing site changes first). Even though most A/B testing tools can do this, they are the only ones to advertise it.
One major downside is the WYSIWYG editor can break sometimes, deleting the code and test setup before it goes live, so just be sure to save a copy of the code before you hit publish.
G2 Rating: 4.2/5.0 (239 Reviews)
Pricing: Starts at $199/mo. Anything more requires customized pricing according to your needs. And you’ll have to contact the VWO sales team for yours.
Do they offer a free plan? Yes, you can get started with a 30-day free trial.
Cost per 100,000 visitors: ~$500
Pros:
- Flexible customization options to adapt your test to a lot of scenarios
- Easy to plan and execute tests with little coding knowledge
- Dedicated support team to guide you through any challenges
- Has the ability to track long-term goals
- Ability to group tests together
- Useful recordings for observing users and troubleshooting
Cons:
- There’s a learning curve to understand the full functionality of VWO
- Pricing plans change frequently
- They only test client side
Client Side, Server Side or Both? Client side only.
Frequentist or Bayesian? Bayesian approach.
Does it support Full Stack? Yes.
Core Web Vitals Ready? Yes.
Has ML or AI? No.
Do they serve Enterprise? Yes. Their enterprise clients include Ubisoft, AMD, Penn Foster, Optimeria, and IMB Bank.
Do they offer customer support? Yes, you can call them when you need help. Or consult their resource page.
Do they care about privacy? Yes. They comply with the major privacy laws and regulations but they store some visitor behavior data on their servers.
Do they care about the world? Recently, the chairman of Wingify (the brand behind VWO) Paras Chopra, tweeted their support for a couple of COVID relief initiatives including setting up a 10-bed COVID-care facility in Delhi.
What this A/B testing tool looks like in action:
#8: Kameleoon
Kameleoon is a French-based A/B testing platform that has been praised for its UI, personalization settings, and integrations. It’s also the tool of choice among Healthcare companies thanks to its emphasis on data protection as well as Fintech companies seeking data-driven solutions with an eye towards privacy concerns.
G2 Rating: 4.7/5.0 (24 Reviews)
Pricing: Customized according to your requirements. You get your unique price by contacting the sales team.
Do they offer a free plan? Yes.
Cost per 100,000 visitors: N/A
Pros
- Easy to set up click tracking
- Smooth integration with many other apps
- Has an easy-to-use WYSIWYG editor for non-developers
- Advanced anti-flicker technology
- Knowledgeable and helpful support team
- Accurate and detailed planning and execution of tests
Cons
- WYSIWYG editor loads slowly
- The reporting dashboard could use a bit more personalization
- Need developer-level skill to implement some complex scenarios
- Cannot archive tests
- Old browsers like IE7 are excluded from the tests
Client Side, Server Side or Both? Both.
Frequentist or Bayesian? I honestly couldn’t find out which they use.
Does it support Full Stack? Yes.
Core Web Vitals Ready? Yes.
Has ML or AI? Has AI personalization options.
Client Side, Server Side or Both? Both. You can make edits on the WYSIWYG editor as well as use Kameleoon’s in-built code editor to edit CSS and JS.
Do they serve Enterprise? Yes. Lexus, Unity, Providence, Toyota, and Rakuten use Kameleoon.
Do they offer customer support? Yes. You can even get a dedicated account manager to assist you with complicated projects.
Do they care about privacy? They are HIPAA, GDPR and CCPA compliant. They also have a flexible consent management feature to adapt your privacy policy for each A/B test.
Do they care about the world? I couldn’t find evidence of any charity they sponsor or environmental causes they support. Although, it’s interesting to know they have two live chameleons in their Germany and France offices.
A/B testing software review from TrustRadius:
What this A/B testing tool looks like in action:
#9: OmniConvert
Primarily a testing tool for e-commerce, but that’s not to say that you can’t use this for other business types and sites. Another fantastic beginner to mid-level all-in-one platform for those looking to start and then scale up their testing program.
OmniConvert’s suite of tools includes four complementary platforms: Explore which is their A/B testing platform, Reveal which is a customer retention and churn tracker, Adapt which is an automated CRO platform and Survey, a feedback tool for qualitative analysis (all on separate pricing).
G2 Rating: 4.5/5.0 (70 Reviews)
Pricing: Plans start at $167 a month on an annual plan for 50k visitors. ($320 a month on monthly pricing). Additional costs apply if using Survey or other tools.
Do they offer a free plan? Yes.
Cost per 100,000 visitors: N/A
Pros:
- Integrates with a suite of features and also 3rd party tools
- Simple setup for tests
- Multiple segmentation, targeting and personalisation features
- Integrates easily with your analytics
- Great customer support
Cons:
- Not the cheapest solution out there at this range
- Intuitive UI but not for absolute beginners
- Sometimes the visual editor can bug out.
Client Side, Server Side or Both? Both.
Frequentist or Bayesian? Both.
Does it support Full Stack? Yes.
Core Web Vitals Ready? Yes. This tool loads fast.
Has ML or AI? Yes.
Do they serve Enterprise? Yes. It’s the tool of choice for Orange mobile, WWF, Samsung and more.
Do they offer Customer support? They have a resource hub with tips, video tutorials, help community, live chat and more.
Do they care about Privacy? They comply with applicable data protection laws in the countries where their products are used.
Do they care about the World? I couldn’t find any record of charity/community work, but they may simply not share that with their audience.
A/B testing software review from TrustRadius:
What this split testing tool looks like in action:
#10: Dynamic Yield
Dynamic Yield is another Enterprise focused experimentation platform.
Ideal for very high volume sites, it offers A/B testing, omnichannel personalization and predictive test improvements via machine learning along with many other features.
Because of this, however, it’s not ideal for beginners.
G2 Rating: 4.3/5.0 (82 Reviews)
Pricing: Pricing only available after speaking to sales.
Do they offer a free trial? You have to request a demo first.
Cost per 100,000 visitors: N/A
Pros:
- Multi-year, award-winning software in the A/B testing and AI space.
- Has advanced personalization capabilities to run predictive targeting of messages to your audience.
- Full stack integration
- Omnichannel testing and tracking
- Fantastic support and education
Cons:
- Can be overwhelming to set up due to the large number of options
- Some essential features require developer setup assistance
- Not the best integration with analytics
Client Side, Server Side or Both? Client-side and server-side testing.
Frequentist or Bayesian? Bayesian approach.
Does it support Full Stack? Yes.
Core Web Vitals Ready? Yes. This tool loads fast.
Has ML or AI? Yes.
Do they serve Enterprise? Yes (Almost exclusively so). They are the tool of choice for companies such as Urban Outfitters, Lacoste, Pacsun and Hallmark.
Do they offer customer support? Yes. They offer call and email support, along with a basic knowledge base to help with installation.
Do they care about privacy? Dynamic Yield is fully GDPR compliant.
Do they care about the world? They actually do something very cool, which is to offer free use of their tool to charities that meet certain criteria. They also do work in the community.
A/B testing software review from TrustRadius:
What this A/B testing platform looks like in action:
Learn More About A/B Testing Tools
What Are A/B Testing Tools?
A/B testing tools:
- Allow you to test 2 (or more) elements on your website, to see which change drives the most favorable output.
- Use a form of statistics (usually Frequentist or Bayesian) to measure the actions that your visitors take during that test and then provide the results. You can run that test to get a statistical significance and level of confidence that if you push that test result live, you can expect similar results from the rest of your audience.
- Usually, come in two varieties. Either open source which allow you to keep your data in-house and use right away, or a closed source that usually asks you to contact them before you can get started.
How Can You Set Up an A/B Test?
We cover how to set up and run A/B tests in far more detail here, but let’s give you a quick overview.
How do you do an A/B test on a website?
First, come up with a hypothesis or idea of how you think you can improve or fix a page to get more conversions.
Then, create the assets needed for your particular test i.e. if you are testing different images.
Set the test up inside the A/B testing software and test a single specific element at a time to see which works best. (A multivariate test lets you test multiple variations, but in an A/B test, it is just 2 options. It helps give faster results, especially on lower traffic sites. With A/B testing, you can pinpoint the exact element that impacts user behavior the most.)
Then, make sure the test runs long enough to get accurate results.
And finally, analyze and learn from the outcomes.
Key Features Needed in an A/B Testing Tool
Before we break down the growing trends in A/B testing and the best tools for it, let’s look at the key features that every optimization platform should have:
- Simple setup and installation
- Customer support
- Easy to use UI
- WYSIWYG editor
- Code editor for custom tests
- Flicker free testing so as not to negatively affect your test
- A/B testing, split URL testing, multivariate testing, multipage testing capabilities
- Targeting and segmentation abilities
- Personalization
- 99% uptime so it doesn’t break mid test
- Unlimited test capabilities
- Unlimited variations
- Multi-filter targeting options
- Detailed reporting that you can dive into
- 3rd party integrations and API
- Security, privacy and data protection
Anything else is either a bonus or nice to have, but without these core features, you may struggle to run tests effectively.
Top Trends in A/B Testing & Why You Should Care When Choosing A/B Testing Tools
What does the future of CRO hold?
As we look at it, there are 4 major trends in the CRO industry right now:
1. Using CRO as a more holistic tool for larger data analysis,
2. Concern over how testing might affect organic traffic and page experience,
3. User privacy and how this relates to testing and tracking,
4. Machine learning in A/B testing — what is it and is it worth it?
1. A/B Testing for Decision Making
Traditionally, CRO and A/B testing have been all about website optimization:
How do we get the traffic on our web pages to take more of the action that we want?
Nowadays, however, it feels like conversion rate optimization and the entire landscape are evolving to a more holistic and strategic view. (Even the name CRO doesn’t quite fit with how it’s changing.)
Rather than just A/B testing software to measure conversion rates, marketing teams are now looking at it from the perspective of measuring and understanding all audience interactions to make larger business decisions.
CRO as a practice isn’t drastically changing, but its place in the organization has shifted.
What was once the domain of a sole team, or in some sad cases, a sole individual, is now an operating system by which data-driven teams across a company make decisions.
I’ve seen the shift from decentralized models to centralized teams, where all CRO efforts are owned and guarded by a core team of experts and specialists. Turns out, this doesn’t scale super well.
Now this model, in many mature organizations, is turned into a “center of excellence” model. In this way of operating, the CRO team simply enables and supports the rest of the organization’s experimentation and optimization efforts through education, process management, resource allotment, and product management (typically it’s at this stage that company’s invest in custom tooling or data pipelines to enable more sophisticated experimentation and greater scale).
Additionally, “CRO” as once practiced, is essentially called “web strategy” or “website product management” now.
Which makes a ton of sense, because growth product managers on the product side are doing the exact same work and calling it something different. This blending of the marketing and product side has yet to fully eclipse the previous silos, but when it does, it’s going to enable much more interesting full-funnel experimentation.
The tools that will help to enable this are the ones that balance server and client-side experimentation, have robust integrations, and have advanced features like experiment swim lanes as well as a mix of marketer-centric features (like your typical WYSIWYG editor) and product-centric features like feature flags.
Alex Birkett, BeOmniscient
Let’s say you run some experiments on a few web pages and you see that your audience is resonating with a particular angle and it’s converting more.
In the past, we would have focused on that simply as a better way to convert.
But now we might start to look at this deeper. Not only can we pass this information on to other areas of the company, but we can research this angle more.
- Is this a change in the industry that’s occurring?
- Is this becoming more important to our audience and something we should focus on?
- Why is it important to them?
Learning about this could help us not only convert better but improve conversions front to back. This can then improve everything from ad results to home page offers, upsells, average sales and repeat sales, etc.
Not only that, but it can help marketers look at new product offerings, and even change company-wide positioning or pricing.
(Maybe moving from a single price model to a recurring model to improve retention while also giving more breathing room for profitable acquisition.)
How Does CXO Affect Your A/B Testing Tool Choices?
Ideally, we’re looking at quantitative testing tools (like Convert Experiences) that can integrate with other optimization tools you might use to help you fulfill these needs.
These could be QA tools so that you can learn more about the reasons why this is important to your audience, customer success tools, or even communication tools so that you can share the information internally.
Convert has 100+ integrations with other tools in your tech stack.
High-quality, measurable data is critical to decision-making. And part of having good quality data is test design: deciding when and where to trigger tests in the funnel. Picking a powerful tool with advanced targeting capabilities, like Convert Experiences, will make test design easy. You can trigger tests after certain goals have been met, like clicking on particular CTAs or visiting specific areas of a website.
2. A/B Testing that Minimizes Impact on Page Experience
Google recently updated their Page Experience ranking algorithm, by adding in their Core Web Vitals update.
We cover this in detail here (along with some best practices on how to make sure your tests don’t mess up your score), but here’s a quick overview.
The CWV update is all about improving the user experience on the page:
- How fast it loads
- How quickly the user can understand what the page is about (LCP)
- How soon they can interact with the page
- How quickly the page responds after that interaction request (TBT)
- How much the page elements shift as the page loads or is used (LCS)
How Does Page Experience Affect Your A/B Testing Tool Choices?
If you’re looking specifically at how your testing solutions may affect your Core Web Vitals score, then you need to look at:
- Tool Speed: How fast does the testing platform work?
- Flicker Rate: Does the experiment cause flickering elements?
- How does it test the element: Does it hide, move, or cause layout shifts?
It may seem like a small thing to care about now, but it’s clear that Google is making big changes that are focused on page and user experience.
You really don’t want your testing tool to start affecting your traffic or customer experience.
3. A/B Testing that Respects Privacy
In the last few years, there’s been a large push towards improving user privacy online.
Everything from how we handle users’ data and consent, to how we track and what we track, and now we’re seeing specific devices and browsers remove user information or limit tracking time frames.
This is a good thing for the user and something we should respect as site or app owners. (Here at Convert, we don’t even work with other 3rd party tools unless they meet GDPR – even if it’s not a client-facing tool).
However, these bylaws and features only affect client-side tools and not server-side testing tools.
What’s the Difference Between Client-Side and Server-Side Tools?
Client-side tools are where the tool is on your website via a Javascript code in your header. It tests and tracks by making changes in the user’s web browser to show different versions of page designs, etc.
Server-side tools are instead installed on your web server.
They each have pros and cons: Client-side tools are usually faster and easier to set up new campaigns, while server-side tools allow you to track things you might not usually track with a standard tool.
The main difference when it comes to privacy is how these new laws and tools are implemented.
You see, pop-up blockers and things like this can stop client-side tools from working until the user agrees to it. The browser or device sees the cookie event and stops it from triggering and reporting back.
(Users without blockers in place will hopefully see a GDPR consent form asking them if they agree to be tracked and share particular user data. Some sites, however, will track as standard and ask them to opt-out instead).
The thing is, server-side tools are not stopped by any of this. They don’t rely on the web browser cookies to send information as it’s tracking all your site events and sending them to your server instead. This means that server-side tools can still track events and user ids, then feed that information into the testing platform, 3rd party analytics, or ad platforms.
You could in theory track away to your heart’s content and never tell anyone. (While risking a potential lawsuit if they get access to your server data.)
But just because you can track all that information, should you?
A/B testing can be the best way to mitigate the risks of damaging the user experience in the future. If you think about it, testing is all about proving a hypothesis that one version is better than the others. So you really don’t need to store anything more than the number of users seeing these versions and the goals you’re tracking. Just that… two simple numbers, variationID and GoalID, and the number of times they are seen. You should set up your A/B testing in this clean way, nothing more.
You lose the trust of your users if you try to slice and dice your data in search of a winner. That’s not what A/B testing is for, that’s analytics — don’t mix it up.
Mitigate the risk of hurting your users and get an occasional win for the business. It will win you trust, save you money and so make you the best version of what you can be for your users.
If you keep it clean, you won’t be impacted by privacy laws. You don’t store any personal data and don’t track users. Your tracking usage… that is long-term the future of this niche in CRO.
Dennis van der Heijden, Convert.com
How Does Privacy Affect Your A/B Testing Tool Choices?
Regardless of if you use client-side tools for the convenience of set up or server-side tools for more in-depth tracking, you should still as a responsible business meet standard and upcoming privacy changes:
- Respect Do Not Track browsers
- Ask for consent
- Track events instead of user data
- Anonymise IDs, etc.
With that in mind, be sure to look at your experimentation platform’s privacy goals and what they allow you to do with it:
- Do they meet privacy requirements as put down by the GDPR and similar laws?
- Do they store data safely so your audience doesn’t get leaked and you don’t get fined?
- Do they allow you to test and respect users’ privacy while doing so? I.e. can you set those settings in your test campaigns, even if it’s a server-side tool where technically you could get away with it?
- Do they allow you to adjust elements of your experiment for different location laws?
- Finally, you might want to check if your tool is client-side only, or if it can be set up server-side also.
But remember: Just because you can track all that information with a server-side tool, it doesn’t mean that you should. Use a tool that cares about your audience’s privacy!
(We removed a lot of our in-depth user tracking details from our tool because of this. We can still track what’s needed for controlled experiments, but we don’t need to do it while knowing too much about the user).
4. A/B Testing Powered by Data & Machine Learning
The final trend over the last few years has been around integrating Machine Learning into CRO testing:
CRO is beginning to change and will very apparently change over the next 5 years, in two fundamental ways. CRO is becoming more empirical and data-sophisticated. (…)
In many ways, this is just a reflection of what the FAANG (Facebook, Amazon, Apple, Netflix and Google) companies have known and practiced for the last decade. Having realized the imperative of testing everything across the business and running as many experiments as technically feasible, FAANG companies have hired scientists to develop what are quite literally fundamental advancements to the field of statistics. These advancements allow them to run more experiments, run experiments faster, and measure increasingly complex outcomes (e.g. Bing measuring “did a user successfully find what they were searching for?“, or Netflix measuring how different approaches to loading videos on slow connections impacts user frustration).
The areas where CRO was previously thought of as more of an “art form” (guiding program strategy, roadmap prioritization) will increasingly rely on science and machines that can make more optimal, and empirical decisions about how to prioritize potential treatments to test and what audiences to test on.
Ryan Lucht, Senior Growth Strategist, Cro Metrics
What Is Machine Learning?
Machine learning is a type of artificial intelligence, but in simple terms, it’s a pattern recognition machine. If you give it enough data points (and it does need a lot), then it’s able to find connections in that data that humans might miss, and then predict potential future actions.
For example, Netflix uses machine learning to understand more about each user so that it can recommend shows that it thinks you might like.
This helps Netflix to improve their customer experience, lower churn, and even learn what genres of content they can invest in.
How Does Machine Learning Affect CRO?
Machine Learning’s real benefit is for companies that have a lot of traffic and conversions already, want to scale their testing program, and gather as much data as possible.
It allows them to set up multivariate campaigns, let the tool learn from actions taken, and optimize tests live. Even more interesting, it can learn from ‘almost’ conversions so you can start to gather a lot of data about those very warm leads who are close to converting.
Amazon is a great example of using machine learning for real-time personalization.
They can adjust front page recommendations, upsells, pricing, and even landing pages, all based on previous user behavior data of people who are similar to you (same interests), or by tracking your past interactions and predicting potential future actions. This personalization alone helps them to improve their sales by 35%.
How Do Machine Learning and AI Affect Your A/B Testing Tool Choices?
The first thing to understand is that Machine Learning tools are not for everyone.
ML works best when it can be given more data. Unless you’re doing around 100,000 website visitors a month, or 1,000 conversions at the minimum, then it’s not for you. (You might be able to get away with it if you had bulk historical data perhaps, but your tests will be stunted if you’re not hitting that 1000 conversion per month threshold.)
Like any testing process, machine learning is only as good as the data that you are feeding it. If something is broken in the test, or you’re feeding it the wrong or not enough data, then it will struggle to give you the answers you’re looking for.
You still need to analyze why you are getting the results back. It’s not a robot you can plug in and ‘do CRO’ for you.
What if you wanted it to help you scale out and run multiple experiments at once, or to start deep learning about large data sets?
An A/B testing tool with Machine Learning can help you do this much faster than you could on your own.
Conclusion
So there you have it. The 4 emerging trends at the forefront of conversion rate optimization right now, how they might affect your tool buying decision, and the 10 best A/B testing tools that we think either meet those requirements or are working towards it.
If you’re looking to get an A/B testing platform and want a future-proof choice, you can’t go far wrong with this list.
