5 Ways Digital Marketers Use Call Tracking to Grow Their Business
Blog /A/B Testing

Using Google Optimize & Other Free A/B Testing Tools? Shift When You See These 5 Things…

14th Jul 2021 –

Can an A/B testing tool truly be free?

No, not really. You may think that paying for the service is just a matter of handing over your credit card information but have you ever considered what else you might be giving up in exchange if somebody asks nothing from you upfront?

Sure, there are quite a few free and open source A/B testing tools out there.

But have you ever wondered if the reason you’re not paying for a service that costs money to build and maintain is that you are paying in some other way?

When you’re starting out, free tools like Google Optimize are great. They help you understand and apply the fundamentals of experimentation.

But as you advance, there are things that’ll make you want to rethink your preferred A/B testing software. And that’s not because free tools don’t get the job done, but because there are limitations you’d want to shoot past.

Let’s show you…

Is There Any Such Thing as Free A/B Testing?

If money is the only factor you’re considering, then A/B testing can be free.

There are many tools out there that cost $0 to run tests and give you results.

But if you’re looking at it from a business point of view, factoring in other resources that contribute to success or failure, then you’d probably want to look deeper.

Many of these free tools are designed for first-time testers to get their feet wet in marketing experimentation. When these marketers want to test further or implement more advanced strategies, they quickly find that their free tools can’t perform these.

And if it was a freemium tool, you’ll see your path to advanced features blocked by a “sign up for a pro plan” notice.

But that’s not all.

For most free tools, you’ll be sacrificing more than just advanced features.

Free vs. Freemium A/B Testing Tools (With Examples)

One of the best free A/B testing tools is Google Optimize — the successor of Google Analytics Content Experiments.

Google Optimize gets you started with:

  • A/B, A/B/n, Multivariate, and Split URL testing
  • Delivering personalized experiences to your web visitors
  • Creating variants of your site with a responsive WYSIWYG editor
  • Editing the HTML, JS, and CSS of your site in a basic code editor
  • Integrating with Google Analytics for extracting valuable insights, and more…

Freemium tools, on the other hand, take it from where free tools stop. You start with a free plan that gives you basic features or use pro features for a limited time, and then you have to pay.

But it is usually worth it. These are common features you can expect from using them:

  • No limits on the number of tests you can run at the same time
  • No limits on your number of goals
  • Ability to plan and execute more complex tests
  • Run tests on apps and websites (not just websites)
  • Integrate with many other marketing apps you use
  • Customer support

And much more depending on the tool you’re using.

One example of this is Nelio, an open-source A/B testing software for WordPress and WooCommerce sites. It is free for 500 page views per month. If you have up to 5000 visitors, you’ll need to pay the starting price of $29/month. It comes with unlimited experiments and technical support.

Also, you should know there’s another type of free tool you can opt for — Open source A/B testing tools.

What’s different about them?

Free vs. Open Source A/B Testing Tools (With Examples)

Open source A/B testing tools provide some premium benefits while being free.

Some examples are:

1. Wasabi

This is a totally free open-source A/B testing tool by Intuit. So what’s the catch? It is no longer under active development and there’s no support. Also, you need some technical knowledge to deploy this 100% API-driven experimentation platform.

2. Vanity

Vanity also lets you run experiments on your websites and apps. But just like Wasabi, you may need a developer to help install and configure Vanity to run your A/B tests.

3. Mojito

It’s an open-source experimentation stack that lets you build and run tests with git.

4. Split

Split is a rack-based A/B testing framework that allows for maximum customization and extensibility.

But of course, there’s no support team. And that’s tough because deploying these tools usually requires a strong technical background.

It’s just you, the tool, and your experiments. If anything goes wrong or you don’t know how to take a certain action, you have to hope it’s in the documentation or there’s a help community/forum to run to.

Also, no one is obligated to get rid of any bugs you may find.

One free tool stands heads and shoulders over all these open source tools. That’s Google Optimize.

Although a favorite among marketers, it wasn’t designed to be a total experimentation solution. But there are times when using Google Optimize makes perfect sense.

When Is Using a Free A/B Testing Tool like Google Optimize Okay?

Besides being a fantastic introduction into the world of website A/B testing software, free tools like Google Optimize are great for:

  • Proving that a new design idea works on your site
  • Making minor changes towards improving your website conversion rate
  • Getting assurance that a proposed idea will not lower your site’s current performance, and
  • Generally, helping you make informed decisions that boost your bottom line with your existing traffic

But ultimately, optimization teams eventually grow past this.

Lorenzo Carreri, CRO and Experimentation Consultant, said:

Free A/B testing tools are good for those that are starting out in experimentation. But there are a few limitations, like anything else that is free in this world. First of all, depending on the tool, oftentimes you are limited by the number of experiments you can run, so if you want to increase experimentation velocity, that’s a challenge. Secondly, in my experience, I’ve come across bugs and issues in the way the tool randomizes the traffic. And this of course completely pollutes and invalidates your tests. As you can imagine, this is quite annoying because you have just wasted a lot of hours designing, building, and QAing your test. And you are now stuck with your experimentation program until the bug/issue gets fixed.

The Google Optimize Debate

What Is Google Optimize?

Google Optimize lets you run A/B tests to improve your website performance for free. It has a simple interface that anyone can easily figure out for their first test — and it integrates with Google Analytics perfectly.

It’s also compatible with Google Ads, Google BigQuery, Accelerated Mobile Pages (AMP), etc. And apart from A/B tests, you can run redirect, personalization, multivariate, and banner tests on Google Optimize.

Other great things about Google Optimize is that you can:

  • Easily troubleshoot on your own with little technical knowledge,
  • Define a detailed audience segmentation without stress, and
  • View test reports in an easy-to-understand interface and terms.

This is perfectly suited to A/B testing for small businesses. But when you’re actively working on growth, you’ll quickly find that you’re limited in so many ways.

Since Google Optimize is a free tool, they don’t use a lot of resources. For example, you can’t upload a picture directly to the platform. It has to be hosted somewhere else.

When your needs grow past a certain point, Google Optimize will no longer be able to handle your tests. You’ll have to upgrade to the paid option, Google Optimize 360.

Other limitations are:

  • You can only 5 tests at a time
  • You cannot set more than 3 goals
  • You can only test websites, not apps
  • Your tests can’t run for more than 90 days
  • In multivariate testing, the maximum number of variables is 16
  • You cannot write more than 10,240 characters of code in Google Optimize’s code editor
  • And if you’re not using Google Analytics, it can be a hassle to integrate Google Optimize with your “alien” analytics tool

Now, don’t get me wrong. We are not bashing this beloved tool. We’re only offering you a different perspective.

google optimize tutorial

Google Optimize wasn’t designed to be the beginning and end of your A/B testing. That’s why they offer the advanced 360 version at a custom price.

Besides, looking at both Optimize and Optimize 360 as one service shows that, in essence, Google Optimize is a freemium tool.

If you want to scale your A/B testing, switching to a paid option is the smart way to go.

5 Reasons to Shift from Google Optimize

As convenient as no-cost tools like Google Optimize are for A/B testing, there comes a time to make a shift.

We recommend making the switch when the 5 following issues start to impact test velocity or introduce opacity in how the results of critical tests are calculated:

1. A Too Generous Interpretation of Bayesian

In Optimize’s documentation, Google states that they’re using the Bayesian inference “to better answer the questions that marketers actually have.”

Here’s a bit of background on it:

Bayesian inference is a method of making educated guesses by using previous information to update the probability of an event taking place.

Here’s an example to illustrate this in simpler terms.

Imagine you were blindfolded and asked to pick from a basket of 3 oranges and 1 apple. The chances that you’ll pick an apple are pretty slim, right? 1 in 4, or 0.25, so a 25% chance of picking an apple.

What if you were told that the last 3 people who were blindfolded and asked to choose, all picked an orange? How does that make you feel about your chances of picking an apple now? Pretty confident, right? That’s because you know that there were only 3 oranges, and if 3 people picked them already, you have a 100% chance of picking the apple.

So you are mentally adjusting the probability of the event of picking an apple being true based on knowledge of prior events.

And that’s the framework Google Optimize uses — And many would argue that it lacks rigor and is too opaque, especially when it comes to “prior” selection!

Here’s what I mean.

In the documentation, their answer to “What is ‘probability to be best’?” goes:

Bayesian priors are modeled beliefs about how we think a variant or experiment will behave. […] For Optimize, we use a variety of priors.

Note that this is predictive and not based on final results. This is in addition to the fact that prior assumptions/beliefs are factored into your data. And Google Optimize doesn’t let you know what these prior assumptions/beliefs are exactly.

On the other end of the spectrum, we have a frequentist-based statistical methodology used by other testing tools.

According to Wikipedia,

Frequentist probability or frequentism is an interpretation of probability; it defines an event’s probability as the limit of its relative frequency in many trials. Probabilities can be found (in principle) by a repeatable objective process (and are thus ideally devoid of opinion).

Frequentist inference allows you to make clear “go” or “no-go” decisions after running well-defined random experiments with minimal risk of influencing the results with biases.

This is what Convert Experiences uses for its tests.

Convert Experiences experiments frequentist two tailed Z test

2. Feature Restrictions

Google isn’t interested in offering all of its testing capabilities to everyone. This would be counterintuitive and lead to wasted money on resources such as servers which cost real dollars. Without feature limitations on the free Google Optimize, lots of people would be using it, putting immense pressure on the company.

So it just doesn’t make sense for Google, which is a for-profit business, to do that.

If you’re just starting out, this doesn’t affect your optimization goals that much. But when you’re growing, the feature restrictions of Google Optimize will stifle your growth.

Like we mentioned above, these restrictions include a maximum of:

  • 5 tests
  • 3 goals
  • 90 days runtime
  • 16 variables, and
  • 10,240 characters in the code editor

And there’s more…

You:

  • Cannot test complex features
  • Cannot add goals while the test is running
  • Cannot customize a Google Analytics audience

Also, there’s no feature to push winners to personalizations. This allows you to validate these winners by attempting to replicate the lifts you got from the experiments without permanently changing things immediately.

Convert Experiences has that feature.

3. Lacking Support

The truth is, no business-minded developer will pay for a dedicated support team to serve non-paying customers.

So as far as support goes, the best you can get from Google Optimize is their resource hub.

On the plus side, it is very detailed. You get everything from tips, Google Optimize video tutorials, community, onboarding tips, etc. But there is no live support.

No one to call, no live chat, no email support. You can only get that when you opt for the paid option — Google Optimize 360.

Google Optimize Resource Hub

4. SEO Impact as a Result of Testing

There are certain best practices for running A/B tests without negatively impacting your SEO.

But, if you run A/B tests using Google Optimize, you risk hurting your SEO anyway.

That is because of flickering — which is what happens when a user visits the page you’re testing and sees the original version of that page for a split second before their variant is displayed.

Not only does this impact the quality of your test results, but it also hurts the user experience.

And Google is big on user experience with respect to SEO. 

To solve this problem, Google Optimize has an anti-flicker snippet you can add to the page.

This anti-flicker snippet works by making your website elements transparent (or, invisible) until the variant from Google Optimize’s JavaScript loads. So, instead of seeing a split second of the original, the page remains white until the variant is ready.

But there’s a problem with this snippet that even Google Optimize recognizes with this warning on the “install the anti-flicker snippet” help page.

They say: 

Only install the anti-flicker snippet when necessary as it can impact on page performance.

Because it prolongs the page loading time. So, the users viewing the variant have a poorer experience than those who get the control.

So despite your effort to remove flicker with the snippet, your SEO still takes a hit.

Convert Experiences serves your variations fast, without any annoying flickering.

5. Tested Visitor Count Inflation

Your visitor count is the number of unique monthly visitors coming to your site and participating in your active tests. This count is not how many times (sessions) a certain visitor has visited your site, only how many of these unique visitors there are.

Google Optimize has a way of logging your visitor count that negatively affects your test results.

Google Optimize uses the Google Analytics rule where a session is measured as 30 minutes and can sometimes be stretched to 4 hours. What this means for you is that if someone visits your site at 9 AM and then returns by 2 PM (5 hours later), Google Optimize logs that as 2 unique visitors.

This makes your conversion rates lower than they actually are, making your results less accurate and impressive.

Why is this so? Google Optimize is a free testing platform. To minimize the use of server storage for keeping user data, they have to keep these sessions short.

So, if you need better results, consider a paid alternative that’s worth every buck you invest in it.

How to Choose An Alternative to Free A/B Testing Tools

You will hit natural stopping points with free testing solutions. And this can impede the results you get from your experiments.

But that’s fine. This only means you’ve advanced to the point where you’re no longer a starter.

And even if you’re not at an enterprise level yet, you can find the right tool that’s suited for that sandwich position you’re in.

Here’s how:

1. Figure Out Your “Why

Why are you choosing to switch over to a paid tool?

When you understand what limitations you’re looking to leave behind and the features you want to start using, it’ll help you make a great decision.

Are you trying to get rid of flickering? Are you looking to run more concurrent tests? Or are you searching for expert support as you scale your business?

When you’ve found the features that are fuelling your migration, the next thing to do is evaluate the available tools for certain criteria.

2. Do an A/B Testing Tools Comparison

Not all tools are created equal. Of course, features are one differentiating factor. But there’s more to this.

Even if they offer the same features for the same price, there are other below-the-surface characteristics that may matter to you and are worthy of evaluation.

You may have questions such as:

  • What’s their stance on user privacy?
  • What pricing model are they using?
  • Do their features make sense compared to the cost?
  • Are they showing care for the world?

Need a list of tools to evaluate? Check here.
Or take a look at this comprehensive A/B testing tools comparison to find the best option for your needs.

3. Shortlist Your Preferred Tools

At this point, you may have found 3 or 4 tools that fit your idea of an ideal tool.

For some tools, your initial evaluation may hit a stone wall when you’re checking out pricing. Many A/B testing tools only reveal a custom price after many talks with sales reps.

When you’ve figured out your shortlisted preferred tools, test them with a free trial. And then, find your winner.

As you’re making the shift from Google Optimize (or any other free A/B testing tool), you may be walking into a more complex environment. Paid tools are more equipped and can be complicated compared to free alternatives.

That’s why your winner should come with great customer support. Whether it is live chat or 24/7 phone and email support, you’ll need reliable support you can always fall back on so that your work doesn’t slow down because of any challenge.

Go for a tool whose users speak of the support team this way:

  • “Customer support team is fast, knowledgeable and solution oriented.”
  • “Support is always friendly and helpful.”
  • “Support is top-notch and very responsive.”

The chosen tool should have the right support team to help you make a smooth switch from free to paid A/B testing tools and scale your testing program.

The Shortcut to Finding an Alternative for Free A/B Testing Tools

If you need to find an alternative to a free A/B testing tool but don’t want to shell out the cash for it yet, get a free trial of Convert Experiences. It gives you free access for 15 days so that you can evaluate how well it fits your needs before making any purchase decisions. You can integrate with over 100+ tools and run A/B tests on your terms. No limits.

Conclusion

Free A/B testing tools like Google Optimize are ideal for kickstarting your experience with testing UI, UX, websites, apps, etc. But naturally, these are designed to have limits since they’re not bringing in revenue to the owners of the tools.

What this means for you is that when you need to do more than basic experimentation, the features of these free and open source testing tools will limit you.

To cross over this barrier and take your optimization to the next level, you have to go for the paid alternatives. The amazing thing is that you can get an affordable & powerful tool like Convert Experiences which delivers limitless features with a dedicated support team to help you maximize your ROI.

Originally published July 14, 2021 - Updated July 23, 2021

Start 15-Day Free Trial Of Convert Experiences.

Quick form submission
Quick submission.
No credit card needed
No credit card needed.
You can always change your preferences later.
Check Icon
You're Almost Done.
Managing Marketing Team
Managing Tech Team
Hypothesizing Experiments
Coding and QA of Experiments
Convert is committed to protecting your privacy.

You've successfully submitted your details to start your Convert Experiences 15-day free trial.

We are setting up your account for you. You'll receive a welcome message to your registered email with your login access. Please explore the app on desktop for an optimal experience.