5 Ways Digital Marketers Use Call Tracking to Grow Their Business
Blog /A/B Testing

A/B Testing: A Guide You’ll Want to Bookmark

02nd Oct 2020 – A/B Testing Ultimate Guide

A/B testing is a MUST for any growth-driven organization. It doesn’t matter what industry you are in. If you have an online property as one or more of your customers’ touchpoints, you want to make it a well oiled-machine that converts the greatest amount of visitors with the least amount of effort.

A/B testing allows you to do this. 

A/B testing is part of a larger conversion rate optimization (CRO) strategy, where you collect qualitative and quantitative data to improve actions taken by your visitors across the customer life-cycle with your company. 

This A/B testing tutorial will go over everything you need to know about A/B testing to make you a better CRO expert, no matter what your specific role is. This includes:

  • Definition and benefits of A/B testing.
  • What and how to test.
  • How to overcome common A/B testing challenges and mistakes. 

Let’s dig in…

What is A/B Testing and How Does A/B Testing Work?

What is A/B Testing?

A/B testing is an experiment that compares two or more versions of a specific marketing asset to determine the best performer.

Two versions are tested against each other. A control or original, usually A, and a variation of A, usually called B. Traffic is split at random so that one group sees version A while the other sees version B. Statistical analysis is used to determine the winner of each test.

Alt Text: What is A/B Testing? A variation(b) against a control (A)
A/B testing compares a control against a variation and statistically determines a winner.

Once you set up your A/B test, your visitors will see either your control page or a variation of that page. 

It is random who sees which page.

Metrics like the number of conversions, currency transactions for each page, etc, are calculated and tracked.

A/B Testing Guide, examples of A/B Testing software, Convert Experiences, Analytics Dashboard.
Example of A/B testing software: Convert Experiences, Analytics Dashboard.

NOTE: This guide focuses on testing website elements, but the concept of A/B testing can also apply to systems and processes. 

Based on the results, you can determine which page performed best. Answering the question: Did visitors have positive, negative, or neutral reactions to the changes on the page?

Doing many tests means compound data that you can use to decide how to optimize. 

A/B testing is one test in your experimentation program. 

Before we jump into the details of A/B testing, let’s take a quick look at other types of experiments. 

Other Types of Tests

A/B testing is one of the most common forms of experimentation. But it is not the only type. It’s important to lay these out, as a testing program runs in unison with the elements of your entire experiment program. 

It is like you are a research team, gathering data from many areas and a variety of ways to get a 360° view. 

Let’s review the different tests before we dive deep into the world of A/B testing. 

What is A/A Testing?

An A/A test is an A/B test that compares identical versions against each other. 

A/B Testing Guide, what is an A/A Test?
An A/A test is used for quality control. You shouldn’t see any statistical significance in these tests. Therefore, there shouldn’t be any winners.

A/A tests are used to check quality control, baselines, and to ensure statistical calibration of the testing tool used. In this test, A is not the control, rather, the entire test is a control.

Since there are no differences between your two versions, expect your A/B testing tool to report this lack of difference. 

Why and When Do You Run An A/A Test? 

One reason to run an A/A test is to determine your baseline metrics like conversions before implementing a test. 

This makes the data you collect during testing much more valuable, as you have a benchmark to compare it against. 

The second reason, as mentioned above, is to check the accuracy of your A/B testing software

What is A/B/n Testing?

A/B/n tests are another type of A/B test used to test more than two variations against each other. In this case, “n” represents the number of different variants, where “n” is two or greater with no limits (nth). 

In an A/B/n test, A is the control, and B and nth changes are variations. 

A/B Testing Guide, what is an A/B/n test
Example of a A/B/N Test. The Control is tested against more than one variation.

A/B/n tests are great to use when you have multiple versions of a page you want to test at once. It saves time, and you can get the highest converting page up quickly. 

However, remember that for every additional variant added in an A/B/n test, the time to reach statistical significance will increase.

People commonly use the term A/B/n interchangeably with multivariable testing, but they are different. A/B/n test compares the performance of one change, for instance, the wording of a headline. You would test variations that include different copy for that one element (the headline).

As you will see in the definition for multivariable test, it is different than A/B/N test.

Multivariable Testing (MVT)

In an A/B test, one specific change is tested. However, in multivariate testing (MVT), a group of elements on a page are tested. A group comprises two or more elements which each have two variants.

A/B testing, example of multivariate test testing four versions with different combinations of images and headlines.
A/B testing, example of multivariate test testing four versions with different combinations of images and headlines.

What is Split URL Testing?

Split URL testing is similar to A/B testing. The terms are used interchangeably, but they are not the same.

Split URL testing compares the performance of distinct URLs. So it is an A/B whose variants are actually URLs, not elements on the page. They are great when you want to try a new design that doesn’t resemble your original design. 

A/B testing ultimate Guide, Split URL Testing definition
Split URL Testing compares two different websites with different URLs against each other.

They are more complex than front-end A/B testing. They can require some technical know-how (how to build a website). 

For example, if you are looking to overhaul your home page, split testing is ideal. You can quickly identify which tool optimizes conversions

Benefits of Split Testing

A challenge associated with testing is, websites need a base threshold of visitors to validate a test. However, split tests run well for low-traffic sites. 

Another bonus, it is easy to see the winner. Which site got the most conversions?

Multipage Testing

Multipage testing, aka funnel testing, allows you to test a series of pages at one time. 

There are two ways to perform a multipage test: 

  1. You can create new variants for each page of your funnel, OR
  2. You can change specific elements within the funnel and see how it affects your entire funnel.
A/B Testing Guide example of a multipage test testing different elements across different pages.
An example of a multipage test testing unique elements across different pages.

To get a clear understanding of the difference between A/B testing, multivariate testing, and multiple page testing, check out this article: 

Now that you understand the various types of tests, time to share the merits of A/B testing. 

Why Should You A/B Test?

Building the case for A/B testing is a simple equation of logic: if you don’t test, you don’t know. And guessing in business quickly turns into a losing game. 

Your business may see hopeful spikes of success, but they will quickly turn into “Spikes of Nope” if you don’t have a formal way of experimenting, tracking, and replicating what works and what doesn’t. 

Even if you guess correctly 50% of the time, which is rare, without data and an understanding of why it worked, your growth will stagnate.

A/B testing lets you put on your data-driven glasses to make decisions steeped in growth and optimization. There is no other system that will give you the data you need to make optimization-based decisions.

It’s not surprising the nickname for A/B testing is:

Always Be Testing.” 

-A/B Testing

Why A/B test? Here are a few benefits of implementing A/B testing in your business. 

17 Benefits of A/B Testing

  1. Optimize ROI from existing traffic and lower acquisition cost.
  2. Solve the prospect’s pain points.
  3. Create/design a conversion-focused website and marketing assets. 
  4. Make proven and tested modifications. 
  5. Reduce Bounce Rate.
  6. Increase the consumption of content.
  7. Test new ideas with minimum investment.
  8. Ask data-driven questions.
  9. Perpetuate a growth mindset within your organization.
  10. Take the guesswork out of website optimization.
  11. Enhance User Experience on your site.
  12. Learn about your prospects, clients, and your business.
  13. Test theories and opinions.
  14. Improve marketing campaigns.
  15. Decrease ad spend with tested ad elements.
  16. Validate new features and design elements.
  17. Optimize across user touchpoints.

This is not an exhaustive list but you can see the many benefits of A/B testing. 

The benefits of A/B testing with tool like Convert Experience
With an A/B testing software like Convert, you can reap the benefits of A/B testing.

So, now that you know the benefits of A/B testing, let’s look at how to perform an A/B test. 

How Do You Perform an A/B Test? The A/B Testing Process 

Just like scientists and doctors follow specific protocols, the same is true for A/B testing. This is a continuous process of going through each step to refine your marketing by extracting data from your test results. 

We will go over the general steps that lead to successful A/B testing, using Convert Experiences software as an example where it applies. 

Essential Steps to Perform a Successful A/B Test

A/B Testing Protocol, the essential steps for experimentation
Essential Steps to Perform a Successful A/B Test

Step 1: Research and Collect Data

You want to be strategic about what you test. The best way to do this is to investigate your current analytics. How is your site performing? What are your current conversion goals on each page? Are you meeting them? Can they be improved?

Use quantitative and qualitative tools to find and analyze this data. 

Questions to Ask Yourself Before You A/B Test

Here are a few additional stimulating questions you can ask during the research process:  

  • What are your top-performing pages? Can you make them perform even better?
  • Where are people leaving your funnel?
    You can think of this as a hole in your bucket, you want to seal it as soon as possible. 
  • What pages are performing poorly, high visits but high bounce rates?
    This may show the page is not meeting searchers’ intent. Can you make changes to get people to stay on the page longer?
  • Leverage the 80/20 rule. What minor changes can you make (the 20%) that will cause higher (the 80%) conversions? 

You can find these opportunities through research. Once you are armed with information, time to set a goal. 

Step 2: Set a Goal

Define a clear goal. 

How will you measure the results of your test? 

There are many goals that can be set, and they will be unique to your organization. Your goal may be to increase the number of clicks on your CTA button, while another business may want to increase the number of sales, or the number of email sign-ups.  

Make sure the goal is specific and clear. Any member of your team should be able to read your goal and understand it. 

Now that you have a clear goal, time to create a hypothesis.

Step 3: Formulate Hypothesis

Now that you have a clear goal it is time to generate a hypothesis. Your hypothesis expresses your goal and why you believe it will have a positive impact. It gives your test direction. 

You can use a free hypothesis-generating tool like Convert’s Hypothesis Generator to frame a credible hypothesis for big learnings and lifts.  

If you don’t want to use a hypothesis generator you can also use a Hypothesis Generation Toolkit that promises to help you identify what to test with the promise of giving you “data scientist wings.” 

You may come up with multiple hypotheses. In this case, you can prioritize them. This is super simple using Convert’s prioritization tool (PIE and ICE) within the software. It will help you organize how confident you are about it winning, the impact it will have, and how easy it will be to implement. 

Prioritizing hypothesis in Convert's A/B testing software using of the PIE and ICE Prioritization models
Hypothesis Prioritization inside Convert, using PIE and ICE scores

The models are as follows:

Description of the PIE and ICE Prioritization models
Description of the PIE and ICE Prioritization models

Based on your responses and observations, choose what will make the most sense for your next action. Less effort, bigger result. 

Step 4: Estimate your MDE, Sample Size, Statistical Power, and How Long You Will Run Your Test

Identify the sample size, which is the correct amount of visitors that need to be bucketed to experiment for accurate results. 

Even if your A/B testing software calculates this for you, have a general understanding of these statistics. It will help you spot issues in your testing for abnormalities. 

For example, what if you know your test should run for 28 days, but you completed after only 14 days? Or maybe you estimated you need around 20,000 visitors to reach statistical significance, but you see your test is still running with 120,0000 visitors. In both instances, you want to take a close look at your test.  

Step 5: Determine Which Audience To Target With Test

Some tools, like Convert Experiences, allow you to segment the audience that will see the test. 

Step 6: Create Variations

Creating your variation should be easy since your research revealed what to test, why you are testing it, and how you want to test your hypothesis. 

It is even easier using a testing platform like Convert, you simply make the change you prioritized in Step 4 in the editor, or via code. This may be changing a headline, copy in a CTA button, hiding an element, the possibilities are endless. 

Step 7: Run Test

Now you are ready to run your test!

A/B testing guide, Once you get through Steps 1 through Steps 6, you are ready to run your test.
Once you get through Steps 1 through Steps 6, you are ready to run your test.

This is the point where your visitors are selected by random to experience your control or the variation. How they respond is tracked, calculated, compared. 

Step 8: Result Analysis and Deployment

Along with the planning phase, this is probably the most important step in the testing process. It is like getting a list of stocks that have the greatest yield in the market, but then never buying. It is not the information that is powerful, but what you do with it. With A/B testing, it’s what you do after the completion of the test that will translate into higher conversions. 

Analyzing your A/B test results will give you the information you need to take your next step. Ultimately your results should lead to a specific action. Remember, research and data are only as good as their applications.

Another thing to remember, there are no bad test results. They are always neutral, giving you the information you need to better understand and connect with your customers. 

Luckily, top testing software like Convert makes it easy to implement the winner of your test and lets you know if there is a statistically significant difference between your control and variance.  

Let these results inform your next test. 

Now that you know the theory of testing, and how to set up an A/B test, let’s look at the various elements that you can test.

What Can You A/B Test?

A/B testing Ultimate Guide, what should I A/B test?

The short answer would be, TEST EVERYTHING. But doing that wouldn’t be strategic or a good use of your resources. 

Here is a list of the most common elements that move the dial in a positive direction for your business if you get them right. 

Copy: Headlines, Subheads, and Body Copy

Take away all the fancy images, colors, backgrounds, animations, videos, etc, and you are left with plain copy


Maybe. But guess what? 

Copy sells. Copy converts! 

Imagine you visit a website that had nothing but buttons, photos and not an ounce of text on it. It would be also impossible to share your message let alone convert a visitor.

Now think about the opposite scenario, a no-frills website with only text, no colors, videos, or images. Could you make a sale? Could you get your visitors to take action? 

Of course! 

While the site would be simple and maybe aesthetically lacking, using the right words could make the difference. 

There are two main types of copy you can test. They are:

  • Headlines and sub-headlines
  • Body

A/B Testing Headlines 

Great copywriters spend hours crafting the perfect headlines. You will not make one conversion if you cannot gain your visitors’ attention. 

Imagine your headline is like the window of a brick-and-mortar storefront business. Your window treatment has the power to attract or repel. In the virtual space, your headline does the same thing.  

Marketers have found changing a headline can single-handedly increase conversions. That’s why seasoned copywriters spend hours coming up with the perfect headline.

For headline success, keep it concise, and be sure to tell your visitor what’s in it for them if they click and read more. 

Use A/B testing to test the actual words, tone, font, size, etc.

A/B Testing Body Copy

Once you entice your visitor to click with your headline, you will need to hold their attention and give them what you promised. This is the job of your body copy. 

Every line of copy should encourage the visitor to continue reading it or take the Call-to-action you assigned. 

Even if your body copy is selling a product, offer value, even if the person doesn’t buy.

You can A/B test copy format, style, emotional tone, ease of reading, etc.   

A/B Testing Content Depth

In addition to actual words that make up the content, length of content also plays into conversions, and therefore, can and should be tested. 

As a rule of thumb, write as much copy as you need to move your prospect through the buying process, but not a word more. 

Approach length not as a standard word count, but as, “what points do I have to make to satisfy the prospect and move them to action? Can I say it more concisely without compromising my message?”

When you A/B test content depth you are looking at, “What happens if I add or delete copy (change length)? Does it increase or decrease conversions?  

A/B Testing Design and Layout

While the copy is important, the form does matter. 

Have you ever walked into someone’s house or a space that had a weird layout? Didn’t it make you feel strange? In some cases, you probably wanted to turn right around and leave. 
Well, the same thing happens with your website visitors. 

It is this element of layout that contributes to how long people stay on your site (aka Bounce Rate).

So A/B test different designs. This may include colors, element placement, overall style and themes, ease of flow (does the page move logically), and types of engagement elements. This may also include removing elements from the page. 

Design and layout are crucial for homepages, landing pages, product pages. Pages where you want prospects to take massive actions. This may include filling out forms, which can also be A/B tested.

A/B Testing Forms

You want your visitors to give you information. The way you do this online is via forms. 

It may not seem intuitive, but changes in the way you collect information from your prospects can change the way they respond. Changing one field can increase or decrease the percentage of sign-ups by 5%-25%.

Filling out a form may be the main CTA you have on a page. If so, finding the perfect balance between getting all the information you need, without causing too much friction in your prospects can feel like art. 

A/B testing different form copy, styles, location, with pop-ups, without pop-up, colors, etc. can give you valuable information about what reduces resistance so your customers fill out and submit the forms on your pages. 

A/B Testing CTA

Determining what Call-to-Action your prospects and visitors respond to the most can single-handedly increase your conversion rates. This is why you have done so much work building a website and testing, right? You want your visitors to act. 

People commonly A/B test colors of CTA buttons, that’s a start, but don’t stop there. You can also test the copy inside the button, placement, and size. 

Be relentless with your testing of CTAs until they are 100% dialed-in. Then, test those.  

A/B Testing Navigation

Navigation is another element associated with the flow of your webpage. Because they are commonplace requirements, their importance can sometimes be overlooked. However, your navigation is tied to the UX of your site. 

Make sure your navigation is clear, structured, easy to follow, and intuitive. If I am looking for product pricing I don’t expect it to be under the testimonials tab. 

While experts website designers suggest placing navigation bars in standard locations, like horizontal across the top of your page, you can A/B test different placements. You can also test different navigation copy. For example, changing ‘Testimonials’ to ‘What People are Saying.’

A/B Testing Social Proof 

Prospects are naturally skeptical of the accolades you give your business. They are more likely to trust their peers, people who they relate to and identify with. This is why social proof in the form of reviews and customer testimonials are important for conversion rate.

A/B testing your social proof can help you determine where it is best placed on your page, which objection it should address, what layout contributes to better conversion (with a picture or without, title, location, first name, or first and last name). You can test many variations. 

A/B Testing Images and Videos

Images, videos, and audio add engagement elements to a webpage. They each can be A/B tested for optimization. 

A/B Testing Images

Images can convey messages in seconds and reiterate ideas simply. This is why they are so powerful. 

Imagine you visit an ecommerce store. Think about what your experience would be without product images?

Testing images is as simple as trying various images. Don’t stop there, be creative. Test themes of images, like color palettes, with people, without people, copy of captions, etc. 

A/B Testing Videos

Video adds value. But only when it’s watched. Just like your headline, you want to make sure the thumbnail is attention-catching. 

Besides A/B testing the thumbnail of a video, you can test various videos. Which one gets played most often? Watched the longest? 

For ecommerce websites, product descriptions have the tough job of replacing “touch-it, feel-it, taste-it, try-it-on” merchandise and the help of a salesperson. 

Optimizing this copy could increase your sales and conversion by percentages (the sky is the limit). 

A/B Testing Landing Pages

You create landing pages for one purpose – to convert visitors. So it makes perfect sense to A/B test them to discover opportunities for greater conversions.

Now you know how to set up your A/B test, the elements you can test, and the benefits of testing to your optimization. But what if your company doesn’t have a formal experimentation program?

Starting Your A/B Testing Program

Best case scenario you already have a robust experimentation program, with full C-suite buy-in and the support of peers across departments. 

But if you don’t, don’t let that stop you from testing. 

However, you are going to want to formalize your testing program. You will need:

  • Company and peer support
  • A/B Testing Software 
  • To understand the statistics involved in A/B testing. 
  • Identify areas to test 
  • Start with pages that already convert well and have a good amount of visitors. Then test all the elements mentioned in the last section. 

How Convert’s A/B Testing Software Can Help You Refine Your Testing Program and Boost Conversions

A/B testing software, Convert's A/B Testing Platform has the right features to Launch Winning A/B Tests
Launch Winning A/B Test with Convert.

The best way to experience our tool is by playing around with it. Check it out for free so you can see the full functionality of a powerful A/B testing tool used by [Names of Businesses we can share who used our tool].

Convert is good for the non-coder and with a robust code editor for coders and developers alike. 

The best way to check out a tool is by exploring it. Get full access to Convert for 15-days for free. A/B test, check out all our integrations and see why so many optimizers chose Convert as an Optimizely alternative.

Access our A/B testing tool for over two weeks to do all the testing you would like. 

The sections that follow will provide more in-depth information on A/B testing to move you from beginner to expert. 

Let’s highlight some challenges experienced when doing A/B test.

What Are the Challenges of A/B Testing? 

Challenges are inherent in all kinds of testing, A/B testing is not excluded from this. Challenges can always frustrate, but knowing what to expect can ease that reaction.  Especially when you understand how powerful A/B testing is in your “Get More Conversions” arsenal. 

Here are a few challenges you may encounter with A/B testing and how to combat them.

A/B Testing Challenge #1: What to Test?

Just like when you have a lot of things to do on your To-Do list – Prioritize, Prioritize, Prioritize! 

Prioritization, data, and analytics drive what to test.  

For specific tips, review the section “What to Test.”

A/B Testing Challenge #2: Generating Hypotheses

It may seem simple, but coming up with a data-driven hypothesis can be a challenge. It requires research and a solid interpretation of data. It’s important not to make an “educated guess” based on your knowledge. Make sure your proposal is data generated. 

Check out the section on generating a hypothesis to get specific guidelines on hypothesis formulation. 

A/B Testing Challenge #3: Calculating Sample Size

Statistical calculations may not be your cup of tea. However, understanding how to calculate sample size, and understanding why it is important to the success of every A/B test is critical. 

Take the shortcut approach and let your testing tool do the heavy lifting for you. But understanding how sample size influences your testing will give you an advantage. 

Convert's A/B Testing Statistical Calculator
Convert’s A/B Testing Statistical Calculator

A/B Testing Challenge #4: Analyzing Your Test Results

This is the fun part of A/B testing, because this is your opportunity to get insights into what worked and what didn’t, and why. Why did the form with more fields get a significant lift? Why does the ugly picture above the CTA continuously beat out the variation?  

Whether your A/B test meets your hypothesis or not, reflect on all the data. Pass or fail, there are jewels of information to gleam in every A/B test developed within a strict A/B testing protocol. 

Poor interpretation of results can lead to bad decisions and negatively affect other marketing campaigns you integrate this data into. So take the step of post-analysis and deployment seriously and don’t get caught in this trap.

A/B Testing Challenge #5: Maintaining a Testing Culture

What good is your knowledge of A/B testing or all your fancy A/B tools if your culture doesn’t support experimentation? It will be a challenge getting co-workers and members of the C-suite to test

It also will prevent you from iterating, which is the foundation of A/B testing. Keep testing. 

A/B Testing Challenge #6: Annoying Flickering Effect

Flicker occurs when your visitor sees your control page for a few seconds before it changes to your treatment or variation page. It may not seem like much, but the brain processes this data (seeing the original page) and it changes how the visitor interacts with your variation, ultimately affecting the results of your A/B test.  

You might be losing customers & revenue because of your A/B Testing tool!

As little as one second of blinking or Flicker of Original Content (FOOC) damages your reputation. It frustrates traffic and negatively impacts conversions by 10% or more!

If your variations aren’t performing well because of the persistent blink, this whitepaper will show you new ways to increase site speed, optimize images, and streamline code to beat FOOC.

-Claudiu Rogoveanu, Co-Founder & Chief Technical Officer at Convert.com

A/B Testing Challenge #7: Cumbersome Visual Editors

Some testing platforms have clunky visual editors that make it difficult to change your control. 

Convert’s visual editor is smooth, seamless to use.

Convert’s Visual Editor

You can play around with this visual editor for 15-days. Free. No credit card required. Full access. 

A/B Testing Challenge #8: Difficulty Creating Testing Goals

Difficulties to create goals and select elements to track clicks. No clear flow when setting up experiments. This goes back to the importance of testing. Following the above step-by-step guide should help minimize this occurrence. 

A/B Testing Challenge #9: Keeping Changes

What happens when the winner of your test is variation B? It means you have to implement the change on your website. 

With many tools, there is no straightforward way to keep changes on the platform until they implement the changes on their server. 

Since all teams will be on board before any testing begins. Alert the team responsible for making changes that depending on the winner of the test, a request for them to change an element may come down the pike soon.   

A/B Testing Challenge #10: Separating Code

No simple way to separate code when there are multipage experiments.  

A/B Testing Challenge #11: Excluding Internal Traffic From Results

Marketers know this all too well. Google Analytics data is flooded with metrics that represent internal traffic. This also happens with your A/B testing. It is difficult to exclude internal traffic from experiment reports.

A/B Testing Challenge #12: Advance Testing Requires Technical (Dev) Support 

If you have lots of traffic, an enormous site, require segmentation, or special code, you will need to get your Dev team involved. More of an issue if you don’t have a development team or your development team doesn’t have the capacity to support testing efforts. 

If the latter is the case, it may be time to restructure your experimentation program. 

A/B Testing Challenge #13: Third-Party Integration

You never use just one tool. Many times you may want to see data from your test (or import data into your test) from third-party applications. 

Not all testing platforms allow you to do this seamlessly

Convert integrations with over 100 tools. 

Convert Integrates with over 100 tools
Convert’s Integrations

A/B Testing Challenge #14: QA the Experiment 

As you can see, there are many areas that can create issues in your A/B test if you are not careful. That’s why quality control of your experience is important. Unfortunately, many people overlook this step and rely heavily on it. Especially when you rely heavily on fancy tools. 

Yes, let the tool do the work, but doing a QA look before pressing the start button is never a bad idea. 

A/B Testing Challenge #15: A/B Testing on Mobile

According to DataReportal, there are 5.15 billion unique mobile phone users. The ability to test on different devices is in demand. And the desire for people to a/b test on mobile is also increasing. Tool accessibility on different devices is critical.

Challenges trying to A/B test with mobile

When picking a tool, be mindful of this limitation of accessibility.

As you can see, you can hit many snafus. 

There is a close relationship between A/B Testing challenges and A/B testing mistakes. Time to go over potential common pitfalls made in A/B testing and how to avoid them. 

What Are the Mistakes to Avoid While A/B Testing? 

With so many steps and concepts that have to work together, the possibility of mistakes is plentiful. 

Here are a few common mistakes made in A/B testing and how to avoid them.

Mistake #1: Not Planning Within Your CRO Strategy

Worse than this is not planning at all.

Planning is a step in the A/B testing process that you can’t ignore or skip. Planning includes gathering data, reviewing your overall conversion strategies and desired goals, and creating a solid hypothesis

We know testing can be fun, but you’re not in it for enjoyment, you are A/B testing to drive business growth and find opportunities in your marketing to convert better.  

Also, sometimes marketers are tempted to follow the claims and paths of industries gurus. Sticking to your plan will discourage you or your team from doing this. A change in CTA may have gotten them a 24% lift, but your business, website, goals, and conversion focus are different. Having a plan will keep this reminder at the forefront. 

Mistake #2: Testing Too Much at Once

The power of A/B testing lies in its head-to-head competitiveness of one element at one time. If you run too many tests at the same time how do you know which one is responsible for positive or negative changes?

If you have multiple elements you would like to test, which you should, use a prioritization tool to focus your efforts. Also, be sure you are using a tool that can handle multiple tests. With Convert, you can run multiple A/B tests on different pages at the same time.  

Mistake #3: Not Reaching Statistical Significance

Allow every test you run to reach statistical significance. Even if you think you can identify a clear winner prior to reaching statistical significance, stopping your test early will invalidate the results. This is especially important when using the Frequentist statistical model in your A/B test. 

It’s like taking a cake out of the oven too early. It may look done, but when you inspect it you realize you should have let it cook for the full duration of time allotted. 

Mistake #4: Testing for Incorrect Duration

As stated in A/B testing mistake #3, you must run your tests for their full duration. When you don’t, you cannot reach statistical significance.  

When you think about test duration, think about the “Goldilocks Rule.” You want the length of your test to be “just right,” not too long or too short. 

The way to avoid this is to shut off a test only when a winner is declared by your A/B testing software.  

Mistake #5: Using Above or Below Calculated Required Traffic

Using different amounts of traffic, higher or lower, can have a negative effect on your A/B test. Use correct traffic numbers to ensure conclusive results. 

Mistake #6: Failing to Iterate

To reap the full benefits of A/B testing, you MUST build iteration into your plan. Each A/B test sets you up for another test. 

It is this iterative process that allows you to refine your website, identify opportunities, and give your customer the experience they want and deserve, which translates into higher conversions for you.

It is tempting to stop testing after one successful test. But remember the acronym for A/B testing. Always Be Testing. Let this be your model.

Mistake #7: Not considering external factors

Holidays, days of the week, hours of the day, weather, the overall state of society – all these things and more influence how visitors engage with your site.  

When analyzing results, factor in these elements.   

Mistake #8: Using the Wrong or Bad Tools

Prioritizing experimentation and A/B testing means investing in the right tools. Would you try to build a house with just a screwdriver and a hammer? Sure, you can do it, but it would be an inefficient and poor use of resources. 

Think of your business as a house, you are building it from the ground up and need the right tools to do the job. 

Don’t fall into the trap of looking at the upfront investment without looking at the ROI of your A/B testing investment

Taking our house example again, a good drill may cost you several hundred dollars, but calculate what it will save you in time (which is money). 

Not sure which tool will give you the bang for your buck? Convert takes the risk out of the decision by giving you 15 days to try our A/B testing tool. No credit card required and all features are available.  

The best way to check out a tool is by exploring it. Get full access to Convert for 15-days for free. A/B test, check out all our integrations and see why so many optimizers chose Convert as an Optimizely alternative.

Mistake #9: Not Being Creative In Your A/B Testing

A/B testing is a beast of a tool if you use it in its full capacity. Unfortunately, many people use it simply to test color variations of buttons. Yes, this is a valid element to test, but don’t stop there. 

If you need inspiration review the session on, “What should I Test?”

The same thing goes with your A/B testing software. Explore it. What can do that? 

Convert provides a custom onboarding session with our support team to not just get you up to speed using the software immediately but showing you ways to use it within your testing program. 

Knowing the potential A/B testing challenges and pitfalls makes it easier for you to avoid or fix them if they arise.   

Here are a few more tips to help you improve your A/B test results.


I hope this article inspired you with real examples and a guideline to follow to start A/B testing today, or re-energize your A/B testing campaigns. Test often, test consistently, test within a strategy.   


Originally published October 02, 2020 - Updated February 26, 2021

Start 15-Day Free Trial Of Convert Experiences.

Quick form submission
Quick submission.
No credit card needed
No credit card needed.
You can always change your preferences later.
Check Icon
You're Almost Done.
Managing Marketing Team
Managing Tech Team
Hypothesizing Experiments
Coding and QA of Experiments
Convert is committed to protecting your privacy.

You've successfully submitted your details to start your Convert Experiences 15-day free trial.

We are setting up your account for you. You'll receive a welcome message to your registered email with your login access. Please explore the app on desktop for an optimal experience.