How to A/B Test on Low-Traffic Sites

Disha Sharma
By
April 25, 2019 ·

Trigger Warning:

We know there is a ton of debate raging around statistical significance, the 95% mythical (or not?) mark, testing and traffic, and more.

If you are an optimizer who feels strongly about this stuff, you might get triggered. Well, the comments are open and we can have a really enlightening conversation.

Let’s begin!

Testing doesn’t work for low-traffic websites.

That’s for sure.

Because to derive statistically significant insights, you need to show your experiment’s variants to (at least) thousands of users.

In fact, at Convert Experiences, we often recommend our users run A/B tests — which are the simplest CRO experiments — only if they can send about 10K visitors to each variant.

Also, we suggest recording around a 1000 conversions to yield conclusive evidence of the superiority of the control or variant.

This kind of explains why CRO isn’t a million customer kind of market as there are just a few 100k websites (or businesses) with the footfall to test with any success and the budgets for the pricey (rapidly going “enterprise”) optimization solutions!

Should Low-Traffic Sites Rule Out Testing?

Given that an average small business only gets a few hundred visitors a day to its website, A/B test experiments don’t make sense for them.

But:

That doesn’t mean a small business can’t run experiments at all.

In fact, there are many experiments that even a small business can run.

And there are many benefits to running them.

For example, if you own a small business, running experiments will help you move away from gut-based toward data-driven decision-making. It will force you to look at data … and with in-your-face numbers, you’ll stop defaulting to *I think* strategies. This data-backed approach alone will transform how you build and grow your business.

Running experiments will also teach you to learn by trying, rather than accepting the status quo. It will inspire you to see if the principles-believed-to-work actually work for you.

Finally, optimizing will prepare you for some “real” CRO when you have “enough” traffic as it helps develop the right mindset for testing — a prerequisite for effective CRO.

So start running whatever experiments you can over whatever marketing channels you use. Here are some of your options.

Testing with Email

Surprisingly, email marketing is one channel that lets you experiment with different ideas, without needing tons of subscribers (or “contacts”).

“Says who?”

Says every email marketing software provider right from MailChimp, ConstantContact, and Campaign Monitor to HubSpot (that supports email marketing in its marketing automation software).

Let’s start with HubSpot.

So if you want to run an A/B test using HubSpot’s email platform, all you’ll need are 1,000 contacts: 1,000 contacts is also our benchmark for running A/B tests on samples of email sends. — HubSpot

Campaign Monitor also says that you can run A/B tests even with small emails lists. Here’s the math behind it:

Now, you may be thinking, “My list size is only 500 subscribers, how do you expect me to run a successful test?” My answer is that you’ll need to set your sights on a larger MDE [Minimum Detectable Effect (MDE)]. When you increase the MDE, your required sample size decreases. So instead of needing 592 subscribers per variation to detect a 20% relative effect, you would only need 94 subscribers per variation to detect a 50% relative effect. Note that the baseline conversion rate also plays a part in your sample size – the lower your conversion rate (in this case, open %), the more subscribers you’ll need to run an accurate test. — Campaign Monitor

So if you’ve about 500 or 1000 subscribers, you’re good to test, provided you have a responsive database that is interested in what you have to say.

However, a few email marketing providers recommend a slightly higher contact base.

For instance, email marketing solution provider ConstantContact recommends a higher sample size at twice the minimum size that HubSpot allows. It suggests that for meaningful A/B testing, the sample size must be at least 1,000 contacts. So, if you’ve 2000 subscribers, you should be good to test (with your A and B versions each getting 50% of the total contacts as test traffic).

For subject line testing, we recommend testing each line on a sample size of at least 1,000 contacts if possible. 1,000 users for each version is typically a good benchmark for any A/B test, but if you don’t have enough contacts or visitors to hit those numbers, it’s still worth running the test. — ConstantContact

MailChimp also recommends somewhat larger lists to run A/B tests. It suggests that you must have at least 5,000 contacts to run tests. But because so many small business contact bases are much smaller than that, it recommends testing ideas across the full list:

We recommend you send combinations to at least 5,000 subscribed contacts to get the most useful data from your test. If the size of your list or segment doesn’t allow for 5,000 subscribers in each combination, consider testing your entire list and use the campaign results to inform future campaign content decisions.MailChimp

While it’s true that having a huge sample size will help you reach statistical significance much sooner (and with much more confidence), but you can start testing with your small list as well!

Here are 150+ email split testing ideas for your email campaigns.

Also, here’s a quick guide on email A/B testing.

Recommended Resource:
An AI junkie? What’s in store for email marketing campaigns in 2019? Here are “Pros and Cons of Implementing Artificial Intelligence in Your Email Marketing Campaign in 2019

Testing with Social Media

Social media is another channel that lets you test your ideas, no matter how small your social following might be. Besides, in a way, on social media, nothing actually limits your reach.

So for example, at Convert Experiences, even we don’t have tens of thousands of followers, yet our awesome Social Seller keeps experimenting with elements of our social media copy such as its hashtags, images, and more.

And her tests bring some great wins for us. For instance, her experiment using a certain set of hashtags once won us a retweet from Google Analytics!

experiment using a certain set of hashtags once won us a retweet from Google Analytics!

Now, that’s some social media exposure, isn’t it?

You might say that it’s easy to dismiss quick wins as (maybe) flukes BUT when you see a set of hashtags routinely earn more exposure than the others you test it against, you’ve more than a test result … what you have is a trend.

You can run a host of social media A/B tests to uncover such trends and boost your social media performance. We’ve published a great primer on social media A/B testing here. Read it to learn how you can start with social media A/B testing, the different social media A/B tests you can run, and the top social media A/B testing best practices that you must keep in mind. 

A/B Testing with Paid Ads (PPC)

Just like email and social media marketing channels, the PPC marketing channel also lets you experiment with your ad copies and all the other elements of your ad campaigns.

Whether it’s Google Adwords or Facebook ads, most paid networks support experiments so you can get the highest ROI.

But to get statistically significant results from your PPC experiments, you need a few thousand impressions of each ad copy. Google recommends testing ad groups that get at least 5000 impressions/week.

So how much do a few thousand ad impressions cost?

It depends a lot on the platform and the industry (among other factors), but there’s data to help you do the budgeting.

For example, AdStage analyzed over 110 million ad impressions (on Google Adwords) in Q1 2018 and found that on an average, you need $116.91 to get a thousand impressions (CPM) on Google Ads. Last year, it was about 13% cheaper.

Social media networks like Facebook also offer accessible CPM options.

Data from Statista research states that the average CPM (cost-per-mille or cost per 1,000 impressions) for Facebook is USD 1.26. Other Facebook CPM average cost estimates from AdExpresso and  WordStream fall in the $7-10 range.

The thing is that you can start testing ad copies even on a limited budget.

An (Almost) “Traffic-Proof” Way of Experimenting: Personalization

Personalization or personalizing a visitor’s website experience based on what you know about them is another common form of experimentation.

The best part about personalization is that even low-traffic websites can work with them as the traffic constraints that apply to CRO tests don’t apply to personalization.

The reason behind this is that with personalization, the end goal isn’t to optimize conversion rates. Instead, its goal is to offer a more relevant user experience that leads to more conversions or higher customer satisfaction and retention.

For example, you could create different versions of your product page to show to your customers and non-customers (or leads).

On your product page for your customers, you could show content such as product guides and videos, quick links to access support and more, so that they get a more rewarding product experience. Whereas the version you show your leads — for example, those who signed up for a trial — could focus on your product’s features or case studies to help them convert.

About 93% of companies see better conversions with personalization.

And you can start with personalization right away.

Plus, with a tool like Convert Experiences, you can set up personalized website experiences within minutes. Just set the targeting rules and you’ll be able to show personalized messaging to your different audience segments — prompting each segment to move to the next stage of their buying journey.

Wrapping it up …

While low website traffic can restrict you from running substantive CRO experiments, it can’t stop you from “learning.”

And that’s what’s important — because once you shift your focus to learning by trying rather than accepting the norm, you’ll start developing a culture of experimentation, a prerequisite for effective CRO.

It doesn’t matter what growth stage you’re at, you can invest in developing a culture of testing. It will come in handy when you actually have enough traffic to your website to go for hardcore, definitive CRO experiments.

Mobile reading? Scan this QR code and take this blog with you, wherever you go.
Originally published April 25, 2019 - Updated June 20, 2024
Written By
Disha Sharma
Disha Sharma
Disha Sharma
Content crafter at Convert. Passionate about CRO and marketing.

Start Your 15-Day Free Trial Right Now.
No Credit Card Required

You can always change your preferences later.
You're Almost Done.
I manage a marketing team
I manage a tech team
I research and/or hypothesize experiments
I code & QA experiments
Convert is committed to protecting your privacy.

Important. Please Read.

  • Check your inbox for the password to Convert’s trial account.
  • Log in using the link provided in that email.

This sign up flow is built for maximum security. You’re worth it!