Blog /A/B Testing

How to Drive More Profits With Experimentation: Full Funnel Testing Guide for 2022

18th Jan 2022 – Full Funnel Testing in 2022

Over 250 million dollars.

That’s the total revenue one company gained from all the tests that they ran.

But it’s misleading to show you this number without more context about their story.

The company had a very mature and aggressive testing model. Yet still, as Deborah O’Malley revealed in an interview we did with her, three out of four of those tests were not winners. In fact, about 75% of their tests had either negligible lifts or no gains at all.

But the winning tests made up for all that.

With such potential to increase revenue, experimentation is undoubtedly worth investing in.

Don’t give in because testing isn’t making your business more profits yet. You need to expect, as Deborah advised, to weed through the losers to get to the winners.

If your first marketing efforts failed to bring customers, would you stop trying? Or you’d keep trying until you find what works?

This full-funnel testing guide is where you’ll find what works to increase your winning testing odds. And drive more profits.

Read on to learn:

  • Why testing and optimizing an isolated part of the conversion funnel can backfire,
  • What is full-funnel testing,
  • How to optimize an entire funnel, and
  • How to increase profits with full-funnel testing.

Let’s dive right in.

Why CRO Can’t Guarantee Profits

Without a doubt, investing in conversion rate optimization can increase profits.

According to VentureBeat, the average ROI from a CRO tool is 223%.

But there are no guarantees. A company may have the best tools, but without a good strategy, their CRO efforts can be in vain.

Here’s what I mean. Picture a software selling company that wants to improve conversions and trial signups. So they remove some form fields from their landing page and the click-through rate jumps 10%.

But since those leads weren’t well qualified, they didn’t end up purchasing after the free trial.

Their CRO efforts brought no profits.

In other cases, increasing conversions in an isolated part of the marketing funnel can backfire:

Conversion rate is a snapshot in time. It is what happens in one isolated part of the funnel. Without the context of all the funnel stages, it can be irrelevant at best and dangerous at worst.


If you remove most of the form fields from your sign-up, you will (probably) see a significant improvement in your conversion rate. But the conversions will be garbage.


Your MQL numbers will drop. And at the end of the day, you will be onboarding customers who aren’t properly qualified and pose a high churn threat

Trina Moitra, Head of Marketing at Convert

A similar scenario once happened to a client Annika Thompson from Speero worked with. In an interview Trina did with her, Annika mentioned that they were doing really well getting website visitors to purchase:

And they even told us, ‘Like yeah, that specific kind of landing page where those people are going… that’s properly optimized… The conversion rate there couldn’t be any higher.

They were crushing landing page optimization. But at the same time, they were hurting their business:

But then, looking at the number of people who ever bought for the second time, it was pretty much close to zero. So their retention rate was kind of around 13%… and then putting the cost of acquisition into that equation, it turned out that every single new customer that purchased on their website was hurting their business in the long run… You can hurt your business even when you think that you’re growing.

If the cost of acquisition outweighs the customer lifetime value (LTV), optimizing to get more customers becomes more like suicide.

Especially given the fact that over the past few years, according to ProfitWell, customer acquisition costs have risen 50%.

Without retention as a guard rail and optimizing for both retention and revenue, CRO can go astray.

That’s why CRO needs to be guided by a more comprehensive approach, putting the customer at the heart of it.

This comprehensive strategy is called customer experience optimization (CXO).

What Is CXO and Why It’s a More Effective Optimization Approach

As clear from the example above, getting more customers isn’t always positive.

Core questions like how valuable those customers are aren’t usually asked with the usual optimization tactics. That’s because, with CRO, it’s about more conversions. That’s all.

Yet such questions are crucial. And not answering them before optimizing may lead to tragic results. Customer experience optimization is about asking such questions:

  • Will this change bring more customers with low LTV who could hurt the business in the long run?
  • How to also optimize the funnel for retention and get more customers with high lifetime value?

Instead of focusing solely on conversion rate, CXO guides your CRO efforts to fix the problems customers face across the entire funnel with a focus on retention.

It adds other key metrics such as customer LTV and churn rate to the equation.

CXO is about optimizing for the long term. And it looks beyond immediate metrics around the sale:

It’s looking at the whole customer journey and all those touchpoints. So we’re not just looking at… what’s happening around the purchase. But we’re looking at what’s kind of going on before the purchase and what’s going on after it. So it’s kind of from acquisition to advocacy.

Annika Thompson, Speero

3 Ideas You Need to Ditch to Have a Profitable Optimization Program

Before getting into full-funnel testing, get rid of false concepts or ideas that hold you back.

1. Conversion Rate Optimization Is the Silver-Bullet

Picture this scenario. An eCommerce selling headphones give their visitors a 70% discount on their product to sell more. Sure, the sales are likely to jump. But even if conversions double, their revenue will drop.

Why?

Before the offer, say they were getting 100 customers a day, each paying $100 for a pair of headphones. After optimizing, even if they get 200 customers, their daily revenue would be $6,000.

Since each customer now pays $30 for the product, their revenue would drop almost half even if conversions doubled.

Sure, this is an exaggerated scenario.

But the principle holds true: CRO only looks at fixing part of the problem. And it might not always be so apparent that what you’re trying to fix is actually hurting your business.

Not considering retention had a hefty cost on the client Annika worked with. The site’s CRO efforts worked, and they were getting customers like crazy. But since their churn rate was high, the cost of acquiring those customers outweighed their ​lifetime value. And they were hurting the business in the long run.

That’s why CRO isn’t a silver bullet. It is often short-sighted. To be effective, CRO should tie back to a good CXO strategy, designed for the long term.

With CXO, you are thinking by default what does that change mean for the end goals of the business. Rather than a small part of optimization, like how many people move from one page to another.

Annika Thompson

With a focus on retention, CXO helps you see optimizing through a broader lens. And centers the efforts on delighting the customer:

CXO puts the customer kind of in the heart of the whole kind of optimization and experimentation process.

Annika Thompson

2. Retention Is a Customer Success Problem

Back to that website Annika talked about in the interview.

Since they didn’t make retention a guardrail, their CRO efforts backfired. And each new customer ended up hurting their business in the long run.

Well, one might think, they did well with CRO. Isn’t retention the duty of their customer success team? Shouldn’t they work harder to keep those new customers?

It’s like handing a woodcutter a bad rusty saw then blaming them for being slow.

Retaining customers who aren’t well qualified is too hard. In order to maximize the chance of retaining customers, optimizers should aim for well-qualified clients with higher LTV and deep interest in the product or service.

Retention isn’t just a customer success problem. It’s everyone’s problem. Retention loops are responsible for funding the growth of a business.

According to research by Frederick of Bain & Company, increasing customer retention rates by 5% increases profits by 25% to 95%.

That’s why optimizers should go for both retention and revenue. And, by its nature, CXO targets both.

Customer experience optimization is also a silo breaker. It gets the CS team, optimization, and other teams to work together to offer the customer a better, consistent experience.

It can centralize learning from across departments and teams in favor of a better customer experience.

In the example above, with CXO in mind, the company’s CS team would help the optimization team learn why most customers don’t purchase again. And also show them the customer segments with high LTV, why they buy again, and such.

Then, optimizers could use these insights to create strong hypotheses and conduct better tests.

3. Customer Experience Optimization Is Based on Soft Metrics Like NPS

I’ll leave it to Convert’s Head of Marketing, Trina, to remind us what CXO is:

CXO is the most complete measure of how the incentivized actions you prompt users to take through your CRO efforts translate into retention, revenue, and profits. And that’s what you are optimizing for anyway.

Soft metrics like NPS can help you measure your customers’ satisfaction levels.

But the focus of customer experience optimization is on retention. And getting more customers with high LTV.

After looking at data from 2,000 companies, ProfitWell found that NPS is not even a strong indicator of retention.

Hard customer retention metrics such as customer churn and customer LTV are the true indicators of retention.

To run a profitable testing program, focus on those hard retention metrics.

What Is Full Funnel Testing?

With the concepts you need to unlearn out of the way, let’s see what funnel testing actually is.

In full funnel testing, you don’t stop at optimizing the homepage. You look at optimizing the entire funnel from the moment a potential customer lands on your site until they convert into a customer… and even beyond.

In full funnel testing, each successive test tries to find solutions to a specific block/resistance/frustration that stops visitors from converting and becoming high LTV customers.

It’s about optimizing for the entire buyer’s journey and for different buying scenarios. This gives your audience a more relevant, consistent, and better user experience.

The opposite is testing and optimizing in an isolated part of the funnel. We’ve seen earlier why this can give you misleading results:

  • The software company that optimized to get more free trailers didn’t profit from them. Since those sign-ups weren’t from qualified leads, they didn’t end up buying.
  • The business Annika worked with had a high conversion rate on their landing page. But since those new customers’ LTV was low, they were hurting the business in the long run.

Both thought they were growing because they had improved conversions. But we saw that didn’t translate into revenue, retention, and profits.

That’s why the full-funnel testing approach is crucial for a profitable experimentation program.

If conversions jump, but revenue drops—like with the headphones company who offered crazy discounts—it’s a losing test.

Since you won’t only be testing a single part of the funnel, full-funnel testing can give you more coherent metrics across the funnel.

Optimizing one landing page or the homepage alone won’t likely cause a significant increase in sales. To see a big change, you have to reduce the issues potential customers face all along the sales funnel, especially high leakage places like the product or checkout pages.

A 5 Step Process to Increase Profits With Full-Funnel Testing

Without a solid strategy, even full-funnel testing can go astray.

Some 74% of optimizers with a structured approach to conversion also claim improved sales.

Alex Birkett for CXL

The following steps will help you approach full-funnel testing in a structured way. And increase your odds of having more winning, profitable tests.

1. Clarify Your Main Goal

So… Why are you testing in the first place? Does it impact your bottom line?

Not all goals carry the same weight. Some will sustain your business while others won’t.

First things first… You need to identify key metrics that drive the success of your product or company. These KPIs will influence decisions you make for both day-to-day operations as well as future strategy.

To set up prioritization correctly, find out what your core metrics are. Then work backwards from there to understand what tests are likely to yield results for those specific KPIs.

If you have a product that is trying to grow its user base, then the Cost of Customer Acquisition will likely be a core metric. Churn and Lifetime Customer Value might also come into play.

If you’re more interested in improving user experience, depth of customer engagement might become a higher priority since it represents the quality of relationships you have with customers. Retention and referral rates may play a role here as well.

2. Factor in Different Teams & Departments

To drive more profits with testing, involve different teams and departments. That is true when optimizing not only to get more customers but to also give them a better experience.

Optimization isn’t only a job for testers. Developers, data scientists, marketers, and product managers all need to take part in it too. Besides, who else understands where the problem areas are better than them?

The goal is to move away from simply testing for more conversions and even thinking of experimentation as being contained to one department, and instead, moving towards a culture of experimentation where every member of the company shares their unique know how.

How does this work? It’s actually quite simple. The more you bring experimentation into the daily processes of your company, the better results you will get. By filtering down your experiments only through testing teams, you are missing out on crucial points of knowledge needed to optimize your site or product.

3. Understand Your Existing Data

99.5% of collected data never gets used or analyzed.

Data without critical thinkers to draw insights from it are just numbers.

It’s crucial to understand your data to pick what to test and create solid hypotheses.

Without that, you’d be testing random stuff:

It’s really, really important that you don’t just perform random acts of testing… Properly plan and prioritize the test that you think has the biggest opportunity to win… Whatever you do, ensure that you can justify the best test to run first with as much quantitative evidence as you have.

Deborah O’Malley, Guess The Test

After clarifying your big “why” behind testing, ask strategic questions like

  • What are the worst-performing web pages in my funnel?
  • Why don’t new customers buy again?
  • How much does it cost to acquire each new customer?
  • Who are the customers with high LTV to target?

Then, look at your data to find the answers. An analytics tool like Google Analytics shows you the pages in your funnel with the most leakage of conversions. It’s logical to start testing those worst-performing pages first.

Plug Data Gaps

To better leverage data for full-funnel testing, plug data gaps.

In her article for Reforge, Crystal Widjaja stressed the importance of understanding how to leverage data at each stage:

This means understanding how data should be leveraged at each stage to meet the needs of the business today in preparation for what the company will need in the (near) future.

Without that, you may end up underbuilding or overbuilding.

Underbuilding is when the data stage is behind the testing program’s maturity. It is when a business is testing a lot but isn’t prepared to use data for more strategic testing:

  • The optimization team isn’t educated enough or equipped to use data and draw good insights from it.
  • Data infrastructure is underbuilt (the data is old, unorganized, or there is no system to organize and share data).

What this looks like:

  • Months or years of data not being tracked,
  • The team has been testing and optimizing without using data for very long.

It’s also important not to overbuild.

Overbuilding is when the data maturity is ahead of the optimization program’s maturity.

It is when a business is building data infrastructure that fits a very mature testing program while they are still at the early stages of testing. It is when optimizers collect data not aligned with their strategy. Data they won’t or can’t use anytime soon. Such as doing many surveys and asking customers random questions.

It’s a waste of resources. Use your existing data for growth instead of overbuilding.

Plug the data gaps. If your business’ data stage fits a more advanced testing program, focus on helping your testing program catch up. If it’s the other way around, do the opposite.

4. Choose Your Testing Strategy

Full funnel testing is about optimizing the entire funnel for the full buyer’s journey. And not just an isolated part of the funnel.

It looks to enhance the target customers’ experience by reducing the blocks/frictions/frustrations in the funnel that prevent them from turning into high LTV customers.

To do so, having a solid testing strategy is a must:

Strategy = Problem + Guiding Principle + Tactics.

  • The problem is the change you want to see happen (your big testing goal).
  • The guiding principle is what you will always keep in mind as you test and optimize. Also known as the Overall Evaluation Criterion.

The OEC should be able to detect improvements over a short period of time while being a good predictor of long-term goals so we’re able to run experiments using it.

Dave Redfern

“If you can’t measure it, you can’t improve it” is what Peter Drucker used to say.

Only looking at metrics around the sale, like conversion rate, is a critical mistake many make. This is why guiding your CRO efforts with CXO is crucial. CXO reminds you of other incorporating retention rate and customer LTV into your roadmap.

When it comes to choosing the actual tactics you’ll use, first decide the types of tests that work best in your case.

Is it going to be concurrent A/B testing, split testing, or multivariate testing on separate pages across the funnel? Or will you do multipage testing to tackle all changes in one go?

Multipage testing is similar to running a bunch of A/B/N and multivariate tests together on multiple pages across the sales funnel.

Here’s a short definition of multipage or funnel testing from Disha Sharma:

Multipage testing is a form of experimentation where you test changes to particular elements across multiple pages.

There are two ways to go about this.

You either take all the pages of your sales funnel and create new versions of each. This new set of pages make your “challenger” sales funnel and you then run it against the original sales funnel (or the “control”). This is called “Funnel Testing”.

Or you can test how the presence/absence of recurring element(s), like security badges, can impact conversions across an entire funnel. This is classic or conventional multipage testing

The best A/B testing tools can handle all these. Convert Experiences can help you run whichever type of experiments you choose, including A/A tests, A/B tests, split URL tests, MVT tests, multipage experiments, and personalizations.

Once you know the types of tests you want to run, you’ll need to think about prioritization.

Prioritization models help you sort out your testing process and choose which test to run first. CEO of Convert, Dennis van der Heijden, explains the popular models in this post: What Prioritization Model to Pick: PIE, ICE, TIR, or PXL.

5. Have a Critical Thinker Onboard

As I wrote earlier, data without critical thinkers to draw insights from it are just numbers.

The same applies to testing tools. Without a good data-driven team, don’t expect to run a profitable testing program.

Properly planning and prioritizing tests requires critical thinking. So does coming up with decent hypotheses.

And knowing that testing a part of the conversion funnel in isolation can give misleading results needs critical thinking.

Critical thinking is necessary to construct a solid full funnel testing strategy.

According to Conversion Director, Jonny Longden, true analysis is:

Critical thinking employed to solve business problems, supported by the interrogation and interpretation of data.

Train your optimization team to think more critically. Consider formal training in critical thinking.

At least one data-driven critical thinker who has business acumen–and should also be involved in the business–should be onboard. If there is no good critical thinker in the team, hire one.

Test, Learn, Iterate

You’ve gone through all the steps.

Let’s recap:

You’ve determined the main goal behind your full-funnel testing program.

Using an A/B testing significance calculator like Convert’s, you found that your site does have enough traffic for testing. After understanding your existing data, you created a testing strategy.

You came up with some hypotheses and using prioritization models, data, and logic, you chose what test to run first.

Maybe then you decided that implementing a multipage test would prove or disprove a hypothesis straight away.

Now what? Keep testing, iterate?

Before running more tests, learn from each test, even non-winners.

Founder of GuessTheTest, Deborah, said in the interview:

It doesn’t matter as much if the test wins or loses if you’re learning something from it, and if it’s properly executed… a test, if it’s done right, is never a failure, and you should never see it that way.

First, check the test’s validity. It’s not enough for a test to reach 95% statistical significance or higher to show valid results:

Having a control and variant with just 50 total conversions and only a small difference between them isn’t enough to get any insight into your audience —even if the test results were somehow statistically significant.

In his guide for learning from A/B testing, Daniel wrote that most experts agree that around 300+ conversions per variant is usually a good rule of thumb. He also emphasized running the test for long enough to get valid, accurate results:

We recommend you get a statistical significance of 95% or higher, but ALSO run the test over 2-4 sales cycles, with 300+ conversions, and with a large enough sample size.

You can even go one step further and

After learning from a test—whether a winner or losing one—iterate. Running tests is a crucial part of a successful marketing strategy. Spin the testing flywheel.

After the first tests, measure the value they provided by the decisions they helped make. To increase interest in the testing program, convey this value broadly. Think meetings or sending test results newsletters, especially those that brought significant value.

This interest should translate into further investing in the testing infrastructure and improving data quality. With every turn of the flywheel, your business should have a more mature and profitable full funnel testing program.

Convert Experiences can run all optimization experiments your business needs to grow, even in data privacy-sensitive industries.

Try the app for a full 15 day no obligations free trial and start your journey for growth.

Originally published January 18, 2022 - Updated April 06, 2022

Start 15-Day Free Trial With Us.

Quick form submission
Quick submission.
No credit card needed
No credit card needed.
You can always change your preferences later.
Check Icon
You're Almost Done.
Managing Marketing Team
Managing Tech Team
Hypothesizing Experiments
Coding and QA of Experiments
Convert is committed to protecting your privacy.

You’ve Unlocked Your Free Trial.

  • Check your inbox for the password to Convert’s trial account.
  • Log in using the link provided in that email.

This sign up flow is built for maximum security. You’re worth it!

PS: Gifts await you once you join us.