How to Work With an A/B Testing Agency (And Hit All Your Growth Goals)

Hard truth: Outsourcing your A/B testing does not guarantee growth.

We wish it were that simple. Even with world-class experts, the wrong expectations mixed with the wrong directions often lead to fruitless collaborations.

On the flip side, there’s a smarter and much more productive way to collaborate with your A/B testing agency. It’s the secret sauce behind the success stories that attracted you to working with CRO experts.

In this article, we’ll show you that secret sauce — a clear step-by-step blueprint to collaborate and partner with the best CRO or A/B testing agencies. By the end, you’ll know how to work with CRO agencies to hit all your growth goals.

Let’s dig in…

Four Great Reasons to Onboard an A/B Testing Agency

When you dive into these four reasons, you’ll find star ratings for collaboration. We’re calling it the collaboration index.

For this, we mapped out the level of collaboration that is essential for unlocking success in onboarding a CRO agency, and they are tied to each of these four reasons (or benefits).

To secure these benefits, you’re going to have to commit a measure of collaboration into your partnership. That’s what these stars represent.

For instance, if you’re just looking for assistance with A/B test build, the collaboration doesn’t necessarily have to be exhaustive, but it still has to be there. That’s why it gets two stars.

Now that’s settled, so let’s proceed.

1. Assistance with A/B Test Build

COLLABORATION: ⭐⭐

If your optimization program has matured up to a certain point, you’re definitely not a lone wolf running the entire operation from start to finish. You need engineers and developers.

But the problem here is most optimization departments don’t have dedicated engineers and developers.

When you hit a hypothesis you feel confident of testing, it hurts your efficiency if you have to beg and borrow resources from the Dev team to implement them.

Moreover, though the underlying aspect of “coding” remains the same, developing a product and building out a test are two very different contexts requiring a mindset shift.

So, don’t rely on in-house agents who prioritize something other than successfully setting up tests to scale your program. There’s a better strategy that also injects agility into your experimentation program…

Outsourcing.

In this setting, this isn’t only desirable but also logical. Not only do agencies have a dedicated Dev team, but they also specialize in build—including Convert partners BrillMark & EchoLogyx effectively 10X-ing your in-house efforts.

Clients who want to increase their testing program by multiple folds should be seeking out to bring external agencies onboard. Based on my experience, there is a limit as to how much they can increase their testing program with the internal team. For example, if the internal team manages to develop two tests per month based on the resource available to support with the A/B testing development and QA, and they want to increase that to 10 tests per month, they are better off using an agency who can provide that dedicated support.

Tasin Reza, Echologyx

2. Assistance with Telemetry

COLLABORATION: ⭐⭐⭐

We’ve talked a lot about the importance of setting up the right data collection channels and treating qualitative and quantitative data with respect to eliminating bias.

A/B testing agencies understand why different platforms report different numbers for the same event or action. They can take stock of how pernicious this discrepancy is and help the in-house team translate this into the impact on the final test results.

Without this invaluable insight, how can you trust your results when data sources often disagree on critical metrics?

As Chad Sanderson mentions, to believe that differing data inputs will reconcile themselves in the result output is naive. A/B testing agencies bring this much-needed data literacy to the mix.

It’s part of the reason Alex Birkett (quoted below) works with agencies – to add the needed integrity to A/B testing results. As you can imagine, it requires even more collaborative input than just getting help with the test build.

In the vast majority of cases I’ve worked with an A/B testing agency, I’ve hired them to help exclusively with experiment development and quality assurance. This is due to the common fact that most organizations’ limiting resource is engineers. Using an outsourced A/B test development agency has allowed me to increase the throughput of quality experience, and I can trust the fact that the experiments are well designed because they’re used to working on the platform I’m using.

Recently, I’ve considered getting a full service firm to help with A/B test development as well as two other bottlenecks: telemetry and customer research. Setting up the proper data pipelines is tricky and getting your A/B testing firm to do this aligns incentives well (they clearly want to appropriately attribute their value, so they want the right analytics set up). And customer research is always lacking, even if I do a lot of it myself. The more the better, and the specialists sometimes have tools that are hard to access.

Alex Birkett, co-founder of Omniscient Digital

The next reason requires just as much collaboration for success as outsourcing for assistance with telemetry.

3. Assistance with Customer Research

COLLABORATION: ⭐⭐⭐

Customer research is, without a doubt, the most expensive, time-consuming, and tedious aspect of ideating a credible hypothesis. But, as the hierarchy of evidence suggests, it is the foundation on which controlled online experiments (A/B tests) sit.

There are a few things that can go wrong here:

• In-house teams do not have the know-how to structure proper customer research questions.
  
  They may be misleading, biased, and ultimately seeking to validate an existing opinion, rather than probing a problem brought to light by the quantitative data.
  
  An external researcher will bring a more neutral perspective to digging for this goldmine.

• In-house teams do not have the bandwidth to immerse themselves in a mind-boggling array of perspectives that only a true expert can.
  
  What are all the angles to explore? How do you produce better hypotheses with this level of perspective without drawing from a rich and varied experience?

4. Assistance with Evangelizing Experimentation

COLLABORATION: ⭐⭐⭐⭐⭐

Yes, experimentation buy-in should come from the C-suite.

That “remit” Manuel Da Costa talks about is crucial here. Experimentation being embedded in the DNA of a business is

• Part enthusiasm around the promise of “more” winning tests bring.
• Part understanding of how these disparate successes in the early days of a CRO program can build from optimizing for a metric to becoming the decision making support for the company.
• Part “experimentation is non-negotiable” mandate from the leadership that proactively chooses scientific workability over gut instinct.

Here’s how having an agency onboard contributes to each part:

• The tests benefit from their vast experience in process and QA. Convert’s in-house data suggests that agencies drive more significant tests.
• They take care of the technical aspects of deploying tests giving Changemakers the confidence to be vocal about the impact of company-wide experimentation. It does not hurt that they assist with org-wide education.
• They can help experimentation leads in the initial stages of C-suite buy-in because of their track record of proving the power of experimentation across industries and verticals. The eventual lobbying that might be needed benefits from their mature POV.

And here’s a guide on how CRO agencies price their services and offerings, so you know what to expect when working with one.

We’ve covered great reasons for onboarding an A/B testing agency. Now, we’ll show you the wrong reasons to have your A/B testing outsourced.

Three Really Bad Reasons to Outsource

1. You Want To See Your Experiments Win ALL.THE.TIME.

You get the experimentation results you deserve.

This post and its comments are proof:

If you bring an agency on board to look for winners, they’ll look for winners. And there are a hundred subtle and not-so-subtle ways to influence everything from data query to results interpretation in order to have a portfolio of winning tests that never drive learning or bottom-line impact.

The result: You will probably declare that running A/B tests with an agency is not your thing. But you would have gotten exactly what you’re looking for – WINNING TESTS.

Agencies should be partners. They should have the freedom to investigate the issues that are actually holding back revenue and performance. Especially if you are contracting them to not just build, but also research and design the experiments.

2. You Want to Leverage the Agency’s “Intuition” to Cut Corners in Testing Rigor

Experimentation follows the scientific process. So, there’s no accurate way to tap into the scientist’s (read: your A/B testing agency’s) experience to find the quickest route to wins.

When onboarding an agency, you’re presenting them with a unique challenge; not a situation where they can say, “You know, we’ve seen this before. This is what usually works. Let’s do that.”

When potential clients approach us about working with them on CRO/experimentation, they often ask about our past experience in that niche or sector. For example, what experience do you have of working with luxury travel brands/high ticket furniture/financial services/etc?

This very firmly misunderstands what experimentation is.

If you hire a brand or creative agency, they are going to use their intuition and creative powers to design something which you are likely not going to test. You, therefore, need to know what kind of intuition and creativity you are buying and whether they ‘get’ your audience. You are trusting someone to make an untested communication to your customers.

Conversely, the entire point of experimentation – which is ultimately the replication of the scientific method in business – is to remove opinion, bias and intuition by:

a) using research and data to formulate ideas

b) testing those ideas with your real customers and

c) building on learnings such that the bigger things you do have been proven effective before you do them

You are buying a process and method which requires a lot of expertise to execute correctly, not past ‘experience’ of a sector.

We are guided by data, not by anyone’s opinions of your sector or customers.

Jonny Longden (via Test Everything newsletter)

If you are planning to leverage an A/B testing agency’s intuition and not feel completely in the dark when trying to diagnose a problem or provide a more integrated roadmap to CRO success, you need to work towards better client-agency collaboration through clear communication and a flow system.

3. You Have No Intention of Igniting a Passion for Testing In-House And Want the Agency as a Forever Crutch

You don’t view bringing an agency on board as an opportunity to springboard into something greater in-house. You might score a few winning tests – and they may or may not convert into real business benefits – but one thing is for sure: consistent experimentation will never take off.

Experimentation can’t be a tactic. Like PPC, it may run divorced from the other departments and (still) survive.

The real power of running tests lies in reframing how every team member makes key decisions at work. It is to realize that we experiment anyway and file that as “taking a risk”. Having a sound scientific structure underlying these risks and gambles is what drives a company towards foresight, greater clarity into what is working and, most importantly, consistent success.

Making sure you and your CRO agency are a good fit in the short and long-term is key to creating a synergistic relationship that succeeds in meeting metrics and exceeding expectations.

Now, let’s summarize the best and the worst reasons to outsource your testing to an agency. Check out this 3-minute clip from Convert’s course on A/B testing.

A Simple Blueprint to Work with A/B Testing Agencies

1. Get Clear

You want to know why you are bringing the agency onboard. Choose KPIs — both for the program & the business — that you’ll strive to improve.

There should always be macro-level business targets and micro-level targets to achieve those higher ones. The testing teams need to focus on those overarching goals via research and innovation which in turn can be tested out.

The key here is to communicate this with the business stakeholders – how the experimentation will help the business to take the right decisions – towards achieving the expected KPIs. The roadmap for the experimentation should be created based on this; however, at the same time, the roadmap should be flexible enough to support the urgent business requirements. And finally, there needs to be a dedicated person to drive this with the support of a dedicated team.

Tasin Reza

Be careful here. KPIs related only to the program will result in more tests (higher test velocity), maybe fewer tests that are deployed with bugs, and a shorter time to deploy. These are all great signs of a healthy program.

But unless there is clarity in what the experimentation machine is supposed to chip away — AKA the problem it should solve — you’ll thrive at optimization, but you won’t move your company forward.

Without clarity, there’ll be a gap between product/site strategy (driving the KPIs that impact business) and experimentation strategy (driving the KPIs that promote a vibrant CRO program).

To bridge this gap, maintain a focus on learning and improving user experience. As you keep an eye on revenue metrics, ensure you don’t develop tunnel vision and lose sight of what experimentation is really about.

For a program that’s dependent on budget, you’re going to have to report in terms of ROI as well.

Tim Mehta, Experimentation Program Manager at Microsoft Store, shared in this interview with Speero that it helps if leadership is innovation-focused because then you can show ROI in terms of the insights you’re pulling from experiments while speaking the language of potential revenue and potential revenue loss avoided metrics.

And then, don’t just share “wins” and “losses”. Those terms don’t mean the same thing outside the CRO sphere. It will be better interpreted and translated to valuable business decisions if presented as recommendations instead.

2. Identify Stakeholders

It’s easier to introduce an agency if there is complete clarity about how the experimentation program will run. You don’t have to wait to get started and correct errors as they occur. That’s ineffective and costly.

Instead, streamline the process from the onset with the RACI assignment matrix. This chart helps you map out the key objectives, milestones, tasks, and responsibilities in your project for each of the stakeholders involved.

It’s also how you figure out who your key stakeholders are (i.e., those that need to stay informed).

RACI is an acronym that spells out how you structure the project:

• Responsible – Who is responsible for completing the tasks or making decisions? This is usually a group of people.

• Accountable – Who will sign off on tasks and make sure milestones are completed? Who will be assigning tasks to those responsible for completing them? This is usually the project leader and is a singular role so that they can give the final decision.

• Consulted – Who needs to be consulted on the project’s tasks? Who needs to give input on a task to help it be completed?

• Informed – Who needs to be informed on the project’s status? Who needs updates on progress but doesn’t need day-to-day input?

We’ve talked about how to build and use a RACI chart for your experimentation project. Here’s an example of how it’s used to organize communication:

Focus on stakeholders who must be informed of what the agency is doing and the ones whose work will be directly affected. Because onboarding an agency doesn’t have to disrupt regular business operations.

When you hire an experienced CRO agency, make sure to use the RACI matrix to be able to spot those with the expertise to deliver results and bring varied solutions to any future problems.

3. Onboard Well

Think of onboarding an agency as you would onboarding a new team member. You know you’ll have to prepare necessary resources for them to get started with ease.

There are a lot of things they won’t know if you haven’t worked together before. For example: the internal workings and various use cases of your site, your prioritization matrix, etc. So, how can you pass on that knowledge?

It’s imperative that they (agencies) know how to work within the parameters of our program, including our prioritization matrix, our sprint planning, the QA checklist we use for launch, and our design team. Luckily, most of this is automated to a large extent and laid out on an Airtable that anyone can access.

Alex Birkett

I have noticed that clients forget what outside people know about their internal website or apps which need to be optimised. The internal team is very close with the website itself. They know the ins and outs of all the different scenarios, use cases, and anomalies that the website presents. And they sometimes forget that someone from an external agency doesn’t necessarily have the same depth of knowledge of their website. This can provide some challenges with the agency at the initial stage. Sharing internal knowledge that is required to help the business with their CRO programme at the beginning is very important. That way, the agencies will know what to expect and what to consider when they are coming up with a test idea or developing and QAing the tests.

Tasin Reza

In case you are starting from scratch, give your agency the time it needs to understand your business and create the following:

• Customer Research Handbook
  
  This is an internal document/database that gives the agency precious insights into who the customer is. It’ll contain data gathered from focus groups, interviews, email surveys, analytics analysis as well as data-backed ideal customer profiles and demographic info.

• Prioritization Matrix
  
  You cannot test everything. According to David Mannheim in his article, most programs are limited by:
  
  • The purpose of testing in the organization
  • Availability of traffic
  • The desired change
  • The amount of time available

In an ocean of test ideas, you need a system to decide on what test to implement and what to leave for later. If you’ve figured this out, your agency needs to know too.

• Q&A Checklist
  
  When tests are completed, how do you validate the accuracy of the results? A lot of steps go into executing a test and that means multiple touchpoints where something could go wrong.
  
  You need to share the steps you take to reflect on what’s been done to verify results. We have a Q&A checklist you can use, so you don’t have to start from scratch.

• Rudimentary Learning Repository
  
  This is where you keep data and documentation on past tests. Information here can improve the effectiveness and quality of your future tests to outstanding levels.
  
  If you don’t have a learning repository, you’re losing out on insights that can boost the success of your experimentation program.

So, why do you need to give your agency time to understand your business and absorb information from these resources? Because processes are a pillar of Experimentation Maturity. Creating some ground rules is a key aspect of rigorous onboarding. Don’t miss out.

4. Establish Trust

Give them the freedom to report flat & losing tests. Understand that most of the tests you run will lose. This is great, why? Because you are actively eliminating what may not work and moving towards changes, features, and decisions that have a higher chance of driving significant positive impact.

If you’re only hearing about the wins, you’ll be losing out of 90% of your insights. Because 9 out of 10 tests are usually a failure.

Also, if your agencies have to explain themselves every time you see a flat test, they’ll feel their services will be more valuable elsewhere – at least, the agencies that understand experimentation and its potential will.

You don’t want that. Instead, you want to bring your communication A-game and build trust. Here’s how:

• Communicate openly with your agency

Speak and listen. Talk about your organization’s goals and what you expect from your relationship with your agency. And then listen to what they expect of you and your team.

Make sure you acknowledge these demands and opinions. Even during subsequent meetings, as the project progresses, let them know that their feedback and ideas are valuable to your team.

If you have concerns, express them honestly. More often than not, your CRO agency is familiar with these concerns and knows how to address them in a healthy manner. They can also help you map a secure pathway to your goals.

That’s something you can always look back on when things are fuzzy and know you’re on the right path.

• Set mutually beneficial communication goals

Anytime you partner with an agency, you’ll have to set clear communication expectations. That’ll include communication medium, time and frequency of meetings, talking points, etc.

Defining these things beforehand not only smooths the work process in the long term, but that amount of certainty fosters trust. Both parties know what to expect, so you can relax and meet other obligations.

But there’s one more thing: You don’t just set this up and ride with it to the end. You’ll have to check back from time to time to ensure it’s working as intended. Ask your agency if they’re getting critical information on time, for instance.

P.S.: If you’re changing anything, let other stakeholders know.

• Handle conflicts like champs

We love to have a rosy ride, but the fact is, conflicts will occur. If you’re micromanaging, you’re stacking your odds in the favor of conflict. Please, don’t micromanage.

But in the event that you disagree on something with your CRO agency, what do you do?

Channel your inner HR manager and handle those tough conversations like a pro. Understand that those moments are not a “you-vs-agency” contest for you to win. Instead, that’s your partner and you have a common goal — growth by optimization (or something else).

Reassure them about the goal, revisit it together, and work your way up from there.

Do you need to change something? Does that change take you closer to the original goals of the project? Is there something you misunderstood? Or is there something you can explain better?

More questions during a moment like this can save time in resolution.

Finally, you give trust to get trust. Trust your agency to do their best job and deliver results, and they’re more likely to extend that trust back to you.

If you relate this way with your CRO partners, you’ll nurture a quality of relationship that gets them excited about going above and beyond for your organization.

5. Extract Learning

Build your learning repository. You cannot afford to let go of the insights from previous tests and deny your QA team the opportunity to diagnose any broken campaigns faster.

It’ll also help you smooth the process of reporting processes and results to your bosses and boost your test velocity (as you have ideas sprinkled all over your experimentation history).

But this can only be valuable if it is accessible and properly structured. A productive learning repo begins with you documenting every learning this way:

1. Record all the metadata from your tests

Highlight test type, pages you ran the test on, touchpoints, audiences, and growth surface areas. Condense this info as well as hypotheses, metrics, and outcomes, into a metadata scorecard.

Preserve this in an experimentation repository tool. When starting, you can use Google slides, Notion, or Airtable to keep and share this info.

2. Record for different learning styles

Not everyone will be a fan of the way you prefer to save information. Some thrive with text, others love visuals like images and videos. When recording your learning from previous tests, add screenshots, and screen recordings where you can.

3. Tag your tests

Anyone going through the repository should be able to instantly identify the purpose of a test. Was it iterating or disrupting? Or was it for optimization or innovation? Develop the tags that suit your experimentation program and use them to sort tests.

This learning repository will be invaluable to you and your CRO agency. They can explore the history of your experimentation program without asking too many questions.

And they can quickly diagnose a problem or, even better, design a plan that doesn’t waste time on tests you’ve already done.

Also, don’t think of your experimentation program as a separate entity from other processes in your organization. Other team members need to get on board with what your agency is doing. That’s when you organize one-on-ones so they can emulate and absorb how agency experts think about and approach testing.

While these 1:1s are integral in promoting an experimentation culture, you should also match it up with access to aggregated data and reports.

That’s one way to tap into the experience of your CRO agency experts. Another way is by asking for their input regarding structuring your optimization team (centralized or decentralized CoE). Since the experts have foresight and precedents, they can nudge you in the right direction based on your experimentation roadmap.

Conclusion

Now that you know how to work with CRO agencies for maximum productivity, you can start hitting your growth goals with as little friction as possible. There may be instances where you doubt your progress but, as long as you’re providing sufficient collaboration and have the right expectations, your partnership will be fruitful.