How To Communicate Your A/B Testing Results Like the Experts22nd Oct 2020 –
Wouldn’t it be great if you could slap your A/B test hypothesis, control/treatment, and results on the desk of your stakeholders and walk away confident they would understand everything?
Better yet, what if you could use telepathy to beam all your A/B testing plans and results to the people who cared about them?
This may come in the future, but today we still have to share experiment plans and results in the old fashion way – presentation style.
But presentations take different forms and contain a variety of information.
Maybe that’s why there is no agreed-upon industry standard on how to present your A/B testing process to key-players. Everyone has their own styles and methods.
In this article, we will go over the important factors for communicating your A/B testing plans and results so they are understood, and actionable, including:
- What to share
- Experts tips, tricks, and suggestions
- And a gift – a downloadable Presentation Template.
Here we go…
The Importance of Reporting A/B Testing Data In a Digestible Form
What’s more important, the A/B testing plan and results or communicating your plan and results?
One is not more important than the other. They actually depend on each other.
As important as the actual testing is, it is just as important to share the results in meaningful ways. This sharing supports the trust cycle:
It takes a lot of planning and energy to move through all the steps of an A/B test from start to finish. How well you share information along your A/B testing journey is what will transform data points from numbers to potential optimization actions.
If you are an agency, your client relationship depends on this.
For in-house and agencies alike, this is where you have the opportunity to put on your sales hat and show-off all the hard work you and your team did.
While on the surface, delivering your results should be simple, you can run into several complications if not done correctly.
Common mistakes include:
- presenting data only,
- not preparing for pushback,
- not showing impact, and
- not sharing potential next steps.
Let’s take a quick look at each of these mistakes…
Sharing Data Only Without an Explanation
Has someone shared information with you and you wondered, “what the ?!@#*& are they talking about?”
It is a magical day when data sings its purpose and meaning, so others understand it without analysis or interpretation. This doesn’t happen by accident. It takes a songwriter, someone who understands the meaning of the data, to put it in story form so others understand it.
Don’t leave stakeholders to their own devices to interpret the data. They may come up with a different story than you intended, or worse, just not understand at all. If they don’t understand it, it is useless to them. It is your job to make sure they are empowered with the information they need.
“The combination of data and storytelling is powerful and it can give confidence to teams worried about changing their websites or who feel overwhelmed with the number of ideas floating around.”
How well you communicate the important details of your projects adds value and may be the difference between a one time purchase or a returning customer who refers others.
It doesn’t mean they are going to like or agree with your hypothesis or the results. Which leads me to the next trap to watch out for…
Somebody May Not Like Your Hypothesis or A/B Test Results
Be prepared for this possibility.
Just because you have data, logic doesn’t always prevail.
Your A/B test treatment may beat a control that has been working for over ten years. You could be presenting your findings to the team that built that original page, and they are still vested in its success.
This is one of many potential situations, so prepare for nay-sayers.
Just because you have data doesn’t mean everyone will be receptive to its implications. Prepare for this. Tests are unbiased, but people are not. They may have interests, desires, and stakes, which don’t always align with what the test uncovers.
The point here is to not ignore the potential that emotions may play into the mix.
Which is why you want to share the upside as it impacts the business and sets them up to take next actions.
This is critical. Your report must speak to the bottom line.
Let’s explore the specific elements that must be included in your A/B test report.
No matter how you share the information associated with your A/B test, the contents should always include the following elements:
Overview: What is the purpose of your test? Why are you running it?
Details: This section should include relative details like test description, variations (descriptions or illustrations), test dates, duration, segmentation, number of visitors, data you are planning to capture, etc.
Outcome: Be specific here. Identify the winner, percentage lift/loss, and other meaningful A/B testing statistics associated with the experiment.
Insights: Time to share what you learned and interpret the meaning of the numbers. This is where you let the data sing its song. Did anything surprise you? How does this test inform your next test?
Impact: Lead with the bottom line. How do these results influence revenue? Remember, strip everything away and ways to increase revenue is what you are looking to uncover with your a/b test.
We asked Nick Swekosky, CEO of Market Metrics how he empowers his audience to translate results into next steps, he offered:
“Always attribute the results of your A/B test to the desired business outcome and the actions that affected the A/B test.”
This layout can be done for pre-tests and for tests that don’t have a winner.
Deb Cinkus, CEO and Owner of Polished Geek explains what this looks like pre-test. She shared her presentation in a CRO FB group.
Cinkus hits it on the head with her presentation layout. Every step of the way she’s tying benefits in and answering the question, “why does this matter?”
Cinkus suggests sharing A/B testing information via a presentation. Let’s explore other formats.
Having a basic structure or template you can use to convey information to your clients and internal teams can take the guesswork out of communication, allowing you to be more efficient.
There are several ways to share information, from simple to complex. But one thing must remain the same – it must be clear and actionable, and at a minimum, contain the sections laid out above.
Potential formats include:
- Combination of all the above
“typically creates graphics from the A/B testing platforms that we use. We try to use graphics and organized charts. We outline exactly what the results mean and our detailed recommendations on what to do next. Technically, we empower our clients to implement the strategies on their own. However, they almost always have us continue with more content marketing and development projects.”
Additional Tips, Tactics, and Suggestions From The Field
- Educate When Needed
Don’t assume your clients understand the testing process.
- Know Your Audience
Always considers who is receiving the message. Marketing 101: Tailor your message to your audience. Calibrate your messaging to fit the needs of the receiver.
David LaVine of RocLogic Marketing, LLC illustrates this with two tiers, clients who require less information and clients who require more. His template is as follows:
For a bottom-line client, provide:
1. The topic to test
2. What you hope to learn
3. The category of measurement criteria (e.g. ranking, engagement,
conversion rate, CTR, etc)
For clients that require nuts-and-bolts client:
1. Explain what you hope to learn via telecon. This gives them a
chance to ask the “why’s” and gives you a chance to answer
contextually and immediately, without the rabbit hole of painful
2. Create an enumerated list of less than ~8 steps illustrating the
testing process. Anything more detailed than that is likely to
encourage the client to get wrapped around the axle.
3. Set expectations upfront that if the results don’t show a significant
improvement, the option to do nothing is totally valid.
“When presenting A/B testing to clients, we typically talk about the process and value of the exercise through to them.
Depending on the client, some may need more visual explanations in which case we’d use slides to illustrate the test visually, while for more digitally versed clients, planning stems from a thorough conversation with the goal of getting a better understanding of variables in question for the test.”
- Remove Outlier
This is common with retail stores. You may have a small purchase of $10 and a large purchase of $100,000. You want to remove these outliers.
Convert allows you to do this seamlessly in our A/B testing tool.
- INSIGHTS, INSIGHTS, INSIGHTS. I mentioned this early, but it is so important it is worth repeating. Answer, “how the results affect the bottom line?”
- Standardize, But Customize.
You are a busy CRO, you don’t have time to create a unique report for every A/B test you do. Create a standardized format, but don’t copy and paste. You may miss something important.
- Lead with Value.
Don’t make them wait for the juicy part, give it to them upfront. What’s the lift? Don’t get into the weeds. High-end exclusive content. And keep the format simple.
- Easy Access.
Make sure your results are easy to access. Make them shareable.
- Use visuals, illustrations, and images.
Especially pictures of your control and treatment. They make the case and are much easier to understand.
- Aggregate Big Amounts of Data.
If you test often, you may not want to share every single test, in that case, aggregate the data and share in a newsletter-style format a set period like bi-weekly or once a month.
Stacy Caprio, founder of Accelerated Growth Monitoring and HER CEO seconds this advice. She says,
“I have found presenting one A/B test win at a time to be most effective, instead of sending a long email or a report to a client every month. People see results best when you present them as a win, instead of sending a page-long report they have to dig through to find the A/B test wins for the month.”
- Use Nuclear Language and Minimize Jargon.
Transmitted in clear language that’s understood. Can’t take away words unless its meaning becomes compromised.
Different people have different expectations and as much as we try to keep A/B testing test planning to be the same, it rarely is.
We want to offer the best for our customers, therefore sometimes the process takes different turns to satisfy the needs of those that are paying sizeable sums of money for the product and service that we provide.
We make it clear to the clients, especially in the early stages, so they are aware of the steps we take to get them what they want.
Time To Present Your A/B Testing Plans and Results With Confidence
Hopefully, you now have a formalized process for presenting your A/B test results as part of your experimentation protocol. Now that you know exactly what to include in your presentation, you can combat potential pitfalls, and have a template you can work from, clearly expressing the impactful elements of your test to inspire action should be simple.