Getting Everyone on Board: 13 Tips for Gaining Buy-In for Experimentation

Experimentation and innovation are inherently intertwined.

Companies that run more experiments learn faster, iterate to optimal outcomes, and lean on data to facilitate better business decisions.

But I’m preaching to the choir.

If you’re reading this blog post, you know the power of experimentation.

But spreading that excitement to the rest of your organization, getting buy-in from key stakeholders, and getting the air cover required to build out a solid program? Not easy.

I’ve consulted for several clients as well as run experiments at HubSpot and Workato. I’ve seen companies from the earliest, primordial stages of experimentation, and I’ve also seen the ones you read about in case studies.

While no two companies and their cultures are the same, I’ve come across a few “soft skills” that will help you build the case for experimentation and conversion rate optimization at your company.

Some tips that I’ll cover include:

1. Have an executive sponsor

There’s idealism, and then there’s realism.

And here’s the reality: you’re not going to enter a company that doesn’t care about experimentation and change their mind from the bottom up.

It might happen if you join a startup early enough (you’ll have plenty of other challenges there). But if you join a company of sufficient size, you’ll need to lean on the support of an influential executive to make the case for experimentation.

Manuel Da Costa talks about this in his article on getting buy-in for CRO:

You’re highly unlikely to get broad senior management buy-in straight away, so it’s worth trying to pick a senior manager who you think is most amenable to adopting a conversion optimization strategy. Picking a HiPPO leader is not the way forward. Focus on senior managers who:

• Take a data-driven approach.
• Love to innovate and experiment.

Have a budget.

Luckily the “arc of the [experimentation] universe is long, but it bends toward justice”. In other words, if you work in tech, you have a high likelihood of joining an organization with a leader who is excited about experimentation.

They may need education on the nitty gritty details, but because of programs like CXL Institute, Reforge, etc., and the myriad conferences and blogs in our space, it’s not uncommon to incorporate experimentation into product and marketing teams anymore. 

It’s very clear from the data and my personal experience: low maturity organizations lack senior sponsorship.

At the very least, you’ll need the following:

• Leaders must understand the value of experimentation
• Leaders must align on KPIs and goals for the experimentation program
• Leaders must create conditions of psychological safety to allow for failure and tinkering

2. Make a strong business case for experimentation

Experimentation requires investment, and like it or not, investment requires justification.

I tell content marketers the same thing: you’ve got to prove your case and build a vision together.

One of my favorite tools to do this is a simple growth model.

Simply model out the key funnels in your given domain (this applies to email marketers, web product managers, growth product managers, etc.). Take all of your input metrics (like traffic to a given landing page, traffic per channel, etc.) and your output metrics (like conversion rates, or revenue per visitor) and establish baselines and historical trends.

Now apply some assumptions. Calculate the business improvement if you were to move the conversion rate by a minimum amount, a reasonable amount, and an exciting amount. Calculate the resources you would need to feasibly achieve those outcomes. Present the case.

For what it’s worth, building a model like this should be a core part of your research and planning process anyway. It’s also a really effective communication tool to get executive buy-in.

Yes, experimentation is about learning. But learning doesn’t pay the bills, so your leadership is still going to want to see some sort of quantitative forecasting.

Manuel echoes this point in his blog post, emphasizing that you should do some pre-work before presenting the case:

Come prepared with case studies and hard data. If you can do some tests on your own before meeting with management to demonstrate the power of CRO on revenue, great, but if you can’t, you’ll want to focus on showing stakeholders several things:

• The gaps and opportunities on the website.
• The potential results of CRO and the ROI, especially in terms of revenue.
• How much is being invested in driving traffic vs converting it.
• How being customer-centric benefits the company and brand broadly.

How an experimentation culture works for other companies and the impact it’s had on their growth.

3. Crawl, walk, run

Don’t bite off more than you can chew and try not to over promise too early.

I’ve made that mistake out of naive eagerness.

In other words, don’t try to “democratize experimentation” or test everything or run predictive mass personalization before you know how to run an A/B test and can showcase the value of doing so.

Manuel Da Costa talks about this with regards to the speed of experimentation:

Certain behaviors and practices that are established in this “speed phase” around test design and analysis become harder to break in the long run and experimentation programs get stuck in this high-velocity, low-quality mode.

A big challenge for companies trying to scale experimentation is they try to onboard new teams to the experimentation practice but find out that in a few months (or less), the teams are not running tests or running poor-quality experiments.

Why? Because of the need for speed, they onboarded teams without a clear strategy and a proper process. They treated it like a box-ticking exercise.

And honestly, you’ll need to titrate your own ambition to stay sane. Make sure you understand the maturity level of your organization and solve the biggest current problem. Then find the next one. Then the next one. And repeat forever, until you’re looked up to like booking.com or Netflix.

But start out slow, and maybe figure out how to design, implement, communicate, and document an experiment first.

4. Show your work and celebrate wins

Another hard lesson I’ve learned is that you don’t just need to do the work, you also need to talk about the work you’re doing.

In most large organizations, it’s hard to know what anyone else is working on. Experimentation, already esoteric to many, isn’t as visible as other efforts like shipping features or sending emails.

When I was at Workato, I made sure to communicate my team’s efforts in the following ways:

• I sent a weekly experimentation recap email to all team members, stakeholders, and executives, as well as individuals on other teams.
• I celebrated every win, especially when it was a collaborative effort with someone on a different team.
• I built a Slack channel with automated notifications when an experiment was launched or concluded.
• I held a weekly experimentation meeting to walk through past results, current experiments, and the current backlog.

I also just constantly evangelized the work my team was accomplishing.

When I was earlier in my career, I didn’t know how to evangelize for myself. I had a great manager who helped me do this at HubSpot, and I saw the value: you get more recognition, more resources, and more trust to build for bigger initiatives.

As I look back on those experiences, I am reminded of the transformative power that effective evangelism holds for experimentation programs. Proactively communicating and advocating for our work is crucial. Here’s a glimpse into how Speero does it:

5. Skip the nerd speak

You and I may care about Bayesian versus Frequentist statistics. Your boss doesn’t.

CRO industry insiders often get bogged down in minutiae and forget what leaders really care about: innovation, moving the needle, growth.

In communicating experimentation results, I learned always to have an executive summary, lots of visuals, and a very clear answer to “so what?” I included addendums and technical notes in case people were curious or wanted to drill down, but I never lead with those details.

Save those conversations for 1:1s with your analysts and data scientists or for lobby bar conversations with fellow CRO nerds at conferences.

6. Educate stakeholders on the role of experimentation in business growth

Not everyone understands the true value of experimentation, but you can lead them to their own “aha moment.”

I used to do this with design and brand teams, who were typically very curious to ascribe quantitative rigor and value to their work, but didn’t know how to accomplish that. Showing them how to set up a simple experiment was a giant unlock for their own work, and from then on, they would love to work with my experimentation team on rolling out more tests.

There will be varying levels of interest in experimentation throughout the organization. Oftentimes, teams set up educational resources, but they tend to be too homogenous. Not everyone needs to understand the ins and outs of p-values.

That’s why I often preferred a mixture of passive and active education.

Passively, I would set up a small Wiki with process documentation and additional reading and resources. I would recommend courses for people with certain specialties.

Speero found that only 9% of businesses strongly agree that they have an organization-wide education program, but it was much more likely that mature organizations would have one.

Actively, I would spend time and schedule 1:1s with people who would be great allies to our experimentation efforts.

The role of education is quite underrated in experimentation. Fortunately, there are so many resources available now, including courses like Reforge and CXL Institute, blogs (like this one), and books (I recommend Experimentation Works for business leaders and executives).

7. Make sure your data is trustworthy

Many naive leaders think a CRO specialist or experimentation specialist is just supposed to know what works and apply it.

It’s not about sprinkling on some social proof here and some trust symbols over there, or slapping on urgency to your headlines or CTAs (though those could all be great tactics).

Experimentation is a process for reducing uncertainty in decision making, thus capping your downside risk and enabling creative innovation.

It’s a methodology that leans on data as feedback to verify the efficacy of a new treatment to an experience. We use that data to make better decisions as to what the experience should be.

So if your data is bad, your experiments will be bad.

More so, if you have anyone with numerical rigor overseeing your experimentation efforts, you’re eventually going to get roasted when they start asking great questions about the data quality. I’ve seen people get skewered for both lacking knowledge about the data as well as for reporting shoddy data.

It’s your job to make sure this stuff is in order. If it’s not, and you don’t have the resources or ability to change that, make some noise.

8. Involve stakeholders in the experimentation process

People hate on the HiPPO. What I’ve learned is that the HiPPO is a reality. And by the way, in your career, you’ll either die an experimentation hero or live long enough to become the HiPPO.

People have ideas. They’re not all great ideas, but who cares? Some are strategically important, some are whimsical, but keeping a percentage of your experiment portfolio open for these ideas builds trust and collaboration.

So take your CMO’s idea and prioritize it high in the backlog. Run the experiment. Communicate the results. If necessary, iterate on the hypothesis.

It’s like the Ikea Effect: people place a higher value on things they help build or create.

Additionally, invite them to workshops, brainstorming sessions, meetings. Keep them in the loop. They should feel like they’re a core part of your experimentation efforts.

9. Incorporate other teams

Next up, invite people from other teams to your experimentation potluck dinner. Everyone brings something to the table, and together, you start to go from a centralized and guarded experimentation function to a company that experiments together.

Speero found a mixture of centralized, decentralized, and center of excellence models among the most mature organizations. The thing in common is that they have dedicated teams working on experiments:

I’ve found that bringing in other departments is valuable in many ways, including leading to more diverse and creative experiment ideas. It also allows experimentation to scale more easily throughout the company, though processes and guardrails are important.

10. Have a process

Process is key in experimentation to drive repeatable results; it’s also an extremely effective way to communicate that you and your company take experimentation seriously.

When you can point to documentation, workflows, and rituals that your team runs, it’s less likely that leadership will look at experimentation as whimsical or something stapled onto existing efforts.

It’s also easier to onboard people into the program.

Make sure everyone understands the steps involved in running an A/B test, from hypothesis generation to analysis of results. This will create transparency and ensure that everyone is on the same page.

11. Trigger FOMO

Look, I hate to say it, but one of the easiest ways to get some executives on board with an initiative is to show that innovative companies or your competitors are already doing it.

Use examples from your industry or even other businesses to demonstrate how experimentation is an essential tool for staying ahead.

Use case studies or statistics that highlight the impact experimentation has had on revenue or market share to show the power of experimentation.

Just avoid getting into the rut of directly copying competitors.

12. Avoid success theater

While I mentioned celebrating wins (and I think that’s very important), it’s also important not to get sucked into the trap of punishing failures.

Manuel Da Costa put this well:

The most important part is to create a culture where failure is okay. It shouldn’t be punished, it should be rewarded.

Facebook had a mantra: “Move fast and break things.” Really, what this meant was that their policy was to test iteratively and if they failed, at least they learned something valuable as they moved forward.

I wrote a post for HubSpot a few years ago outlining specific ways to avoid success theater as you scale.

13. Make experimentation cheap and easy

Whether you want to build a decentralized experimentation model where every team is running their own experiments, or you just want to run more experiments yourself, it’s important to remove the friction to running tests.

At the very least, you need a solid experimentation platform.

Analytics should be well-integrated into your testing tool.

You should have some basic design and development resources to run tests.

Ideally, you have a formalized process that automates a lot of the workflows that need to happen to get an experiment live.

I’ve learned that the harder something is to implement, the less likely it will stick in an organization’s DNA.

If it takes a whole quarter to get a simple experiment live, it’s unlikely your program will thrive. Work, as a program manager, on making the program run more smoothly.

This is largely how booking.com has become a ubiquitous experimentation case study, and it’s how Lukas Vermeer has spoken about his experimentation platform initiatives at Vista:

It does this by decreasing the cost and friction involved in experiments—as well as increasing the value of the results. For example by creating standard tools to automate repetitive experimentation related tasks, and by facilitating sharing of results and peer review.

Conclusion

Outside of technical know-how and strategic intelligence, learning how to get buy-in and leverage support and resources for your experimentation program is a huge skill.

It’s mostly art, very little science. And how you do so will vary from company to company depending on the resources and existing culture.

Luckily, the world is moving in a direction where experimentation is supported and understood. Speero’s data shows that there was a big maturity jump even in the last year:

The key thing to remember is getting buy-in and communicating the value of your program isn’t a waste of time. It’s a critical part of the job.

It doesn’t just lend additional development resources to your team (though it can do that). Building and communicating the case for experimentation spreads its value throughout the organization. It opens your company up to an innovative mindset. It unlocks the dormant creativity of individuals within the company.