Experimentation is Human Nature™. Use This Realization to Fuel Your First Test.

Curiosity drives us.

Picture a child who dismantles a toy, not to break it but to understand how it works. That’s us. We don’t grow out of that. We all have an innate desire to explore, to understand, and to improve.

Yet, for many, the word “experimentation” feels distant, perhaps even clinical. But it isn’t just about lab procedures and petri dishes. At its heart, experimentation is the practice of exploring, testing and iterating ideas and processes in pursuit of better outcomes.

It’s about tapping into that childlike curiosity that drives us to understand and improve. And applying it in a structured, methodical way.

Appealing to that experimenter inside every individual and tying it back to the context of business goals… that’s experimentation.

Doing this right puts you on the right side of this snarky remark:

Experimentation legends John Ostrowski and Simon Girardin recently sat down with us to discuss:

• The true essence of experimentation and its role in modern businesses
• Real-world challenges and triumphs in establishing an experimentation culture
• The pitfalls of treating experimentation as just another business tactic
• Insights on how to foster a genuine, data-driven experimentation mindset

Here’s why we did this interview:

1. Experimentation is Human Nature: This is a core idea in our strategic narrative. It’s a cure for the old way: Thinking the tool is the be-all and end-all of optimization and going for exorbitant annual contracts that leave little wiggle room to invest in the right people and processes.
   
   The new way is not to just optimize or A/B test, but instead to continuously explore new ground using the very spontaneous and instinctual instrument of experimentation.

From finding the perfect workout routine to crafting the most relevant ABM campaign, experimentation (and iteration) is the bedrock of human innovation.

If experimentation is human nature, then we also want to…

2. Explore how this propensity to experiment can be used: Specifically, in two different scenarios:
   • The scenario where folks who have no exposure to optimization get to understand continuous improvement through the more familiar idea of experimentation.
     
     With the hope of igniting their interest so they make the transition from opinion-based to data-driven more easily.
   • The scenario where an organization is starting its “CRO” program from scratch and the intentions and mindset it needs to have in place. Such as those around the acceptance of data-driven decision-making, i.e. team members not afraid to be proven wrong.
     
     And how these intentions and mindsets are a given when humans experiment in everyday lives, without the pressure of KPIs or the siren calls of revenue. All of which can be a more robust and effective springboard into experimentation, rather than looking to competitors for inspiration.

At the end of the day, ladies and gentlemen, experimentation requires infrastructure and rigor. Yet, the concept of experimentation—the ideas you have to seed in your team to actually get to the point where an experimentation program is possible—is as innate to humans as the desire to sleep or eat. Primal, if you may.

Start with the Right Intention: Everyone is an Experimenter

In this video clip, John Ostrowski is sharing his experience with Nick at Wise Publishing. Nick already grasped two foundational concepts:

1. That it is important to give the entire team data to base their work on, and
2. It is equally important to empower them to bring these data-informed ideas to the table and get a shot at having them improve the reader experience

Combine those two and you’ve defined experimentation.

This enables everyone to build on the existing foundation of experience and knowledge and try new things. It also gives novel ideas a shot at pushing the boundaries of what’s considered “good” and “acceptable” within that ecosystem.

The beauty of experimentation is its universal appeal. Most leaders intuitively get it. They know they have to place bets and watch them play out. Sure, they might not always think in terms of statistical significance or metric choice, but the core intention is always present.

And you can use this to pave the way for an experimentation program.

This brings us to the next vital element of preparing the foundation for experimentation: hiring folks who already know they should bring data to any argument and all proposals. Because, let’s face it, teaching this to grown-ups is tough.

Getting it right from the start—hiring those who inherently value experimentation—is the only antidote to friction later down the line. You know, those conflicts where the experimentation team is battling the entrenched mindsets of the rest of the organization to deploy the first few tests.

Move Away from the Idea that Experimentation is Just Another Tactical Channel

At Convert, we believe Experimentation is Human Nature. It’s not just a strategy; it’s a way of life. But, we were curious, have Simon and John come across teams where experimentation was treated as just another tactic?

They were aligned on this one…

John said that experimentation as a tactical channel (where the right intention is missing) is something he actively avoids in his consultation positions.

Simon said that in his vast experience, he’s seen many teams value data. While they might not collect it in a way where the insights are easy to find (something Conversion Advocates excels at), the “data-informed” bug has bitten everyone.

But he went on to talk about an instance in the past (and you can look back on his journey to becoming a CRO lead at one of the best optimization agencies in the world) where he had been on teams where bias ruled the roost. All changes were based on opinion and bias and there was pushback when data-triangulated updates were proposed.

When people misunderstand experimentation, they use it to:

• Impress VCs with “something dynamic” in a quarter:

• Produce quick results under pressure
• Fix poor performance across the board
• Chase revenue without doing anything “extra” (or add any real value)

Both Simon and John would rather not work at all than work with teams where the data-informed mindset (including the mindset of testing ideas and long-held beliefs) isn’t given the time of day.

This makes us think: It’s time the C-suite wakes up and scrutinizes the impact of closed-off mindsets.

They not only reject data that may show where their business is falling behind competitors, but they repel professionals like John and Simon who can help with guidance and locating which motor in the organization is idle and which could use an upgrade.

This strands the business in a quagmire of negative feedback. Leaders listen to the yes-folks who in turn perpetuate this vicious cycle of outdated playbooks and a lack of regard for customer research.

As highlighted in Speero’s post below, you can choose the ultimate goal of your experimentation program. It can be about understanding customers or going for quick wins. But not having a strategy is letting a competitive advantage slip out of your hands.

Break the Cycle. OWLs Keep HiPPOs in Line

OWL: Objective Wisdom through Logic

Owls are often associated with wisdom and insight, making them a fitting symbol for those who prioritize data and logic over subjective opinions.

It all starts there… bringing on people with the right mindset.

While many in the experimentation space focus more on tools, techniques, and strategies in experimentation, only a handful emphasize the significance of structuring the right team and hiring right. Among these voices, Manuel Da Costa and Jeremy Epperson stand out, echoing through our daily feeds.

#TeamOverTools. Because at the end of the day, a tool is only as good as the hand that wields it.

Take a look at Booking.com. In their bar-raising experimentation program, they map bringing in the right people to the company’s growth in general. When you scrutinize their job descriptions, you’ll spot a recurring theme: critical thinking, data-driven decision-making, and a passion for experimentation.

Do an “Experimentation Onboarding” for All Key Roles

Hiring the right talent is just the beginning. Integrating them into the experimentation fold moves things further in the right direction.

That’s where onboarding comes in.

At Conversion Advocates, Simon Girardin runs “mini onboarding” sessions for newly hired C-suite members and directors. These sessions aren’t your typical orientation. They dive into

1. Roadblocks: A complete overview of the conversion roadblocks identified so far, the insights they have triangulated, and what they will be testing to remove said roadblocks.
2. Challenges: A candid discussion about the challenges the experimentation program faces on its journey to maturity. They open up about the glut of test ideas pouring in from all departments, and the need to properly prioritize them into an impactful backlog and experimentation roadmap.

And Ronny Kohavi also recommends something akin to this!

Buy-in from executives and managers must happen at multiple different levels and include:

• Engaging in the process of establishing shared goals and agreeing on the high-level goal metrics and guardrail metrics and ideally codifying tradeoffs as steps to establishing an [Overall Evaluation Criterion].
• Setting goals in terms of improvements to metrics instead of goals to ship features X and Y. There is a fundamental shift that happens when teams change from shipping a feature when it does not hurt key metrics, to NOT SHIPPING a feature unless it improves key metrics. Using experiments as a guardrail is a difficult cultural change, especially for large, established teams to make as they shift towards a data-informed culture.

Excerpt from Page 60 of Ronny Kohavi et. al’s Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

Such sessions do more than just inform. They immerse new hires in the company’s experimentation culture. They get a firsthand look at the hurdles, share the collective enthusiasm for experimentation, and feel a part of the mission from day one.

Getting on the same page with data-minded hires and setting the right expectations is crucial for the success of the program.

How to Find Your OWL Allies?

Tricky business to find allies to further secure your buy-in, right?

We have two absolutely golden nuggets from John Ostrowski & Simon Girardin.

John’s approach? Dive into the company’s OKRs and goals first. He zeroes in on roles facing the steepest revenue challenges, offering a smoother path via data-driven decisions and experimentation.

Simon adds to this starting point. He suggests scouting for team members showcasing:

1. Curiosity: Do they genuinely want to understand the business’s inner workings, or are they content with internal assumptions?
2. Humility: Are they open to having their hypotheses challenged? Can they update cherished decisions based on fresh data?
3. Radical Honesty: Will they cherry-pick metrics to force wins? Can they resist the pull of skewed incentives?

At the end of the day, starting an experimentation program can hit roadblocks if the team doesn’t prioritize experiments.

This is where company leaders come into play. They’ve likely run thousands of experiments (not necessarily A/B tests) to get their products off the ground—their version of the most viable solution for a particular market problem.

Leaders must rekindle their love of trying new things to make existing things work better or to create new processes and campaigns from scratch. With A/B testing, they get the added safety net of statistical validations, reducing risks.

If your business venture is an adventure, experimentation is the armor!

Inspire Interest in Folks with No CRO or Experimentation Background

This illustrates what Simon is talking about:

You can see the alternative to experimentation is accidental gains. Simon explains why accidental gains aren’t what you want:

• You can only measure success after the fact
• You have to fully invest in the feature before reaping rewards (if any)
• Poor performance will affect 100% of traffic and often for periods of 1 to 2 months
• Performance is evaluated on a sequential basis, which can be influenced by (sometimes unknown) external variables
• Projects are larger and take a lot of time
• Successes create a massive growth spike, and then
• They are followed by long periods of stagnation

Meanwhile, experimentation allows teams to capture more frequent gains. Though incremental, they stack. And every loss is mitigated by being shown to only 50% of traffic, for a max of 2 weeks.

Ultimately, both curves will have fluctuations. Each company will have its own growth curve. However, experimentation creates a larger area under the curve, which means extra revenue and profits.

So, how do you secure experimentation buy-in to move to incremental gains?

The first conversation with the team regarding this could be awkward, especially if there’s no experimentation infrastructure to speak of.

For data-driven marketers and anyone with prior optimization experience, settling into a broad experimentation mindset isn’t a huge hurdle to climb.

But for folks with little to no experience, it’s probably best not to start with CRO. CRO is more popular in comparison but it may be interpreted as just another growth tactic, which isn’t aligned with the foundations we’re setting here.

You have to make experimentation feel natural and familiar to them first. Highlighting that “we already do this, let’s double down on it” makes it easier to get these folks on board when starting from scratch.

When Simon does this, he starts by sharing his story from a CRO specialist to a program lead. And then shows practical examples of experimentation and its successes. This excites beginners to build on something they already do informally.

Blank Canvas: Getting Started After That Critical First Conversation

After recognizing the innate human tendency to experiment, the next step is to channel that into a structured, business-focused experimentation program.

Here we’ll give you a blueprint to guide you through the transition from that pivotal first conversation to your first test:

1. Understand Your Holistic Growth Goals

Conduct an audit to get started here. An audit helps you understand and map out, where your org is and where you want it to go; plus it puts a spotlight on the end goal.

Since you can’t do this on your own, John suggested doing this exercise:

Get your stakeholders in a room and ask, “How do we grow?” It’s very likely you’ll get different responses. If that happens, that’s a symptom of misalignment. It needs to be addressed first.

This process should encompass every aspect of your brand in the form of customer touchpoints, not just acquisition. Because this is where the myopic view of improving acquisition conversion rates is born (which creates a fixation on CRO while sidelining the broader potential of experimentation).

At this stage, you’ll also want to consider a maturity assessment. Why? Chances are there’s some experimentation ‘activity’ that already exists. And your org would be willing to 2X, 3X, or 10X that existing activity.

Tools like Speero’s Experimentation Program Maturity Audit or Microsoft’s Experimentation Maturity Model can help you explain the path better (and sell it) with solid definitions for different dimensions like metrics, platform, team structure and organization, etc. They also help you prioritize the blocks to focus on first.

2. Take Stock of Larger Projects in the Pipeline

Find out how these releases or projects could impact ongoing tests. Are there any overlaps?

This is similar to understanding your holistic growth goals: if you do not have it figured out, it could relegate experimentation to top-of-business elements like imagery and value proposition.

3. Dive into Innovation with Experimentation

Try testing in areas related to potential new releases. It gets you closer to the serious impacts that win buy-in for experimentation. Resources like How to Win Big with A/B Testing can offer additional insights here.

A word of caution: Executives might be risk-averse in the beginning. You can overcome this hesitancy by showcasing the tangible benefits of experimentation, as Simon mentioned in his post:

Utilizing the ability to create low-fidelity prototypes and making a tangible (positive) difference to the bottom line of an investment that’s already approved is a compelling introduction to experimentation.

4. Understand the Limitations of Controlled Experiments

While controlled experiments are the gold standard, you may not always have the luxury to run them, especially in the early stages.

While rigor is of paramount importance in an experimentation program, it is also key to have a process in place when experiments are not feasible.

What do you do in these instances?

John recommends blending data from customer surveys, competitor analysis, and preference tests to zero in on signals that stand up to these compounding insights. The customer data-driven signals can then be used to inform decisions.

He also advocates maintaining “Decision Journals”.

Proposed by Nobel laureate Daniel Kahneman, a decision journal serves as a tool to capture your thought process, the context, the uncertainties, and the expected outcomes at the time you made the decision.

Over time, this journal becomes a repository of decisions (or “decision log”), allowing you and your teams to revisit them, assess outcomes, and refine your decision-making skills.

5. Cultivate Empathy for Team Members

Not everyone will immediately embrace the concept of business experimentation.

Extend empathy to such team members and help them get into experimentation by sharing examples of work they have done where informal experimentation was already at play.

Remember, the journey of establishing an experimentation program is less about the tech and more about managing leadership expectations and the aftermath of change which is introduced as a result of the “question and then test everything” mindset.

For those seeking guidance here, Ruben’s Udemy course is a valuable resource. Additionally, integrating empathy-building exercises into the experimentation program can create a more inclusive environment.

If team members voice concerns or skepticism, host open discussions or ‘rant sessions’. These can often lead to transformative ‘wow moments’ when they begin to see the value of experimentation.

And don’t forget self-empathy. Leading the charge in establishing an experimentation culture can be challenging. Avoid being overly critical of yourself.

In the end, we aren’t striving to make perfect decisions.

It is about making better decisions every day so all business goals—not just funnel-focused metrics—can benefit.

Conclusion: How Convert Supports the Sentiment of “Experimentation is Human Nature”

In three ways:

1. Technology: A/B testing platforms are tools, and by definition, can only produce results as good as the user. Poor use gives bad results; maestro-level use creates wonders.
   
   We understand this. That’s why we’ve built Convert Experiences with intentionality. We are also the foremost believers in “experimentation is human nature”. Hence, our relentless push for everyone to tap into this innate curiosity to make data-driven decisions.
   
   Great experimentation programs are built by people, not tools. So, we’ve never reconciled with the idea of 100,000-dollar annual contracts for enterprise A/B testing features. Instead, we offer self-service pricing.
   
   We believe in empowering teams, not emptying pockets. For $499/month, you get access to essential testing features you can use anywhere (including full stack). This also comes with safety features like SRM checks and collision prevention; and advanced targeting and goal engines.

2. Process: Convert Experiences is built for the hands-on tester who values integrations, access, and versatility. With 90+ one-tag integrations, it blends seamlessly into any tech  stack. And with our Data Sources feature, you can set up test triggers based on input from other apps. Our robust API extends this integration even further.

3. Culture: Cultural roadblocks litter the road to starting an experimentation program. Campaigning for a data-informed culture is extra difficult when you’re battling with:
   • A steep budget barrier: With affordable monthly plans that provide all the features you need to maintain experimentation rigor, you no longer have to watch from the sidelines as opinions, instead of data-backed decisions, rule the day.
   • Admin: Our support team is 10x faster so you can spend more time testing and less time struggling with tool issues.
   • Onboarding headaches: Our 10-step onboarding ensures everything that should work, works — from tracking code installation to glitchless A/A tests.

With Convert Experiences, you can ignite a movement at your organization.