Testing Mind Map Series: How to Think Like a CRO Pro (Part 75)

Interview with Hannah Davis

There’s the experimentation everyone talks about. And then there’s how it actually happens.

We’re hunting for signals in the noise to bring you conversations with people who live in the data. The ones who obsess over test design and know how to get buy-in when the numbers aren’t pretty.

They’ve built systems that scale. Weathered the failed tests. Convinced the unconvincible stakeholders.

And now they’re here: opening up their playbooks and sharing the good stuff.

This week, we’re chatting with Hannah Davis, Managing Director, CRO at Brainlabs.

Hannah, tell us about yourself. What inspired you to get into testing & optimization?

I started my career in the field of product management, so my entry into experimentation was as a means to merely validate and de-risk feature releases. Once you are able to provide a statistically proven answer to ‘Did this have a positive impact on our business?’, it becomes almost uncomfortable to be in situations where subjective decisions are made. I opted to spend the rest of my career where experimentation and data-driven decisions are the core of the business due to this. 

How many years have you been testing for?

Around 10 years.

What’s the one resource you recommend to aspiring testers & optimizers?

The career-defining resource that transformed my approach to optimisation was David Kelley’s ‘Creative Confidence.’ This book shattered the misconception that creativity is reserved for artists, revealing instead that experimental thinking is a muscle anyone can develop through deliberate practice and mindset shifts.

In my experience, the greatest challenge organisations face in optimisation isn’t generating insights, it’s translating those insights into effective, testable solutions. Kelley’s work establishes the foundational mindset necessary for powerful ideation, particularly by reframing failure as an essential part of the discovery process.

My favorite passage from the book perfectly captures the counterintuitive mathematics of meaningful experimentation:

“The surprising, compelling mathematics of innovation: if you want more success, you have to be prepared to shrug off more failure.”

What makes this resource so valuable for aspiring testers and optimisers isn’t just its approach to creativity, but how it fundamentally reshapes your relationship with uncertainty and learning, the true prerequisites for groundbreaking optimisation work.

Answer in 5 words or less: What is the discipline of optimization to you?

Thoughtful experiences, proven through experimentation.

How do you treat qualitative & quantitative data to minimize bias?

The deepest understanding of customer behavior occurs when qualitative meets quantitative data. At Brainlabs, we take an approach that some might consider unconventional, we begin with qualitative data when reviewing a digital experience for the first time. By starting with expert reviews where we actively embody the customer, we can explore behavioral patterns that might not be immediately visible in macro metrics in the analytics funnel.

For example, in a recent e-commerce project, our qualitative review identified subtle friction points in the product comparison process that weren’t showing up in the standard funnel metrics but were significantly impacting purchase decisions.

To safeguard against confirmation bias, we then rigorously validate these initial insights with quantitative data. This tells us both if the problem truly exists at scale and quantifies its impact on conversion performance. We use structured frameworks and multiple reviewers to ensure our qualitative assessments remain objective.

This methodology allows us to look beyond macro metrics like ‘add to cart rate’ or ‘checkout abandonment.’ While these indicators provide valuable signals, they rarely pinpoint what’s actually holding customers back. Our approach is ultimately more efficient. It combines human insight with data validation to identify specific opportunities that purely quantitative analysis might miss or take longer to uncover.

How (to you) is experimentation different from CRO?

I see experimentation and CRO as related but distinct approaches to improving digital experiences.

Experimentation is broader in scope, it’s a scientific methodology for testing hypotheses across any aspect of business. It embraces learning from failure, explores new possibilities, and can address fundamental questions about product-market fit, user needs, or business models. True experimentation is comfortable with uncertainty and values insights over immediate wins.

CRO to me more specifically focuses on optimising conversion metrics within existing journeys. It typically operates within established frameworks, often emphasising incremental improvements to specific KPIs like click-through rates, form completions, or purchases. CRO tends to be more tactical and results-focused.

The key difference lies in their objectives: CRO aims to improve defined conversion metrics, while experimentation seeks to generate learning and insights that may lead to more transformative changes and change the culture of an organisation.

Talk to us about the cool CRO stuff you’ve done in the last 1-2 years.

Over the past two years, AI has significantly transformed our approach to CRO, accelerating both our velocity and precision in ways that benefit Brainlabs and our clients. While we’re strategically implementing AI to streamline the more process-heavy aspects of our CRO practice, we maintain a deliberate balance. Our core value proposition remains deeply rooted in behavioral science principles, which is why we prioritise the human element in our work. We focus on crafting thoughtful solutions that respond directly to observed behavioral triggers, combining AI’s analytical power with the irreplaceable human insight needed to truly understand and influence customer behavior.

Cheers for reading! If you’ve caught the CRO bug… you’re in good company here. Be sure to check back often, we have fresh interviews dropping twice a month.

And if you’re in the mood for a binge read, have a gander at our earlier interviews with Gursimran Gujral, Haley Carpenter, Rishi Rawat, Sina Fak, Eden Bidani, Jakub Linowski, Shiva Manjunath, Deborah O’Malley, Andra Baragan, Rich Page, Ruben de Boer, Abi Hough, Alex Birkett, John Ostrowski, Ryan Levander, Ryan Thomas, Bhavik Patel, Siobhan Solberg, Tim Mehta, Rommil Santiago, Steph Le Prevost, Nils Koppelmann, Danielle Schwolow, Kevin Szpak, Marianne Stjernvall, Christoph Böcker, Max Bradley, Samuel Hess, Riccardo Vandra, Lukas Petrauskas, Gabriela Florea, Sean Clanchy, Ryan Webb, Tracy Laranjo, Lucia van den Brink, LeAnn Reyes, Lucrezia Platé, Daniel Jones, May Chin, Kyle Hearnshaw, Gerda Vogt-Thomas, Melanie Kyrklund, Sahil Patel, Lucas Vos, David Sanchez del Real, Oliver Kenyon, David Stepien, Maria Luiza de Lange, Callum Dreniw, Shirley Lee, Rúben Marinheiro, Lorik Mullaademi, Sergio Simarro Villalba, Georgiana Hunter-Cozens, Asmir Muminovic, Edd Saunders, Marc Uitterhoeve, Zander Aycock, Eduardo Marconi Pinheiro Lima, Linda Bustos, Marouscha Dorenbos, Cristina Molina, Tim Donets, Jarrah Hemmant, Cristina Giorgetti, Tom van den Berg, Tyler Hudson, Oliver West, Brian Poe, Carlos Trujillo, Eddie Aguilar, Matt Tilling, and Jake Sapirstein, and Nils Stotz.