Testing Mind Map Series: How to Think Like a CRO Pro (Part 87)

Interview with Daniel Mullins

There’s the experimentation everyone talks about. And then there’s how it actually happens.

We’re hunting for signals in the noise to bring you conversations with people who live in the data. The ones who obsess over test design and know how to secure buy-in even when results are complex.

They’ve built systems that scale. Weathered the failed tests. Convinced the unconvincible stakeholders.

And now they’re here: opening up their playbooks and sharing the good stuff.

This week, we’re chatting with Daniel Mullins, a customer value focused product leader with a track record for delivering measurable outcomes across ecommerce, booking platforms, and financial services.

Daniel, tell us about yourself. What inspired you to get into testing & optimization?

I’m very competitive and always want to try to find a better way to do something or to improve a score, etc. Early in my career, I found a love for web analytics and data, which fed my desire to find ways to improve the numbers I was observing.

Testing and optimization then organically became part of my roles, starting with data analysis and insight generation, and moving into A/B or Multivariate testing. 

How many years have you been testing for?

In total, I’ve been testing for nearly 15 years now.

Answer in 5 words or fewer: What is the discipline of optimization to you?

Finding the best way.

What are the top 3 things people MUST understand before they start optimizing?

What are you trying to do or understand?

Why are you doing this?

How are you going to measure success or learning?

You’ve recently made the shift to product. How does your experimentation background help in this pivot?

It enables teams to be more rigorous and focussed when it comes to deciding what to work on, why to work on those things and really importantly, how we are going to measure the impact.

Any advice for optimizers looking to work with product teams?

Go for it. If you want something, go and get it. Join new groups online (I’ve done this through LinkedIn), shadow your colleagues, go to ProductTank meetups or similar and get out there and network. It’s a lot of fun when you really get it all moving.

Talk to us about the unique experiments you’ve run over the years. 

The best experiments I’ve been involved in over the years have been the ones where new functionality has been released via the testing platforms. These ‘controlled releases’ have had the power to really unite the teams I’ve worked in around a common priority and desire to deliver a positive outcome for our business, as well as our customers.

These releases are the culmination of hours of work comprising data analytics, customer research, user testing, and tech development time. There has been significant investment of time and effort to build the new shippable product.

Releasing through the controlled environment of the testing platform provides three very important outcomes:

1. A control group to measure the new data against while limiting the exposure to a controlled percentage of traffic
2. Focus on truly measuring the impact of the newly shipped product/functionality/feature
3. True collaboration between the diverse functions/stakeholders in the team

The last point about collaboration has been the overwhelming benefit from introducing this ‘controlled release’ process. The teams are excited to see the data flow and watch as their work comes to life. This period of ‘hyper care and attention’ really gels the squads to maximise the release potential and focus on making the new product/functionality/feature the ‘best it can be’. Even when the newly shipped product/functionality/feature is not performing vs the control, the teams have been energised and focussed, using the data to understand and pin-point where the problems are or might be and provide solutions quickly during this period of ‘hyper care and attention’.

Cheers for reading! If you’ve caught the CRO bug… you’re in good company here. Be sure to check back often, we have fresh interviews dropping twice a month.

And if you’re in the mood for a binge read, have a gander at our earlier interviews with Gursimran Gujral, Haley Carpenter, Rishi Rawat, Sina Fak, Eden Bidani, Jakub Linowski, Shiva Manjunath, Deborah O’Malley, Andra Baragan, Rich Page, Ruben de Boer, Abi Hough, Alex Birkett, John Ostrowski, Ryan Levander, Ryan Thomas, Bhavik Patel, Siobhan Solberg, Tim Mehta, Rommil Santiago, Steph Le Prevost, Nils Koppelmann, Danielle Schwolow, Kevin Szpak, Marianne Stjernvall, Christoph Böcker, Max Bradley, Samuel Hess, Riccardo Vandra, Lukas Petrauskas, Gabriela Florea, Sean Clanchy, Ryan Webb, Tracy Laranjo, Lucia van den Brink, LeAnn Reyes, Lucrezia Platé, Daniel Jones, May Chin, Kyle Hearnshaw, Gerda Vogt-Thomas, Melanie Kyrklund, Sahil Patel, Lucas Vos, David Sanchez del Real, Oliver Kenyon, David Stepien, Maria Luiza de Lange, Callum Dreniw, Shirley Lee, Rúben Marinheiro, Lorik Mullaademi, Sergio Simarro Villalba, Georgiana Hunter-Cozens, Asmir Muminovic, Edd Saunders, Marc Uitterhoeve, Zander Aycock, Eduardo Marconi Pinheiro Lima, Linda Bustos, Marouscha Dorenbos, Cristina Molina, Tim Donets, Jarrah Hemmant, Cristina Giorgetti, Tom van den Berg, Tyler Hudson, Oliver West, Brian Poe, Carlos Trujillo, Eddie Aguilar, Matt Tilling, Jake Sapirstein, Nils Stotz, Hannah Davis, Jon Crowder, Mike Fawcett, Greg Wendel, Sadie Neve, Cristina McGuire, Richard Joe, Ruud van der Veer, Merritt Aho, Felipe Henrique Fogarolli, Riccardo Oricchio, and Bruno Borges.