Testing Mind Map Series: How to Think Like a CRO Pro (Part 74)

Interview with Nils Stotz

There’s the experimentation everyone talks about. And then there’s how it actually happens.

We’re hunting for signals in the noise to bring you conversations with people who live in the data. The ones who obsess over test design and know how to get buy-in when the numbers aren’t pretty.

They’ve built systems that scale. Weathered the failed tests. Convinced the unconvincible stakeholders.

And now they’re here: opening up their playbooks and sharing the good stuff.

This week, we’re chatting with Nils Stotz, Product & Experimentation leader at Zalando.

Nils, tell us about yourself. What inspired you to get into testing & optimization?

I’m Nils and I lead the experimentation platform team at Zalando. I love talking about experimentation and sharing content about it on LinkedIn. When I was working at a very small startup, we used a CRM tool that allowed A/B testing. This was my first exposure to the concept because it resonated with me so much, and it was so easy to set it up. From then on, I’ve tried to test everything and work in a super data-based way.

How many years have you been testing for?

Around 8 years.

What’s the one resource you recommend to aspiring testers & optimizers?

Follow me on LinkedIn haha. The knowledge is quite scattered, but generally, I would recommend ‘Testing Business Ideas’ by Alex Osterwalder, ‘Experimentation Works’ by Stefan Thomke. The material from Speero is also really excellent!

Answer in 5 words or less: What is the discipline of optimization to you?

Less stressful ways of working!

What are the top 3 things people MUST understand before they start optimizing?

• It is not bad to be wrong—it leads you closer to your goal and is part of the game.
• The quantity of experiments will eventually lead to an increased quality of experiments.
• Understanding what it means when a test is statistically significant and when it’s not.

How do you treat qualitative & quantitative data to minimize bias?

I approach both as methods to learn and/or minimize risk. In my opinion, the skill that PMs must master in the mid and long term is finding a method-problem fit. This means fully understanding the toolkit available to solve problems both quantitatively and qualitatively.

How (to you) is experimentation different from CRO?

That is a very deep question, but I think CRO is only one part of experimentation with a very explicit intention. There are many more reasons to conduct experiments beyond just trying to increase a conversion rate.

But I want to emphasize that both areas can learn from each other and should exchange more. While in-house experimentation teams have a lot of in-depth knowledge about particular cultural challenges and domains in their work, only CRO agencies can really provide a holistic view because they see a lot more different companies.

Talk to us about some of the unique experiments you’ve run over the years.

My favorite experiments are actually those where you remove something and observe the effect. I have executed many experiments where this led to very significant improvements in conversion rates. A nicer way to think about it is not really just one specific experiment, but rather an experiment roadmap that is really aiming to validate various aspects of a product and is able to produce multiple similar experiments for exactly that.

Cheers for reading! If you’ve caught the CRO bug… you’re in good company here. Be sure to check back often, we have fresh interviews dropping twice a month.

And if you’re in the mood for a binge read, have a gander at our earlier interviews with Gursimran Gujral, Haley Carpenter, Rishi Rawat, Sina Fak, Eden Bidani, Jakub Linowski, Shiva Manjunath, Deborah O’Malley, Andra Baragan, Rich Page, Ruben de Boer, Abi Hough, Alex Birkett, John Ostrowski, Ryan Levander, Ryan Thomas, Bhavik Patel, Siobhan Solberg, Tim Mehta, Rommil Santiago, Steph Le Prevost, Nils Koppelmann, Danielle Schwolow, Kevin Szpak, Marianne Stjernvall, Christoph Böcker, Max Bradley, Samuel Hess, Riccardo Vandra, Lukas Petrauskas, Gabriela Florea, Sean Clanchy, Ryan Webb, Tracy Laranjo, Lucia van den Brink, LeAnn Reyes, Lucrezia Platé, Daniel Jones, May Chin, Kyle Hearnshaw, Gerda Vogt-Thomas, Melanie Kyrklund, Sahil Patel, Lucas Vos, David Sanchez del Real, Oliver Kenyon, David Stepien, Maria Luiza de Lange, Callum Dreniw, Shirley Lee, Rúben Marinheiro, Lorik Mullaademi, Sergio Simarro Villalba, Georgiana Hunter-Cozens, Asmir Muminovic, Edd Saunders, Marc Uitterhoeve, Zander Aycock, Eduardo Marconi Pinheiro Lima, Linda Bustos, Marouscha Dorenbos, Cristina Molina, Tim Donets, Jarrah Hemmant, Cristina Giorgetti, Tom van den Berg, Tyler Hudson, Oliver West, Brian Poe, Carlos Trujillo, Eddie Aguilar, and Matt Tilling, Jake Sapirstein.