Blog /Optimization

Testing Mind Map Series: How to Think Like a CRO Pro (Part 11)

18th Mar 2022 – Interview with Ruben de Boer

Interview with Ruben de Boer

Are you someone who is looking to get more out of experimentation?

Our guest today has been doing this for years and can help you get started on your own experimentation journey. Plus, he has some great advice on how to make sure your experiments give you the right ROI.

Once you have all that figured out, you’ll want to keep track of all the learnings so you can apply them across your company. Our expert has tons of advice on how to do just that.

Let’s dive in.

Ruben, tell us about yourself. What inspired you to get into testing & optimization?

In 2009, I helped a company that taught people to become happier and more confident with improved social skills. At that time, it really struck me the importance of supporting and inspiring others through what you love to do. I also became fascinated with the question:

If we can change people’s behavior offline, how can we do this online?

This is how I got into the research of human behavior and psychology. Data has always been my thing. I enjoy finding patterns and solutions in data (at school, I was much better at data topics than any other topic).

And I love optimization and getting results. Not just in my work but also in my personal life.

This combination of interests and passions naturally pushed me to work in testing & optimization.

How many years have you been optimizing for? What’s the one resource you recommend to aspiring testers & optimizers?

In 2009, I started optimizing websites and analyzing business and online data. It wasn’t until 2011 that I did my first A/B test. Of course, I made all the mistakes one can make when starting with Conversion Rate Optimization.

My number one resource is meeting other specialists in this field. Share ideas and experiences and learn from each other. At events, don’t just get ideas from the speakers, but also people in the audience. By helping each other and learning together, we can bring ourselves and our work to the next level.

Answer in 5 words or less: What is the discipline of optimization to you?

Continuous improvement mindset, learning, and experimentation.

What are the top 3 things people MUST understand before they start optimizing?

First, get the basics right. Specialists will often run experiments based on common practices. These practices, however, have been tested on different websites (if tested at all) and might not work for your website or (digital) product at all.

Get your test ideas from your data, user, and scientific research. And when you complete an experiment, think about what you learned and what new test ideas you can come up with based on those learnings. These sources will help you craft solid test ideas with a greater chance of finding winning variations.

Another basic proceeding, where I see many specialists go wrong, is statistics. For every experiment, do your pre-test analysis. Calculate the minimum sample size and test duration, and only stop the test after you reached those.

If you find out that you have insufficient traffic for A/B testing in your pre-test calculations, find other means to validate your test ideas, for instance, through usability testing.

Second, the process is key. Without a proper process in place, you will not be successful. To optimize your process, you have to understand the Return of Investment (ROI) of experimentation:

ROI = (Quantity * Quality) / Costs

Quantity is the number of experiments you run. Quality is the win rate and value per winner. Costs are time and money, so your efficiency.

With the right process in place (including a tool to keep track of your process, progress, and insights!), you can optimize your ROI and experimentation success.

And third, understand that your job is not just to run A/B tests. A large proportion of your work should be dedicated to change management. To thrive and work towards an experimentation culture, A/B testing is not enough. You need to change the culture, mindset, and the way your entire organization works. Start by involving people, especially higher management.

How do you treat qualitative & quantitative data so it can tell you an unbiased story?

In a nutshell:

The first step is to conduct thorough research using various resources, like web data, customer service, surveys, polls, and scientific articles.

The second step in the process is to turn the insights of your research into solid overarching hypotheses. Aim for five to ten hypotheses that state something about your (potential) customers, not related to a single element or web page.

For example:

  • Customers find our products too expensive
  • People need more information about our service before signing up
  • People become our customers as they have the need to belong

You can then validate these hypotheses through several related experiments, also based on your research. This is called a meta-analysis.

Why? A single A/B test is prone to errors. Your experiment could result in a false positive, or the reason for a winning experiment can be different than the one stated in your hypothesis. With a meta-analysis, we get closest to the truth. Therefore, our insights are more reliable and we create better follow-up experiments.

The meta-analysis is the highest quality of proof you can get, based on the hierarchy of evidence.

Hierarchy of Evidence

The final step in the process involves keeping track of which overarching hypothesis works and where in the customer journey (on which page), and prioritizing your test ideas based on that, to maximize your win rate and learnings.

What kind of learning program have you set up for your optimization team? And why did you take this specific approach?

The process laid out above will help you gather reliable insights from which you and your organization can really learn.

When you document those meta-analyses well (in a tool like Airtable), it becomes easier to see where your success lies. By tagging every experiment with the overarching hypothesis and psychological direction, you get an incredible amount of knowledge on your potential customers’ needs (and problems), while also knowing where in the customer journey to address them, and through what type of experiments.

Once you have done a lot of experiments and start getting a clear picture, you can share these insights and the whole organization can learn and benefit.

What is the most annoying optimization myth you wish would go away?

I will mention two 🙂

The incredible uplifts experimentation can accomplish (with a single A/B test). It is mainly some vendors to blame for this, as in their cases and on their landing pages, they advertise with huge uplifts, resulting in false expectations.

Organizations that use experimentation for better decision-making are often the ones that thrive. Not the ones who focus on money solely.

And second, the name Conversion Rate Optimization in general. It gives the wrong impression and is too narrow for what specialists in experimentation can accomplish.

Ruben de Boer Infographic

Sometimes, finding the right test to run next can feel like a difficult task. Download the infographic above to use when inspiration becomes hard to find!

Hopefully, our interview with Ruben will help guide your experimentation strategy in the right direction! What advice resonated most with you?

Be sure to stay tuned for our next interview with a CRO expert who takes us through even more advanced strategies! And if you haven’t already, check out our interviews with Gursimran Gujral, Haley Carpenter, Rishi Rawat, Sina Fak, Eden Bidani, Jakub Linowski, Shiva ManjunathAndra Baragan, and our latest with Rich Page.

CRO Master
CRO Master
Originally published March 18, 2022 - Updated November 11, 2022

Start Your 15-Day Free Trial Right Now.
No Credit Card Required

You can always change your preferences later.
You're Almost Done.
Managing Marketing Team
Managing Tech Team
Hypothesizing Experiments
Coding and QA of Experiments
Convert is committed to protecting your privacy.

Important. Please Read.

  • Check your inbox for the password to Convert’s trial account.
  • Log in using the link provided in that email.

This sign up flow is built for maximum security. You’re worth it!