Naughty or Nice? The Experimentation Community Reflects, Confesses & Pays It Forward (2024 Edition)

Experimentation is a journey: equal parts thrilling, humbling, and unpredictable. Behind every successful test is a story of ingenuity and collaboration. Behind every misstep is a lesson learned the hard way. And behind it all are the people who dedicate themselves to improving, optimising, and moving things forward.

As the year drew to a close, it felt right to pause and reflect on what 2024 has brought us in the world of experimentation. I reached out to the community with three questions: 

What are you proud of? What didn’t go quite to plan? And who made a difference in your year? 

The answers I received were honest, thoughtful, and often surprising.

Some respondents shared test wins that went beyond expectations, from redesigns that transformed user experiences to quiet changes with outsized impact. Others were candid about mistakes that led to late-night scrambles, tough lessons, or just a good laugh. And many took the time to celebrate the teammates, mentors, and friends who showed up when it mattered most.

In this end-of-year blog, we’ll revisit the highlights of 2024 by making our list: 

• the “nice” moments that made us proud, 
• the “naughty” ones that taught us something valuable, and 
• the nominations that remind us of the incredible people driving experimentation forward.

The Nice List: Wins That Made 2024 Shine

Experimentation isn’t just about running tests, it’s about making an impact, whether through data-driven decisions, collaboration, or tackling personal challenges. This year’s “nice” stories reflect the resilience, creativity, and growth that define the experimentation community. Here’s a look at 2024’s proudest moments.

1. Transformative Test Wins

When experimentation works, it doesn’t just validate ideas, it drives real change.

• One respondent shared a massive win: improving the accessibility of their product, which significantly increased conversions, reduced legal risk, and retained a major client. This wasn’t just a win in numbers, it demonstrated how experimentation can positively impact users, businesses, and broader social outcomes​. As more people spoke about accessibility this year, in terms of the ethical responsibility of accessibility as well as the legal and business cases, we thought this one was important to highlight. 
• Deborah O’Malley highlighted an experiment from the GuessTheTest Best In Test Awards, which challenged the assumption that add-ons always boost conversions. Instead, the test found that simplifying the funnel improved results, saving the client from implementing a losing variant. This is a textbook example of why testing matters, it challenges assumptions and drives smarter decisions.

Transformative wins like these go beyond surface metrics. They shift strategies and show the importance of asking (and testing!) bold questions.

2. Collaboration and Courage

Experimentation thrives on collaboration, and for many, 2024 was about harnessing the collective power of their teams while pushing the boundaries of individual growth.

• Georgiana Hunter-Cozens celebrated a strong partnership with her UX team at Creative CX that led to impactful test wins for a travel client. “We combined data with user testing to deliver more robust insights and recommendations,” she shared. This collaboration between analytics and design shows how cross-functional teams can unlock powerful outcomes—and underscores the importance of robust user testing as part of the experimentation process.
• But Georgiana’s year wasn’t just about teamwork. She stepped outside her comfort zone to take on new challenges, putting herself forward as a speaker. “I delivered my first talk at ExElite, followed by webinars, and joined the WiE x Convert Speak Up competition. It’s been terrifying but so rewarding,” she said. We look forward to seeing more talks and posts from Georgiana this coming year. 
• Aly Balmes also took a leap of faith, reaching out to the CEO of an automation company about their CRO efforts. “Not only did they respond, but we launched two tests in the first month,” they said. This contribution from Aly reminded me of a discussion recently in the Women in Experimentation Slack where several members discussed how they often report bugs to websites they buy from – maybe we will get a response one day!

Whether it’s building team collaboration or stepping into the spotlight, the rewards of 2024 extended far beyond the tests themselves.

3. The Quiet Wins That Matter Most

Not every win is in the spotlight and it can be the quieter successes that make someone’s year.

Craig Sullivan reflected on the importance of lifting others up and creating opportunities for your peers:

I recommended several people for roles this year quietly and without expecting anything in return. It costs nothing, but it can make a real difference. It’s a gift you should give as often as you can, especially in a tough year for many in the experimentation industry.

This act of “paying it forward” is something anyone can adopt. Take a moment to think about the talented people in your network: Is there someone whose skills deserve recognition? A colleague who might be perfect for an opportunity you’ve heard about? 

Even a small gesture like sharing someone’s work, nominating them for an award, or offering a recommendation can have a meaningful impact. And, as Craig points out, it’s about doing a nice thing when no one’s looking and trusting that the ripple effects will inspire others to do the same.

The Naughty List: Lessons Learned the Hard Way

Mistakes are inevitable in experimentation: sometimes they’re frustrating, sometimes they’re funny, but they’re always a chance to learn. 

This year’s “naughty” stories reveal the moments that didn’t go to plan and, more importantly, how to avoid them in the future.

1. Testing Too Soon and Sharing Too Early

One respondent admitted to celebrating prematurely:

It was my very first experiment, and maybe it’s just me, but I checked the analytics every day, haha. When I saw the bounce rate drop after a week, I practically shouted the news to the team like I’d just discovered fire. Then… one week later, the bounce rate bounced right back to its old number

Early data can be exciting, but results often fluctuate until a test reaches statistical significance. To avoid jumping the gun, set clear thresholds for when results are ready to share and communicate these expectations with your team. Consider using tools that flag incomplete data or remind you when it’s safe to draw conclusions. Patience ensures that decisions are built on solid evidence, not momentary excitement.

2. A Lesson in Patience and Perspective

Another respondent described jumping to conclusions based on initial data:

The early data looked amazing, and I jumped to conclusions, sending out a celebratory message to the team. A week later, when the experiment finished, the results had completely flipped—what seemed like a clear winner had turned into a false positive due to sample bias.

Sample bias and incomplete data can lead to misleading conclusions. Build protocols into your testing process to flag data anomalies early. Waiting for statistical significance ensures you’re acting on reliable insights, not premature trends. And finally, if you have to share early results, always include caveats about the test’s current status to manage expectations.

3. Overconfidence and Costly Rollouts

Iqbal Ali shared a misstep with far-reaching consequences:

I became overly confident with the rollout of test-triggering functionality, and it took two months to catch the issue. It invalidated at least a dozen tests.

The impact was compounded by the time and resources spent on experiments that could no longer be trusted​ – oops. Thank you for not going anonymous, Iqbal! 

Overconfidence often leads to missed details. Introducing robust quality assurance (QA) processes can catch errors before they escalate. Iqbal’s experience also shows us the importance of ongoing monitoring after rollout to identify problems early. Regularly reviewing your testing infrastructure and running small-scale pilots before full deployment can save significant time and effort.

4. Revisiting Old Work with New Eyes

Craig Sullivan reflected on a mistake from “many Christmasses ago” early in his experimentation career:

I found an old slide deck from 2007. Looking at it now, I realise it was full of statistical errors and flawed advice, which I shared with others at the time. It’s humbling to think about how wrong I was, but also a reminder of how far I’ve come.

Craig’s reflection also carried a valuable lesson:

Everyone starts out with particular ideas about A/B testing, and over time, biases and assumptions get ironed out as you keep exploring knowledge beyond your comfort zone

Revisiting past work isn’t just about self-critique, it’s also an opportunity to learn (and cringe!). Craig’s story underscores the importance of staying curious and committed to professional development. Engage with new research, collaborate with peers, and challenge outdated practices. Being too sure that you know everything will stop you from being open to learning about what you don’t. 

These “naughty” moments may have caused a few red faces, but they also brought invaluable lessons. Mistakes are part of the journey, but by implementing better processes, embracing humility, and learning from these experiences, we can turn missteps into meaningful progress. After all, as one respondent aptly put it, “The only real failure is the one you don’t learn from.”

Nominations: The People Who Made a Difference in 2024

These nominations were gathered as part of our end-of-year survey, where we asked the community who made a meaningful difference in their work this year. The responses celebrated problem-solvers, mentors, and unsung heroes whose contributions had a lasting impact.

1. Problem Solvers and Innovators

These nominees stood out for their ability to solve tricky challenges and make experimentation better for everyone.

• Iqbal Ali nominated Sani (Slobodan Manic): “I’m forever bugging him about development-related stuff on Ressada, and he’s always incredibly patient, answering the weirdest of questions with a straight face”. Sani is a well loved part of our community and runs one of the best experimentation podcasts out there “No Hacks” – check it out to hear from his impressive roster of guests.
• Aly Balmes nominated Silvia Guardingo de la Riva: “She handed me cheat codes for understanding cookies and CRO dev concepts. I feel like I gained level 10 skills when I was still at level 3”. A perfect example of paying it forward and sharing knowledge with those newer to the industry than you!

2. Mentors and Coaches

These individuals made a difference by guiding, teaching, and elevating others.

• Sanne Maach Abrahamsson nominated Nicolai Ahle Bjerg: “Nicolai Ahle Bjerg is a Senior CRO Specialist on our team at Truelinked. He’s incredibly talented, passionate, and innovative, but he’s also an amazing person who taught me and the team so much this year.” Nicolai’s mentorship has clearly left a lasting impression​!
• Patrizia Giampa nominated Linda Bustos: “My favorite industry interaction this year was with a long-time peer, Linda Bustos. Way back when, Linda gave me some of my first exposure in the industry by publishing an article I wrote on her blog, Get Elastic. I was ecstatic at the exposure. This year, we reconnected on LinkedIn, shared funny stories, groaned at bad puns, and even did a couple of webinars together, including one with Convert. Linda is a true OG in the experimentation world, and I’m grateful for her insights and contributions through her Ecom Ideas posts”. I’m sure we have all seen Linda’s wonderful Ecom Ideas posts but if not you can find them here.
• Kelly Wortham nominated Georgi Georgiev: “Georgi Georgiev is an amazing teacher. Ironically, when I was having a hard time understanding a concept, he asked me to pre-read one of his upcoming blog posts. It helped me better understand that concept, and then, because he’s incredibly kind, he thanked me when he published it. Thank you, Georgi. Keep teaching and always be awesome. I look forward to the next collaboration.” Georgi’s teaching and expertise have been a big part of our community and many see him as the authority on statistics in optimisation – you can read his great content on medium and buy his book here.

Stephen Situ mentioned Marcella Sullivan: “Marcella plugged me in for two talks this year. Legend!”. Thank you Stephen for the nomination – I was there to see how amazing you were for both talks! The first was in summer at Experimentation Elite in Birmingham, the TLC summit Stephen spoke at returns in Summer 2025 and the other was at the December London Experimentation Meetup hosted by Creative CX.

3. Builders of Community and Opportunity

These nominees went above and beyond to create opportunities and foster connections within the community.

• Georgiana Hunter-Cozens highlighted Women in Experimentation (WiE): “WiE has encouraged me to speak up and get more involved, with incredible support from people like Trina from Convert, Marina Ostrowski, and Kelly Wortham”. Watch this space for lots WiE events in the coming year – and if you are a woman in the industry, join the Slack channel here! 
• Craig Sullivan nominated Jacob Funnel: “Jacob encouraged me to share my Looker Studio dashboard more broadly. He suggested I turn it into a short course so others could use it practically. His encouragement inspired me to rethink how I share knowledge, and it’s made a big difference.” Jacob’s a skilled web analytics consultant and you can see more about what he does on his LinkedIn.

Kelly Wortham also celebrated Ton Wesseling and the Conversion Hotel crew: “I’m so proud of the launch of Experimentation Island. It’s been a lot of work (fun work) but I can’t wait for folks to experience the culture and collaboration we’ve built. This is about more than the content—it’s about creating a space where experimenters can connect and grow together”. You can find more about Experimentation Island here. Kelly, Ton and team are balancing out the EU skew of experimentation events by giving their conference a US home!

4. The Unsung Heroes

Some contributions may fly under the radar, but their impact has been significant.

• Deborah O’Malley highlighted Linda Bustos (again!): “Linda is a true OG in the experimentation world. Her insights and webinars have shaped how I approach eCommerce. I’m so grateful for her contributions.” Linda’s double nomination just goes to show her impact!

Erin Weigel nominated Lucas Bernardi: “Lucas has always been there for me to answer weird and random questions about data analysis and statistics. He’s a hidden gem.” Quiet, consistent support like Lucas’s often goes unnoticed but makes all the difference​. You can check out Erin and Lucas’ joint course with TLC here.

Conclusion

2024 has been a year of creativity, learning, and persistence in experimentation. From the proud moments to the lessons learned from missteps, these stories have shown the many ways experimenters continue to push boundaries. At its core, experimentation is as much about people as it is about processes.

The nominations are a testament to the community’s generosity and spirit. They celebrate those who solve complex problems, guide others through challenges, and quietly support their peers. These contributions remind us that success is not just about results but also about the connections we build and the influence we have on each other’s growth.

As we look ahead to 2025, there is every reason to be optimistic. Experimentation will continue to evolve, shaped by the creativity and commitment of those who work to improve it. Let’s carry forward the lessons of this year, celebrate our wins, and embrace the challenges yet to come. 

Here’s to another year of experimenting, learning from each other, and achieving great things!