Testing Mind Map Series: How to Think Like a CRO Pro, Part 95 with Craig Kistler

Craig Kistler, who leads experimentation at Signet Jewelers, is the latest expert to open up their playbook for our Testing Mind Map series. These interviews look past the tidy version of experimentation and into the day-to-day: how evidence fares against conviction, which test wins survive contact with production, and where judgment sits now that AI has commoditized execution.

Craig, tell us about yourself. What inspired you to get into testing & optimization?

I kind of fell into testing and optimization by chance. My background was UX. I had built a number of UX teams and worked with some pretty large brands, either building or helping establish their UX practice.

Then I went to Signet as the first and only UX designer and started building the UX practice there.

Around that time, another team brought in an A/B testing tool and started experimenting with some of the ideas coming out of my UX work. That was really the moment it clicked for me.

Up until then, we could run usability studies, watch people struggle, ask them questions, and come away with a pretty good idea of what needed to change. Experimentation gave me a way to take those ideas and put them in front of real shoppers at scale to see what they actually did.

UX research helped us find the problem. Experimentation helped us see whether we actually solved it.

We started seeing success pretty quickly, both in business results and in what we were learning about customer behavior. From there, the practice just started snowballing.

Answer in 5 words or less: What is the discipline of optimization to you?

Using experimentation to optimize experiences.

How is AI influencing experimentation for you? How do you think about incorporating AI in your workflows?

AI is definitely making it easier to explore and iterate on ideas, but I don’t think that’s where the real challenge in experimentation has ever been.

The harder part is still judgment. What is worth changing? Why do we think it will help? Who is the change actually for? What should we measure? And once we have the result, what do we do with it?

That’s where I’ve found AI most useful, not as a replacement for thinking, but as a way to think more effectively, surface patterns faster, and reduce the busy work so I can spend more time focusing on the actual insights and decisions.

I use it as a thinking partner throughout the process. I’ve used it to take analytics outputs and turn them into a more structured analysis, summarize and organize experiment learnings, challenge hypotheses, troubleshoot code, and help pull together information that might otherwise sit across a bunch of different tools.

One of the areas I’ve probably found most interesting is rapid prototyping. I can take an idea that would normally require a lot of explanation, build a working version pretty quickly, and put it in front of users before asking a development team to invest in it. That does not tell me whether the idea will perform in an experiment, but it can tell me pretty quickly whether people understand it and whether there is something worth pursuing.

I think about the overall process as Problem, Bet, Evidence, Extension.

What customer problem are we trying to solve?

What are we betting will make that experience better?

What did the behavior and the data actually show us?

And then what do we do with it? Roll it out, revise it, personalize it, test another version, or just walk away?

AI can help at every stage of that process. What I would not do is use it as an experiment factory. If the underlying problem is weak, AI just helps you produce more bad ideas faster.

That is the part I think teams need to be careful with. AI is reducing the cost of execution. It is not reducing the need for judgment.

Where is personalization headed? Has AI driven its resurgence? 

I’ve been focused on personalization for more than eight years, so from my standpoint, AI did not create the need for personalization. But I do think it has helped drive some of the renewed attention around it.

Part of that is practical. AI is making it easier to analyze more data, identify patterns, and act on signals faster. And part of it, honestly, is that AI and personalization have become selling points for just about every vendor. I’m not saying that is necessarily a good thing, but it has definitely brought personalization back into a lot of conversations.

For me, the high-level strategy really has not changed that much. I’ve never been overly focused on the idea of one-to-one personalization. I tend to think about larger groups of people with similar needs or intent. For example, what is this visitor trying to do, what do they need from the experience right now, and how can we help them move forward?

What is changing is how much smarter we can get about those groups. We have access to more signals and newer tools that can help us understand what someone may need while they are actually interacting with a page or moving through the site. I often say it takes those basic segments and puts them on steroids. The segment gives you a starting point, but now you can understand much more about what is happening within that segment in the moment.

That is where I think this gets really interesting.

Most ecommerce sites are still essentially one-size-fits-all. We design a PDP or category page for everyone, which means the page ends up carrying every message that might possibly matter to somebody.

I think that starts to go away.

Instead of designing one static page and trying to make it work for everyone, the page itself should become dynamic. What information gets emphasized, what guidance appears, what content comes next, even how the page is assembled could change based on what that visitor needs at that moment.

AI can help make that practical because it gives us a better ability to understand signals and respond to them quickly. So I don’t see the future of personalization as building a completely unique website for every individual. I see it as getting much more accurate about the visitor’s needs and letting the experience adapt around them.

The page should change because the shopper’s needs change.

Talk to us about the unique experiments/personalizations you’ve run over the years.

Some of my favorite experiments are the ones that challenged something everyone assumed was true.

One example was around friction. The conventional wisdom is fewer clicks, fewer steps, get people to the product as quickly as possible. But we saw people land on category pages with thousands of products, scroll through them, and leave. The “frictionless” experience was not actually helping them.

So we did the opposite. We put something in their way.

We asked them what they were looking for before showing them products and used those answers to narrow the selection. We added a click, but gave them direction. It worked, and it reinforced something I still believe: guidance is not the same thing as friction. Sometimes an extra step (friction) makes the experience easier.

Another experiment started with something almost every ecommerce site does: immediately asking a new visitor for their email address in exchange for a discount.

I always thought that was a strange value exchange. You just arrived. You may not even know whether you like the brand yet, and the first thing we do is ask you for something.

We flipped this on its head and instead offered help.

We asked what they were shopping for and used their answer to create a more relevant experience as they moved through the site. Rather than saying, “Give me your email,” the experience essentially said, “Tell me what you need and I’ll make this more useful.”

That performed very well for us and became a much bigger lesson around personalization. Sometimes the best way to learn about intent is simply to ask, as long as the customer gets something useful in return.

Then there are experiments where the lesson takes a lot longer.

We had an idea internally that people felt very strongly about. It was something competitors were doing, so there was a belief that we needed it too. We tested it and it did not work.

But the belief was strong enough that one losing experiment was not going to settle the question.

So we changed the execution. We changed the placement. We tried connecting it to different parts of the shopping journey. We kept asking whether the idea was right but we had just implemented it in the wrong way.

Eventually, after several iterations, we had enough evidence to say this was not an execution problem. It just was not solving a meaningful customer problem for us.

I actually think that is an important part of experimentation that gets overlooked. Sometimes you need patience not to prove an idea works, but to give a strongly held belief enough opportunities to fail that people are finally willing to let it go.

Those are usually the experiments I find most valuable. Not because they produced a winning variation, but because they changed how we thought about the customer.

Last but not least, AI taking over repetitive tasks and simplifying execution. How has that changed the way you describe work? 

AI is definitely making a lot of the work faster. Things that used to take hours can take minutes. Prototypes can be built faster. Analysis can happen faster. Ideas are easier to explore. I think all of that is useful.

But I don’t think AI solves the harder part of the work.

You still need to understand your customers. What are they trying to accomplish? Where are they struggling? Why are they behaving the way they are? What does the experience actually need to do for them?

My concern is that as execution gets easier, teams start optimizing for speed. How many experiments can we launch? How many variations can we create? How much content can we produce?

That can become a distraction.

If you don’t have a solid understanding of the customer problem, AI just gives you the ability to move in the wrong direction faster.

So I think the work shifts. There is less value in simply being able to produce the thing and more value in knowing which thing is worth producing in the first place.

That means understanding customer behavior, identifying the right problems, making good bets, and then interpreting what you learn. AI can help with all of those things, but the team still needs to understand the experience they are trying to create.

For me, the goal isn’t to use AI to do more work. It’s to use AI to remove the repetitive work so we can spend more time on the parts that actually require judgment.

Speed is useful. But not if we lose sight of the experience we are trying to improve.

Cheers for reading! If you’ve caught the CRO bug… you’re in good company here. Be sure to check back often, we have fresh interviews dropping twice a month. And if you’re in the mood for a binge read, have a gander at our earlier interviews with Gursimran Gujral, Haley Carpenter, Rishi Rawat, Sina Fak, Eden Bidani, Jakub Linowski, Shiva Manjunath, Deborah O’Malley, Andra Baragan, Rich Page, Ruben de Boer, Abi Hough, Alex Birkett, John Ostrowski, Ryan Levander, Ryan Thomas, Bhavik Patel, Siobhan Solberg, Tim Mehta, Rommil Santiago, Steph Le Prevost, Nils Koppelmann, Danielle Schwolow, Kevin Szpak, Marianne Stjernvall, Christoph Böcker, Max Bradley, Samuel Hess, Riccardo Vandra, Lukas Petrauskas, Gabriela Florea, Sean Clanchy, Ryan Webb, Tracy Laranjo, Lucia van den Brink, LeAnn Reyes, Lucrezia Platé, Daniel Jones, May Chin, Kyle Hearnshaw, Gerda Vogt-Thomas, Melanie Kyrklund, Sahil Patel, Lucas Vos, David Sanchez del Real, Oliver Kenyon, David Stepien, Maria Luiza de Lange, Callum Dreniw, Shirley Lee, Rúben Marinheiro, Lorik Mullaademi, Sergio Simarro Villalba, Georgiana Hunter-Cozens, Asmir Muminovic, Edd Saunders, Marc Uitterhoeve, Zander Aycock, Eduardo Marconi Pinheiro Lima, Linda Bustos, Marouscha Dorenbos, Cristina Molina, Tim Donets, Jarrah Hemmant, Cristina Giorgetti, Tom van den Berg, Tyler Hudson, Oliver West, Brian Poe, Carlos Trujillo, Eddie Aguilar, Matt Tilling, Jake Sapirstein, Nils Stotz, Hannah Davis, Jon Crowder, Mike Fawcett, Greg Wendel, Sadie Neve, Cristina McGuire, Richard Joe, Ruud van der Veer, Merritt Aho, Felipe Henrique Fogarolli, Riccardo Oricchio, Bruno Borges, Daniel Mullins, Matthew Bass, Pieter Boonstra, Simbar Dube, Dzifa Mensah, Katie Faulkner, Andrea Bronzini, and Sanne Maach Abrahamsson.