We solemnly promise that each bit of content you read here
is thoughtfully curated or crafted by a human, for a human. We spend hundreds of hours every month speaking to experts with decades of experience. Our blogs are more a journalistic report on the state of experimentation than a single perspective article. Enjoy!
We all want to make good data-driven decisions from our experiments. After all, that’s why we run experiments in the first place. But how many of us actually feel confident navigating the complexities of p-values and standard deviations to make truly well-informed and confident choices from our test results? If...
Strategically combining targeting and triggering in your A/B tests produces reliable results by focusing tests on the right audience at the right time. Key points: Targeting controls who is included in the experiment while triggering defines when the variation is applied during the user's experience. Targeting narrows the audience based...
If you’re planning to halt your A/B testing program for the holidays, you may be missing the most wonderful time of the year to capture customer insights. While it’s true – the holiday season is very different from rest-of-year, and shifts in traffic and customer behavior can pollute your existing...
The Minimum Detectable Effect (MDE) is the smallest true effect an A/B test can detect with sufficient statistical power, factoring in test duration, sample size and practical significance. There's a big tradeoff between MDE and test duration. If your MDE is too high, you risk missing real improvements, but if...
This article explores ten mental models driving successful experimentation, drawing on insights from 64 leading CROs. Key points: Mental models, like the scientific method, feedback loops, and probabilistic thinking, are essential parts of the experimenter’s toolkit. The importance of a scientific approach to learning, including well-crafted hypotheses, is universally acknowledged...
The "messy middle" of A/B testing—when hypotheses have to be given a form so they go live as a treatment or intervention—demands a blend of data-driven insight and creative problem-solving. You may recall our exploration of triangulation: combining quantitative, qualitative, and visual data to uncover what users do and why...
