We solemnly promise that each bit of content you read here
is thoughtfully curated or crafted by a human, for a human. We spend hundreds of hours every month speaking to experts with decades of experience. Our blogs are more a journalistic report on the state of experimentation than a single perspective article. Enjoy!
Frequentist and Bayesian are the two main approaches A/B testing tools use to crunch results. One uses a fixed, pre-determined sample size and p-values. The other gives probabilities like “there’s an 85% chance this version is better.” Which should you trust? It depends on your traffic, test design, and the...
Synopsis: There are 3 major conversion rate optimization (CRO) agency pricing models: Hourly project model Monthly subscription retainer model Variable incentive model Within the monthly subscription retainer model, the usual CRO agency pricing per month is: $2,000$16,000$30,000LowAveragePremium Within the hourly project model, the usual costs for the services of a...
Hard truth: Outsourcing your A/B testing does not guarantee growth. We wish it were that simple. Even with world-class experts, the wrong expectations mixed with the wrong directions often lead to fruitless collaborations. On the flip side, there’s a smarter and much more productive way to collaborate with your A/B...
A/B Testing is an experiment that compares two or more versions of a specific marketing asset to determine the best performer.
Isn’t multivariate testing just several A/B tests running at the same time? No. That’s one of the most popular myths about MVT. It’s much more nuanced than that. Unlike A/B tests that compare one change at a time, multivariate tests evaluate multiple variables and their combinations simultaneously. It’s an efficient...
Some A/B tests are doomed before the first conversion rolls in. If your traffic split is off, even slightly, your results might look clean while quietly misleading you. That’s Sample Ratio Mismatch: when test variants don’t get the exposure they’re supposed to, and the assumptions behind your stats fall apart....
