Sequential Testing

What is Sequential Testing?

Sequential testing is a statistical method used in online experiments that allows experimenters to monitor data continuously or at scheduled intervals and stop the test early based on accumulating results.

Unlike fixed-sample tests—which require collecting a predetermined amount of data before analyzing results—sequential testing introduces interim evaluations governed by formal stopping rules that control statistical error rates.

It is designed to avoid the pitfalls of peeking (or optional stopping), which can drastically increase the risk of false positives. With sequential testing, stopping early can be statistically valid, if implemented using proper models.

How Does Sequential Testing Work?

Sequential tests rely on decision rules that allow an experiment to be stopped early for one of two reasons:

• Efficacy: Evidence emerges early that the variation is significantly better than the control.
• Futility: Evidence suggests there is no effect or even a negative effect, making continued testing unnecessary.

These decision points are governed by statistical boundaries, such as alpha-spending functions (for efficacy) and beta-spending functions (for futility). Techniques like group sequential designs and sequential probability ratio tests are common implementations.

Why Use It?

Sequential testing is especially useful when:

• Speed of decision-making is a priority.
• You want to reduce exposure to underperforming variants.
• You’re optimizing for time-sensitive features, like flash sales or promotional banners.
• Resource constraints make long tests costly.
• You want to avoid wasting traffic or time once a clear result has emerged.

Because sequential tests often stop earlier, they can use fewer samples on average while maintaining the same statistical power, reducing costs and speeding up experimentation cycles.

“Sequential testing is useful for optimizing time-sensitive assets or when real-time decision-making is critical.

Sequential testing offers a robust framework for building automated decision-making systems. In digital advertising, for example, sequential testing can be used to test variations of time-sensitive promotions or event-based campaigns. It allows teams to quickly identify the best-performing variations and optimize their spending by showing them to everyone.

Other common use cases include ephemeral content, ramps, fraud & failure detection. Sequential experiments are an extremely powerful tool in an experimenter’s armory.”

Vijay Krishnan, Data Science Manager

Risks and Limitations

Sequential testing introduces statistical and practical complexities:

• Biased Estimates: P-values and confidence intervals from sequential tests are not unbiased in the same way as fixed-horizon tests. Interpret with care.
• Threats to Generalizability: Early stopping may produce results based on unrepresentative early users, which might not hold up at scale.
• Complexity: Requires more sophisticated planning, tools, and statistical literacy.
• False Confidence: Teams might mistake early significant results for conclusive truth without understanding the model’s assumptions.

Best Practices

To use sequential testing responsibly:

• Use Proper Statistical Models: Don’t just peek. Use frameworks like AGILE or alpha-spending functions.
• Pre-Plan Decision Rules: Define efficacy and futility boundaries before running the experiment.
• Validate with A/A Tests: Confirm your platform can handle repeated analysis without bias.
• Interpret Results with Caution: Understand and communicate the increased bias risk in early-stopped tests.
• Replicate When Possible: Particularly for surprising results or high-stakes decisions.