Hypothesis

What is a Hypothesis in A/B Testing?

A hypothesis is a focused, testable statement that predicts what will happen when you make a specific change in an A/B test. It forms the foundation of any controlled experiment, providing clarity on what you’re testing, why you’re testing it, and what outcome you expect.

There are three types of hypotheses often used in A/B testing:

• Substantive hypothesis: A high-level claim about how a change will impact behavior or outcomes (e.g., “Adding testimonials will increase sign-ups by building trust“).
• Precise claim: A specific, quantifiable prediction (e.g., “Reducing form fields will improve completion rate by at least 10%“).
• Statistical hypothesis: The formal null/alternative model used in significance testing (e.g., “There is no difference in conversion rate between the control and variant“).

Where Hypotheses Fit in the Experimentation Process

Hypotheses sit at the heart of the experimentation lifecycle:

1. Start with an observation or insight: Derived from user research, analytics, or business challenges.
2. Formulate a hypothesis: Define the change, the expected outcome, and the reason behind your expectation.
3. Design the experiment: Choose metrics, select audiences, and build variants, all guided by the hypothesis.
4. Run and analyze: Test results are interpreted through the lens of your hypothesis—was your prediction supported or refuted?
5. Learn and iterate: The hypothesis shapes how you extract insight and plan future tests.

A strong hypothesis doesn’t just kick off the experiment; it drives the design, measurement, and interpretation. Without one, experimentation loses focus and becomes reactive guesswork.

“Your hypothesis forms the basis of your experiment. It’s a crucial building block. The same test could be considered a success or failure based on whether the data supports or disproves the hypothesis.

A good hypothesis should summarise what you’re changing, why, and what you think will happen. You can take it further and explain why you think the change will occur. For example, ‘we believe that changing the color of the CTA from red to green will draw more attention, therefore increasing the click rate because psychologically, green is a positive color associated with go rather than stop.

I also find hypotheses help me focus on analyzing the data that relates to the test rather than losing track.”

Georgiana Hunter-Cozens, Senior Strategy Consultant at CreativeCX

How to Write a Strong Hypothesis

A clear, useful hypothesis should include:

• The change you’re making (independent variable)
• The expected outcome (dependent variable/metric)
• The reasoning behind your expectation

Common A/B Testing Hypothesis Structures

• “If we change [X], then [Y] will happen.”
• “We believe [change] will improve [metric] because [reason].”
• “Our team believes testing [action] will improve [outcome]; we’ll know this because [metric] changes significantly.”

Hypothesis Example:

“If we change the CTA color from red to green, then the click-through rate will increase, because green is associated with ‘go’ and is easier to visually scan on our existing page layout.”

What Makes a Hypothesis Strong?

• Specific: Clearly identifies the change and outcome
• Measurable: Tied to a specific metric (e.g., bounce rate, form completion)
• Testable: Can be evaluated through a controlled experiment
• Falsifiable: Possible to disprove the prediction
• Grounded in data or user behavior: Based on actual observations, not speculation
• Aligned with business goals: Helps drive meaningful, strategic outcomes
• Simple and focused: Avoids testing multiple changes at once

Best Practices for Hypothesis Writing

• Start from observed behavior or a real user pain point
• Test only one major change at a time
• Make the reasoning behind your hypothesis explicit, even if not in the statement
• Keep language simple and aligned with how your team works
• Ensure the test requires experimentation, not something existing data could answer
• Collaborate and document, hypotheses improve with input and transparency

Common Pitfalls to Avoid

• Vagueness: “Let’s test a new layout” without explaining why or what success looks like
• Over-complexity: Testing too many variables in one variant dilutes learning
• Trying to prove the mechanism: You can observe that a change works; proving why it works often requires deeper or multiple studies
• Running tests without a hypothesis: Leads to wasted effort and unclear analysis
• Assuming qualitative insight is enough: Surveys and interviews inform hypotheses, they don’t replace experimental validation