Convert’s Free A/B Test Hypothesis Generator:
3 Tools to Take You from Idea to Design

Many people define a hypothesis as an “educated guess”.

To be more precise, a properly constructed hypothesis predicts a possible outcome to an experiment or a test where one variable (the independent one) is tweaked and/or modified and the impact is measured by the change in behavior of another variable (generally the dependent one).

A hypothesis should be specific (it should clearly define what is being altered and what is the expected impact), data-driven (the changes being made to the independent variable should be based on historic data or theories that have been proven in the past), and testable (it should be possible to conduct the proposed test in a controlled environment to establish the relationship between the variables involved, and disprove the hypothesis – should it be untrue.)

An A/B test should be treated with the same rigour as tests conducted in laboratories. That is an easy way to guarantee better hypotheses, more relevant experiments, and ultimately more profitable optimization programs.

The focus of an A/B test should be on first extracting a learning, and then monetizing it in the form of increased registration completions, better cart conversions and more revenue.

If that is true, then an A/B test hypothesis is not very different from a regular scientific hypothesis. With a couple of interesting points to note:

• Most scientific hypotheses proceed with one independent variable and one dependent variable, for the sake of simplicity.
  
  But in A/B tests, there might be changes made to several independent variables at the same time. Under such circumstances it is good to explore the relationship between the independent variables to make sure that they do not inadvertently impact one another.
  
  For example changing both the value proposition and button copy of a landing page to determine improvement in click through or completion rates is tricky. Reaching a point where the browser is compelled to click the button could easily have been impacted by the value proposition (as in a strong hook and heading). So what caused the improvement in the dependent variable? Was it the change to the first element or the second one?
• The concept of Operational Definition is non-negotiable in most laboratory experiments. And comes baked with the question of ethics or morality.
  
  Operation Definition is the specific process that will be used to quantify the change in the value/behavior of the independent variable in the test.
  
  As an example, if a test wishes to measure the level of frustration that subjects experience when they are exposed to certain stimuli, researchers must be careful to define exactly how they will measure the output or frustration. Should they allow the test subjects to act out, in which case they may hurt or harm other individuals. Or should they use a non-invasive technique like an fMRI scan to monitor brain activity and collect the needed data.
  
  In A/B tests however, since data is collected through relatively inanimate channels like analytics dashboards, generally little thought is spared to Operational Definition and the impact of A/B testing on the human subjects (site traffic in this case).

According to an analysis of over 28,000 tests run using the Convert Experiences platform, only 1 in 5 tests proves to be statistically significant.

While more and more debate is opening up around sticking to the concept of 95% statistical significance, it is still a valid rule of thumb for optimizers who do not want to get into the fray with peeking vs. no peeking, and custom stopping rules for experiments.

There might be a multitude of reasons why a test does not reach statistical significance. But framing a tenable hypothesis that already proves itself logistically feasible on paper is a better starting point than a hastily assembled assumption.

Moreover, the aim of an A/B test may be to extract a learning, but some learnings come with heavy costs. 26% decrease in conversion rates to be specific.

A robust hypothesis may not be the answer to all testing woes, but it does help prioritisation of possible solutions and leads testing teams to pick low hanging fruits.