See in real time how much money your experimentation program can realize, given your testing roadmap, your traffic (risk appetite), and the costs of good & bad decisions for your business.
Disclaimer: This calculator is merely an indicator of the hypothetical revenue potential of your testing program and isn't definitive. We recommend experimentation be a part of every process you execute as a business, amplifying the ability to make better decisions. It isn’t a stand-alone revenue channel.
Frequently Asked Questions
In statistical terms, a confidence level represents the probability that a parameter will fall between two set values. In this tool, it's the degree to which you are willing to accept that the projected outcomes (number of correct/wrong decisions) are accurate. A 95% confidence level, for example, implies that you would expect the real outcome to lie within the projected range 95 out of 100 times if the test was repeated.
Calculating the financial impact of a good business decision involves assessing various factors such as increased revenue, cost savings, return on investment (ROI), and overall profitability. Here are some resources and tools that businesses can use to make these calculations:
Financial Modeling Software: Tools like Excel or more advanced software like Quantrix, Planful, or Vena Solutions offer robust capabilities for financial modeling. These tools can help in forecasting, budgeting, and simulating different scenarios to understand the financial impact of decisions.
Business Intelligence Tools: Tools like Tableau, Power BI, or Looker can analyze data from various aspects of the business to provide insights into potential financial impacts.
Professional Consultation: Consulting with financial analysts, accountants, or business advisors can provide expert insights tailored to the specific context of the business.
Industry Benchmarks and Reports: Resources like industry reports, benchmarking studies, and market analysis can provide context and comparative data to estimate the potential financial impact of a decision. Start here if you do not have the means to analyze data to understand how correct and incorrect decisions may impact your financial bottom-lines.
Each of these resources can be tailored to the specific needs and context of the business to help accurately calculate the financial impact of decisions.
These numbers are derivatives of your chosen confidence level. The formula for calculating 'Winning Tests' is Winning Tests = Confidence Level x Total Number of Tests. 'Losing Tests' is computed as Losing Tests = Total Number of Tests - Winning Tests. Essentially, these numbers reflect worst-case scenarios based on your acceptable risk.
ROI stands for 'Return on Investment'. In the context of this tool, ROI is computed as ROI = (Winning Tests x Cost of Good Decision) - (Losing Tests x Cost of Bad Decision). It's a straightforward but powerful metric to gauge the financial efficacy of your decision-making.
These are monetary values that you attribute to the outcomes of each test. They represent the financial impact of making a correct or incorrect decision based on the test outcomes, affecting your bottom-line accordingly.
An Excel sheet serves as a manual ledger for tracking your decision costs. Given that these costs are arbitrary and based on your own historical data, an Excel sheet can provide a more tailored and historical perspective on decision-making efficacy over time.
The tool assumes a simplified model of decision-making and does not account for externalities or more complex statistical models like Bayesian inference, game theory considerations, or multi-armed bandit problems. Also, ROI, as calculated here, is an estimation and not a guarantee.