A/B Testing Software Open Source: FAQs Answered

An A/B testing open source software solution is one that allows you - the optimizer - to be in complete control of the code that drives the engine serving variations to traffic, collecting results, and creating reports to help you zero in on web page changes that can boost important KPIs.

You have the privilege of owning the experiments you hypothesise and the customer data you collect, across all the channels you wish to improve conversions for.

Most A/B testing software open source solutions lead with the promise of better security, more customizations, ease of deployment, and a robust API that delights developers looking to dig deeper into a platform and amp up its capabilities.

The debate of open source vs. commercial 3rd party tools is one that rages across multiple industries and disciplines.

But in the realm of optimization and A/B testing where most recognized industry players look to serve enterprise customers, open source is emerging as a popular alternative to lower the entry into A/B testing without getting locked into costly annual contracts.

Technically you can’t outgrow an open source A/B testing solution. It though can outgrow your organization in terms of the maintenance it needs to continue delivering desired results.

Under such circumstances if you are shifting to a commercially maintained tool, you should look to “import” as many advantages of the open source format as you can.

Ask these questions:

• Does the new A/B testing tool have flexible pricing with the ability to upgrade, downgrade and cancel - without sales rep intervention?
• Does the new A/B testing tool give you access to its API without an exorbitant price tag?
• Does the new A/B testing tool have responsive support? After all this is something that open source alternatives can not bring to the table. You should look to take advantage of being able to reach tool experts who will solve your problems for you.
• Does the new A/B testing tool have advanced developer options? You will probably not have the freedom to simply commission the features you want added to the vendor tool, but opt for a solution that allows you to play with the code in small but significant ways. Like having a Code Editor. Or the ability to add custom Javascript to your projects, experiments and variants.
• Does the new A/B testing tool have easy integrations? This is something that you will enjoy if you are switching from the open source format. The ability to add integrations with a few clicks saves a lot of time and enriches the optimization process with responsibly collected data.
• Does the new A/B testing tool have exhaustive documentation? One of the drawbacks of an open source A/B testing solution might be the potential lack of documentation. It is good to make sure that the vendor maintained tool comes with well written, well organized and easy to find support articles.

• Your organization is big on flexibility and agility. With an open source A/B testing solution you do not need to rely on the features provided by the vendor. Custom development is possible. You can add features that your optimization and A/B testing program needs. Open source is synonymous with the freedom to solve an issue in multiple ways with a greater chance of finding a solution that really benefits your company.
• You have extremely high test velocity. When you have complicated workflows and are running thousands of tests a month (think - Facebook and Amazon), it makes sense to take an open source A/B testing software solution and build it into a platform that can support your scaling needs.
• You like the idea of the “wisdom of the crowd”. This is one of the core strengths of an A/B testing software open source solution. In most cases there is an involved and ever-growing community of people who are really passionate about the tool adding to its capabilities on a daily and monthly basis. You can incorporate these tweaks and upgrades if it makes sense for your optimization needs.
• You don’t want to battle with sales reps and licenses. Open source is - open. You can get started with the deployment right away. As long as you have the skills in the organization to customize it, install it, and scale it according to the documentation, you do not need to jump through hoops to get on calls with sales reps or go through the hassles of license management. Generally the initial spend on an A/B testing open source solution is relatively low.

When it comes to investing in an open source A/B testing software solution, you must keep in mind that nothing is truly free.

You will spend hours installing and updating the tool to get it to work just the way your organization needs it.

The last thing you want is for the community around the project to disappear or fall into disengagement taking away most of the advantages of the open source format.

Evaluate projects based on the following criteria:

A robust hypothesis may not be the answer to all testing woes, but it does help prioritisation of possible solutions and leads testing teams to pick low hanging fruits.

• Funding: Where is the project deriving its funding from? How stable is that source?
• Legacy: How long has the project been around? A thriving community that’s seen years is less likely to fall into disengagement than something which sprouted overnight. But this does not mean you should go by longevity alone.
• Code Activity/Release History: This is crucial too. How well is the source code being maintained? What updates are being made and how frequently? Where are these releases logged?
• User Community: We return to the people who power an open source tool. Look at how many passionate, talented individuals are in the community and how many are joining the cause? How diligent are they about keeping the documentation clear and updated?

Based on our research and feedback on the internet, here are some open source A/B testing frameworks that may be worth your time:

• ABBA
• SIXPACK
• PROCTOR
• PHPA-B
• VANITY
• COHORTS
• ALEPHBET
• WASABI