What Security Criteria Should You Evaluate When Buying an A/B Testing Tool?

Dionysia Kontotasiou
February 27, 2023 ·
What Security Criteria Should You Evaluate When Buying an A:B Testing Tool?

Well, if you are a part of the experimentation ecosystem, you already know that Google Optimize is bidding us goodbye. Time to look for a few Google Optimize alternatives!

Yes, you need to free up your schedule, sit in for a ton of demos, and take trials.

You know what to look for, but there’s so much out there that it’s hard to figure out which features are most important.

When looking for an A/B testing tool, among other things, you should also evaluate the security of the platform. And in the context of experimentation, security means three things:

  1. Preventing leaking of data
  2. Preventing manipulation attempts
  3. Encrypting all communication channels between your systems and the A/B testing tool provider’s system.

To help you make the right choice, in this article we’re going to share a list of security questions to consider when choosing an A/B testing platform.

Is the A/B Testing Platform Compliant With Any Security Standards and Certifications?

Over the past ten years, more and more enterprises have started requiring security and compliance certifications from SaaS and other service providers before awarding contracts.

This has increased the demand for certifications such as SOC 2, ISO 27001, etc.

By obtaining these accreditations, organizations can standardize their cybersecurity precautions to protect data and preserve their brand integrity. Some of the most popular ones for A/B testing are:

  • ISO 27001: This is a standard for information security. Your A/B testing platform must comply with best practices in information security and regulations in order to be certified.
  • SOC2: The Service Organization Control 2 (SOC2) certification ensures that your A/B testing tool has a comprehensive set of internal controls in place.
  • PCI-DSS: This standard ensures that credit card information is maintained in a secure environment if your A/B testing platform processes, stores, or transmits it.

Check if the A/B testing platform you intend to use meets any of the above criteria, or better, all of them.

Did you know that Convert’s storage and information processing infrastructure is fully hosted by ISO 27001 and SOC 2 certified cloud service providers, AWS and Hetzner?

Convert uses Amazon Web Services (AWS) for its production servers and databases. Convert and AWS both take security seriously. Amazon employs cutting-edge data security measures, as well as physical access restrictions at server locations. To comply with GDPR, Convert also uses Hetzner Online GmbH for the login site.

Finally, you can find Convert’s PCI-DSS SAQ A-EP self-assessment here.

Note: Convert’s support team can answer any questions you may have, including security-related inquiries. Contact support@convert.com if you wish to report a problem or find the answer you need.

How Often Does the A/B Testing Platform Perform Penetration Tests?

Penetration testing is a process that allows an organization or individual to evaluate their security systems by simulating real-world attacks. Penetration tests or pen tests should be performed regularly and at different intervals depending on the type of data being stored and its sensitivity.

When you conduct A/B tests, a pen test could protect you from

  • Unprotected access points
  • Weak passwords
  • SQL injection attacks
  • Code injection
  • Cross-site scripting
  • Data breaches
  • Phishing attacks

Convert frequently hires external security companies to evaluate its services besides the security controls carried out internally by the Convert security team.

Convert does penetration tests twice a year to find any security flaws or vulnerabilities and provides the required security patches as quickly as possible after such flaws are found.

To conduct an independent penetration test of the Convert application, customers are encouraged to contact support@convert.com to obtain both Convert’s and its hosting provider’s permission.

Does the A/B Testing Tool Offer a Bug Bounty Program?

A bug bounty program rewards ethical hackers who find and tell the application’s developer of vulnerabilities or issues. This allows businesses to increase the security of their systems by leveraging the hacker community.

Since bounty programs attract a diverse group of hackers with varying skill sets, companies using bounty programs are at an advantage over those who use less experienced security teams.

That’s why businesses often use bounty programs in addition to normal penetration testing to assess the security of their applications throughout the development process.

No technology is faultless, and Convert believes that finding flaws in any technology requires collaboration with knowledgeable security experts from throughout the world. Convert gives the capacity to continuously monitor and assess its application, detect hazards as needed, and take appropriate action.

We advise contacting Convert’s support team if you think you’ve discovered a security flaw in our product.

Does the A/B Testing Platform Encrypt Data Both in Transit and at Rest?

Encryption is a security best practice that protects data in transit and at rest, preventing unauthorized access to the information.

Most A/B testing tools encrypt your data at the highest level possible (SSL/TLS). This means that while it’s being transmitted over the internet, it’s protected by encryption technology that keeps prying eyes from seeing what you’re doing with your customer data.

If you’re using an A/B testing tool that doesn’t offer encryption as part of its service or only offers limited protection for sensitive information, it’s time to look elsewhere!

Documents and database backups for Convert customers in the US data centers are stored in the AWS US west region, whereas for customers in the EU data centers, they are stored in the AWS Ireland region. Amazon manages the backend servers and provides encryption in transit. BambooHR encrypts sensitive fields at rest in the BambooHR database. If the customer requests it, any custom fields created by the implementation or customer service team can be encrypted at rest and marked as sensitive. The BambooHR database and file backups are encrypted at rest, and only IT personnel have access to them.

Does the A/B Testing Platform Support Single Sign-On?

With Single Sign-On (SSO), users can log in to multiple connected but independent software systems with a single ID and password. By sharing the authentication token between multiple applications, SSO authentication allows users to remember fewer passwords, which improves security.

This makes it an important security feature to look for in an A/B testing platform.

Convert Experiences single sign on

Convert’s SSO feature is available on selected plans only. Learn more about our pricing plans and their features.

Who Will Have Access to Your Data?

A/B testing platforms handling PII data should adhere to the principle of least privilege, which states that employees should only have access to the data they need to do their duties. By doing this, only a portion of each vendor’s staff will be allowed access to the technology stack’s storage areas or transit zones, reducing the risk of both malicious actors and user breaches.

Convert has a very tight data access policy, by applying the principle of least privilege. Only Convert’s dev team has access to the data the tool collects. We always know who is authorized to access data from a user account, so access rights to data are only granted in name. Users’ rights are updated frequently, and a review is done at least once every three months.

What Type of Information Is Logged? How Long Is It Available?

When choosing an A/B testing platform, you should also consider the tool’s logging capabilities.

  • What information is logged, and how long are logs available?
  • How long is data stored in the cloud?
  • How long is data stored on its servers?
  • How long is data stored on your client’s devices (if applicable)?

It is important to know how long you have to retrieve information from these sources when you need to conduct an audit or investigation.

Convert offers two types of logs:

Live Logs available through the UI (you can keep these for a week depending on the number of views, conversions, and transactions per day). Live Logs track how end users are interacting with web pages in real time. They capture information like timestamps when a goal is triggered, the event type that was triggered, variations displayed to the end user, and much more.

Debugging Logs are kept here: logs.convertexperimetns.com (for 7 days only). Debugging Logs are used for debugging purposes and are not accessible via the UI.

Convert Experiences live logs

Is There an Incident Response Plan in Place?

When you’re evaluating an A/B testing tool, you need to consider how the company handles security incidents. It is especially important if you plan to use the software in a production environment and want to be sure that your data is protected. For example,

  • Do they have an incident response plan?
  • What steps would they take if their system was compromised?
  • How will they communicate with customers about the breach?
  • Who will be responsible for handling these situations?
  • How often does their team meet up internally and externally (e.g., with law enforcement) to discuss potential threats or best practices surrounding them?
  • If there’s ever a breach or other issue involving user data, what kind of information would need to be released publicly so that users can take action accordingly?

Convert has an Incident Response Plan in place. Contact us to ask for a copy.

What Happens to Your Data if You Choose to Stop Taking A/B Testing Services?

Compromises in your A/B testing platform compromise your own systems as well. If your data is stored on a vendor’s system, you are at risk.

When you stop using an A/B testing tool, make sure all of your data has been deleted from their databases. Some providers treat this final phase as an afterthought, but it’s imperative that they delete all of your data. The last thing you want is massive data repositories or breadcrumbs scattered all over the vendor ecosystem. This increases exposure and overall risk.

When you stop using Convert, Convert will give you access to your data and help you export it. If there are any issues with exporting your data, Convert will work with you on a case-by-case basis.

How Do You Audit User Activity on Your A/B Testing Platform?

It is not always possible to see what activity is taking place within an A/B testing tool. Some tools don’t allow you to see critical information, like

  • Who made a change to a particular Split URL test
  • Who added a GA integration to a particular deployment
  • Who deleted some critical code, preventing an experience from running

A log of user activity is crucial to securing a platform as it ensures users are accountable for their actions and can be used as evidence as needed in the event of a security breach.

Convert logs most actions performed in the app, such as creating or modifying an experiment, adding or removing audiences, etc.

The Change History feature shows a record of user activity for each of your projects.

You can troubleshoot an experiment by looking at the change history to see what changes were made, when they were made, and who made them. Detailed change histories provide additional security for teams with multiple collaborators.

Convert Experiences change history
Convert Experiences change history log

Does the A/B Testing Vendor Choose Third-Party Vendors Carefully?

Make sure that the vendor you’re considering has a good track record and is reputable.

  • Check for tool security issues or if the vendor works with third parties that could threaten your data.
  • Read reviews from previous customers. Find out if there are any common complaints about poor performance or worse, data breaches due to poor security practices on the vendor’s part.

See what users are saying about Convert on G2, TrustRadius, and Capterra.

Does the A/B Testing Tool Have the Visitors’ Consent To Process the Personal Data Collected?

Choosing an A/B testing platform should also include asking the following questions:

  • Did the vendor recommend updating your privacy policy?
  • Has the visitor’s consent been obtained for the tool to process their personal data?
  • Is the privacy policy easy to find on your website, and does it explain how you collect and use visitor data, including personal information (e.g., names, email addresses)?
  • Does it include information about how visitors can withdraw their consent at any time?
  • Do you have a procedure in place for obtaining consent from visitors who want to receive marketing messages via email or SMS?

Convert’s commitment to transparency extends to both security and privacy.

What Data Does the A/B Testing Platform Collect? (e.g. Traffic, Clicks, Page Views, Demographic Data, etc.)

Testing data collected from your website(s) is transmitted to the A/B testing vendor’s servers and stored there. Traffic, visitors, IP addresses, and location are some of the most common data collected by A/B testing tools. These can be used to make decisions about whether a test is successful or not.

You can use A/B testing tools to determine how different versions of your website perform in real-world environments. In order to do so, your tool needs to collect visitor behavior data, including

  • Traffic information: How many visitors came to your site, where they came from (referrals), and which pages they viewed on your site.
  • Visitor information: The device that each visitor uses (iPad vs iPhone vs desktop), the operating system on the device, their geographic location (city and country), what browser they use, and other similar characteristics.
  • IP address information: An IP address uniquely identifies every computer connected to the internet at any given time. Because these tools keep track of every visitor who comes to your site, they can tell if someone has visited more than once or if someone visited multiple times within a short period of time
  • User behavior data: This can help you understand how your users react to a new design or layout. You can also see how long they spend on a particular page and which elements attract their attention more than others.
  • Number of conversions. You can, for example, use an A/B test on a landing page for your product to see how each variation performs in terms of conversions in comparison with the original.

What Tracking Technology Does the A/B Testing Platform Use? (e.g. Fingerprinting, Local Storage, Web Storage, Cookie, Pixel, etc.)

Several methods are used by A/B testing platforms to track users across devices.

Some companies rely on fingerprinting, which is the process of identifying a device by its unique fingerprint. This method is the most accurate, but it’s also more invasive and requires more work from the user.

Cookies are another common method for tracking users, though they’re far less accurate than fingerprinting because they rely on a lot of data being sent back and forth between the browser and the server every time a user visits a page. Cookies can also be disabled by users who want to avoid being tracked.

Other methods include local storage, which stores data locally on your computer, or mobile advertising IDs (MAIDs), which identify mobile devices based on unique identifiers.

Always ask your A/B testing tool what method it uses.

Convert uses 3 different cookies to track visitors during A/B tests.

We hope this article helped you understand why security is important in A/B testing tools, and how to evaluate your options. Taking the time to thoroughly review these criteria will help ensure that you ultimately select a platform that is secure and reliable. Here is a quick comparison of the most important security features in a few popular A/B testing tools:

Security standards Pen tests
Bug bounty Encryption SSO Access to data Logs Incident Response Plan
Convert ISO + SOC 2 infrastructure, PCI-DSS compliant Principle of least privilege
Optimizely ISO certified, SOC 2 infrastructure, PCI-DSS compliant Principle of least privilege
VWO ISO certified, SOC 2 infrastructure, PCI-DSS compliant Principle of least privilege
A/B Tasty ISO + SOC 2 infrastructure, PCI-DSS compliant Unknown
Adobe Target ISO + SOC 2 certified, PCI-DSS compliant Unknown
User Activity Audits Third-Party Vendors
Data Collected Technology being used Cancel Service User Consent
Convert Link to Privacy Policy Cookies Export and delete data
Optimizely Link to Privacy Policy Cookies, LocalStorage Export and delete data
VWO Link to Privacy Policy Cookies, LocalStorage Export and delete data
A/B Tasty Link to Privacy Policy Cookies, LocalStorage Export and delete data
Adobe Target Link to Privacy Policy Cookies Export and delete data
Basic Plan Free Trial Banner
Basic Plan Free Trial Banner Vertical
Originally published February 27, 2023 - Updated December 14, 2023
Mobile reading? Scan this QR code and take this blog with you, wherever you go.
Dionysia Kontotasiou
Dionysia Kontotasiou Convert's Head of Integration and Privacy, helping customers with technical queries.
Carmen Apostu
Carmen Apostu In her role as Head of Content at Convert, Carmen is dedicated to delivering top-notch content that people can’t help but read through. Connect with Carmen on LinkedIn for any inquiries or requests.

Start Your 15-Day Free Trial Right Now.
No Credit Card Required

You can always change your preferences later.
You're Almost Done.
I manage a marketing team
I manage a tech team
I research and/or hypothesize experiments
I code & QA experiments
Convert is committed to protecting your privacy.

Important. Please Read.

  • Check your inbox for the password to Convert’s trial account.
  • Log in using the link provided in that email.

This sign up flow is built for maximum security. You’re worth it!