The Guide to Ethical A/B Testing: The Missing Component of Your Optimization Program
Back in 2014 Facebook was on the receiving end of severe backlash when it revealed that it had grouped users into an “emotional contagion” study that blatantly manipulated emotions by showing “optimistic” or “depressing” feeds to those who were chosen.
The most disturbing aspect of the whole debacle was the fact that the people who were experimented on were unaware of the fact that they were being manipulated.
Let us put a pin in that thought and ask you:
Do the visitors to your website know that they are being tested on?
The answer is probably no. After all, you are not trying to manipulate their emotions… are you?
The truth is, marketing is the art and science of impacting emotions. And A/B testing is how that impact is isolated and quantified.
Make people feel more secure about a website and they’ll buy more. If the website is actually secure, you are enhancing the user experience by squashing needless fears. If the website does not have a back-end to justify the trust signals, then it is blatant manipulation.
Like any effective technique, A/B testing can do a world of good by letting businesses present relevant offers in a way that helps people.
Or it can do a world of harm through deception, manipulation, and even by treating data collected for an experiment in a lackadaisical way, leaving it vulnerable to breaches.
What is Ethical Optimization & Why Should You Care?
A/B testing is here to stay and will only become more powerful as Artificial Intelligence develops.
If you look into the future of optimization, you can see AI proposing hypotheses that are a thousand times more likely to impact the behavior of site visitors than anything we can conceive right now.
And data are the foundation stones gradually building this future into existence.
This is why the famed GDPR (General Data Protection Regulation) was a big deal and will continue to be so.
It is a rap on the fingers of tech giants and businesses that treat people as nothing more than visitor counts on their online properties. Even though the GDPR is a regulation, strangely enough, it humanizes people by forcing companies to see them as individuals who might protest to their data being misused.
What’s more… the GDPR was a precursor of other initiatives like the ePrivacy Directive and the California Privacy Rights Act.
As brands like Netflix and Amazon scale their testing programs, it is only a matter of time until the question of ethics in A/B testing becomes mainstream and gets its own set of guidelines.
If you plan to comply with each new regulation when it comes out, you will stifle innovation in your business and will be running to avoid landmines.
The better approach is to embed ethical A/B testing in your company and make it a part of your organization’s culture.
Ethical A/B testing is testing that treats site visitors as humans. That is all.
When ethics are at the core of your company, you automatically:
● Respect user privacy while collecting data to frame hypotheses.
● Weigh the impact of your test on their psyche and psychological well-being to rule out manipulation.
● Take adequate care to store and process their data in secure ways.
● Respect their consent and allow them to opt-out of experiments.
In short, you become transparent and accountable.
And you get compliant where any regulation – past, present, or future – is concerned through your daily business practices.
In this guide, we are breaking down the steps you can take to minimize data privacy hassles during testing and the considerations to keep top of mind for transparent, ethical A/B testing.
GDPR, CCPA 2.0, and Beyond: How Have They Changed A/B Testing & Analytics
Europe’s reaction to the data misuse was the Data Protection law, the GDPR (General Data Protection Regulation), which forces any company handing data to walk the extra mile to safeguard their client’s data.
The US’s reaction was the CCPA (California Consumer Privacy Act), Nevada SB 220, and more recently, the California Privacy Rights Act 2023.
The aim of these laws is essentially about two things, the ethical use of personal data and keeping that personal data secure.
This has forced companies from all over the world to start strengthening their data security and privacy.
With these privacy laws, the EU and the US also introduced a new legal requirement: privacy by design.
At its core, privacy by design calls for the inclusion of data protection from the onset of the designing of systems, rather than an addition.
Also, the conditions for consent have been strengthened and companies are no longer able to use long illegible terms and conditions full of legalese.
These laws introduced data portability – the right for a data subject to receive the personal data concerning them – which they have previously provided in a ‘commonly use and machine-readable format’ and have the right to transmit that data to another controller.
However, accommodating the above changes should not be driven by fear of consequence.
Rather, companies and marketers should consider how well their values support areas of business practice and how scenarios might play out that will lead to their systems and procedures facing challenges. This might come from an individual tested visitor that their data should be moved or deleted or a hacker trying to gain personal data illegally. It might also come with a Data Subject Access Request (DSAR).
Whatever the requests around personal data, marketers should know, understand and commit to appropriate behaviors and values.
It is that which will allow choices to be made to protect the individual and the company.
This is why ethical A/B testing is so important: it impacts how values are communicated within an organization, how they are demonstrated by leadership, and how they are embodied in day-to-day working relationships.
The Cost of GDPR Non-Compliance: Wake Up to the Numbers
Here is a list of the fines that brands big and small have incurred since the GDPR was enforced. It is a scary compilation in terms of the magnitude of money that the businesses have had to shell out.
But scarier still are the reasons for the breaches.
They display a flagrant disregard for the privacy of users/customers and drive home the need to adopt an organization-wide culture of ethical business.
Company Name | Knubbels.de |
---|---|
Date of Fine | 21/11/2018 |
Data Protection Authority | LfDI Baden-Württemberg |
Value of fine | €20,000.00 |
DPR Article breached | Art. 32 (1) (a) GDPR (obligation to pseudonymized and encrypt personal data) |
Reason for breach | Passwords stored unencrypted and unhashed. Personal information was stolen from 330,000 customers following a hacker attack |
Date breach filed | 9/8/2018 |
Action taken by company | Improvements to IT architecture in coordination with LfDI |
Company Name | Hospital Barreiro Montijo |
---|---|
Date of Fine | 24/10/2018 |
Data Protection Authority | Comissão Nacional de Protecção de Dados (CNPD) |
Value of fine | €400,000.00 |
DPR Article breached | Article 25 relating to privacy by design |
Reason for breach | Too many users in the hospital had access to patient data |
Date breach filed | unknown |
Action taken by company | unknown |
Company Name | Austrian small local business. Name not known |
---|---|
Date of Fine | 1/10/2018 |
Data Protection Authority | Austrian Data Protection Authority (“DSB”) |
Value of fine | €4,800.00 |
DPR Article breached | Not known |
Reason for breach | CCTV camera in front of his establishment that also recorded a large part of the sidewalk |
Date breach filed | unknown |
Action taken by company | unknown |
Company Name | |
---|---|
Date of Fine | 21/01/2019 |
Data Protection Authority | CNIL |
Value of fine | €50,000,000.00 |
DPR Article breached | Not known |
Reason for breach | Lack of transparency, inadequate information, and lack of valid consent regarding ads personalization |
Date breach filed | 25/05/2018 |
Action taken by company | Not yet known |
Company Name | Bisnode |
---|---|
Date of Fine | 15/03/19 |
Data Protection Authority | Polish Data Protection office |
Value of fine | 220,000 approx |
DPR Article breached | Art 14 – Right to be informed (subject data rights) |
Reason for breach | Did not inform about the processing of data. Created a database enabling verification of the credibility of these entities |
Date breach filed | 25/05/2018 |
Action taken by company | Likely to appeal although not yet known |
Company Name | UAB MisterTango |
---|---|
Date of Fine | 16/05/2019 |
Data Protection Authority | Lithuania State Data Protection Inspectorate |
Value of fine | €61,500.00 |
DPR Article breached | Not known |
Reason for breach | Inappropriate data processing, disclosing personal data and failing to report a breach |
Date breach filed | |
Action taken by company | Likely to appeal although not yet known |
Company Name | Plaintiff name not known (mayor in Belgium) |
---|---|
Date of Fine | 28/05/19 |
Data Protection Authority | Belgium DPA |
Value of fine | €2,000.00 |
DPR Article breached | Not known |
Reason for breach | Misuse of personal data by a mayor for campaign purposes |
Date breach filed | Unknown |
Action taken by company | Not yet known |
Company Name | La Liga |
---|---|
Date of Fine | 6/12/2019 |
Data Protection Authority | La Agencia de Protección de Datos, (AEPD) |
Value of fine | €250,000.00 |
DPR Article breached | Not known |
Reason for breach | Users were not explicitly informed of the intended use of the microphone and geolocation permissions. These were being used to identify venues showing matches without paying |
Date breach filed | Unknown |
Action taken by company | Intend to appeal stating AEPD “has not made the necessary effort to understand how the technology works.” |
Company Name | SERGIC |
---|---|
Date of Fine | 28/5/19 |
Data Protection Authority | CNIL |
Value of fine | €400,000.00 |
DPR Article breached | Article 32 |
Reason for breach | – The company had not put in place a procedure to authenticate users of its website to ensure that the persons accessing the documents were the ones who had uploaded them – The company kept the documents uploaded by candidates for an unlimited period |
Date breach filed | 8/12/2018 |
Action taken by company | Unknown |
None of these examples can be directly traced back to A/B testing. But many of the mindsets that have led to these breaches and fines pervade optimization in companies as well.
It is time to wake up and change that.
How to Get Started with Ethical A/B Testing: 10 Solid Considerations To Keep in Mind
In any type of research that involves human participants, it’s important to consider the ethics of the research project.
That is also the case when you do A/B testing. You are responsible for your participants’ wellbeing, for representing them honestly, and for keeping their personal information safe.
Here, we will go over some of the most important considerations for ethical A/B testing.
A/B tests that involve the processing of personal data must provide information about the data protection provisions. It is more likely that your tests raise higher ethics risks if they involve:
- Processing of ‘special categories’ of personal data (formerly known as ‘sensitive data’);
- Processing of personal data concerning children, vulnerable people, or people who have not given their consent to participate in the tests;
- Complex processing operations and/or the processing of personal data on a large scale and/or systematic monitoring of a publicly accessible area on a large scale;
- Data processing techniques that are invasive and deemed to pose a risk to the rights and freedoms of the tested visitor, or techniques that are vulnerable to misuse;
- Collecting data outside the EU or transferring personal data collected in the EU to entities in non-EU countries.
Consideration #1: Testing, Not Deception
A distinction must be drawn between traditional A/B testing and an alternative form of experimentation where algorithm results are modified for a fraction of users for supposed research purposes.
Ding..ding… Facebook 2014 anyone?
In A/B testing, interface design characteristics – such as the arrangement of buttons, layout, or explanatory text – are blocked or rearranged to test their effect. Many online companies routinely perform A/B testing with their users to assess the impact of website design changes.
However, a new form of experimentation emerges when the programming code of a website’s algorithm is altered to induce deception with manipulated results.
This is a deep form of testing, which we call code/deception or C/D experimentation to distinguish it from the surface level testing associated with A/B testing.
C/D experimentation should be distinguished from the ongoing efforts of online companies aimed at improving their algorithms for operational purposes.
Such cases of optimization do not involve deception because the objective is to produce better (more accurate) results for all the users. In contrast, in C/D experimentation the results of the algorithm are altered (i.e. distorted or falsified) for research purposes.
Consideration #2: Look Out for the User’s Best Interest
As explained by Isaac Wardle of Team Croco, you should aim for an alignment between company and user interests.
Ideally, behavioral scientists must ask their collaborating companies what their intentions are and how their intentions align with those of the people they’re working with, often employees or customers.
When intentions are misaligned, researchers and companies need to place greater care on how behavioral insights are used and for what ends.
Here is a list of questions to ask before each test goes live:
- What are we looking to gain in terms of KPI improvements from the test?
- What perception change are we looking to induce through the tests?
- Is this perception shift justified? (Think back to the example of the site with trust signals that can’t be supported by the back-end).
- Will inducing this perception put the tested visitors in any form of physical, mental, or financial risk?
- Is the A/B test worth the cost? Think of the loss of goodwill, opportunities, and customers if the approach is risky and there are chances of things going wrong.
Consideration #3: Transparency and Honesty
You should be honest with your tested visitors about the purpose of your A/B tests, who you are doing it for, and how you are going to use the results.
That way, participants can give their informed consent and won’t be surprised if they encounter your results later.
In some cases, though, you might not be able to tell the tested visitors everything right away. Sometimes, knowing which experiment you are conducting might influence their reactions.
It could be that they like or dislike your brand or that they have experience with your product or service that will influence what they expect from it.
Other times, knowing the purpose of your experiments will influence how visitors act or navigate your site, because they will want to give you the results they think you are looking for. A nice gesture from their side of things but certainly not what you want if your experiments are to provide a solid foundation for a real-world endeavor involving users.
Consideration #4: Keep Your Biases to the Side
When you analyze your tests or present your results, always represent what your tested visitors have said and done honestly.
When we frame hypotheses, we often have preconceived notions about what we think the results will look like—or what we want the results to look like.
It’s important not to search for examples of what you expect your tested visitors to do. That’s subjective and misleading, rather like reacting to reality before it happens. After all, you can’t choreograph real people in their environment; you have to monitor them instead.
Be open and listen to what tested visitors are saying and doing. This might sound obvious, but it can be difficult in practice, given that marketers are human, too.
When communicating your results, make clear the parameters on which you are basing your results. Make sure you count how many of the tested visitors said or did that interesting thing that fits perfectly with your idea for a new design.
Was it all of them, most of them, only a handful, or maybe just one?
Insertion of bias in A/B testing results not only costs business money (when the desired results are not achieved), but also leads to the deployment of variants that do not enhance the experience of visitors, and in many cases may traumatize them, especially if financial transactions are involved and the UX is particularly unpleasant.
Consideration #5: Obtain Consent and Permission if PII Is Involved
Make sure that you get each of your tested visitors’ informed consent (if you are using personal information that can identify them) for participating in your A/B experiment either verbally or in writing. Informed consent requires that your participants have a clear idea of what you are doing and what you will use the experiment for.
Most subjects probably won’t bother to read the information, they’ll just click through to the website as fast as possible.
And if the subjects do read about the study, try not to give out information that might influence them. Say we are evaluating the impact of different shades of blue. Reading about it will almost certainly change how they react to colors when they get to the website, and hence bias the results of the study.
Thus, always ask for consent if you are intending to store personal information, but try to do it in a neutral way.
Consideration #6: Add Easy Opt-Outs
In Facebook’s notorious “social contagion” experiment, people whose news feeds were manipulated did not get any advance notice and there was no way for them to opt-out from any research activities conducted on the site. This is extremely problematic.
Users must be allowed to opt out from A/B testing easily.
Consideration #7: Acknowledge that Data Points Are People (and Hidden Harm Is Real)
One of the most fundamental rules of responsible and ethical A/B testing is the steadfast recognition that most data represent or impact people.
Starting with the assumption that all data are people until proven otherwise puts the task of dissociating data from human subjects on the right track.
Even though it is obvious that you should never do anything that might be harmful to your tested visitors, there is a difference between that harm and the hidden, indirect harm that can rear its ugly head farther down the road.
You can unintentionally cause harm if you don’t carefully consider how you interact with your tested visitors and how you handle their data. The risks mustn’t outweigh any benefits they can get from your results.
Bart Schutz, a behavioral psychologist and an expert in A/B testing unpacks the concept of hidden harm:
If the cleanliness of a hostel or hotel is associated in the minds of women with safety, then tests that highlight cleanliness of accommodations in areas with high crime rates can actually push women to book with places that are clean, but unsafe.
Consideration #8: Guard Against the Re-Identification of Your Data
When datasets thought to be anonymized are combined with other variables, it may result in unexpected re-identification, much like a chemical reaction resulting from the addition of a final ingredient.
While the identificatory power of birthdate, gender, and zip code is well known, there are some other parameters—particularly the metadata associated with digital activity—that may be as or even more useful for identifying individuals. IPs, geolocation, customer IDs and tags, time zones, transaction IDs, timestamps can be used to reidentify people.
Hence, identify possible vectors of re-identification in your test data. Work to minimize them in your published results to the greatest extent possible.
Leverage Pseudonymisation and Anonymisation
One of the best ways to mitigate the ethical concerns arising from the use of personal data is to anonymize it so that it no longer relates to identifiable persons.
Data that no longer relate to identifiable persons, such as aggregate and statistical data, or data that has otherwise been rendered anonymous (so the subject cannot be re-identified), are not personal data and are therefore outside the scope of data protection laws.
However, even if you plan to use only anonymized datasets, your A/B tests may still raise significant ethical issues.
These could relate to the origins of the data or how it was obtained. You must therefore specify the source of the datasets you intend to use in your tests and address any ethics issues that arise.
You must also consider the potential for misuse of the methodology or findings, and the risk of harm to the group or community that the data concerns.
Where it is necessary to retain a link between the tested visitors and their personal data, you should, wherever possible, pseudonymize the data to protect the subject’s privacy and minimize the risk to their fundamental rights in the event of unauthorized access.
Consideration #9: Do Not Target Children with Your A/B Tests
All A/B tests involving children and young people raise significant ethical issues, as the subjects may be less aware of the risks and consequences of their participation. This also applies to the processing of their personal data.
Most importantly, children are impressionable and any hidden harm arising from testing on them is likely to be multiplied and ingrained.
If your tests involve collecting data from children, you must follow the GDPR note on informed consent, in particular, the provisions on obtaining the consent of a parent/legal representative and, where appropriate, the assent of the child.
As that guidance makes clear, any information you address to a child must be in age-appropriate and plain language that they can easily understand. You must also apply the principle of protection by design to test on data concerning children and minimize the collection and processing of their data.
The GDPR establishes special safeguards for children concerning ‘information society services’, a broad term covering all internet service providers, including social media platforms. These include a requirement for verified parental consent in respect of information society services offered directly to children aged under 16.
If you are collecting data from children, you must ensure that you observe the national and EU/US law safeguards and explain in your Privacy Policy how you will obtain and verify the parent/legal representative’s consent.
Consideration #10: Stay Away from Cloaking
Is A/B Testing allowed by Google?
Am I going to get penalized in Google Search results because of cloaking?
Google suggests that if they detect cloaking on your site you may be removed entirely from the Google index.
So what does cloaking mean? Simply put, you display different content to search engine bots and to humans, to manipulate your search engine rankings.
Most of the cloaking scripts identify the IP of the user agent (humans or search engine bots) and based on a predefined list of IPs of search engine bots guess if the visitor is a search engine or a human.
Other scripts use “traps” to identify robots. Based on who’s visiting your site, you can set up your web server to serve the tricky content to the search engine and nice-looking content to the human.
Some examples of cloaking include:
- Serving a page of HTML text to search engines, while showing a page of images or Flash to users
- Inserting text or keywords into a page only when the User-agent requesting the page is a search engine, not a human visitor
There are simple ways to avoid being fined for cloaking:
- Do not distinct on Googlebot User-Agent
- Use rel=”canonical”
- Use 302s for redirects
- Only run the experiment “as long as necessary”.
For more details on how to stay away from cloaking, please take a look here. Or read more about Consent Rate Optimization, a new discipline in CRO.
Making Ethical A/B Testing Easier: Go With a Tool That Understands Data Privacy
No A/B testing solution can argue the moral high ground of your tests for you.
But the responsibility of handling data with care, and keeping you on the right side of compliant testing practices is something that you can delegate to the right tool.
Here are 7 must-have features you should be looking for in a privacy-conscious tool:
Feature #1: Data Anonymization – Testing without Requiring Consent
An important principle in the GDPR is data minimization.
This means that in the context of personal data, product and service providers should only collect, store and process what is adequate, relevant, and limited to their business case.
There is no clear definition of what personal data should be collected and what should not be. It is completely based on the specific use case.
To practice the data minimization principle, we anonymized visitors’ IDs in our tracking by grouping hundreds of website visitors into visitor groups that only count the presence of the visitor.
Individual visitors are not stored in Convert Experiences. It will not be possible to connect group counts to individual visitors in any way.
The GDPR allowed us to take a hard look at what we were storing in Convert and what the use case was for keeping it in an increasingly privacy-centric environment.
Does your A/B testing platform require consent from tested visitors?
Feature #2: Convenient GDPR Warnings
Tools like Convert Experiences have introduced warnings to inform customers of GDPR-related settings or options used in their Projects or Experiments:
- Convert Experiences has traditionally allowed the grouping of site visitors by conditions like location and behavior. These groups are referred to as custom segments. However, post-GDPR, if the Segmentation feature is enabled, this can be interpreted by Privacy Authorities in Europe as a way to identify data subjects. To inform users, we have inserted conspicuous warnings that activate if segmentation is enabled for at least one audience.
- Audiences built with Personal Data: A GDPR warning exists in Saved Audiences and on the Experience Summary pages when Audiences are built with cookies or JavaScript conditions, or if Time Zone, City, Region, Customer ID, or Customer Tags have been
- Cross-Domain Tracking: The cross-domain cookie is by default turned off for all projects in Convert Experiences. Turning it on activates another warning:
- Personalization Experiences may contain small segments (under 100 unique visitors) and this may be interpreted by Privacy Authorities as identification of data subjects. For that reason, we’ve added a warning to the summary of any Personalization Experience.
The purpose of these warnings is to ensure that our users understand which features may be viewed as potential “identification” of data subjects by EU authorities.
It is difficult to memorize the gist of the GDPR mandates!
By using Convert Experiences you work with a tool that can do a lot, but also punctuates its potential with reminders that certain actions are now interpreted differently in EU countries.
You can turn off GDPR warnings.
Does your A/B testing tool offer these safeguards?
Feature #3: Track User Actions with Change History
Have multiple people collaborating on your tests? You must look out for unanticipated changes introduced into your A/B tests.
A changelog is critical in this regard. Tools like Convert Experiences log most actions that can be made in a Project; for example, creating an experiment, modifying a variation, adding and removing audiences, and more. The Change History shows a record of user activity for each of your projects.
If an experiment seems to behave strangely or stops working correctly, you can troubleshoot by checking the change history to see what changes were made, when they were made, and who they were made by.
This detailed change history creates an activity trail that provides additional security to individuals and teams with multiple collaborators.
Are you confident that your tests are being deployed as they were hypothesized?
Feature #4: Two Factor Authentication
Two Factor Authentication (2FA) increases the security of a testing tool by adding a second level of authentication when signing in. Instead of relying only on a password, with two-factor authentication enabled, you will be required to enter a code that you access from your mobile device. That way you can rest easy, knowing that your account is protected even if your password is compromised.
We’ve also built a secure Single Sign-On System (SSO) in Convert Experiences for improved security and ease of use.
Is your A/B testing platform still on single password authentication?
Feature #5: Respect for User Settings (Opt-out and DNT)
Your A/B testing tool should provide an opt-out feature that allows visitors to be excluded.
Each Convert customer needs to have this opt-out form on their site, giving their website visitors the right to object to this statistical research.
Your software should also recognize Do Not Track (DNT) because we think it’s important to have a simple way to control how the end-user information gets used.
Convert Experiences honors DNT as a signal for how you and your end-users want us to use data.
The technical implementation on how Convert will support this field can be explained with DNT’s three possible values:
- Do Not Track (Opt-out of tracking)
- Track (Opt into tracking)
- Null – No preference
By default, web browsers use the null value (no preference), indicating that the end-user hasn’t expressed a desire of whether they want to be tracked or not.
Since 2018, Convert does not load the scripts/experiments when option #1, Do Not Track (Opt-out of tracking), in the browser, is set and load them with the other two options.
Especially with the latest browser settings (Apple Safari with ITP 2.2 and Mozilla Firefox with ETP), it is evident that DNT, Opt-Out and other browser settings that tested visitors use while navigating your site should be respected.
Does your A/B testing solution respect DNT settings and Opt-Outs?
Feature #6: Transparent Stats Engine
A/B testing is a technique based on statistical methods and analysis. That said, you do not need to be a statistician to understand the concepts involved or the results given to you by your favorite A/B testing framework.
But it is good to know the mathematical equations used to calculate the statistics and metrics surrounding your test and understand what the results mean to you and how they can potentially impact your tested visitors.
We at Convert are very transparent about the algorithms we use to calculate statistical confidence and winning variations. You can find more details here.
We use a 2-tailed Z-test at a .05 statistical significance level (95% confidence) (that is .025 for each tail being a normal symmetric distribution), with the option to change this between 95%-99%.
Do you know how your A/B testing tool reaches conclusions about winning variants? Click here to use our A/B Testing Significance Calculator.
Feature #7: Ethical Tool with Ethical Partners
Just working with a compliant testing tool from an ethical vendor is not enough. We live in an interconnected world and no SaaS company stands alone.
Choose a solution that has built an ecosystem of conscious partners.
At Convert, we have a set of questions that we use to partner with any new third-party vendor:
- Where are your data and applications stored?
- Is that data ever moved out of the EEA?
- Do you ever transfer data between data centers outside of the EU?
- Do you always inform me when my data is being transferred?
- Do you have a Data Protection Officer?
- What data controls and risk management processes do you have in place?
- How do you manage the version release process on your platform to ensure an adequate level of data protection?
- Who can access my data, under what circumstances and what can they see? Is this access tracked?
- Can I audit your security and technical measures on the protection of data?
- Do you have in place a security breach notification process?
- Are you GDPR compliant?
Is your A/B testing tool partnering with ethical vendors?