How to Ensure Accurate Results from your A/B Tests Despite Apple’s ITP
- What Is ITP?
- How does ITP work?
- Why Did Safari ITP Come into Being?
- How Has ITP Evolved Over the Years?
- Is Your A/B Testing Impacted by Safari ITP?
- What Tools Can You Use to Pin Down ITP Impact?
- What Does the Impact of ITP Look Like on A/B Test Results?
- A Live A/B Test Was Used to Simulate the Impact of ITP
- What Happens if You Ignore the Impact of ITP in A/B Test Results?
- Convert’s Solution to Getting Around ITP
- Closing Thoughts
What Is ITP?
Intelligent Tracking Prevention (ITP) is an important Safari component that helps protect users from visitor tracking. It is a feature of Webkit, an open-source browser engine that powers Apple’s default browser.
It was originally designed to block third-party cookies (trackers that identify users across different domains) for Safari users. Over time, it evolved into also restricting some first-party cookies that act as trackers. This is where the importance of ITP lies. It does an excellent job at preventing malicious visitor tracking.
How does ITP work?
ITP uses the Machine Learning Classifier, a machine-learning model, that identifies domains with cross-domain tracking capabilities and then restricts cookie creation under those sites.
Why Did Safari ITP Come into Being?
For a long time, before ITP was developed, third-party data was the main source of website visitor information for the marketing, analytics, and advertising industries.
Visitors, however, did not have a clue about who was able to access their personal data and what they could do with it. The lack of transparency regarding who could access the information made it difficult to know if companies were using this responsibly or not.
The solution came in a series of unexpected privacy laws (GDPR, CCPA, CPRA, LGPD, ePrivacy), which enabled users to understand when their personal information is being collected by companies or accessed improperly, giving them the ability to decide how this is used–including whether or not they want it collected at all!
With the introduction of these laws that protect user privacy, browsers were forced to comply. Apple was the first to incorporate privacy features into its browser. In 2017, it launched Intelligent Tracking Prevention (ITP) to protect the privacy of its users.
How Has ITP Evolved Over the Years?
ITP has changed over the years, each version introducing new restrictions. Let’s go through the most important versions of ITP and see what changes each of them brought about.
Share this infographic on your site:
Is Your A/B Testing Impacted by Safari ITP?
If you’re conducting experiments with any of the following, ITP may have an impact on your results:
Third-Party Cookies
Because ITP does not allow third-party cookies, you will most likely be negatively impacted and should consider switching to another solution.
First-Party Cookies
These cookies will be deleted from the user’s browser after 7 days, which may affect your data and user experience.
Local Storage
This data will be deleted from the user’s browser after 7 days, which may have an influence on your test results and user experience if the conversion path is normally longer than a week.
Server-Side Testing
This is the safest solution at the moment, but make sure your testing tools support server-side bucketing and segmentation.
What Tools Can You Use to Pin Down ITP Impact?
If you’re still not sure if you’ve been impacted by ITP, here are some tools that can help you spot its impact right away.
Google Analytics Reports
You can use your Google Analytics reports and look for the changes we cover below. You will be mainly using the Browser & OS report and the Source/Medium report to identify the ITP impact on your data.
ITP Impact Calculator for Google Analytics Data
You can also use the ITP Impact Calculator to see how ITP 2.1, 2.2 and 2.3 may have impacted your GA data. There is also an article where you will find an explanation of the output of this tool.
What Does the Impact of ITP Look Like on A/B Test Results?
Now that you know how to spot if you’re affected by ITP, you may be wondering how ITP can influence your A/B test results.
So let’s look at how much of an effect ITP would have on your A/B test results and some common signals to look for in your A/B test reports.
Fewer Returning Visitors with Safari 12.1 and Higher
One tell-tale sign that Intelligent Tracking Prevention is taking place is when the number of new users who use browsers that force ITP changes increases, while the number of returning users decreases. Meaning you get more new visitors and fewer returning visitors than usual.
This is because, when ITP happens, cookies can no longer do their job. ITP’s 24-hour and 7-day visitor targeting limitations alter the meaning of “returning visitors”.
Those visitors who come back to your website within 7 days of their last visit are accurately reported as “returning visitors”. While users who interacted with your site repeatedly but that converted or made a purchase after 7 days are classified incorrectly as “new visitors”.
This redefinition of “returning visitors” means that ITP inhibits proper marketing attribution, as well as market segmentation and retargeting.
Next, let’s show you exactly how you can spot this potential issue in your Google Analytics reports.
Filter the “Browser” report by Browsers containing “Safari”. Set the metric visualized in the chart to “New Users” and “Returning Users”, and the granularity to “Month”. This allows you to see that fewer returning users are captured over time.
You can also use the ITP Impact Calculator for this. The pie charts show how the number of new and returning users has varied over time. In this case, only 15.8% of users with ITP 2.1 or higher may be identified as returning users. 22.4% of other browsers can be identified as returning users.
Higher % New Sessions
Due to the fact that returning visitors cannot be easily recognized, another thing you will notice is that the “count of sessions” dimension will be impacted.
This means you will see a higher percentage of Safari users with a single session count.
Let’s look at this dimension in Google Analytics.
Filter the “Browser” report to any Browser containing “Safari”. Set the metric visualized in the chart to “% New Sessions” and the granularity to “Month”. This shows you the “% New Sessions” trends over time.
You can also use the ITP Impact Calculator and see how the “% New Sessions” has changed after March 2019 when ITP 2.1 took effect. Because the duration of the cookies has been reduced, you will see more “% New Sessions” for Safari compared to your site overall to understand the macro and micro implications.
Wrongly Declared Winning Variations
This is the worst fear of any tester. Nobody wants to declare false winners, yet the reality is ITP may be a culprit. Because of ITP, you might notice some irregularities with your declared A/B test winners.
Since visitors can only be classified correctly as new vs returning for 7 days, you have a very short time with valid data to declare a winner. But if your site does not get enough traffic before the expiration date, the wrong winner might be declared or no winner declared at all.
See what this would look like in the Convert Experiences app:
No Marketing Attribution
One of the biggest nightmares for marketers is not being able to attribute conversions properly. This means that you can’t track which marketing campaigns are resulting in sales, and so you can’t determine which ones are worth investing in.
Nowadays, most marketers use query parameters like UTM parameters to attribute conversions.
But these can be flagged by the ITP, which limits relevant cookies from 7 days to 1 day. This gives you a very short window to attribute conversions properly. Furthermore, as we’ve shown you previously, returning visitors are not reliably categorized beyond 7 days. This misdirects credit for the visitor’s conversion to the last channel of engagement or transaction.
In rare circumstances, ITP may even be able to remove all of the extra information from URLs. When UTM parameters are lost, there’s no way to link conversion to a specific channel or source, rendering your analytics meaningless.
Let’s take two examples.
Visitor A comes from an affiliate site on Monday, then comes back a week later to purchase:
- Before ITP: sale is attributed to “Affiliate”
- After ITP: sale is attributed to “Direct”
- Why: the 2nd visit is more than one day after the 1st
Visitor B clicks through from a Facebook ad to your most recent blog post and purchases:
- Before ITP: sale is attributed to “Facebook”
- After ITP: sale is attributed to “Direct”
- Why: the visit to the blog is not linked to the visit on another domain
Go to your Google Analytics dashboard and select Acquisition, then All Traffic from the drop-down menu. To see a list of how people are finding your site, go to Source/Medium.
If you rely on traffic from affiliates like Facebook and LinkedIn, ITP may push your web analytics program to show inflated unique visitors. Affiliates like Facebook and LinkedIn can easily recognize unique visitor traffic from these sources in regular, non-ITP conditions. Visitors to your site would bring a unique visitor identifier with them via the URL, which the affiliate would compare against a cookie dropped on the browser before that. The affiliate would then monitor visitors across domains using the cookie and visitor identification.
Below you can see the number of unique visitors coming from Facebook and LinkedIn without ITP impact:
With ITP, things are a little different. ITP now employs a blacklist of domains capable of cross-site tracking. If ITP blacklists your affiliates’ sites and they employ something that looks like a visitor identifier, the cookies they drop will have a 24-hour expiry period instead of the usual 7-day one. As a result, you’ll only be able to track users within 24 hours.
If you were to load the same report as before, but that now shows an ITP impact, you will notice a decrease in “Users”:
No Persistent User Profile Creation
With ITP, user profile creation is no longer available. The user profile will also be blurry. The user profile tracker is no longer able to issue a persistent user or customer ID to visitors when using ITP. This is a dynamic situation. The mechanism of “retargeting” is also affected. To provide the most appropriate content and adverts, retargeting relies largely on the user’s permanent ID.
Meaningful Personalizations Only for New Visitors
Personalizations will be impacted depending on the logic that underpins them.
Real-time personalization, in which the user’s experience is shaped by their behavior during that particular session, will be unaffected. However, recognizing their preferences and behaviors when they return will no longer work unless they return to your site within 7 days.
Web Functionality Issues
ITP has consequences on other areas as well, such as web functionality and affiliate marketing. If you use script-writable cookies or any other form of script-writable storage to save visitor data, you’ll really feel the effects of ITP. Your website may request cookie consent on a regular basis, reset language and currency preferences to the default settings, and so on.
End-To-End Performance
Because of the restriction on first-party cookies on your website, ITP has an impact on your digital marketing and, as a result, your business as a whole. Because advertising and analytics platforms rely on these cookies for tracking and reporting, the implementation of ITP means that marketing data insights can no longer be completely trusted.
A Live A/B Test Was Used to Simulate the Impact of ITP
Consider a website that receives 30% of its traffic from Safari users. It receives 200,000 unique monthly visitors and has a conversion rate of 3% on average.
We created a simulation of what would happen if a two-page A/B test were conducted (the Original and Variant1). Variant1, according to the theory, outperforms the Original.
On our graph, however, it is shown that when ITP is not removed, the Original is doing better.
The simulation shows that, because of ITP, the experimentation tool will report that the Original is the winner after 7 days.
This is the opposite of the results obtained when the impact of ITP is removed, which shows that Variant1 should be the actual winning result.
Essentially, ITP causes your A/B test to produce completely inaccurate and unreliable results. There is something you can do about it though. Read on to see what that is.
What Happens if You Ignore the Impact of ITP in A/B Test Results?
On their Tracking Prevention Policy Apple stated:
We treat circumvention of shipping anti-tracking measures with the same seriousness as exploitation of security vulnerabilities. If a party attempts to circumvent our tracking prevention methods, we may add additional restrictions without prior notice.
It means that if your website tries to circumvent Apple’s built-in anti-tracking features, it will be treated as malware.
If you try to use a grace period, you may not see an impact in your Analytics reporting right away, but you will see an inconsistency between Google Analytics/your experimentation platform and other external providers that take ITP changes into consideration.
Convert’s Solution to Getting Around ITP
Let’s start with an example to understand what ITP may do to your Convert experiences and reports (if you don’t apply our solution below).
For example, if your funnel has three steps and a visitor completes the final step on the eighth day, the flow of the funnel will never be complete, even if the visitor completes all of the steps. This is due to the fact that the data gathered throughout the customers’ journey may expire after 7 days.
Beyond this, the visitor is treated as a new visitor and the data collection process begins anew.
The results become unclear because the same user is traced many times.
This might also mean that if a visitor returns to the website after 7 days, they will be offered a different version of the same test than what was previously displayed.
This can then lead to inconsistencies in the user experience and have an influence on campaign accuracy.
Visitors that arrive on your site from a domain with cross-domain tracking capabilities and a query string or fragment identifier in the URL are recognized as new visitors if they arrive 24 hours after their previous visit, even though they landed on the site a bit longer before that.
This is because cookies hold information about which variation was shown to the user.
So what could go wrong in your Convert experiences and reports?
- Someone who comes back several times during an experience may have a different variation each time.
- Due to varying privacy protection standards in each browser, the same activity may perform differently.
- Conversions may not be appropriately attributed to the experiences offered over long conversion windows (longer than 7 days).
- As clean separation of variation influence gets more difficult, your optimization program will notice more “inconclusive” or “flat” outcomes.
- Because returning visitors are mislabeled as new visitors, experiments are likely to exceed the pre-calculated minimum sample size earlier than expected.
- Users will not be identifiable across visits.
Fortunately, none of the above has to happen to you.
In order to overcome the issues above, we investigated numerous techniques to mitigate the impact of ITP, before deciding to relocate the cookie generating process from the browser to the server.
Because the new cookie duration restrictions apply only to browser-created cookies, we moved the cookie issuance part to your web server, which means the cookies will be created by your server rather than the users’ browsers.
We’ve written a comprehensive guide to help with server-side cookie creation. Check it out here. Please do not hesitate to contact our team if you require assistance in making the necessary changes to your web server infrastructure.
And here’s a rundown of which indicators are likely to be impacted in your Convert Experiences account and how you can exclude them from your reports.
For Users that Cannot Apply Our ITP Solution
If the above solution isn’t possible on your side, Convert offers filtering your experience results by a default set of user segments, which includes Browser, right out of the box.
Use the Browser segment option to display results that include and exclude Safari to check for cross-browser consistency of an experiment’s results.
Here’s how:
Go to the “Report” tab of your experience:
From the Segment drop-down, use the “Filter by Browser” checkboxes:
Select all of the browsers listed in the menu, excluding Safari.
This allows you to view results with all typical reporting figures and statistical significance calculations, with and without the population of visitors who may be affected by ITP.
Just keep in mind that other browsers might force these privacy changes by default so this workaround is only applicable until ITP is forced to all other browsers (similar to Safari).
Closing Thoughts
How are businesses supposed to manage this terrain given the ever-changing privacy requirements while also improving marketing efforts, preserving a competitive advantage, and satisfying a growing demand for individualized consumer experiences?
Unfortunately, there is no simple solution.
The main thing to remember is that if your company hasn’t already, it’s important to make sure your teams are aware of the privacy concerns and the potential consequences of data collecting, results analysis, and optimization.
It would be a critical error to turn a blind eye or discount the potential effects just because Safari may represent a small portion of your traffic.
While Safari has been at the forefront of increasing data protection, Firefox and Chrome are also taking steps to improve their privacy. As a result, these limits may affect a larger number of your site’s users.
If your teams are struggling to keep up with all of the new privacy updates and determine how they will affect the business, give us a shout.
Convert’s three major strategic solutions are focused on updating important aspects of the business that may be impacted by changing privacy regulations, such as:
- Enhancing the Client Experience
- Boosting Your Marketing Results
- Gaining A Competitive Advantage
We’re now assisting our clients with navigating the ITP updates and ensuring that the essential workarounds are in place across their various systems. Let us know if we can do the same for you.
