Why The Data Doesn’t Match Up When Analyzing Test Results (GA4 + Convert)

There’s little worse than when your company loses trust in its data. To be clear, it’s rarely the data itself that’s the problem, but rather the people and processes around it, since raw data is rarely the “end product”.

Accuracy doesn’t exist with marketing data, and it never will. Accuracy is for accounting, not marketing data. And conditions will continue to get worse over time (privacy changes, generative AI, increasingly fragmented user journeys, etc). But if you are using trends and patterns, this isn’t scary, but more so an adjustment of strategy.

Even data that doesn’t need transformation (the “T” in “ETL” known as Extract, Transform, and Load in the data world) still needs a narrative to tell the story. That is, if you want the insight to have value and hopefully be actioned.

If raw data is rarely the end product, what causes us to lose trust in our data?

Three main culprits: data literacy, transformations, and documentation (or lack thereof).

Data Literacy Between GA4 and Convert

There might be a better word as it kind of sounds accusatory to say, “you aren’t data literate”, but that is the truth many times 🙊

Many users who adopted GA4 treated it as an extension of Universal Analytics (which it absolutely wasn’t). This approach is like trying to put diesel in an electric car. GA4 should have been treated as a completely new analytics platform – because that’s exactly what it is.

It was Google’s fault for naming it “GA4” in the first place, but they wanted the product adoption. I digress…

… GA4 is way more powerful than Universal Analytics. However, the learning curve is steeper.

Most Common Metric Challenges in GA4

The biggest stumbling block is the scoping of metrics. Total users vs sessions vs views. Yes, these are all different metrics in GA4.

And the hierarchy is as follows:

• User Scope: The highest level. Encompasses all sessions and events for a user. (Total users)
• Session Scope: Encompasses all events within a single session. (Sessions)
• View Scope: Lives beneath the sessions metric, and a user can have multiple views in a website session. Like viewing different pages. A view is a “hit” scope in GA4’s data model. (Views)
• Event Scope: Focuses on individual user interactions within a session. (Total events)
• Item Scope: Specific to individual products or services within e-commerce events. (view_item)

What you need to know is that Total Users is THE metric to use for testing. While I rarely make such definitive statements, using any other metric risks biasing your test results.

Unless you are exclusively setting your test audience to be new users, you could have a return visitor to your test who would behave very differently than the first time they came to your site. I usually have an agenda when I’m a return visitor to a site.

If you are using sessions or views as your primary test metrics, be careful, as you could be introducing bias without realizing it.

We Write Hypotheses for Humans, Not Sessions or Views

We write hypotheses for users, meaning the real people behind the data, not for each session or visit. Our goal is to measure the impact on unique individuals, so the total users metric in GA4 gives us the most accurate picture of how our experiments influence actual behavior. This ties our analysis directly to the people we are trying to understand and improve experiences for.

It also shouldn’t matter how many times a user returns in a given date range – what matters is whether they completed the action or not. The focus is on the outcome for each person, regardless of visit frequency (aside from rare edge cases).

Transformations Challenges in GA4 Data

Transformations don’t need to be a word only reserved for data pipelines (Reverse ETLs or ETL, as previously mentioned). Every time you add an audience or report filter, create a segment, or throw in a comparison, you’re transforming the data in GA4.

Again, transformations are necessary in order to share the insights you want. But this is also where trouble can creep in.

Data Thresholding

If you are performing a transformation (let’s use the most basic one, a filter, for example) and if the filter you are using reveals a small data sample, you could be subject to thresholding:

Thresholding is a privacy measure that limits or hides certain data when there’s a risk that users could be identified. It’s meant to protect user privacy and comply with data regulations.

Data Sampling

Data sampling in GA4 means you might not see every single data point in your reports. Instead, GA4 looks at a sample of your data and uses it to estimate the overall numbers. This helps GA4 run reports faster, especially with lots of data or complex filters.

Key points:

• GA4 uses only a portion of your data to build certain reports.
• Sampling usually happens in custom or complex reports.
• The numbers you see are estimates, not exact counts.

It will look something like this:

Documentation to the Rescue: Creating Consistent Tracking Across Platforms

Always leave a paper trail for your future self. At minimum, use consistent test ID names between GA4 and Convert to avoid confusion when referencing tests.

The fewer differences between platforms, the better.

Consider how conversions (also known as key events) are set up in GA4 and Convert. While you can configure GA4 through GTM (see my previous article for doing that with Generative AI) and set up Convert conversions via JavaScript test experiment code, there’s a better way.

Since you’re already using GTM for GA4, deploy Convert conversions (aka goals) through GTM as well. This ensures consistent data deployment and reduces the chance of errors.

Don’t Chase Accuracy

One key takeaway here is to NOT chase accuracy with data. Focus on what the data is telling you, not whether the numbers match perfectly across platforms. Accuracy doesn’t exist with marketing data and it never will.

… but that is ok! What we need are consistent trends and patterns leading to a useful truth.

Next time you hear “Shopify shows 100 sales last week, GA4 shows 92, and Convert shows 88,” don’t immediately dive into troubleshooting. Instead, reframe the situation: How do Shopify’s numbers compare to the previous week? Then, examine the same week-over-week comparison in GA4 and Convert.

If Shopify’s sales increased by 10% from the previous week, GA4 and Convert should reflect similar trends.

A good rule of thumb: if your bottom-of-funnel conversion tracking (purchase, lead, free trial, phone call, etc.) is off by more than 5% between systems, investigate why.

That said, you should also “put your money” where your actions are. Ask yourself, “Based on the actions we’ve taken historically as a business, has 5% (or whatever your % difference is here) been enough of a difference for us to change our actions?”

Oftentimes, that change isn’t significant enough. The bigger point is that it’s happening to ALL conversions and not just one type. Trends and patterns typically hold across traffic sources—it’s rare to see something like Meta being 5% off while GA4 isn’t. If you are using unified triggers in GTM, this will definitely be solved for.

This approach prevents you from fixating on minor discrepancies that wouldn’t change your decision-making anyway.

Happy analyzing! 🤓