Why Ad Platform Testing Alone Misses the Mark: Run Experiments That Impact Real Business Metrics

If you thought the phrase “let’s test that” was loosely thrown around too much with website testing, you probably haven’t seen many paid advertising ‘experiment’ reports.

Can you run statistically valid tests in paid advertising platforms? Yes.

Should you even bother testing in paid advertising platforms? Yes. (And I’ll help you with the right measurement mindset to have.)

When done right, running tests in paid advertising platforms has an outsized return on investment. It’s one of the biggest incremental revenue levers you can pull as a business.

After reading this, you will have a much better idea of what is possible and why it’s important to have a process.

(D2C, B2B, B2C…we’re all H2H – Human to Human here, so yes, this will work for you… and I’m still meaning that for you, B2B Enterprise Accounts) 😉

The Paid Advertising Platforms

There are the big two:

Google: Search (Google Search Ads), Display (Google Display Network), YouTube, Discover, Gmail Sponsored Promotions, Google Maps, Google Shopping/Performance Max

Meta: Facebook, Instagram, Messenger, WhatsApp, Facebook Audience Network, Meta Reels

These two make up well over 50% of the ad spend from advertisers. Unless you have some incredibly odd ICP (who doesn’t use Google or Meta’s ecosystem), then you should really focus your efforts on these two primarily. There are hundreds of millions of dollars in ROAS out there for every business between these two.

Yes, LinkedIn is more popular for B2B, of course, but it’s also more expensive and nowhere near as good an algorithm as Meta with creative targeting.

If you are in Ecommerce, then you need to be aware of Amazon’s continued rise:

BUT, I think Amazon is where users go at a pretty distinct stage of their journey. You aren’t going to Amazon for discovery; you are generally going once you are problem, solution, and product aware already. You are probably going because you already went to the website and you want it faster, or cheaper, or both.

This is one of the many reasons Paid Media Advertisers need to be shifting towards MMM instead of in-platform metrics and attribution.

What “Testing” Capabilities Does Each One Have

Google Ads has these built-in experimentation tools:

What an “experiment” looks like in Google Ads (and what to watch out for):

Meta offers three testing types: A/B Testing, Brand Lift, and Conversion Lift. Here’s the A/B testing interface:

You can’t even configure to test for “custom” events of your own language; you have to use Meta’s standard events.

Here is a rundown of all the ways in which you can test on both Meta and Google Ads:

• Creative (Ad variation) testing
  • Test different images, videos, headlines, and ad copy to determine which creative performs best.
  • Dynamic creative testing: Upload multiple elements and let Meta auto-generate and test combinations.
• Audience testing
  • Compare different targeting criteria, demographics, interests, or custom audiences to find the most responsive segment.
• Placement testing
  • Test performance across placements (Feed, Stories, Reels, Messenger, Right Column, Audience Network, etc.).
• Bidding/Optimization strategy testing
  • Split test bidding strategies (CPC, CPM, oCPM) or optimization events.
• Campaign objective/Type testing
  • Test different campaign objectives (Conversions, Traffic, Awareness, etc.) or purchase events.
• Landing page testing
  • Send users to different landing pages to identify which drives better results.
• Incrementality/Conversion lift testing
  • Run coordinated holdout or lift studies to measure incremental impact versus a control group.
• Ad ranking
  • Let Meta algorithms rotate/auto-optimize and retrospectively assess top ads (less controlled than A/B).

Here are the testing methods in most ad platforms that I recommend prioritizing first: Creative testing, Campaign objective testing, Landing page testing, and Conversion lift testing (really just a subset of Campaign objective testing).

Let me explain each one of my reasons behind those.

Creative Testing

This is everything in Meta nowadays. Since Andromeda rolled out on November 1st, 2024, Meta has fundamentally changed as an advertising platform.

There is no more: demographic targeting, interest or behavioral targeting, lookalike audiences, to name a few.

Creative is your targeting now.

So when something is emphasized, like creative testing and iteration, it’s for the right reason. If you are serious about running ads on Meta, you will be iterating and testing constantly.

I personally don’t do any “Experiments” using Meta’s A/B testing feature.

What I do instead is launch creatives on the same day and see which perform best. Sounds kind of basic, and in theory it’s similar, but I don’t need Meta to tell me what the stats are or what they need to be for a conclusive result.

By far the biggest issue with using Meta’s own testing is that Meta doesn’t know what “success” looks like for a business. Even when conversion tracking is set up via EdgeTagging (the gold standard nowadays), you still need to check your bank account to see the real results.

Here is an example of another tool that helps me see when batches of creatives were launched on the same day:

I don’t use this for my own Meta ads accounts (this is for competitor intelligence), but the concept of launching ads at the same time and ads continuing to run is correct.

I would highly recommend your “success” metric to be what benefits your business (sales), not micro-conversions. Even if that means you have a lead generation business model and you have to import offline conversions.

If you start to optimize too high up in the funnel, it could send the wrong data to the algorithm.

I highly recommend doing your creative testing on Meta, not Google. Take the learnings from Meta and apply the concepts to Google. Watch to make sure it’s driving the right business impact, but Meta is going to be a much better platform for “creative assessment” vs Google. YouTube costs more money to “learn from” and generally takes longer.

Campaign Objective/Type Testing

As I mentioned in the previous section, tread carefully when optimizing for goals that are higher up in the funnel. Make sure there is “declared intent” behind a user action. This mainly applies to lead generation business models where importing offline conversions is usually more complex, but is very important.

It’s very easy for Google and Meta to find users who will opt in for something but won’t be willing to give you money.

Add extra form fields so you are paying for quality leads, ones that convert to customers at a decent rate.

There are situations where there needs to be more awareness of the problem, solution, or just overall brand in the target ICP’s eyes. In this case, campaigns designed for consumption can make sense.

Make sure to set the hypothesis appropriately as a result.

Landing Page Testing

Landing page testing is what we are all more used to and where the larger levers usually exist. One of my favorite ways to start with a business is around offer testing. That usually has the biggest lift earlier on. UX is, of course, important, but it has never compared to changing the offer (or perception of the offer) in my experiences.

This will never not be important, though I also feel it isn’t where you should start from day one, either.

Incrementality/Conversion Lift Testing

I would call this more of a way of testing and not a placement or type of testing you have available to you.

If you have the data for this, you should absolutely do it.

And here is the good news: you don’t need a lot of data here. If you are spending $10k/month, then you probably have enough data.

The mindset to have with this type of “testing” – I’d call it more B/A testing (before/after testing) rather than A/B testing:

“If we increase the budget on Meta by 20%, do we see a 20% lift in sales?”

Assuming 2 things with this statement:

1. Your campaign goal was purchases
2. Your time lag is quick (less than a day)
   1. If it’s more than a day, then you might need to look into the following week to check the incremental lifts

Spend and sales are the two most popular variables to isolate (properly) with incremental testing.

Too many marketers tend to get caught up in various forms of “engagement” and it doesn’t lead to any noticeable business impact.

Why Positive Constraints Help You Focus on Fundamentals

I am a fan of starting to test on ad platforms first (before your website) for two reasons:

1. You will get quicker feedback loops
2. You can only test so many different things

To the second point, you can test these things: ad copy, headline, creative, call to action buttons.

That’s it. That’s also “freeing” in my opinion.

Think about what you can test on a website… everything. Where do you even start?

Hopefully, from the bottom up (forms/checkout first and then work your way up), but even then, working your way up the funnel pages can feel paralyzing.

When you boil marketing down to the basics, it largely comes down to messaging. Or “copy” as I like to say.

If we don’t have a hook that is resonating with our audience, it doesn’t make any sense to come up with new CTAs at the end of the video, for example.

Here is an actual metric in Meta to look out for: Hook rate.

Hook rate = The percentage of times your video played for at least 3 seconds.

Again, if your videos aren’t getting played for 3 seconds or more, it’s a complete waste of time to focus on testing different offers at the end of the video.

It’s no different than worrying about the pricing table mid-way down the page if no one even scrolls more than 25%. You are solving a problem that doesn’t need solving.

P.S. Hook rates should be over 50% in Meta.

This anatomy of a video might be helpful:

What Tests I’ve Found To Work Best

I’ve run quite a few tests over the years (both on websites and in ad platforms). Here are the types of tests that have consistently driven the best performance and learnings:

• Offer tests (can do both in ads and on landing pages)
• Creative tests (just for ads, as the landing page should largely be congruent with the ad creative)
• Hooks (just for ads)
• Talent on camera (just for ads, but again, should be congruent on the page)

Here is a process document I use when working with clients, sharing the areas in which we test:

How to Know If Ad Platform Testing Is For You

This all might seem a bit complex, but the good news is that the barrier to entry here is quite low. The learning rate is quite high when approached correctly, instead of only thinking about website testing.

If you want to get started in Ad Platform testing, here are the requirements:

• Spend at least $5k/month per platform
• Be open to measuring off-platform to see true results
  • Shopify/BigCommerce or Ecommerce platform for Ecom
  • Salesforce/Hubspot or CRM platform for Lead Gen
• You are committed to “feeding the algos” with creative and copy iterations
• You are open to using spreadsheets for pre-test planning and post-test analysis

Here is a helpful tool from Koalatative regarding pre-test planning: https://koalatative.com/tools/pre-test-calculator/

The In-Platform Mirage

Beware of doing all of your testing in-platform (Meta, Google, etc). It is ok to do, but it needs to be taken with a big grain of salt. It will always look better than it really is and you can only over test “so much” if you are doing it in-platform.

In-platform testing is usually one of the last places I will test. Very few businesses have a thoroughly vetted/tested front end offer and that is almost always where you should start.

It’s quite risky to run ads without a proven offer in a business. If you don’t have a proven offer and money model, start there.