Agency vs Freelance vs In-House CRO: Which CRO Setup Actually Works?

The trade-offs, the hidden costs, and what really drives results in experimentation

I’ve got the full bingo card over the last 11 years: I’ve worked agency-side, in-house, and freelance. No prize for the win, but it does mean I’ve seen the good, the bad, and everything in between.

One thing I see far too often: when companies decide to invest in CRO, the default move is to hire an agency. That can be a huge mistake.

Not because agencies are bad (there are plenty of great agencies out there), but because they’re not the right fit for every organization. And realistically, an agency isn’t going to tell you you’re not ready for them, and why would they?

So what is the right setup? It’s the one that drives the most learning for your business at its current stage.

The question most companies get wrong

Most companies frame this as a resourcing question: “Who’s going to run the A/B tests?” It’s a fair question, but it’s not the most important one. The real question is: “Who owns our experimentation strategy and the learning that comes from it?”

That’s the difference between doing CRO and actually building an experimentation capability. And it’s the latter you should be aiming for.

Because no matter the setup, one thing remains true: you can’t permanently outsource CRO. For it to be effective, experimentation has to become part of your DNA.

Finding the right fit

Which bed is the right comfort for Goldilocks — and your brand? Here’s a cheat sheet to help you decide whether you need an agency, a freelancer, to double down in-house, or a mix of everything.

1. Agency

Sometimes the reality is simple: you have the strategy, but not the team to execute it. You need research, design, development, QA, and analysis — and hiring all of that at once just isn’t realistic. A CRO agency can be a powerful way to kickstart (or support) your program.

It can also help you generate early wins that build leadership buy-in, proving that CRO is a growth lever worth investing in.

But agencies should kickstart your process, not become a permanent fixture. Ideally, they should enable your team to take over, not create long-term dependency. As Lucia van den Brink, founder of The Initial, puts it:

Our goal is to help internalize experimentation as a capability within the organization. This is how we differentiate what we do from many other CRO agencies that talk about the Start Date of collaboration, but never the End Date.

Here’s the catch: research from Convert found that 90% of agencies say they offer “strategy,” but that definition varies wildly. For some, it’s test prioritization; for others, it’s embedding experimentation into your operating model. Before you sign anything, make sure you’re clear on which version you’re actually getting.

One more thing to watch: the same research found that 60% of agency practitioners run two or fewer tests per month. If the velocity is that low, it’s worth understanding why before committing budget. It’s not always the agency’s fault — traffic, dev resources, and internal approval processes all play a role — but it’s a conversation you should have upfront.

2. Freelance/consultant

Freelancers often get more deeply embedded than agencies. They typically work with fewer clients, and without layers of account management, the thinking can feel sharper and more strategic.

That said, they’re usually strongest in one core area. Don’t expect a single CRO freelancer to lead strategy, design, development, and analysis all at once.

While many will have a solid grasp of the full process, they’re far more impactful when supported by a team. Freelancers work best when you need a specific skill or when you want senior guidance to help grow your team’s CRO capability.

They’re especially useful for filling a strategic gap, or in the early stages of optimization when budget doesn’t yet stretch to a full-time hire. Ideally, they either upskill someone internally or help you reach the point where CRO is driving enough impact to justify building out the team. Here is how Ruben de Boer, an Independent Experimentation & Decision Strategy Leader, explains it:

A freelancer is the right choice when an organization already has a team and wants to build experimentation as a capability, not outsource it. You bring in an experienced expert to help teams improve how they think, work, and make decisions, so experimentation continues to grow long after the freelancer is gone. That experience also helps teams move faster, avoid common mistakes, and overcome obstacles much more quickly.

One important distinction: a freelance CRO strategist is very different from a freelance experiment developer. One shapes your program; the other executes within it. Make sure you’re clear on which one you actually need.

3. In-house

Long-term, in-house is always a strong choice. Knowledge stays within the business, context compounds over time, and things move faster without the constant back-and-forth of briefing. Every experiment generates learning, and you want that living inside your organization, not buried in someone else’s Slack channel.

That said, in-house comes at a cost, especially if you need multiple roles. It’s not just salaries, but tooling, management time, and opportunity cost. There’s also the inherent risk that comes with hiring.

That’s why this setup tends to make the most sense once you’ve validated that experimentation drives results and you’re ready to scale. At that point, bringing things in-house helps embed experimentation into everyday decision-making, not just as a tactic, but as a core capability to ensure you maximise the value from your experimentation program. This is how they approach it at Audible, explains Beatriz Tavares, Global Acquisition & Experimentation Manager:

Experimentation is part of our DNA and our team’s mindset. Customer obsession is one of Audible’s core principles, which is why we invest in in-house experimentation to learn faster, apply those learnings to improve the value we deliver to our customers, and drive growth over time.

4. A mix

Because CRO is inherently multidisciplinary, it’s rarely a case of picking just one option, and most organizations end up using a mix. Some common combinations include:

1. Agency for execution, in-house for strategy: Works when you have a strong internal leader who knows what to test but needs hands to build and analyze experiments.
2. Freelance/consultant for strategy, in-house for execution: Ideal when your team is capable but too close to the product to spot broader patterns. It’s a great way to kickstart a program.
3. Agency for capability building, transition to in-house: The gold standard of agency relationships. They set up the program, train your team, and eventually step back, which is something many agencies struggle to do effectively.

For any setup to succeed, one rule is non-negotiable: clarity on who owns the learning. Insights must be documented and fed into future decisions, or it doesn’t matter which model you choose.

Hidden costs to consider

An agency or consultant will, naturally, want to sell to you. A good one will flag hidden costs and be transparent, but just in case they don’t, here are a few things to keep top of mind:

• Knowledge retention. When an agency or freelancer leaves, do the learnings stay? Your experiment history is just as valuable as the results themselves. Make sure insights are documented somewhere your team can access and build on.
• Tool fragmentation. Agencies often bring their own tech stack. When the engagement ends, can your team continue using those tools? Can the data transfer smoothly? Getting this wrong means starting from scratch.
• Context. Every new person needs time to ramp up. In CRO, context is everything, and the best test ideas come from a deep understanding of the product, the customer, and the history of what’s been tried.

How to choose the best option for you

While there are many factors to consider, the most important is your stage, and what setup makes sense for where you are right now:

1. Pre-product-market-fit / very early stage

Don’t invest in CRO yet. Focus on qualitative research: talk to users, analyze heatmaps, and recordings. You probably can’t run A/B tests yet, but there are plenty of other ways to test and learn.

2. Early growth (some traction, limited traffic)

A freelance strategist or consultant can help identify what to optimize and set up lightweight experiments. You likely don’t have enough volume for a full agency program or to justify that level of spend.

3. Scaling (meaningful traffic, proven model)

This is where the agency vs. in-house debate usually kicks in. If you’re testing the value of experimentation or don’t have many in-house resources, start with an agency. If experimentation has proven its impact, begin building in-house.

4. Mature (experimentation is a core capability)

At this stage, your goal should be an in-house team, potentially supplemented by specialist freelancers for specific skills. Agencies should only come in for discrete projects, capability gaps, or short-term support.

Are we building a system to learn?

At the end of the day, this isn’t a game of bingo, and there’s no single “better” model. The goal is to build a system that helps you learn.

You can’t simply buy CRO or outsource it; it has to become part of your company’s DNA. Your internal team needs to absorb the knowledge, get smarter with every experiment, and embed that learning into future decisions. Only then will it truly scale.