A Guide to Collaborating With Your A/B Testing Tool Vendor to Run More (and Better) Experiments
You’ve probably heard this myth: We use 10% of our brains.
Back when most people thought it was true, we fantasized about the possibility of using more. There’s even a whole movie about it starring Scarlett Johansson — Lucy.
But with A/B testing tools, many optimization teams are definitely below 100% usage. Yes, you run tests and deploy winners from time to time, but do you have strong ties with your A/B testing tool vendor?
Beyond onboarding and providing support, your tool vendor can be your partner in experimentation. You can benefit from their expertise and experience in working with customers across industries and verticals to run more robust experiments.
This is especially important right now given the fact that budget cuts are the norm and optimization program leaders have to optimize their experimentation funding.
In this guide, we’ll show you how to collaborate with vendors more effectively, so you can get the maximum return from your testing investment.
- What Does an Economic Slowdown Mean For Your Optimization Program? (And Why Your Vendor is an Under-Utilized Resource)
5 Ways to Improve Vendor-Team Collaboration for More Tests and More Wins
- 1. Think of Your Tool (and Vendor) as an Investment
- 2. Be Strategic in Your Communication
- 3. Define Areas of Responsibility
- 4. Prepare for the Trough of Disillusionment
- 5. Take Advantage of Education & Enablement
- Key Takeaways
What Does an Economic Slowdown Mean For Your Optimization Program? (And Why Your Vendor is an Under-Utilized Resource)
“Is this another tech bubble burst?”
That’s been the headlines since May. But the truth is investors are no longer very confident they’ll make as much money from the tech industry as they put in it.
Cash investments won’t come as they used to, so startups now have to adjust to their new valuations and plan their budgets accordingly.
This isn’t just affecting startups; it is industry wide. Even Google is adjusting.
Addressing staff about it, the CEO of Google, Sundar Pichai said, “We need to be more entrepreneurial, working with greater urgency, sharper focus, and more hunger than we’ve shown on sunnier days”.
Departments will revise their growth plans to align with sliced budgets — “greater urgency” and “sharper focus”. Budget cuts might get to your optimization program, but what does that mean for your next steps?
Spend less on experimentation? Sure. But how do you do that and still pull the maximum ROI?
Test more. Increase your test velocity to better understand the changing consumer behavior. Now is the time to stop making random changes and leverage testing as a validated way of aligning experiences with what potential buyers want.
Now more than ever, you need more clarity into what works — so you can ship profit-driving changes faster and more frequently.
Scarcity breeds clarity.Sundar Pichai, Google CEO
Running more tests is just one aspect. Running these tests at a lower cost is another. And there’s more to it than just getting a more affordable A/B testing tool.
Let me illustrate with the A/B testing flywheel:
You can speed up the flywheel and lower the cost of A/B testing by improving steps 1 and 5. That is, running more A/B tests to support more decisions and lower the human cost of A/B testing.
A truly data-driven organization uses tests to support decisions and, for beginners, it’s your first test that kickstarts the flywheel. Each time you return to step 1, you’re building up on what you’ve already learned in the previous round. And round and round we go.
Step 5 is possible when you streamline the testing process by investing in improving your experimentation infrastructure (step 4). Tests should be easy to set up, require fewer manual interventions, and cost less. Results should be easy to understand and practical for non-statisticians.
That way, interest in using A/B tests to support decisions grows naturally, and more people use the A/B testing tool to find improvements in your organization. Now you’re testing more and pulling higher ROI from experimentation across various departments. Not just marketing.
You see, the impact of step 4 — investing in the A/B testing tool — makes this a win-win. And this guide is about this type of investment: Developing a collaborative relationship with your A/B testing tool vendor so that you have a partner on your optimization journey.
Your partner in optimization holds you accountable, works with your in-house team or agency to provide excellent tool-specific training, and improves your process efficiencies by sharing what worked with other successful users. This means you cut down ship time on your tests and reduce the human cost of testing even further.
With the flywheel functioning this way, there’ll be no need to copy tests. That isn’t even a thing. Instead, you iterate based on your previous A/B testing learnings and—through seamless collaboration with your A/B testing tool vendor—the learnings of others to advance your experimentation program.
You may even scoff at meta-analysis but you can’t deny the fact that where experimentation team management and structure is concerned, standing on the shoulders of others who have gone before you makes sense. There is no need to reinvent the wheel; an idea we borrowed from Positive John’s Positive Experiments newsletter.
5 Ways to Improve Vendor-Team Collaboration for More Tests and More Wins
Now for the juicy part.
Here are 5 ways you can be more than just another A/B testing tool user and create a relationship with your vendor that takes your optimization program to the next level:
1. Think of Your Tool (and Vendor) as an Investment
Rather than treating your A/B testing tool as a cost center, consider the value it will add to your program. And getting a bug-free, capable tool isn’t enough for that. You need to know how to use it to its full capacity.
That doesn’t mean diving into tons of features you’ll never need. Instead, are there ways your tool can improve the quality of the growth-advancing insights you extract that you probably don’t know about?
If you look for this value, you will find it—and that can mean the difference between getting nothing back from your A/B testing tool versus producing the impact that spins your A/B testing flywheel faster.
Besides, this will also help with the trough of disillusionment further on (more on this below) and help you keep an open mind when listening to your vendor’s feedback and suggestions.
It is essential to be open to feedback from your vendor. Their insights and observations can be extremely valuable in helping you fine-tune your optimization efforts.
Also, keeping an open mind when working with a vendor is vital. They may have different ways of doing things than you are used to, but if you are open to their suggestions, you may be surprised at the results.David Scott, CEO of Neutypechic
Find out ways you can reach the support team: Email, chat, and phone support. And which are available with your current plan.
Of course, make sure you opt for a vendor team that actually cares about your success and isn’t just making a marketing promise. They should have a legacy of stellar support, collaboration, and cooperation.
Whether you have an in-house team or are working with an A/B testing agency, this relationship with a reliable, knowledgeable vendor support team comes through in the clutch — especially during an economic slowdown.
If you’re an experimentation program manager or leader, a lot of this is indeed your job. But that doesn’t mean you can’t lean on the expertise of your tool vendor for some support. Often, they’ll have amazing customer success teams that can help train your team not only on their specific platform, but on the best practices that their most successful clients are using to get results. Why reinvent the wheel? Use their experience and pattern matching to inform your program.
This isn’t just to flatter Convert, but their ability to help educate the team and help out with integrations, test ideas, and process improvements has been one big reason I’ve continually chosen the platform over other competitors at various companies.Alex Birkett, Co-Founder of Omniscient Digital
2. Be Strategic in Your Communication
Not all communication is effective. You’re only going to see value from any collaboration when the communication aligns with specific goals, allows a smooth two-way flow of ideas, and is done through the right channels.
So, before you engage your A/B testing tool, chalk out a clear plan of action. Your strategic communication should follow these tips:
Have you experienced your first moment of “value” with the A/B testing tool? Yes? That’s great. Share it and ask for feedback. What could be better, and what are some rewarding next step ideas?
Let Concerns Be Known
If you haven’t gotten your first win, talk to your vendor. Express your concern and work with them to take you and your team to that first value moment. Or, if you’re already past the time, how can they help you find consistent value?
The alternative is to give up and find a new A/B testing tool you think will get you those wins faster.
But switching tools is a costly and cumbersome process. Avoid it till you don’t have a choice because you’ve exhausted your options, and the tool isn’t covering your most common and important use cases. Or if the tool vendor doesn’t care about your business and treats you like a revenue number on the dashboard.
Break it Down into 6-week Sprints
Break down your big A/B testing goals into bite-sized pieces to complete in a month and a half.
Focus on keeping everyone aligned with those goals — your team and your tool vendor’s customer success team. Make sure everyone understands how your organization measures success.
Structure Your Meetings
Have an agenda that sticks to the essentials. Come prepared to make the most of the limited time you have.
Keep track of your meetings by making recordings (and if you’re on Zoom, use an automatic note-taking app like Fathom).
Andra Baragan talks about a similar structure for the internal meetings at Ontrack Digital:
Make sure everyone on your team is super familiar with this structure. You’ll spend less time figuring out what’s going on and more time extracting value from your collaboration with your tool vendor.
Also, make sure you drum up enthusiasm in your team. Someone from your A/B testing program should be present in all the meetings — to probe, ask questions and answer them, as needed.
Set Up a Cadence of Meetings
Trust us when we say your A/B testing tool vendor understands the importance of you experiencing ongoing value with their app. Because if you don’t, you might switch thinking you got a bad tool.
It’s likely on their churn reduction agenda. So, why won’t you take advantage of this?
So, reach out. Arrange a meeting routine (because one meeting doesn’t cut it). This should be an ongoing relationship where you overcome challenges and come back with new ones.
Establish a routine of meetings. It’s critical to schedule time for all individuals involved in the optimization process to meet and discuss what needs to happen next on a regular basis. This will allow you to maintain track of your statistics and keep everyone informed about how your team’s efforts are progressing.Matthew Post, Co-founder of WCAG Pros
And how frequently should you have these meetings with your A/B testing tool vendor? Bi-weekly? Monthly?
Establish a regular cadence of communication with your vendor. This can be weekly, bi-weekly, or monthly, but make sure to touch base regularly so that you can stay up-to-date on new features and capabilities, and provide feedback on how the tool is performing for you.Mushfiq Sarker, CEO, StreamSEO
Find a routine that keeps pace with your experimentation program.
Establish a Single Point of Contact with Your Vendor
This is the person responsible for handling your requests and representing your vendor’s team in all matters concerning you.
It’s better than being passed from hand to hand, feels more personal, and is actually more productive in the long run.
Consider negotiating for or securing a single point of contact with the vendor. This ensures that you will have someone who thoroughly understands your demands as the length of time you utilize A/B testing expands or your priorities change. A solid working relationship with a single point of contact makes optimizing savings and avoiding costs easier.David Bitton, CMO at DoorLoop
3. Define Areas of Responsibility
Define what you and the vendor are responsible for. Outlining expectations right from the start sets the relationship up for success and speeds up your optimization progress.
While it may seem obvious, the strategy we used that had the greatest impact was establishing areas of responsibility from the start and conveying this expectation consistently throughout the engagement. This offered clarity for all parties on desired results and the resources required to accomplish those outcomes.
Pinpointing resources and desired results ensures that the budget is allocated appropriately. We also routinely assess collaborative processes to detect any concerns and address them.David Bitton
Now, it’s not enough to say who does what and when. Codify this in your legal agreement. This is why some users prefer annual contracts. It is easier to refer to the agreement and identify who dropped the ball.
But even if you don’t have a legally binding agreement, bringing this up in your orientation call will ease the road ahead.
So, keep a record of the agreement. We’re humans; we forget. And if someone else comes on board (on your team or your vendor’s) this agreement gets everyone on the same beat.
In your internal team, you can keep track of responsibilities in your experimentation program with a RACI matrix.
A RACI matrix is a chart that maps out key objectives, milestones, tasks, and responsibilities in a project for everyone involved.
RACI means responsible, accountable, consulted, and informed:
- Responsible: Who is responsible for completing the tasks such as making decisions, executing tests, recording tests, etc.?
- Accountable: Who delegates tasks and ensures they are completed on time?
- Consulted: Whose input needs to be consulted on tasks? Who is involved in providing feedback to take projects to completion?
- Informed: Who needs to be up-to-date with progress?
We’ve talked about how to build and use a RACI chart for your experimentation project. Here’s how it looks:
Share the RACI matrix of your experimentation program with your vendor. They will appreciate the transparency and will better navigate requests from your team.
You can add an extra layer of transparency by setting up a real-time tracker of your experience with the testing tool and sharing this with your vendor.
It’s also important to set up a process for monitoring progress and communicating with each other about what works and doesn’t work as you use this tool together. This is especially important if there are multiple people involved in using this tool at once (like if your marketing team uses it while your engineering team also uses it).Ryan Stewart, Managing Partner of The Blueprint Training
This can be a document on Notion, Confluence, or Google Doc including screenshots or short video recordings of your experience. Your A/B testing tool vendor will love it! And if you hit a snag while working on experimentation projects, you’ll get unblocked faster.
On that note, we sincerely hope that you are in communication with your vendor. And that they offered you an orientation session!
Your orientation session is a welcome that familiarizes you with the vendor’s team, sets expectations for the journey, and segues into your onboarding. Which is between 6 and 12 months.
Remember, scripted responses from the support team don’t make up your orientation session.
4. Prepare for the Trough of Disillusionment
You’re either in it or headed towards it.
If you haven’t of the trough of disillusionment, it’s a fancy name for the third stage of the Gartner Hype Cycle; the let-down after expectations and hype around an idea or product has hit its peak.
It’s like a harsh call to the reality that comes with humility and less confidence in your tool. But on the plus side, it makes your team more open-minded to new ideas and adjustments.
And if you’re not the only one in the trough of disillusionment, you’ll have to reassure other stakeholders that you can make something out of the money spent so far.
If you initiated A/B testing in your organization, but everyone else didn’t get a proper orientation and instead had unrealistic expectations, you might find yourself under pressure. Because A/B tests only add certainty to ideas and decisions.
If you’re already out of the trough of disillusionment, then this section doesn’t apply.
For those in it, here’s an out:
Start with mapping out your needs from the tool.
- Why did you invest in this technology/brand?
- What stage was your optimization program that prompted the need for your A/B testing tool?
- What needs were you trying to meet?
- What was the marketing message that stuck out to you and your team?
Those were your expectations and you’re in the trough because they don’t match your current reality. I’ll tell you what to do with this soon.
In my opinion, one of the most important responsibilities of a testing leader is to ensure that the organization’s expectations are appropriately managed. When it comes to the program, the level of scrutiny will increase in direct proportion to the size of the technology budget. The magnitude of the expenditure contributes significantly to the depth of the trough.
Developing a proof of value prior to making a significant investment in technology is the most effective strategy for managing expectations. Before you go out and buy a Ferrari, you should get your program off the ground by using a solution that is either cheap or free that will create some early value.Gerrid Smith, Director of E-commerce, Fortis Medical Billing
The trough of disillusionment is why massive tool budgets don’t bode well for experimentation programs unless the price tag can be unequivocally justified. For instance,
- Edge or advanced use cases are NOT the right reason to get an enterprise A/B testing app.
- Edge or advanced use cases don’t drive significant returns since the problem(s) solved improve things for a very small number of users or tests.
- But the bigger the budget, the greater the scrutiny and the expectation. You can’t escape that.
TL;DR: Don’t needlessly pump up the pressure.
If you can’t navigate the trough of disillusionment, you won’t develop a mutually respectful collaboration with your vendor. Your team members will view it as a futile exercise.
Has the trough already damaged your morale? It’s not too late. Get on a call with your vendor, lay out the expectations you’ve outlined above, express your concern, and decide on the next steps.
Don’t be afraid to ask questions when something doesn’t work like you thought it would! Your vendor is there to help and they want your business, so they’ll do whatever they can to make sure you’re happy and satisfied with the product they’re selling you!Ryan Stewart
When you get on a call with your tool vendor to discuss these, here are some talking points:
- What’s the origin of the disillusionment?
- Clearly define the “value” or “output” that’ll make the juice worth the squeeze (that is, justify the price tag)
- Is it possible to realize this value with the tool?
- If yes, what will this timeline look like?
- Who will be responsible for what?
- How frequently will you touch base to discuss progress?
- What grievances does your team have? Allow your vendor to address them.
If you’ve got a great vendor, this will be a fruitful conversation that brings you a fresh perspective and points you towards a successful optimization program.
Our users love the fact that they get Dennis (our CEO) and Claudiu (our CTO) on call. So, if this ever comes up, this conversation with our execs is just a call away.
5. Take Advantage of Education & Enablement
Successful A/B testing requires some technical know-how. That’s why terms like statistical significance look foreign at first glance for folks not well versed in statistics.
To get the most out of A/B testing, you need to have a clear idea of
- How randomized online experiments (like split testing and A/B testing) work
- How to use your tools to plan, set up, and execute tests; and
- How to interpret test results, extract insights, and implement them to consolidate your wins.
Education is absolutely vital.
Acquiring some of this knowledge is all on you and your team. But what if you have an A/B testing vendor that’s happy to assist you on this? Even better!
It should be a criterion when choosing an A/B testing tool.
When selecting an A/B testing tool vendor, you should look at it like an integrated and complementary partnership. While that sounds banal, most buyers are simply looking to check a few boxes. Does it enable personalization? Does it integrate with Google Analytics? How much does it cost?
These things are important, but they’re table stakes. I’ve worked on several A/B testing programs, and I can tell you that it’s rarely the technical side of things that limit the program. Rather, the limiting function tends to be the cultural side: education, enablement, process optimization, and accessibility.
So when working with a vendor, I like to audit my current program and see what the biggest limiting dimensions are. In most cases, this has been education and enablement. Simply put, not enough people on the team have been well-trained on experimentation platforms nor the process by which an experiment should be ideated, run, and analyzed.Alex Birkett
To outline what you need from your vendor in terms of education and enablement resources, pinpoint what you already have (experience and talent) and share your optimization goals with them so they’ll recommend resources to fill in the gap.
Experimentation vendors should provide you with
Get onboarding that’s customized to your specific use cases (so you don’t spend time learning what you don’t need) and have them point you to relevant support docs to get you started.
Often, the cost of onboarding is baked into the cost for the first year or charged separately.
- Tool training
This is a no-brainer, but it’s sometimes overlooked. Make sure you receive training on how to use the tool before you start using it.
- Access to a network of experts
Here’s another idea to expand your enablement: Leverage your vendor’s network to find CRO agencies/experts to scale with. Even just following them on LinkedIn will expose you to a world of insights that you can take home to your optimization program.
You’d also want to get featured in their case studies and marketing drives — it is free publicity after all. The more enthusiasm you can ignite for what your CRO program is doing for your brand, the more buy-in currency you collect.
- Learning resources
Explore free resources like support articles, webinar recordings, educational blog posts, etc.
Make use of all the resources your vendor offers. In addition to the actual A/B testing tool, most vendors offer services like consulting, training, and support. Leveraging these resources can help you get the most out of your investment and optimize your testing program for maximum impact.Mushfiq Sarker
Then, there’s even better: Find out if they provide special access to external training.
Convert offers the CRO mini-degree from CXL (for free) and GoodUI access at a reduced cost.
The true brilliance of your optimization program could be in a profitable relationship with your A/B testing tool vendor.
It’s a success driver when you tap into the expertise they have from working with other teams from various industries to grow your program. But this collaborative relationship has to be structured.
You have to
- Get the mindset that your A/B testing tool is an investment
- Use effective communication to facilitate a cross-pollination of ideas
- Define areas of responsibility
- Prepare for the trough of disillusionment, and
- Take advantage of education and enablement resources to up your game.
Not only do optimization program leaders get to optimize their experimentation budget this way, but they also get to create valuable insights that provide immunity against the budget cuts happening in the current economy.
Scan this QR code and take this blog with you, wherever you go.