Shopify Announced 150+ Updates, Including Rollouts as A/B Testing. What Does This Mean for the Experimentation Space?
In case you didn’t notice, there is a consolidation rush happening.
The world is moving towards what we like to call the “zero-click” everything.
While consumers want their insights without clicking through to websites, the SaaS landscape is collapsing into agent-handled workflows and micro-apps that allow a few big platforms to purportedly build an entire ecosystem of capabilities — into one unified solution.
Shopify’s ‘26 Winter edition plays to the front row.
Let’s look at what each announced upgrade means for the experimentation space.
Shopify’s RenAIssance Era. Released 10th December 2026.
Shopify goes ‘AI-native’.
This doesn’t come as a shock. We were all expecting it.
Straight from the giant’s mouth:
Build faster with an AI assistant that deeply understands the Shopify platform, powered by the same Dev MCP Server whether you’re in Cursor, Claude, or the dev assistant on shopify.dev. AI agents can now handle the full development workflow end-to-end: scaffolding apps, running GraphQL operations, and generating validated code across Admin, UI extensions, Liquid, and Hydrogen
While on the surface, it is focused on empowering their thriving community of developers with the freedom and the fluidity they need to vibe-code their next creations, there is a lot to unpack for experimentation and growth practitioners as well.
The Elephant in the Room: Shopify’s A/B Testing
Only it’s not dedicated A/B testing. It is Rollouts.
This Edition, we are opening up our new Rollouts feature in early access. Rollouts gives you the ability to stage a set of changes to your online store and schedule it for a specific time….
Within Rollouts, we’ve snuck in a very handy feature: you can control the percentage of traffic you want the rollout to apply to and can simultaneously launch more than one set of features. This in effect turns a rollout to an A/B test.
What are Rollouts?
Feature rollout is a software development strategy for gradually introducing a new product or feature to users. Its main goal is to ensure the product—whether a minimum viable product (MVP) or a fully-developed feature—performs as intended before reaching the entire user base.
Rollouts are for already-validated products & ideas. A/B testing is about probing assumptions, about questioning the intent to “spend & build”, and about learning what works for diverse audiences.
About a year ago, we asked Anastasia and Michael to explain how they work with rollouts at KonversionsKRAFT. Their elegant approach clarifies the difference.
Sometimes the step [after a successful feature A/B test and] before the native development would be to put the variant on 100% of audience traffic.
However, we highly recommend limiting the timeframe for this adjustment for two reasons:
For some tools this means really high costs due to the increased amount of impressions.
The experiment code is still fragile and the changes on the control-side can affect the variant functionality.
What we practice a lot is the phased rollout strategy that follows the following logic.
We might:
- Rollout to a smaller amount of users and then gradually increase the amount of the affected traffic (soft roll out)
- Rollout to one segment of users (for example if we want to reduce the quality assurance effort). The segmentation can be both technical (by browser/device) or socio-demographic (a customer group with a specific consumer behavior).
- Rollout in one country as a starting point.
Rollouts & A/B testing are complementary. Kathryn Mueller from ROI Revolution says:
The Rollouts feature allows ecommerce merchants to A/B test site changes directly within Shopify – an excellent starting point for teams new to optimization.
And for brands experienced in conversion optimization, Rollouts can complement a traditional A/B testing program for a layered approach: using Rollouts for theme-level changes while leveraging Convert Experiences for sophisticated testing that drives measurable revenue impact.
Here’s what Convert’s lead developer, Ahmed Abbas, who has been studying the updates since the announcement hit, commented:
Shopify’s Rollouts enable merchants to stage theme changes and control what percentage of traffic sees them — primarily to avoid breaking their store during deployments.
It’s fundamentally designed for risk mitigation, not conversion optimization. The friction is high: testing a headline change requires duplicating your entire theme.
Rollouts and A/B testing are complementary — one deploys safely, the other discovers value.
Remarkably similar to what Ecom CRO & Experimentation expert Gijs Wierda chimed in with:
They mention that testing ‘naturally expands’ as you get more traffic, and the focus on ‘Rollouts’ suggests a workflow based on version control. This implies that a developer needs to code a full theme variation to run a test, rather than using an agile tool for direct DOM manipulation. It also seems to lack essential CRO features like granular targeting or custom goal tracking.
There is very little to go by right now.
And it isn’t clear what targeting, segmentation, and goal options Shopify will make available.
Our CEO Dennis van der Heijden however sees opportunity in Shopify’s Rollouts:
Shopify’s theme testing was hacky at best. The Rollouts – in whatever capacity – might unlock more elegant solutions and architecture for a series of experimentation challenges that currently stumps testers & specialist tool vendors. Convert is looking into options to improve its own platform taking advantage of the new updates.
Experimentation legend Craig Sullivan is cautiously optimistic.
There’s very little detail on this, but it could be a game changer for Shopify Merchants, if it’s robust and trustworthy.
Having the capability to run A/B tests instead of a 100% rollout, is an excellent way to ‘de-risk’ deployments, as many of us testers already know – so the fact that they are building this as a native feature, will massively increase experimentation on the platform. My only worry is that there are very few details on how Shopify will ensure that the results are reliable, particularly for stores that have lower traffic levels. Of course this will probably be taken care of if you have higher volumes, but I worry that if not implemented with guardrails, it could lead to shopify stores making poor decisions based on limited data.
And Enavi’s human CRO proponent Anthony Morgan says:
As with anything, we need to consider the upsides and the downsides. On the upside — more Shopify stores will talk about experimentation. They will be exposed to A/B testing. Once they get a taste for it, hopefully the drive will be there to improve the confidence of all decisions they take moving forward. Today, only 20% of Shopify stores test.
Out of that 20% though, a handful test well. My reservations stem from the trend of self-service testing intensifying — leading to even more false positives and winners that do not move the needle.
We’ve recently run a panel on this very problem. The sugar-rush of A/B test wins that do not improve KPIs: Watch it here
What Does Shopify’s New Rollouts as A/B Testing Feature Mean for the Experimentation Space?
- Experimenters want store owners to note that rollouts are not designed for A/B testing.
- Vendors & experimenters are happy that A/B testing is going mainstream. The exposure from Shopify elevates it from a “nice-to-have” to a necessity.
- The space is waiting to see what learning & targeting tools are available with the update and whether the platform can support robust, statistically-savvy A/B testing.
SimGym: Synthetic QA Tool
SimGym is an intriguing synthetic QA tool that uses AI shoppers trained on Shopify-wide data to stress-test store changes before launch.
Think of it as a pre-flight check — useful for catching obvious UX problems (broken buttons, confusing flows) before a Rollout. However, simulated behavior is not real customer data.
Shopify though – out of all providers – has the data depth & reserves to model a believable facsimile of human response. Recent research shows that LLMs are improving in this regard.
Craig Sullivan layers a different perspective:
Note the phrase ‘results might differ from actual customer behaviour’ is doing some heavy lifting in their announcement. But this might actually outperform humans at stopping them from shipping bad UX work. In my experience working with over 30 stores, product quality is an issue that AI can help to improve, especially if they don’t have enough traffic to AB test their changes.
All-in-all, Emily Isted of HypeDigital sums it up:
SimGym is where things get interesting. Pre-validating changes with AI shoppers could become a useful screening layer, though I’m watching closely to see how well-simulated behaviour translates to real conversion impact.
Tangle: Open-Source ML Experimentation Platform
This is probably the update least exciting to growth & experimentation practitioners.
Tangle is an internal ML pipeline tool Shopify has open-sourced — essentially infrastructure for data scientists building machine learning models. It allows Shopify engineers and ML teams to iterate on recommendation algorithms, fraud detection, and search ranking.
The choice to go open-source is interesting. This may imply user interactions will add to the sophistication & robustness of a basic platform.
It definitely indicates that Shopify is building a moat and seeking its slice of the AI infrastructure monopoly pie.
Shopify Heatmaps: New Way to Visualize Report Data
This isn’t the investigative heatmap we love.
Think turning human behavior into a snapshot of interactions with the site – via the visualization of clicks, taps, and scrolling.
Shopify’s heatmaps will take the data from its analytics engine, and help you see its distribution better. It sits more as a “new report type” than actual user research.
Unlisted Products & Checkout Customization per Market
Unlisted products (direct-link-only items) open possibilities for stealth price testing — you could create product variants only accessible to test segments.
While checkout customizations personalize checkout and customer account pages for different countries and B2B buyers, directly in the editor!
Together, sophisticated merchants can hack together segmented pricing tests. Something that shifts the definition of what is possible when you add a specialist tool like Convert, and its price testing power to the mix.
150+ Updates Mean What?
It’s good news for the Conversion Rate Optimization & experimentation industry.
Traditionally, we’ve spent thousands of dollars educating clients & customers about why testing or any sort of pre-launch validation is a no-brainer.
We can begin to conserve those funds now.
Especially clubbed with Google’s recent Optimize requiem, many more stores/businesses will know A/B testing is important.
The focus moving forward should be on promoting how to test well, and doing so with thoughtful & proven-effective AI support.
So, what’s your take?
Written By
Trina Moitra