Segmentation

What is Segmentation?

Segmentation means splitting your users into different groups based on shared traits like demographics, behavior, or device type.

In experimentation, you can use segmentation before a test (to design it) or after a test (to better understand the results). Either way, the goal is the same: to see how different groups respond to your changes.

Segmentation can be based on almost anything you can measure reliably, including:

• Demographics like age, income, or education
• Geographic location
• Device type, operating system, or browser
• User behavior, like how often they visit or what features they use
• Acquisition source (e.g., ads, organic traffic)
• Subscription length or customer tenure

The most important part isn’t how many segments you create. It’s whether they are relevant and actionable for the decisions you want to make.

“Segmentation helps you speak the same language as your customers. Your messaging and campaigns will resonate better if you group customers by similar attributes, characteristics, behaviors, and needs.

You can split your customers into segments based on attributes such as acquisition channel, time zone, or Recency, Frequency, and Monetary Value (RFM.) But you should choose segments that are relevant and actionable.

Showing sandals to people in warmer regions makes sense if you sell shoes. But if you sell software, weather is less relevant than job seniority or company size.

Start by thinking about how you’ll gather customer data, and then prioritize your initiatives based on segment size and value. Segmentation is also essential in the post-test analysis to help you get deeper insights. For example, test metrics segmented by new vs. returning users, desktop vs. mobile, Android vs. iOS.”

Carmen Albisteanu, Growth Strategy Analyst at Salt Bank

Why Does Segmentation Matter in Experimentation?

Segmentation plays a role before and after your test.

Before a test, segmentation helps you:

• Define eligibility criteria. You want only users who will experience the change meaningfully.
• Ensure randomization works. Good randomization spreads different user types evenly across variants.
• Build more inclusive products. Some changes may affect certain user groups differently—good segmentation catches that risk early.

After a test, segmentation helps you:

• Find out if different groups responded differently.
• Detect issues like bugs that only affect certain browsers or devices.
• Discover hidden wins—or problems—that aren’t visible at the aggregate level.
• Answer stakeholder questions about how a change impacts key customer groups.

For example, if a test looks neutral overall, segmenting by mobile device could reveal that Android users had a negative experience while iOS users saw a boost.

Without segmentation, you risk rolling out changes that help some users but hurt others and not even realizing it.

Best Practices for Segmentation

Good segmentation isn’t about slicing your data thinner. It’s about choosing smart, meaningful cuts.

• Pre-select important segments. Decide before the test starts.
• Use fixed attributes. Don’t segment by things the test itself could change.
• Keep randomization clean. Make sure users are evenly distributed across variants.
• Watch for false positives. Correct for multiple testing if you slice data many ways.
• Validate surprising findings. Always consider running a follow-up test if you find something unexpected.

Also, make sure results are easy to understand for everyone. Use clear visuals and simple language in post-test reports.

And if your test uses triggering (only showing a change to users who take an action), always check how the non-triggered group behaved too. It’s a critical validation step.