Skip to main content

Sample Ratio Mismatch (SRM)

Sample Ratio Mismatch (SRM) occurs when the actual distribution of users across experiment variations does not match the expected traffic allocation.

For example, if an experiment is set to a 50/50 split, you would expect roughly 50% of users in each variation. If instead you see something like 60/40 or 70/30 without explanation, this indicates a Sample Ratio Mismatch.

Why SRM matters​

SRM is a critical data quality issue.

When traffic is not distributed as expected, it often means that:

  • The experiment setup is flawed
  • Some users are not being tracked correctly
  • The results may be biased or invalid

If SRM is present, you should not trust the results of the experiment until the issue is identified and resolved.

Automatic SRM detection​

Lyftio automatically checks for Sample Ratio Mismatch on the results page.

  • Traffic distribution is continuously validated against the expected allocation
  • If a mismatch is detected, an alert is displayed

This helps you quickly identify potential data integrity issues without needing to manually calculate or monitor traffic splits.

How to identify SRM​

You may have SRM if:

  • Traffic split differs significantly from allocation (e.g. 50/50 → 65/35)
  • One variation consistently receives more or less traffic than expected
  • The imbalance persists over time (not just small random fluctuations)

In properly randomized experiments, small deviations are normal, but large or consistent imbalances are not.

Common causes of SRM​

1. Tracking issues​

Not all users are being counted correctly.

  • Script not loaded for all users
  • JavaScript errors preventing execution
  • Events not firing consistently

2. Conditional script execution​

The experiment script does not run for all visitors.

  • Script blocked by consent settings
  • Script conditionally loaded based on page or user state
  • Ad blockers interfering with execution

3. Redirects and navigation issues​

Users may not reach the intended variation.

  • Redirect loops or delays
  • Users leaving before assignment completes
  • Split URL tests with inconsistent page behavior

4. Bot or internal traffic​

Non-human or biased traffic can skew distribution.

  • Bots hitting only certain pages
  • Internal testing affecting one variation more than others

5. Implementation differences between variations​

Variations themselves may affect tracking.

  • One variation breaks tracking scripts
  • One variation loads slower or fails to render
  • Differences in page structure affecting event binding

What to do if you detect SRM​

If you suspect SRM:

  1. Stop the experiment
  2. Verify script installation across all pages
  3. Check for console errors
  4. Validate audience conditions
  5. Ensure tracking and goals fire correctly
  6. Review recent changes to the site or experiment

Only resume or relaunch the experiment once the issue is resolved.

Summary​

Sample Ratio Mismatch indicates that your experiment traffic is not being distributed as intended.

  • It is a sign of invalid or biased data
  • It is usually caused by implementation or tracking issues
  • It must be resolved before drawing conclusions

Detecting and fixing SRM early ensures your A/B testing results remain reliable and trustworthy.