Skip to main content

False positives in A/A tests

Why high probabilities can appear in A/A tests​

Seeing a probability of 95% or higher to beat baseline in an A/A test can feel alarming—but in Bayesian A/B testing, this is expected behavior.

In an A/A test, both variants are identical. Any observed difference in conversion rate is therefore pure random noise. However, Bayesian models still calculate probabilities based on the data they observe, even when the difference is meaningless.

This means:

  • The system is not broken
  • The math is working as designed
  • The result should not be acted on

What Bayesian probability actually means​

Bayesian A/B testing does not answer the question:

“Is this difference real?”

Instead, it answers:

“Given the observed data, what is the probability that variant B’s true conversion rate is higher than variant A’s?”

This is a conditional probability, based on assumptions and observed variance.

A probability of 95.59% means:

“If one variant were better, variant B is more likely than variant A given the data so far.”

It does not mean:

  • The effect is real
  • The test is finished
  • The uplift is meaningful
  • You should ship the change

Bayesian probability is not the same as statistical certainty.

Why false positives are unavoidable​

Bayesian models are designed to be responsive. Early in a test—or with limited data—random variation can temporarily favor one variant, producing a high probability.

This behavior is unavoidable and expected:

  • Random noise exists in all experiments
  • Bayesian models will sometimes favor one variant strongly by chance
  • A/A tests make this visible because no true effect exists

Seeing false positives in A/A tests is therefore evidence that the system is behaving correctly, not incorrectly.

Key takeaway​

  • Bayesian A/B tests will occasionally produce false positives
  • A/A tests occasionally showing ≥95% probability is normal
  • Probability alone is not a decision rule

To learn how to safely evaluate A/B test results and avoid acting on noise, see How to evaluate A/B test results correctly