A/B testing compares two or more variants of a feature by serving them to randomly assigned slices of live users at the same time. Each variant logs a predefined success metric — conversion, task completion, time on task, revenue per user — and a statistical test decides whether the difference between variants is real or noise. The variant that wins on the metric becomes the default for everyone.

The defining property is the randomized, concurrent comparison: because users are split at random and measured over the same period, differences in the metric can be attributed to the variant rather than to seasonality, traffic mix, or chance.

A/B testing: a randomized splitter sends half of live user traffic to Variant A (control) and half to Variant B (treatment), both served in the same time window; per-user metrics feed an experiment-analysis panel that compares a primary metric against guardrails and promotes the winning variant to 100% of users.

Both variants run at the same time on a random, even split, so the measured difference reflects the change itself — and the variant that wins the primary metric without breaching its guardrails is promoted to everyone.

How It Works

  • Define one primary success metric and the guardrail metrics that must not regress before the test starts.
  • Randomly assign each user to control or a treatment, keeping assignment sticky so a user always sees the same variant.
  • Serve the variants concurrently and log the metric per user; calculate the required sample size up front.
  • When the sample reaches statistical power, run a significance test, promote the winner, and remove the losing variant.

Failure Modes

  • Peeking: stopping the test the moment it looks significant inflates false positives, so a real-looking win fails to replicate.
  • Underpowered tests: too little traffic or too short a run reads noise as a result, and the “winner” is chance.
  • Metric myopia: optimizing a narrow metric (clicks) while a guardrail (revenue, retention) quietly regresses ships a net-negative change.
  • Canary deployment also splits live traffic between versions, but it gates on operational health (errors, latency) to release safely — A/B testing gates on a business or user metric to learn which variant is better.
  • Feature flags are the delivery mechanism both share: they route users to variants inside one running version.
  • Multivariate testing varies several factors at once to measure their interactions, at the cost of needing far more traffic.
  • Interleaving compares ranking variants within a single user’s result list, reaching significance with less traffic than A/B for search and recommendation.

References