ab-testing-platform.js·5 scenarios
A/B Testing Platform — Optimizely/GrowthBook-class experimentation system: Client SDK with in-process bucketing, Control Plane with config CDN and SSE kill-switch, Event Ingest via Kafka+Flink, Stats engine with mSPRT sequential testing, Druid for real-time agg, Guardrail auto-stop. 5 scenarios + 2 ADRs.
A/B testing platform is the system that lets product teams change a product scientifically instead of shipping by opinion. It looks simple from the outside: define variant A and B, split traffic, collect conversions, show a dashboard. At real scale it becomes a distributed decision system with strict latency, huge event volume, statistical correctness, guardrails, privacy boundaries, and operational rollback.
The important scale assumption is asymmetric load. Experiment assignment can happen tens of times per page view because every page may check feature flags, layouts, ranking models, prices, and onboarding variants. With 500M monthly users, 30 page views per user per day, and 50 checks per page, assignment can reach hundreds of billions of evaluations per day and tens of millions of peak RPS. The assignment path must be local, deterministic, and almost free.
Events are the opposite: exposures, clicks, purchases, revenue, errors, latency, and retention signals arrive as a massive stream that needs deduplication, joins, aggregation, and statistical analysis. This case trains ::concept{slug="stream-processing"}, ::concept{slug="ad-click-aggregator"}, ::concept{slug="model-serving"}, guardrail thinking, and the difference between online serving and offline truth.
Separate the platform into three loops. The first loop is assignment: given user, context, and experiment config, decide variant deterministically in under a few milliseconds. The second loop is measurement: log exposures and outcomes, join them correctly, aggregate metrics, detect sample-ratio mismatch, and feed dashboards. The third loop is control: create experiments, publish configs, roll back bad variants, and stop experiments when guardrails are breached.
The main invariant is sticky assignment. The same eligible user should normally receive the same variant for the same experiment until the allocation or targeting rules intentionally change. A typical implementation hashes user_id + experiment_key into a large bucket range, then maps bucket intervals to variants. This avoids a central assignment database on the hot path.
Exposure is not the same as eligibility. A user should count only after the product actually had a chance to show the variant. Logging an exposure too early biases results; logging it too late loses attribution. This one detail decides whether the platform is trustworthy.
The diagram shows a client or app using an Assignment SDK. The SDK pulls versioned experiment config from a Config CDN or cache and evaluates locally. The Experiment Service and Admin UI form the control plane: PMs define variants, allocations, targeting rules, metrics, and guardrails. The hot assignment path does not call the Experiment Service directly.
The event path starts with exposure and outcome events flowing to collectors and Kafka. Flink joins, deduplicates, checks sample-ratio mismatch, and produces real-time aggregates. Druid or ClickHouse serves low-latency dashboards. S3/Iceberg stores raw and joined event history for re-analysis. A Stats Engine reads the warehouse and computes confidence intervals, CUPED adjustments, sequential tests, and final recommendations. Guardrail Auto-stop watches operational and business metrics and can push allocation changes back to the Experiment Service.
The diagram also highlights a kill-switch channel. Normal config TTL can be 30 seconds, but a dangerous variant needs faster propagation. This is why many platforms use a separate push or short-TTL path for emergency disable.
Assignment scenario teaches why the SDK-with-pulled-config pattern wins at scale. The app asks the SDK whether checkout-v3 is enabled. The SDK uses cached config, hashes the user into a bucket, returns variant B, and emits exposure asynchronously. There is no RPC in the render path, so assignment survives temporary control-plane outages.
Outcome scenario teaches attribution. A purchase event is useless for an experiment unless it can be joined to a valid exposure within a defined attribution window. Stream processing joins user, experiment, variant, timestamp, and metric context, then updates real-time aggregates and archives joined facts.
Stats scenario teaches that dashboards are not merely counters. The Stats Engine must handle confidence, power, sequential peeking, multiple comparisons, novelty effects, and variance reduction. A reported +2.3% conversion is only meaningful if the sample is valid and the test design supports the conclusion.
Guardrail scenario teaches operational rollback. If variant B increases latency, errors, refunds, or revenue loss beyond a threshold, the Guardrail service forces allocation back to control and invalidates config. The platform should protect users before a human opens the dashboard.
Sample Ratio Mismatch scenario teaches the most common trust failure. If expected 50/50 traffic becomes 53/47 at large sample size, assignment, targeting, logging, bot filtering, or event loss is broken. Results should be marked invalid, not interpreted as product impact.
Client-side or SDK-side assignment gives very low latency and removes the central assignment service from the hot path. It also means config distribution, cache invalidation, privacy, and SDK correctness become critical. Server-side assignment is simpler to audit and can hide sensitive targeting rules, but at tens of millions of RPS it adds cost and failure coupling.
Deterministic hashing gives sticky assignment without storage, but makes changes to traffic allocation subtle. If you rebalance buckets incorrectly, users can switch variants and contaminate results. A robust platform versions configs, freezes completed experiment assignments, and treats ramp-up as an explicit operation.
Real-time analytics gives fast detection and confidence for operators, but raw event history remains the source for final analysis. Real-time systems can drop, duplicate, or late-arrive events. Batch recomputation from immutable logs is slower but more trustworthy.
Frequent peeking at results is dangerous. Sequential tests or always-valid methods can support continuous monitoring, but naive p-values after repeated peeks inflate false positives. Experiment platforms must encode statistical discipline into the product, not rely on every PM to remember it.
Microsoft ExP, Optimizely, LaunchDarkly, Statsig, GrowthBook, Netflix experimentation infrastructure, Meta Deltoid-like analytics, and Google internal platforms all converge on similar principles: deterministic assignment, versioned config, massive event pipelines, self-service dashboards, and guardrails. The details differ by domain. A search-ranking platform cares about long-term retention and query satisfaction; an ecommerce platform cares about revenue, refunds, fraud, and latency; a messaging product cares about delivery, abuse, and notification fatigue.
Feature-flag systems and experiment systems overlap but are not identical. Feature flags optimize release control and kill switches. Experiment platforms optimize causal measurement. Mature companies combine both, but they keep statistical semantics visible.
Do not make every assignment an RPC to a central service. It looks clean in a small design, but at hundreds of billions of evaluations per day it becomes expensive, fragile, and too slow for rendering.
Do not log exposure when the user merely qualifies. If the variant was never rendered or used, the exposure should not count. Early exposure logging creates bias, especially when some variants affect rendering or navigation.
Do not ignore deduplication. Mobile retries, offline flushes, ad blockers, and collector retries create duplicates. Exposure keys often include user, experiment, variant, and time window; outcomes need idempotency keys or event IDs.
Do not let teams ship on unpowered tests, SRM, or cherry-picked metrics. A platform should warn about insufficient sample size, broken splits, novelty periods, and guardrail regressions.
Do not store only aggregates. Without raw or semi-raw immutable events, you cannot re-run analysis after a metric definition bug, attribution bug, bot filter change, or privacy deletion request.
Do not build a full A/B platform for a small product with low traffic and rare experiments. A hosted feature flag or experiment tool is cheaper and less risky until event volume, privacy, or custom statistics justify the investment.
Do not use A/B testing for changes where randomization is unethical, illegal, or impossible to isolate. Pricing, credit, healthcare, safety, and employment decisions may require stricter governance or different experimental designs.
Do not use online experiments when the metric takes months to mature and the product cannot wait. In those cases, offline evaluation, user research, shadow launches, or phased rollouts may be more appropriate.
Study ::concept{slug="stream-processing"} for exposure/outcome joins and late events, ::concept{slug="ad-click-aggregator"} for high-volume metric aggregation, ::concept{slug="model-serving"} for experiment-driven ML rollout, ::concept{slug="caching-strategies"} for SDK config delivery, and ::concept{slug="pacelc-theorem"} for latency vs consistency choices in assignment and analytics stores. Also read Kohavi et al., Trustworthy Online Controlled Experiments, because most production failures in experimentation are methodological, not infrastructural.