Experiment Engine
Guardrails, watch-outs and traffic split
The minimums before a winner is called, watch-outs that flag a version that does worse elsewhere, and the traffic split warning.
Minimums
By default, a winner is not called before 7 days, or before each version has 100 visitors and 10 orders. Early results swing, and judging a test in its first days is the most common way to pick the wrong version. A banner explains this until the earliest verdict day.
Watch-outs
On the Goal step, Watch-outs lists metrics that should not get worse. If a version earns more of your main goal but another metric drops, the verdict is Mixed result and tells you which. Look closely before you ship it.
Traffic split
Traffic split looks off appears when visitors split differently from what you set, by more than chance explains. It makes results unreliable, so the verdict turns to We can’t trust these numbers yet. Common causes are an app embed that is off, a pixel that is disconnected, or a page that redirects some shoppers before the test runs.
Health checks
- The app embed is off, or the tracking pixel is disconnected.
- No visitors have been counted for several hours.
- Profit is hidden when costs are missing for too many sales; select Add costs.