A/B Testing Offers: Sample Size, Stopping

An offer test fails in two ways. The first is stopping too soon: the variant is ahead on Tuesday, you call it, and the lead was noise. The second is never reaching a verdict because the store does not have the traffic. Both come from skipping two numbers before the test starts, the sample size each arm needs and the day you will stop. This post covers A/B testing promotions on a small Shopify store, with a sample size table you can check, a plain explanation of why daily peeking makes false wins, a stopping rule to write down in advance, and what to do when the store is too small.

What is being compared

An A/B test splits visitors into two groups at random. Group A sees the store as it is. Group B sees the same store plus the offer, such as a gift tier. The comparison is between the groups over the same dates, so a holiday or a weekday cannot explain the difference.

Choose the one metric before you start. Conversion rate, the share of visitors who order, is the simplest, and the sample size math below is for it. Revenue per visitor includes order value, which varies a lot from order to order, so it usually needs more visitors for the same confidence. If you test an offer that changes margin, such as a free gift, a lift in conversion can still lose money, so judge the result in profit per visitor and use AOV vs Profit per Order to define it. Write the metric down. If you pick it after seeing the results, you will pick the one that looks best.

How many visitors each arm needs

The NIST/SEMATECH e-Handbook of Statistical Methods gives the sample size for a one-sample test of a proportion in section 7.2.4.2. For a two-sided test it is N at least ((z for alpha over 2 times the square root of p0 times (1 minus p0), plus z for beta times the square root of p1 times (1 minus p1)) divided by delta) squared, using the normal approximation, where delta is the change you want to detect. Section 7.3.3 gives the matching two-proportion test, whose statistic divides the difference between the two sample proportions by the square root of the pooled proportion times (1 minus the pooled proportion) times (1 over n1 plus 1 over n2), and says that if the samples are reasonably large the normal approximation to the binomial can be used.

NIST does not print a ready table for two proportions, so the figures below are our adaptation of those formulas to two groups of equal size: a two-sided test at the 5 percent level with 80 percent power, using the normal approximation. The sample size is per arm, in visitors.

Read the 3 percent row as an example. A store converting at 3 percent that wants to detect a rise to 3.6 percent, a 20 percent relative lift, needs 13,914 visitors in each arm, 27,828 in all. Aiming at a 10 percent lift, from 3.0 to 3.3 percent, takes 53,211 per arm. Halving the lift you want to see roughly quadruples the visitors you need, which is the reason small effects are expensive to confirm.

The same math run the other way tells you what a given amount of traffic can detect. At a 3 percent baseline:

A store with 500 visitors a day, split evenly, gives each arm 250 a day, or 7,000 in 28 days. That detects a lift of about 29 percent at a 3 percent baseline. A gift tier that lifts conversion by 10 percent is real and invisible to that test. These figures are examples of the method. Use your own baseline.

Why checking every day makes false wins

A test at the 5 percent level is designed to be read once, at the end. If there is no real difference, a single read at the planned sample size declares a winner about 1 time in 20. Reading every day and stopping at the first read that looks significant is a different test, because each look is a new chance for noise to cross the line.

We ran the numbers rather than quote anyone's figure. Take two identical arms, both converting at 3 percent, with 500 visitors per arm per day for 28 days, so no offer has any effect. We simulated 4,000 such tests and applied the same 5 percent test every day. In roughly 27 to 28 percent of them (two runs with different random seeds gave 28.2 and 27.1 percent), at least one daily look declared a significant difference. Looking only once, on day 28, flagged about 5 percent, the rate the test promises. So a rule of "stop when it is significant" turned a 1-in-20 false win into more than 1 in 4. The 27 to 28 percent is a result of this setup, not a constant, and it grows with the number of looks.

The lesson is not to avoid looking. It is that looking cannot change the decision. Check traffic and errors every day if you want, and decide only at the planned end.

A stopping rule you write before the start

Put four lines in a note before the first visitor is split, with the date.

1. Metric: conversion rate, or profit per visitor, one of them.
2. Sample: the visitors per arm from the table, rounded up to whole weeks so each weekday appears equally often.
3. Stop date: the day that sample is expected to be reached at your normal traffic, plus no extension for a promising result.
4. Decision: ship the variant if it beats control on the metric and the gap is larger than the noise, keep the control if not, and call it inconclusive if the gap is inside the noise, without a second reading of the same data.

One early stop is reasonable: harm. If the variant is clearly losing money, for instance the gift costs more than it earns on every order so far, end the test and say so in the note. Stopping early for a win is the thing that inflates false wins.

When the store is too small

If the table says you need 14,000 visitors per arm and you get 7,000, you have three honest options. Test a bigger change: a gift tier at a visibly lower threshold, not one that differs by a dollar. Run longer, accepting that the season may change. Or give up the split and compare periods. Incrementality for Small Stores: Holdouts shows how to hold back a group of past customers when visitors are too few, and How to Measure Whether a Promotion Worked covers matched weeks, with their weaker conclusions. Conversion Rate Benchmarks can help you judge whether your own baseline is stable enough to use in the tables above.

How you split visitors is up to the tool you use, such as a theme split, a landing page test or alternating weeks. Whatever you use, assign people at random and never let the same person see both versions.

Building gift tiers

For a test of a tiered gift, start from the Gift tiers template in Freebies Engine by Log Studio Lab, described in the catalog as "More spent, more gifts, in as many steps as you like." Keep arm B to one change, such as adding a tier, so you know what caused the result. Plan requirements are on the pricing page. To plan a first simple test, Free Sample With Purchase: A Test Plan is a good start.