Incrementality for Small Stores: Holdouts

An offer report tells you how many orders used a discount. It cannot tell you how many of those orders would have happened anyway. That second number is incrementality, and it is the one that decides whether an offer made money or only gave some away. The standard way to measure it is a holdout: a group of customers who could have received the offer and deliberately did not. This page is the small-store version of incrementality ecommerce testing, with no data team and a customer list in the hundreds or low thousands. It covers how to split the list fairly, how Shopify's customer segments help, how many customers and how long you need, and how to read the result as profit per customer. The numbers in the worked example are invented for arithmetic.

Incremental versus attributed, in one example

Suppose an email goes to 2,000 past buyers offering a free gift above a cart value, and 100 of them order within two weeks, 80 of them using the offer. The offer report shows 80 orders. Is that 80 sales created?

Only if none of those 80 would have ordered without it. Some of them were on the way to buying regardless, because past buyers reorder. If a group of 2,000 similar customers who did not get the email produced 100 orders in the same two weeks, the offer created nothing: it gave a gift to people already buying. If that group produced 80, the offer created 20 orders on 2,000, a lift of 1 percentage point. The answer lives in the group that did not get the offer. That group is the holdout.

How to Measure Whether a Promotion Worked compares time windows and shows how to catch pull-forward. A holdout compares two groups in the same window, which removes the season and the weekday from the comparison entirely. That is why it is the stronger test, and why it is worth the effort when the decision is big enough.

Designing the split

A holdout is only as good as the split. Three rules keep it fair.

1. Choose the population before the split. Pick the group that could receive the offer, such as customers who ordered in the last 180 days and are subscribed to email. Everyone in the population goes into one of two groups.
2. Randomize, or alternate down a sorted list. Put the customers in a spreadsheet, add a column of random numbers and sort by it, then alternate: treatment, holdout, treatment, holdout. Or sort by last order date and alternate down the list, which also balances how recently each group bought. Do not split by anything the customer did on their own, and in particular not by "people who did not open the email". That group is made of people who are different from the openers before the offer starts, so any difference you measure is not the offer's.
3. Decide before the offer who is in each group, and do not move anyone. Moving customers after you see who ordered turns the test into a story.

Keep the holdout the same size as the treatment group when you can. A 50-50 split gives the most statistical power from a given list. If a 50 percent holdout feels too costly, 20 to 30 percent works with a larger list, but it widens the noise on the smaller group, so use the table below to check.

Limiting an offer to a segment in Shopify

Shopify's Help Center says customer segments let you offer discounts to groups of customers who have similar characteristics, that an automatic discount can have up to 5 customer segments and a discount code up to 100, and that customer segment eligibility is a filter on the discount list. The segmentation page says segments are dynamic: customers are automatically added to or removed from a segment as they meet the criteria. That is useful in general and a trap for a holdout, because a holdout must not change after you assign it.

The solution is to make the criterion something that does not change. Shopify's segment query language supports filtering by customer tags, with a clause such as customer_tags CONTAINS 'vip'. So tag the treatment group and the holdout group once, with tags such as offer-test-a and offer-test-b, and build the segment on the tag. Segment filters on order history, such as number of orders or last order date, move with every purchase and are not safe to use for the assignment.

Two cautions. First, the pages we read do not describe how a shopper who is not recognized, such as a guest who has not logged in, is treated when a discount is limited to a segment. Test with a logged-out cart and with a logged-in cart before launch, and check what each sees. Second, a segment limits who can use a native Shopify discount. It does not stop the holdout from seeing a sitewide banner. For the holdout to be clean, deliver the offer only to the treatment group, by email or SMS to their addresses, and do not advertise it to everyone. Check whether the tool you build the offer with can be limited to a segment, because a gift that adds itself to any qualifying cart reaches the holdout too, and we do not claim our templates can be limited this way. If yours cannot, use a native segment-limited discount for the test, or fall back to matched weeks.

How long, how many

The answer depends on how often your customers buy. As a plain-language guide, run the offer for at least one full purchase cycle of your usual repeat customer, often two to four weeks for a store whose customers reorder monthly, and then watch the weeks after. How to Measure Whether a Promotion Worked covers the comparison windows and the dip afterward, so this page does not repeat it.

For how many, use the noise band. Suppose the baseline purchase rate in your population is 4 percent over the window. The standard error of the difference between two groups of n customers each is the square root of 2 × 0.04 × 0.96 ÷ n. Twice that is a rough band inside which a difference could be chance.

With 500 customers in each group, an offer has to lift the purchase rate by more than 2.5 points, from 4 to 6.5 percent, before you can tell it from luck. A small store with a list that size can only detect a big effect. That is not a reason to skip the test. It is a reason to test offers that are meant to make a big difference, and to treat a smaller measured lift as a direction, not a finding.

Reading the result: lift per customer, then profit per customer

Do the arithmetic per customer, not per order, because the offer may change who buys as well as how much.

Using the list of 2,000 per group, a 55 percent gross margin and a gift with a landed cost of 4.20 dollars given on 70 of the treatment orders:

The lift is about 0.50 dollars per customer, roughly 991 dollars over the group. The purchase rate went from 4.0 to 5.2 percent, a difference of 1.2 points, and the band for 2,000 per group is about 1.2 points. So the result is right at the edge of noise. The honest reading is that the offer probably helped and the test does not prove it. Extend the list, repeat the test, or run it longer before you commit the gift to every customer.

If you measure only revenue, you can miss the other half. A lift in orders that costs more in gifts and discounts than it adds in margin shows up in profit per customer and in no other number. AOV vs Profit per Order: What to Track explains why, and Break-Even Discount: Extra Volume You Need gives the volume a discount must add.

When a holdout is not possible

Some stores cannot withhold an offer: the banner is sitewide, the list is too short, or the owner will not leave customers out. Then use matched weeks, with the same weekdays, the same season and no other promotion in either window, and accept that the answer is weaker. The same noise rules apply, so record the number of orders in each window.

Setting up the gift

Start from the Spend X, get a gift template in Freebies Engine by Log Studio Lab, described in the catalog as "Shoppers who reach a cart value get a gift added automatically." Set the threshold and the gift cost from your own numbers. Because the gift is added automatically to a qualifying cart, check the setup steps for an audience or eligibility option before you rely on it for a holdout. For a first read on a gift, Free Sample With Purchase: A Test Plan is a simpler test. Plan requirements are on the pricing page.