Measuring recovery with a holdout group
What an attributed order can't tell you
ACR credits an order to a campaign when it used the campaign’s discount code, or when the shopper engaged with the campaign inside the attribution window, 14 days by default. That rule, set out in how ACR counts a recovered order, records which campaign touched the order. It cannot say whether the shopper would have bought without hearing from you, and some would have: they meant to come back, and the text arrived first.
A holdout measures that part. You keep a share of abandoned checkouts away from the campaign, count orders in both groups the same way, and read the difference as a range.
Split by shopper, before anyone is contacted
A coin flip for each shopper is the ideal, because it leaves only chance between the groups. ACR has no built-in no-contact group. Here is what its tools can do instead.
| Tool | No-contact group? | What to know |
|---|---|---|
| A/B test | No | Splits shoppers at random between two versions of a campaign, and both versions contact them. If one version cannot reach a shopper, ACR offers them to the other. |
| Start condition | Close to it | Start conditions have no random field. A rule on something unrelated to buying, such as the first letter of the email address, can stand in for one. |
| Dispatch limit, or the trigger switched off on set days | By date | Shoppers past the cap, or on the off days, are not contacted. The split follows the calendar, so a sale or an ad push can land in one group only. |
We would use the start condition. Pick a few starting letters for the held-out group and add a condition that leaves those email addresses out of the campaign. Then:
- Expect uneven groups, because letters are not spread evenly, and compare rates rather than totals.
- Remember it is not truly random. Names follow patterns of language and region, so check that the two groups look alike on cart total and country before you trust the result.
- Decide where shoppers with no email address go. A condition on a field the shopper does not have counts as not met, so write the rule to put phone-only shoppers where you intend, and count them the same way later.
- Make sure no other campaign on the same lead source picks up the held-out shoppers, because a shopper one campaign skips is offered to the next in line.
Held-out shoppers appear in their history as skipped, with the reason “Start conditions not met for this campaign.” A shopper every campaign skipped does not count toward your plan’s runs.
Count orders the same way for both groups
ACR’s credited orders cannot be the scorecard, because a held-out shopper never has a run and so can never be credited. Count from your store’s records instead:
- Take every abandoned checkout from the test period. Shopify’s help center says abandoned checkouts older than three months are deleted from the admin, so pull them before then.
- Mark each one with the group the letter rule put it in.
- Match each shopper to orders by email address or phone number, and count the ones placed within 14 days of leaving checkout. Fourteen days matches ACR’s default attribution window.
- Keep everyone the rule put in the campaign group, including shoppers ACR could not reach for lack of consent or a phone number. The held-out group has people like them too, and dropping them from one side tilts the result.
Run it long enough
How long depends on how many checkouts you get and how small a difference you care about. A sample size calculator such as Evan Miller’s takes a baseline conversion rate and the smallest change worth detecting, and gives the number of shoppers needed “per variation”. For the baseline, use your best guess at how often uncontacted shoppers buy. The calculator assumes two equal groups, so with a smaller holdout, get at least that many shoppers into the holdout itself.
Then run whole weeks, so every weekday appears equally often. Change nothing in the campaign while it runs, and keep counting for 14 days after the last shopper enters, so everyone gets the full window.
Do the arithmetic
Here is a four-week test at a store we made up, with 5,000 abandoned checkouts split roughly 80 to 20.
| Campaign group | Held out | |
|---|---|---|
| Shoppers | 4,000 | 1,000 |
| Ordered within 14 days | 380 | 70 |
| Order rate | 9.5% | 7.0% |
| Orders ACR credited | 210 | None possible |
The campaign group ordered at 9.5% against 7.0% for shoppers nobody contacted, a lift of 2.5 points. On 4,000 shoppers that is 100 extra orders. About 280 of the group’s 380 orders would have happened without the campaign. ACR credited 210, more than twice the lift; the gap is shoppers who were coming back anyway and clicked, replied or used the code on the way.
Putting a range on the lift
Both rates carry sampling error. The standard error of the difference is √(0.07 × 0.93 ÷ 1,000 + 0.095 × 0.905 ÷ 4,000) ≈ 0.0093, or 0.93 points.
A 95% interval is the lift plus or minus 1.96 standard errors: 2.5 ± 1.8 points, so about 0.7 to 4.3 points.
On 4,000 shoppers, that is between about 27 and 173 extra orders.
At an example average order of $90, the extra orders are worth about $9,000, somewhere between roughly $2,400 and $15,600, against the $18,900 ACR credited. Set what recovery costs against the margin on those extra orders. The test had a price as well: the held-out shoppers missed the lift, about 25 orders here.
Where ACR's A/B tests fit
A holdout tells you whether recovery adds orders. An A/B test tells you which of two versions adds more. ACR’s tests split new shoppers at random between two copies of a campaign until a target number have been sent, and show each version’s sends, runs, engaged runs, recovered runs, recovery rate and revenue. Two cautions when you read them:
- “Recovered” counts runs that ended in a credited order, a positive reply or a hand-off, so it is broader than orders.
- Revenue is credited revenue, under the same rule as everywhere else. If only one version offers a discount code, it collects code credit from shoppers who would have bought anyway, and the other cannot. When the versions differ in their offer, count orders per shopper from your store’s records, as in a holdout.
To test timing, change the first step’s delay in one version, since both versions share the abandonment window. Choosing an abandonment window explains why.