Dunning A/B Test Planner: Experiments That Read

Most dunning A/B tests produce a winner that is not real. The volumes are small, the audiences differ week to week, and a three-point gap on forty payments is noise wearing a lab coat. This dunning A/B test planner fixes the planning, not the maths: a hypothesis card for every experiment, a test order that spends your limited volume on the questions worth asking, holdout rules that keep comparisons honest, and a log format you can run a year of experiments on without fooling yourself.

It sits between measurement and action. If you have not instrumented the basics yet, start with how to measure dunning performance and keep your running numbers in the recovery KPI template. This planner assumes you can count failures, attempts, outreach sends, and recoveries by cohort — that is all the instrumentation a honest test needs.

The hypothesis card

Every test starts as a card, filled in before the test runs. A card you cannot fill in is a test you are not ready to run. Copy this template verbatim for each experiment.

Field	What to write
Test name and number	Sequential, dated. Example: T07 — SMS before second retry.
Observation	The measured fact that prompted the test, with its source and window. "Recoveries from the second retry ran below the first across the last two full cycles" — not "retries feel weak."
Hypothesis	One sentence in the form: changing X for segment Y will move metric Z, because reason R. If you cannot name the because, you have a tweak, not a hypothesis.
Segment	Exactly which failures enter the test — decline family, plan type, new versus established subscriber. Mixing families in one test is the most common way dunning results get muddied, because an expired card and an insufficient-funds decline respond to completely different levers.
Variant and control	What each arm receives, in full. The control is your current live sequence, unchanged — not last quarter's, not the one you meant to ship.
Primary metric	One metric, chosen in advance: usually recovery rate on the tested failure cohort by the end of the full recovery window. Secondary observations may be noted, but the decision rests on the primary metric alone.
Window	The test runs until every failure in both arms has completed a full recovery cycle — roughly sixteen days per failure in a typical sequence, plus entry time. Calendar deadlines that cut cycles short are banned.
Decision rule	Written in advance: what difference, on what raw counts, leads to adopt, reject, or extend. Example: adopt if the variant recovers more on at least equal volume and the gap survives removing the single best day.
Result and decision	Filled at close: raw counts for both arms, the decision taken, and the date it shipped or was discarded.

What to test, in what order

Volume is the constraint, so spend it where leverage lives. Test in this order and do not skip ahead: a brilliant subject line on a broken retry schedule is a rounding error with good copy.

1. Retry timing for insufficient funds. The largest decline family at 40.6% of failed payments, according to Churnkey analysis of Stripe data. Timing changes move this family more than any copy change ever will.
2. Channel addition. Adding SMS before a key retry, or a second email touch. Context: each additional dunning email touch adds roughly 1–2 points of recovery, according to Churnkey analysis of Stripe data — that is the honest scale of outreach gains, and your test should be read against it, not against hope.
3. Send timing of outreach relative to retries. The day before versus the day of an attempt.
4. Card-update page friction. Fewer fields, clearer amount and date, pre-filled where your processor allows. Measure updates completed per page visit, not email clicks.
5. Subject lines and sender name. Real effects, small effects. Test these only once the structural items above are settled.
6. Offer framing in late-sequence and deflection steps. Discount versus Pause versus Service emphasis, by cancellation reason. High value per decision, but the audience is small, so these tests need long windows.

Holdout and honesty rules

• Split at the failure, not the subscriber's convenience. Assign each new failure to an arm at random when it occurs, and keep that subscriber in the same arm for the whole cycle. Letting staff, timing, or plan type influence assignment quietly rigs the result.
• Keep a true holdout for structural tests. When testing a whole-sequence change, hold back a small control arm on the old sequence for the full test. Without it, seasonality — payday concentration, holidays, your own growth — will happily impersonate your improvement.
• Run full cycles, always. A failure counted on day two has not finished being recoverable. Comparing arms mid-cycle systematically favours whichever arm front-loads its attempts.
• Report raw counts first. "41 of 118 recovered versus 33 of 121" is information. "Up 8 points" on those numbers is a story. Decision rules in this planner are written in counts for exactly this reason.
• Never stop early on a good day. A variant that jumps ahead on Tuesday and is declared the winner on Wednesday is the single most reliable way to ship noise. The decision rule names the window in advance; the window governs.
• One test per segment at a time. Overlapping tests on the same failures make both unreadable. Queue tests; the log below is the queue.
• Record the losses. A rejected variant stays in the log with its numbers. Otherwise the same idea returns every year wearing a new name, and pays the testing cost again.

Sample honesty, stated plainly: at typical subscription volumes, a single month of failures can only reliably detect large differences. Small gaps need multiple full cycles pooled. That is not a defect in your programme — it is the arithmetic of the audience size. Plan for fewer, bigger, longer tests, and size the prize before you start with the recovery calculator. If your current recovery sits well inside the published ranges on the failed payment recovery benchmarks page, the headroom justifies patient testing; if you are already near the top, structural coverage — channels, backfill, deflection — usually beats further micro-tests.

The twelve-experiment test log

Keep one log for the whole programme. Twelve rows is a realistic year: one test a month, each run to a full cycle, each decided on counts. Copy this table into your tracker and fill it as you go.

#	Test	Segment	Primary metric	Control (recovered / total)	Variant (recovered / total)	Decision
T01	Payday-aligned second retry	Insufficient funds	Recovery by end of cycle	— / —	— / —	Pending
T02	SMS before second retry	All soft declines	Recovery by end of cycle	— / —	— / —	Pending
T03	Outreach day before vs day of attempt	All soft declines	Recovery by end of cycle	— / —	— / —	Pending
T04	Third email touch added	All soft declines	Recovery by end of cycle	— / —	— / —	Pending
T05	Card-update page: short form	Expired / replaced card	Updates per page visit	— / —	— / —	Pending
T06	Sender name: person vs brand	All dunning email	Update-link clicks per send	— / —	— / —	Pending
T07	Subject: specific amount vs general	All dunning email	Recovery by end of cycle	— / —	— / —	Pending
T08	Longer window: extra late attempt	Insufficient funds	Recovery by end of cycle	— / —	— / —	Pending
T09	Late-sequence help offer (human reply)	High-value plans	Recovery by end of cycle	— / —	— / —	Pending
T10	Deflection lead offer: Pause vs Discount	Timing-reason cancels	Saves per flow entry	— / —	— / —	Pending
T11	Pre-dunning expiry warning at 14 days	Cards expiring next month	Failures prevented	— / —	— / —	Pending
T12	Annual-plan sequence: earlier personal outreach	Annual plans only	Recovery by end of cycle	— / —	— / —	Pending

The bottom line

Honest dunning tests are slow, and that is the price of results you can trust: a card written before the test, a control that stays untouched, full cycles before any verdict, and decisions made on raw counts. Run twelve of those a year and your sequence improves on evidence while competitors improve on anecdotes. If you would rather run experiments on a sequence that already handles decline-aware retries, outreach, and deflection at one flat $179/mo, look at the plan on the pricing page and use the 14-day trial as your first full-cycle test.

Frequently asked questions

How long should a dunning A/B test run?

Until every failure in both arms has completed a full recovery cycle, plus the time it takes those failures to enter the test. For most sequences that means several weeks at minimum. Ending on a calendar date or a good day cuts cycles short and biases the comparison.

How big a difference can I trust at low volume?

Only large differences are readable at small volumes. Report raw counts for both arms, pool multiple cycles when the gap is small, and treat any close result as undecided rather than as a win for whichever arm is ahead.

What should I test first?

Retry timing for insufficient-funds failures, because that is the largest decline family and timing is its main lever. Then channel additions and outreach timing. Subject lines and sender names come last — their effects are real but small compared with structural changes.

Do I need a holdout group that gets no dunning?

No — a holdout means keeping the current live sequence running for a control arm, not withholding recovery entirely. Never test by denying one group the outreach another group receives; test variant against your existing sequence instead.

Can I run two dunning tests at once?

Only on completely separate segments, such as annual plans in one test and monthly plans in another, with no subscriber appearing in both. Overlapping tests on the same failures make both results unreadable.