Back to E-commerce Tips

How to A/B Test Upsell Offers After Checkout (and Trust the Result)

E-commerce Tips
How to A/B Test Upsell Offers After Checkout (and Trust the Result)

You've set up a post-purchase offer and buyers are accepting it. Now the questions start. Would a bigger discount earn more, or just give away margin? Is the shorter heading better? Would a different product do better? You can guess, or you can A/B test your upsell offers: show two versions to comparable buyers at the same time and let the orders answer.

This guide covers what to test, which number to judge by, how long to run a test, and the mistakes that make a test pick the wrong winner.


Why A/B test upsell offers after checkout

A post-purchase offer is a good place to test. Every buyer who reaches the page sees exactly one version, the decision is a simple yes or no, and the result is money you can count. There's no long funnel between the change and the outcome.

It's also a place where intuition fails. A bigger discount often raises the acceptance rate and lowers the revenue per buyer. A product that sounds like an obvious match may get ignored. A version that wins on Tuesday may lose over a full week. Testing replaces opinions with orders.

The core rule is simple: split buyers randomly between two versions, at the same time. Comparing this month to last month isn't a test, because the season, your traffic and your ads all changed too.


What to test on a post-purchase offer

Test changes big enough to matter. Swapping one word in the button rarely moves anything you can measure; changing the offer itself often does. Good candidates, roughly from biggest to smallest effect:

  1. The product. A complement versus a consumable, a cheap add-on versus a premium one. See our post on the best products to upsell after checkout for ideas.
  2. The discount. No discount versus 10%, or 10% versus 20%. Our post on how much discount to put on an upsell explains why bigger isn't always better.
  3. Shipping. Free shipping because it travels in the same parcel, versus paid shipping with a deeper discount.
  4. The heading and text. A plain product name versus a reason to add it now.
  5. The design. One large image versus a short product list, with or without a countdown timer.

Change one thing per test. If Version B has a new product, a new discount and a new heading, and it wins, you don't know which change did it. You also can't reuse what you learned on your other offers.


Pick the right metric before you start

The acceptance rate is the number everyone looks at first, and the wrong one to decide on. A version that gives the product away will win on acceptances every time.

Judge a test by revenue per checkout that saw the offer: total upsell revenue from a version, divided by the number of checkouts that were shown it. It rewards both more buyers saying yes and each yes being worth more, and it penalizes a discount that's too generous.

Two other numbers are worth a glance, but not as the deciding metric:

  • Acceptance rate, to understand why a version won. Fewer, bigger yeses and more, smaller yeses can earn the same.
  • Returns on the upsold product, a few weeks later. A version that wins by selling things buyers send back hasn't really won.

Write down which version you expect to win and why before you start. It keeps you honest when the results come in.


How long to run an A/B test on upsell offers

A test needs two things: enough orders, and enough time.

Enough orders, because acceptance is a yes or no, and small numbers swing wildly. If 10 buyers out of 100 accept one version and 14 out of 100 accept the other, that difference could easily be luck. Most stores need hundreds of checkouts per version before a difference means much, and more if the two versions are close.

Enough time, because buyers behave differently on weekdays and weekends, after a newsletter or during a sale. Run a test for at least one full week, ideally two, so each version sees the same mix of days.

Don't stop at the first good-looking number

This is the most common mistake. You check the test every morning, and on day three Version B looks clearly ahead, so you end it. The problem is that early results swing, and if you check often enough, one version will look ahead by chance at some point. Statistician Evan Miller shows that repeatedly checking a test and stopping as soon as it looks significant can make a no-difference test report a winner far more often than the 5% you'd expect (How Not To Run an A/B Test).

The fix is to decide in advance how long the test runs, or to use a tool whose winner rule already accounts for time and sample size, and then leave it alone. If you change a running test halfway, start counting again.


Reading the results

When the test ends, you'll land in one of three places:

  • A clear winner. Keep it, and make it the new baseline for your next test.
  • No clear difference. That's a result too: the change didn't matter enough to detect. Keep whichever version is simpler or cheaper, and test something bigger next.
  • A winner on acceptances, a loser on revenue. Trust revenue per checkout. The extra yeses cost more than they earned.

Keep a short log of each test: what changed, what you expected, what happened. After a few tests, patterns appear that apply across your offers.


Running A/B tests in Elite Post Purchase

In Elite Post Purchase, you test a whole flow. In the flow's editor, scroll to Advanced and click Create A/B test. Version B starts as a copy of the flow, Version A; you change it yourself, choose Describe it and AI drafts Version B from your words (products, collections, discounts, shipping, texts and layout), or choose Get ideas for test suggestions written for that flow. Both versions share the flow's name, time limit and conditions. When you click Start test, each matching checkout gets one version, 50/50.

The test's page shows Shown checkouts, Accept rate and Gross revenue per checkout for each version. Its hint compares the versions on gross revenue per checkout and calls a winner only after 14 days, 300 shown checkouts and 20 accepting checkouts per version, with at least a 95% chance. When you're done, End test, keep A or End test, keep B makes that version the flow. The full walkthrough is in A/B test a flow, and Read your analytics covers the numbers outside a test.

Stop guessing which offer works. Test one change, wait for real numbers, and keep what earns more.

E-commerceShopifyPost-PurchaseUpsellingA/B Testing