What you’re testing
The unit of a test is a complete cart — an entire experience, not a single block or setting. You compare two of your saved carts wholesale. Any draft cart can be the challenger.Creating a test
From the Carts page, choose Create test. You’ll need a live cart and at least two carts total. Then:- Your live cart is the control automatically — you can’t change it.
- Pick one challenger from your other carts.
- Set the traffic allocation — the split between the two, defaulting to 50/50. Each side is between 1% and 99%, and the two must add up to 100%.
- Give the test a name (required).
- Publish & start test.
While a test runs
- Shoppers are stickily bucketed into one variant, so a returning shopper keeps seeing the same cart for the length of the test.
- Publishing and unpublishing are locked, and the variant carts are read-only in the editor — end the test to make changes.
The significance indicator
While a test runs, Aftersell shows a significance progress bar: the impressions collected so far against a target sample size. The target is a total across the test, sized so the smallest variant still gathers about 3,500 impressions — roughly 7,000 in total for a 50/50 split, and proportionally more for an uneven one (an 80/20 split needs about 17,600). As the test runs, impressions accumulate and the bar fills toward that target. The bar appears only while a test is live. Once you end the test the bar disappears; the banner keeps showing the test’s start and end dates, and you read the outcome from the per-variant results.The bar is a readiness indicator — it tells you whether you’ve gathered enough traffic to make a confident call. It is not a statistical p-value, and Aftersell does not auto-declare a winner. Let the test gather enough data, then compare the variants and choose.
Ending a test and choosing a winner
When you end a test, you see each variant’s Revenue, conversion rate (CCR), and impressions side by side, and you pick the winner yourself. The winner is published live and the other cart returns to draft.The end-test Revenue figure is broader than the one on the Test analytics page. Here it’s every conversion attributed to the cart. On Test analytics the Added revenue metric is narrowed to upsell, add-on and subscription revenue. Expect the two numbers to differ for the same test, and compare variants within one surface rather than across them.
What can and can’t be tested
- Can: any two whole cart configurations. Swap layouts, blocks, copy, design — build a challenger cart however you like and test it against the live one.
- A caveat on metrics: the Added revenue metric on Test analytics covers upsell, add-on, and subscription revenue — a subset of everything a cart earns rather than a full partition. Treat it as a like-for-like read between the two variants rather than as each variant’s total contribution. The end-test screen uses the broader figure described above.