Skip to main content
Shopify Rollouts lets you run a controlled experiment between two themes on your live storefront. You can use it to compare Upcart against a different cart app head to head: one theme runs Upcart, the other runs the other app, and Shopify splits real shopper traffic between them and reports which one converts better. This is different from an Upcart A/B test, which compares two of your own Upcart cart variations against each other. Use an Upcart A/B test to tune Upcart itself; use a Shopify Rollouts experiment to decide between Upcart and another app.
Rollouts is a native Shopify feature, not an Upcart one, and it lives in your Shopify admin under Markets → Rollouts. Running an experiment (the A/B test this page describes) requires the Shopify Grow plan or higher; launch-only rollouts are available on lower plans.

​
How it works

A Rollouts experiment has two sides:
  • Control - your currently live, published theme. You can’t change which theme is the control; it is always the published one.
  • Treatment - one theme that you select. This is the variation you are testing against the control.
You set how traffic is split between the two. On the scheduled start, Shopify creates a dedicated copy of the treatment theme for the experiment to run against, then splits shoppers between the control and that treatment copy and records how each side performs. An experiment is always two themes only: the control (your live theme) and one treatment theme. You can’t add a third.

​
Before you start

Set up two themes so each side runs a different cart:
  • Your live (published) theme running Upcart as the control. Make sure the Upcart app embed is enabled in that theme and Upcart is turned on, so the control actually serves the Upcart cart rather than the theme’s default cart.
  • A second theme with the other cart app’s app embed enabled (and Upcart’s embed turned off), set up to serve that app’s cart. This is the treatment.
App embeds are toggled per theme in Online Store → Themes → Customize → App embeds, so each theme can carry a different cart app.

​
Set up the experiment

  1. In your Shopify admin, go to Markets → Rollouts.
  2. Create an experiment and choose your treatment theme. The control is your live theme and is selected for you.
  3. Set the traffic split to divide visitors between the control and the treatment, and keep the traffic allocation at 100% so the whole eligible audience is in the test.
  4. Schedule a start time. An experiment can’t start immediately. You pick from 30-minute time slots, and you have to choose a slot that hasn’t passed yet.
  5. Confirm. On the scheduled start, Shopify duplicates your treatment theme and begins splitting traffic.
For an unbiased test, use a 50/50 traffic split so each cart gets the same share of traffic. An uneven split skews which cart appears to win and makes the result harder to trust.
Allocation and split are different settings. Allocation is the share of eligible visitors who enter the rollout; the split is how that audience divides between control and treatment. They multiply, so a 50% allocation with a 50/50 split shows the treatment to only a quarter of visitors. For a straight A/B test, keep allocation at 100% and set the split.

​
Read the results

Once the experiment is live, click into it again to see the share of sessions allocated to the control and the treatment, along with these metrics for each side: Compare the two sides on the metric that matters most to you, usually conversion rate, to decide which cart performed better.

​
See both versions yourself

To QA both carts on your live store, you need Shopify to assign you to each side in turn. Clearing your cookies and refreshing makes Shopify re-roll which version you land on. A few things to know:
  • It isn’t deterministic. Even on a 50/50 split, the experiment doesn’t flip you back and forth. After you clear cookies and refresh you might land on the same theme and cart again. Repeat the clear-and-refresh until you land on the other side.
  • You can clear just the one cookie. Instead of clearing every cookie, open your browser’s developer tools, go to the Application tab, and delete the _shopify_essential cookie. Removing it makes Shopify treat you as a new visitor and re-assign you.
On a password-protected storefront, deleting _shopify_essential also clears your storefront password authorization, so you’ll have to re-enter the store password.
Need help? If you have questions about setting up a Rollouts experiment with Upcart, reach out to our support team via the in-app chat.