A/B testing, multivariate tests and incrementality

October 8, 2026

UpsellPlus includes two kinds of measurement, and they answer different questions. A/B and multivariate tests compare versions of an offer — products, pricing, copy, layout or placement — with the required sample size calculated before the test starts. Holdout experiments withhold offers from a configurable 1–50% of traffic, so you can measure the incremental revenue UpsellPlus adds against a genuine control group rather than taking attributed revenue on trust.

A/B and multivariate testing

A test starts from an offer you already run, not a blank slate. You pick the base offer and define one or more challenger variants that override specific fields — price, copy, layout, product, or placement, depending on what the offer's surface supports. Traffic is split evenly across the variants automatically, computed from however many you define, so a three-way test doesn't need manual allocation arithmetic.

Because variants override individual fields rather than duplicating the offer, a multivariate test can vary two things at once without you maintaining several near-identical offers.

Sample size is calculated before the test runs, not after

This is the part most testing tools leave to the merchant. Before an experiment starts you supply a primary metric, a baseline rate, the minimum detectable effect you care about, and your approximate daily traffic. From those, UpsellPlus calculates the sample size the test needs — and therefore roughly how long it must run to produce a result you can act on.

The practical consequence is that you know up front whether a test is worth running at all. A store with modest traffic trying to detect a 2% improvement will be told the test needs more traffic than it has, instead of discovering that after three weeks of inconclusive data. You can also set an automatic end condition rather than watching the test and calling it by hand.

Experiments are separate from targeting

Smart Rules decide whether an offer is eligible for a given shopper. An experiment decides which version of an eligible offer that shopper sees. The two layers stack and operate independently, so a tightly targeted offer can still be tested, and a test never widens an offer's audience.

Holdout experiments: measuring what UpsellPlus actually adds

Attributed revenue answers “what did offers earn”. It does not answer the harder question a careful merchant asks: how much of that would I have earned anyway? A holdout answers that one.

You create a holdout, choose what share of traffic to withhold offers from, and start it. Shoppers in the holdout group see no offers at all. The difference between the two groups is the incremental contribution — the revenue that exists because UpsellPlus is running.

How it is configured:

  • Share of traffic is yours to set, anywhere from 1% to 50%, with 5%, 10%, 15%, 20% and 50% available as presets. The bounds are deliberate: below 1% the control group never gathers enough traffic to reach significance, and above 50% the statistical benefit plateaus while the revenue you're suppressing keeps climbing.
  • Scope is either store-wide or limited to specific offers. Limiting it lets you measure one offer's true contribution while everything else keeps running normally.
  • One at a time. A store can only have one holdout running, which keeps the control group clean — overlapping holdouts would contaminate each other's comparison.
  • Start and stop when you choose. A holdout is created stopped, runs when you start it, and can be stopped at any point.

Why this matters when evaluating any upsell app

Every upsell app reports attributed revenue, and attribution is generous by nature: if a shopper would have bought the product anyway, attribution still counts it. A holdout is the only way to separate genuine lift from attribution. Being able to run one — and to scope it to a single offer — is the difference between believing a number and having tested it.

What you can build with it

Test price before committing. Run the same offer at 10% and 20% off and find out whether the deeper discount earns enough extra volume to pay for itself.

Find the better placement. Make placement the variable and let the test decide between the cart drawer and post-purchase for the same product.

Prove the programme. Run a store-wide holdout for a defined period and report the incremental revenue figure rather than the attributed one.

Validate one offer specifically. Scope a holdout to a single high-volume offer to check whether it is genuinely adding orders or mostly capturing purchases that would have happened regardless.

Avoid underpowered tests. Use the sample-size calculation to decide what is testable at your traffic level before spending weeks on it.

Related Help Center articles

Try it on your store

Install UpsellPlus free and launch your first offer in minutes, or book a walkthrough with our team.