A/B and multivariate testing
A test starts from an offer you already run, not a blank slate. You pick the base offer and define one or more challenger variants that override specific fields — price, copy, layout, product, or placement, depending on what the offer's surface supports. Traffic is split evenly across the variants automatically, computed from however many you define, so a three-way test doesn't need manual allocation arithmetic.
Because variants override individual fields rather than duplicating the offer, a multivariate test can vary two things at once without you maintaining several near-identical offers.
Sample size is calculated before the test runs, not after
This is the part most testing tools leave to the merchant. Before an experiment starts you supply a primary metric, a baseline rate, the minimum detectable effect you care about, and your approximate daily traffic. From those, UpsellPlus calculates the sample size the test needs — and therefore roughly how long it must run to produce a result you can act on.
The practical consequence is that you know up front whether a test is worth running at all. A store with modest traffic trying to detect a 2% improvement will be told the test needs more traffic than it has, instead of discovering that after three weeks of inconclusive data. You can also set an automatic end condition rather than watching the test and calling it by hand.
Experiments are separate from targeting
Smart Rules decide whether an offer is eligible for a given shopper. An experiment decides which version of an eligible offer that shopper sees. The two layers stack and operate independently, so a tightly targeted offer can still be tested, and a test never widens an offer's audience.
Holdout experiments: measuring what UpsellPlus actually adds
Attributed revenue answers “what did offers earn”. It does not answer the harder question a careful merchant asks: how much of that would I have earned anyway? A holdout answers that one.
You create a holdout, choose what share of traffic to withhold offers from, and start it. Shoppers in the holdout group see no offers at all. The difference between the two groups is the incremental contribution — the revenue that exists because UpsellPlus is running.
How it is configured:
- Share of traffic is yours to set, anywhere from 1% to 50%, with 5%, 10%, 15%, 20% and 50% available as presets. The bounds are deliberate: below 1% the control group never gathers enough traffic to reach significance, and above 50% the statistical benefit plateaus while the revenue you're suppressing keeps climbing.
- Scope is either store-wide or limited to specific offers. Limiting it lets you measure one offer's true contribution while everything else keeps running normally.
- One at a time. A store can only have one holdout running, which keeps the control group clean — overlapping holdouts would contaminate each other's comparison.
- Start and stop when you choose. A holdout is created stopped, runs when you start it, and can be stopped at any point.
Why this matters when evaluating any upsell app
Every upsell app reports attributed revenue, and attribution is generous by nature: if a shopper would have bought the product anyway, attribution still counts it. A holdout is the only way to separate genuine lift from attribution. Being able to run one — and to scope it to a single offer — is the difference between believing a number and having tested it.
