Your ad platform says it generated 500 orders, but how many of those customers would have purchased anyway?
That is the question attribution cannot answer on its own.
Incrementality testing compares what happened when advertising was present with a credible estimate of what would have happened without it. The difference is the incremental result: the sales, leads or revenue that the advertising actually added.
For ecommerce brands spending heavily across Meta, Google, TikTok and other paid channels, that distinction can change where the next marketing dollar should go.
What Is Incrementality Testing?
Incrementality testing measures whether marketing caused an outcome that would not otherwise have happened.
The simplest version compares two groups:
- a treatment group exposed to the advertising
- a control or holdout group that is not
If the treatment group produces meaningfully more sales than the comparable control group, that difference is the estimated incremental lift.
This is different from attribution.
Attribution asks:
Which marketing touchpoint should receive credit for this sale?
Incrementality asks:
Would this sale still have happened without the marketing?
Both questions are useful, but they solve different measurement problems.
Why Can Platform ROAS Overstate an Ad’s Contribution?
Imagine a repeat customer searches your brand name, clicks a paid search ad and buys a product they were already planning to purchase.
The ad platform may correctly attribute that conversion to the ad. But did the ad create a new sale? Not necessarily.
The customer may have bought through the organic result, typed the URL directly or returned through email if the ad had not appeared.
The same problem can happen with:
- brand search
- retargeting
- existing customers
- high-intent shopping campaigns
- promotional periods
A channel can therefore report excellent ROAS while some of the attributed revenue would have existed without the advertising.
That does not mean platform reporting is useless.
It means attributed ROAS and incremental ROAS answer different questions.
For a broader look at which numbers deserve attention, see The Darl’s guide to marketing metrics that actually matter.
A Simple Ecommerce Example
Consider a hypothetical ecommerce test.
Two comparable groups of customers each contain enough users to make a meaningful comparison.
During the test:
- the group exposed to advertising generates 1,100 orders
- the holdout group generates 1,000 orders
The advertising is associated with 1,100 purchases in the exposed group, but the comparison suggests that roughly 1,000 of those purchases might have happened anyway.
The estimated incremental contribution is therefore:
100 additional orders
If those 100 incremental orders generated $12,000 in revenue and the advertising cost $6,000:
Incremental ROAS = $12,000 ÷ $6,000 = 2.0
That is very different from calculating ROAS as though all 1,100 orders were caused by the ads.
The numbers above are purely illustrative. Real tests require adequate sample size, a credible control group and enough time for conversions to occur.
Holdout, Geo Test or Conversion Lift?
There is no single incrementality test that fits every ecommerce brand.
Audience Holdout Test
A portion of an eligible audience is intentionally prevented from receiving the advertising.
You then compare conversion behavior between exposed and holdout groups.
This is often the cleanest structure when users can be randomly assigned and kept separate.
Best suited to questions such as:
Does this campaign generate additional purchases among this audience?
The limitation is practical: not every channel or campaign setup allows a clean user-level holdout.
Geo Incrementality Test
Instead of withholding ads from individual users, you compare geographic markets.
Ads continue in the test markets and are reduced or removed in comparable holdout markets.
This can be useful when you want to evaluate:
- an entire paid channel
- a large campaign
- regional advertising
- activity that cannot easily be separated at user level
The challenge is finding regions that behave similarly enough before the experiment.
A strong sales week in one city and a weak week in another does not automatically prove the ads caused the difference.
Platform Conversion Lift Study
Platforms such as Meta and Google can provide lift-testing capabilities for eligible advertisers and campaigns.
These experiments create treatment and control groups and estimate the additional conversions produced by advertising.
They can be convenient because much of the experiment infrastructure already exists inside the platform.
But they generally answer a platform-specific question.
A Meta lift study helps you understand Meta’s incremental contribution. It does not automatically explain the interaction between Meta, Google, email, organic search and the rest of the customer journey.
Which Test Should You Choose?
Start with the business question.
| Question | Possible Approach |
| Does one Meta campaign create additional purchases? | Platform lift or audience holdout |
| Does Meta as a whole add sales? | Geo holdout |
| Is branded paid search creating additional demand? | Geo or controlled holdout |
| Does advertising in a new market create lift? | Geo test |
| Should we move budget from one major channel to another? | Channel-level incrementality test |
Do not choose the method because it sounds sophisticated.
Choose the simplest design capable of answering the decision you actually need to make.
How to Design a Useful Incrementality Test
A good experiment begins with one clear question.
Bad:
Are our ads working?
Better:
Does our Meta prospecting activity create additional first-time purchases?
Better still:
If Meta prospecting is withheld from a comparable group, does total first-time customer revenue fall enough to justify the current spend?
That gives the test a measurable outcome and a decision attached to it.
Read more: Product Page SEO for AI Search
Change One Important Thing at a Time
An incrementality experiment becomes difficult to interpret when the business changes several major variables at once.
Imagine you:
- stop Meta ads in the holdout market
- launch a new product
- start a 25% sale
- change shipping prices
- launch an influencer campaign
- increase Google Ads spend
Then revenue rises in the treatment market.
What caused it?
You no longer have a clean answer.
During the test window, keep unrelated differences between test and control groups as limited as reasonably possible.
That does not mean the entire business has to stop operating.
It means changes capable of materially affecting the measured outcome need to be planned around the experiment or accounted for in its interpretation.
Define the Outcome Before the Test Starts
Decide what success means before seeing the data.
For an ecommerce business, the primary outcome could be:
- total orders
- first-time customer orders
- revenue
- contribution margin
- subscription starts
- another commercially relevant conversion
Avoid selecting whichever metric looks best after the experiment ends.
That creates bias.
The measurement should match the question you asked at the beginning.
How Do You Calculate Incremental Lift?
A simplified lift calculation is:
Incremental Lift = Treatment Outcome − Expected Outcome Without Advertising
The difficult part is not the subtraction.
It is estimating the counterfactual:
What would the treatment group have done without the advertising?
That is why control-group quality matters.
For a randomized user-level test, the holdout provides the comparison.
For a geo experiment, historical behavior and matched markets can help estimate what the treatment regions would have done without the intervention.
Once incremental revenue is estimated:
Incremental ROAS = Incremental Revenue ÷ Advertising Spend
This can be much more useful for budget decisions than platform-reported ROAS alone.
How Should the Result Change Your Budget?
Suppose a channel reports strong attributed ROAS but shows relatively little incremental lift.
That does not automatically mean:
Turn the channel off.
The result should trigger investigation.
The channel may:
- be capturing demand rather than creating it
- work better for certain audiences
- be more incremental in prospecting than retargeting
- support another channel indirectly
- perform differently during promotional periods
The next decision may be to reduce a specific campaign, change audience strategy or run another narrower test.
On the other hand, a channel with modest platform attribution but strong measured incrementality may deserve more investment.
The question becomes:
Which spend creates business we would not otherwise receive?
That is a much stronger basis for framing the marketing budget decision than ranking channels only by their dashboards.
Incrementality Testing and Creative Testing Solve Different Problems
Creative testing asks:
Which message or execution performs better?
Incrementality testing asks:
Did the advertising create additional business?
You need both.
You may identify a high-performing Meta creative, but that does not prove the campaign as a whole is incremental.
Likewise, a channel may be demonstrably incremental while still containing weak creative that can be improved.
If you are working on the first problem, see The Darl’s Meta Ads creative testing framework.
What Can Ruin an Incrementality Test?
Several mistakes can make the result difficult to trust.
Test and Control Groups Are Not Comparable
If the treatment market historically generates twice as much revenue as the holdout market, a simple comparison will be misleading.
The Test Is Too Short
Ecommerce behavior fluctuates by weekday, promotion, pay cycle, season and purchase consideration period.
A few days may not provide enough signal.
Too Many Things Change
Major pricing, product, promotional or channel changes can contaminate the comparison.
The Wrong Outcome Is Used
If the purpose is to understand total business impact, using only platform-attributed conversions defeats much of the point of the experiment.
The Result Is Called Too Early
Early differences can disappear as more data arrives.
Agree on the intended duration and stopping rules before the test begins.
When Is a Brand Too Small for a Reliable Test?
This is an important question because incrementality testing is not automatically reliable just because you created a holdout group.
Small brands may struggle when:
- conversion volume is low
- daily sales fluctuate heavily
- only a few regions generate meaningful revenue
- the expected lift is small
- the purchase cycle is long
- withholding enough traffic would materially hurt the business
Imagine a store averaging six orders per day.
If one group produces seven orders and another produces five, that difference may simply be normal variation.
You should not build a major budget decision around weak statistical evidence.
Sometimes the correct conclusion is:
We do not yet have enough data for a confident incrementality estimate.
That is better than manufacturing certainty.
What Can Smaller Ecommerce Brands Do Instead?
If a formal lift study is not feasible yet, you can still improve measurement.
Track:
- blended revenue
- total acquisition cost
- new customer revenue
- contribution margin
- branded search demand
- repeat purchase
- channel spend changes over time
You can also use carefully planned natural experiments when budgets or campaigns genuinely change.
These methods do not provide the same causal certainty as a well-designed randomized experiment, but they can produce useful directional evidence.
The important thing is to label the evidence correctly.
A correlation is not automatically incrementality.
Should You Test Retargeting or Prospecting First?
The answer depends on where the uncertainty and budget risk are highest.
Retargeting is often a useful candidate because it targets people already close to the brand or purchase.
That creates an obvious incrementality question:
Would many of these people have bought anyway?
Brand search can raise a similar issue.
Prospecting can also be tested, especially when it represents a large share of spend and the business needs evidence that the channel is creating new customer demand.
Prioritize the test with the greatest potential to change a meaningful budget decision.
Incrementality Is Not a Replacement for Attribution
Attribution still helps marketers understand:
- customer paths
- campaign performance
- optimization signals
- channel interactions
- conversion journeys
Incrementality answers a narrower but important causal question.
Marketing teams therefore should not replace all attribution reporting with lift tests.
A practical measurement stack might include:
| Measurement | Question |
| Platform attribution | Which campaigns are receiving credit? |
| GA4 / analytics | How do visitors behave after arrival? |
| Blended metrics | Is the business growing efficiently overall? |
| Incrementality tests | What did advertising actually add? |
| Customer economics | Was the added growth profitable? |
Each layer gives you a different piece of the answer.
A Practical Ecommerce Incrementality Checklist
Before launching a test, confirm:
- ☐ We have one clear business question.
- ☐ Treatment and control groups can be kept meaningfully separate.
- ☐ The groups are comparable before the test.
- ☐ Conversion tracking is reliable.
- ☐ The primary outcome is defined in advance.
- ☐ The test is long enough to reflect the normal purchase cycle.
- ☐ Major promotions or unrelated changes will not distort one group.
- ☐ We know how the result will affect a real budget decision.
- ☐ We will evaluate incremental rather than only attributed revenue.
- ☐ We are prepared to report uncertainty if the evidence is weak.
If several of those boxes cannot be checked, fix the test design before spending money on the experiment.
Frequently Asked Questions
What is incrementality testing in marketing?
Incrementality testing estimates the conversions or revenue caused by marketing that would not have happened without it.
What is the difference between attribution and incrementality?
Attribution decides which marketing touchpoint receives credit for a conversion. Incrementality asks whether the marketing activity actually caused an additional conversion.
What is incremental ROAS?
Incremental ROAS is incremental revenue divided by the advertising spend used to generate that lift.
What is a holdout test?
A holdout test intentionally prevents a comparable group from receiving the marketing being tested. Its results are compared with the exposed group.
What is a geo incrementality test?
A geo test uses geographic regions as treatment and control groups to estimate whether advertising created additional sales or revenue.
How long should an incrementality test run?
There is no universal duration. The test needs enough time and conversion volume to distinguish the expected effect from normal business variation and should account for the normal purchase cycle.
Can small ecommerce brands run incrementality tests?
Sometimes, but low conversion volume can make results too noisy to support confident decisions. Smaller brands should assess feasibility before sacrificing revenue to a holdout.
Does a high ROAS mean ads are incremental?
No. High attributed ROAS can include customers who would have purchased anyway. Incrementality testing is designed to estimate the additional business caused by advertising.
The Question Behind the Dashboard
Performance marketing becomes much more useful when the conversation moves from:
“Which platform reported the highest ROAS?”
to:
“What additional business did this spend create?”
Incrementality testing helps answer that question.
It does not make measurement perfectly certain, and not every ecommerce brand has enough data to run a reliable test immediately.
But when the test is feasible, it gives marketers something attribution alone cannot provide: evidence about what would have happened if the advertising had not run.
That evidence can change which campaigns you scale, which channels you reduce and where the next dollar should go.
If you want to evaluate paid media based on business impact rather than platform attribution alone, review paid media measurement with The Darl.
