Join our mailing list
Assess Your Channel Measurement
The Ladder of Incrementality
You can think of incrementality as a ladder of options that get closer to measuring true business value as you climb
-
•
Method Pros Cons Randomized controlled trials Purest form of experimentation which gives the best read on incrementality Impossible to implement for certain tactics (e.g. linear TV) and hard for others. Not often offered. Ghost ads Purest form of randomized control trial (RCT) testing Proprietary method that is limited to walled garden offerings PSA placebo experiments Purest form of randomized control trial (RCT) testing Proprietary method that is limited to walled garden offerings Intent to treat or per protocol Close to purest form of randomized control trial (RCT) testing Unclear if those not reached with ads are somehow different, introducing noise into the test A/B Tests Advantages depend on how the test is implemented Term is used haphazardly and could refer to any form of RCT testing -
•
Method Pros Cons Twinning A typical process to make unexposed consumers as well matched as possible to those exposed. Twinning is done on factors (like demos) that are presumed to be important. These factors can vary. Weighting A typical process to make unexposed consumers as well matched as possible to those exposed Weighting is done on factors (like demos) that are presumed to be important. These factors can vary. Counterfactual Modeling Closely matches the intent of measuring incrementality: what would have happened had the consumers not been exposed to the treatment variable? Model structure and parameter estimation need to be set up in valid ways Doubly robust estimators Estimators combine two modeling approaches to reduce bias and improve the accuracy of causal effect estimation. By using both models, doubly robust estimators provide more consistent estimates of the causal effect even if either one of the models is mis-specified. No downside to adding doubly robust estimators to other weighting or matching protocols. Targeted maximum likelihood Statistical method used for estimating treatment effects in observational studies or complex data settings. It can also incorporate prior knowledge Complicated and subject to model misspecification. Pre/post designs Eliminates the issue of matching a test and control set of IDs Requires persistent IDs which are not always available. There can also be exogenous factors that affect conversion patterns from the pre to post period that are not adjusted for. Spike modeling Less data burden as this method can be run off of more readily available aggregated data. Aggregated data can be divorced from the underlying process and miss the true effect of the causal variables. Geo-based testing Test marketing is easy to understand and a traditional method for measuring treatment effects Factors can vary across markets, such as retailers being different, that make the results non-comparable -
•
Method Pros Cons Counting Simple and widely available Rarely reflects incrementality Expert Opinion There are always experts who have seasoned insights Opinions made be wrong in specific instances
Download the Research
The series introduces the concept of incrementality, describes various popular methodologies commonly used to estimate advertising ROI, and reveals a set of actionable steps marketers can take to dramatically improve the accuracy of performance measurement.
-
The What and Why of Incrementality:
An introduction of the importance of measuring incremental impact -
The Ladder of Incrementality:
A description of various popular methodologies for estimating advertising ROI -
Climbing the Ladder of Incrementality:
A set of actionable steps advertisers can take to dramatically improve accuracy of performance measurement and optimization
Watch the webinar: Measuring Incrementality to Untangle Marketing Contribution
Recorded September 13, 2023 as part of MMA Global's Marketing Measurement Great Debates.