Why Test for Incrementality
Understanding the incrementality of advertising spend is one of the biggest advertising efficiency step changes there is. Bigger than creative. Bigger than multi-touch attribution. Despite that, incrementality tests are rare: rare to discuss and rare to implement. All of the biggest and best advertisers run them.
Incrementality tests are not limited to display campaigns and Public Service Announcement (PSA) tests. They can be run with any creative type (brand v. direct response) or medium (including search advertising).
Incrementality tests help with more than just the incremental cost per conversion. They also help understand if advertising helps with re-activation and retention.
Some of the largest retention gains come not from email campaigns but from mass media campaigns. Continuing the emotional connection a user has with a product is an underappreciated way to retain a user.
Incrementality Tests and Cookies
Testing bifurcates around cookies. Tests can use cookies or be agnostic to cookies. Tests that use cookies will have greater power, can run with smaller sample sizes, and work with a smaller lift.
Cookie methods are the easiest to implement, but are limited due to user behavior (switching devices, deleting cookies) and overall changes in user privacy settings and regulation.
Tests without cookies work regardless of changing privacy controls, cookie deletion, multi-device use, and can pick up effects on other parts of product usage such as re-activation and retention. Re-activated and retained cookies might not be part of an acquisition incrementality test, and so will be missed.
Overview of Test Types
| Test Type | Benefits | Concerns | When To Use |
| Geo Test | Works regardless of changing privacy settings around cookies or multiple devices | Statistical power | Non-digital tests or when digital spend is sufficient to detect at scale |
| A/B Test | Works on smaller ad budgets | Issues around cookies | Digital tests with low lift |
| Public Service Announcement | Ad platforms make these easy to do | True incrementality | Display campaigns when other options aren’t available |
| Multivariate | See the holistic benefits of multiple advertising channels | Test setup is complex, more expensive to run | To understand the benefits of multi-channel campaigns. |
Approaches
Geo Tests
Geo tests get around privacy issues related to cookies, and multiple devices, and work across a broad set of advertising medium: TV, out-of-home, radio, SEM, and display. Setting up the tests requires understand what targeting options are available for the media, and if that inventory is comparable (For example, National v Location TV options).
It’s important to validate that the geo groups had good correlation for some period of time before the test intervention is run.
Once groups are selected, the test is run for the appropriate period of time. Factors that influence test length are: time to convert, time for media to ramp up, minimum contract lengths, etc.
Choosing a greater number of smaller geos can provide a more robust test than a smaller number of larger, matched markets. This is because a test of say, NY vs LA could be thrown off by differences in situation in NY (maybe a hurricane in NY, for example). More pleneitufl, smaller geos, helps insulates against the impact to one geo.
This type of test can be analyzed in a number of ways. A traditional hypothesis testing framework could be used with sufficient information. For example, if the test is done through a digital ad network that provides impressions, clicks, and conversion data.
Another method is to use causal inference, and a tool like Causal Impact, to compare test and control geos. One of causal inference’s strengths is that it can get around traditional limitations of cookie based testing (multiple devices, privacy controls, cookie deletion).
A/B Tests
A/B tests can apply to several advertising activities: retargeting, email, and seo. A/B testing for retargeting and email purposes is identical to a typical website a/b testing. A control group will receive no treatment (email, display ad) and a test group will receive the treatment. The test results can be analyzed using typical hypothesis thing techniques.
Public Service Announcements
Display tests are pretty common, although this feels like a more old school method. Every user gets cookies for the test, although a portion of user s will see a public service announcement instead of the advertisement.
Multivariate Advertising Tests
A fancy way of saying multiple types of advertising activity on the same users. This could involve a control (no treatment), a test with one advertising type (display), and another method online video (OLV). The goal here is to determine whether multiple advertising media and messages have additive benefits that produce more efficient results than just one advertising method alone.
Multivariate tests, generally, are more complex to do well than a/b tests. Setting these up in an advertising environment takes more care and coordination as well.
Getting It Right
Incrementality testing illuminates the value of advertising, which helps with medium planning and creative. It also helps with budget allocation. Testing can be done with any number of advertising channels and is relatively straightforward.
Are you doing incrementality testing today? If not, which method do you think you could get started with first?
Sources
Focus on Growth 