Incrementality Testing: True Value of Your Ads

Why Test for Incrementality

Understanding the incrementality of advertising spend is one of the biggest advertising efficiency step changes there is. Bigger than creative. Bigger than multi-touch attribution. Despite that, incrementality tests are rare: rare to discuss and rare to implement. All of the biggest and best advertisers run them.

Incrementality tests are not limited to display campaigns and Public Service Announcement (PSA) tests. They can be run with any creative type (brand v. direct response) or medium (including search advertising).

Incrementality tests help with more than just the incremental cost per conversion. They also help understand if advertising helps with re-activation and retention.

Some of the largest retention gains come not from email campaigns but from mass media campaigns. Continuing the emotional connection a user has with a product is an underappreciated way to retain a user. 

Incrementality Tests and Cookies

Testing bifurcates around cookies. Tests can use cookies or be agnostic to cookies. Tests that use cookies will have greater power, can run with smaller sample sizes, and work with a smaller lift. 

Cookie methods are the easiest to implement, but are limited due to user behavior (switching devices, deleting cookies) and overall changes in user privacy settings and regulation.

Tests without cookies work regardless of changing privacy controls, cookie deletion, multi-device use, and can pick up effects on other parts of product usage such as re-activation and retention. Re-activated and retained cookies might not be part of an acquisition incrementality test, and so will be missed. 

Overview of Test Types

Test TypeBenefitsConcernsWhen To Use
Geo TestWorks regardless of changing privacy settings around cookies or multiple devicesStatistical powerNon-digital tests or when digital spend is sufficient to detect at scale
A/B TestWorks on smaller ad budgetsIssues around cookiesDigital tests with low lift
Public Service AnnouncementAd platforms make these easy to doTrue incrementalityDisplay campaigns when other options aren’t available
MultivariateSee the holistic benefits of multiple advertising channelsTest setup is complex, more expensive to runTo understand the benefits of multi-channel campaigns.


Geo Tests 

Geo tests get around privacy issues related to cookies, and multiple devices, and work across a broad set of advertising medium: TV, out-of-home, radio, SEM, and display. Setting up the tests requires understand what targeting options are available for the media, and if that inventory is comparable (For example, National v Location TV options). 

It’s important to validate that the geo groups had good correlation for some period of time before the test intervention is run. 

Once groups are selected, the test is run for the appropriate period of time. Factors that influence test length are:  time to convert, time for media to ramp up, minimum contract lengths, etc.

Choosing a greater number of smaller geos can provide a more robust test than a smaller number of larger, matched markets. This is because a test of say, NY vs LA could be thrown off by differences in situation in NY (maybe a hurricane in NY, for example). More pleneitufl, smaller geos, helps insulates against the impact to one geo.

This type of test can be analyzed in a number of ways. A traditional hypothesis testing framework could be used with sufficient information. For example, if the test is done through a digital ad network that provides impressions, clicks, and conversion data. 

Another method is to use causal inference, and a tool like Causal Impact, to compare test and control geos. One of causal inference’s strengths is that it can get around traditional limitations of cookie based testing (multiple devices, privacy controls, cookie deletion). 

A/B Tests

A/B tests can apply to several advertising activities: retargeting, email, and seo. A/B testing for retargeting and email purposes is identical to a typical website a/b testing. A control group will receive no treatment (email, display ad) and a test group will receive the treatment. The test results can be analyzed using typical hypothesis thing techniques. 


Public Service Announcements

Display tests are pretty common, although this feels like a more old school method. Every user gets cookies for the test, although a portion of user s will see a public service announcement instead of the advertisement. 

Multivariate Advertising Tests

A fancy way of saying multiple types of advertising activity on the same users. This could involve a control (no treatment), a test with one advertising type (display), and another method online video (OLV). The goal here is to determine whether multiple advertising media and messages have additive benefits that produce more efficient results than just one advertising method alone.

Multivariate tests, generally, are more complex to do well than a/b tests. Setting these up in an advertising environment takes more care and coordination as well.

Getting It Right

Incrementality testing illuminates the value of advertising, which helps with medium planning and creative. It also helps with budget allocation. Testing can be done with any number of advertising channels and is relatively straightforward.

Are you doing incrementality testing today? If not, which method do you think you could get started with first?


Click to access kellogg_fb_whitepaper.pdf

Click to access 45950.pdf

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: