Beware of the black box tv attribution solution

Near the end of the movie The Wizard of Oz, Wizard says to a desperate Dorothy, “Do you presume to criticize the great Oz?”

This, of course, occurs just as Toto pulls back the curtain, and Dorothy unloads on him like Greta Thunberg at a climate change conference.

We have met with three clients in the past year who are deeply unsatisfied with their media partner’s proprietary TV attribution model. These so-called black box models lack transparency; their owners have invested significant time and resources in creating “proprietary” algorithms and thus desire to protect their intellectual property. Understandable. So, what’s the problem?

The problem is the fact that attribution models provide value through understanding and interpreting their probabilistic viewpoint, not through trusting them as gospel. Yet some black box solutions claim to be just that.

We have looked at developing our own attribution model over the years; we do after all have hundreds of thousands of data points and 35 years of expertise. The conclusion is always the same however: all attribution models are flawed and there are plenty of third-party models from extremely well-funded companies such as TVSquared, Visual IQ, Google 360 and others. Our decision is to be model agnostic and instead help clients understand their strengths and shortcomings.

But back to the black box. Our client’s frustration ranged from lack of transparency to outright suspicion. “The temptation when you are trying to justify your model is to put your thumb on the scale,” says Michael Evert, Vice President of Data and Analytics at Northern Lights Direct. “Intentionally or not, it introduces a form of bias that cannot be discounted.”

“We would rather understand where the client is on their data continuum, and introduce the appropriate third-party attribution model,” says Evert. “Are they at the beginning of their data journey or are they full on using multiple models and looking at multi-touch, cross-channel attribution? One attribution model will not work in every situation, and we would never make such a claim. We would rather advise the client on their model selection and help them interpret results, based on any given model’s strengths and weaknesses.”

Attribution is becoming more and more important. We recognise the temptation to build a proprietary model leveraging our own learnings. But as the frustrated clients pointed out, it’s hard to buy the solution and the interpretation from the same source. Beware the black box.