Skip to contenttop

The Emperor's Rating System

The Nielsen model is outdated. So why do advertisers keep saluting it?

by Robert Duboff
The Nielsen television ratings are bloodied but unbowed. Periodically, people take shots at them (most recently by a consortium of groups and individuals concerned about possible under-representation of minorities on Nielsen panels in New York and other major markets), but Nielsen perseveres as a measuring stick that literally has a multimillion-dollar impact.
Nielsen itself has changed dramatically over the years, starting as a respected research business at the dawn of the business market research movement before becoming a part of Dun & Bradstreet. Nielsen was altered, if not damaged, by that company and its consultants who made it a more profitable business and then sold it off. Now it is a very profitable part of VNU, a Dutch information and media company. Nielsen continues to have a monopoly on the television rating business despite a few rivals nipping at its heels and the best efforts of potential new providers over the years.
The power and influence of Nielsen media research is truly amazing. As Nielsen's website proudly proclaims, its "information is the currency in all the transactions between buyers and sellers, which adds up to more than $60 billion in national and local advertising spending in the U.S. each year." The core of the national panel now is the meters installed in 5,100 households across the United States. Sounds impressive! But when Nielsen terms its sample as "randomly selected and recruited" and that the meters measure "two things - what program is being tuned and who is watching," the problems arise.
Where to start? First, the sample may be random in its design, but less than half the people contacted by Nielsen choose to participate. (According to The NTI Pocketpiece report as of July 25-31, 2005, Nielsen's response rate for its famous panel was 41.3 percent.) This low level of acceptance seriously undermines the credibility of the sample because it means that those who do accept are quite likely to be different from the majority who don't want to participate.
To make matters more problematic, not all households in the Nielsen Panel "turn in" data every day. In a real sense, the Nielsen sample can have a different profile every day. According to recent reports, Nielsen has a substantial "faulting" problem. From January 2005 through the end of July 2005, from 10 percent to 15 percent of the panel households were unable to provide useful data for any number of reasons, including non-cooperation and technical failures. The failures are not random across the panel. A disproportionately high number of household with minorities and young persons "fault out." This results in a sample that no longer looks like the population it purports to measure. (Hence the latest problems surfacing in New York and elsewhere.)
However, even if you want to accept the sample (which few experts who are not paid by Nielsen do), the problems with what is being measured are worse. Nielsen says it is measuring who is watching but, at best, it counts who was in the room when ads were being shown. Were the viewers watching or talking or eating or reading? Nielsen doesn't know - so neither does its customers.
Quantitative research is evaluated on its reliability (the same results would be obtained with a parallel sample over time) and validity (it is measuring the reality it is designed to assess). Nielsen ratings are neither.
Back in the day, before computers and cell phones and TiVo's, maybe knowing the set was on when the ad was shown was sufficient to infer the ad was being watched. That inference now flies in the face of common sense.
Yet advertisers still shell out millions for this ersatz measuring stick of how many people their TV ads reached - and money changes hands again once the ratings are in. It's as if the election polls decided the primaries and final elections.
Why has this happened, and why does it continue? One reason is that hard numbers drive out soft ones. The industry needs numbers to decide whether and how much to pay for commercial time. At the same time, the buyers and sellers don't really understand (or want to understand) market research. So the Nielsen numbers continue to be saluted, just as the Emperor was.
Nicholas P. Schiavone, former head of research for NBC and now a strategic research consultant, has closely followed ratings issues. His explanation: "Insofar as Nielsen ratings are considered the coin of the realm, even if they have all but lost their intrinsic value, as long as people see everyone else trading on them, then they feel obligated to trade on them as well."
There's yet one more level of absurdity. Even if Nielsen numbers were reliable and valid and actually measured viewers actually watching ads, the numbers would be interesting, but not truly useful, which many call a third aspect for evaluating the quality of quantitative research.
Exposure to advertising of any kind ought to be understood simply as an analytic starting point. The real test of an ad is whether it ultimately helps to motivate an action (typically an incremental purchase of some sort). The industry seems relieved at being able to measure eyeballs (or pretend that it can) for each ad produced and aired. This avoids the more important question of whether or not the ad affected those who saw it. It's kind of like the Emperor parable: All the focus on the clothes distracts the subject from the real issue of whether he could get the job done. Nielsen, at least as it is now performing, cannot.
Robert Duboff is CEO and cofounder of HawkPartners, a marketing consultancy based in Cambridge, Mass.
Printer version (PDF) below:
emperors_rating.pdf26.41 KB