<$BlogRSDUrl$>

Maintaining Optimism in the Face of Reality. Occasional observations on the state of the world, society, business and politics. Usually anchored by facts, always augmented by opinion.


John Zogby and the Heisenberg Uncertainty Principle  | e-mail post

An apology, as I am abusing the Heisenberg Uncertainty Principle in my title. Strictly speaking, the principle applies to the field of physics, quantum mechanics more specifically, and in casual interpretation suggests that the accuracy of measurement is necessarily influenced or compromised by the very act of measuring. This is not precisely accurate however, and the curious can read more about the subject.

Despite its inaccuracy, my reference to it in the title is based on the common interpretation, and my real point is that John Zogby is not so much an outstanding pollster as he is an outstanding showman, and his role as pollster celebre may actually be of more value to those for whom is a partisan than those seeking news.

In their Election Issue, The New Yorker ran a pretty in-depth piece about Zogby written by Larissa MacFarquhar. (If you're curious about Ms MacFarquhar's leanings, you can read her glowing piece on Michael Moore, "The Populist: Michael Moore Can Make You Cry" as well.) The article on Zogby was what prompted me to buy that particular issue, although I might subscribe again. Not surprisingly, the Election Issue bordered on Kerry campaign literature.

Zogby is certainly a smart guy and fairly creative. You can get a sense of his unconventional approach to getting a feel for voter sentiment:
"How do I get a handle on this election or any other?" he asked the road builders. "I asked one question the Saturday before the election in 2000. I called my call center in Utica and said, 'Put this in the poll: "You live in the land of Oz, and the candidates are the Tin Man, who’s all brains and no heart, and the Scarecrow, who’s all heart and no brains. Who would you vote for?"' The next day, I called Utica and said, 'Whaddaya got?' They said, 'Well we’ve got Gore—,' I said, 'I don’t care about Gore. What’s Oz?' It was 46.2 for the Tin Man and 46.2 for the Scarecrow. It was right there that I knew I wasn’t going to know what was going to happen. But I asked this question again two weeks ago and the Tin Man led by ten points."
Clearly, John Kerry is the Tin Man.

Setting aside his pop-psychology. In any event, the article presents a very sympathetic portrayal of John Zogby and his sense of exclusion from the "clubby" world of pollsters. The truth is a little more complex, though:
Zogby thinks of himself as a natural maverick who stands outside the clubby world of the other pollsters because he finds it pompous and stuffy and because he isn’t a joiner anyway. But it is also true that he uses techniques that are frowned upon by AAPOR, the American Association for Public Opinion Research, as unscientific or unethical.
Of course, after listing the litany of complaints the "polling establishment" has with John Zogby's methods, MacFarquhar brushes them aside simply, "But Zogby doesn’t want to be scientific: he wants to be right." How nice. The fact is, his track record isn't half as good as his spin is, though. But before looking at that, what are the complaints those clubby pollsters have:
It's of interest to note that Zogby probably doesn't particularly care for the non-partisan, non-profit Pew Research Center, as the article notes that when Pew asked for information about his statistical adjustments in the 1998 race, he wouldn't give them up. Most pollsters work with Pew, as they serve a very important function in analyzing aggregate polling results and as a non-profit, aren't really competitors to the main business of pollsters, which is usually corporate research.

Now, what about Zogby's accuracy? As MacFarquhar said, "Zogby doesn’t want to be scientific: he wants to be right." He rose to some prominence when he predicted a much closer election in 1996 between Dole and Clinton, and then got some kudos again for calling Al Gore with the popular vote in 2000. None of this is nearly as impressive as George Gallup's first poll where he not only called FDR's reelection, but also predicted well in advance the incorrect results of the nation's previously-revered presidential poll from Literary Digest within one point. The main thing is that this is a state-by-state electoral vote race, and I don't think that's where Zogby shines.

Obviously, the proof has got to be in the pudding, so we're still a week away from knowing, but in the 2002 research conducted by the non-profit National Council on Public Polls tells a story . (The NCPP is another one of those "clubs" that Zogby is apparently too much of a "maverick" to join, or maybe it's just their disclosure standards that preclude him.) In any event, the 2002 NCPP results (which I have augmented with an accuracy percentage in the last column:

PollsterRacesAvgError*
WrongAccuracy
Mason-Dixon23
1.80%196%
Zogby International
172.50%
571%
Research 2000
13
2.10%
285%
Gallup7
1.40%
186%
Quinnipiac College
4
2.00%
0100%
All Others
952.70%
1287%
Total159
2.40%2187%

* The AvgError column is the "Average Error on Candidate" is best described byt he NCPP in footnote 4 of their results: "The candidate error reported here is half the error on the margin between the top two candidates. The error was calculated by subtracting the margin between the top two candidates in a poll from the margin between the same candidates in the election. For example, if a race was won by 55% to 45% the margin is 10 percentage points. If a poll reported a lead of only 47% to 43% with 10% undecided, the 4-point margin in the poll would be off by 6 percentage points. The candidate error in this case was counted as 3 points, half the error on the margin. No method of judging the error works perfectly."

71%. Hmm. Lower than any other major pollster, lower than the average. Even if we don't grade him on the curve, he got a C- in 2002. Ironically, you can buy Zogby's hindsight-based analysis of the 2002 election, "Decision 2002: Why The Republicans Gained."

I'll put my bias out there, I've been a fan of Mason-Dixon polls for years. They have a very good reputation and a good track record to point to, as the 2002 numbers attest. I'll certainly be curious to see how this race plays out. I'll also be honest that I don't respect Zogby, other than possibly as a businessman. I think he is a far better self-promoter than he is a pollster; his results and his approach both attest to that. I pimped on him last year as an aside about his comments on polling in Iraq (when he erroneously claimed to have conducted the first public opinion polls over there). I've also pointed out comparative analysis that indicates his systematic liberal bias in national polling.

I think for a good chunk of the election cycle, Zogby has been favoring Kerry. The tracking polls he's doing now for Reuters are starting to show Bush with a lead, which I think may be Zogby trying to preserve his reputation by shifting to more objective polling. Who knows? Maybe he's even using some of those crazy AAPOR guidelines he often thinks are too restrictive.

More on this later, I'm sure. In the meantime, Jay Cost points me to DJ Drummond's post in which he also takes Zogby to task: "All in all, Zogby’s habit of confusing his personal opinion with data-driven conclusions, his admitted practice of manipulating the respondent pool and his demographic weights, by standards not accepted anywhere else, along with mixing Internet polls with telephone interview results, forces me to reject his polls as unacceptable; they simply cannot be verified, and I strongly warn the reader that there is no established benchmark for the Zogby reports, even using previous Zogby polls, because he has changed his practices from his own history."



e-mail post | Link Cosmos | [Permalink]  |  | Thursday, October 28, 2004
Comments: Post a Comment

This page is powered by Blogger. Isn't yours?