Maintaining Optimism in the Face of Reality. Occasional observations on the state of the world, society, business and politics. Usually anchored by facts, always augmented by opinion.
Despite its inaccuracy, my reference to it in the title is based on the common interpretation, and my real point is that John Zogby is not so much an outstanding pollster as he is an outstanding showman, and his role as pollster celebre may actually be of more value to those for whom is a partisan than those seeking news.
In their Election Issue, The New Yorker ran a pretty in-depth piece about Zogby written by Larissa MacFarquhar. (If you're curious about Ms MacFarquhar's leanings, you can read her glowing piece on Michael Moore, "The Populist: Michael Moore Can Make You Cry" as well.) The article on Zogby was what prompted me to buy that particular issue, although I might subscribe again. Not surprisingly, the Election Issue bordered on Kerry campaign literature.
Zogby is certainly a smart guy and fairly creative. You can get a sense of his unconventional approach to getting a feel for voter sentiment:
"How do I get a handle on this election or any other?" he asked the road builders. "I asked one question the Saturday before the election in 2000. I called my call center in Utica and said, 'Put this in the poll: "You live in the land of Oz, and the candidates are the Tin Man, who’s all brains and no heart, and the Scarecrow, who’s all heart and no brains. Who would you vote for?"' The next day, I called Utica and said, 'Whaddaya got?' They said, 'Well we’ve got Gore—,' I said, 'I don’t care about Gore. What’s Oz?' It was 46.2 for the Tin Man and 46.2 for the Scarecrow. It was right there that I knew I wasn’t going to know what was going to happen. But I asked this question again two weeks ago and the Tin Man led by ten points."Clearly, John Kerry is the Tin Man.
Setting aside his pop-psychology. In any event, the article presents a very sympathetic portrayal of John Zogby and his sense of exclusion from the "clubby" world of pollsters. The truth is a little more complex, though:
Zogby thinks of himself as a natural maverick who stands outside the clubby world of the other pollsters because he finds it pompous and stuffy and because he isn’t a joiner anyway. But it is also true that he uses techniques that are frowned upon by AAPOR, the American Association for Public Opinion Research, as unscientific or unethical.Of course, after listing the litany of complaints the "polling establishment" has with John Zogby's methods, MacFarquhar brushes them aside simply, "But Zogby doesn’t want to be scientific: he wants to be right." How nice. The fact is, his track record isn't half as good as his spin is, though. But before looking at that, what are the complaints those clubby pollsters have:
- Because Zogby's cheap, he doesn't use random-digit dialing, he uses listed phone numbers. This is a no-no, although Zogby defends that there is no demographic difference between the groups. The article notes most pollsters say those with unlisted numbers are disproportionately rich or poor. Maybe Zogby thinks it balances out. However, think about it John: the rich don't want to be bothered by people for different reasons than the poor, who mostly are looking to avoid skip-tracers and bill collectors; which group do you think is more likely to vote?
- Zogby doesn't feel the need to disclose his clients in published polls, which the AAPOR considers unethical. He claims that if his polls are objective (big if), it shouldn't matter who is paying, and he does require the release of all the poll data if the client wants to release it to the public. But that's not exactly right either:
But if a client wishes to publish only part of a survey—the results that he likes—Zogby offers another, rather underhanded option: he will conduct a second survey, using only the questions from the first one whose answers served the client’s purposes, and the client can publish that and keep the first poll private.
- He frequently includes very leading questions in his polls. An example cited in the article
"If you had to choose one of the following ways to wage the war against terrorism, which would you choose?" and offers these options: "Changing America’s Middle East policy," "Use of strategic nuclear weapons," "Attacking Iran," "Contracting with mercenaries," "Using U.S. Military Covert/Special Forces," "U.S. Air Force bombing," and "Biological warfare." If there exist choices beyond Arafat or war, they are not offered.Even the sympathetic MacFarquhar notes that some of his polls can sound like "push polls," polls that are more designed to influence opinion than simply measure it. (Although in fairness, they aren't exactly push polls, as they are defined by a lack of actual data collection, they're simply for influencing opinion. (BTW, about those Muslim-American polls, don't forget that John's brother James Zogby is the founder and President of the Arab-American Institute.)
- The real whopper, though, the reason that some in the AAPOR consider Zogby a hack, is that he weights for party identification. Everyone weights polls for demographics (race, sex, income, etc), because you know what those are from census numbers and it lets calibrate your sample with the actual population, but the whole point of polling is to gather political opinion. "Traditional research people trained in statistics just roll their eyes," Andrew Kohut, the former head of Gallup and now the president of Pew, says. "Party is a loose attitude, not a demographic."
Now, what about Zogby's accuracy? As MacFarquhar said, "Zogby doesn’t want to be scientific: he wants to be right." He rose to some prominence when he predicted a much closer election in 1996 between Dole and Clinton, and then got some kudos again for calling Al Gore with the popular vote in 2000. None of this is nearly as impressive as George Gallup's first poll where he not only called FDR's reelection, but also predicted well in advance the incorrect results of the nation's previously-revered presidential poll from Literary Digest within one point. The main thing is that this is a state-by-state electoral vote race, and I don't think that's where Zogby shines.
Obviously, the proof has got to be in the pudding, so we're still a week away from knowing, but in the 2002 research conducted by the non-profit National Council on Public Polls tells a story . (The NCPP is another one of those "clubs" that Zogby is apparently too much of a "maverick" to join, or maybe it's just their disclosure standards that preclude him.) In any event, the 2002 NCPP results (which I have augmented with an accuracy percentage in the last column:
* The AvgError column is the "Average Error on Candidate" is best described byt he NCPP in footnote 4 of their results: "The candidate error reported here is half the error on the margin between the top two candidates. The error was calculated by subtracting the margin between the top two candidates in a poll from the margin between the same candidates in the election. For example, if a race was won by 55% to 45% the margin is 10 percentage points. If a poll reported a lead of only 47% to 43% with 10% undecided, the 4-point margin in the poll would be off by 6 percentage points. The candidate error in this case was counted as 3 points, half the error on the margin. No method of judging the error works perfectly."
71%. Hmm. Lower than any other major pollster, lower than the average. Even if we don't grade him on the curve, he got a C- in 2002. Ironically, you can buy Zogby's hindsight-based analysis of the 2002 election, "Decision 2002: Why The Republicans Gained."
I'll put my bias out there, I've been a fan of Mason-Dixon polls for years. They have a very good reputation and a good track record to point to, as the 2002 numbers attest. I'll certainly be curious to see how this race plays out. I'll also be honest that I don't respect Zogby, other than possibly as a businessman. I think he is a far better self-promoter than he is a pollster; his results and his approach both attest to that. I pimped on him last year as an aside about his comments on polling in Iraq (when he erroneously claimed to have conducted the first public opinion polls over there). I've also pointed out comparative analysis that indicates his systematic liberal bias in national polling.
I think for a good chunk of the election cycle, Zogby has been favoring Kerry. The tracking polls he's doing now for Reuters are starting to show Bush with a lead, which I think may be Zogby trying to preserve his reputation by shifting to more objective polling. Who knows? Maybe he's even using some of those crazy AAPOR guidelines he often thinks are too restrictive.
More on this later, I'm sure. In the meantime, Jay Cost points me to DJ Drummond's post in which he also takes Zogby to task: "All in all, Zogby’s habit of confusing his personal opinion with data-driven conclusions, his admitted practice of manipulating the respondent pool and his demographic weights, by standards not accepted anywhere else, along with mixing Internet polls with telephone interview results, forces me to reject his polls as unacceptable; they simply cannot be verified, and I strongly warn the reader that there is no established benchmark for the Zogby reports, even using previous Zogby polls, because he has changed his practices from his own history."
e-mail post | Link Cosmos | [Permalink] | | Thursday, October 28, 2004