Chris Vance is on a mission to kill the messenger again. Four polls last week show Patty Murray with a substantial lead over Dino Rossi, yet Vance once again has been compelled to explain to his readers that The Elway Poll “often has been somewhat off” in its polls of Washington political races. Really?
Vance says, “the only way to measure a pollster’s accuracy is to compare their final poll to actual election results, if they release a poll within a week or so of election day.” There are many things wrong with that assertion, but let’s accept it for the sake of argument. The first thing to know that The Elway Poll has never done or published a poll within two weeks of an election. I believe that the purpose of public surveys is to explain the election dynamics, not to predict the outcome. No public purpose is served by last-day polls. I know this sounds quaint in the current atmosphere, but there it is.
Vance goes on to explain that SurveyUSA “routinely releases a poll just before the election, and in 2008 they were within 1 percentage point of the actual Washington state results for governor and president.” He also notes that Rasmussen’s final polls “came in earlier” but were also “largely accurate.”
As evidence that The Elway Poll is “off,” Vance abandons his own “last week of the election” criterion and compares those last-day polls to an Elway Poll in September of 2004 and a poll in mid October of 2008. The new criteria is that our 2004 poll was “in contrast to what other polls were showing at the time.” And the 2008 poll overestimated the size of Gregoire’s eventual victory over Rossi.
So let’s accept those criteria and look at the record. How has The Elway Poll done at foreshadowing election results?
Since 1992, The Elway Poll has published a total of 29 polls in September and October on races for President, US Senator and Governor. The eventual winner was leading in 25 of those 29 polls (we had 86 percent right). Two of the four we missed were in the 2000 Senate race — we had Gorton ahead by 1 point in September and 3 points in October. Sen. Gorton lost by 1 point, well within the margin of error, but we’ll count them as misses. Our other two misses were in the 1992 Governor’s race, where we had Eikenberry ahead of Lowry in both September and October. Note that all four of our misses involved having the Republican ahead, in contrast to the assertion that The Elway Poll has a Democrat bias.
Across those 29 polls, we have underestimated the Republican candidate 27 times — by an average of 7 points. That is, the Republican ended up with a higher percentage of the vote than he received in our poll. We under-estimated the Democrat candidate 25 times — also by an average of 7 points. How can that be? As I have noted elsewhere, we typically ask a “soft” ballot question and do not push undecided voters. This results in larger “undecided” numbers in our polls than in others. Since there are no “undecided” voters in election results, the candidates’ vote totals will be higher than the poll results.
Next, let’s look at the specific polls Vance cites as evidence of our inaccuracy. The Elway Poll in September 2004 showed Gregoire at 49 percent, which is the vote she got in November. We showed Murray at 57 percent; she got 55 percent. That may be “in sharp contrast to what other polls were showing at the time,” as Vance asserts, but it hardly supports his other assertion that our polls “dramatically overstated the vote the two Democrats would receive two months later.”
How about that other criteron for judging pollsters work, “what other polls were saying at the time”? In the 2006 Senate race, the story line all year was that it was a very tight race. There were eight polls published between late April and mid June. Our two polls at that time showed Cantwell with leads over Republican Mike McGavick of 14 and 22 points (average equals 18; remember that number). The other six polls showed Cantwell’s lead at 4-7 points. Rasmussen had it at 5 and 4. In September, there were seven published polls. Cantwell’s lead in ours was 18. The other six ranged from 6 to 17 points — both the 6 and the 17 were from Rasmussen polls, conducted two weeks apart. Our October poll (two weeks out) showed Cantwell ahead by 18. SurveyUSA’s last poll, one week later, had Cantwell ahead by 7. She won by 18 points.
In the 2008 Governor’s race our polls from July on consistently showed Gregoire well ahead of Dino Rossi, while no other poll showed a lead of more than 4 points. Our September poll, which Vance omits in his critique of our polls, had Gregoire up by 8 points. That same week, SurveyUSA had Rossi up by 1 and Rasmussen had Rossi up by 6. True, we bounced up to a 12-point Gregoire lead in October, but during the same week, SurveyUSA had her lead at 1 point and Rasmussen had it a 2 points — both “too close to call.” Gregoire won by 6. So yes, we were “in contrast to what other polls were showing at the time.”
This year’s story line is familiar. The race is believed to be tight. The national polls are all over the yard, but The Elway Poll has consistently shown a comfortable lead for the Democrat. Partisans on the short end of the data attack The Elway Poll.
Am I saying that Murray will win by 13 points? No. All I am saying is all any pollster can ever say: I am 95 percent certain that my last poll was accurate within the margin of error (4.6%) at the time it was taken. Remember, any survey is only a snapshot it time and polls do not predict the future. All that boring stuff matters about margin of error and confidence intervals and the rest.
I think the problem that Vance and his friends have with The Elway Poll is that it keeps indicating that Republicans are losing. But they keep losing. What am I supposed to do?
