Sunday, February 13, 2011

Pollsters advising us to be wary of polls

This Canadian Press report today on polling and the abundance of polls that are littering the Canadian political landscape these days provides some welcome perspective: "Pollsters advise voters to be wary of polls ahead of possible spring vote!" A must read for all following Canadian politics and hopefully it will get some wide coverage due to its importance. Since I somehow got into the act of beating a drum on this issue this week, criticizing what seemed to me a ridiculously early poll on the perimeter deal and as well, criticizing that death penalty poll from late January, thought I would link to it as a good follow-up.

Frank Graves of Ekos, Allan Gregg of Harris-Decima, Andre Turcotte of Carleton University (pollster and communications prof), Jaideep Mukerji of Angus Reid and Michael Marzolini all provide some input to the piece, giving it a good dose of credibility.

There is lots of useful information canvassed in the report, including Gregg's remark that sets the tone: the "dirty little secret of the polling business...is that our ability to yield results accurately from samples that reflect the total population has probably never been worse in the 30 to 35 years that the discipline has been active in Canada." That's significant. The move in the population from land lines to cell phones and difficulties in getting people to respond are all contributing to the dynamic. These polls should be qualified to a much greater extent but the "unholy alliance" between media and pollsters doesn't permit that "dirty little secret" to take hold. Pollsters want coverage, it's good for business. Media want horse races and excitement, it's good for their business. At least the pollsters like Gregg and Graves are being quite honest here about the dynamic that's come to be.

Also raised, the advent of online polling and the difficulties in terms of how they are being presented. The report states that their usage is "...more controversial when it comes to surveys of the general population, which is what political polls purport to be." Online polls "may be skewed" as a result of the selection process for respondents (younger, connected to internet, induced by pay, etc.). Therefore, reporting margins of error with these polls, something that can only be done if you have randomly sampled the entire population, is "misleading and prohibited" according to the Marketing Research and Intelligence Association (MRIA), the polling industry's "voluntary self-regulating body." But pollsters are doing so. Something to keep in mind when you see an online poll with a reported margin of error. For example, that death penalty poll, raised in the blog post linked to above, was such a case.

Other key points:
  • "Turcotte says political polls for the media are "not research anymore" so much as marketing and promotional tools." (!!!!!!!!!!!!!)
  • the insufficient attention to margins of error in the reporting on polls:
"Take a poll that suggests Tory support stands at 35 per cent, the Liberals at 30. If the MOE is, say, 2 percentage points, that means Tory support could be as high as 37 and the Liberals as low as 28, a nine point gap. Or the Tories could be as low as 33 and the Liberals as high as 32, a one point gap.

If support falls within those ranges the following week, it should be reported as no change — but rarely is. A two or three point change is more likely to be touted as one party surging or the other collapsing."
Marzolini is quoted near the end of the report, touching on what may be the very unfortunate psychological effect that is being created in the population when constant horse race numbers are presented. "...voters, — "just want to get the score for the game; they don't want to watch the game."

A much needed check-up on polling in Canadian politics.