On Polls and Pollsters

There have been two recent polls on the Liberal leadership race: a Strategic Counsel poll of 1000 Liberal members nationally, and an Ekos poll of 1000 members in Ontario and Quebec. Both purport to show a more or less even run between Ignatieff and Rae, with Dion polling strong.

Should we believe these numbers? Here's two reasons why I don't believe them yet. First, it's quite hard to draw a good sample of party members. They aren't always keen to share their opinions. And, as is becoming clear, a good portion of them probably don't even know that they're members, meaning they won't attend the DEMs. One good indication of how good the sample is the screens used to weed out effective non-members. Another is the number of calls each firm made to a name drawn from the list before giving up (the more the better). A final one is the response rate. To my knowledge, neither Ekos nor the SC have provided this information.

Second, the weighting of responses matters a lot. I assume that provincial weights have been applied. But, given that ridings have vastly uneven numbers of members, unless weights were applied to ridings (and the sample was inordinately large), it's unlikely that the regional support levels are anywhere near accurate to the conventional margins of error.

I've emailed principals in both firms with questions about their samples. Fairly straightforward ones, in fact. I've heard nothing back.

Now, this may just be because they don't respond to lowly PhD students who only know this stuff in theory but not in practice. Or, it may be because they don't care. But, either way, it speaks to either their professionalism or the quality of their polls, or both. All I know is that I can answer straightforward and simple questions about all the modelling decisions that I make.

Polls aren't magic, but they are good tools for understanding how a population thinks. Provided they are done correctly. I'd love to be assured that this is the case.