How to read a poll

As readers of this blog know, I am not wild about public polls – they tend to focus people on the “horse race” at the expense of other areas of the campaign and way too often their read of the horse race or of changes in it are misleading. Nonetheless, they seem to proliferate so here is a short primer of what to look for to evaluate how real they are – or are not:

1. Do the demographics of the poll match those of the electorate? The distribution in the poll by age, gender, partisanship, race, education and geography should match that of the electorate. Now, these factors vary in the electorate depending on voter registration and turnout so the exact distribution for a future election is unknowable. Additionally, while we all have access to U.S. Census data, most of us do not have access to special modeled voter files that tell us this information for earlier elections. So some “guesstimating” is necessary for casual consumers of polls.

Still, the electorate isn’t radically different from the adult population, except perhaps by age as older people are more likely to vote than younger people. One thing to always watch out for is percent college educated because people with four-year college degrees are only a little more likely to vote but much more likely to complete polls. In Mississippi, 24 percent of adults over age 25 have four-year college degrees. The most recent MSToday poll of registered voters over age 18 had the figure at 21 percent. That is not unreasonable, although perhaps a tad low. On the other hand, I have seen polls that had the figure over 40 percent in Mississippi, which is not at all reasonable. It matters a lot because Governor Tate Reeves has more support among white voters without college experience than among white voters with four year college degrees. It also mattered a lot in producing the polling errors of 2016 as Hillary Clinton had a lot more support than Donald Trump among voters with four-year college degrees.

2. Is the partisanship correct? In states with party registration, like California or Florida, you can see whether the number of Democrats, Republicans and independents (or decline to state voters as they are known in California) is correct. With a special modeled file, statisticians have estimated the probability of each voters’ partisanship in every state and polls that use those files rely on that modeling. When neither of those is available, partisanship can rely on party self-identification or on prior vote. Both of those methods are somewhat problematic. Self-identification is an attitude and can fluctuate over time – someone may see themselves as a Republican today but start thinking next week they are more of an independent, particularly if they anticipate crossing party lines in the next election. Prior vote – for whom people voted in the last election – relies on their memory, and there is a tendency to recall voting for the winner. Still recalled Presidential vote, since people felt pretty strongly about that one, can be a useful measure and ground those who say they are independents as leaning one way or the other in reality. In the last two MSToday polls, party self-identification shifted quite a bit from 35 percent Democratic in January to 27 percent in April while the Republican percent went up two points from 38 to 40 percent and the independent percentage went down five points. It may be that voters are feeling less Democratic but it is likely that some of the change was the result of sample fluctuation. If the state’s underlying partisanship is the same, the partisanship of the two samples should have been more similar than it was.

3. Know the real sample sizes (or be cautious of them). Every poll these days has been weighted. That means that when the data collection is done, the pollster looks at the sample and up-weights or down-weights respondents in some groups to reflect their representation in the electorate. If they do not have enough young people, or people without college experience, or voters in the Delta, they count those they do have extra – as if they were 1.1 persons (or more) instead of 1 – and they down-weight people in groups that are over-represented. Small weights make the poll better but larger weights – or multiple weights – can make a very small group of people count for too much of the poll. I once did a poll that was low on both Republicans and African Americans and made the rookie mistake of up-weighting both those groups at the same time creating a sample that had a lot of Black Republicans, which made it appear (wrongly) that my candidate was slipping among Black people.

Weighting can be tricky and as response biases have gotten worse, it matters more. Very few public polls report their weights or the actual sample sizes they collected. Ask for them – or know that in telephone polls the pollster has probably up-weighted younger voters – especially younger men – African Americans and Hispanics. In on-line polls, they have almost certainly up-weighted voters without college experience, and seniors. In either case, the actual sample sizes of these groups are likely smaller than they appear.

4. Take it all with many grains of salt. Polls can be very useful in understanding how other people are thinking about the world or about an election. But they used to be more of an exact science than they are because people used to be easier to reach. If everyone in the population of interest (people who will vote in the next election) is equally likely to be in the poll, then all the laws of probability apply and you know their opinions within a mathematical margin of error. But as the response rate to polls has plummeted, and in ways that are not at all random, those laws no longer apply. The collection has biases that have been adjusted by the pollster in line with their assumptions. The best pollster making the most studied assumptions still misses the mark sometimes. And changes in a horse race for an election that won’t happen for months may – or may not – mean anything at all. I hope to see more news coverage of what candidates are doing and saying, and leave the internal processes and strategic judgments to their campaigns – although I am still something of a poll addict and will look, even while shaking my head and wishing for more coverage of who these people are, what makes them tick, and what they would do if they win the office that they seek.

Right now, like most Mississippians, I know a lot more about Tate Reeves and the kind of leader he is than I do about Brandon Presley. That will change as the candidates, their campaigns, and the press each tell us more. All we really know right now is that Mississippians aren’t satisfied with the status quo leaving room for the challenger, whom most of us don’t know very well yet.

Author: dianefeldman1764

In December of 2018, I closed down the polling firm I operated for nearly 30 years. I continue to consult and write on research and politics, while living here in Jackson, Mississippi. Jackson is on the Pearl River and so I named the blog View from the Pearl. All views are my own, newly unfettered from the need to run a polling business or please anyone. Please click Follow to receive posts in email.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: