The core problem with polling is that people do not wish to be polled. Those who answer their phones when the caller is unknown to them are unusual and atypical. And even many who do answer do not choose to complete the poll.
This year’s telephone polling results were closer to the final election results than in 2016. Much of the improvement, however, was in the nature of the mid-term electorate and not because the polls themselves were better. The mid-term electorate was highly polarized, and rabid partisans are easier to poll than voters in the middle. Polls were still wrong when those in the middle did not break proportionately to the partisans.
Back in the 1980s, polling achieved representative samples of voters by calling phone numbers at random. The definition of random is that everyone in the universe of interest (people who will vote in the next election) has an equal chance of being polled. With the advent of cell phones, caller ID, and over-polling, samples have not been random for a while – not since the last century anyway.
Pollsters replaced random samples with representative ones. Political parties and commercial enterprises have “modeled” files – for every name on the voter file, there is information on the likely age, gender, race or ethnicity and, using statistics, the chances that individual will vote as a Democrat or Republican. If the sample matches the distribution of these measures on the file, then it is representative and the poll should be correct.
There are three problems (at least) with that methodology: (1) there may be demographics the pollster is not balancing that are important; pollsters got the 2016 election wrong in part because they included too few voters without college experience in samples and college and non-college voters were more different politically than they had been before. (2) rather than letting the research determine the demographics of the electorate, the pollster needs to make assumptions about who will turn out to make the sample representative – including how many Democrats and how many Republicans. When those assumptions are wrong so are the polls. This year, conventional wisdom was correct and so the polls looked better.
The third problem is perhaps the most difficult and follows from the first two: pollsters “weight” the data to their assumptions. If there are not enough voters under 30 in the sample (and they are harder to reach) then pollsters count the under 30 voters they did reach extra – up weighting the number of interviews with young people to what they “should” have been according to assumptions. Often, however, the sample of one group or the other wasn’t only too small, but was an inadequate representation in the first place – a skewed sample of young people is still skewed when you pretend it is bigger than it actually was.
The problems can be minimized by making more calls to reduce the need to up-weight the data. If 30 percent of some groups of voters complete interviews but only 10 percent of other groups, just make three times the number of calls to the hard to reach group. That is what my firm and others did this year. It is, however, an expensive proposition and still does not insure that the people who completed interviews are representative of those who did not.
Next Post: The Self-Selecting Internet
ViewfromthePearl.com 