Polling is Leaving Out Poor People

Those who follow such matters already know that pollsters under-sampled white, non-college voters in 2016. Then, in 2020, Trump voters exhibited greater than average response bias as they were less likely than others in their demographic to respond to polls.

The problems with polling are not only about Trump voters, or about election projection for that matter. The core problem is that some people are less likely to respond to polls. Pollsters “correct” for this by up-weighting those who do respond – counting their responses extra and assuming the respondents represent their demographic. Some groups who are not Trump voters but consistently require up-weighting are low income people, people in minority communities, lower propensity voters, and young people.

Low income people and lower propensity voters (groups that overlap significantly) have always been harder to poll. Some of the difference is behavioral. Low income people are often less available – more likely to work nights, to move frequently, or to use a burner phone without any listing. They may also associate polls with the government, or the media, or other elites – the establishment if you will – and have little interest in unnecessary interaction with those (which is likely part of the problem with Trump voters).

Question wording is also often a problem. If people are asked to choose among response alternatives that do not reflect their views or concerns, they are more likely to terminate the interview. Many polls on COVID vaccination do not include cost as a barrier, assuming that people know the vaccine is free although free health care is outside the experience of most people, particularly those who are lower income.

Pollsters’ increasing use of online panels may be making the problem of getting a representative sample of low income people worse. Such panels are recruited in advance and demographically “balanced” to represent the population.

The first problem is that rather than eliminating response biases they are simply injecting bias earlier in the process as the panel consists of people who have agreed in advance to be polled.

Second, online panels eliminate some low income people from polling samples entirely. In 2019, 86.6 percent of households had some form of internet access, including 72 percent with smart phones. But the percent varies by state, ethnicity, and income, according to the ACS https://nces.ed.gov/programs/digest/d17/tables/dt17_702.60.asp which has been clear about the problems in needing to weight census data in 2020 given the low response rates of low income people https://www.census.gov/newsroom/blogs/research-matters/2020/09/pandemic-affect-survey-response.html.

Finally, if panel recruitment is by phone or mail, it may be skipping those who are more transient or who do not respond to such calls for all the reasons described above. And even with pre-recruitment, most panels are up-weighting low income people because they are not responding at the same rate as other panelists even when the recruitment is more balanced.

Does the exclusion of low income people from polls matter? Superficially it may not matter very much to political campaign strategists because they are interested in likely voters and willingness to be polled and vote propensity are related (per Pew Research studies). However, the relative absence of low income voters may misinform the campaign about what is on people’s minds, especially in lower income states and districts. If the campaign is considering investment in organizing low income communities, the exclusion reduces the potential for that strategy.

Not-for-profit organizations that wish to provide services to low income people should be very careful about relying on polls. Research has shown large response biases in health care research (https://link.springer.com/article/10.1007/s11606-020-05677-6), for example. Collecting data on site or in person may be far more valuable, and personal interviews are becoming feasible once again.

Most of the publicly released polls on issues like COVID vaccination are reporting data by income. In some cases, the income categories are cruder than they should be (e.g. below $40K as the lowest). In virtually all public surveys, the data are weighted but information on the degree of weighting applied is unavailable. If, as in Mississippi, nearly 20 percent of the population of interest is below the poverty line, how many were interviewed in a sample of 500 before weighting? If there were only 50, that wasn’t a meaningful sample from which to weight.

Every consumer of polls should know what the unweighted data looks like. And every consumer of polls should be a little skeptical of results in groups that required significant weighting or were unbalanced demographically without it. If your interest is in a group that is up-weighted, like lower income people, you may have learned less than you think.

None of this should suggest that such polls are without value. But they shouldn’t be seen as all encompassing. There is no substitute for conversation, and articles like these https://www.nytimes.com/2021/04/30/health/covid-vaccine-hesitancy-white-republican.html may be more useful and informative than some of the published online panel data in understanding what lower income communities are thinking and feeling on issues of concern.

There are other groups who are under – or over- represented in polls. Under sampling low income people seems both egregious and important at this time. But, as I have written before, the core problems on sampling call for new research methodologies as well as for greater care by pollsters and greater caution from those who consume data.