Week 6: Measuring public opinion: attitudes, samples, and nonresponse

In plain words

Week 6 of the My Vote Forward seminar studies how public opinion is measured, and how measurements go wrong. Students read John Zaller’s theory of how people answer survey questions, Robert Groves on why low response rates may or may not bias results, the American Association for Public Opinion Research task force on samples that people opt into, a study that forecast an election from Xbox users, and Pew Research Center’s methods research.

A poll does not simply read people’s minds. The wording, the order of questions, who agrees to answer, and how answers are weighted all change the result. A large number of answers from people who chose to take part can still be badly wrong. This week teaches students to ask what a number actually measured before trusting it.

Defined terms

• Probability sample: a sample in which every member of the population has a known, nonzero chance of selection.
• Nonprobability (opt-in) sample: a sample made of people who chose to participate or were recruited without known selection probabilities. See opt-in sample.
• Nonresponse bias: error that arises when people who do not respond differ from those who do on what is being measured. See nonresponse bias.
• Weighting: adjusting the influence of each response so the sample matches known population characteristics. See weighting.
• Multilevel regression and poststratification (MRP): a modeling method that estimates opinion within many small population groups, then combines them according to the groups’ known sizes.
• Acquiescence bias: the tendency of some respondents to agree with statements regardless of content.

Learning objectives

By the end of week 6, a participant should be able to:

1. State Zaller’s four premises and explain what they imply about the stability of survey answers.
2. Explain, with Groves, why nonresponse rates and nonresponse bias are different things.
3. Summarize the AAPOR task force’s conclusions about nonprobability samples, including its transparency standard.
4. Describe how Wang and coauthors used MRP to adjust a highly nonrepresentative sample, and what conditions made that possible.
5. Use Pew’s research to explain why weighting opt-in samples removes only part of their bias.
6. Apply these lessons to judge a published priority ranking.

Core readings

1. John R. Zaller, The Nature and Origins of Mass Opinion (Cambridge University Press, 1992), chapters 1 to 3.⁠¹
2. Robert M. Groves, “Nonresponse Rates and Nonresponse Bias in Household Surveys,” Public Opinion Quarterly 70(5), 2006, pp. 646 to 675.⁠²
3. American Association for Public Opinion Research, Report of the AAPOR Task Force on Non-Probability Sampling (2013), executive summary and conclusions.⁠³
4. Wei Wang, David Rothschild, Sharad Goel, and Andrew Gelman, “Forecasting Elections with Non-representative Polls,” International Journal of Forecasting 31(3), 2015, pp. 980 to 991.⁠⁴
5. Pew Research Center, “Writing Survey Questions”.⁠⁵
6. Pew Research Center, “For Weighting Online Opt-In Samples, What Matters Most?” (January 26, 2018).⁠⁶

Recommended: Reg Baker and coauthors, “Research Synthesis: AAPOR Report on Online Panels,” Public Opinion Quarterly 74(4), 2010, pp. 711 to 781.⁠⁷

Lecture notes

From preferences to measurements

Measuring public opinion is the subject of week 6, and it is where the course’s theoretical questions meet data. Weeks 1 to 3 asked whether officials hear the public accurately. Week 5 asked whether individual rankings can be combined meaningfully. Week 6 asks a prior question: when someone records what a person prefers, what exactly has been recorded? The site’s chapters on measurement, question design, and opinion and choice treat these issues in the context of the My Vote Forward framework. This lecture places them in the research tradition.

Zaller: answers constructed on the fly

John Zaller’s The Nature and Origins of Mass Opinion develops “a comprehensive theory to explain how people acquire political information from elites and the mass media and convert it into political preferences.” The theory rests on four premises, as the book’s description states them. First, “individuals differ substantially in their attention to politics and therefore in their exposure to elite sources of political information.” Second, “people react critically to political communication only to the extent that they are knowledgeable about political affairs.” Third, “people rarely have fixed attitudes on specific issues; rather, they construct ‘preference statements’ on the fly as they confront each issue raised.” Fourth, in constructing these statements, “people make the greatest use of ideas that are, for various reasons, the most immediately salient to them”.⁠¹

The third and fourth premises matter most for this seminar. If people construct answers when asked, using whatever considerations are most salient at that moment, then the act of asking shapes the answer. The order of questions, a news story that morning, or an example in the question wording can change which considerations come to mind. Zaller also emphasizes “the role of political elites in establishing the terms of political discourse,” which connects to week 7 on agenda setting.

Zaller’s theory does not say opinion is meaningless. It says opinion is a distribution of considerations that a question samples from. A priority ranking is subject to the same process: what a person ranks first may depend on what they were thinking about when asked. That is an argument for stable, carefully designed questions and for repeated measurement, not for abandoning measurement.

Question design in practice

Pew Research Center’s guide to writing survey questions turns these concerns into practice. On question order, it states that “surveyors must be attentive to how questions early in a questionnaire may have unintended effects on how respondents answer subsequent questions.” On agree and disagree formats, it reports that “compared with the better educated and better informed, less educated and less informed respondents have a greater tendency to agree with such statements,” a pattern “sometimes called an ‘acquiescence bias,’” which is “even more pronounced when there’s an interviewer present.” Its recommended alternative is to “offer respondents a choice between alternative statements”.⁠⁵ The guide also stresses using the same wording and placement when measuring change over time.

For a priority list, these lessons translate directly. The list of subjects shown, their order, their labels, and any examples attached to them can all change rankings. Randomizing order and testing alternative labels are standard defenses.

Groves: response rates and bias are different

Robert Groves’s 2006 article addresses a worry that has grown as fewer people answer surveys. He observes that “many surveys of the U.S. household population are experiencing higher refusal rates,” and that “nonresponse can, but need not, induce nonresponse bias in survey estimates.” He notes “cases when the linkage between nonresponse rates and nonresponse biases is absent,” and yet, in a synthesis of studies, finds “the bias often present.” He poses the question directly: “what advantage probability sample surveys have if they suffer from high nonresponse rates.” His answer is that, because adjustment for nonresponse requires auxiliary variables, “the answer depends on the nature of the design and the quality of the auxiliary variables”.⁠²

The lesson is subtle and important. A low response rate is a warning, not a verdict. Bias arises when the reasons people do not respond are related to what is being measured. If people with strong views on a subject are more likely to respond to a survey about it, the survey will overstate the importance of that subject, whatever its response rate. A priorities channel is especially exposed to this mechanism, because people may choose to take part precisely because they care about a particular subject.

Nonprobability samples: the AAPOR standard

The American Association for Public Opinion Research convened a task force on nonprobability sampling, whose 2013 report remains a standard reference. Its conclusions include that “unlike probability sampling, there is no single framework that adequately encompasses all of non-probability sampling,” which is “a collection of methods” whose properties cannot be described all at once. The report insists that “transparency is essential”: “whenever non-probability sampling methods are used, there is a higher burden than that carried by probability samples to describe the methods used to draw the sample, collect the data, and make inferences.” It adds that “too many online surveys consistently fail to include information that is adequate to assess their methodology”.⁠³

The task force does not say nonprobability samples are useless. It says their validity rests on modeling assumptions that must be stated and on adjustment variables that must be good. This is the standard the site’s published releases chapter adopts.

The Xbox study: what modeling can do

Wei Wang, David Rothschild, Sharad Goel, and Andrew Gelman provide the most cited demonstration that a badly skewed sample can be useful. They built forecasts “from a novel and highly non-representative survey dataset: a series of daily voter intention polls for the 2012 presidential election conducted on the Xbox gaming platform.” After adjusting the responses “via multilevel regression and poststratification,” they obtained estimates “in line with the forecasts from leading poll analysts, which were based on aggregating hundreds of traditional polls conducted during the election cycle.” They conclude that nonrepresentative polling “shows promise not only for election forecasting, but also for measuring public opinion” more broadly.⁠⁴

Students should read this result with its conditions in view. The authors could check their adjusted estimates against a known benchmark: the actual election and many traditional polls. Their adjustment variables, such as demographics and past vote, were measured for both the sample and the population. A priorities channel lacks both advantages. There is no election that reveals the “true” public priority ranking, and the variables that predict participation in a priorities channel may not be the ones available for weighting.

Pew: weighting removes only part of the bias

Pew Research Center tested this directly for online opt-in samples. Across scenarios with “sample sizes as large as 8,000 interviews,” it reports that “none of the evaluated procedures reduced the average estimated bias across 24 benchmarks below 6 percentage points,” down from “8.4 points unweighted.” That means “even the most effective adjustment strategy was only able to remove about 30% of the original bias.” Pew also found that “choosing the right variables for weighting is more important than choosing the right statistical method”.⁠⁶

The site’s measurement chapter discusses further Pew findings: that opt-in samples overrepresent civically and politically engaged people, and that increasing sample size narrows the margin of error around an estimate without reducing its bias. That last point is the single most important lesson of the week. A larger opt-in sample is more precisely wrong, not more right.

Putting the week together

The readings converge on a set of questions to ask of any published measure of public opinion, including a priority ranking. What population is it meant to describe? How were people selected, and who chose not to take part? How were questions worded and ordered? What adjustments were made, using which variables, and how well do those variables predict both participation and the answers? Is there any benchmark against which accuracy can be checked? If the answers are missing, the number should be treated as a description of participants, not of the public. The site’s objection page is this just a poll? applies these questions to My Vote Forward.

A worked example: bigger is not better

Hypothetical example: a city invites residents to rank priorities online and receives 20,000 responses. Suppose, for illustration, that residents who use public transit are twice as likely to respond as other residents, because a transit advocacy group promoted the survey. The result will overstate transit’s rank. Collecting 40,000 responses through the same channel would not fix this; it would produce a tighter estimate around the same distorted value, which is the pattern Pew describes when it reports that larger opt-in samples narrowed margins of error without reducing bias. Weighting on age, income, and neighborhood would help only to the extent those variables capture the difference between transit users and others. A weighting variable for transit use itself would help more, but the city would need a reliable population benchmark for transit use, and a priorities channel rarely has one for the attribute that actually drives participation.

The example also shows why disclosure matters. A release that reported the outreach channels used, the share of responses arriving through the advocacy group’s link, and the results with and without those responses would let readers judge the bias for themselves. That is the transparency burden AAPOR places on nonprobability designs.

Discussion questions

1. If people construct answers on the fly, as Zaller argues, is there such a thing as a person’s “true” priority? What would it mean to measure it?
2. Groves says nonresponse need not cause bias. Describe a priorities channel in which low participation would cause little bias, and one in which it would cause a lot.
3. The AAPOR task force places a higher transparency burden on nonprobability samples. List the disclosures you would require before trusting a published priority ranking.
4. Why could the Xbox study check its accuracy, and why can a priorities channel not do the same? What could substitute for a benchmark?
5. Pew found that the best weighting removed about 30 percent of opt-in bias. Should an organization that cannot do better publish opt-in priority results at all?
6. How would you test whether the order of subjects on a priority list changes rankings?

Assignment

Measurement audit (1,500 words). Find one published poll, consultation result, or online “public priorities” ranking from a government body, news organization, or advocacy group. Using the AAPOR transparency standard and the readings, audit it: identify the population, sampling method, wording, order, weighting, and any benchmark; list what is disclosed and what is missing; and estimate, in qualitative terms, the likely direction of bias. End with a rewritten one-paragraph summary of the result that states only what the evidence supports.

Evidence

Zaller’s premises are quoted from the book’s published description. Groves, and Wang and coauthors, are quoted from their abstracts. The AAPOR conclusions are quoted from the 2013 task force report. Pew’s guidance and weighting findings are quoted from the two Pew pages. All were opened on the review date. The recommended 2010 AAPOR online panels synthesis was verified bibliographically; no claim on this page relies on it.

Competing interpretations

Survey methodologists disagree about how much nonprobability samples can be trusted. One view, supported by studies like Wang and coauthors, holds that with good models and auxiliary data, opt-in samples can approach probability samples at far lower cost. Another view, supported by Pew’s weighting experiments, holds that opt-in samples carry biases that adjustment cannot reliably remove, especially for groups and topics where participation is related to the answer. Zaller’s theory is accepted by many as a description of survey response, while others argue that aggregate public opinion is more stable and meaningful than individual-level instability suggests.

Application to this framework

My Vote Forward would collect priorities from people who choose to take part, which makes it a nonprobability design by default. This week sets the standard its published releases must meet: full disclosure of method, honest description of who took part, randomized presentation of subjects, and a refusal to describe participant results as the public’s view without evidence. The framework’s measurement chapter should be judged against Groves, AAPOR, and Pew.

Unresolved questions

• What benchmark, if any, could validate a priority ranking, given that no election reveals the true ranking?
• Could a priorities channel recruit a probability sample panel alongside open participation, and report both? What would that cost?
• How unstable are individual priority rankings over short periods? No public data on this exists for the framework.