How to Evaluate Public Opinion Polls

Public opinion polls are reported on every day, it seems, in newspapers and magazines. A well-done poll can provide valuable information about the public's priorities and viewpoints. A poorly constructed poll may mislead readers. Becoming an informed consumer of public opinion research can help you assess the value and validity of the data you encounter.

Ask Your Own Basic Questions
The first questions to ask when evaluating poll results are: "who conducted the poll?" and "who paid for it?" The biases of a poll's sponsors can be reflected in its design — skewing results. Following are a few other things to look for when considering poll results.
Who was polled? Public opinion research seeks to understand the opinion of an entire group by asking questions of a scientifically constructed sample of the group. Critical to understanding any poll data is knowing what group was sampled. Are the opinions reported those of all adults or just voters; all Californians or just those who live in cities; all parents or just those with school-age children?
To be statistically valid, the sample itself must be randomly selected and of adequate size. In general, the larger the size of the sample, the more reliable the results.
Any report based on the opinions of people who actively choose to participate — such as magazine readers who mail back a survey or callers to talk radio shows — is not built on a representative sample of the larger group. While such surveys might be interesting, they cannot be construed to represent the opinions of a larger group.
How were the questions worded? The exact wording of survey questions can have a major influence on the results. Taking a close look at how the questions were worded is one of the most important things a reader can do to evaluate public opinion research. The order in which questions are asked can also affect a poll's results.
Two types of questions frequently used in polls are "forced choice" questions, where respondents are provided with a list of answers to choose from, and "open-ended" questions, which allow respondents to give whatever answer they like. Open-ended questions are best used when the researchers know little about the public's opinions on a particular topic. The responses to an open-ended question reveal the entire range of beliefs people hold on the topic. The varied answers to open-ended questions are grouped into larger categories by the poll takers for reporting purposes. This must be done carefully to avoid biasing the conclusions.
When asking questions about public policy issues such as education, forced choice questions work well when public opinion researchers already have some idea of the options the public is considering. Asking respondents to pick only one option from a list, or to rank the options, can reveal their priorities among alternatives in a way that an open-ended question cannot. What is included on the list of choices is crucial. The results only identify the public's priorities among the alternatives offered; if an option that has meaningful public support is left off, the results may be confusing or incomplete. In addition, the order in which the choices are presented must be systematically rotated during the polling to ensure unbiased results.

What was the timing of the research? A poll is a "snapshot" of the opinions of a group of people at a particular time. It is important to consider the timing of a survey. Did it solicit opinion on a particular topic soon after a related event occurred? An example would be conducting a survey on air safety a week after a major air disaster. The results might reveal more about the public's emotional response to the disaster than about their long-term beliefs about the dangers of flying.
To understand how steady a particular opinion is, or how it changes over time, researchers sometimes ask the exact same question to different samples of a target population at regular intervals. Much important research has come from this kind of "longitudinal" polling.
What was the margin of error? Reports on poll results usually include a margin of error. Typically 5% or less, it is a function of how many people were sampled. A 5% margin of error on a poll means there is a 95% probability that the results represent "true public opinion", the results that would be obtained if every single member of the public were polled. In addition, if a result is 80% on a question, a 5% margin of error indicates that true public opinion lies between 75% and 85%.

Breaking Out Opinions of Subgroups
Polls often report on the opinions of subgroups of the general population, such as racial and ethnic groups, men and women, or different age groups.
If a subgroup makes up only a small part of the public as a whole, researchers may "oversample" it, meaning that they survey more people from that group than would appear in their random sample of the public. The oversample assures that enough members of the group are surveyed to make their responses statistically valid. But when the public as a whole is studied, the oversampled group's opinions are adjusted to carry weight equal to its proportion of the public.
Differences in opinion between subgroups and the public as a whole must vary by more than twice the poll's margin of error to be statistically meaningful.

Measuring the Quality of Public Opinion
Daniel Yankelovich, a leader in the field of public opinion research and founder of Public Agenda, identifies three components of quality to look for in public opinion findings. They are:
Volatility or firmness. Is the opinion a deeply held belief, or an off-the-top-of-the-head reply? Public opinion researchers have a number of techniques for trying to assess this. The most common is asking the same question twice, to see if opinions shift due to changes in wording or in response to issues raised or information provided between the two questions.
Consistency. Does the opinion fit in with, or contradict, other beliefs the respondent holds? For example, the public may endorse trade protectionism when it is thinking about preserving jobs, but reject it when the focus is on consumer values such as lower prices, better quality, and more choice.
Accepting responsibility for the consequences of its views. For Yankelovich, this is the essence of quality public opinion. It includes, yet goes beyond, firmness and consistency. Public opinion surveys can attempt to measure this by asking a series of related questions that explore the respondent's understanding of and feelings about the possible consequences of various choices.
In order to reach this kind of thoughtful judgment, the public needs information on the consequences of specific policy choices. It needs adequate time to digest events and work through the conflicting values raised by many of today's complex issues. Well-designed polls can measure public opinion. Providing the public with unbiased information on policy issues and tools to help them weigh the consequences of policy choices can improve it.

Public Opinion and Public Policy
Good public opinion research can allow the hopes and fears of the people to be heard. Poor public opinion research may reflect the bias of the researchers or merely highlight the issue of the hour. Anyone who plans to act on poll results — policymakers, advocates, and journalists, for instance — needs to be able to tell the difference. They can evaluate the statistical validity of a poll, search for consistent results in multiple polls over time, and consider the quality of the public's judgement to help guide how much weight any given poll research should be given.