How to design opinion polls that lie.
The meaning of statistical terms used in polls.
"There are ... lies, damn lies, and
Most results announced by public opinion pollsters in recent years are quite meaningful, understandable and helpful. They are simple attempts to assess the public's opinion on a single topic. They are particularly useful to politicians who often gear their voting decisions to match either the majority opinion of the public, or the majority opinion of the voters within their party, or whichever vote they feel will maximize their chance of re-election.
However, when the topic of a poll is very controversial, or if the stakes are very high, some polls may be designed not for accuracy but to achieve a certain goal. Often the goal is not to estimate public opinion but to convince the public to change their mind, or to convince movers and shakers that public opinion is very different from what it really is. Currently, some polls about abortion access and same-sex marriage appear to have this type of ulterior motive.
Numerous U.S. national polls and polls within states have been conducted on same-sex marriage (SSM) by secular, political, and religious polling groups. Some have been sponsored by lesbian, gay, bisexual, transgender (LGBT) groups; others by religious or secular groups. There is often great variations in the results -- to the point that some wonder if fraud had taken place.
Some cooking of the results may have happened. For example, we have noticed one political polling group that asks their customers what their goals are, and then designs a poll and an explanation of the data to meet those goals. However, we suspect that fraud is generally not the cause of differences among polls. Sometimes, it can be the result of carelessness, ignorance and/or chance.
The following techniques can warp polling results; we should all look out for them:
- The exact question asked: For example, the question: "Do you believe that marriage should be between one man and one woman" will produce very different results than "Do you believe that same-sex couples should be allowed to marry?" The respondent might answer the former question differently depending upon whether they are answering with regard to themselves only, or their family, or their faith group, or the entire population. Some people may respond differently depending upon whether they view marriage as a religious ritual, a religious sacrament, and/or as a secular ceremony at city hall.
Changing the phrase "same-sex couples" to "homosexual couples" will probably also bias the result because many people use the term "homosexual" as a snarl word with heavy negative emotional overtones. Inserting a clause like: "change the definition of marriage to..." will probably raise red flags and lower the apparent support for SSM greatly. Similarly, changing the phrase "be allowed to" into "have the right to" may well increase support for SSM, because of the important that Americans place on individual rights.
It would be fascinating to see a poll that randomly selects a question on SSM from a group of, say, five alternatives, and then reports on how the responses differ among the questions asked. We have never seen such a poll. They probably don't exist, because they would reveal to the public how much the exact wording of the question does matter. That might well lower the credibility of polling in the public's eyes.
- Questions asked prior to the question on SSM can set up the respondent to answer in a desired way. See the example below.
- Simply by phoning only land lines, or only cell phones, or some ratio of land lines and cell phones, can result in an entirely different age distribution of respondents, and thus shift the end results.
- By simply phoning land lines during normal working hours on weekdays will shift the age distribution towards older, retired adults, and thus shift the end results without the bias being too obvious.
- Lacking data: Some polls are reported without the number of respondents and the margin of error. This makes the data essentially meaningless. See the discussion below.
- A polling group can simply propose many alternatives to the respondents. For example, ask them which arrangement they would prefer:
- For same-sex couples to be allowed to marry, or
- To enter into a civil union with all of the benefits, protections, and responsibilities of marriage except that they can't refer to their relationship as a marriage, or
- To enter into a domestic partnership with some of the benefits of marriage, or
- No state recognition of the relationship at all.
Giving many options would be almost certain to result in only a minority of respondents favoring same-sex marriage. The polling group could then accurately, but deceptively, describe the poll as indicating that most adults oppose SSM. What they don't reveal is that each of the options is opposed by most of the respondents.
An example of bias in the design of a public opinion poll:
Two large polls on the topic of same-sex marriage were
conducted in Canada in the early '00's by different polling companies. The results were obtained 3 and 4 years before access to marriage was widened across that country to include all loving, committed same-sex couples:
||2001-JUN: A Canadian Press / Leger
Marketing survey found that 65.4% of Canadian adults supported
same-sex marriage rights.
||2002-JUN: Strategic Counsel conducted a poll on behalf
of the fundamentalist Christian group Focus on the Family Canada. They found that 46% of Canadian adults supported same-sex marriage. 2,3,4
There are suggestions that support in both Canada and the U.S. for same-sex
marriage was increasing during that decade by about 1.5 percentage points a year. Why did these two polls show an almost 20 percentage point
drop in support over 12 months?
A major component of the reason may be found in the questions that were asked before the question on SSM:
||In the Canadian Press survey, Question 2 asked "In your opinion,
should homosexuals have the same rights as heterosexuals?" 76% of the
subjects responded "Yes." There is no record in their report as to
the nature of Question 3. However, Question 4 was worded "In
your opinion, should we grant homosexuals the following legal
rights...same-sex marriage?" Since more than three out of four
subjects already said that they were in favor of equal rights for gays and
lesbians, many would probably feel pressured to affirm same-sex marriage.
Otherwise, they might feel that they would be judged
inconsistent in their views. So this poll probably overestimated support for SSM significantly.
||In the Focus on the Family Canada survey, the two questions
(#59 and 60) that preceded the question on same-sex marriage (#60) were
whether "marriage is an outdated institution" and "Is it alright
for married people to have sex with people other that their spouse?"
Those questions may have preconditioned the subjects to feel defensively
towards marriage just before they were asked about their support for
same-sex marriage. This poll probably underestimated support for SSM significantly.
Please note that we are not implying that the bias in these polls was
intentional. It may have been the result of incompetence, carelessness, or simply a random
effect caused by the way in which the questions were sorted.
More details on these polls.
About the accuracy of polls and the importance of the margin of error:
One term associated with polling is the "true value:" This would be the value that one would obtain if every person (or, more likely, every adult) in the population was asked the question. A typical poll only involves 1,000 to 2,000 respondents. Thus, for state polls, it is only practical to take a sample involving a small percentage of the total population. The end result will be an estimate of the true value; the actual true value will be forever unknown.
It is impossible to understand the accuracy of a poll without knowing its margin of error. A typical margin of error value in professionally run polls is ±3 percentage points and involves a sample size of about 1,000 respondents.
Most margins of error are calculated at the 95% confidence level. In the case where the margin of error is ±3 percentage points, this means that one can have 95% confidence that the true value is within 3 percentage points of the reported percentage; anywhere from 3 percentage points higher to 3 percentage points lower than the reported percentage.
- 52.7% of the respondents in a particular poll are found to be in favor of the legalization of SSM, and
- the margin of error is ±3.1 percentage points at the 95% confidence level, then
One could say with 95% confidence that the true percentage of adults who favored the legalization of SSM is between 49.6% and 55.8%. 5
The margin of error is mainly a function of the number of respondents who actually contributed their opinion to the poll -- often referred to as "N." In cases where the total population is less than 20,000, the size of the population also becomes an important factor. Fortunately, the smallest state poll would be in Rhode Island which has a population of a little over 1 million. Thus the total population is not important in determining the accuracy of state polls; only the size of the sample is.
American Research Group, has a free Margin of Error Calculator 6 that computes the margin of error for a given population and sample size. They also have a free Sample Size Calculator that computes the required sample size among a given total population in order to achieve a desired margin of error. 7
For a large population like a city, state or province, or country -- even Monaco -- the actual population doesn't have a significant effect on the margin of error and the following relationship between the size of the sample "N" and margin of error for 95% confidence is:
|Size of sample "N"
||Margin of error
±2.17 perc. points
±3.08 perc. points
±4.37 perc. points
±6.19 perc. points
A sample size of about 1000 and a margin of error of ±3.1 percentage points is very commonly found among polling companies.
The following information sources were used to prepare and update the above
essay. The hyperlinks are not necessarily still active today.
- Variously attributed to Benjamin Disraeli, Alfred Marshall, Mark Twain, etc.
- "Canadian Perceptions of Homosexuality," Canadian Press /
Leger Marketing, 2001-JUN, at: http://www.legermarketing.com/ This is a PDF file. Survey data: N = 1507. Margin of error is ±2.6%
- "U.S. Public opinion polls on same-sex marriage," Marriage
Equality New York, at: http://www.marriageequalityny.org/
- "Canadian Attitudes on the Family: Complete Report 2002,"
Focus on the Family Canada at: http://www.fotf.ca/ This is also a PDF file.
- "Margin of error," Wikipedia, at: http://en.wikipedia.org/wiki/
- "Margin of Error Calculator," American Research Group, at: http://www.americanresearchgroup.com/
- Sample Size Calculator," American Research Group, at: http://www.americanresearchgroup.com/
Copyright © 2011 by Ontario Consultants on Religious
Originally written: 2011-JUL-02
Latest update: 2011-JUL-02
Author: B.A. Robinson