Accuracy of public opinion surveys
gay marriage (a.k.a. same-sex
Part 1 of two parts
Numerous opinion studies of American adults have been conducted by polling agencies down through the years to determine the degree of national support and opposition to marriage equality. The polls attempt to determine:
Do adults favor restricting marriage to only the voluntary union of one woman and one man, who meet the requirements of the marriage law in their state, territory, or district where they live? These laws typically set a minimum age for marriage, sometimes allow marriage for persons of lower age with parental consent, and specify how closely related the spouses can be before a marriage is forbidden. For example, in some areas of the U.S., first cousins are allowed to marry; elsewhere they are forbidden because such marriages typically have double the incidence of genetic abnormalities among their children.
Do adults favor allowing engaged couples of either the opposite sex or the same sex to marry if they meet the same requirements of age and genetic closeness?
Typically, public opinion surveys will not precisely agree on the level of support and opposition, even if they ask the same question, are conducted on the same day, and select the people that they will poll using the same randomizing techniques. That is because most polls only ask the beliefs of a relatively small number of adults -- perhaps 1,000. So, even if the same polling agency conducts successive polls in exactly the same way, they will usually arrive at slightly different answers. What they can and should do is to report the result that they measured, along with a computed "margin of error" value associated with that result.
For example, a recent national U.S. poll conducted among 2,189 randomly selected adults might report that:
- 2,000 adults were contacted,
- 60% favor marriage equality,
- 32% are opposed,
- 8% are either undecided or declined to answer the question, and
- the margin of sampling error was ~+mn~2.2 percentage points.
The "margin of sampling error," which is often called the "margin of error," is mainly determined by the total number of persons who were interviewed. That is often referred to as the sample size. In the above hypothetical case, it means that if the agency were to repeat their survey a very large number of times at the same time, that 95% of the results would lie within plus or minus 2.1 percentage points.
Thus, suppose that a polling group conducted a national survey in early January among 2,180 people and obtained the above results. If they performed the survey again in late June and found that 62.0% of adults favor marriage equality, the increase would fall just inside the margin of error. Most ethical polling agencies would not claim that their results show that public support had actually increased over the six months. The apparent increase would be within the margin of error.
The Pew Research Center is one of the most highly respected polling agency in the U.S. They explain:
"Because surveys only talk to a sample of the population, we know that the result probably won’t exactly match the “true” result that we would get if we interviewed everyone in the population. The margin of sampling error describes how close we can reasonably expect a survey result to fall, relative to the true population value. A margin of error of plus or minus 3 percentage points at the 95% confidence level means that if we fielded the same survey 100 times, we would expect the result to be within 3 percentage points of the true population value 95 of those times. 1
As the number of the persons interviewed increases, the margin of error decreases. Using statistical methods, the following values can be computed for U.S. national polls:
Sample size (number of people interviewed)
The margin of error also varies slightly with the total population size. For example, a massive U.S. national poll that sampled the opinion of 16,000 persons from among the country's total population of about 242 million adults, the margin of error is ~+mn~0.77% as shown above. 2 If the same survey were conducted among 16,000 of the 500,000 adults who lived in a particular city, the margin of error would be reduced to ~+mn~0.76%.
A free "margin of error" calculator can be found at the American Research Group, Inc. 3 We recommend that you save the link. It is a very useful tool to have on hand if you are studying a poll result in which the margin is not stated, but the number of interviews is shown.
If you are scanning the Internet and find the result of a poll, check to see the precise question asked, and either the sample size or margin of error. If you find all three missing, we recommend that you ignore the result completely.
Unfortunately, there are many ways to design a poll so that it gives results that are not accurate. Some techniques are:
- Designing the question to be asked so that it contains certain negative trigger words like "redefine marriage" "gay lifestyle"
- Not defining the scope of the question. For example whether it refers to whether your gay/bisexual family members should be able to marry, or whether the teachings of your church support marriage equality, the wording of the laws regarding marriage, etc.
- Ask a bunch of questions first, that deal with negative topics like child sexual abuse, sexually transmitted infections, etc.
- Conduct the survey by phoning residences during normal working hours and days. This will bias your results by sampling a larger percentage of seniors. Seniors tend to have lower approval values for marriage equality.
- Conduct the survey using only land lines. Including cell phones will give more accurate values because they include more young adults who tend to have greater approval of marriage equality.
For more information, see our essay: "How public opinion polls lie, either intentionally or by accident."
A consolidated graph by Pew Research:
This graph (repeated here from the previous essay) shows results from their marriage equality surveys, 2001 to 2016:
The following information sources were used to prepare and update the above
essay. The hyperlinks are not necessarily still active today.
Andrew Mercer, "5 key things to know about the margin of error in election polls," Pew Research Center, 2016-SEP-08, at: http://www.pewresearch.org/
"How many adults live in the USA?," IAC Publishing, at: https://www.reference.com/ (The 2013 census reported about 242 million)
"margin of error Calculator," American Research Group, Inc, 2015, at: http://www.americanresearchgroup.com/
How you may have arrived here:
Copyright © 2016 by Ontario Consultants on
Original posting: 2016-DEC-08
Latest update : 2016-DEC-09
Author: B.A. Robinson