Sampling Error: What it Means

Surveys based on a random sample of respondents are subject to sampling error – a calculation of how closely the results reflect the attitudes or characteristics of the full population that's been sampled. Since sampling error can be quantified, it's frequently reported along with survey results to underscore that those results are an estimate only.

Sampling error, however, is oversimplified when presented as a single number in reports that may include subgroups, poll-to-poll changes, lopsided margins and results measured on the difference. Sampling error in such cases cannot be described accurately in a brief television or radio story or on-screen graphic.

Sampling error assumes a probability sample – a random, representative sample of a full population in which all respondents have a known (and not zero) probability of selection. Given that prerequisite, sampling error is based largely on sample size, but also on the division of opinions or characteristics measured and on the level of confidence the surveyor seeks. A larger sample has a lower error margin. A result of 90-10 percent has a smaller error margin than a 50-50 result; when more people agree, there's less chance of error in the estimate. And a result computed at the 90 percent confidence level has a smaller error margin than a result computed at 95 percent confidence.

Assuming a 50-50 division in opinion calculated at a 95 percent confidence level, a sample of 1,000 adults – common in ABC News polls – has a margin of sampling error of plus or minus 3 percentage points. The error margin is higher for subgroups, since their sample size is smaller. Given customary subgroup sizes, for 800 whites the error margin would be plus or minus 3.5 points; for 560 women, +/- 4 points; for 280 Republicans, +/- 6 points. Click here for a list of examples using averages from recent ABC News polls.

At a 90-10 division of opinion, rather than 50-50, still at 95 percent confidence, sampling error for 1,000 interviews is +/- 2 points, not 3. For a sample of 100 cases – roughly the minimum sample size ABC News will report – the error margin is +/- 10 points at a 50-50 percent division, +/- 8.5 points at 75-25 percent and +/- 6 points at 90-10 percent. (ABC News polls at times will oversample small populations to increase their sample size to a level we consider reliably reportable.)

As noted, the confidence level is the third chief variable in sampling error. (There are other factors in some surveys, such as design effects – see the addition to the end of this piece - and finite-population adjustments, which we'll leave aside here.) The 3-point error margin at 95 percent confidence for a sample of 1,000 declines to a +/- 2.5 points at 90 percent confidence and +/- 2 points at 80 percent confidence.

(Some organizations round sampling error to whole numbers; others report them to the decimal. ABC's practice is to round them to the half. That acknowledges the differences caused by sample size – 800 and 1,500 both round to +/-3; better to show the former as 3.5 and the latter as 2.5 – without suggesting the level of precision in the data implied by entirely unrounded decimals, e.g. +/-3.3.)

The calculations above are based on a single sample, using a standard formula – multiply the division in opinion (e.g., .50 times .50 for a 50/50 split), divide the result by the sample size, take the square root and multiply by the so-called "critical value"– for 95 percent confidence, 1.96.

Page
  • 1
  • |
  • 2
  • |
  • 3
Join the Discussion
You are using an outdated version of Internet Explorer. Please click here to upgrade your browser in order to comment.
blog comments powered by Disqus
 
You Might Also Like...
See It, Share It
PHOTO: The first explosion knocked down a runner at the finish line of the 117th Boston Marathon.
John Tlumacki/The Boston Globe via Getty Images
PHOTO: The scene near the finish line of the Boston Marathon is seen in this April 16, 2013 file photo. Inset, suspects Tamerlan and Dzhokhar Tsarnaev are seen. Tamerlan Tsarnaev died after a gunfight with police and Dzhokhar Tsarnaev, was captured.
Elise Amendola/AP Photo; Inset: Lowell Sun, FBI/AP Photo
PHOTO: Pulaski Township Police Sgt. Chad Adam seen here in this undated Facebook photo, went undercover as an Amish woman.
Pulaski Township Police Department/Facebook