Sampling Error: What it Means

Oct. 8, 2008— -- Surveys based on a random sample of respondents are subject to sampling error – a calculation of how closely the results reflect the attitudes or characteristics of the full population that's been sampled. Since sampling error can be quantified, it's frequently reported along with survey results to underscore that those results are an estimate only.

Sampling error, however, is oversimplified when presented as a single number in reports that may include subgroups, poll-to-poll changes, lopsided margins and results measured on the difference. Sampling error in such cases cannot be described accurately in a brief television or radio story or on-screen graphic.

Sampling error assumes a probability sample – a random, representative sample of a full population in which all respondents have a known (and not zero) probability of selection. Given that prerequisite, sampling error is based largely on sample size, but also on the division of opinions or characteristics measured and on the level of confidence the surveyor seeks. A larger sample has a lower error margin. A result of 90-10 percent has a smaller error margin than a 50-50 result; when more people agree, there's less chance of error in the estimate. And a result computed at the 90 percent confidence level has a smaller error margin than a result computed at 95 percent confidence.

Assuming a 50-50 division in opinion calculated at a 95 percent confidence level, a sample of 1,000 adults – common in ABC News polls – has a margin of sampling error of plus or minus 3 percentage points. The error margin is higher for subgroups, since their sample size is smaller. Given customary subgroup sizes, for 800 whites the error margin would be plus or minus 3.5 points; for 560 women, +/- 4 points; for 280 Republicans, +/- 6 points. Click here for a list of examples using averages from recent ABC News polls.

At a 90-10 division of opinion, rather than 50-50, still at 95 percent confidence, sampling error for 1,000 interviews is +/- 2 points, not 3. For a sample of 100 cases – roughly the minimum sample size ABC News will report – the error margin is +/- 10 points at a 50-50 percent division, +/- 8.5 points at 75-25 percent and +/- 6 points at 90-10 percent. (ABC News polls at times will oversample small populations to increase their sample size to a level we consider reliably reportable.)