TNSI and its predecessor, Chilton Research Services, have been ABC News' primary fieldwork provider since 1979. ABC News closely oversees TNS sampling and fieldwork procedures.

Sampling

A sample of telephone households in the continental United States is selected via random digit dialing (RDD) procedures, in which all telephone numbers, listed and unlisted, have an equal probability of selection.

RDD samples are produced by Survey Sampling International from a sampling frame that includes all active telephone area codes and exchanges. SSI first purchases a database of all listed telephone numbers. Active blocks — contiguous groups of 100 phone numbers for which more than one residential number is listed — are added to the database, and listed business numbers are removed. The listed database is updated on a four- to six-week rolling basis, 25 percent of listings at a time. The file of business numbers is updated quarterly.

Each telephone exchange is assigned to the county where it's most prevalent. In the first stage of selection, the database is sorted by state and county, and the number of telephone numbers to be sampled within each county is determined using systematic sampling procedures from a random start, such that each county is assigned a sample size proportional to its share of possible numbers. In the second stage of selection, telephone numbers are sorted within county by area code, exchange and active block, and using systematic sampling procedures from a random start, individual phone numbers within each county are selected. The sampled phone numbers are pre-dialed via a non-ringing auto-dialer to reduce dialing of non-working numbers.

The last stage of sampling is respondent selection within the household: Interviewers ask to speak to the household member age 18 or over at home who's had the last birthday. To compensate for the fact that women tend to be easier to reach, in-house selection is stratified by sex, with interviewers asking to speak with the male household member 75 percent of the time and the female 25 percent of the time. If a person of the selected sex is unavailable, the interviewer asks to speak with the person of the other sex who had the last birthday.

Interviewing

Phone numbers are released for interviewing in replicates by region to allow for sample control. In multi-night polls, numbers are called multiple times during the field period. Interviews are conducted via computer-assisted telephone interviewing (CATI). The professional interviewers employed by TNS, and their supervisors, are extensively trained in interviewing practices, including techniques designed to achieve the highest possible respondent cooperation.

Weighting

Final data are weighted using demographic information from the Census to adjust for sampling and nonsampling deviations from population values. Respondents are classified into one of 48 cells based on age, race, sex and education (32 cells are used for smaller-sample surveys). Weights are assigned so the proportion in each of these 48 cells matches the actual population proportion according to the Census Bureau's most recent Current Population Survey.

Sampling Error

Poll results may deviate from full population values because a sample, not a census, is interviewed. Sampling error can be calculated when probability sampling methods, such as those described here, are employed. The standard formula to calculate sampling error at the 95 percent confidence level is: (SQRT(.25/sample size))*1.96. There can be other sources of differences in polls, such as the wording and order of questions.

Pre-election Polls

Pre-election polling presents particular challenges. As Election Day approaches, these polls are most relevant and accurate if conducted among voters. Yet actual voters are an unknown population — one that exists only on (or, with absentees, shortly before) Election Day. Pre-election polls make their best estimate of this population. Our practice at ABC News is to develop a range of "likely voter" models, employing elements such as self-reported voter registration, intention to vote, attention to the race, past voting, age, respondents' knowledge of their polling places and political party identification. We evaluate the level of voter turnout produced by these models and diagnose differences across models when they occur.

The use of political party identification in likely voter models is a subject of debate among opinion researchers. It's used commonly by campaign pollsters, less so among academic researchers. After an extensive evaluation of the issue, ABC News began employing party ID as a factor in our likely voter modeling for our tracking poll in 2000, and we continue the practice in our 2004 tracking poll. (A tracking poll is a series of consecutive, one-night, stand-alone polls reported in a rolling multi-night average. Ours is conducted among 600 general population respondents per night, using a nightly mix of fresh and redialed random telephone numbers.)

We made a detailed presentation on our 2000 tracking poll, including an examination of the effects of party ID as a factor in modeling, at the 2001 annual meeting of the American Association for Public Opinion Research (AAPOR). It showed that party ID factoring in 2000 had essentially no effect on our estimate of vote preferences — no more than a single point on any given day.

Proponents of using party ID in likely voter modeling point out that party ID has been remarkably stable in exit polls conducted in presidential elections since 1984 — Democrats accounting for either 38 percent or 39 percent of voters, Republicans 35 percent and independents 26 percent or 27 percent. (That stability is impressive given the differing vote margins in these elections — Rep +18, Rep +8, Dem +6, Dem +9, tie.) Opponents of the practice note that party ID can and does change, and that polls measuring the dynamics of the race — rather than simply attempting to predict its outcome — need to measure and report this change, not suppress it.

Our practice is informed by the fact that, in all our polling, we see night-to-night variability in party ID that appears to represent trendless sampling variability rather than actual changes in partisan self-identification. It also appears to us that some likely voter models (although not ones that we use) may accentuate this short-term variability in party ID. This affects portrayals of the race, given the very high correlation between party ID and vote preference. Rather than reporting actual changes in opinion, these surveys instead may be reporting who's moving into and out of their likely voter models. That's meaningful if it represents true movement of potential voters into and out of the pool of presumed actual voters, but not if it only represents an artifact of the likely voter model itself. Claims that this movement is meaningful seem to be contradicted by its trendless variability and by the remarkable consistency in party ID in actual turnout in the last five presidential elections.

We do not use party ID as a factor in our pre-election polls before tracking begins. These polls, done well in advance of Election Day, are not predictive, and do not seek to model actual turnout. The shifts in allegiance they record often appear as consistent, multi-night, event-based changes, rather than trendless, night-to-night variability. We noted and reported, for example, shifts in party ID around the 2004 conventions — more Democratic self-identification after the Democratic National Convention, more Republican self-identification after that party's convention.

Tracking polls, done in the final weeks of the campaign, are seen as more predictive. They need to sharpen their best estimate of actual likely voters, and not let the accuracy of their portrayal of the race fall victim to sampling variability or model-induced fluctuations.

Keeping in mind that actual change can occur, but also that random movement can distort, our solution is to compute an average of party ID as measured in our nightly tracking poll, and party ID as measured in recent presidential elections. This averaging approach allows us to pick up real movement in party ID while constraining random variability. It reflects our conclusion that, on one hand, the stability in party ID in the last five elections is persuasive, but not necessarily fully predictive; and, on the other, that some variability in party ID in tracking polls may be real, but that it also can reflect sampling or modeling variability, rather than true movement in voter attitudes.

Some critics of using party ID from exit polls in likely voter modeling point out that it's the equivalent of weighting a poll to a poll, which increases sampling error. It still, however, may improve the estimate. Exit polls are based on much larger samples than tracking polls — at least 13,000 voters in each election since 1992 — with correspondingly low margins of sampling error, less than one percentage point. Exit polls also are based on samples of actual voters, rather than likely voter estimates. And they're post-stratified to actual vote, which is highly correlated with party ID. All these increase the reliability of exit poll estimates. Opponents of using party ID in modeling also note that it introduces judgment into the process. However, judgment is required across all components of likely voter modeling — what elements to include, how to compute them, what turnout to anticipate.

While our modeling is intended to produce the best possible estimate, we reject the myth of pinpoint accuracy in pre-election polls. A good final poll, rigorously conducted and with accurate modeling, should come within a few points of each candidate's actual support. Any more indicates a problem, but any closer is the luck of the draw. Winning the horse-race lottery is not sufficient grounds for a substantive evaluation of the quality of any pre-election poll.

Indeed, while good polling produces the best available estimate of the candidates' standing at any point in time, that is not the sole or even the main reason ABC News engages in pre-election polling. We conduct these surveys as part of our effort to cover the election fully and well, by independently measuring the concerns and interests of likely voters and voter groups, and reporting how these inform their decisions in the deliberative process under way.

Response Rates

Response rates are a complex issue. Rates are computed for "contact," that is, the number of households reached out of total telephone numbers dialed (excluding an estimate of nonworking and business numbers); and "cooperation," the number of individuals who complete interviews out of total households reached. The two together produce the "response rate." There is no single, agreed-upon means of calculating response rates (including, for example, how to estimate nonworking and business numbers).

Even given a probability sample, it cannot be assumed that a higher response rate ensures greater data integrity. Research over many years, including a variety of studies reported at the annual meeting of AAPOR in May 2003, has found no significant biases as a result of response rate differences. As far back as 1981, in "Questions & Answers in Attitude Surveys," Professor Howard Schuman of the University of Michigan, describing two samples with different response rates but similar results, reported, "Apparently the answers and associations we investigate are largely unrelated to factors affecting these response rate differences." (p332.) For more details on response rate issues, click here.

In spring 2003 ABC News and the Washington Post produced detailed sample dispositions for five randomly selected ABC/Post surveys at the request of Professor Jon Krosnick, then of Ohio State University, and his associates for their use in a study of response rate differences. The cooperation rate calculations produced by Krosnick's team for these five surveys ranged from 43 to 62 percent, averaging 52 percent. The response rate calculations produced by Krosnick's team ranged from 25 to 32 percent based on what the AAPOR describes as a "very conservative" estimate of the number of business and nonworking numbers in the sample; it would be 31 to 42 percent based on a less conservative estimate reported in the June 2000 issue of Public Opinion Quarterly. The difference underscores one of the many factors that make the issue so complex, and response rate comparisons so tenuous.

Standards

The ABC News Polling Unit vets all survey research presented to ABC News to ensure it meets our standards for disclosure, validity, reliability, and unbiased content. We recommend that our news division not report research that fails to meet these standards.

On disclosure, in addition to the identities of the research sponsor and field work provider, we require a detailed statement of methodology, the full questionnaire and complete marginal data. If any of these are lacking, we recommend against reporting the results. Proprietary research is not exempted.

Methodologically, in all or nearly all cases we require a probability sample, with high levels of coverage of a credible sampling frame. Self-selected or so-called "convenience" samples, including Internet, e-mail, "blast fax," call-in, street intercept, and non-probability mail-in samples do not meet our standards for validity and reliability, and we recommend against reporting them.

We do accept some probability-sample surveys that do not meet our own methodological standards — in terms of within-household respondent selection, for example — but may recommend cautious use of such data, with qualifying language. We recommend against reporting others, such as pre-recorded autodialed surveys, even when a random-digit dialed telephone sample is employed.

In terms of content, we examine methodological statements for misleading or false claims, questionnaires for leading or biasing wording or ordering, and analyses and news releases for inaccurate or selective conclusions.

In addition to recommending against reporting surveys that do not meet these standards, we promote and strongly encourage the reporting of good-quality polls that break new ground in opinion research.