ABCNEWS Statement on Elections
— -- STATEMENT OF ABC NEWS
CONCERNING THE 2000 ELECTION PROJECTIONS
February 8, 2001
On November 7, 2000, the United States set out to elect a new President. Leading up to Election Day, all indications were that the race would be very close. As it turned out, it was the closest race in recent history (in terms of electoral votes) and was not resolved for over a month, when the Supreme Court of the United States ended the legal challenges to the vote count in the State of Florida.
During the evening hours of November 7 and the early morning hours of November 8, ABC News projected the winner of the presidential race in 49 states and the District of Columbia.¹ In each of these but one, ABC News' projections were correct. But in what turned out to be the key state of Florida, ABC News made two projections, one of them mistaken and the other premature.
In this statement, we discuss the practices and procedures followed by ABC News in making its election projections, the reasons for the flawed projections in Florida, and the steps we are taking to prevent a recurrence of the mistakes we made on Election Night. This statement follows a comprehensive review conducted by ABC News, ABC's in-house counsel, and ABC's outside counsel. Among other things, we: (1) reviewed transcripts and videotapes of ABC News' election coverage; (2) interviewed members of the ABC News team responsible for making election projections and producing ABC News' election coverage; (3) reviewed archival copies of computer screens containing some of the data and statistical models provided to ABC News on Election Day by Voter News Service ("VNS"); and (4) reviewed post-election reports by VNS and others analyzing the Election Night projections.
A. ABC News' Practices and Procedures in Projecting Election Results
1. Background
For many years, ABC News has included in its election coverage projections of likely winners in individual races. When done properly, such projections provide our viewers with highly reliable and timely insights into the election, including the likely outcome of races in the various states.
These projections go beyond mere reporting of actual vote tallies as they come in. They involve the interpretation of sophisticated statistical models that evaluate the exit poll and actual vote data in various ways. Although grounded in mathematics and science, projections of likely outcomes always depend in critical part on the informed judgments of knowledgeable analysts. To make these judgments, ABC News relies on teams of experts, including political scientists and statisticians, with the experience and acumen necessary to interpret the data properly.
2. The Role of the Voter News Service
In 2000, as in previous years, ABC News relied heavily on data and statistical modeling from VNS in making its election projections. VNS was established in 1994 through the merger of two predecessor organizations. The first, the News Election Service, was founded in 1964 and collected raw vote data. The second, Voter Research and Surveys, was formed in 1989 and did three things: (1) polled voters as they exited the polling place; (2) reported the exit poll results; and (3) used statistical models to help project race results based on the exit polls and on vote data. Since 1994, VNS has performed the functions of both of these predecessor organizations.
VNS was founded in 1994 by ABC, NBC, CBS, CNN, and the Associated Press; in 1997, Fox became a full member. The members collectively share the costs of operating VNS and govern it through a Board of Directors consisting of one voting representative from each member. VNS data and analysis are provided to all of the members equally. In addition, many other news organizations subscribe to VNS, paying fees that partially defray operating expenses in exchange for projections and some VNS data. By sharing the costs among the members and subscribers, VNS collectively provides a better service to all than any individual member or subscriber could afford on its own. As one of the owners of VNS, with a substantial role in its management and supervision, ABC News (with the other members) is ultimately responsible for the reliability and accuracy of VNS' product.
Each election, VNS collects survey responses provided to VNS personnel by voters as they leave polling places ("exit polls"), actual vote tallies from selected precincts, and the vote count from all reporting counties, cities, or towns nationwide. VNS loads this material into a central computer that feeds information to VNS members and subscribers. VNS also provides members with statistical analyses under various VNS models, which use different methods to extrapolate from the data received by VNS at that juncture. Some models include pre-election poll data; some include exit poll data; some models use geographical distinctions within a state; some reflect political distinctions within a state; others rely on actual vote tallies as reported by precincts or by counties. VNS models also estimate the remaining vote outstanding for precincts not yet reporting, and provide a check on the accuracy of exit poll data by comparing that data to actual vote tallies as they come in.
Before polls close and actual vote tallies become available, the VNS models rely upon exit poll data provided by VNS, together with various pre-election poll data. As actual vote tallies become available from precincts, these data are included in the models, replacing exit poll data. When county data is available, it is included in several other models.
Early in the afternoon of Election Day, VNS begins providing its members with preliminary results of the first exit polls taken. As the day progresses, VNS updates these results, including later exit poll data and actual vote tabulations. All of these data are provided to VNS members so that they may evaluate for themselves whether and when to project the results in a race. VNS also makes its own projections for the benefit of its members and subscribers.
Prior to the election of 2000, those most closely involved with VNS can remember only one instance in which a VNS projection ultimately proved wrong. In the 1996 New Hampshire Senate race, exit poll data available at the time the polls closed showed the Democratic candidate to have such a commanding lead that VNS members (including ABC News) projected the outcome of the race. When the actual vote tallies began to come in, it became apparent that the exit poll data were seriously flawed and the projections were retracted. After the 1996 error, VNS conducted its own study and retained outside experts to examine what happened and how to avoid similar mistakes in the future. Despite intensive study, the experts could find no definitive solution to the problem that could be applied across states and elections. In 2000, ABC News included as part of its team two of the experts most knowledgeable about the 1996 New Hampshire race, so that they could brief the other ABC News experts responsible for making projections.
3. ABC News' Decision Desk
On Election Day, ABC News employs a separate unit of professionals charged with deciding whether and when to make projections. This group is generally referred to collectively as the ABC News "Decision Desk." It is headed by an expert journalist who is experienced both in covering previous elections and in the statistical analysis of elections.
The Decision Desk on November 7 consisted in principal part of four decision teams. Each decision team included two individuals chosen for their experience in covering past elections and/or their background in statistics or political science. The 50 states and the District of Columbia were divided among three teams, each responsible for making projections of statewide elections (for President, for Senate, and for Governor) in certain states. The fourth team was assigned to follow certain races in the House of Representatives that ABC News had identified in advance as having particular importance in the election. In order for ABC News to make a projection in any race, both members of the decision team assigned to that race had to agree that the projection was justified.
The ABC News Decision Desk also included a team of two individuals reporting directly to the head of the Desk and responsible for monitoring the overall presidential race and the battle for control of the Senate. ABC News also retained two experts who were posted at VNS headquarters as a liaison between the ABC News Decision Desk and VNS.
The Decision Desk independently studies the data as they are reported by VNS, analyzes the results indicated by the VNS application of the statistical models to these data, and decides whether and when it is appropriate for ABC News to project an outcome in each race. Sometimes this projection comes after VNS has made its own projection; often it comes before VNS does so.
If the margin reflected in exit polls is sufficiently large to justify making a projection based on these data alone, ABC News may be able to project results at the time polls close. In determining whether it can make an accurate projection based on exit polls and other research at the time of poll closing, ABC News relies on the following considerations:
1. the extent to which all of the exit poll data from the several exit polls taken throughout the day are complete and available;
2. the size of the lead for a particular candidate indicated in the exit poll models;
3. a comparison of the margin indicated in the exit polls with a statistical calculation of the margin of error;
4. a comparison of the results generated by the different VNS statistical models;
5. prior estimates of the race in the particular state;
6. special factors that might affect the reliability of exit poll results, such as the size of past absentee balloting in the state and the distance restrictions placed on VNS personnel conducting local exit polls;
7. past experience in projecting results in the particular state or in states having similar characteristics; and
8. any special messages from VNS concerning problems or irregularities that may have arisen during the collection of exit poll data.
If exit poll results do not demonstrate a sufficiently decisive lead for one candidate after assessment of these factors, then ABC News considers the following factors as actual vote tallies arrive, in addition to exit poll data, pre-election estimates, and the patterns of prior elections:
1. the size of the lead and the margin of error indicated in the various models as actual votes from the precincts are substituted for exit poll results;
2. the size of the lead and the statistical margin of error indicated in the models analyzing actual vote tallies in selected precincts and in overall county results;
3. the degree to which the exit poll data for the state differed from actual vote tallies in precincts where exit poll data are available;
4. the number, percentage, and location of precincts from which actual vote tallies have been received;
5. the number, percentage and location of counties from which actual vote tallies have been received;
6. the likely outstanding absentee vote not yet counted;
7. the likely other vote not yet counted; and
8. the percentage of the remaining vote outstanding that the trailing candidate would have to garner in order to prevail.
In each case, the question is whether ABC News in its independent journalistic judgment can conclude that the data make it appropriate to project the outcome in a given race - that is, that the results are decisive enough to make a projection with great confidence that it will be right.
In addition to these steps, ABC News makes an effort to say explicitly on the air that it is reporting a projection, not an accomplished fact. And, since 1985, ABC News has followed the policy of not projecting any statewide race until the polls in the state have closed. In the few states with multiple poll closing times, ABC News has not projected the results of races until the substantial majority of the polls have closed.
B. ABC News' Election Projections in 2000
On the morning of November 8, ABC News initiated an investigation to determine the cause of its erroneous and premature projections in the Florida presidential race and the measures it should take to avoid similar problems in the future. In addition, VNS conducted its own internal investigation and - at the urging of ABC News and other members - commissioned a thorough review of its actions by outside experts. These investigations lead us to the following primary findings and conclusions about the problems in Florida and some areas for improvement.
1. The Flawed Florida Projections
a. ABC News' Projection for Vice President Gore
The first of ABC News' Florida presidential projections in the 2000 election came shortly before 8:00 p.m. EST on November 7. VNS projected that Mr. Gore would win the presidential race in Florida at 7:52 p.m., after some 90% of the polls in state had closed (but before polls in the panhandle of the state were to close at 8:00 p.m.). At 7:55, ABC News Radio reported that ABC News projected that Mr. Gore would win in Florida. At that time, the ABC Television Network was in commercial and local broadcast time. ABC News did not project Mr. Gore to win in Florida on the ABC Television Network until shortly after 8:00 p.m., after all polls in the state had closed.
A review of the computer data preserved by VNS that night² indicates that, as of 6:40 p.m., exit poll data and statistical models revealed a lead for Mr. Gore in Florida, but not by a sufficient margin to warrant a projection. As a result, ABC News did not project a winner in the Florida race for President when 90% of the Florida polls closed at 7:00 p.m.
By 7:40 p.m., VNS was reporting actual vote tallies from eight of the sample precincts. These actual vote tallies significantly reduced the margin of error for the projections under the various statistical models, and therefore indicated a greater confidence in the accuracy of the projections. Based on these indications, and in accord with the factors listed above, the ABC News Decision Desk projected that Mr. Gore would win in Florida shortly before 8:00 p.m. At 8:10 p.m. and 8:40 p.m., the numbers continued to show a solid lead for Mr. Gore in Florida. In fact, at that time, a comparison of the actual vote tallies with the exit poll data in those eight precincts suggested that the VNS exit poll models had actually understated the extent of Mr. Gore's lead in the state.
At 9:10 p.m., the statistical models continued to indicate sufficient strength to justify the projection for Mr. Gore - even with 36% of the county vote tallies reported. But the precinct models showed some weakening. At about 9:40 p.m., VNS began to send messages to its members that called into question the accuracy of the Florida projection. And by 10:10 p.m., the models incorporating real vote tallies were mixed, with some continuing to project Mr. Gore the winner with sufficient confidence to warrant the projection, but the county vote model pointing to Mr. Bush, although without sufficient certainty of accuracy to warrant a projection in his favor. At approximately 10:00 p.m., ABC News withdrew its projection that Mr. Gore would prevail in Florida.
Based on subsequent reviews, it is now apparent that there were three principal causes of the flawed projection for Mr. Gore in Florida. First, the exit poll data actually overstated - rather than understated - Mr. Gore's lead. Although the risk of such error normally is checked by the comparison of the exit poll data in the first several precincts with actual vote tallies in these precincts, in this particular case the first precincts reporting actual vote were atypical of the larger sample.
Second, the first Florida projection was based in part on a mistaken estimation of the absentee vote in Florida. Although VNS anticipated a significant absentee vote in the state and expected that the absentee vote would favor Mr. Bush, it turned out that there was a much larger vote than anticipated, and that vote was slightly more favorable to Governor Bush than predicted.
Finally, the projection for Mr. Gore resulted in part from the VNS system failing to choose the most appropriate past race to use in its models. It chose the 1998 Governor's race, rather than the 1996 presidential race. As it turned out, the latter was more similar in pattern and result than the former. Absent any of these three VNS errors - the mistaken check on the exit poll data, the mistaken estimation of the absentee vote, and the choice of the wrong past race to use in its models - ABC News likely would not have found the data sufficient to make the first projection in the Florida presidential race.
b. ABC News' Projection for Governor Bush
From 10:10 p.m. until 1:40 a.m. EST, the data and statistical models continued to be mixed, with all but one model (the one that included actual vote count from the counties) indicating that Mr. Gore would prevail. By 2:10 a.m., however, the data provided by VNS indicated that a full 96% of all precincts had reported and that, given the projections of the remaining outstanding vote, Mr. Gore would have had to win over 63% of the remaining vote to prevail. Based on these data and these estimates, ABC News projected at 2:20 a.m. that Mr. Bush would prevail in the Florida presidential race.
It now appears that two primary factors precipitated the second ABC News projection for President in Florida. First, raw vote data coming into VNS from Volusia County significantly overstated Mr. Bush's totals and significantly understated Mr. Gore's totals. Normally such variations in a single county would not be significant, but with the race in Florida as close as it turned out to be, this variation alone led ABC News to have more confidence in its projection than was warranted.
Second, the VNS model projected significantly fewer outstanding votes at 2:10 a.m. than in fact was the case, leading the VNS model to underestimate the outstanding vote and thereby to overestimate the percentage of the vote that Mr. Gore would have to receive to prevail. Once again, these mistakes in the data and the models led ABC News to make a flawed projection.
2. Projections in Other States
As noted above, ABC News made 49 other projections of state races for President on November 7-8, and each one was correct. Overall, leaving Florida aside, ABC News correctly projected that Mr. Bush would prevail in 29 states and that Mr. Gore would prevail in 19 states and the District of Columbia. Of these, ABC News projected the winner at the time of poll closing in 28 states, including 16 states for Mr. Bush and 11 states and the District of Columbia for Mr. Gore. Of the 21 states in which ABC News waited to project a presidential winner until after polls had closed (other than Florida), ABC News projected 13 for Mr. Bush and 8 for Mr. Gore.
Some have questioned not the accuracy, but the timing of some of our projections. In particular, they have questioned the delay in the projection of some of the states won by Mr. Bush. Based on our review and the analyses described above, the timing of each of these projections is fully explainable by the data available at various times of the night, the application of uniform statistical models to those data, and prior experience with the states involved. There is no basis whatsoever for concluding that there was any intentional bias on the part of anyone who took part in the projection process at ABC News.
In determining whether and when to make a projection, there are a variety of factors ABC News considers, as set forth above. Significantly, however, the ultimate actual margin of victory in a state in no way indicates the speed with which a projection could have been made with a sufficient assurance of accuracy. Rather, the question is whether at any given time the margin shown by the exit poll (and, when available, actual vote tallies) is sufficiently larger than the statistical margin of error that ABC News can make a projection with a high level of confidence that it will be accurate. This statistical margin of error can be affected by a number of things, including the number of precincts sampled, the relative homogeneity of the state in question, and the percentage of actual vote available at the time of a projection.
This last factor is extremely important, as the ability of VNS and ABC News to assess the accuracy of close exit poll data and the certainty of a projection often depends on how quickly a given state reports actual vote data. In some states, precincts and counties report vote totals relatively quickly, and may facilitate an earlier projection; in others, vote totals arrive more slowly, and may delay the projection of the race.
A careful review of ABC News projections in states other than Florida shows that in each case - whether for Mr. Gore or for Mr. Bush - the projection was not only accurate, but was made at a reasonable time, given the available data and the need for great confidence in its accuracy.
Some have also raised concerns that, before all polls have closed in a state with multiple poll closing times, (like Florida) projections of race could influence voting behavior in other parts of the state. There have also been claims that projecting winners in presidential races in states where polls have closed may affect voting behavior in other states where polls remain open. These questions have been researched extensively for many years, with no clear answer.
C. Areas for Improvement
Based on our experience in making the Florida projections, our post-election review of ABC News' practices and procedures in general, and our review of other reports about the events of Election Night, we have identified the following primary areas that warrant further attention and improvement.
1. Improve the Accuracy and Reliability of VNS' Data and Statistical Analyses
VNS has a long and enviable track record of accuracy in projecting races over the years. In the November election, however, VNS fell short.
In the days since the election, VNS' performance has been analyzed extensively by internal and external investigations. Most recently, the VNS Board of Directors commissioned and released a detailed report by an independent consultant, the Research Triangle Institute ("RTI") of North Carolina, containing comprehensive findings and recommendations.
On the basis of the RTI report and our own review, we believe VNS must do a better job in future elections in the following principal areas:
First, it must account more accurately for the size and likely outcome of absentee ballots. In 2000, VNS made special efforts to account for absentee ballots in Washington, Oregon, and California by conducting telephone polling targeted at these ballots. In Florida, however (as noted above), it relied upon a rough estimate of the size and the outcome of the ballots. In the future, one or both of two things must happen: (1) VNS should make similar, targeted efforts in all states where there is reason to believe that absentee balloting may affect the accuracy of projections; and/or (2) ABC News will need to take into account the increased risk of inaccuracy from VNS' not having made such efforts and, as a result, be more conservative in its projections in those states.
Second, VNS must do a better job of quality control. For example, in 2000, there was a plan to include in the VNS computer models process data from the separate Associated Press reporting of actual vote tallies. This would have provided a check on the accuracy of VNS' data. ABC News believed on Election Night that this had been done; as it turned out, it was not. In the future, the AP (or other, similar sources of data) should be included in election projections -- either by VNS itself or by ABC News if VNS is unable to include such data.
Third, VNS must make adjustments in its statistical models to ensure that the best past election in a state is selected for comparison with exit poll and raw vote data. VNS itself has suggested that one solution may be to consult more than a single past election.
Fourth, VNS must devote further study to the causes of discrepancies between exit poll data and actual vote returns within the same precinct. In any given race, such discrepancies can favor either candidate, but on average over time they have been shown to favor Democratic candidates somewhat more than Republican. Experts believe these discrepancies may result in part from the refusal of some people to respond to exit poll surveys as they leave the polls. However, studies have not shown a clear direct relationship between overall response rate and exit poll accuracy. Moreover, the size of the discrepancy and whether it favors the Republican or the Democrat is highly variable from precinct to precinct, state to state, and year to year, and experts have yet to develop an overall statistical solution. Nevertheless, we must work to develop reliable ways either to reduce the causes of the discrepancies or to compensate for them.
Fifth, VNS must improve its system to provide better correction on inaccuracies in exit poll data. As discussed above, it was previously thought that the comparison of actual vote totals in six selected precincts against exit poll results in those same precincts would give some reasonably accurate indication of the reliability of the exit poll data in all the precincts. The experience in 2000 with the exit poll data in Florida demonstrates that this is not the case.
Finally, VNS must do a better job estimating the outstanding vote. This is a crucial piece of information in any close election.
Based on its review, ABC News at this point believes that with improvements such as those outlined above VNS can remain a highly reliable means for analyzing and reporting election results. Any system, however, no matter how sophisticated and how reliable, is inherently fallible. ABC News will remain open, therefore, to any reasonable alternative sources of information that can help to improve the accuracy of its election projections.
2. Further Insulate ABC News' Decision Deskfrom Competitive Pressure
In the past, and in 2000, the ABC News Decision Desk has been located separately from the remainder of ABC News editorial operations on Election Night. Communications between the Decision Desk and those responsible for ABC News election coverage have been structured through a single senior producer located in the control room.
Until now, however, ABC News has not sought to restrict access of members of its Decision Desk to the reporting of other news organizations, including competing television news organizations. It was thought that the knowledge of what other credible news organizations were and were not projecting could be helpful to the Desk in determining when it was appropriate to make a projection.
Competition, in news reporting as in other enterprises, can be a good thing. It can spur us to work harder, do better, be faster. But competition that encourages a journalist to report a story prematurely is bad. In the particular instance of the Decision Desk, it is most important that the individuals making projections do so based on two things: the data provided to them (from exit polls, from actual vote tallies, and from statistical models), and their own experience and judgment. They should not be distracted or influenced by the decisions of other news organizations.
3. Improve the Manner in which ABC News Reports and Discusses its Projections
Although valid statistical methods applied to raw vote tallies and exit poll data are the best and most accurate means of projecting election outcomes, they also pose risks if not properly used and explained. First and foremost, they are projections - not actual results of elections. As such, there is always some margin for error, some chance that they will ultimately be proved wrong.
ABC News for some time has attempted to describe its predictions of the likely winners in election contests as "projections." We try to avoid saying on air that we are "calling" a race with the implication that the election is truly over simply because we feel sufficiently confident in the statistics to make a prediction. This policy was generally followed on November 7-8 in the initial announcements of projections in the presidential races in individual states.
In reviewing the transcript of ABC News' Election Night coverage, however, it appears that ABC News did not make sufficiently clear to our audience the nature of projections in election races in several respects. First, after having initially described the prediction as a "projection," ABC News journalists on the air sometimes later referred to the candidate as having "won" the race in a particular state. Second, we were not always as careful in our use of language in making projections in races for the House of Representatives, the Senate, and Governors. Third, ABC News did not explain adequately to our viewers what a projection means. We did not make it clear that, as with any statistical projection, there is a margin of error. We would not be reporting the projection unless, according to our analysis, the margin of error is sufficiently small. But we need to do a better job of pointing out that our projection can be wrong.
D. Conclusion
In light of our review, ABC News will implement the following changes and clarifications to its practices and procedures in making election projections. Many of these measures were announced on November 22, 2000.
1. ABC News will project the winner in a race in a given state only after the last scheduled poll closing time in that state.
2. ABC News continues to support a uniform national poll closing time. We also support efforts to reform balloting processes to enable faster and more reliable official tabulation and reporting of vote totals.
3. ABC News will continue to make projections only if they are justified by ABC News' independent analysis of the data and the relevant statistical models.
4. In making and discussing projections, ABC News will explain to its viewers that they are informed, statistically based estimates of the probable results of elections. Projections are not reports of the actual, final results of elections.
5. ABC News will take all reasonable steps to insulate those involved directly in making projections from the pressures of competition from other news organizations.
6. ABC News will ensure that voting data from the Associated Press is fully incorporated into its projections, providing a check on inaccurate information. ABC News will remain open to additional sources of information on election night, including national exit polling conducted by organizations other than VNS.
7. ABC News will support a continuing comprehensive review of and improvement in the operation of VNS, including improvements in the collection of data, reporting of data, and application of statistical models to those data. Further, ABC News will provide its share of resources to ensure that these improvements and upgrades are made as quickly as possible.
¹ ABC News never projected a winner in the presidential race in Oregon, which was resolved several days later, after the last of the absentee ballots was counted. In addition to the projections in the race for President, ABC News made 45 projections in races for the Senate and for Governor, all of which were correct. ABC News never projected a winner in the key Senate race in Washington, which was ultimately resolved several days later as well.
² VNS automatically archived the primary data (or "decision screens") it sent to members at only 10 and 40 minutes past each hour on Election Day. Minute-by-minute screens reflecting then-available VNS data were not retained.