Will Digital Era Mean End of History?

July 9, 2001 — -- Every October, a centuries-old Buddhist sect in Kyoto, Japan, prays for our culture — hoping in a special "information service" that today's history and records will be open to future generations, and not fade away like the ghosts of so many failed dot-coms.

"Not too long ago before the computer was invented, people recorded important matters by committing them to paper," reads the Web site of the Daioh Temple of Rinzai Zen Buddhism. "As we enter the 21st century, however, most of the knowledge gained presently is through the use of electronic-aided devices, whether they be televisions, radios or PCs."

The transformation presents issues not only on a spiritual level, but also on a cultural one for archivists and historians, who worry that historically important information will be lost to digital deterioration, changing computer formats, carelessness or willful destruction.

"If the last century was built on the exploitation of oil … this next century is all about intellectual property and knowledge, and right now we are mishandling the most important resource the world has," says Brewster Kahle, founder of the Internet Archive, a nonprofit group trying to preserve a record of the Internet by periodically scanning and storing as much as possible.

More in This Series:

Why Bits Can Byte the Dust

Archivists Scramble to Save Digital Era

Preservation Can Be Message to Future

Virtual Museum Remembers Dead Web Sites

Although there is considerable disagreement over the extent to which the historical record is in jeopardy — with many feeling it is quite secure — archivists and preservation officials agree there has been a change in the way we are recording history. And, they say, old ways of preserving that record need to be rethought.

Information on paper could be stored in a static archive and checked occasionally for signs of decay, but digital information requires frequent conversion from old storage media, like computer disks and magnetic tapes, to newer ones.

"If these materials are not actively maintained, we will lose them," Kahle says. "If we drop the ball for 10 or 15 years, which is a heartbeat in the digital field, we may never be able to recover the information again in a cost-effective way."

Archivists note it is common for early examples of developing mediums, such as film, television and publishing, not to survive, but they are eager to preserve a more representative record this time.

Questions About Data Storage

In the early days of computers, some users assumed data would be safe for decades. But though documented examples of historically significant digital data loss are rare, archivists claim some early government tapes, or the records on them, already have deteriorated, sometimes making them unreadable.

There are implications for home computer users, too. Although no independent aging test results are commonly available for CDs, archivists claim common media in use now — such as CDs and floppy disks — have proven unreliable and uneven in quality. They say that while some hold up, others can suddenly become unreadable after just a matter of years, rather than decades or centuries, which manufacturers often claim their own laboratory testing shows.

Katherine Cochrane, president of the CD-Info Co. Inc., a consultant company and publisher of CD technology, says home users without sophisticated testing equipment might not know in time if their disks are damaged, because error-correction functions of devices such as home music CD players may mask data errors until it is too late.

Ken Thibodeau, director of the electronic records archives program at America's National Archives, where government data is stored, says despite the lack of public and independent test results to document it, the decay is real. He says the Archives have received tapes, CDs and other media from government agencies that were poorly stored or manufactured, and had significant deterioration.

"We didn't lose any data, but we had to do a lot of work to make the copies," Thibodeau says. "By having two copies at two different locations, we were able to recover the media."

Adding to the uncertainty, data storage industry watchers say, is the rapid pace of evolving technology.

"It's very hard to keep up with software and hardware," says Patsy Baudoin, e-journal archiving project planner at the Massachusetts Institute of Technology libraries. "Each one of our computers is some sort of island that's specifically configured for our needs.

"My own [doctoral] dissertation, I can't read" without paying a fee for a private company to extract it, she adds. "It's on large floppies. … I could try and track down the software, but it wouldn't be easy."

To combat that problem, archivists now seek to preserve data in standard, simple formats such as ASCII and HTML, rather than common but proprietary word processing programs and other software that are more likely to become inaccessible.

Archivists say they now recognize the problems of physical decay and obsolescence and feel they can manage for them by moving the data to fresh storage materials periodically. They say it also helps that the cost of hard drives and other storage devices are falling rapidly, making regular digital duplication on that relatively stable format less expensive.

Laws vs. Cultural Heritage

But they may have less of a prayer controlling what the Daioh Temple Buddhists say is thoughtless discarding or erasure of some digital documents and software — particularly because some archivists feel digitization and the law are costing them control over archivable information.

"We've got laws and policies working against our cultural heritage," says Kahle, the Internet archivist. "In this world where the copyright term lasts more than 100 years, which is far longer than the durability of electronic media — floppies, disk drives — how do libraries perform their traditional roles of preservation and access?"

In the past, Kahle and others argue, far-flung libraries could save copies of published material for posterity, and give the historical record a better chance of living into the future. But now, they say, publishers sometimes assert their copyrights, licensing access to their collections online but forbidding local storage of electronic matter. Some say that leaves the publisher, and nobody else, responsible for saving the historical record of their publications.

"If a company gives an indication that they don't want something archived, we won't archive it," Kahle says of the efforts of his organization, which attempts to periodically create a snapshot of the Internet. "For instance, The New York Times, the newspaper of record, refuses to be recorded."

ABCNEWS.com also places some material, including some graphics and interactive quizzes, off-limits to the "robots" Kahle's Internet Archive uses to scan the Internet.

In addition, searchable databases such as Lexis-Nexis, which save articles in digital form, often only maintain the textual aspects of a publication — without advertisements, photos or illustrations. And according to a recent Supreme Court decision, those databases are now forbidden in some cases from maintaining copies of freelance articles without paying the writers an additional fee.

The result, publishers say, is that the articles simply will be deleted from the databases and the publication's Web site, perhaps leaving a gap in the historical record.

Archivists say it is useful to have data saved in original form, because seemingly disposable aspects, such as advertisements in old newspapers, can often shed historical light on the time of publication.

The Urge to Purge

Liability concerns also can frustrate digital data preservation, Kahle says. If a person or company is sued, the thinking goes, he or it might be ordered to turn over all records, no matter how innocuous, and would then have to pay an expensive lawyer tens of thousands of dollars to review them.

"This encourages companies and people to purge their archives," Kahle says. "If you had to pay some lawyer that gets paid hundreds of dollars an hour just to go through something to find that it's not relevant, those are real costs."

The co-founder of a company that creates e-mail systems designed to keep messages off the record agrees that potentially high legal costs encourage companies to obscure casual e-mail correspondence. But he says companies always have had certain power to decide what remains private, so little new information is being lost.

"These policies by and large mirror what's happening with paper. People don't save every piece of paper," says Jeff Ubois of Omniva Policy Systems, formerly known as Disappearing Inc. "Do people who want to have confidential communications have the right to have those communications? Traditionally, the Western legal tradition says: 'Yes, they do.'"