WASHINGTON -- For 40 years, U.S. presidents have begun each day with a top-secret, personal briefing on security threats and global affairs obtained largely from covert spy missions, clandestine satellite surveillance and other highly classified intelligence sources.
Now, however, the President's Daily Brief and other crucial intelligence reports often rely less on secrets from risky espionage missions than on material that's available to just about anyone.
Intelligence officers have gleaned insights on Iran's nuclear capabilities from photos on the Internet. They've scooped up documents, including a terrorist training manual, at international conferences and public forums. They've found information in foreign university libraries and newscasts.
Such material is known as "open-source intelligence" or, in the acronym-laden parlance of the 16 federal agencies that make up the U.S. intelligence community, OSINT. The explosion of information available via the Internet and other public sources has pushed the collection and analysis of that material to the top of the official priority list in the spy world, intelligence officials say.
The change hasn't been easy in a bureaucracy that often measures success by its ability to steal secrets. Federal commissions repeatedly have criticized the intelligence community for not moving more quickly and aggressively to exploit open-source information.
It's a challenging task, given the mountains of material to sift through. Every potentially useful nugget must be vetted because enemy states and terror groups, such as al-Qaeda, sometimes use the Internet and other open channels to put out misleading information.
Yet officials say agencies are overcoming such obstacles and unearthing increasingly valuable troves of intelligence.
"It's no longer unusual to see open-source material in the President's Daily Brief … (and) it's often a very important component of the information that's incorporated into our intelligence analyses," says Frances Townsend, who until January was President Bush's assistant national security adviser for homeland security and counterterrorism.
Whether it's developments in Russian politics, the spread of avian flu, the rise of Islamic fundamentalism in Asia or the technological capacity of enemy states, "there's been a significant shift toward relying more on open-source information," Townsend adds. And "a lot of what we know about our (terrorist) adversaries comes from statements and videos they put on the Internet."
The intelligence community is investing heavily to improve its collection of open-source information.
The CIA has set up an Open Source Center, based in a nondescript office building in suburban Washington, where officers pore over everything from al-Qaeda-backed websites to papers distributed at science and technology symposiums, says Douglas Naquin, the center's director.
Other agencies, such as the FBI and the Defense Intelligence Agency, are training scores of analysts to mine open sources and giving many of them desktop Internet access. That's a big change in a world in which computers in such agencies have long been designed to prevent data from flowing to or from the public realm.
At the same time, national security officials also are grappling with the flip side of the open-source phenomenon: making sure sensitive information held by the government, businesses and even individuals doesn't slip into the same sort of public outlets that U.S. intelligence agencies are scrutinizing.
Intelligence officials see it all as a necessary evolution.
Open sources can provide up to 90% of the information needed to meet most U.S. intelligence needs, Deputy Director of National Intelligence Thomas Fingar said in a recent speech. Harnessing that information "is terribly important," he said. "It ought to be a normal part of what we do, not being fixated on secrets dribbling into the computer's in-box."
But progress has been slow.
Robert David Steele, a former CIA and Marine Corps intelligence officer, gives the intelligence community a D+ for its use of information available from the Internet, commercial satellite imagery and other open sources.
"There's still a cult of secrecy — nothing is seen as important unless it's classified," says Steele, founder of OSS.Net, a commercial intelligence provider for private companies and the government.
Agencies still aren't investing enough in training and technology to use open sources, he says, so analysts lack language and computer skills, and many use outdated hardware and software that make searches slow or cumbersome.
"There are lots of problems," Steele says. "It's hardware, it's software, it's mind-set, but most of all, it's lack of leadership vision."
Insight into Iranian nukes
Perhaps the greatest evidence yet of the intelligence community's new embrace of open-source information emerged in December, when top U.S. officials noted that photos available through public media factored into a new assessment by U.S. intelligence analysts that Iran had suspended efforts to build a nuclear weapon in 2003.
The new National Intelligence Estimate on Iran's nuclear program reversed previous analyses, which had said Iran had an ongoing nuclear arms program.
The shift was based partly on public photos from Iran's Natanz nuclear facility, including pictures from a media tour and United Nations inspections, according to the Office of the Director of National Intelligence.
By settling important questions about such matters as the size of certain pipes within the facility, the photos provided important clues to its capabilities.
The pictures weren't available when earlier assessments of Iran's nuclear program were developed. And if they had been, the CIA's Open Source Center and other initiatives to cull publicly available information still were in their infancy.
Even now, intelligence agencies struggle with harnessing all the open-source information that's available and grabbing useful nuggets, Naquin acknowledges. "The volume is overwhelming."
Since 2005, when the center opened, the government has increased spending substantially on technology and training to help analysts find useful information, Naquin says, though precise dollar and staffing figures are classified.
"With so many sources out there (to monitor), we're required to use technology much more than we have in the past to help pinpoint useful information," he adds.
A slow change
Intelligence agencies have used open source information for decades, but it amounted mostly to monitoring foreign news broadcasts. As the information age dawned, those agencies were slow to seize on all the material that began pouring into the public domain.
As far back as 1996, a congressional commission created to study intelligence issues noted that tremendous amounts of open-source information had become "readily available," but the intelligence community had been "inexplicably slow" in using it.
Nearly a decade later, little had changed, according to another commission assembled by the White House to assess the intelligence community's failed pre-war assessment that Iraq was developing weapons of mass destruction.
Open sources "offer vast intelligence possibilities," said the commission's 2005 report. "Regrettably, all too frequently these 'non-secret' sources are undervalued and underused."
Steele, the former Marine and CIA officer, notes that the intelligence community's investment in using open sources still is a tiny fraction of what's spent on collecting secrets, and he says the proportion should be reversed.
"I'm not a librarian saying open sources are cool and we can do this," he says. "I'm a very good former spy saying open sources are cool and we can do this."
Some intelligence officials still see secret information as more reliable, but increasingly "there's much more cultural acceptance" of open-source material, says Charlie Allen, undersecretary of intelligence at the Department of Homeland Security.
"It began to change dramatically in the late '90s, and it really took hold after 9/11," says Allen, who pushed to expand open-source collection during decades as a top CIA official.
"Open source is the world of the future."
Open-source successes, such as the new assessment of Iran's nuclear program, have helped push that change. In another instance, Naquin says, an intelligence officer picked up a terrorist training manual that was distributed at a public event in Southeast Asia.
Acknowledgements of open-source coups are rare, because officials don't want to reveal the sort of information they find useful.
Plus, open-source information does not often lead to "eureka" moments in the intelligence world, says Wayne Murphy, assistant director in the FBI's Intelligence Directorate. More often, its main use is to "add perspective and context" to material gathered through classified means.
Additionally, he says, such information helps officials better focus classified missions on material that can't be obtained elsewhere. "Increasingly … we are asked to look at open source first, then show the added value we get by applying (classified) assets."
'Know your enemy'
Ellen Tudisco, chief of the Defense Intelligence Agency's open-source office, acknowledges that there has been "a sort of legacy attachment to classified sources." However, she says, "we've come to realize that you can know your enemy by looking at what they say, and they say it in open sources."
The DIA's experience reflects both the growing acceptance of open-source information and the hurdles intelligence agencies face in trying to use it.
In the 18 months since her office was created, Tudisco's staff has grown from two to 15.
She says it will take at least nine more months to reach the goal of giving all DIA analysts ready access to the Internet and teaching them to work securely in the unclassified world.
The FBI faces similar challenges. About 11,000 FBI personnel now have desktop Internet access, says the FBI's Murphy.
Another 19,000 still need it, and Murphy says it will take until the end of 2009 before they all get the necessary equipment and are trained to make the most of it.
"Just giving someone a computer and an Internet account doesn't make them an open-source analyst," Murphy says.
The FBI also is one of several security agencies that are re-evaluating their approach to seeing that sensitive government and private-sector information doesn't leak out through the same sort of open sources that the intelligence community is trying to exploit. That means setting new priorities for protecting data.
"When secrets are harder to keep … what do we really have to protect?" asks Joel Brenner, counterintelligence executive for the director of national intelligence.
It's a question that government, businesses and individuals all must face, he adds, noting that the FBI is working not only with the Pentagon but also with private industry to identify what sort of information needs to be protected and how best to do it.
"Now, counterintelligence is a problem for everyone who has secrets to keep and … lives on a network," Brenner says. "And that's all of us."