A recently issued fact sheet from the agency warns that al Qaeda's "demonstrated capability to carry out sophisticated attacks against sizeable structures – such as ships, large office buildings, embassies, and hotels – makes it one of the greatest potential threats to the Olympics."
The warning is part of a security assessment the U.S. government has issued in advance of the Games, which are expected to bring 60,000 to 135,000 spectators to the coastal city each day of the competitions.
The State Department said that although there have not been any "specific, credible terrorist threats" to the Games so far, "in the post-September 11th world, the threat from international terrorist groups at major public events is always a principal concern."
The assessment said that as "security increases in and around Olympic venues, terrorists could shift their focus to more unprotected Olympic venues, open public spaces, hotels, railway and other transportation systems, churches, restaurants, and other sites not associated with the Olympics."
Threats from other extremist groups, like those that carried out the London subway and bus bombings in 2005 and the Madrid train bombings in 2004, are also a possibility, the State Department sheet said.
Former FBI agent Brad Garrett, now an ABC News consultant, said that in light of the attempted Christmas Day underwear bombing, law enforcement are going to be particularly worried about security at the upcoming Canadian games.
"Will they go to full body scanners? I'd be shocked if they're not thinking about that," Garrett said. "If guys are willing to try this on a plane, in theory, it's a lot easier to get it into an event.
Law enforcement will be on alert, "looking for every piece of intelligence possible that links known individuals to wanting to do something there," he added.
The Olympic Games have been targeted by attackers before.
Two people were killed and over 100 injured in Atlanta, GA at the 1996 Summer Olympics when a pipe bomb was detonated in the Centennial Olympic Park.
At the 1972 Summer Olympics in Munich, 11 Israeli athletes and coaches were murdered after a militant group took them hostage.
The Canadian government has allocated $900 million for security for the Olympic Games, five times the original estimate. More than 15,000 individuals will make up the security forces, from the Royal Canadian Mounted Police, the Canadian Forces, private security firms and additional Canadian police agencies.
The RCMP set up a specialized Vancouver 2010 Integrated Security Unit to coordinate security forces, which will also be assisted by U.S. officials.
Americans traveling to the Vancouver Olympic Games are urged by the State Department to register with the U.S. Consulate in Vancouver through the Internet-Based Registration System, which enables U.S. officials to locate travelers in the case of an emergency.