Harry "Skip" Brandon, a former deputy assistant director at the FBI detailed to national security and counterterrorism, called the investigation's results "frightening."
"A federal office building represents to an awful lot of people the U.S. government and if anybody has a problem with the U.S. government , they're going to take it out, where they can get publicity, and it's going to be, most likely, in a federal office building," he said.
"For a terrorist group, it's hugely symbolic. There is no question about that," Brandon continued. "So the federal buildings should be very well protected. It's appalling to me that we see that, in fact, they're not even minimally protected."
Another problem, said both Brandon and Lieberman, is the slashed funding, which results in smaller staffs and contributes to diminished training for the guards tasked with providing security.
"I suspect that part of the problem here is money," Lieberman said. "And we've got to be very careful. If there was ever a case of being penny-wise and pound-foolish, this is it."
"Part of the problem here I'm afraid is that we're not investing the money to hire the best people to be the guards and then to train them effectively," he added.
Lieberman said that in the short term, he'd like to make certain that the Department of Homeland Security devotes extra attention to the FPS, and to later push to give it more prominence within the department because of its critical role. He also suggested moving the service out from under the umbrella of ICE.
"This has been a mess. But it's a mess with very serious consequences for our homeland security, and it's an urgent matter based on this GAO report for Secretary Napolitano and anybody in a position of responsibility in the Department of Homeland Security to act on now," Lieberman said.
As for the training aspect, Brandon said that too often in government, officials lose sight of "what I would say, are front line people, whether they're the building guard, the telephone operator, the people that move the mail, and they represent the greatest vulnerability."
"The first responsibility of the federal government is to provide security," said Lieberman. "If you don't have that, then anything else we do is not worth anything."
ABC News' Theresa Cook and Olivia Hallihan contributed to this report.