But killing or capturing bin Laden and his top lieutenants is a job exclusively designated to a special task force, variously known as Task Force 121 or Task Force 626. No one else can do the deed. And the go order, this officer said, can be a long time coming.
"When you get an actionable intelligence, then that intelligence has to be confirmed," he said. "Once confirmed, then there is a planning cycle that's entered into, three courses of action. Courses of action have to be approved at higher headquarters. Coordination with other elements has to be made. And then, of course, it has to be approved at the highest levels of our government. Well, in this world, people don't stand still that long, particularly people who … know they're being hunted."
In a sense even today, more than four years after Sept. 11, the gloves aren't entirely off in the fight against al Qaeda.
"The Pakistanis certainly are encouraged to do it," Keane said. "The Pakistanis will tell us they are doing it. But the facts speak for themselves."
The former senior officer of the U.S. Special Forces who spoke to ABC News anonymously suggested bin Laden might be living fairly comfortably right now.
"If he's living in the village -- and he probably does move around to different villages and different homes; they have lots [of] money, and there's lots of money in that area, in any event -- I think he's eating very well," the former officer said. "I think he's probably got more than ample and satisfactory living conditions."
He said his old colleagues thought bin Laden might be trying to disguise himself.
"One hears from friends and colleagues and so on that he has changed his appearance significantly, maybe even had some reconstructive facial surgery, shaved," he said. "I think in my own personal opinion, that's why we have not seen a video of him in well over a year."
Clarke says that bin Laden's direct command of al Qaeda may be weakened by his isolation, but that may be a small consolation.
"He probably doesn't have much of an operational capability left," Clarke said. "But how much does it take? It took 19 people to do 9/11."
ABC News' Terry Moran originally reported this story for "Nightline" on Jan. 19, 2006.