"You could see me sort of cringe on camera when the president said that," Brown said, adding that he knew that the administration hadn't grasped the severity of the devastation.
Brown announced his resignation Sept. 12, 2005, saying the negative publicity surrounding his leadership was a distraction from the job at hand.
The White House refused to discuss his resignation, except to say he was not fired. Brown now admits that he was forced out by the Bush administration.
"They probably would have fired me anyway, if I had gone public with what was really happening," he said, "but at least I would have called attention to how farcical the Department of Homeland Security had become with respect to the way that it does business."
Congressman Thompson said he knew nothing of Brown's contention that he had been fired.
Either way, Thompson said, the "question is, are we a better FEMA or a better Department of Homeland Security. ... The true test will come with the next Katrina-like event."
In the meantime, on this 5th anniversary of Hurricane Katrina, Brown will return to New Orleans for a brief visit to the place where many people still blame him for city's chaos and suffering.
Calvin Lawrence Jr. contributed to this story.