Despite the criticism, Brown has accused others in the administration, including White House officials and Chertoff, of dragging their feet and ignoring his warnings of widespread flooding in New Orleans before the storm hit.
Chertoff said there was "no reason to doubt his commitment."
"If I knew then what I know now about Mr. Brown's agenda, I would have done something different," Chertoff added.
He reiterated earlier statements that he did not realize that levees in New Orleans had been breached on the day of the storm -- despite Brown's claims to the contrary.
"When I went to bed, it was my belief … that actually the storm had not done the worst that could be imagined," Chertoff said.
Collins told Chertoff that "you did seem curiously disengaged to me" when he attended a conference on avian flu in Atlanta on Aug. 30, instead of rushing to the storm scene.
Also today, a group of Republican Congress members released a report that sharply criticizes the Bush administration's handling of the hurricanes that hit the nation's Gulf Coast last year.
According to portions of the draft obtained by ABC News, the report says Chertoff executed his responsibilities "late, ineffectively or not at all."
The report, produced by an 11-member House select committee of Republicans, said that if Sept. 11 was a failure of imagination, "Katrina was a failure of initiative" because it was widely known that the Gulf Coast was positioned to be severely damaged by hurricane-related flooding. "This crisis was not only predictable, it was predicted," the report says.
Democrats refused to participate.
On Monday, Chertoff announced a major reorganization of the Federal Emergency Management Agency, which calls for the following:
The addition of a "substantial" number of permanent employees to serve as a core disaster work force
New communications equipment that will still work when power fails
Special teams to coordinate information sharing between agencies
On Tuesday, the White House said President Bush had full confidence in Chertoff and had not considered asking him to step down.
"Secretary Chertoff is doing a great job," White House Press Secretary Scott McClellan said, answering a reporter's question about Chertoff. "The president appreciates his strong leadership. He is someone who is committed to doing everything he can to protect the American people and to continuing to take steps to make sure we are better prepared to respond to the threats that we face."
Brown told a Senate panel last week that he had dealt directly with the White House in Katrina's aftermath rather than with what he described as a bumbling Homeland Security bureaucracy. Brown said the agency under Chertoff's leadership had been preoccupied with terror threats at the expense of preparing for natural disasters.
The Bush administration has been pushing back aggressively against criticism from Brown and congressional investigators.
Chertoff on Monday rejected criticism that his agency was preoccupied with terror threats at the expense of preparing for natural disasters like Hurricane Katrina.
"I want to tell you I unequivocally and strongly reject this attempt to drive a wedge between our concerns about terrorism and our concerns about natural disasters," Chertoff said.