After President Bush nominated Bernard Kerik to the top job at the Department of Homeland Security, an avalanche of tabloid stories put the former New York City police commissioner's personal and professional life under the microscope and badly tarnished his reputation.
But to this day, the man who became a national hero for his leadership during the Sept. 11, 2001, attacks says there is only one reason why he withdrew his name from consideration: his family's nanny.
"It was absolutely the only reason," Kerik told ABC News "Primetime Live" co-anchor Chris Cuomo. "There was no other issue on the table. There was nothing else said, there was no other inferences, innuendo."
While he was completing documents required for his Senate confirmation, Kerik says he discovered he hadn't paid all the required taxes for his family's nanny. Kerik soon learned she was using a friend's Social Security number and turned out to be an illegal alien. He then notified the White House that he needed to withdraw.
The next day, another wave of allegations appeared in the media linking him to organized crime and accusing him of misusing an apartment at Ground Zero.
When President Bush tapped Kerik to run the Department of Homeland Security in December 2004, it appeared to be the culmination of the ultimate rags-to-riches story.
Born to an alcoholic prostitute, Kerik dropped out of high school, but had become a decorated cop and New York City's police commissioner.
He saw his star rise along with that of his friend and mentor, New York City Mayor Rudolph Giuliani. After guiding the city through one of the nation's most desperate times, he and Giuliani retired to form a prestigious consulting firm.
"To come from where I had, I often say I've lived the American dream," Kerik said.
But he added, "When you get that American dream, there's a thousand people at the top to despise you for it … No matter what you do, no matter how great you become, there is always someone there to slap you down."
Kerik said as he moved toward the national stage, the media had distorted his past. "If you took everything that's been written and everything that's been said, there are 10, 12 events that people have capitalized on … out of a 30-year career," Kerik said.
Some media raised suspicions that Kerik had connections to organized crime, but he said, "I am as connected to organized crime as I am to Saddam Hussein or bin Laden. It's just absolutely untrue."
There were other accusations that Kerik carried on an extramarital affair. Kerik told Cuomo that he has weathered the storm.
He also asserts that he did not lie to the White House about his nanny, and stresses that the discovery of the problem and his decision to withdraw was all his.
"I found it. I identified the problem. I made a decision based on what I found. And I think that decision was the right decision for my family and for the president and for the Department of Homeland Security," he said.
After all the criticism, Kerik says he now wonders if he should have stood his ground.
"If I knew that I was going to come under all this criticism, and have to fight this battle started, was it right for me to step down? Maybe I should have just fought it," he said.
The former commissioner says the day he had to withdraw his name from the nomination process was "the worst" day of his life.