"I just told her I was going to get some help and I wanted to make sure she would take care of [his daughter] Sunny," James said. He said Bullock agreed, and said that she was "proud" of him.
But while in rehab, James' reputation took another hit when US Weekly magazine ran a photo of him wearing a German S.S. officer's hat, posing in a Nazi salute. James said the photo, which was taken before his marriage to Bullock, was meant as a joke, but he understands why it has stirred such controversy.
"I could tell by the look on my face it was a joke that was funny then, probably for a minute, but then looking at it in the context of now and in my life, it's not funny," he said. "There's not a racist bone in my body."
After months of keeping a low profile, James said one of the reasons why he sat down for the "Nightline" interview was to dispel these rumors.
"I think out of everything, this is why I'm doing this interview the most, because you know, dealing with losing my marriage and my son and embarrassing everyone and decimating my life, but to be called a racist on top of that is, it makes me really sad," James said.
James told his side of the story and opened up on his past, his time in rehab and what made him cheat, in his first television appearance since news of his affairs came out in March. At one point in the interview, while speaking about the reasons why he was unfaithful, James teared up and asked for a moment to collect himself.
"I've spent my whole life with the kids and with her trying to ... protect her from everything, but the most hurt she's ever been caused is by the person that wanted to protect her," James said. "It's terrible, and she's going to be hurt for a long time."
Watch the exclusive interview with Jesse James tonight on a special edition of "Nightline" at 11:35 p.m. ET/PT.