Reed Saxon/AP Photo

"It's just one of the disastrous situations I've created and the worst thing that I've ever done," former California Gov. Arnold Schwarzenegger told me this morning on "GMA" when I asked him about his affair with the housekeeper who he fathered a child with and kept hidden from his wife for years.

"It hurt so many people and the mistakes and failures I've had in the past, and I write about all of them in the book…But this is the worst because it has affected so many people," he said.

The affair with Mildred Baena occurred in 1996, according to Schwarzenegger's new autobiography "Total Recall," when Maria Shriver and their children were away. The following August Baena gave birth to Joseph, but it wasn't until the boy was older that Schwarzenegger saw the resemblance.

"While Mildred and I barely discussed it, from then on I paid for his schooling and helped financially with him and her other kids," Schwarzenegger writes.

Shriver asked her husband if Joseph was his son years earlier, but he denied it. She again confronted Schwarzenegger about Joseph the morning after he left the governor's office in January 2011, he writes. This time he admitted to the indiscretion.

Shriver filed for divorce in July 2011.

When I asked why he didn't tell his wife about the affair earlier, he said he didn't know how.

"I was just, I think, too scared about losing everything, too scared about losing her, losing the family and I just didn't know how," he said.

Schwarzenegger told me he continued to employ Baena as housekeeper because he thought it would make the situation worse if he fired her.

"It's worse when you start getting rid of people because it's kind of like punishing somebody for being in this situation so that's the best way that I could have handled it," he said on "GMA."

The former governor said he sees Joseph as a son, and takes care of him and his mother financially.

Schwarzenegger's book is out today. But despite writing an autobiography, he said he doesn't like looking backwards and described writing about the affair as "painful but necessary."

"When you look at the whole thing it was just so horrible, what it did to the kids, what it did to my wife, because they are really fantastic," he said.

The biggest lesson he learned, he said, that "one stupid mistake that maybe you think is good, that moment can really destroy a lot of things and can hurt a lot of people that don't deserve to be hurt like that."

"I mean, Maria does not deserve to be hurt like that and nor do the kids and nor do everyone in the extended family so everyone has been hurt but I'm trying to rebuild all that," he said.

But Schwarzenegger acknowledges repairing those relationships will take time.

"I've had a good relationship with the kids and I've had a very good relationship with Maria but it takes time to rebuild the trust and everything and I'm doing everything I can to work on that because I want them back together again because I think we were such a terrific family and just because of my doing and doing this stupid thing, I got into the situation that I'm in," he said.

And as for a reconciliation with Shriver?

"Maybe Maria doesn't say that, but I'm sure that Maria wishes that we could all be together and I definitely wish that we could all be together," Schwarzenegger told me.