It's supposed to be one of the happiest moments of your life, but for some, that joy isn't so automatic.

Photographer and author Phillip Toledano confesses it wasn't all sugar and spice and everything nice when it came to the birth of his now 4-year-old daughter, Loulou.

"I was waiting for this tsunami of love and nothing happened," Toledano, of New York City, told ABC News. "I was very perplexed, sort of, by my lack of immediate connection."

In an excerpt from his new book, "The Reluctant Father," on People.com, Toledano admits when Loulou was born, he had trouble developing an emotional bond, even referring to his baby girl as an alien.

"Loulou is screaming her head off and I said to Carla [his wife], 'God, doesn't that just make you crazy?' And Carla said, 'I know, don't you want to pick her up and hold her?' And I said, 'No! I want to throw her out the window,'" Toledano said while laughing.

The new father also revealed he wasn't prepared for the change in his marriage with wife Carla Serrano of nine years.

"I was not number one anymore, and I want to be number one," he explained.

"I think I got angry because it was a very special time for me," Carla recalled. "You know, guys don't get an opportunity to really talk about it."

Toledano, who was 40 when Loulou was born, hopes that by writing the book, "it will make people more likely to be honest with each other.

"People make mistakes," he added. "And I'm flawed like everyone else."

"All men, including my husband, are capable of transformation," said Carla.

For Toledano, that transformation happened as Loulou began to grow. Now, he says he's come full circle.

"She [Loulou] is such an odd, peculiar, lovely, funny thing," Toledano said, fighting back tears. "That's what's so lovely."