Jacqueline Kennedy's Audio Tapes: 'John John' Says 'He's Gone to Heaven'

"John John" Kennedy was 3 when asked by an interviewer what happened to his dad.

Sept. 13, 2011— -- When a young John Kennedy, Jr. – still affectionately known as "John John" - wandered into a session where his mother, Jacqueline was recording an oral history of her slain husband's presidency, there's a moment when time stands still. The tape keeps rolling.

The interviewer, historian and family friend Arthur Schlesinger Jr., took the opportunity to ask the boy a question on his tape recorder.

"John, what happened to your father?" Schlesinger asked.

"Well, he's gone to heaven," the 3-year-old replied.

John Jr. was born 16 days after his father was elected president, and his father's funeral was held on his third birthday. The young boy, standing up straight to salute his father's casket, brought the nation to tears.

But when gently prodded by Schlesinger about what he remembered, the boy adopted the tactic kids everywhere use to ward off prying adults by saying mischievously, "I don't remember any-thing."

John made his escape seconds later, but the moment was a reminder that this President was also a father, who interrupted naps, interrupted the White House school and lined his bathtub with floatie toys for the boy who would insist on piling into the tub with his dad.

That oral history the former first lady was recording nearly 50 years ago will be released this week in a book titled "Jacqueline Kennedy: Historic Conversations on Life with John F. Kennedy."

ABC News' Diane Sawyer will host a prime-time, two-hour special based on the tapes tonight, featuring exclusive, never-before-heard extended audio of Jacqueline Kennedy's oral history, rare photographs, plus an exclusive interview with Caroline Kennedy.

One of the many personal memories Jacqueline Kennedy lovingly shares is her husband's frequent morning wake-ups in the White House.

"The television, gosh sometimes it was loud ... there'd be cartoons, and there was this awful exercise man, Jack La...," Jacqueline recalled, referring to 1960s exercise guru Jack LaLanne.

Daughter Caroline and toddler son John would be rolling on the floor doing exercises to Jack LaLanne with their father encouraging them.

"He'd have them tumbling around. He loved those children tumbling around him," she said.

According to Caroline Kennedy, her father didn't like to read to his children, preferring to make up stories instead. Many of them starred Caroline and a pony, and in his stories she frequently won the Grand National horse race, beating a Miss Shaw and Mrs. Throttlebottom. In his tales, there were also sharks, a girl named Maybelle who hid in the woods, and sometimes the kids would join him on a PT boat and sink a Japanese destroyer.

Watch the two-hour Diane Sawyer Special "Jacqueline Kennedy: In Her Own Words" today at 9:00 p.m. ET on ABC.

Like the Pied Piper, the President was a magnet for children, often a disruptive one for teachers at the White House school attended by Caroline and the children of other members of the administration.

"He'd always come out in the garden during their recess in the morning and clap his hands, and all the little things from school would come running," Jacqueline Kennedy fondly remembered.

Life with John Kennedy was a blur of dignitaries, travel -- the family plane was named The Caroline -- and foreign visitors. "There'd be 50 Lithuanians arriving with folk dolls for Caroline or something at 11 in the morning, then they'd go," she said, apparently exaggerating to make a point.

There are famous scenes of John Jr. tumbling out from beneath his father's Oval Office desk during high-level sessions on public policy or the boy banging away on the typewriter of the president's secretary.

As much as Kennedy reveled in a string of inaugural balls, the morning after being sworn in he began badgering his wife to bring the kids to the White House. They had been staying with relatives while the White House was being repainted and prepared for the new first family.

"He couldn't wait to get the children back. And all that end [of the White House] smelled so of paint, but he'd keep saying 'You've got to bring them back soon.' He really missed them," she said.

In the early days of the presidency, it was Caroline's privilege to walk to the Oval Office with her father each morning. "Later on, it used to be John's treat to walk to the office with him every day," Jacqueline Kennedy recalled.

The family's time in the White House also had moments that were personally painful. On Aug. 7, 1963, Kennedy gave birth to a third child, Patrick Kennedy. The boy was born more than five weeks premature and died two days later. The president was devastated.

"He came back from Boston to me in the hospital and he walked in the morning about 8 in my room, and just sobbed and put his arms around me," she said.

Kennedy also recalled her anger at comedian Vaughn Meader, who had a best-selling album lampooning the first family's children. After its release, the Women's Press Club had a reporter dress up as Caroline and ride a tricycle in an annual political skit.

"I listened to one side and then I threw it away," she said of the Meader album, "The First Family." And the following year, she broke tradition and refused to attend the Women's Press Club dinner in protest of the Caroline skit.

In a forward to her mother's oral history, Caroline Kennedy recalled her mother's love of history, and how she wanted to preserve her husband's legacy by recording an oral history documenting his Presidency. She would spend endless hours in meetings behind closed doors with her husband's aides and friends, ensuring his days as President were properly preserved.

Both of her parents loved history, Caroline Kennedy writes, and she has kept her father's dog-eared and marked-up books.

"I still have his books on the Civil War and English parliamentary history, as well as his annotated copy of 'The Federalist Papers,'" she wrote.

Caroline Kennedy recently shared her mother's oral history with her own children.

"They loved the stories about their grandfather," she wrote, "and how insightful yet irreverent their grandmother was."

To Pre-Order "Jacqueline Kennedy: Historic Conversations on Life With John. F. Kennedy"