"He couldn't wait to get the children back. And all that end [of the White House] smelled so of paint, but he'd keep saying 'You've got to bring them back soon.' He really missed them," she said.
In the early days of the presidency, it was Caroline's privilege to walk to the Oval Office with her father each morning. "Later on, it used to be John's treat to walk to the office with him every day," Jacqueline Kennedy recalled.
The family's time in the White House also had moments that were personally painful. On Aug. 7, 1963, Kennedy gave birth to a third child, Patrick Kennedy. The boy was born more than five weeks premature and died two days later. The president was devastated.
"He came back from Boston to me in the hospital and he walked in the morning about 8 in my room, and just sobbed and put his arms around me," she said.
Kennedy also recalled her anger at comedian Vaughn Meader, who had a best-selling album lampooning the first family's children. After its release, the Women's Press Club had a reporter dress up as Caroline and ride a tricycle in an annual political skit.
"I listened to one side and then I threw it away," she said of the Meader album, "The First Family." And the following year, she broke tradition and refused to attend the Women's Press Club dinner in protest of the Caroline skit.
In a forward to her mother's oral history, Caroline Kennedy recalled her mother's love of history, and how she wanted to preserve her husband's legacy by recording an oral history documenting his Presidency. She would spend endless hours in meetings behind closed doors with her husband's aides and friends, ensuring his days as President were properly preserved.
Both of her parents loved history, Caroline Kennedy writes, and she has kept her father's dog-eared and marked-up books.
"I still have his books on the Civil War and English parliamentary history, as well as his annotated copy of 'The Federalist Papers,'" she wrote.
Caroline Kennedy recently shared her mother's oral history with her own children.
"They loved the stories about their grandfather," she wrote, "and how insightful yet irreverent their grandmother was."