"We had a big party at our house the night before," she told ABC News' Christiane Amanpour. "It was going to snow. It was in our garden in Georgetown. That was, sort of, the send-off and everybody was very up and excited."
Smith sat down with "Nightline" to mark the 50th anniversary of her brother's inauguration, and in a rare, exclusive interview she discussed how her brother's words still are relevant today.
Watch the full interview on "Nightline" tonight at 11:35 p.m. ET
"[What] we all remember is that very famous line about, 'Ask not what your country can do for you, ask what you can do for your country,'" Smith said. "And I think that sort of sums up his philosophy. That was a very strong sentiment that he believed deeply in."
The Kennedys are well known for their devotion to public service. Smith made her own contribution when she served as ambassador to Ireland under President Bill Clinton, helping to broker a historic peace agreement involving Northern Ireland.
In addition, no other family in public life has experienced as much tragic loss as the Kennedys. Smith is familiar with the kind of political violence waged on Rep. Gabrielle Giffords, D-Ariz., in Tucson, Ariz.
"It's happened, we all know, in the past," Smith said. "I don't think we should blame a whole group of people for it. I think, probably, it calls our attention to our country and, 'Are we doing the best we can for everybody?'"
Smith believes politicians are more than capable of working together to improve the current political tone.
"I think that if we cross the aisles, as all my brothers did" gaps can be bridged, Smith said. "I think they honestly felt that people ... march to their own drummer and that they have a point of view, and their point of view should be respected. I mean, Teddy, of course, is well known for crossing the aisle."
She chuckled, "I think coming from a big family, it helps you to keep your sense of humor about certain things ... even politics."
Politics has been the Kennedys' family business more than 70 years. Smith said she and her father, Joseph P. Kennedy Sr., an ambassador to the United Kingdom, are the first father-daughter ambassadors in the country's history.
"My father was a very powerful influence, well, always, through our life," she said. "He taught us very much that ... we were very lucky and that we should make a contribution to country, that we were fortunate to live in America."
Although she came from a life of privilege, Smith said her parents' focus always was on their family.
"I don't ever remember a dinner party, a cocktail party in our house ever," she said. "It was always family."
Smith is one of the last direct descendents of what many consider to be America's royal family. Their incredible political influence was demonstrated in the 2008 presidential election.
"Caroline Kennedy and Ted Kennedy not only endorsed Barack Obama, but say explicitly that he was the heir to John F. Kennedy," said Jonathan Alter, author of "The Promise," a book about Obama's rise to the presidency. "[Obama] would not have beaten Hillary Clinton in the Democratic primaries, on Super Tuesday, without the help of the Kennedy family."
Smith echoed her family's decision to support Obama in the race.
"I think he is a good president," she said. "I think he's had difficulty, big difficulties, in the situation he came into, and I think that he handled it very well."
After Rep. Patrick Kennedy, D-R.I., decided not to seek re-election this past November, no Kennedy is serving in a national office for the first time in 63 years. Smith said it's something she doesn't even think about.
"I know one is going to crop up," she said, laughing. "I'll have to go to another fundraiser."