Hyannis Port Lovingly Remembers Ted Kennedy

As the nation mourns the death of Sen. Edward Kennedy, his absence will be deeply felt in the small community where he grew up, spent his summers and lived out his last days -- Hyannis Port, Mass.

The Kennedy family has been inextricably linked with this seaside town since Joseph P. Kennedy first bought a home here in 1929, three years before his youngest son was born. Ted Kennedy used to describe the big Dutch colonial house as a scrapbook. It did not house any art, but instead was adorned with hundreds and hundreds of photographs.

VIDEO: Political World Remembers Ted KennedyPlay

It wasn't unusual to see Ted Kennedy walking down the dock at the Hyannis Port Yacht Club with his Portuguese water dogs Sunny and Splash by his side on the way to sail his beloved 50-foot wooden schooner, Mya. The senator and members of his extended family could often be spotted on Main Street in Hyannis, eating ice cream at Four Seas in nearby Centerville or attending Sunday Mass at Our Lady of Victory Church.

Visit ABC News' special section on Sen. Ted Kennedy.

Watch video of historic moments in the life of Ted Kennedy.

Living in CamelotPlay

Howard Penn, a close friend of the senator, described first meeting the "Kennedy boys" Jack, Bob and Ted in the 1950s, when they walked into his family's clothing shop, Puritan of Cape Cod, on Main Street.

"They were full of life, all of them," said Penn, whose friendship with Ted grew over the years. "He used to like to drive and he had an old, blue, convertible, a Pontiac I think, and he would jump in it and drive off to church."

Penn has no doubt that even though the senator spent much of his time in Washington, his heart was in Hyannis Port.

"Most definitely they were Cape people. They loved Cape Cod," said Penn.

And just as the Kennedy family loved Cape Cod, Cape Codders loved them back. Deborah Converse lives in Hyannis and runs the John F. Kennedy Hyannis Museum in Hyannis.

"They're our neighbors, and we respect their privacy. Teddy used to be out and about a lot before he got sick. You don't rush up to them, but we treasure them just the same," she said.

Kennedy in particular was beloved by the locals, according to Converse. "He would always donate his paintings to nonprofit groups here. He had student scholars that he would bring in to the museum every year. He was always very thoughtful, especially to any staff member. He would do anything you asked of him. He was just a really good guy," said Converse.

More than 50,000 visitors every year stream through the doors of the museum and most stop and stare at the large black-and-white photographs of the family at the six-acre Kennedy compound in Hyannis Port. The "compound" includes several adjacent houses owned by members of the Kennedy clan.

Converse said the photographs are fascinating to people for a very simple reason "The house is right on the water and absolutely beautiful. The images show the family on the beach and playing flag football. I always thought wouldn't it be wonderful to have a summer like that. It's kind of dreamy."

See photos of Sen. Ted Kennedy's life.

See photos of the Kennedy family's history of privilege and loss.

Seeking to share that dream, it seems, is what motivates so many people to come to Hyannis every summer. Visitors started coming in droves in the early 1960s when John F. Kennedy was president. And they've continued coming with each new Kennedy celebration or tragedy, curious to catch a glimpse or share in the experience of this political dynasty.

The collection of clapboard houses on Marchant Street off Scudder Avenue in Hyannis Port is known as the Kennedy Compound. The houses are largely hidden from view behind a stockade fence and down a private road, so the only way to get a good look is from the water. The original home bought by Joseph and Rose Kennedy has a large wraparound porch and overlooks Nantucket Sound. That house, now owned by Ted Kennedy and his wife, Victoria, is where the senator spent most of his time after his seizure in 2008 and subsequent diagnosis of a brain tumor.

There are several other family homes in the area. Maria Shriver owns a house nearby on Atlantic Avenue, and on a summer day it is not unusual to see various Kennedy or Shriver family members walking the streets or riding bikes on their way down to the beach or heading down to the small, private Hyannis Port Yacht Club to go sailing.

Kennedy's 50-foot wooden schooner is anchored here. In the final months of his life, although it was clearly physically difficult for him, Kennedy would ride in a golf cart slowly down the dock and, with his family supporting him, would board his boat for a sail. The senator often said that sailing was one of the greatest pleasures of his life.

Penn often sailed with the senator and competed against him in a Cape Cod sailing race called the Figawi. The race starts in Hyannis and heads to Nantucket. It's a Cape tradition held on Memorial Day weekend every year. Kennedy was a fierce competitor.

"He takes it very seriously when he is out there. He knows where he is going. He charts the course and when he is behind the wheel he is the one who calls the shots," said Penn, who added, softly,"the water was his life."

A short drive from the Kennedy Compound is Our Lady of Victory Church in Centerville, the Catholic church that many Kennedy family members attended in the summer months. Like so many places in Kennedy's life it has been the focus of both sadness and joy. Joy when the senator escorted his niece, Caroline Kennedy, down the aisle of this intimate country church on her wedding day July 19, 1986. And sorrow when he attended the funeral of his nephew, 39-year-old Michael Kennedy, after a tragic skiing accident in 1998. On that day, the altar was a sea of white roses, lilies and poinsettias. A family friend, Andy Williams, sang "Ave Maria," and a parish priest from Waterford, Ireland, helped celebrate the Mass.

But that is not the only church to figure prominently in Kennedy family life on the Cape.

St. Francis Xavier Church on South Street in Hyannis was the church Rose and Joe Kennedy chose to attend when they first bought their home in Hyannis Port in the 1920s. For the next 60 years, St. Francis Xavier was a summer parish for most of the Kennedy family. It was also the site of several important family events so much so, that the locals often call it the "Kennedy church."

When Joe Kennedy Jr. died Aug. 12, 1944, his parents held a service at St. Francis Xavier. The main altar of the church is now a memorial to him. President John F. Kennedy attended Sunday Mass with his family there during the summer. Church historians recall that the Kennedys would sit in the second row of pews and Secret Service would bracket the president in the front and third rows.

Until she died in 1995, Rose Kennedy, the family matriarch, would regularly take her place in the front row of the east wing of the church. It is also where Patrick J. Kennedy, Ted Kennedy's youngest son, was baptized and where Maria Shriver, Kennedy's niece, married Arnold Schwarzenegger in 1986.

Just steps away from the "Kennedy Church" is the John F. Kennedy Hyannis Museum, where an iconic statue of the former president walking through sea grass has formed the backdrop for thousands of visitors' photos. Inside the museum, room after room is chockful of Kennedy memorabilia.

"In looking back, I realized the Kennedy saga was so much about what could have been. But Teddy really did it. He really accomplished so much, despite all the horrible things that happened in his life. He did what the others didn't," said Converse.

Now that Sen. Edward M. Kennedy, the last Kennedy brother, has died, many wonder if the Kennedy mystique will die with him, if the thousands of people who come to Hyannis Port every year just to brush up against an American political dynasty will keep coming?

But most residents can't imagine Cape Cod without the Kennedys or their followers. And they will tell you that as long as there are Kennedys, those Kennedys will come to Cape Cod, and as long as they keep coming, people from all over the world will too. Dotty, a docent at the museum who didn't want to use her full name, put it simply, "They're like our royalty."

ABC News' David Wright contributed to this report.