While in Oak Bluffs, the Obamas might visit their longtime family friend Charles Ogletree, a professor at Harvard Law School. Filmmaker Spike Lee also has a home in the town.
Closer to the Obama family's vacation rental in Chilmark, other prominent African Americans also have summer homes, including longtime Bill Clinton adviser Vernon Jordan and Harvard Professor Henry Louis Gates Jr.
Chicago friend and White House senior adviser Valerie Jarrett is also taking time away from work on Martha's Vineyard this month.
During times in the nation's history when Jim Crow segregated public schools, restaurants and transportation, Oak Bluffs remained an integrated community and retreat for many families from discrimination.
"African American families and individuals started coming here right after the Civil War," Hayden said. "They came primarily as individuals, solo men and women because there were tremendous job opportunities here. Some decided to stay over into the fall and others came early in the spring."
As those working-class individuals and families eventually became middle-class, African Americans began purchasing homes in Oak Bluffs from what was formerly a Methodist Revival Camp.
Jill Nelson, author of "Finding Martha's Vineyard: African Americans at Home on an Island" traveled to Martha's Vineyard with her family for the summer months for 50 years.
"Certainly the Vineyard is not a racial utopia, but it was and is better than most places," Nelson wrote in her book.
"On the Vineyard we were insulated from many of the racial assumptions and expectations, most of them negative, that at the least intruded upon and at worst defined many of our lives off-island."
For Nelson, Oak Bluffs was about family and traditions -- traditions that continue for families like Bob Hayden's. This summer, the Haydens will enjoy a vacation that includes three generations of their family tree.
"My three adult children came with me in the '60s and '70s and now as adults in their thirties and forties, they come every summer, now with their children, my grandchildren," he said.