Opening this weekend, 'Lee Daniels' The Butler' chronicles the life of late White House butler Eugene Allen through his decades of service in the White House.

Former White House storeroom manager Bill Hamilton has had an equally impressive run at 1600 Pennsylvania Ave., spending the last 55 years there before his retirement earlier this summer, making him the longest serving White House staff member ever.

In an interview for "This Week," Hamilton said the first rule for a White House staff member is to check your politics at the door.

"When a president comes in, it doesn't make a difference whether he's Democrat or Republican, you're there for the president," Hamilton told ABC News' Reena Ninan.

Hamilton joined the White House staff at the age of 20, going on to serve 11 presidents from Eisenhower to Obama over the last half-century.

"It was being part of history," Hamilton says. "Nixon resigned from the presidency, that was a hard time and I didn't think I would be there when the President [Kennedy] was assassinated… and to get the first black president I didn't think that would ever happen in my lifetime."

ABC News

Hamilton explained what it was like being a black man in the White House during the civil rights movement.

"Even where there was the March on Washington and Martin Luther King, I was the only one who went out. No one would go out. They said, 'We can't go out there, you're going to lose your job. I said, you won't lose your job… I went out to listen to him speak… it was really very thrilling."

While the fight for civil rights waged on in the streets of Washington, Hamilton says the White House was not immune to the symptoms of racial prejudice and inequality.

"Oh, it was very hard," says Hamilton. "They were promoting other people, but weren't promoting blacks."

But Hamilton did stand up for his fellow White House staff members to gain one measure of equality, delivering the Johnson White House an ultimatum - either the black staff members would receive more pay or they wouldn't work at an upcoming state dinner.

"We're not going to work the state dinner. This is the only way we're going to get a raise," Hamilton said. "Two days later we had our raise."

After witnessing decades of history from the White House before his retirement, one thing remains that Hamilton says he wishes he had seen: the first woman president.

"If Mrs. Clinton was to win, I would say I wish I was still working there," Hamilton says.

Like "This Week" on Facebook here. You can also follow the show on Twitter here.