Filner, whose parents were liberal fundraisers, met King at his home in New York City when he was 13. Later, still in his teens, he was jailed in 1961 with black protestors at the notorious Parchman jail in Mississippi who had ordered coffee in an all-white bus waiting room.
He went on to represent California in Congress from 1993 to 2012. Filner said he was one of three congressmen whose vote put Obama "over the top" in the super delegate count at the Democratic convention in his 2012 bid for the nomination.
"I have always dedicated myself to nonviolent social change," Filner, 70, said. "The system still needs our efforts."
But he cautions that those who backed Obama's vision for hope and change "have now gone home.
"The town hall meetings on health care have been taken over by the Tea Party," Filner said. "We should have been there. …We can't leave [Obama] on his own."
Filner, who has worked with both the Clinton and Obama White House, faults the 44th president for being "more aloof."
"Clinton had an enormous capacity to identify with average people," he said. "Obama should be organizing, as opposed to all these media events. … That's what King was all about, organizing.
"They are both great orators," Filner added. "But King was seen as the outsider trying to get in."
The civil rights landscape has also changed, according to Filner. Young people are "looking inward" rather than engaging in civil disobedience.
"It's an economic fear," he said. "We didn't have that sense of economic retribution in the '60s. People today are afraid their career will be hurt."
But Benjamin Jealous, at 39, the youngest-ever president of the NAACP, has more faith when it comes to King's dream of "judging a child by the strength of his character, not the color of his skin."
"This generation that is rising is the most diverse we have ever had in this country," he said. "It's also the most inherently inclusive and the most embracing of racial and gender equality. … It's in their DNA."
The son of a long line of civil rights activists, Jealous successfully fought to help make same-sex marriage legal in Maryland and is working to abolish the death penalty there.
He credits Obama for supporting the undocumented immigrant "Dreamers," "evolving" on gay marriage and working to repeal "don't ask, don't tell," the policy that banned gays from openly serving in the military.
He said Obama's second term is "an important victory for future generations." But he warned that Americans must "keep the pressure on" to rid the nation of racism.
"It's time for us to recognize our children, whether they are poor or of color, they are all of our children," Jealous said. "We, more than any other country on earth, litter our language with difference before we get our nouns of commonality. We literally say 'poor white child' or 'that black child' when we need to say American child when it comes to stopping poverty."
When Obama takes the oath of office Monday with King's "traveling Bible" -- along with a second one that belonged to Abraham Lincoln -- as Americans remember the civil rights leaders's sacrifices, it will be "a moment rich with symbolism," Jealous said.
"One cannot help but feel our ancestors, many of them taken from us without mercy, are looking down from heaven and winking back at us."