"He demonstrated, I think, real courage, real sense of this place during the early '60s. I think about my late sister-in-law who integrated [the] University of Alabama," Holder said. His late sister-in-law, Vivian Malone Jones, was the first black graduate of the university.
"[Alabama Gov.] George Wallace [was] standing in the door, she was the young black student who was admitted on that day as a result of his [Robert Kennedy's] efforts; he pushed the Kennedy administration, I think, in the area of civil rights, he's a legend in this building," Holder said.
The main Justice Department building bears Robert Kennedy's name.
Holder's remarks in a February speech that the United States was "a nation of cowards" when it comes to race relations, spurred controversy. Though he said he stands by his comments, the attorney general also said he believes the country has made progress along the lines of race.
"Race is a tough issue. Wherever it has been found, whether in the United States or even in other countries, it's an issue that has divided us, I think, in the past. It's an issue that if unaddressed, I think, can divide us in the future. And what that speech was about," Holder said.
On the fallout surrounding the recent arrest of Harvard University professor Henry Louis Gates Jr., which has triggered a national uproar about the role of law enforcement and reignited a debate on race relations, Holder said he agreed with Obama, calling it a "teachable moment" for the nation to improve communication between different groups.
"Based on what I know of Professor Gates and what I've heard about Sgt. [James] Crowley, my guess is that they both think that they might have reacted, perhaps, a little differently and this situation might have -- might have been avoided," Holder said.
Though he declined to take sides, Holder, the nation's first African-American attorney general, said that as a young man, he, himself, had been profiled by police during a traffic stop -- an incident that left a lasting impression on him.
"I was a young college student driving from New York to Washington, stopped on a highway and told to open the trunk of my car, because the police officer told me he wanted to search it for weapons. I was at a rest stop. And, you know, I was a college kid. I didn't know quite what my rights were," Holder said.
"People walked by as I was opening this trunk and the officer was looking in there," he said. "And I remember, as I got back in the car and continued on my journey how humiliated I felt, how angry I got."
Holder said that the incident has been an impetus for his work to counter racial profiling at the Justice Department and work toward establishing trust between communities. And while law enforcement is far from being colorblind, he said that agency is pushing for change.
"I think we've made a lot of progress in this country," he said. "We still have a ways to go. But I'm pretty confident that the America of the 21st century can be all that we want it to be."