"We have made a lot of progress. The fact that we have an African-American attorney general, an African-American president, I think, is extremely significant. But it is not an indication that all of the problems that we have confronted as a nation over the years are now resolved," he said.
"I think we have an obligation. We have to continue the fight of all the people who I mentioned in my speech to really kind of ultimately get this nation to the place where I think it can and should be."
After his speech, Holder also said he would be heading to Guantanamo Bay next week, accompanied by acting head of the Justice Department's National Security Division, Matthew Olsen.
"We need to have our feet on the ground to really see what is really going on at that facility, to see how people are being detained, to talk to people down there about the interrogation techniques being used," Holder said.
As part of Obama's executive order to eventually close the facility, Holder has been tasked with chairing the committee to implement Guantanamo's closure.
Holder will coordinate the review of the controversial detention facility with other Cabinet officials from the departments of Defense, State and Homeland Security as well as the director of National Intelligence and the chairman of the Joint Chiefs of Staff.
ABC News' Gary Langer contributed to this story.