"We want to move forward. We don't want to look back in that regard," Holder said. "My obligation as the head of the Justice Department is to make sure that the laws are followed and to the extent that we find that laws were broken, to hold people accountable.
"I think the department, at least some of the people who worked, simply lost their way," he said, citing tactics like waterboarding, which he considers torture. "I will follow the facts and the law wherever it takes me."
Another issue weighing heavy on the attorney general is how best to shut down the detention camp at Guantanamo Bay.
"Guantanamo is a place that has served as a recruiting tool for those who want to do us harm," he said.
Holder declined to say how many detainees could face trial in the United States, but said U.S. attorney offices around the country have been reviewing the cases for possible prosecution in those districts.
"At this point, a significant number have been referred to the Justice Department for possible prosecution and those cases have been referred out [to] four or five U.S. attorneys' offices. And they're in the process right now of looking ... at where they can make cases against these individuals. I don't know what the number, the final number is going to be," he said.
Holder also said some of the detainees could be held indefinitely.
"The possibility exists that there could be people who are held in a preventative way under the laws of war," he said. "If that happens, we'd only do so by creating a system that had due process.
"I think that by closing Guantanamo, by prosecuting people, be it in Article III courts, or in military commissions, we will make the American people safer than they are now," Holder continued.
On Wednesday, Holder met with Senate Judiciary Committee Chairman Patrick Leahy, whose panel has oversight over the Justice Department.
The Vermont senator said Holder's job "is probably the toughest job there is, because it is the one part of the Cabinet where certainly you have to reflect the president, but you've got to be independent."
Leahy was highly critical of one of Holder's predecessors, Alberto Gonzales, who resigned under fire for allowing political ideology infiltrate the Justice Department. Gonzales had long been a friend and ally of George W. Bush, and critics said Gonzales operated the department as an arm of the White House.
Leahy acknowledged Holder's close ties to President Obama, but said his approach to the department is more in line with that of Robert Kennedy, who served in his brother John F. Kennedy's Cabinet. Recalling a time when he met Attorney General Kennedy, Leahy said he was struck by his independence from the White House.
"Of course he was as loyal to the president as anybody could be. He was the president's closest advisor, most trusted person. He was his brother. But he was independent as attorney general, [which] is probably the best model for being attorney general," Leahy said. "I think Eric Holder fits that model."
On the wall in Holder's office at the Justice Department hang portraits of several past attorneys general -- Kennedy is one of them.
Holder told ABC News that Kennedy serves as an "inspiration" for many reasons, but one that hits close to home.