Larry Summers: Harvard's Fallen Star

Stephanopoulos pressed, asking, "Is what you said wrong? Or was it just that a university president can't say it?"

Summers expressed remorse over the comments but explained, "What's really important is to try and turn that kind of heat into light by focusing on very constructive things that we can do and that institutions across this country can do, finding ways to assure people that people can pursue a career that gets them to the top, while at the same time meeting their responsibilities to their families."

Despite majority support from both students and the Harvard board, and just as it appeared Summers had weathered the storm, he resigned effective before the start of the next term.

"At the moment, it's the most difficult professional decision I've ever made and hope I'll ever have to make," Summers told ABC News. "I felt the controversy around me personally, whether it was right or whether it was wrong, was at a point where the agenda, the things I cared about -- equal opportunity, pushing forward in science, becoming more supportive of public service, engaging internationally, really having a faculty and a university that worked for the students -- I thought those things would happen faster without the distractions of arguments around things I was doing or things I was saying.

"And so," Summers said, "I thought it was the right thing to step down."

In spite of the controversy, Summers leaves a legacy that includes enabling middle-class families with an annual income less than $60,000 to send a child to Harvard without any family contribution toward tuition. The former treasury secretary also pushed for substantial investments in biomedical sciences, particularly stem cell research.

Summers took issue with those who might critique today's college-aged generation.

"Our students are not lazy," Summers said. "Whether it's the kids who publish the school newspaper, who work 70 hours a week doing that, or the almost infinite number of hours that go into hundreds of student theatrical productions each year, or the level of commitment that it takes to play a varsity sport -- even in the Ivy League -- our students are focused."