The president had also been expected to push the banks to increase lending to small businesses and to participate in the administration's housing help program announced in February to help millions of homeowners modify the terms of their mortgages and avoid foreclosure, Summers said.
Since then, almost 760,000 Americans have signed up for assistance, but banks have accepted only 31,382 applications.
The question for the president now is whether his stern words will be heeded.
Obama told ABC News' Barbara Walters last year that bank executives should forgo their Christmas bonuses.
"If you are already worth tens of millions of dollars and you are having to lay off workers," he said, "the least you can do is say, 'I'm willing to make some sacrifice as well.'"
They did not, paying themselves at least $20 billion in bonuses in 2008.
"That is the height of irresponsibility," Obama said in January. "It is shameful."
Several major banks have set aside money for bonuses this year but have yet to pay them out.
The president said today that he's most frustrated that the same banks that benefited from taxpayer assistance now have lobbyists on Capitol Hill fighting against financial regulatory overhaul.
"If they wish to fight common sense consumer protections," Obama said after the meeting, "that's a fight I'm more than willing to have."
ABC News' Matthew Jaffe and Sunlen Miller contributed to this report.