(Image Credit: Image credit: Charles Dharapak/AP)

Douglas Shulman, the former commissioner of the Internal Revenue Service, said today he was "dismayed and saddened" that his agency had improperly targeting conservative groups, but declined to offer a direct apology and dismissed suggestions that he mislead Congress.

The testimony from Shulman, who was making his first public appearance since the IRS controversy broke into the open two weeks ago, did not satisfy members of the Senate Finance Committee. He faced more than three hours of stern questioning from Democrats and Republicans, but said "I don't believe I was aware" when asked why he had not informed Congress about potential problems percolating at the IRS office in Cincinnati.

"I agree that this is an issue that when someone spotted it, they should have brought it up the chain and they didn't," Shulman said. "Why they didn't, I don't know."

READ MORE: IRS Official Lois Lerner to Take the Fifth

Shulman, who was first appointed by President George W. Bush, acknowledged that the scandal had placed a "blemish" on the IRS. But he said the task facing the IRS was "very, very, very, very difficult," given the rapid rise of groups seeking tax-exempt status.

Shulman, whose term ended last year, said he was aware back then that the inspector general overseeing the IRS was reviewing reports of conservative groups' being targeted. But he said he didn't know the details and only became aware of them recently, six months after leaving his post.

Get more politics with ABC News.

He said he was not directly involved with approving tax-exempt requests by groups. He said he thought it would be "inappropriate to get involved with cases" because he was a political appointee.

Sen. Max Baucus of Montana, chairman of the committee, ordered a second round of questioning after he said the answers had been unsatisfactory.

"The American people have every right to be outraged," the Democrat said.

Sen. John Cornyn, R-Texas, was among the committee members who expressed outrage with Shulman.

"The buck doesn't stop with you?" asked Cornyn, who repeatedly pressed for an apology.

"I certainly am not personally responsible for creating a list that had inappropriate criteria on it," Shulman said.

He added, "This happened on my watch. I very much regret that this happened on my watch."

With a stern tone, Cornyn replied, "I don't think that qualifies as an apology."

The senators also repeatedly pressed Steven Miller, the outgoing acting IRS commissioner who was dismissed last week by President Obama, for new details about what officials in Washington were aware of the targeting practices - and when.

Miller said he assumed responsibility for the unusual nature of how the controversy burst into public view. A question was planted at a May 10 meeting with tax lawyers, a week before the Treasury Department's inspector general was set to release an investigation accusing the IRS of misconduct in its treatment of tea party and other conservative groups.

"We thought we'd get out an apology," Miller said. "Obviously, the entire thing was an incredibly bad idea."

As he did during an appearance before a House committee last week, Miller also apologized "for the mistakes made and the poor service."

He said "partisanship or the perception of partisanship does not belong in the IRS," but adding that the decisions were not politically motivated.

"I think foolish mistakes were made by people trying to be more efficient," Miller said, "not partisan."

Sen. Orrin Hatch of Utah, the ranking Republican on the committee, pressed the two IRS officials on why they have not corrected their previous testimony to Congress when they denied that tea party and other conservative groups were being singled out for exhaustive reviews. He accused Miller of committing a "lie by omission."

"Why did you mislead me?" Hatch said.

"I did not lie, sir," Miller replied.

The inspector general overseeing the IRS, J. Russell George, said he had no reason to believe the conservative groups were targeted because of political bias by IRS agents. But he said his office continued to investigate.

"This matter is not over as far as we are concerned," George said.