A fed-up Attorney General Eric Holder is accusing congressional critics of launching "unprecedented, unwarranted, ugly and divisive" attacks on him and the Obama administration.

During a speech to the National Action Network, a group founded by Rev. Al Sharpton, in New York on Wednesday lauding the organization's effort to advance racial equality, a heated Holder went a little off-script.

"Forget about me [specifically]. Look at the way the attorney general of the United States was treated yesterday by a House committee," Holder told the crowd. "What attorney general has ever had to deal with that kind of treatment? What president has ever had to deal with that kind of treatment?"

Eric Holder Wishes Critic 'Good Luck With Your Asparagus' After Testy Exchange

Sharpton Opens Civil Rights Annual; Obama Expected

The remarks drew loud applause from the crowd.

On Tuesday, while testifying before a House panel, Holder engaged in a testy back-and-forth with two of his most outspoken Republican antagonists. One accused Holder of violating federal law and questioned whether he should even be allowed to testify before lawmakers, while Rep. Louie Gohmert, R-Texas, seemed to threaten Holder with another contempt citation and angrily told Holder he didn't need to be lectured by the nation's top law enforcement official.

On Wednesday, with a finger raised, Holder told the crowd in New York that his tenure as attorney general has been "defined by significant strides … even in the face of unprecedented, unwarranted, ugly and divisive adversity."

The day before, after Gohmert ran out of time to ask questions, Holder leaned into the microphone in front of him and wished Gohmert "good luck with your asparagus." It was a not-so-subtle dig at Gohmert's unique remark last year when he warned Holder not to "cast aspersions on my asparagus."

Holder has had a tense relationship with several Republican lawmakers since the Republican-led House held him in contempt of Congress two years ago for failing to turn over certain documents tied to the Fast and Furious scandal involving botched firearms sting operations despite a congressional subpoena seeking those documents.

In asking Holder on Tuesday about the Justice Department's refusal to turn over documents in an unrelated terrorism case, Gohmert again raised the issue of contempt, saying, "I realize that contempt is not a big deal to our attorney general, but it is important that we have proper oversight."

A visibly upset Holder, leaning back in his chair, shot back, "You don't want to go there, buddy. You don't want to go there, OK?"

"You should not assume that that is not a big deal to me," Holder added, pointing a finger toward the congressman. "I think it was inappropriate. I think it was unjust. But never think that was not a big deal to me. Don't ever think that."

Gohmert pressed the issue.

"There have been no indications that it was a big deal, because your department has still not been forthcoming in producing the documents that were the subject of the contempt," Gohmert said.

"I don't need lectures from you about contempt," Gohmert added.

"And I don't need lectures from you either," Holder responded.

The documents at issue were mostly from the months after the public first learned that an Bureau of Alcohol, Tobacco, Firearms and Explosives investigation called Fast and Furious put guns into the hands of criminals in Mexico. Republican lawmakers subpoenaed the Justice Department for internal emails dated after Feb. 4, 2011, when department officials realized they would have to retract a letter to Congress denying any such thing happened.

But President Obama then asserted executive privilege over the documents, and Republicans filed suit in federal court.

Holder's "good luck with your asparagus" comment Tuesday was a reference to a remark by Gohmert at a hearing nearly a year ago. At that hearing, Gohmert and Holder got into another heated back-and-forth over how the FBI handled the case of alleged Boston bomber Tamerlan Tsarnaev, who had been interviewed by the FBI years before the attack over possible radicalization.

Gohmert said he was "trying to determine how the FBI blew the opportunity to save people's lives," but Holder defended the FBI's work, telling Gohmert, "You don't know what the FBI did. You don't know what the FBI's interaction was with the Russians. You don't know what questions were put to the Russians, whether those questions were responded to. You simply do not know that. And you have characterized the FBI as being not thorough or taken exception to my characterization of them as being thorough. I know what the FBI did. You cannot know what I know."

In trying to get a chance to respond to Holder's comments, a flustered Gohmert shot back, "[Do not] cast aspersions on my asparagus."

Gohmert has used the phrase at least once before. During a House Judiciary Committee discussion in July 2007 over business law, he said, "I've had aspersions cast on our asparagus in East Texas." And on Wednesday, he told TheBlaze.com that the phrase was often used by a Texas lawyer decades ago.

Asked Thursday about Holder's recent run-ins with Republicans, House Speaker John Boehner insisted, "There's no issue of race here."

"The American people have not been told the truth about what happened in Fast and Furious," he said, adding that the Obama administration has also not been "forthcoming" about other scandals.

"We've been going through all of these hearings, having to hold people in contempt because they've made it impossible to get to the documents," Boehner said. "They owe the American people the truth."

ABC News' John Parkinson contributed to this report.

(J. Countess/Getty Images)