This morning on "This Week," White House Senior Adviser Dan Pfeiffer told ABC's George Stephanopoulos that the legality surrounding the targeting of conservative groups by the Internal Revenue Service is "irrelevant," but called the behavior "outrageous and inexcusable."

"I can't speak to the law here. The law is irrelevant. The activity was outrageous and inexcusable, and it was stopped and it needs to be fixed so we ensure it never happens again," Pfeiffer said on "This Week."

Stephanopoulos asked Pfeiffer if he really thought the law is "irrelevant."

"What I mean is, whether it's legal or illegal is not important to the fact that the conduct doesn't matter. The Department of Justice has said they're looking into the legality of this. The president is not going to wait for that. We have to make sure it doesn't happen again, regardless of how that turns out," Pfeiffer said.

ABC

Following the interview, Pfeiffer tweeted "Before folks quoting me out of context get too far ahead of themselves, of course the law matters, IRS conduct is wrong even if legal."

Pfeiffer said that there was "absolutely not" any communication between the White House and IRS to exert pressure to more closely scrutinize conservative groups applying for tax-exempt status. He added that while the White House would cooperate with Congress in investigating the matter, it would not participate in "partisan fishing expeditions."

"We are going to work with Congress, as the president said, in legitimate oversight," Pfeiffer said. "What we're not going to participate in is partisan fishing expeditions designed to distract from the real issues at hand."

Sen. Rob Portman, R-Ohio, who appeared on "This Week" following Pfeiffer, said that in addition to congressional committee investigations now taking place, he believed a special counsel will be needed to look into the actions of the IRS.

"I also think a special counsel is going to end up being necessary here, because it has to be independent of the White House. What we do know is that politics was put ahead of the public interest, " Portman said.

"I just find it very hard to believe that lower-level employees here in Cincinnati, Ohio took this on themselves. And that it went on for a couple of years without anybody knowing about it," Portman said of claims that the additional scrutiny of conservative groups was centered in one IRS office in Cincinnati, without direction from Washington.

Rep. Tom Price, R-Ga., described the targeting of conservative groups as "really chilling stuff," and said Congress will continue to investigate the matter.

"I think it's really important to appreciate what Senator Portman started with and that was that the IG report was an audit, it was not an investigation," Price said of the Treasury Inspector General report released last week. "This is just the beginning of this process. And we need to get to the bottom of it. We need to find out who made those decisions, hold them to account and see how high up the chain it went. And that's exactly what we'll do."

But Price said it was too early to say if a special counsel would be needed.

"I think it's premature to determine that. I do believe that the committees of jurisdiction in the House and in the Senate need to continue their investigation and determine exactly who made these decisions," he said.

Like "This Week" on Facebook here. You can also follow the show on Twitter here.

Go here to find out when "This Week" is on in your area.