It would be "foolish" to rule out the possibility of President Obama being impeached, despite House Speaker John Boehner's direct denials that there are any plans to do so, because of the power Sen. Ted Cruz has in the House, White House senior adviser Dan Pfeiffer said today.

"When the House takes an unprecedented step to sue the President of the United States … even though he is issuing executive orders at the lowest rate in 100 years. I think it would be foolish to discount the possibility," Pfeiffer told ABC's George Stephanopoulos on "This Week."

"John Boehner may have the gavel, but Ted Cruz has the power," Pfeiffer said earlier in the interview.

Boehner said last week that there is no plan to impeach the president, blaming Democrats for the speculation.

"Listen, it's all a scam started by Democrats at the White House," Boehner said Tuesday.

Last month, Pfeiffer told reporters at a Christian Science Monitor breakfast he didn't consider impeachment out of the question.

"I would not discount that possibility. I think that Speaker Boehner, by going down this path of this lawsuit, has opened the door to Republicans possibly considering impeachment at some point in the future."

On "This Week" Sunday, Pfeiffer also pushed back on reports on potential action the president is considering taking to address the crisis at the southern U.S. border and broader immigration issues facing the country.

"I think the reports you're seeing are uninformed speculation," Pfeiffer said.

"[T]he president has asked the Attorney General and the Secretary of Homeland Security to present him with recommendations by the end of the summer. They have not done that yet. So let's wait and see what those are before we make judgments about them," he said.

Pfeiffer also defended CIA director John Brennan, who this week issued an apology after it was revealed the CIA spied on the computers of Senate staffers producing the Senate Intelligence Committee's new report on CIA interrogation policies. Brennan had previously denied the CIA had spied on Senate computers.

"John Brennan is a man of great integrity and ability," Pfeiffer said.

"I think what we have to do is look at how Director Brennan responded to this," the White House adviser added. "When there were allegations of improper conduct, he was the one who referred it to the inspector general."

ABC's Molly Nagle contributed to this article.

Like "This Week" on Facebook here. You can also follow the show on Twitter here.

Go here to find out when "This Week" is on in your area.