Michael Dreeben, arguing for Smith's team, faced questions from Justice Samuel Alito on whether or not presidents can make a "mistake" given the many competing pressures they are under in their day to day duties.

"Presidents have to make a lot of tough decisions about enforcing the law and they have to make decisions about questions that are unsettled," Alito said, then asking if a "mistake" makes a commander in chief criminally liable.

"Making a mistake is not what lands you in a criminal prosecution," Dreeben said.

Later he raised some of the specific accusations in the charges against Trump: "It is difficult for me to understand how there could be a serious constitutional question about saying, 'You can't use fraud to defeat the [certification of the winner of the presidential election], you can't obstruct it through deception, you can't deprive millions of voters of their right to have their vote counted for the candidate who they chose.'"

In a recurring point of interest for the court as it questioned the government, Justice Brett Kavanaugh raised the question of the "risk" of a "creative prosecutor" using "vague" criminal statutes -- including obstruction and conspiracy, which Trump faces -- against any president if they can't claim immunity.

In response, Department of Justice attorney Michael Dreeben said the question about the risk is "very serious" and "obviously it is a question that this court has to evaluate." He argued there is a "balanced protection" with "accountability" for the presidency.

Both Kavanaugh and Justice Samuel Alito appeared skeptical of Smith's use of at least some of the conspiracy and fraud-related charges in the case against Trump. Alito said to Dreeben: "I don't want to dispute the particular application of that [conspiracy statute] to the particular facts here, but would you not agree that is a peculiarly open-ended statutory prohibition? And that fraud under that provision, unlike under most other fraud provisions, does not have to do -- doesn't require any impairment of a property interest?"

Dreeben responded: "It is designed to protect the functions of the United States government and it is difficult to think of a more critical function than the certification of who won the election."

Justice John Roberts began his line of questioning by raising concerns about the opinion issued by the D.C. Circuit Court of Appeals in their rejection of Trump's claims of immunity.

Roberts said the statement that "a former president can be prosecuted for his official acts because the fact of the prosecution means that the former president has allegedly acted in defiance of the laws" concerned him because "as I read it, it says a former president can be prosecuted because he's being prosecuted."

Roberts said such a position could put too much faith in the justice system to act non-politically and out of good faith, and he asked whether the Supreme Court should send the opinion back down to make clear to the circuit court that that is not the law.

Michael Dreeben, arguing for the government, responded that there are "layered safeguards" that protect against malicious prosecution.

"We are not endorsing a regime that we think would expose former presidents to criminal prosecution in bad faith, for political animus, without adequate evidence or politically driven prosecution that would violate the Constitution," Dreeben said.

Asked by Justice Clarence Thomas why previous presidents were not prosecuted for controversial actions, prosecutor Michael Dreeben said "this is a central question."

"The reason why there have not been prior criminal prosecutions is that there were not crimes," he said.

He said there are "layers of safeguards" that ensure that former presidents do not have to "lightly assume criminal liability for any of their official acts."

Trump appeared confident while addressing the presidential immunity hearing during remarks Thursday morning in Manhattan.

"We have a big case today -- this judge wouldn't allow me to go, but we have a big case today at the Supreme Court on presidential immunity, a president has to have immunity," Trump said at a construction site where he met with union members, workers and supporters. "If you don't have immunity, you just have a ceremonial president," he insisted.

When asked about the justices, three of whom he named to the high court, Trump said, "They're very talented people, they're very smart, they know what they're doing."

Trump stopped by the midtown Manhattan site, where the new JPMorgan headquarters is under construction, before heading to the courthouse for his hush money trial. He denies all wrongdoing.

-ABC News' Lalee Ibssa, Soo Rin Kim and Mike Pappano