Murphy confirmed the meeting with Mohammed Javad Zarif in a Medium post on Tuesday.
"I have no delusions about Iran -- they are our adversary, responsible for the killing of thousands of Americans and unacceptable levels of support for terrorist organizations throughout the Middle East," Murphy wrote. "But I think it’s dangerous to not talk to your enemies."
He added, "Discussions and negotiations are a way to ease tensions and reduce the chances for crisis. But Trump, of course, has no such interests."
Trump expressed confusion over the meeting as he boarded Air Force One on Tuesday.
"I saw Senator Murphy met with the Iranians, is that a fact?" Trump asked reporters, adding "I just saw that on the way over. Is there anything I should know? Because that sounds like, to me, a violation of the Logan Act."
"What happened with that?" Trump continued. "They ought to find out about if it’s true. I don’t know."
The law in question -- the Logan Act -- is an obscure statute that has received renewed attention in recent years. Signed into law in 1799, it penalizes private individuals for negotiating or collaborating with foreign governments on issues involving the United States without the federal government's permission. No one has ever been prosecuted under the law.
Trump took to Twitter on Wednesday, going after Murphy.
"Kerry & Murphy illegally violated the Logan Act. This is why Iran is not making a deal. Must be dealt with strongly!" Trump tweeted.
Murphy responded on Twitter, calling Trump out for his Iran policy, calling it a "disastrous failure."
"Iran restarted their nuclear program, fired at our troops, upped support for proxies. Your Iran policy is a disastrous failure," Murphy tweeted, adding that "It’s literally my job to meet with regional leaders."
Trump had previously gone after former Secretary of State John Kerry for taking meetings with Iran over the years, calling for the former senator and top U.S. diplomat to be prosecuted for meeting with Iranian leaders and "telling them what to do."
"If they met, I don’t know what they said," Pompeo told reporters Tuesday. "I hope they were reinforcing America’s foreign policy, not their own."
Pompeo also bashed Zarif: "This guy's been designated by the United States of America. He's the foreign minister for a country that shot down a commercial airliner and has yet to turn over the black boxes. This is the foreign minister of a country that killed an American on Dec. 27."
He added, "And he's the foreign minister of a country that's the world's largest sponsor of state terror and the world's largest sponsor of anti-Semitism."
Murphy acknowledged in his post that he can’t conduct diplomacy on behalf of the U.S. government.
"I don’t know whether my visit with Zarif will make a difference. I’m not the President or the Secretary of State -- I’m just a rank and file U.S. Senator," he wrote. "I cannot conduct diplomacy on behalf of the whole of the U.S. government, and I don’t pretend to be in a position to do so."
He continued, "If Trump isn’t going to talk to Iran, then someone should."
ABC News' Conor Finnegan contributed to this report.