While there, Pompeo says he will raise the plight of the three Americans detained by North Korea, but has no commitments for their release.
The trip – Pompeo's second to the highly secretive country – is a sign of the Trump administration accelerating the diplomatic opening with North Korea, as it pushes for Kim to relinquish his nuclear missile program.
Pompeo left Monday night in secret, stopping in Alaska and Japan, and onward to Pyongyang.
President Donald Trump revealed Pompeo's trip as he announced the U.S. is withdrawing from the Iran nuclear deal.
"Today’s action sends a critical message: The United States no longer makes empty threats. When I make promises, I keep them. In fact, at this very moment, Secretary Pompeo is on his way to North Korea in preparation for my upcoming meeting with Kim Jong Un," Trump said.
The U.S. does not know exactly who Pompeo will meet, but he said, "We're prepared to meet anyone who can speak on behalf of the North Korean government and give us solid answers so we're prepared," to pool reporters traveling with him.
On his first trip, he met with Kim himself, in what was an impromptu sit down that lasted more than an hour.
Pompeo said he hopes North Korea will "do the right thing" by releasing the three Americans. U.S. officials have told ABC News they expect them to be released before the Trump-Kim meeting.
"We've been asking for the release of these detainees for 17 months. We'll talk about it again. It'd be a great gesture if they'd agree to do so," Pompeo added, speaking to reporters.
For his part, Trump said Tuesday, "It would be a great thing if they are [released]. We'll soon be finding out."
The trip's primary goal, however, is to lay the groundwork for the Trump-Kim summit.
Pompeo said that first trip over Easter weekend was more an "intelligence effort," to validate what South Korean officials told the U.S. about Kim's interest in talks. It was a limited diplomatic discussion, aimed at learning, listening, and making sure the U.S. understood the outlines of what's possible, Pompeo said.
Since that first meeting, there have been discussions between the two sides, putting outlines around the substance of the agenda for the summit. Now, Pompeo hopes to nail some down and put in place a framework for a successful summit, including specifics such as the location and date, how long it will go on, and the exact venue.
"We also want to make sure what our expectations are not," he added. "We are not going to head down the path we headed down before. We will not relieve sanctions until such time as we have achieved our objectives. We're not going to do this in small increments, where the world is coerced into relieving economic pressures."
Pompeo said he hopes they outline a set of conditions that give them the opportunity to have a historic, big change in the security relationship between North Korea and the United States.
A senior State Department official added that they "will be listening for signs from North Korea that things have substantially changed since Kim declared on New Year's Eve that he wouldn't mass produce nuclear warheads and the means to deliver them... We are looking for bold steps. Anything less would be repeating the mistakes of the past."
The U.S. goal is incremental, gradual, long-term disarmament, the official added: "We are not going to fall for theatrical pronouncements on the end of their nuclear program."
Traveling with Pompeo are the National Security Council Senior Director for Asian Affairs Matt Pottinger, State Department Policy Planning Director Brian Hook, and spokesperson and acting Under Secretary of State for Public Diplomacy and Public Affairs Heather Nauert.