Pompeo made the remarks during a trip to Ottawa for talks with Prime Minister Justin Trudeau and Canada's foreign minister.
Beijing detained ex-diplomat Michael Kovrig and entrepreneur Michael Spavor on Dec. 10 in an apparent attempt to pressure Canada to release Meng Wanzhou, a top executive at Chinese tech giant Huawei. She was arrested Dec. 1 at the request of U.S. authorities who want her to face fraud charges in the U.S.
"Please do know our team is focused on helping those two Canadians be released," Pompeo said to Trudeau, Foreign Minister Chrystia Freeland and members of the media. "We're working on it diligently. It's wrong that they are being held."
Pompeo said he was there when Trump brought up the detained Canadians with Chinese President Xi Jinping at the G-20 summit in Japan in late June. He said Trump made an unambiguous statement about America's concern about the "inappropriate behavior."
"We will continue to do that until such time as they are home with their families," Pompeo said.
Trudeau said at the start of his meeting with Pompeo that he would bring up plight of the two Canadians and thank the U.S. for its efforts as well as talk about "how we move forward on that."
Beijing threatened grave consequences after Meng was arrested at Vancouver's airport and the detention severely damaged Beijing's relations with Ottawa. China has also stopped importing certain Canadian products like canola and meat. China also re-sentenced a convicted Canadian drug smuggler to death after the Meng arrest as part of an apparent campaign of intimidation and retribution against Canada.
The two detained Canadians have been accused of conspiring together to steal state secrets. No evidence has been provided and they have not been allowed access to family members or lawyers while in custody.
Meng, the daughter of Huawei's founder, is accused of lying to banks about the company's dealing with Iran in violation of U.S. trade sanctions. She is out on bail in Vancouver and living in her multimillion-dollar mansion awaiting extradition proceedings.
Pompeo accused a reporter of "taking the Chinese line" when the journalist compared the detained Canadians to Meng's arrest.
"The arbitrary detention of two Canadians citizens in China is fundamentally different as a human rights matter, as a rule of law matter," Pompeo said.
"They want to talk about these two as if they are equivalent, as if they are morally similar, which they are fundamentally not."
China and the U.S. are currently embroiled in a trade war that has roiled global financial markets. Pompeo said Meng is not a bargaining chip in the U.S. China trade talks. The U.S. has taken pains to emphasize that the trade talks are separate from the U.S. case against Meng.
Pompeo also discussed ratification of the new free trade agreement between the United States, Canada and Mexico during his trip.
Pompeo also said at a news conference that he was "disappointed" that South Korea said Thursday it will terminate an intelligence-sharing deal with Japan that focused on classified information about North Korea. The surprise announcement is likely to set back U.S. efforts to bolster security cooperation with two of its most important allies in the Asian region.
"We're urging each of the two countries to continue to engage," Pompeo said. He said he spoke to his South Korean counterpart earlier Thursday.