"It should not be a diplomatic issue. We have laws that need to be enforced," he said. "My child should not be in Japan, and is in Japan as a result of criminal activity. And that's what has to be straightened out here. It's not for a diplomat to go and decide that."
In the case of Sawyer's ex-wife, she admitted to ABC News that she told the Consulate General of Japan in San Francisco that she lost her Japanese passport, when in reality, she had surrendered it in accordance with a U.S. court order.
Though the mother initially declined to tell ABC News how she got the passport for her son – saying she would release the information when he was an adult – she eventually admitted to using the fake name. Documents obtained by ABC News show she requested Wayne's passport using the first name Issa, a name that Sawyer says has never been associated with the boy. She used her maiden name as his last name.
The Japanese Consulate in San Francisco told ABC News that they only issue passports to minors with the consent of both parents.
"When we issue passports to Japanese minors, we will explain to applicants that a passport will be issued when an application is submitted with consent of both parents, and that is against the United States law for a parent to take a child out of the country without consent of the other parent who has the custody of the child. We thus take strict measures in checking on the consent of both parents on the application before issuing passports."
Still Wayne's mother was able to get one for him.
This could be more than a diplomatic issue. ABC News showed Jacobs, the assistant secretary of state, a video of Sawyer's wife admitting to lying and using a fake name to get passports for herself and her son.
Asked if lying to get a passport could present a national security threat, Jacobs nodded and replied, "That could be."
"It certainly doesn't make me happy," she said.
Unlike success stories of children being returned from countries like Brazil, Pakistan and Syria, the U.S. has been unable to reach any kind of agreement with the Japanese.
Jacobs said a large part of the problem lays with Japan's domestic custody laws, which do not support giving the left-behind parent access to the child.
"The president considers it a priority. Secretary Clinton considers it a priority," Jacobs said. "It's not a question of power, it's a question of persuasion. Japan has to do the right thing."
The close ties between the U.S. and Japan on issues like economic trade and military policies, Jacobs said, have not affected the State Department's willingness to push Japan to return the American children.
"We have many important bilateral issues," she said. "I think we need to take this one, separate it and tackle it from a diplomatic perspective and also really work with the Japanese to find out what really is in the best interest of the children involved."
But some of the fathers say attempts at persuasion are not enough.
"We tend to try to judge the State Department success by whether or not we get to have our children returned or at a minimum see our children," Toland said. "And none of that criteria has been met."
Toland's daughter, Erika, was taken in 2003 by his former wife, Etsuko Futagi, from U.S. military housing in Japan, where he was stationed.