While President Donald Trump has also warned of the urgency, he took the issue a step further in recent days and threatened to free ISIS fighters at Europe's borders if countries don't repatriate them.
U.S. ambassador-at-large for counterterrorism Nathan Sales rang those alarm bells Tuesday, but wouldn't say whether the U.S. was actively planning to carry out Trump's threat.
"It's impossible to predict what tomorrow is going to look like in Syria, let alone two months or six months from now ... You could envision all sorts of other scenarios playing out," Sales told ABC News in response to questions about Trump's comments. "We don't want to assume that the relative stability that we see today is an enduring feature."
The Institute for the Study of War reported Tuesday that ISIS "is preparing to free its loyal fighters and followers from prisons and displacement camps across Syria and Iraq."
The terror group is fundraising and organizing within some camps, according to the report, including the sprawling al Hawl Camp in northern Syria where some 70,000 people from the ISIS caliphate, mostly women and children, now live.
U.S. officials estimate that beyond the 8,000 Iraqi and Syrian ISIS fighters, there are about 2,000 foreign fighters in detention camps, mostly run by local Kurdish forces aligned with the U.S.
Sales said the U.S. has heard "some disturbing reports" about al Hawl and other camps and seen "several attempted prison breaks over the past several months," adding it all "is adding to our sense of urgency to facilitate repatriation."
For months now, the U.S. has been urging countries around the world to repatriate, or take back, their fighters for prosecution or rehabilitation.
But Trump has had a more extreme message.
"At some point, I'm going to have to say, 'I'm sorry, you either take them back or we're going to let them go at your border'," Trump said Friday in the Oval Office. "They mostly come out of Europe, and we've done them a tremendous favor ... So they have to make their decision. Otherwise, we're releasing them at the border."
The threat almost seems to be a good cop/bad cop routine -- Trump's threatening to release these ISIS fighters in countries or at their borders, while U.S. officials push these partner countries to build the political will to accept returned fighters and make reforms to their legal systems to prosecute them effectively.
To that end, the State Department has provided funding and legal know-how to several countries, including examples from American cases. Eight adults and 13 children have been repatriated to the U.S., according to State Department officials; six of those adults are facing criminal charges, according to public indictments.
But the U.S. will not accept any foreign fighters for prosecution in the U.S. or send them to the detention facility at Guantanamo Bay, Sales said.
"We are not prepared to solve this problem for other countries ... If a person was radicalized in country X sufficiently to travel to Syria to fight for ISIS, that's on country X. That's not on the United States now," Sales told ABC News.
In the meantime, the U.S. is working to strengthen the security of these ISIS prison camps, but it is also calling for more funds from other countries, in particular members of the coalition against ISIS, to do that, too.
A panel of experts report funded by Congress and published Tuesday suggested that the U.S. create a special coordinator for this specific issue of ISIS fighter repatriation, given the urgent need and complex issues involved.
The report also includes an implicit criticism of the Trump administration and the president's sudden announcement that he was withdrawing U.S. troops from the country. It recommends the U.S. "maintain consistency" in its policy and ensure "that, going forward, [U.S. partners] are never surprised or undermined by U.S. policy actions or announcements."