Secret Service Scandal: President Obama Calls for 'Thorough, Rigorous' Investigation

One of the women believed to have spent the night with the agents would not leave the room until she was paid for her services. Hotel authorities who then got involved also checked to see who else of the American guests may have signed in female guests -- alleged prostitutes -- for the evening, a senior Obama administration official told ABC News.

Initially, that inspection led the hotel authorities to have questions about 22 Americans -- 17 Secret Service agents and five special operations soldiers who were there to assist the Secret Service, the official said. Their names were reported to the lead U.S. military official on the ground.

U.S. officials have stressed that some of those about whom the hotel raised questions may merely have been attending a party and violating curfew. But Secret Service Director Mark Sullivan made a prompt decision to relieve the entire detail of duty and have them return to Washington for questioning.

"I don't think inspector Sullivan would have all of them leave if there was not evidence of prostitution," King said.

Sources familiar with the situation also said the allegations against the agents include excessive drinking.

The group included special agents, Uniformed Division officers and two supervisors, sources said. None were assigned to the Presidential Protective Division.

Secret Service Scandal Details Emerge Watch Video

Secret Service Caught in Prostitution Scandal Watch Video

Secret Service Caught in Prostitution Scandal Watch Video

"The nature of the allegations, coupled with a zero tolerance policy on personal misconduct, resulted in the Secret Service taking the decisive action to relieve these individuals of their assignment, return them to their place of duty and replace them with additional Secret Service personnel," Assistant Secret Service Director Paul S. Morrissey said in a statement.

"These actions have had no impact on the Secret Service's ability to execute a comprehensive security plan for the President's visit to Cartagena," he said.

Commander Gen. Douglas Fraser, who heads the U.S. Southern Command, said he is "disappointed by the entire incident and that this behavior is not in keeping with the professional standards expected of members of the United States military."

He also pledged a thorough investigation of the military members who may have participated in the incident, and punishment, if appropriate, in accordance with established policies and the Uniform Code of Military Justice.

White House press secretary Jay Carney said Obama was made aware of the allegations but said it "would not be appropriate for the president to characterize something that's being looked into by the Secret Service at this time."

Carney insisted, however, that the incident has not been a distraction for Obama, who is participating in the two-day Summit of the Americas with other Western Hemisphere leaders.

The Secret Service most recently faced public embarrassment and intense scrutiny in November 2009 when several agents allowed two uninvited guests onto White House grounds for a state dinner and photo line with the president. The so-called "Gate-crasher" incident resulted in three agents later being placed on administrative leave.

ABC News' Mary Bruce and Devin Dwyer contributed to this report.