(Hotel Caribe)

A senior administration official briefed on the accusations against the Secret Service agents pulled from duty in Colombia tells ABC News that a heated argument between at least one of the alleged prostitutes and at least one of the Secret Service agents first alerted authorities of the Hotel Caribe in Cartagena to the cavorting between Secret Service agents and prostitutes.

The argument was a dispute over the bill for services rendered, the administration official said.

Hotel Caribe authorities went to investigate the ruckus and learned that there had been some activity between Secret Service agents and prostitutes, the senior administration official said.

Hotel authorities then went down to the reception desk to see who else of the American guests may have signed in female guests - call girls - for the evening.

Initially, this official said, that inspection led the hotel authorities to have questions about 22 Americans - 17 Secret Service agents and five special operations soldiers who were there to assist the Secret Service. Their names were reported to the lead U.S. military official on the ground.

That is not to say that all 22 men had hired prostitutes, the administration official underlined. Some of those about whom the hotel raised questions may merely have been attending a party and violating curfew. Eleven Secret Service agents have been sent back to the United States. The five U.S. special forces members remain in Colombia, per the request of the Secret Service.

The State Department says that adult prostitution is legal in designated "tolerance zones" in Colombia, though restricting the sex trade to those zones has been difficult. "Sexual tourism" is reportedly widespread in Cartagena and other coastal cities.

- Jake Tapper