(Hotel Caribe)

The scandal surrounding Secret Service personnel allegedly drinking excessively and having women - possibly prostitutes - in their rooms in a Colombia hotel where they were preparing for President Obama's visit has widened to include five members of the military who may also have been involved.

The military announced today that five personnel who were assigned to assist the Secret Service have been restricted to quarters and will be sent home to face questions on their alleged participation.

According to the Secret Service, the allegations were made Thursday against 11 special agents and Uniformed Division officers, none of whom are assigned to the Presidential Protective Division.

"The nature of the allegations, coupled with a zero tolerance policy on personal misconduct, resulted in the Secret Service taking the decisive action to relieve these individuals of their assignment, return them to their place of duty and replace them with additional Secret Service personnel," Assistant Director Paul S. Morrissey, U.S. Secret Service Office of Government and Public Affairs, said in a statement today.

"These actions have had no impact on the Secret Service's ability to execute a comprehensive security plan for the President's visit to Cartagena," he said.

All 11 were interviewed today and have been placed on administrative leave, he said.

Two government sources familiar with the situation told ABC News that the allegations against the agents include excessive drinking and contact or attempted contact with prostitutes in their rooms at Cartagena's Hotel Caribe.

If the allegations are proven true they could face reprimands and firing potentially.

According to The Associated Press, an employee at the Hotel Caribe described the agents as "drinking heavily" during their stay. The same employee said the agents arrived at the hotel about a week ago and then left the hotel Thursday.

In addition, the U.S. Southern Command (USSOUTHCOM) announced today that five U.S. service members assigned to Joint Task Force Summit of the Americas in support of the Secret Service violated curfew and may have been involved in inappropriate conduct. The conduct is alleged to have occurred in the same hotel where the recalled Secret Service agents were staying.

The personnel are currently in Colombia but confined to quarters and under orders not to have contact with other individuals. They will return to the United States with the rest of the support contingency at the conclusion of the mission.

USSOUTHCOM Commander Gen. Douglas Fraser said he is "disappointed by the entire incident and that this behavior is not in keeping with the professional standards expected of members of the United States military."

He went on to say that after a thorough investigation, punishment, if appropriate, will take place in accordance with established policies and the Uniform Code of Military Justice.

The White House declined comment, confirming only that it was  informed of the allegations of misconduct before President Obama arrived in Cartagena.

"The president has had, does have confidence in the Secret Service,  and the Service has said, and I would point you to those reports,  this incident had no impact on the president's security," Press Secretary Jay Carney told reporters.

It appears that agents directly protecting the president were not involved.