An investigation into the U.S. Secret Service prostitution scandal by the Department of Homeland Security's Inspector General contradicts Secret Service director Mark Sullivan's adamant assertion before Congress that "this just is not part of our culture," ABC News has learned.

"Thus far, we have not found that this type of behavior was exhibited by any of these individuals before," Sullivan testified before the Senate Homeland Security and Governmental Affairs Committee in May, referring to the 12 agents who were accused of drinking and cavorting with prostitutes in Cartagena, Colombia, ahead of President Obama's visit for the Summit of the Americas.

The report, however, revealed that one of the agents who was in Cartagena during the scandal and picked up a prostitute "admitted to soliciting a prostitute on two previous occasions, once in El Salvador in 2008/2009 and one time in Panama in 2009."

The report also mentioned allegations of similar misconduct by agents on trips to Romania and China. Details from the report, labeled "law enforcement sensitive," were shared with ABC News by sources who had reviewed it.

The investigation found that while Secret Service personnel were still on the ground in Cartagena, one of the supervisors that had engaged in misconduct was alerted that his actions had become known. He, in turn, warned other Secret Service staffers in Colombia that they should not bring prostitutes back to their hotel rooms.

A senior Secret Service official with knowledge of the investigation said Sullivan had been briefed prior to his testimony, and that "while some agents had been truthful regarding their conduct with prostitutes in Cartagena, none had confessed to prior contact with prostitutes. One agent, who later admitted to the OIG that he had indeed engaged in prior misconduct with prostitutes in El Salvador and Panama, had previously denied in an interview with USSS Office of Professional Responsibility that he had not had prior contact with prostitutes."

Sen. Ron Johnson of Wisconsin, the ranking Republican on a key Senate oversight subcommittee, was thoroughly briefed on the report by the subcommittee staff, which spent two days reviewing it. Johnson noted that it is government policy for Secret Service agents to report certain contacts with foreign nationals.

"In the three and a half years prior to the Cartagena incidents, there were only 105 of those foreign national contacts reported," Johnson said. "Once Cartagena occurred and the policy was redistributed, you know that agents were reminded of that, 423 additional contacts were all of a sudden reported. And again, this gives me concern that rotationally this type of behavior is more widespread."

Sullivan is also facing questions about whether he misled lawmakers about the security risks surrounding the scandal.

In May, he testified that the prostitutes' names - when run through U.S. national security and law enforcement databases - did not raise any red flags, with law enforcement concluding that there was "no connection either from … an intelligence perspective or a criminal perspective."

But the inspector general asserted that Secret Service officials knew when Sullivan testified that information about two of the prostitutes had caused what's commonly referred to as "intel hits." One of those hits has since been dismissed and the other is still being investigated, sources told ABC News.

The senior Secret Service official asserted that before his testimony, "Sullivan was briefed as to the current status of the investigation and the facts known at that time. He was briefed that checks of the women's names against national security and law enforcement databases, both in the U.S. and Colombia, had yielded no derogatory information."

The official acknowledged that the Secret Service was told that there had been "potentially… a partial match to the name of one of the women, but at the time, Director Sullivan was briefed that it was not a match. Indeed, the Secret Service, working with other government agencies, was never able to confirm a connection."

The DHS inspector general has faced challenges in his investigation, with 10 senior and current Secret Service officials refusing to grant him or his investigators an interview.

"We are concerned that the inspector general was interfered [with] in terms of his investigation, that was constrained and hampered," Johnson told ABC News.

The inspector general also said the Justice Department denied its request to pursue the legal authority to conduct interviews with the prostitutes, hotel staff, or night club employees in Colombia and to access hotel records. Justice Department officials asserted they provided the inspector general with the documents that they were seeking.

"These are very serious charges - the fact that the Secret Service has been implicated in this kind of behavior that puts the president's life at risk, our national security at risk and we cannot get the answers," Johnson said.

Ten days after the scandal broke, White House Press Secretary Jay Carney told reporters, "There have been no specific, credible allegations of misconduct by anyone on the White House advance team or the White House staff."

The inspector general's report also noted that the White House counsel conducted her own investigation when two staffers - one a soldier who was part of the White House Communications Agency, the other a White House Advance Team volunteer - were also cited in follow-up investigations, after Carney's comments.

The soldier ultimately confessed, but the advance team volunteer denied any wrongdoing. The White House argues that the only information tying the volunteer to the scandal was a hotel log in which a prostitute listed the volunteer's room number as her destination. White House officials noted that a Secret Service agent was similarly implicated - falsely - in the scandal, and that they are convinced of the volunteer's innocence. They have no further information about whom the prostitute was visiting.

Last week, Johnson wrote letters to Director Sullivan, White House Chief of Staff Jack Lew, Department of Homeland Security Secretary Janet Napolitano and Attorney General Eric Holder asking for a detailed description of the findings from their investigations into the scandal.

"Director Mark Sullivan and the Secret Service have conducted a fair and thorough investigation resulting from the Cartagena incident," Secret Service spokesman Ed Donovan told ABC News in a written statement. "The agency response to those with oversight responsibility has been timely and truthful with over a dozen briefings to Congress, hundreds of employee interviews, and tens of thousands of documents turned over to oversight entities. We have remained in close touch with those partners to answer any questions and will continue to respond to the DHS-OIG and Congressional inquiries in that manner. Since 1865, the Secret Service has done its job with excellence and integrity, and the true culture of our employees is demonstrated everyday as we execute both our investigative and protective missions."

"The fact that we've hit a brick wall just makes me highly suspicious that there is something being covered up here and the American public has a right to know," Johnson said.

A senior White House official said this evening that the White House continues to have confidence in Sullivan.

-Jake Tapper and Mary Bruce