The U.S. official said the emails were "innocuous" and mostly about upcoming dinner parties and seeing him on TV.
Allen denied he was involved in an affair, a Pentagon official said.
An intermediary for Allen told ABC News that Allen and his wife are friends with Kelley and her husband and most of the emails were sent from Kelley to Allen's wife.
A U.S. official said Allen may have triggered the investigation when he got an anonymous email a few months ago that was traced to Broadwell. The email had a "Kelley Patrol" return address or subject line and painted Kelley as a seductress, which Allen found alarming and mentioned to Kelley in a subsequent email, the official said.
The official described Kelley as a "nice, bored, rich socialite who drops the honorary from her title ... and tells people she is an ambassador. She gets herself in anything related to Centcom and all the senior people and has been for years."
Panetta cautioned that "no one should leap to any conclusions" about allegations against Allen over the investigation.
Panetta said he supports Allen, who has been in command in Kabul, Afghanistan, since July 2011. He took over that summer for Petraeus, who retired from the Army to take over as the head of the CIA.
"He [Allen] certainly has my continued confidence to lead our forces and to continue the fight," Panetta said at a news conference in Perth, Australia, Wednesday.
Panetta declined to explain the nature of Allen's correspondence with Kelley.
Secretary of State Hillary Rodham Clinton, who appeared with Panetta, declined to comment on the Allen case, but insisted the scandal has not harmed the war effort.
"There has been a lot of conversation, as you might expect, but no concern whatsoever being expressed to us because the mission has been set forth and it's being carried out," Clinton said.
Allen has been nominated as the next commander of U.S. European Command and the commander of NATO forces in Europe -- but, despite President Obama's backing, the nomination has been put on hold. The change of command at NATO is currently slated not to take place until March at the earliest.
Allen was supposed to appear before a Senate confirmation hearing this Thursday alongside his designated replacement, Marine Gen. Joseph Dunford. Panetta said that while the matter is being investigated by the Defense Department IG, Allen will remain in his post as commander of the International Security Assistance Force, based in Kabul.
ABC News' Russell Goldman contributed to this report.