The Inspector General's report suggested that IRS employees did not act from political motives in targeting these groups for additional scrutiny.
But according to the report, IRS employees also did not consider the potential for the very kind of political scandal these revelations have sparked.
Now other groups are raising questions about IRS inquiries they received in recent years that they believe may have been politically motivated.
Franklin Graham, son of evangelical pastor Billy Graham, said that his organizations were audited by the IRS in 2012 and he believes that the incident is linked to the agency's current troubles.
"An IRS agent arrived October 15, 2012, to conduct a review of Samaritan's Purse; and an agent arrived on October 29, 2012, to conduct a review of the Billy Graham Evangelistic Association," Graham wrote in a letter to Obama. "In light of what the IRS admitted to on Friday, May 10, 2013, and subsequent revelations from other sources, I do not believe that the IRS audit of our two organizations last year is a coincidence—or justifiable."
The IG report also found that while the IRS's probing halted or delayed dozens of conservative groups, the agency approved other groups who probably needed additional scrutiny.
These IRS revelations have also put Democrats in a bind. For several years, Senate Democratic lawmakers as well as outside groups have pushed the IRS to crack down on excessive political activity by tax-exempt groups. One letter came from seven Senate Democrats in 2012, another was sent to the IRS in 2010.
Senate Minority Leader Mitch McConnell (R-Ky.) blamed Democrats for creating an environment that made the IRS's misdeeds possible.
"Our Democrat friends should also acknowledge their role in inculcating this culture of intimidation, due to repeated calls for increased IRS scrutiny of groups like the very ones that were targeted," McConnell said on Tuesday.
Today, some liberal groups are coming forward with claims that they also received similar letters from the IRS, suggesting that it was not only conservative groups that received this treatment.
"Progress Texas and the Tea Party strongly disagree on the role of government. Yet, when we applied for tax-exempt status, Progress Texas received the same type of additional scrutiny that Tea Party groups are complaining about," said Ed Espinoza, the group's executive director. "The similar treatment indicates the IRS was likely addressing a flood of 501c4 applications after Citizens United, and undermines the paranoid notion that Tea Party groups were singled out."
Still, the IG report makes it clear that IRS officials flagged dozens of groups – 1/3 of the applications that received extra scrutiny – for no reason other than the fact that their names included the words "tea party" or "patriot" or otherwise suggested conservative viewpoints.
ABC News' Jeff Zeleny and John Parkinson contributed to this report.