Brendan Hoffman/Getty Images

Weeks before the first of Republican voters cast their primary ballots, Newt Gingrich is embroiled in a controversy that threatens to damage his credibility.

Gingrich is refusing to elaborate on his work at Freddie Mac, the government-backed mortgage giant that hired the former House speaker as a consultant months after he left Congress, and paid him more than $1.6 million over eight years.

After saying at a Tea Party town hall today that he would “cheerfully answer any single question,” Gingrich refused to answer reporters’ questions about Freddie Mac. When asked by ABC News why he wouldn’t talk about that issue, the former congressman replied, “They [the Tea Party audience] didn’t ask.”

During the town hall, he reaffirmed his claim that he hasn’t ever done any lobbying “of any kind.”

“I did not influence peddling of any kind,” he said. “But the truth is if you have the reputation — if you just take what people say about me in the debates and say to yourself: ‘Is that a person somebody might have hired for advice?’ — it’s hard to argue they should have hired someone who is truly dumb.”

Gingrich’s campaign hasn’t yet released any documents, as they said Wednesday they would do.

Most conservatives, including Tea Partiers, want Freddie Mac to be eliminated altogether. And Gingrich has yet to make a convincing argument about his eight-year long relationship with the once-beleaguered organization. His comment that he was hired as a “historian” has, in fact, put his credibility into question even more.

“It’s going to be very tough for him to turn this into a positive,” Republican strategist and ABC News’ political contributor Matthew Dowd said on “Good Morning America” today.

Gingrich continues to insist that he did “strategic planning” for Freddie Mae, and denied ever lobbying for the group.

“I spent a fair amount of time trying to suggest a totally different model where, if you wanted to expand the number of people owning homes, you need to expand the number of people capable of owning homes. You don’t just give people a house because they have no idea what they’re doing. So we’re talking about those kinds of concepts,” Gingrich said in a radio interview with Mark Levin Wednesday night.

But sources told Bloomberg News that Gingrich was hired to build bridges with Republican members of Congress who were wary about the organization’s work.

Gingrich has said he doesn’t remember what he was paid, but that it was “a relatively standard Washington fee for that kind of advice.”

The biggest damage to his credibility could come from how vocally he has blasted Freddie Mac and Fannie Mae. In 2008, he demanded that President Obama and other members of Congress return the money they received from them, and more recently said Democrats like Rep. Barney Frank should be jailed for having ties with lobbyists at those organizations.

As he rises in the polls, Gingrich’s inconsistencies are increasingly coming under the spotlight.

In 1989, Gingrich co-sponsored the Global Warming Prevention Act and in 2008 appeared with then-House speaker Nancy Pelosi to warn about global warming. Now he says he doesn’t know whether climate change is really occurring and called that ad “the dumbest single thing” he’s done in recent years.

Gingrich also once advocated imposed mandatory limits on carbon dioxide emissions, but has recently become a vocal critic of the cap-and-trade bills.

“I do know that I’m opposed to cap-and-trade and I’m opposed to any kind of massive government response,” he said Wednesday.

He has also flip-flopped on his views in Libya, first criticizing President Obama for intervening and then saying later he would have sent forces there.

It’s no surprise that the scrutiny on Gingrich increased when he jumped in the polls. From Rep. Michele Bachmann to Herman Cain, almost all GOP candidates have faced the same kind of scrutiny when their poll numbers surged.

But Gingrich carries heavier political baggage compared to his opponents, which may both help and hurt him.

“I think the expiration date on Newt is a little bit longer than the other candidates,” Dowd said.

The former congressman from Georgia has extensive experience building national campaigns, he has close ties with Republican heavyweights, has handled controversies before and has performed relatively well in recent debates.

But his past also opens him up to criticism. Opponents like Bachmann have attacked him on working too closely with Nancy Pelosi in the House.

On a personal life, Gingrich has also taken some heat. He had an affair with his current wife, Callista, while still married to his second wife. The two racked up nearly a half a million debt at Tiffany’s & Co., where Callista Gingrich had a revolving account.

ABC News’ Russell Goldman contributed to this report.