ABC News’ Amy Walter and Michael Falcone report:

For Rick Perry, there’s no going back.

Whether he likes it or not, he is now the proud owner of a position on Social Security that both his Republican opponents and Democrats say is so radical, it will sink his chances of throwing President Obama out of office in 2012.  But it’s unclear whether his tough-talk will really hurt him that much with Republican primary voters.

Less than 24-hours after Perry called Social Security a “Ponzi scheme” at his debut presidential debate in California, declaring that “it’s time to have some provocative language in this country,” the Texas governor appeared to dial back his rhetoric.

Gone from Perry’s stump speech at a rally in Orange County on Thursday was any mention of the phrase “Ponzi scheme” or, for that matter, any reference to Social Security. Nevertheless, he emphasized that “we need to have a nominee that doesn’t blur the lines between themselves and the current resident of the White House.”

And while Perry may not have employed his usual sharp-tongued words about the entitlement program on Thursday, it was clear that for some Perry backers, “Ponzi scheme” might soon become a rallying cry.

Orange County Supervisor Shawn Nelson, a Perry supporter who fired up the crowd before the governor spoke, offered a defense of the Texan’s “colorful” and even “inflammatory” rhetoric. “Sometimes it helps prove your point,” Nelson said.

Some members of the audience even responded by shouting, “Ponzi scheme!”

When asked after the event whether she thought the words Perry uses to describe Social Security were over-the-top, Micki Blair, a teacher from Corona, Calif., said the governor’s characterization was just fine with her.

“I don’t think his rhetoric is too strong,” said Blair, who is leaning toward supporting Perry. “I think that people are mature, have a brain. They can figure out that if you don’t put enough in, you can’t pass it all out. It just doesn’t work that way.”

And, at least one GOP strategist interviewed by ABC News agreed that Social Security is not nearly as motivating an issue among Republican primary voters as it is among Democrats. The strategist, who has worked on many Republican primaries, doesn’t think it hurts Perry “at all” in the nominating process. Instead, it may actually serve to undermine Romney, this person said. Romney’s attacks on Perry’s Social Security position appeal to those Republicans interested in electability –usually the more moderate voters in the party.

Romney’s challenge right now is to appeal to those Republican voters on the right. The most recent ABC/Washington Post poll found that “Perry leads Romney among conservative leaned Republicans by 18 points, largely by dint of a wide 29-point Perry lead among those who describe themselves as ‘very’ conservative. Perry also leads Romney by 30 points, among ‘strong’supporters of the Tea Party political movement.”

And although Perry may have held his tongue on Thursday, his advisers say he is not going to back down.

“It is a Ponzi scheme,” Perry’s top strategist Dave Carney told reporters after this week’s debate at the Reagan Library.

“I think it’s naïve for the political elite to think that Social Security can’t be discussed, can’t be fixed, can’t be done better in this new modern era,” Carney said. He promised that the campaign would release its own plan for reforming the system, but declined to say when.

Perry came under a withering attack from aides to Romney who insist that the Texan’s Social Security stance makes him unelectable in a general election. The Romney campaign distributed a memo Thursday morning with an all-caps subject line: “RICK PERRY: RECKLESS, WRONG ON SOCIAL SECURITY.”

Romney, himself took to the airwaves on Thursday, telling conservative radio host Sean Hannity that “if we nominate someone who the Democrats could correctly characterize as being against Social Security we would be obliterated as a party.”

Some Romney backers suggest that Perry’s stance on Social Security could hurt him in a primary election too, especially in states where the GOP primary is dominated by older voters. In Florida, for example, 44 percent of the electorate in the 2008 Republican primary was 60 or older. In 2000, it was 54 percent.

Huffington Post  Pollster Mark Blumenthal points to a Pew Research Center Survey that showed that 87 percent of Republicans believe that Social Security has been either “very good” or “good” for the country. Yet, that same survey showed that just 37 percent of Republicans (and 39 percent of voters overall) thought that Social Security does an excellent or good job of serving the people in covers. Moreover, just 14 percent of Republicans (and 18 percent of voters overall), think that it’s in excellent or good financial condition.

In other words, attacking Social Security as something that was a bad idea to begin with is unpopular across the board. (In his 2010 book, “Fed Up!” Perry calls it “something we’ve been forced to accept for more than 70 years now.”) But, arguing that it’s not working and needs to be fixed is popular.

In response to the Romney team’s attacks, the Perry campaign followed up with its own missive, headlined, “Mitt Romney’s Social Insecurity,” noting that Romney has had some unkind things to say about the entitlement program too.

“Traditional political rhetoric and tap-dancing don’t comfort Americans deeply concerned about the future of our nation,” Perry’s communication director Ray Sullivan said in a statement accompanying the fact-sheet.

The Huntsman campaign sent around a similar memo on Thursday calling Romney and Perry “two peas in a pod” when it comes to their views on Social Security. And the liberal advocacy group, Americans United for Change, on Friday morning circulated a web video called, “Mitt Romney, Rick Perry, George W. Bush — 3 Amigos, Gunnin’ for your Social Security.”