But even some conservatives are saying that kind of talk is dangerous.
"Even if he was just joking, he was trivializing something that shouldn't be trivialized," said conservative columnist and beliefnet.com blogger Rod Dreher. "And, at a time at our history where people are really anxious, really afraid, this is like throwing a bomb."
The incident comes as debate continues about South Carolina Rep. Joe Wilson's "you lie" outburst, with many speculating about the GOP congressman's motives for interrupting the president during a congressional address. It also follows an August filled with boisterous, sometimes angry, town hall meetings about proposed health care overhaul.
"The mood on the right is fearful, anxious, people don't know where to turn," Dreher said. "They see this popular president and we have no one on our side to match him. We have no leadership, the only leadership we have, it seems, are people like Rush and [talk-radio and TV host] Glenn Beck and a lot of these talking heads whose stock and trade is being outrageous."
Beck has been vocal about the role of race in the political debate.
In July, Beck said on Fox News, "This president, I think, has exposed himself as a guy over and over and over again who has a deep-seated hatred for white people."
Comments like that have some people deeply concerned.
"We're at a dangerous time in this country right now," Dreher said. "Not only on the right, but also on the left. When people feel that their anger justifies anything ? I am afraid they are going to justify, in their own minds, doing anything. This is going to get ugly."
To be fair, any blatant racism is coming from a minority of conservatives.
While prominent conservative talk-show commentators like Limbaugh and Beck have touched the race issue directly, Republicans on Capitol Hill argue that efforts to portray criticism of Obama as being racially motivated will backfire.
GOP congressional leaders say the conservative uprising is driven not by race but by policies that "tax too much, borrow too much and spend too much."
Meanwhile, White House Press Secretary Robert Gibbs did all he could at a briefing Wednesday to downplay talk of race.
"The president does not believe that the criticism comes based on the color of his skin," Gibbs told reporters.
But Carter could not have been more clear about what he sees as driving the criticism.
"It's a racist attitude," Carter said Wednesday night. "And my hope is and my expectation is that, in the future, both Democratic leaders and Republican leaders will take the initiative in condemning that kind of unprecedented attack on the president of the United States."
Democratic political analyst and ABC consultant Donna Brazile said that kind of unified approach and a rational discussion of race in America are what's needed. The president, she said, doesn't have time for any more "beer summits."
"No one is discussing the legacy of racism or segregation," Brazile said. "Rather, this is a hollow conversation that is distracting us from the larger issues where we can find common ground, whether it's health care, the environment. I basically disagree with those who are starting to start up a conversation that is clearly a dead-end subject. We really need to have a conversation that allows us to move forward, not takes us back to a past where we wish we all could have done better."