The local NAACP chapter in Rep. Allen West's South Florida district is distancing itself from its congressman following West's accusation earlier this month that 81 House Democrats are Communists.

The Martin County, Fla., branch of the National Association for the Advancement of Colored People revoked their invitation for West to be the keynote speaker at the group's annual fundraiser last Saturday, instead rescheduling the event, with a new speaker, for September.

"We don't want to look like we support that statement he made," the chapter's President Rev. Jerry Gore told ABC News.

In response to a question at a town hall earlier this month, West said he believes "there's about 78 to 81 members of the Democratic Party that are members of the Communist Party."

He doubled down on the accusation last week, saying he does not regret the statement "whatsoever."

"I'm not going to back down," West said at a press conference last Tuesday. "I'm not going to be afraid about the fact that I called a spade a spade."

Gore said that while he supports many of the things West does as a member of the Congress, as a non-partisan organization the Martin County NAACP "don't want nothing to do with" West's "communist" comments.

"I don't want to be a part of that politics stuff," Gore said. "I don't say things against people like that. We just don't support that kind of stuff."

West's Chief of Staff Jonathan Blyth said the Congressman saw the rescinded invite as a "missed opportunity."

"While the NAACP of Martin County is certainly entitled to its decision, Congressman West believes this was a missed opportunity for a serious exchange of ideas," Blyth said in an email.

Gore said his group has not yet chosen a new keynote speaker for their fundraiser, which will now take place September 15, as the Scripps Treasure Coast Newspapers first reported.

This post was updated at 2:12 PM to include a comment from Rep. West.