Mark Dunaway told ABC News that he's flown that black flag for the 10 years he’s lived in Garwood, New Jersey.
“I’m Muslim, and I fly a flag in front of my home that says I’m a Muslim,” he said.
However, after the Garwood Police Department received a complaint about the flag, officers visited Dunaway's home on Tuesday.
"The Garwood Police follow up with any complaint received," Police Chief Bruce Underhill said in a statement to ABC News.
“Police came by that day on a matter of safety,” Dunaway said, “I had no idea until they pointed it out to me. My reaction was, ‘Are you serious?’”
"Mr. Dunaway was very receptive when we approached him with our concerns and he voluntarily took the flag down," said Chief Underhill.
Dunaway, surprised at the complaint, realized the extent of the controversy when he saw the photo of his home posted on Twitter.
“It totally caught me off guard that someone was offended to that extent,” Dunaway said.
Marc Leibowitz, who posted the photo to Twitter, told ABC News that he was sent the photo by a friend and alerted Homeland Security.
Leibowitz said he doubted a member of ISIS would openly fly the flag, but that the situation was "disturbing and worth looking into," and that he "thought Homeland Security and any relevant authorities should probably be notified."
Ibrahim Hooper, a spokesman for the Council on American-Islamic Relations, told ABC News the flag is different from the ISIS flag and its message is something most Muslims are familiar with.
“The flag is a profession of Muslim faith,” Hooper said.
“Every Muslim in America has this phrase somewhere in their home,” Hopper added. “This man just chose to put it on a flag.”
“This is not the ISIS flag,” said Hooper, noting that the ISIS flag has an additional phrase on the bottom that makes reference to the "Islamic State.”
Hooper added that ISIS is a relatively new organization and Dunaway's flag flying predates it.
“It got totally taken out of context,” said Dunaway, “I am not affiliated with any type of militant group. It was just my way of expressing my religion.”
After hearing Dunaway's explanation, Leibowitz acknowledged Dunaway's First Amendment right, saying, "I don’t think he should be restricted from flying the flag, but I think it is a breach of good taste."
Some residents continue to harbor negative feelings.
"There have been a few threats to damage Mr. Dunaway’s residence on various social media outlets," said Chief Underhill. "This is unacceptable."
Dunaway said he is Muslim, but also "American-born and -raised" and did not mean to offend anyone.