New Yorkers told ABCNews.com that the idea of a mosque is not the problem, but the Cordoba Initiatve is just too close to the city's most painful scar and has struck many as insensitive and cruel.
Robert Engel's office on Church Street, would look right into the proposed center. It's not a prospect he or many of his co-workers are looking forward to, he said.
"I don't think they should do it. It's too close," Engel said. "It's a slap in the face."
While the Cordoba's plans have become a lightning rod for politicians from Buffalo to Alaska, a blink-and-you'll-miss-it mosque a few blocks away draws worshipers instead of critics.
The Web site of the Masjid Manhattan even includes a disclaimer that it is not affiliated with the Cordoba plans.
"Please be advised that we are by no means affiliated with any other organization trying to build anything new in the area of downtown Manhattan."
"Our members are city, state and federal employees, as well as professional employees of the financial area who come to our Masjid to perform their daily prayers," the statement continued. "Masjid Manhattan and its members condemn any type of terrorist acts."
For more than an hour Monday, men wearing everything from neatly pressed suits and corporate ID cards to jeans and T-shirts came to pray at Masjid Manhattan. Three women also quietly made their way in, slipping silently into the women's section separated by two thick curtains.
The mosque has operated in the neighborhood for years, moving in 2008 to its cramped basement space after their previous building was sold in 2008 and they were forced out.
Engel, 36, also said that he's never had any problem with Masjid Manhattan, often passing worshippers on his way to eat in the restaurant above their prayer space.
"I'm fine with them there," he said. "I just think that they shouldn't build one too close to the World Trade Center."
Yanna Agoureev, 46, who has lived near Ground Zero for five years, said she isn't offended by the plans for the Cordoba mosque, but fears trouble for a long time to come if it's built, predicting protestors and clashes with Muslims.
"I'm basically okay with it," she said. "But if it were somewhere else, it would be a bit better for everybody."
Last week, the Anti-Defamation League raised eyebrows when it seemingly strayed from its longstanding reputation for interfaith tolerance, and issued a statement condemning plans for the mosque.
"The controversy which has emerged regarding the building of an Islamic Center at this location is counterproductive to the healing process," the ADL statement read. "Therefore, under these unique circumstances, we believe the City of New York would be better served if an alternative location could be found."
The controversy over the building of the Cordoba Initiative has been furious from the start and has morphed into a political debate that has ensnared Sarah Palin and one of New York's Republican gubernatorial candidates.
"They should just move this thing," Buffalo Republican Carol Paladino told the Wall Street Journal over the weekend. "The vast majority of New Yorkers and Americans have rejected their idea."
Palin earned the ire of Bloomberg and became a late-night punchline last month when she took to Twitter to call on New Yorkers to "refudiate" plans for the mosque, calling it "unnecessary provocation."
Bloomberg quickly shot back at Palin, saying that "everything the United States stands for and New York stands for is tolerance and openness."