Engel, 36, also said that he's never had any problem with Masjid Manhattan, often passing worshippers on his way to eat in the restaurant above their prayer space.
"I'm fine with them there," he said. "I just think that they shouldn't build one too close to the World Trade Center."
The center, which has been called everything from an insult to a "house of evil" by protestors, has already been through several approvals, including the a community financial district and an advisory board.
Yanna Agoureev, 46, who has lived near Ground Zero for five years, said she isn't offended by the plans for the Cordoba mosque, but fears trouble for a long time to come if it's built, predicting protestors and clashes with Muslims.
"I'm basically okay with it," she said. "But if it were somewhere else, it would be a bit better for everybody."
Last week, the Anti-Defamation League raised eyebrows when it seemingly strayed from its longstanding reputation for interfaith tolerance, and issued a statement condemning plans for the mosque.
"The controversy which has emerged regarding the building of an Islamic Center at this location is counterproductive to the healing process," the ADL statement read. "Therefore, under these unique circumstances, we believe the City of New York would be better served if an alternative location could be found."
The mosque is part of a planned 13-story community center that will also house recreational facilities and space for cultural events.
The controversy over the building of the Cordoba Initiative has been heated and furious from the start and has morphed into a political debate that has ensnared Sarah Palin and one of New York's Republican gubernatorial candidates.
"They should just move this thing," Buffalo Republican Carol Paladino told the Wall Street Journal over the weekend. "The vast majority of New Yorkers and Americans have rejected their idea."
Palin earned the ire of New York City Mayor Michael Bloomberg and became a late-night punchline last month when she took to Twitter to call on New Yorkers to "refudiate" plans for the mosque, calling it "unnecessary provocation."
Bloomberg quickly shot back at Palin, saying that "everything the United States stands for and New York stands for is tolerance and openness."