An aggravated harassment complaint has been filed, but no arrests have been made, police added.
"The Confederate flag is a symbol of hate. This week it was paraded through the Capitol and this morning it was used to vandalize the Museum of Jewish Heritage in our city," Mayor Bill de Blasio tweeted.
"We have ZERO tolerance for acts of fascism and bigotry," the mayor added, vowing that "there will be consequences for those responsible."
"This is an atrocious act on our community and on our institution and must be met with swift and forceful response from law enforcement," Jack Kliger, the president and CEO of the museum, said in a statement. "The Confederate flag is a potent symbol of white supremacy, as evidenced by the events at the U.S. Capitol this week. Such hate has now arrived at our doorstep."
This incident comes in the wake of Wednesday's violent siege in Washington, during which supporters of President Donald Trump -- some brandishing Confederate flags and wearing anti-Semitic clothing -- stormed the U.S. Capitol.
The violence claimed at least five lives, including a Capitol Police officer who died Thursday following injuries sustained in the riot, and a 35-year-old woman who was shot and killed by Capitol Police while trying to enter a broken window into the House Chamber.