Emanuel said that he asked for McCarthy's formal resignation this morning.
"To build the trust and confidence of the public and, at this point, in this juncture for the city, given what we're working on, he had become an issue rather than dealing with the issue and a distraction," Emanuel said.
Deputy Superintendent John Escalante will step in to fill McCarthy's place for the time being, Emanuel said. The police board will be tasked with finding a permanent superintendent, he said.
Calls were made for McCarthy's resignation in the wake of the release of dash cam footage showing a police officer shooting a teen in October 2014. The footage was not made public until last week, after a court order.
McCarthy appeared on ABC's Chicago station WLS earlier today saying that he would not resign.
"I'm doing the best job I can do, let's put it that way," he said this morning.
McCarthy was hired as the police department's superintendent in May 2011, and previously served as the chief of police in Newark, New Jersey, and spent years working for the New York City Police Department as well.