The Trump administration had said it would not abide by Maduro's "meaningless" order, calling him the "former" president after recognizing opposition leader Juan Guaidó as interim president.
But the drawdown of staff comes amid fears that diplomats and their families were in danger.
The socialist leader responded by severing diplomatic ties with the U.S., ordering American diplomats out, and announcing Thursday that he would pull his own diplomats from the U.S. and close the Venezuelan embassy and consulates.
The initial American response was to dismiss Maduro outright, and Pompeo warned in a statement Wednesday that the U.S. "will take appropriate actions to hold accountable anyone who endangers the safety and security of our mission and its personnel."
But Maduro still controls the military and security forces in the country, and while Pompeo called on them to respect a "peaceful, democratic transition" to Guaidó's government, the Armed Forces reiterate their allegiance to him on Thursday.
Some American staff will remain in Caracas, although it was unclear how many.
The U.S. Embassy had between 100 and 150 personnel, with family members accompanying diplomats, too. Those families will now be pulled out, after staff were asked to keep their children home from school on Thursday and restrict their movements to certain neighborhoods.
The State Department has set up a task force to manage the ongoing crisis in Venezuela, an official told ABC News, with staff "monitoring the security situation in real time, 24 hours a day, seven days a week."