The former Masters champion is the second South African in the past week to pull out of this summer's Rio de Janeiro Games. Louis Oosthuizen previously cited family and scheduling issues.
Nine-time major winner Gary Player, South Africa's Olympics golf captain, said in a tweet earlier Monday that he was "sad and disappointed" that several of his top players had opted not to represent South Africa in the Games.
"I was sorry to hear that Louis Oosthuizen and Charl Schwartzel have withdrawn their names from consideration," Player wrote. "They have been playing outstanding golf lately. This certainly impacts South Africa's chances to win a gold medal. No doubt they would have made our team even stronger."
PGA Tour spokesman Ty Votaw says Schwartzel informed golf officials last week. Schwartzel was a strong candidate to make the team once Oosthuizen withdrew. That means Jaco Van Zyl would be next in line for South Africa.
Information from The Associated Press was used in this report.