"I think a lot of the Americans that get caught up in this are decent people but they've allowed themselves to be manipulated into believing that they're enemies and that they should hate each other, and that they should try to destroy each other, and this is exactly the wrong thing," Carson told ABC News' "Powerhouse Politics" podcast.
"We need to start thinking more about the things that benefit all of us and get away from this partisanship," he added.
In August, Carson faced what he called a "symptom" of that partisanship when his appearance -- and introduction as secretary of housing and urban development -- at a Trump campaign rally raised the suggestion that he violated the Hatch Act, a law preventing some executive branch officials from participating in political affairs in their capacity as government employees.
The Office of Special Counsel ultimately concluded there was no wrongdoing on the secretary's part becausd he personally "did not refer to his official title or use his official position during the speech," according to a letter from OSC shared in a tweet by former Deputy Secretary of Labor Chris Lu, who filed the complaint.
Carson also said that critiques of the Trump campaign's hosting of the event were misguided and bound to occur no matter how it was funded.
"The purpose of the Trump campaign paying for it though is so that all of that doesn't go on the taxpayer's bill. He's trying to save the taxpayers money," Carson said, arguing that if the rally was held as a White House event and paid for "out of the [government's] coffers, then there would be complaints about that as well."
The secretary, who spent 30 years as director of pediatric neurosurgery at the Johns Hopkins Children's Center in Baltimore, thinks a party-blind ethos can be applied to the ongoing debate over health care.
"If we go back and we just ask ourselves: What are we trying to accomplish? And then create something around that, that is centered primarily around patients and health care providers, I think we would do ourselves a great favor," Carson said, expressing optimism about finding common ground with Democrats on the issue. "I don't think that's a partisan issue."
Carson, the only African-American member of Trump's Cabinet, explained that he understood why some feel disrespected and "threatened" by statues of Confederate leaders around the country.
The status of a monument featuring Confederate Gen. Robert E. Lee prompted the white nationalist rally that sparked the violence in Charlottesville.
He further echoed a stance that has been offered by Trump, questioning the value of erasing history by removing such statues. Any solution, Carson said, should come from the local level.
"That's why we have 50 states. That's why we have multiple communities," he said. "People have freedom to talk about how things should be done and how they want them remembered.”