Former VA Secretary David Shulkin speaks out about what he calls 'shadow government' under Trump

Former Veterans Affairs Secretary David Shulkin joined ABC News' "The Briefing Room" on Friday, expressing concern over a "shadow government" within the Trump administration that, he said, made it more difficult to do his job.

The former cabinet official said political appointees in place during his tenure at the VA often communicated directly with the president, simply going around him. He also said the administration "orchestrated" the plan to oust him.

"You serve at their pleasure, and the president should have the team around him or her that allows them to feel comfortable and to get the advice they want, there's no issue with that," he told ABC News Deputy Political Director MaryAlice Parks. "The issue that I've raised in the book and just laid out the facts -- for people to decide -- is this was being orchestrated by a small number of political appointees who weren't elected and weren't put in place to decide who should be secretary."

When asked about the current administration, Shulkin added, "I think that President Trump is struggling right now."

Shulkin served as VA secretary from February 2017 until his departure in March 2018, after an ethics investigation and alleged misuse of taxpayer dollars.

He said they created the situation by leaking and creating "false information" that had an influence on the other members of government.

In his new book, "It Shouldn't Be This Hard to Serve Your Country: Our Broken Government and the Plight of Veterans," Shulkin describes the nation's capital as "toxic, chaotic and subversive."

When Parks asked if the book, in part, was a way for him to settle the score after he said a "shadow government" of political appointees worked to undermine him, Shulkin said it wasn't.

"I wrote the book because I feel so strongly that our veterans deserve the very best care and services that this country can offer," Shulkin said. "I had found a formula, I think, for working within government to make that better and I wanted to share what was working and what wasn't."

Shulkin's advice? More Americans need to volunteer to serve and help make the government better, including helping to protect whistleblowers.

A report published Thursday by the VA inspector general showed the VA Office of Accountability and Whistleblower Protection failed to meet its stated goals under the Whistleblower Protection Act. The office, specifically created by President Donald Trump to clean up the challenged agency, didn't meet its objectives and created circumstances that could put whistleblowers at further risk for retaliation.

"In its first two years of operation, the OAWP acted in ways that were inconsistent with its statutory authority while it simultaneously floundered in its mission to protect whistleblowers," the report said.

Shulkin said he agreed it was tough for people to speak out even when he was in office, but that it wasn't his intention.

"Well, I think that there's no doubt that this has been an environment that has been tough for many people who have tried to speak out. When I was secretary we passed through Congress, and the president signed, the Accountability and Whistleblower Protection Act," Shulkin added. "We were enacting new legislation to be able to make sure that people felt comfortable speaking out when they saw wrongdoing and that certainly was not the intention or an environment that I supported where there was retaliation."

He expressed concern over partisanship preventing what he feels should be a bipartisan effort.

"Unfortunately, I think that this pattern of behavior, the experience that I had, which I think took away from the ability to focus, in this case on veterans, is not, unfortunately, a unique experience," he said. "I've seen the same thing happen to many other people who have come to serve, many dedicated career professionals, who, trying to do the job and there for the right reason because they believe in government and they believe in their country, being prevented from doing their job every day."

Shulkin said Trump and "others in Washington" should take a lesson on service from veterans.

"Veterans are about serving and putting country first and I think we all have a lot to learn from them," he said. "I think that this is a country that is divided, and it's hard to be a leader of a divided country, and I would hope that he would see that this is the opportunity to really lead in a very different direction."

ABC News' Stephanie Ebbs contributed to this report.