Troubled transition: Presidential historian says 'completely unprecedented' in modern times

Washington DC -- White House historian Kate Andersen Brower describes President Donald Trump's refusal to say if he will attend the inauguration of President-elect Joe Biden as "a willful abandonment of his job. In my view, this is a real reckless decision if he doesn't show up."

Brower tells ABC's Powerhouse Politics podcast hosts Jon Karl and Rick Klein that one has to go back to the 19th century to find three presidents that boycotted their successors' inaugurations: John Adams, John Quincy Adams, and Andrew Johnson.

"It's completely unprecedented in modern times," Brower says.

The former White House reporter, who wrote the bestselling book, "Team of Five: The Presidents Club in the Age of Trump," discusses how many traditions are in jeopardy with a president who refuses to concede, calling into question a smooth transition of power.

On Inauguration Day, the outgoing president and first lady usually welcome the incoming president and first lady with a tea at the White House, an Oval Office meeting, and a shared ride from 1600 Pennsylvania Avenue up to the Capitol for the ceremony.

Brower says she doesn't envision any of those traditions happening with a Trump departure. And ironically his political team could blame the pandemic.

"No one would expect him to ride in the limo with Biden on the way to the swearing in anyway. Right? Because of COVID," she says. "So there are some things that he could get out of doing just because of the virus."

Brower, however, says she feels badly for the White House staff and the "controlled chaos" that describes the stress of moving one first family out, while moving the other one in. The frenzied process is further exacerbated by a disgruntled departing president and the pandemic.

"He's going to make this job a lot harder for about 95 resident staffers who move one president and the other out. And these are you know, these are ushers, butlers, housekeepers. It's kind of an all hands-on deck situation," Brower says. The White House will also have to undergo a deep-cleaning process which requires even more work and stress.

Brower has her eye on a man most Americans have never heard of: the chief White House usher, Timothy Harleth.

"He was hired by the Trumps. He worked at Trump International Hotel. And if he is doing his job, he would be coordinating this move and that would be, you know, flat out going against his boss's wishes because he isn't conceding yet. So I think he's in an impossible situation and we've never seen this before."

What about the traditional letter that some outgoing presidents leave in the top drawer of the Resolute Desk with a patriotic message or a piece of advice from one occupant of the Oval Office to another? In 2016, President Barack Obama reminded Trump that they were both "guardians of those democratic institutions and traditions -- like rule of law, separation of powers, equal protection and civil liberties."

Trump has mentioned how moved he was by Obama's gesture, but Brower admits she has no idea what the outgoing president will do.

"Even if he didn't show up to the inauguration ... he could leave a letter in the drawer of the resolute desk for Biden. God knows what it would say, but I think it would be fascinating."

She's also fascinated by the actual White House building itself, and has expanded her bestselling book audience to include pre-teen children. Her new book is called "Exploring the White House: Inside America's Most Famous Home."

The children's book, complete with maps and photos, describes an underground complex that most White House visitors never see on the two ground floor levels of 1600 Pennsylvania Avenue. The pages detail everything from a flower shop to employee lockers to a pastry chef area.

"It is kind of like this Downton Abbey at the White House. And the book is meant to honor the work too of the African American butlers who during segregation would not be able to sit where they wanted to on the bus, but yet were working in the most famous house in the country, and knew the president well. And the fact that the White House was indeed built by slaves."

"There's something very aspirational and patriotic and exciting about the building itself," Brower adds.