He had spent 14 years as a congressman, eagerly sitting in the House chamber and bringing family members and constituents along.
But Richardson didn't end up making it.
"I thought I'd done something wrong," Richardson told ABC News ahead of the State of the Union in 2018.
The designated survivor is typically a Cabinet-level official who is chosen to skip certain high-profile events -- generally limited to State of the Union addresses and inaugurations -- so that he or she may assume power if a disaster were to occur while the majority of the government is gathered at the event.
This year's designated survivor has not yet been announced. Secretary of Agriculture Sonny Perdue was picked as the designated survivor in 2018.
For the inauguration, Johnson noted how the designated survivor is "serving two administrations," and in his case, he had to withdraw his resignation and make it contingent on the confirmation of his successor.
"I went from being secretary of homeland security with Secret Service protection and the designated survivor, fourth in line to the presidency, to being a private citizen," Johnson told ABC News.
The transition out of the role was a bit less staggering for Richardson in 2000.
Richardson had been planning to spend the weekend with friends in Oxford, Maryland, so when he had to miss the State of the Union on Thursday, Jan. 27, 2000, he and his wife just moved up their trip a few days. He said the small town of Oxford fell within the radius that the Secret Service would allow, wanting the designated survivor to not be in, but not be far from, Washington, D.C. But rather than keep a completely low profile, the buses of security and medical personnel that were required to follow Richardson drew the attention of the townspeople, he said.
"The whole town must have thought there was a national emergency or a fire," Richardson said.
"It caused quite a ruckus," he said.
Once then-President Bill Clinton had returned safely to the White House, Richardson said the Secret Service agents came up to his friend's house, giving the all clear and saying that the security detail would be heading back to Washington.
"I had a glass of wine, maybe two" after they left, Richardson said, noting that he "was pretty alert" during the speech itself.