But, don't fret, fans -- a version of Colbert's portrait will be at the Smithsonian through April 19 in honor of the host. It goes up at 4 p.m. today.
The image will be of Colbert posing in front of his own portrait and will reside in the National Portrait Gallery, the Smithsonian reported on the museum website.
"The portrait, updated every so often with versions of the previous portraits in the background, will occupy prime placement in the museum's 'historic alcove' -- just above the water fountains between the men's and women's restrooms on the second floor," the statement read.
The special hanging dates back to 2008, when Colbert interviewed the director of the museum and convinced him to hang his portrait. It hung for a while back then, until it was moved to the National Museum of American History and came down in 2011.
"A passerby was quoted as saying, 'Colbert is essential to our existence, just like restrooms,'" the Smithsonian joked on its website.
The rest of the Smithsonian statement read, "The new portrait contains at least twice as many Stephens as the previous image did. 'In recognition of the end of Stephen Colbert’s decade-long persona for Comedy Central’s 'The Colbert Report,' the museum has borrowed Colbert’s portrait, which was created for the final season of the show.'"