"I don't think I'll ever get tired of seeing my house on television while I'm in my house watching television," Letterman said on a March 2005 show.
Letterman was a longtime bachelor after his first marriage ended in divorce in 1977.
He has had at least one relationship with a co-worker in the past. Merrill Markoe, a comedian and author who was his longtime writer, was his girlfriend in the 1980s.
Even so, Letterman has joked about others' infidelities. Numerous jokes and at least two of Letterman's famous Top 10 lists highlighted South Carolina Gov. Rick Sanford's extramarital affair. Letterman included the "Top 10 Surprising Facts About Mark Sanford" and the "Top 10 Gov. Mark Sanford Excuses" during the days after the governor admitted to having an affair this June.
Letterman has been a fixture on late night network television since 1982 after a short stint with a morning show on NBC. As host of "Late Night" on NBC and then "The Late Show" on CBS, he is second only to Johnny Carson as the longest-running late night host.
ABC News' Eileen Murphy and Emily Friedman contributed to this report.