Inside the marriages and family life of Hugh Hefner

— -- Later in life, Hugh Hefner may have become know for such reality shows as "Girls Next Door," where crews filmed his interactions with three girlfriends or more at a time.

But the publishing mogul, who died Wednesday at the age of 91 from natural causes, was only married a handful of times -- despite his many girlfriends and polyamorous ways -- and was also very close to his four children, some of whom ran the business after he started to step away.

Here are a the key people who shaped the icon's life.

Tying the Knot

Hefner married his first wife Millie Williams in 1949 and the duo would remain together for about a decade. The two welcomed daughter Christie and son David in their time together.

Hefner's relationship with his first wife was not like what readers of the magazines and watchers of his shows saw in his later years. Williams was the same age as Hefner and the two got together years before the Playboy empire was founded.

In fact, the two were childhood sweethearts, but while Hefner was away in the Army, Williams had an affair, something Hefner later called "the single most devastating experience of my life," according to the New York Times.

After their divorce in 1959, Hefner wouldn't tie the knot again until 1989 to playmate Kimberly Conrad.

And while Hefner definitely promoted his life in the quintessential "Playboy" way prior to tying the knot again, this marriage had a very similar ending to his first.

Conrad and Hefner welcomed two more children, but the duo split in the late 90's and eventually divorced about a decade later.

"During the marriage I was faithful,” he told the Times in 2011. “She was not."

Hefner said he married Conrad after he had dealt with some major health issues and was feeling his age.

Hefner's final wife at the time of his death was Crystal Harris, who he started dating in 2009 and married three years later. The marriage made headlines as he was 86 and she was 26.

Hef at Home

Though his relationships and marriages may have almost always been in flux, Hefner was very supportive of his children, specifically his first-born daughter Christine, who eventually took over the business.

In 1982, she became president of Playboy Enterprises and later become CEO in 1988. She remained in the position until 2009.

Christine eventually went into philanthropy and created the Hugh H. Hefner First Amendment Award in honor of the man who raised her.

Hugh Hefner through the years

1 of 24

American publisher Hugh Hefner is seen in this 1955 portrait sitting in front of a wall collage of female centerfolds from his men's magazine 'Playboy' which he launched in 1953.

Getty Images

In 1977, Hefner spoke to ABC News about bringing his young daughter into the fold at the magazine and expressed his gratitude for the their "very close relationship.

Hefner said he and his daughter forged a bond out of his first divorce.

"When you have that kind of problem, it very much is in the hands of what kind of relationship is sustained between the former husband and wife," he said.

But Christine, now 64, wasn't the only of Hefner's children to work at Playboy.

His youngest son Cooper, 26, was chief creative officer at Playboy Enterprises and before that was the CEO of upstart media company HOP, according to Business Insider.

After news of his father's death, Cooper released a glowing statement in honor of the late icon.

"My father lived an exceptional and impactful life as a media and cultural pioneer and a leading voice behind some of the most significant social and cultural movements of our time in advocating free speech, civil rights and sexual freedom," he told ABC News. "He defined a lifestyle and ethos that lie at the heart of the Playboy brand, one of the most recognizable and enduring in history."

He added, "He will be greatly missed by many, including his wife Crystal, my sister Christie and my brothers David and Marston and all of us at Playboy Enterprises."