May 2, 2009— -- Supreme Court Justice David Souter may be little known outside his small circle of friends -- he shies away from the Washington social scene and prefers to be in his New Hampshire cabin reading books -- but those who know him or work with him say Souter is a funny man who takes serious interest in the law and his work.
The 69-year-old associate justice never hid his dislike for Washington, but Souter is hardly a recluse.
"When you meet the guy you expect to meet some silent Cal Coolidge type of New Englander who talks very little and whatever else goes with that stereotype," said Meir Feder, one of Souter's clerks from the 1990 term and a partner at New York-based Jones Day. "Although he's someone who doesn't like to spend a lot of time out in high society, he is a really warm and engaging person, and in fact quite a talented story teller and sort of off-the-cuff speaker."
His clerks remember him as one who took their ideas seriously and did not impose his higher stature on those below him.
"I was not expecting him to be quite as warm and charming as he turned out to be," said Kermit Roosevelt, who clerked for Souter in 1999 and 2000, and is now a professor at University of Pennsylvania's law school. "His persona was very laid back, unpretentious. You could see this in the discussions he had with the clerks about the cases. He was interested in hearing what we had to say and didn't give us any sense of hierarchy."
Souter's tenure on the court was marked by a willingness to vote outside political party lines, marking him as an independent thinker.
President Obama also reflected on that on Friday, announcing Souter's retirement and calling the justice a "fair minded and independent" judge who combined a "feverish work ethic" with good sense of humor and integrity.
"It's never been a secret he doesn't like Washington. ... There was a lot of artificiality in relationships. People relate to the position rather than the person and that's completely the opposite of he was," Roosevelt said.
Souter has been unabashedly vocal about his dislike of Washington and his desire to return home to New Hampshire.
"Of course, we have all known that his deep love for New Hampshire would take him away from the court some day," Sen. Patrick Leahy, D-Vt., said in a statement Friday.
Although he rarely gives speeches in Washington, Souter raised eyebrows this spring when he made a joke about the fatigue from his work load.
"I find the workload of what I do sufficiently great that when the term of court starts I undergo a sort of annual intellectual lobotomy," he said.
The never-married Souter is an avid reader. He has told his friends for years he wants to sit by the fire in his New Hampshire draft cabin and read his books.
"He was a classic frugal Yankee Republican. He's been known to reheat yesterday's coffee in the microwave," recalled Rebecca Tushnet, who was one of Souter's clerks in 1999 and 2000 and is now a law professor at Georgetown University.
Souter was born in Massachusetts, but spent most of his life in the rural town of Weare, N.H. A graduate of Harvard and Oxford, he climbed up the ranks of the legal world from assistant attorney general of New Hampshire to an associate judge in the state's Supreme Court.
When Souter was plucked out of New Hampshire by President George H.W. Bush in 1990, he was little known outside of the state.
"He really was someone who saw himself as someone working in Washington but not being of Washington, and he has a very strong sense of what he likes to spend his time doing, and hobnobbing isn't high on the list. Hiking and jogging and reading are things that are pretty high on the list," Feder said.
Souter is also a history buff, specifically the history of the Constitution, as evident in his attempt to maintain a continuity of Supreme Court opinions.
"I think he was very interested in... what history could teach us about the rule of law," Roosevelt said.
Though a Republican, Souter deviated from other conservatives on many issues. His opinions on controversial topics like abortion and school prayer irritated some Republicans, many of whom think his nomination was one of the biggest presidential blunders in modern history.
But some who knew him say he did not come in with an agenda and did not care to push decisions in a particular direction. Instead, he came in with an open mind and looked to past cases and the existing law to come to his decision.
"I think he's not necessarily that far from the first President Bush. He's pretty far from the second President Bush," Roosevelt said. "He's a Republican from 30 years ago, and the Republican party now has moved considerably to the right. He doesn't look like a modern Republican, he's not a modern person in a lot of ways."
Probably the biggest case of Souter's 19-year term in the Supreme Court is the abortion case of 1992, Planned Parenthood v. Casey, in which Souter formed part of the trio that kept basic rights to abortion part of American's constitutional rights.
At the time, Souter wrote that the controversial Roe v. Wade decision should not be overturned because it would be "a surrender to political pressure... So to overrule under fire in the absence of the most compelling reason to re-examine a watershed decision would subvert the Court's legitimacy beyond any serious question."
Roosevelt likens the controversial case, in which Souter sided with justices to uphold the abortion made legal in Roe v. Wade, to the Battle of Gettysburg.
"At the time, most people expected that the recent appointments to the court would result in Roe being gutted or overruled," Feder said. "He wrote the part of the opinion stressing how important it was for the court to respect prior decisions and not let major constitutional principles swing back and forth with every new appointment. Aside from the obvious importance of the opinion, it captures a fundamental part of his approach to judging."
Experts say Souter will also be remembered for his opinions on freedom of religion cases. In 1992, in the first major school prayer case looked at by the Supreme Court, Souter voted against allowing prayer at a high school graduation ceremony.
"He's a deeply religious person himself, but has regularly voted to keep government out of religious matters," Feder remembered.
He is also likely to be remembered for cases such Bush v. Gore, in which he dissented from the decision to stop the Florida recount in the 2000 presidential election, and was deeply disillusioned by the court's decision to end the recount, which essentially gave George W. Bush an opening to the White House.
Many recall him as a justice for whom law triumphed over ideology, and say his open mindedness and lack of predictability will be missed.
"He is first of all probably the most decent man I've ever really had dealings with, the best boss anyone could ever hope to have," Feder said. "He is the model of the way a judge should approach things."
ABC News' Ariane de Vogue contributed to this report.