"He really was someone who saw himself as someone working in Washington but not being of Washington, and he has a very strong sense of what he likes to spend his time doing, and hobnobbing isn't high on the list. Hiking and jogging and reading are things that are pretty high on the list," Feder said.
Souter is also a history buff, specifically the history of the Constitution, as evident in his attempt to maintain a continuity of Supreme Court opinions.
"I think he was very interested in... what history could teach us about the rule of law," Roosevelt said.
Though a Republican, Souter deviated from other conservatives on many issues. His opinions on controversial topics like abortion and school prayer irritated some Republicans, many of whom think his nomination was one of the biggest presidential blunders in modern history.
But some who knew him say he did not come in with an agenda and did not care to push decisions in a particular direction. Instead, he came in with an open mind and looked to past cases and the existing law to come to his decision.
"I think he's not necessarily that far from the first President Bush. He's pretty far from the second President Bush," Roosevelt said. "He's a Republican from 30 years ago, and the Republican party now has moved considerably to the right. He doesn't look like a modern Republican, he's not a modern person in a lot of ways."
Probably the biggest case of Souter's 19-year term in the Supreme Court is the abortion case of 1992, Planned Parenthood v. Casey, in which Souter formed part of the trio that kept basic rights to abortion part of American's constitutional rights.
At the time, Souter wrote that the controversial Roe v. Wade decision should not be overturned because it would be "a surrender to political pressure... So to overrule under fire in the absence of the most compelling reason to re-examine a watershed decision would subvert the Court's legitimacy beyond any serious question."
Roosevelt likens the controversial case, in which Souter sided with justices to uphold the abortion made legal in Roe v. Wade, to the Battle of Gettysburg.
"At the time, most people expected that the recent appointments to the court would result in Roe being gutted or overruled," Feder said. "He wrote the part of the opinion stressing how important it was for the court to respect prior decisions and not let major constitutional principles swing back and forth with every new appointment. Aside from the obvious importance of the opinion, it captures a fundamental part of his approach to judging."
Experts say Souter will also be remembered for his opinions on freedom of religion cases. In 1992, in the first major school prayer case looked at by the Supreme Court, Souter voted against allowing prayer at a high school graduation ceremony.
"He's a deeply religious person himself, but has regularly voted to keep government out of religious matters," Feder remembered.
He is also likely to be remembered for cases such Bush v. Gore, in which he dissented from the decision to stop the Florida recount in the 2000 presidential election, and was deeply disillusioned by the court's decision to end the recount, which essentially gave George W. Bush an opening to the White House.
Many recall him as a justice for whom law triumphed over ideology, and say his open mindedness and lack of predictability will be missed.
"He is first of all probably the most decent man I've ever really had dealings with, the best boss anyone could ever hope to have," Feder said. "He is the model of the way a judge should approach things."
ABC News' Ariane de Vogue contributed to this report.