And his precarious relationship with the state's Jewish community became a thorny issue for Jason Carter, whose district contains a sizeable Jewish population. His predecessor, David Adelman, who left his post to become ambassador to Singapore, was also Jewish.
Carter, the son of Jimmy's and Rosalynn's eldest son, Jack, took special note in his campaign to reach out to Jewish communities.
"Ultimately, I spent a lot of time reaching out to the Jewish community in my district and to other communities to make sure that I could build trust between me, Jason Carter and those communities as best as I could and I think I did that," he said.
"I also spent time talking about Israel in particular and what a good state senator in the Georgia Senate can do in regard to Israel, which is not much but it's also not nothing."
Carter, who won the special election to replace Adelman, is guaranteed to keep the state Senate seat for at least the next two years. He is unopposed for the Democratic primary in July and the November race for that Senate seat is likely to be uncontested.
For now, the married father of two little boys said he wants to focus on the issues facing Georgia, specifically concerning education, transportation and water planning, and he has no political ambitions to go to Washington.
"My goal is to be a really good state senator," he said. "I have no interest in going to Washington. I don't want to move."
Carter denounced the political posturing by some states such as Arizona, which passed a controversial immigration law last month, but also said that in some arenas, such as education, states cannot wait for the federal government to fix the system.
Despite the heavy partisanship in the GOP-controlled state legislature, Carter is hopeful he can reach across the political aisle and find common ground with his Republican colleagues.
"Around my district, people are really frustrated with the system and ... the partisanship and the sort of the gridlock that seems to have taken over in Washington," he said. "I think Georgia has some of the same partisanship issues but I think that we should be able to find some common ground."