That's due to factors outside of the policy debates in Congress. The parties themselves have become more ideologically rigid. Conservative lawmakers, especially from the south, began to disappear from the Democratic Party during the 1960s, when the party adopted a pro-civil rights stance. And the GOP lost many of its moderate northeastern and midwestern members soon afterward as southern Democrats migrated to the GOP.
Congressional districts have also become more homogenous along racial and ideological lines, in part because of gerrymandering but also because of natural population shifts. That encourages House members to vote on a more partisan basis in order to avoid constituent backlash or a primary challenge.
And interest groups like the National Rifle Association (NRA) have increasingly "scored" key votes to hold members' feet to the fire, according to Baker.
"You combine all of that, and you've really got the makings of some pretty impressive polarization," he said.
Norman Ornstein of the American Enterprise Institute and Thomas Mann of the Brookings Institution recently co-authored a book that argues Republicans in particular have taken a more radical approach that prioritizes blocking Obama's agenda rather than negotiating on major legislation. Complicating the problem, Obama has also not shown an adeptness for working the inside game of Congress.
Ornstein and Mann wrote this month in a Washington Post op-ed that despite progress on certain issues like immigration, those fundamentals remain unchanged.
"Yes, there are signs of progress in our political system," they wrote. "The universe of problem-solvers in the Senate has increased since the 2012 elections. But the broader pathologies in our politics remain. For all the problems that existed in previous decades, in a system designed not to act with dispatch, there was a strong political center, with responsible bipartisan leadership. The same cannot be said today."
Baker also said that the immigration debate itself could become a lot more partisan than it appears now.
"The opponents of a path to citizenship haven't given up," he said. "My feeling about comprehensive immigration reform is that it remains to be seen."
He has a point. Even when an issue has bipartisan support in opinion polls it doesn't mean that Congress will act to pass it. Take the failed Manchin-Toomey amendment. Nine in ten Americans support expanded background checks, like the legislation proposed. But Republicans senators and a handful of Democrats from rural states were discouraged to vote for it, particularly by the NRA and vocal constituents. As a result, supporters couldn't find 60 votes to break a GOP filibuster.
"[A bipartisan budget deal], along with an immigration overhaul and gun overhaul, should be in the no-brainer category," Ornstein wrote in Roll Call in March. "But the odds may still be with the no-brainers who want to thwart compromise and push confrontation."