(Credit: Jim Cole/AP Photo)

Sen. Bob Smith is fighting to return to the Senate, trying to regain his former title as U.S. Senator from New Hampshire, but he has to first beat another former U.S Senator, one from Massachusetts who is heavily favored in Tuesday's primary, Scott Brown.

Yes, two formers, one battle for the Senate, a fight against incumbent Democratic Sen Jeanne Shaheen that could determine if Democrats retain control. Brown and Shaheen have essentially been fighting a general election battle for months, facing off head to head and ignoring Brown's GOP opponents. In a WMUR University of New Hampshire poll from last month, Brown was only trailing Shaheen by two points, within the margin of error.

That hasn't deterred Smith, who has a 30-year political resume - with several non-traditional moments - but admits Tuesday is his last political campaign, no matter what happens.

"I doubt very much I would be a candidate again," Smith told ABC News. "Some people I might help probably, but not as an elected official."

He's confident, saying he still very much thinks he will be the nominee and doesn't like to think about the possibility of losing, but acknowledges that if he doesn't win, he will stay busy writing a book he is about one-fifth of the way done with. He says this race will be the book's "final chapter."

Smith was first elected to New Hampshire's 1st District House seat in 1984. In 1990, he ran for Senate and won, going on to serve two terms before losing in 2002 to then U.S. Rep. John Sununu.

His second Senate term was not without its share of headlines. In 1999, after a short primary bid for the GOP presidential nomination, he left the Republican Party and launched a long-shot White House campaign as a candidate for the U.S. Taxpayers Party. He had a subsequent presidential bid as an independent, which also was brief. That hurt him in his 2002 bid and he moved to Florida soon after his loss and mounted two bids for the GOP Senate nomination from the Sunshine State in 2004 and 2010, both unsuccessfully.

Now that he's back in New Hampshire, Smith is running as the "true conservative" in the race, but it's not only the fact that he left the GOP that has undermined his claim to that title: In 2004, just before Election Day, he endorsed John Kerry over George W. Bush.

Smith admitted that the endorsement was a "mistake made in anger" when Bush did not endorse him in his 2002 election, something he said Bush promised him.

Smith said most of the reaction he gets on the campaign trail, something he calls the "Dunkin Donuts poll" is positive, but said "I would be lying" if he said people didn't bring up these parts of his record. He says he tells them, "If you can't get past that I understand it fully," and counsels them to vote for another candidate.

He said the one thing people never are upset about is the fact he changed parties briefly, saying conservatives he meets now are "so angry at the establishment and this national party and some in the state party who have all endorsed Brown … it's not the way it's supposed to be, primaries are supposed to be left to the voters."

He said he left the party in 1999 to make a statement that the GOP "can't desert our principles," which he thinks some in the party, notably Brown, have done. The former Massachusetts senator is more moderate than Smith and the Republican Party on issues including abortion and gun control.

Smith says his early stance against what he calls the "catastrophe" with the national debt shows that he was "tea party before the tea party was cool."

Smith may be 73, but he says he's been hitting the campaign trail hard. On Friday, he did 11 appearances, including two radio interviews, a stop at a gun manufacturer, two gun shops, and a block party.

"I did it the old fashioned way," Smith said. "As if I had never run before, I re-introduced myself and I think it has worked."

In their final GOP primary debate on Thursday, Smith was the only one of the three candidates who declined to commit to endorsing Tuesday's victor, saying "I'll make that decision when the primary election is over."

Smith said he did that because he doesn't "like to pre-judge … I like to think I'm going to be the nominee. That's always the way I think, but I also said that I would support every Republican that supports the Republican platform." Another clear non-committal about whether he would endorse Brown or the other challenger, former New Hampshire state Senator Jim Rubens.

He said he isn't wistful as he looks at his final campaign, and he has "no bitterness for the past."

"I think if you are bitter you aren't healing yourself, you have to forgive and move on," he said.

Looking back, he said that during his time in the Senate, Capitol Hill was less divided than it is now, noting, "Ted Kennedy and I were bitter political enemies, but we were friends. I liked him a lot and I think he liked me … that's respect."

Watchers of Smith's bid say his irregular path, despite the years, could still be having an impact.

"Some voters are not forgetting," Director of the New Hampshire Institute of Politics at Saint Anselm College Neil Levesque said.

"Leaving the party, endorsing Kerry, then running for the U.S. Senate in Florida. That's three pretty big things that could be central to voter' minds," Levesque said. "He's certainly been a longtime candidate, but he hasn't been necessarily here for a while. He's been in Florida. It was kind of a surprise then when he came back."

Smith's final message to voters is something he says he hasn't held back from telling Brown on the campaign trail.

"I think Scott Brown is more like Shaheen than he's like me," he said, before recalling Ronald Reagan's famous phrase. "If he's the nominee there are so many issues he agrees with Shaheen on, I don't see him winning, but with me bold colors, not pale pastels … there is a sharp contrast between Shaheen and myself."