Rep. Vance McAllister, who returned to the Capitol today for the first time since video leaked showing him making out with a congressional staffer, maintains that he will not resign, contending that it would be "very selfish of me because of what I did to not give them a voice no more."

"I was elected to fill a role and that's what I'm doing," McAllister, R-La., told ABC News as he arrived at his Congressional office Monday evening. "I'm going to make sure the people still have their voice and work as hard as I can, be their congressman for the time left on my term."

Standing alongside his wife, Kelly, McAllister said his decision not to run for reelection is all about "this lady right here and my family," before complaining about the lack of power freshman lawmakers command.

AP Photo

"It's such a dysfunctional mess up here. One man can't make a change in three to four days a week, and that's all I can come up here and do as a freshman congressman. It would take 10 years to get to where I need to be," he said. "I've asked for forgiveness, I've fallen short, I've done the things I feel like I need to do, I appreciate the privacy everyone has given me for these last three weeks, to be sure to repair the damage that I have done to my family and the people closest to me, and we've done that and it's time to move on."

McAllister and his wife agreed they are confident their marriage will survive.

"Absolutely," McAllister said as his wife nodded in agreement. "That was my number one focus, my number one goal, and that's where I'm at today."

McAllister said he has not spoken to the staffer captured in his embrace on the leaked video, but he was adamant the relationship was over before the news broke. He refused to discuss how the relationship evolved or ended.

"That has nothing to do with me representing the Fifth District as a congressman," he said.

McAllister rode the coattails of America's No. 1 rated cable television program to emerge from a jam-packed primary to win a special election in Louisiana's 5th Congressional district last fall. He was endorsed by "Duck Dynasty" stars Phil and Willie Robertson and had never been to Washington, D.C., before he traveled here to take the oath of office.

The Louisiana Republican who campaigned on a pledge that Washington wouldn't change him, also deflected criticism from state party leaders who called on him to resign.

"Look man, they wasn't with me before I got elected. Why would I think they'd be with me now?" McAllister said. "There ain't no pressure there. I believe in doing what's right."

"I fell short in my values - the values I still hold true to me, and I'll continue to maintain those values and it is what it is, but there's never no pressure to resign from any of the political pundits, or whatever you want to call them, just throwing stones, you know?" he continued. "Maybe they should look at doing their job better and quit worrying about what somebody else is doing in their job. Nobody's perfect."