Actor Kal Penn's Final Thoughts After Leaving the White House: 'The Score Keeping on Capital Hill' Is a 'Bummer'

July 29, 2011— -- Kalpen Modi's job as a White House liaison to young voters for the past two years has been to re-energize them during a time when the job market is bleak, and compromise on Capitol Hill seems impossible. He clearly has had his work cut out for him.

"I think for most young folks who I'm talking to, it doesn't matter if they're on the left or the right, the issues that they cared about were the same," he said. "It is a kind of bummer if you look at the score keeping on the Hill ... and what we have been focusing on is getting folks on the same page, focusing on the solutions."

More recognized by his stage name -- Kal Penn -- the 34-year-old actor is known for playing the lovable stoner Kumar in the "Harold and Kumar" film franchise, or the suicidal Dr. Kutner in the TV series "House, M.D." or a terrorist from "24."

But Penn stunned Hollywood in 2009 when he left to come to work in President Obama's White House as the associate director for the Office of Public Engagement -- a job he has never spoken about on camera until now.

"It's just not our role here," Penn said. "There is a whole team that does media stuff. Our jobs are just keep our heads down and do good work."

On the eve of his departure on July 29, Penn sat down for an exclusive interview with ABC News to talk about his work and his views on the political process.

"I don't know anyone who would say that the White House is particularly glamorous," he said. "I think everyone has an understanding of the impact you can have, and the fact that ... that feeling is indescribable."

Raised by Indian immigrant parents in Montclair, N.J., Penn has been working for Obama's senior advisor Valerie Jarret, slaving away in a small office with five other junior staffers.

"Our office doesn't handle policy, but we help bridge the gap between policies," Penn explained. "So if there's a group that's particularly concerned with an issue, and they want to bring in 10 or 12 folks, we'll put them in a room with some of our policy team and they'll link up that way."

The Office of Public Engagement focuses on creating a dialogue about policy with students from around the country. One of Penn's jobs is to engage the elusive young voters who turned out in droves for Obama in 2008 but who seem to have become more complacent almost three years later.

"I think what I've seen is that there is a realization that change is not a light switch," he said. "That if it was easy to flip on a light switch and change everything someone would have done it before, and it's actually a very laborious process, it's very slow. I wouldn't say it's disillusionment, I would say it's understanding the process."

While several actors have campaigned for Democratic politicians and presidential candidates, Penn may be the only one in recent memory to take a hiatus from acting to work for a politician full-time behind-the-scenes. It's a decision he said he made for personal reasons.

"I had friends who were over in Iraq and Afghanistan," Penn said. "I had buddies who had huge student debt, people who got kicked off their health insurance plans for one reason or another, and so that was my decision to get involved on a personal level."