Actor Kal Penn's Final Thoughts After Leaving the White House: 'The Score Keeping on Capital Hill' Is a 'Bummer'

July 29, 2011— -- Kalpen Modi's job as a White House liaison to young voters for the past two years has been to re-energize them during a time when the job market is bleak, and compromise on Capitol Hill seems impossible. He clearly has had his work cut out for him.

"I think for most young folks who I'm talking to, it doesn't matter if they're on the left or the right, the issues that they cared about were the same," he said. "It is a kind of bummer if you look at the score keeping on the Hill ... and what we have been focusing on is getting folks on the same page, focusing on the solutions."

More recognized by his stage name -- Kal Penn -- the 34-year-old actor is known for playing the lovable stoner Kumar in the "Harold and Kumar" film franchise, or the suicidal Dr. Kutner in the TV series "House, M.D." or a terrorist from "24."

But Penn stunned Hollywood in 2009 when he left to come to work in President Obama's White House as the associate director for the Office of Public Engagement -- a job he has never spoken about on camera until now.

"It's just not our role here," Penn said. "There is a whole team that does media stuff. Our jobs are just keep our heads down and do good work."

On the eve of his departure on July 29, Penn sat down for an exclusive interview with ABC News to talk about his work and his views on the political process.

"I don't know anyone who would say that the White House is particularly glamorous," he said. "I think everyone has an understanding of the impact you can have, and the fact that ... that feeling is indescribable."

Raised by Indian immigrant parents in Montclair, N.J., Penn has been working for Obama's senior advisor Valerie Jarret, slaving away in a small office with five other junior staffers.

"Our office doesn't handle policy, but we help bridge the gap between policies," Penn explained. "So if there's a group that's particularly concerned with an issue, and they want to bring in 10 or 12 folks, we'll put them in a room with some of our policy team and they'll link up that way."

The Office of Public Engagement focuses on creating a dialogue about policy with students from around the country. One of Penn's jobs is to engage the elusive young voters who turned out in droves for Obama in 2008 but who seem to have become more complacent almost three years later.

"I think what I've seen is that there is a realization that change is not a light switch," he said. "That if it was easy to flip on a light switch and change everything someone would have done it before, and it's actually a very laborious process, it's very slow. I wouldn't say it's disillusionment, I would say it's understanding the process."

While several actors have campaigned for Democratic politicians and presidential candidates, Penn may be the only one in recent memory to take a hiatus from acting to work for a politician full-time behind-the-scenes. It's a decision he said he made for personal reasons.

"I had friends who were over in Iraq and Afghanistan," Penn said. "I had buddies who had huge student debt, people who got kicked off their health insurance plans for one reason or another, and so that was my decision to get involved on a personal level."

Kal Penn Says Working at the White House Has Been 'Awe-Inspiring'

Seeing this job as an opportunity to serve his country, Penn said the experience has been "awe-inspiring" and Obama hasn't shown him any special celebrity treatment.

One of his fondest memories was the late night in January when he accepted a painting to be given as a gift to Chinese president Hu Jintao. A White House videographer caught the moment on camera, which aired on the White House's "West Wing Week" video series. Watch the video HERE.

"You're standing there at midnight at the South portico and it's freezing cold outside and you're receiving a painting for the president that he's going to give to a world leader," Penn said. "You kind of take a moment and kind of realize where you are -- it's humbling."

And another one of the perks for working for the president is sometimes Penn and his co-workers staff presidential events, which can be hot spots for celebrity sightings. He said he ran into actor and comedian Chris Rock while working the gate at the Kennedy Center Honors last year.

"I'm a huge Chris Rock fan," Penn said. "Chris Rock sort of looked at me and said, 'Hey, you're that guy!' and I'm like, 'Hey Chris, how you doing?' and he's like 'I heard you were here, you're doing this?' and I said, 'I'll see you inside.'"

There's certainly a disconnect between Penn's coat and tie look and his earnest and wonky persona and his marijuana-loving, munchie-seeking Kumar Patel character -- who is returning to theaters for the "Harold & Kumar" series' third installment this November, "A Very Harold & Kumar 3D Christmas." Penn had to take a leave from the White House to shoot the film, as he was contractually obligated to do.

"Everyone knows the difference between fact and fiction, suspension of disbelief," he said. "Everyone knows that Anthony Hopkins didn't actually eat people. ... I have not actually had an experience where the conversation has been sidetracked by my old life."

The plan, he said, was always to work for two years at the White House. Now on his last day, he heads back to Hollywood, where work waits: a guest role on the hit CBS TV series "How I Met Your Mother," a development deal and, of course, "Harold and Kumar 3." But Penn has clearly enjoyed every minute of his work at the Office of Public Engagement.

"What I love and am passionate about with service," he said, "is this concept that, yes, we elect officials but things don't actually get done until we put our boots to the ground and become part of that solution."