Seeing this job as an opportunity to serve his country, Penn said the experience has been "awe-inspiring" and Obama hasn't shown him any special celebrity treatment.
One of his fondest memories was the late night in January when he accepted a painting to be given as a gift to Chinese president Hu Jintao. A White House videographer caught the moment on camera, which aired on the White House's "West Wing Week" video series. Watch the video HERE.
"You're standing there at midnight at the South portico and it's freezing cold outside and you're receiving a painting for the president that he's going to give to a world leader," Penn said. "You kind of take a moment and kind of realize where you are -- it's humbling."
And another one of the perks for working for the president is sometimes Penn and his co-workers staff presidential events, which can be hot spots for celebrity sightings. He said he ran into actor and comedian Chris Rock while working the gate at the Kennedy Center Honors last year.
"I'm a huge Chris Rock fan," Penn said. "Chris Rock sort of looked at me and said, 'Hey, you're that guy!' and I'm like, 'Hey Chris, how you doing?' and he's like 'I heard you were here, you're doing this?' and I said, 'I'll see you inside.'"
There's certainly a disconnect between Penn's coat and tie look and his earnest and wonky persona and his marijuana-loving, munchie-seeking Kumar Patel character -- who is returning to theaters for the "Harold & Kumar" series' third installment this November, "A Very Harold & Kumar 3D Christmas." Penn had to take a leave from the White House to shoot the film, as he was contractually obligated to do.
"Everyone knows the difference between fact and fiction, suspension of disbelief," he said. "Everyone knows that Anthony Hopkins didn't actually eat people. ... I have not actually had an experience where the conversation has been sidetracked by my old life."
The plan, he said, was always to work for two years at the White House. Now on his last day, he heads back to Hollywood, where work waits: a guest role on the hit CBS TV series "How I Met Your Mother," a development deal and, of course, "Harold and Kumar 3." But Penn has clearly enjoyed every minute of his work at the Office of Public Engagement.
"What I love and am passionate about with service," he said, "is this concept that, yes, we elect officials but things don't actually get done until we put our boots to the ground and become part of that solution."