"What the public doesn't see is how competitive he is," Duncan told ABCNews.com. "He plays to win."
"He's an athlete," he said. "This is an important part of who he is."
Duncan said he and other players don't hold back for fear of hurting the president-elect.
"We don't take it easy," he said. "That would ruin the competitive nature of the games, and that's what it's all about."
Obama told ABC News' Barbara Walters that he considered tearing up the White House bowling alley and replacing it with a basketball court.
"Well, it turns out that there's a tennis court in the White House, and I think the guys in charge say they maybe can put a couple of basketball hoops at either end. So that may be good enough for spring," he said. "And, I hear there's a gym at Camp David. So, we may leave the bowling alley in place, partly because it's clear I'm going to have to practice if I want to get any good."
The half-joke about his bowling abilities referred to a campaign stop in a bowling alley where he quit trying after throwing nothing but gutter balls.
Obama's predecessor was no slouch when it came to fitness: Bush was known to jog on a treadmill aboard Air Force One, cycle, weight lift and stretch six days a week.
On Election Day, Obama played what had become his "lucky charm" basketball game with old high school and law school friends. Duncan and Casey joined them.
"He did really well," Casey told ABCNews.com. "He's actually a very good player."
Casey, who is 48, warns that basketball is not only intense, but dangerous. When he played in an over-30 league at age 31, he broke his nose and severed his Achilles tendon. Duncan also admits to breaking his nose at a game.
"You're running in short, hard sprints and your other parts are moving laterally and jumping," he said. "When you're my age and his age, you're not supposed to be playing. You have to be careful. You're more apt to get hurt."
But both men describe Obama as "smart," playing guard and maintaining his distance.
"He's disciplined and steady," said Casey.
Obama's focus on exercise may actually bode well for the country.
"It gives you more mental endurance and more energy to think clearly," he said. "For a president, that's not a bad thing."
Even Obama's snack food is lean: Planters Trail Mix (nuts, seeds and raisins), MET-Rx chocolate roasted-peanut protein bars and Black Forest Berry Honest Tea, an organic brew that's hard to find.
He does, however, have his temptations: smoked sea-salt caramels in milk chocolate. And there is also the much-reported on-and-off battle with cigarettes.
"I was never really a heavy smoker," Obama told Men's Health in November. "Probably at my peak I was smoking seven or eight a day. More typical was three. So it wasn't a huge challenge with huge withdrawal symptoms.
"There have been a couple of times during the campaign when I fell off the wagon and bummed one, and I had to kick it again," he said. "But I figure, seeing as I'm running for president, I need to cut myself a little slack."
In his interview with Walters last month, Obama talked about his continuing effort to quit the smoking habit.
"Part of what I think comes with this role as president is not that you're perfect, but hopefully you're trying to set a good example for people and that starts with my two kids," he told Walters.