"I was trying to raise myself to be a black man in America and beyond the given of my appearance, no one around me seemed to know exactly what that meant," Obama writes in "Dreams."
The basketball court became a place of refuge. His old friends say they had no idea of his struggle until they read his book more than 20 years later.
"In reading that … it doesn't surprise me at all that he said that because we were all going through our things out there," said Lum. "I mean, I didn't know to the extent of what he was going through for sure but we were all going through our things. And maybe that's why, the basketball court was kind of, a sanctuary for us too. … Kind of bring us together and. … Get us away."
And it would become the very place where Obama would refine perhaps his most important talent: his ability to communicate.
"He could beat anybody in a debate and we wouldn't even realize we got beat because we'd end up agreeing with him," Hale said. "He would be very straight to the point and then he'd just have a way of just getting people to agree."
Today even from a distance, Obama is still very much a part of Punahou, from articles in the school newsletter to fundraisers for his presidential bid.
His old coach remembers the last time he saw Obama in person a few years ago. He says he didn't want to bother the newly famous politician so he stayed off to the side.
"Part way through his speech," McLachlin said, "he kind of caught my eye in the back of the chapel and said, 'Coach Mac, how you doing? You know I used to play basketball here you guys and I really wasn't as good as I thought I was. Was I coach?' and we sort of laughed about it."
McLachlin continued. "He sort of admitted, you know, maybe I pushed the envelope a little bit too much on the minutes thing and I really wasn't as good as I thought I was and it was kind of, I thought, a very cogent remark."
Eldridge agreed. "It's like fatherly pride that I sit here and see that one of my students, you know, is running for president of the United States," he said. "That's almost beyond belief that if something like this happens that, you know, I don't know if I could take it."
Obama's communication skills continue to impress Eldridge. "I e-mail him and he writes, he e-mails me back," he said. "In three days, I got an e-mail in three days from Barry and I'm thinking, 'Geez, how many presidential candidates would write their, you know, former teacher back?"
After graduation Obama left the Punahou School for Columbia University, Harvard Law School and eventually the U.S. Senate. But first, along with his fellow Rat-ballers, he went on to win the state championships -- the best basketball team in school history.
No. 23 wasn't a star back then, but he was a standout. And his teammates are still cheering for him.