But eventually his mother, too, would travel, leaving her son behind -- an absence that hurt more than he said.
"I think in retrospect, it was probably harder on me than I cared to admit," Obama told ABC News' Diane Sawyer. "You know, if you're a young boy or a teenager, you don't want to think that you still need your mom around."
"He wrote beautiful letters to her," Soetoro-Ng said of her older brother. "Sometimes in the corners, he would put little illustrations, little cartoon characters of himself, looking very cool."
Obama was 10 years old when he moved in with his two middle-American grandparents who would raise him and become the stable center of his life.
Looking at a photo, Obama said he saw the joy of his early childhood. Both grandparents came from Kansas. His grandfather was a World War II veteran who had come to Hawaii to find better work. Once there, their daughter threw them a curve.
"They had to absorb an awful lot," Abercrombie said. "Put yourself back 50 years. 'Hi mom, hi dad. You know, guess who's coming to dinner, right?' It's not Sidney Poitier."
Nonetheless, Abercrombie said, the grandfather loved his little boy, Barry.
"His grandfather was the most wonderful guy who loved that little boy," Abercrombie said, "took him everywhere."
There wasn't much money. They all lived in a 950-square-foot apartment. His grandfather's effort to sell furniture, and later insurance, failed. Holding the family together was a 5-foot-3 woman they called "Toot."
"She never got a college education, but is one of the smartest people I know," Obama said of his grandmother, who in a man's world, worked her way up from secretary to bank vice president.
Obama said his iconic image of his grandmother was seeing her come home from work and trading her business outfit and girdle for a muumuu, some slippers and a drink and a cigarette.
"She's where I get my practical streak," he said. "That part of me that's hardheaded, I get from her. She's tough as nails."
Watch "Portrait of a President" for much more on Obama's life and influences, and to read about John McCain's years at the Naval Academy, click here.