They didn't even have enough money to buy coal in coal-rich Welch. They would heat their home with scavenged coal -- but it would only be enough for a few hours. Most of the time, they were cold.
And for food, the Walls siblings often looked to a cafeteria garbage can at their high school.
When Walls became an adolescent, she started yearning for the nice things the other people in the town had. But her mother condemned such trappings as matching curtains "middle class."
Walls thought to herself, "I want a little piece of that middle class," and started putting together her own plans to do so. The day after she finished 11th grade, she got on a bus for New York City.
Walls went to live with a sister who had left for the city before her. She started work immediately -- one of her first jobs was as a nanny -- and at 17 did something her parents had never managed to do: pay the bills.
She also finished her senior year and applied to college. She was accepted at Barnard, Columbia University's sister school. The girl from Appalachia joined the Ivy League.
She thought her past was far behind her, until her parents and youngest sister packed up the van and decided to join their three older children in the city.
They lived in their van until it got towed. And then they lived on the street. "They brought all of the craziness that I've always lived with," Walls recalled.
Finally, Rex and Rose Mary found a home of their own. They moved into an abandoned building.
As her parents became squatters, Walls' career kicked into high gear. Her work also brought her in touch with her future husband.
She was immediately impressed by him: born in Japan, son of a diplomat, educated in a boarding school. "He just seemed a little bit like going after one of the Kennedys," she said.
Walls was initially vague when they talked about her family. Only when their relationship developed did she reveal everything to him. His reaction was exactly the opposite of what she had always feared.
"I was in awe of this woman who had come from this incredible, you know, background and had overcome all of these obstacles and adversity and then pulled herself up," Taylor said. They married in 2002.
The little girl who was once so cold and hungry in Appalachia now lives on a well-groomed street on Manhattan's tony Upper West Side. She has finally found security, stability and true love.
Meanwhile, Rose Mary Walls still lives in poverty just a short subway ride away from her daughter. She is a widow now. Rex died 10 years ago. But she still lives in the squat they found when they first came to New York.
She lives in a room crammed with paints and frames, with no heat or running water or a working stove. She keeps electricity to a bare minimum. And she appears to be still living by the same philosophy she had when Walls was growing up.
If you have a nice house, "you put your whole energy and being into keeping that house up," she said. "You're not living for yourself, you're living for a house."
Walls says she still loves her mother and wishes she could help her more. But Rose Mary refuses -- and the younger Walls believes that unconditional love means letting her mother be who she wants to be.
"I accept her -- she's neat. I don't know if she's loopy or worse, but it works for her. She's happy and she leads the life she wants."