Community activist George Fields said whenever Ali came home, he would unite the area.
"When Ali won the Golden Gloves [amateur boxing tournament], the city of Louisville was in slavery mentality. For all low income not just certain color," Fields told ABC News. "Ali made Louisville known for ... peace.
"And he stood up for that. And he never never did what a lot of other celebrities did -- forget about where they come from," Fields said.
"He made a statement for everybody that lived here," Louisville native Sallie Larid told ABC News. "He made our city great by just showing what he can do for people.
"When he could, he always came back," Larid said. "He just never forgot us. That's the greatest thing."
Louisville native James Carr told ABC News he was a child when he met Ali, who was in the early stages of Parkinson's disease at the time.
"He gave me a jab on my jaw" and "he showed off his boxing moves," Carr said. "And gave me a hug and had a big old grin on his face.
"I'm glad I got the chance to meet him," Carr said. "He had a great impact on this city just from the way he was raised ... and how he was able to persevere through adversity and came back stronger."
Carr said Ali was an icon and role model.
"He was just so positive," Carr said. "A lot of us [as children] said when we grow up we want to be like Muhammad Ali."
Tina Davis and her children Alex and TaNaa took a tour of Ali's boyhood home Saturday to pay their respects.
"His leadership and the impact that he has is so major, not just for people in the world, but really major here in Louisville," TaNaa Davis told ABC News. "I honestly wouldn't feel like a Louisvillian if I did not come and pay some type of respect to him. Now I feel like I have a duty to do something to impact this world because of him."
Alex Davis echoed his sister's sentiment.
"It's like losing a brother. That's what it feels like," he said. "Because everyone here in Louisville is already really close knit, and losing someone with his inspiration and his influence -- it's like losing an icon for most people, but for us, it's like losing a relative."
ABC News' Janice McDonald and Charli James contributed to this report.