Schaffel, who eventually had his own falling out with Michael Jackson and ended up suing him, said that professionally, there had long been tensions between father and son. In 1983, Joe referred derisively to Michael's co-managers as "white help," causing Michael to release a statement disavowing his father's words.
As Michael's downward spiral began, Joe's input waned and his father said he sensed that those surrounding Michael were taking advantage of him. He said that his son, who seemingly surrounded himself with those who didn't have his best interests at heart, "didn't know any better."
As he mulls who might have been around Michael, and what they might have been doing, at the time of his death, Jackson admits to having one regret.
"Maybe that I might have -- should have learned 'em a little more how the streets are instead of -- instead'a keepin' them -- close together," the family patriarch said of teaching his son street smarts to better judge those around him.
"Michael's the type of people he give you his shirt off his back and go and cry about it later," Jackson said. "I remember a time when everybody had a cell phone and he's payin' all them cell phone bills himself 'cause he's nice like that ... Michael didn't care about money."
Still, Joe says he was unable to stop his son from spending what he had.
"There was nothin' I could do about that 'cause it was his money," Jackson said. "Tell him, 'Michael, you gotta -- you gotta be careful with the people that you're around. And make sure tha t-- you are around the right people.'"
In the grandfather's eyes, his son's tragic death paradoxically revealed to the world that Michael's three long-concealed children are bright, poised kids.
"I thought he was a great father. He took his kids to places. And he didn't let 'em watch TV too much because of the -- they had certain times to them to watch because he was trainin' 'em. And those kids about the smartest kids I ever seen, really. 'Cause that's what Michael put into them," Jackson said.
The world has attention turned to the fate of Michael's three children, Prince Michael I, 12, Paris-Michael Katherine, 11, and Prince Michael II, 7, also known as Blanket, who have been under the care of Katherine Jackson at the Hayvenhurst compound, living with five cousins.
In the exclusive reality TV footage obtained by ABC News from before Michael's death, Katherine plays Pictionary in the yard with five of her other grandchildren, who also live at the Encino home, with Joe Jackson looking on. If not for their celebrity name, the video paints a picture of a "normal" family life and upbringing.
"I am just proud of my family and my great grandchildren and I thank god that I have lived long enough to see my great grandchildren and I just, when I think about it I get tears in my eyes," Katherine Jackson said in the footage.
With a custody battle looming between the Jacksons and Debbie Rowe, mother of Michael's two oldest children, some experts say the court may question whether Katherine will have the stamina to keep up with a three children who are dealing with a sudden and dislocating change in their lives. A hearing on the custody issue has been postponed until later this month.
Still, little doubt remains in Joe Jackson's mind who should raise the children.
"Their grandmother, Katherine, and I," he said. "There's no one else to do what we can do for them. You know, which -- keep 'em all together and-- make 'em happy. And feed 'em like they're supposed to be fed. And let 'em get rest. Plenty of sleep. And grow up to be strong Jacksons."