In referring to Jews and the media, Graham said, among other things, "This stranglehold has got to be broken or this country's going to go down the drain." He apologized for his comments, saying he had no memory of the conversation and "They do not reflect my views and I apologize for any offense."
Graham's supporters say that conversation was an aberration.
Jimmy Carter told Gibson: "That was a mistake he made. Since he was so forgiving of others when they made a mistake, I think that, that the natural thing for all of us to do is forgive him because he has publicly apologized for that comment … It may be that President Nixon's feelings along those lines may have influenced Billy Graham in a private conversation."
Other presidents say Graham's role was not one of advisor. "I've never called him on a specific issue. I wouldn't be afraid to if I needed to, but his influence is bigger than a specific issue as far as I'm concerned. He warms your soul," George W. Bush said. "He reminds one of the greater purpose of life. Just talking to the man is beneficial to me."
Bill Clinton said, "I doubt that many presidents wanted to be around him because they thought it would help them politically. I think that they really felt and hoped that by being with Billy Graham, they might have more strength for the work at hand."
He has preached to 215 million people in 185 countries, but Graham does not have a church or an official congregation. In a sense, the presidents, politicians and their families have been his flock. And it is a role he takes seriously.
"I think he thinks the men who get [to the presidency] need the blessing of God in order to do the job," said Michael Duffy, co-author, along with Nancy Gibbs, of a new book about Graham "The Preacher & the Presidents."
Graham has attended or participated in 10 inaugurals since 1953 and has spoken at the funerals of Johnson, Nixon and Ford.
Gibbs said these close relationships developed because "the presidents saw someone who really understood what they were up against and who they could talk to very confidentially."
He's also had a role in the faith of many of these men before their rise to power. A 13-year-old Clinton saw Graham preach in a racially divided Little Rock, Ark., in 1959.
Graham refused to segregate the audience and Clinton remembered how it affected him. "I desperately wanted it to be different and, and he, by what I thought was an act of grace, showed that it could be different."
Yet there are tantalizing questions about Graham. Should a pastor — of any faith — get so close to power?
Did presidents exploit Graham — or did Graham exploit them?
"Every president wanted to be seen with Billy Graham … because he could convey … them a blessing that no one else in American life could," said Duffy.
In a country founded on the separation of church and state, what role does faith play in its most powerful office?
"I don't think the president is any different than anybody else when it comes to wonder," George H.W. Bush told Gibson.
"I prayed more and needed more [spiritual] counseling when I was president than any other four years of my life." said Carter. And George W. Bush said, "I don't say you can be president without faith, I'm sure some have done it. In my case, I don't see how you can be president without belief."
To read more from TIME's Nancy Gibbs and Michael Duffy, authors of "The Preacher & the Presidents," click here.