Perpetually in his mother's shadow, it seemed Anthony Marshall could do no right in her eyes -- especially when it came to Charlene, his third wife. Their courtship and 1992 marriage was marked by scandal in Northeast Harbor, Maine, where Astor had a waterfront estate and Charlene Gilbert -- then the local minister's wife -- fell for Marshall, who was also married at the time.
"I met Tony and fell in love with him, and I thought I was the luckiest girl in the world," Charlene told ABC News' Deborah Roberts in an exclusive interview. "When Tony first told his mother that he was getting divorced, she was thrilled. And she said, 'Oh, this is wonderful, you can escort me to all the dinners and parties.'
"Well, not long after that he told her that he was getting divorced because he wanted to marry me. Then she wasn't so thrilled."
"My grandmother really did not like Charlene," Philip told ABC News. "That was clear."
Philip, who says tensions with his father grew after the marriage to Charlene, told "20/20" that he was suspicious of his stepmother's intentions from the beginning.
"You have to wonder, was there some grand plan," Philip said, "Or was it simply love at first sight? And I think it might be mixed."
When asked by Deborah Roberts if he considered Charlene to be a gold-digger, Philip said the term "emerald digger" might be more appropriate. He suggested that Charlene may have had her eye on his grandmother's famous emerald necklace -- a gift from the late Vincent Astor.
"I think ... Charlene came from a family that, or as her family was, had very little money. And she saw an opportunity," he said.
While Brooke Astor acknowledged that Charlene made her son happy, many close to Astor said she never fully accepted her daughter-in-law -- ultimately snubbing Charlene in her will.
In the very public criminal trial, the prosecution depicted Charlene as a greedy villain, and claimed it was Marshall's motivation to secure funds for his wife at a time when Astor was incompetent to make financial or legal decisions.
"I really believe as though Charlene is driving this," Philip said. "The plan seemed to have been that Charlene wanted as much as she could possibly get. Even a few million was not enough. That she wanted everything."
The prosecution claimed that Marshall had taken some of Astor's valuable artwork, among other things, and tricked her into changing her will, steering millions to Charlene and to him instead of to charity. The defense argued that Astor changed her will after warming up to her daughter-in-law.
"Over time we grew closer," Charlene told "20/20." "She realized that Tony and I truly were deeply in love. ... She would always just thank me profusely for being there and for taking such good care of him and for making him happy."
Philip disagreed, saying that it's not possible that Astor had a change of heart. "I think she probably didn't want much to go to Charlene," he said.
Tony Marshall insists he is innocent of the charges. On legal advice, Charlene and Marshall declined to discuss details involving the case.
Philip told "20/20" that filing for guardianship of his grandmother was never about the money.