"Sean did not have his most basic right respected, which was to be listened to by the Brazilian justice system," his grandmother, Silva Bianchi, told ABC News Tuesday.
Brazilian courts have already turned down the family's initial request to let Sean testify but has not made a final ruling on the matter.
But a U.S. judge said he doesn't believe any ruling by Brazil's courts would be enough to send Sean back.
"Realistically, it's just not going to happen," U.S. District Judge Peter Messette said. "I cannot imagine any judge or any judicial system in the United States sending this boy back."
Goldman, who brought Sean home from Brazil Christmas Eve, said Tuesday that his son hasn't cried since being home and is eager to play.
"He's happy and having fun," Goldman said.
Goldman said the transition from Brazil to Tinton Falls, N.J., had been easy.
"It's almost so seamless I'm taken aback by it," Goldman told a news conference. "He doesn't have the stress, he doesn't have the pressure. He can be a little kid again."
David and Sean spent the first few days in the United States with relatives in Orlando, Fla., after their Christmas Eve flight out of Brazil. They returned to New Jersey this weekend. Sean went immediately to the Christmas tree, opened presents and took instantly to Goldman's cat.
The father admitted that the boy has yet to call him Dad.
"He'll just come to me and say, 'Can I do this?' not calling me anything," Goldman said. "Yeah, it hurts, but you know, he's home. If he doesn't call me Dad for a while, I can live with that because I can live with him."
He said his son is very playful, starting a pillow fight with his father this morning, asking for sharks in the now empty fish tank, playing with Nerf guns and Wii games with his father until midnight his first day home, Goldman said. The boy insisted on going outside to play with his cousin today, despite frigid temperatures.
And when they drove past the Six Flags amusement park on their way home, "He said, 'I want to go there,'" Goldman said.
The boy, who last lived in the house when he was 4, remembered that he liked to bite the bannister as a little kid. He tasted it again.
Sean went with his Brazilian mother to visit her family in Rio de Janeiro five years ago and never returned to the United States. Sean's mother later divorced Goldman, remarried and died while giving birth to a child by her new husband.
When Goldman sought custody of his son, her family began a long and bitter battle to keep the boy in Brazil.
Goldman said Sean had been allowed to speak to his grandmother in Brazil three times so far since arriving in the United States, but Goldman still seemed angry about how hard his former mother-in-law fought the custody ruling and how she delivered the boy to the U.S. consulate.
He was upset, Goldman said, that the grandmother parked outside the consulate and walked the boy through the media horde.
"Why didn't you drive into the consulate?" Goldman said he asked her. "She didn't have an answer."
Goldman said the grandmother did cooperate with one request.
"I told her, you need to tell Sean I was a good father and I will be a good father. You have to do it for him," Goldman said. She complied, he said.