"Sean did not have his most basic right respected, which was to be listened to by the Brazilian justice system," his grandmother, Silva Bianchi, told ABC News Tuesday.
Brazilian courts have already turned down the family's initial request to let Sean testify but has not made a final ruling on the matter.
But a U.S. judge said he doesn't believe any ruling by Brazil's courts would be enough to send Sean back.
"Realistically, it's just not going to happen," U.S. District Judge Peter Messette said. "I cannot imagine any judge or any judicial system in the United States sending this boy back."
Goldman, who brought Sean home from Brazil Christmas Eve, said Tuesday that his son hasn't cried since being home and is eager to play.
"He's happy and having fun," Goldman said.
Goldman said the transition from Brazil to Tinton Falls, N.J., had been easy.
"It's almost so seamless I'm taken aback by it," Goldman told a news conference. "He doesn't have the stress, he doesn't have the pressure. He can be a little kid again."
David and Sean spent the first few days in the United States with relatives in Orlando, Fla., after their Christmas Eve flight out of Brazil. They returned to New Jersey this weekend. Sean went immediately to the Christmas tree, opened presents and took instantly to Goldman's cat.
The father admitted that the boy has yet to call him Dad.
"He'll just come to me and say, 'Can I do this?' not calling me anything," Goldman said. "Yeah, it hurts, but you know, he's home. If he doesn't call me Dad for a while, I can live with that because I can live with him."
He said his son is very playful, starting a pillow fight with his father this morning, asking for sharks in the now empty fish tank, playing with Nerf guns and Wii games with his father until midnight his first day home, Goldman said. The boy insisted on going outside to play with his cousin today, despite frigid temperatures.
And when they drove past the Six Flags amusement park on their way home, "He said, 'I want to go there,'" Goldman said.
The boy, who last lived in the house when he was 4, remembered that he liked to bite the bannister as a little kid. He tasted it again.
Sean went with his Brazilian mother to visit her family in Rio de Janeiro five years ago and never returned to the United States. Sean's mother later divorced Goldman, remarried and died while giving birth to a child by her new husband.
When Goldman sought custody of his son, her family began a long and bitter battle to keep the boy in Brazil.
Goldman said Sean had been allowed to speak to his grandmother in Brazil three times so far since arriving in the United States, but Goldman still seemed angry about how hard his former mother-in-law fought the custody ruling and how she delivered the boy to the U.S. consulate.
Sean Goldman's Brazilian Sister Calling for Him
He was upset, Goldman said, that the grandmother parked outside the consulate and walked the boy through the media horde.
"Why didn't you drive into the consulate?" Goldman said he asked her. "She didn't have an answer."
Goldman said the grandmother did cooperate with one request.
"I told her, you need to tell Sean I was a good father and I will be a good father. You have to do it for him," Goldman said. She complied, he said.
ABC News spoke today with the Brazilian matriarch, Silva Bianchi, who said: "I am very, very sad. My country didn't protect us, protect Sean."
Bianchi said she spoke to Sean today and the conversation reassured her that the boy was all right.
"I am more comfortable now because he's OK with his dad. He told me it's cold, he's sitting by the fireplace and he's playing. I think he's doing OK there and that makes me feel better," she said.
Between tears, Bianchi said it was hard on the rest of the family, particularly Sean's younger sister.
"His sister misses him a lot. She calls for him all day, walks by his room asking for him, looking for him. It's so upsetting," she said.
Goldman said his son would be allowed to see his Brazilian family members "in the appropriate time and under appropriate conditions."
He said the families need to come to a settlement, "something collectively. It can't be this constant battle."