The couple were briefly married following Prince's conception, but divorced six months after Paris' birth. Rowe gave Jackson custody of the children, but sued him in 2006 for breach of contract in an attempt to regain custody. The pair settled out of court for undisclosed terms and Jackson retained custody.
According to Brown, the Jackson family believes Rowe will again try to gain custody of her children.
Rowe's lawyer, Iris Finsilver, would not comment on whether Jackson's ex-wife would try to get custody of the children after Jackson's death, saying only: "Debbie is absolutely inconsolable."
In 2001, Jackson had a third child, a son named Prince Michael II, better known as Blanket, who was carried by a surrogate mother whose name has not be released.
Jackson later admitted making a "terrible mistake" when he dangled his infant son over a hotel balcony in Berlin as he greeted screaming fans.
A year after, following a stunning admission in a television interview that Jackson invited an unrelated adolescent boy to share his bed and offered him wine -- which he called "Jesus juice" -- the pop star was accused of molesting the child.
In between the interview and the 2005 molestation trial, California state child welfare officials interviewed Jackson's children and Rwaramba, the nanny originally from Rwanda.
Jackson was acquitted of the charges in the trial. No further molestation charges were brought against him and he retained custody of his children.
During the investigation, Jackson's parents Joe and Katherine told ABC's Barbara Walters that they were concerned officials would attempt to take away the children. They offered to adopt the children, should the singer lose custody of them.
According to Brown, the parents are again looking to adopt the children but Jackson said he wanted Grace Rwaramba to have custody.
"Michael's wishes were known. He wants Grace to have the kids. They love her," Brown said.
"If the kids had the choice," Brown said, "I think they'd pick Grace."
Before his death, Jackson was in regular contact with his mother but had a notoriously contentious relationship with his father.
Following his acquittal in the molestation trial, Jackson, Rwaramba and the children moved to Bahrain and then later to Ireland.
According to a 2006 profile in the Daily Mail, Rwaramba had become Jackson's closest confidant and gate-keeper, screening his phone calls and setting up meetings.