The Browns are members of the Apostolic United Brethren Church, a fundamentalist break from the Church of Jesus Christ of Latter-day Saints -- the Mormon church, which officially banned polygamy more than 100 years ago as Utah sought statehood.
In making their case, the Browns argue that making polygamous unions illegal violates the free exercise, establishment, free speech and freedom of association clauses of the First Amendment, and the due process and equal protection clauses of the 14th Amendment.
The Browns have faced no allegations of incest, child abuse or child brides, despite the inquiries into their lifestyle, something that could help their case in court.
"We believe that this case represents the strongest factual and legal basis for a challenge to the criminalization of polygamy ever filed in the courts," said the Brown's attorney, Jonathan Turley, a law professor at George Washington University.
A ruling in the Brown's favor would affect tens of thousands of people in polygamous families in the United States.
Since the show first aired, the Browns told "Nightline" they had received many positive reactions from people in their community and strangers who have reached out to them through other means, such as Facebook, but have also faced discrimination.
"My job, I absolutely loved," Meri Brown said. "When I came out as a polygamist, and you know, became public, I was fired. That being said, some of my friends who had no idea have been extremely supportive of me."
In the end, the Browns told ABC News, they were glad they had at least raised awareness about their lifestyle and shown that even though polygamists, they were a loving family.
"We started this so that we would hopefully open and create more tolerance in the world," Janelle Brown said. "I hope that's what we're accomplishing here."
ABC News' Lauren Effron contributed to this story.