NFL Players Association attorney Theodore B. Olson told ABC News on Monday morning in an interview aired on "Good Morning America" that it plans to file the appeal later Monday.
"The facts here are so drastic and so apparent that the court should rehear it," Olson told ABC News.
The court upheld Brady's suspension in a 2-1 decision April 25, deciding that commissioner Roger Goodell's discipline was properly grounded in the collective bargaining agreement and that Brady was treated fairly.
Brady will appeal en banc, asking the entire panel of the 2nd U.S. Circuit Court of Appeals to participate in the hearing. Seven of the 13 judges would have to agree that an en banc session is the right decision before a hearing is held.
If the en banc session isn't granted, Brady could take his case to the U.S. Supreme Court.
Olson told ESPN's "Mike & Mike" that he expects the appellate court to move quickly on a ruling on the hearing. If necessary, he said attorneys will consider asking the court for a stay. That would freeze action in the case and allow Brady to play if the case continues closer to the season.
Monday is the final day for Brady and his legal team to file an appeal after they received a two-week extension from the initial May 9 deadline.
The NFLPA added Olson to its legal team defending Brady days after the court's ruling. Olson has argued 62 cases in the Supreme Court, including the two Bush v. Gore cases arising out of the 2000 presidential election. He has prevailed in more than 75 percent of those arguments.
Olson also represented the NFLPA during the 2011 labor negotiations.
"Our two primary arguments are that the commissioner in the first place conducted an investigation and then the commissioner imposed discipline. Then the commissioner appointed himself as an appellate judge or an arbitrator and then decided something new in the appellate process, abandoning the grounds that were the original basis for the supposed discipline," Olson told ABC News. "That's No. 1, and an appellate judge is supposed to look at the record and make a decision on the basis of what happened before. He departed from what happened before. Secondly he ignored important provisions of the CBA about discipline that might be imposed for equipment violations. He departed from that completely and went off the track."
In requesting the two-week extension, Brady's legal team wrote that the decision "raises significant labor law issues that could have far-reaching consequences for all employees subject to collective bargaining agreements."
On SportsCenter on Monday, NFLPA executive director DeMaurice Smith said Brady is filing the appeal to fight an unfair ruling by the NFL and Goodell.
"The league has a history of being bullies," Smith said. "And today we're filing a brief because the commissioner decided to be a bully when it came to the fair hearing of a player."
If Brady is suspended for the first four games of 2016, he would miss games against the Arizona Cardinals, Miami Dolphins, Houston Texans and Buffalo Bills. He would be eligible to make his regular-season debut in Week 5 against the Cleveland Browns.
Brady signed a two-year contract extension during the offseason that dropped his 2016 salary from $9 million to $1 million. That could save Brady almost $2 million in lost salary during a four-game suspension.
Information from ESPN's Mike Reiss and The Associated Press was used in this report.