Frazier went 21-32-1 in three-plus seasons for the Vikings, including 5-10-1 in 2013. The 54-year-old had one season remaining on his contract with Minnesota.
He will join a Bucs staff led by new coach Lovie Smith, who was hired this week after Greg Schiano's ouster. Also, former California coach Jeff Tedford was officially named the club's new offensive coordinator on Saturday.
The Bucs still might interview another candidate or two for the general manager job, but Ballard tops the list, sources said.
Information from ESPN senior NFL analyst Chris Mortensen was used in this report.