Haslam again denied that the Browns have a negative public image, and he also rebuffed questions about the federal investigation into rebate fraud at his company, Pilot Flying J, saying Tuesday was not the time to discuss that topic.
The shakeup came one month after the Browns hired Pettine after a 25-day search. Banner and Haslam fired first-year coach Rob Chudzinski after a 4-12 season, the team's sixth straight with at least 11 losses.
Banner, who previously worked in Philadelphia, was hired by Haslam to run the team shortly after his ownership was approved by the league in 2012.
Information from ESPN NFL Insider Adam Schefter, The Associated Press and ESPN Stats & Info contributed to this report.