Hoyer, who is the starter heading into training camp, is headed into the last year of his deal, which is scheduled to pay him $1 million this season.
Hoyer's agent, Joe Linta, confirmed to NFL.com that negotiations have started.
"It will be a very difficult deal to do," Linta told the website. "We're always open to talking with the Browns, but we're content to wait and see what happens down the road."
Coach Mike Pettine told reporters on June 13 that Hoyer was "securely ahead" of first-round draft pick Johnny Manziel on the team's depth chart but that he didn't think that gap was insurmountable.
Hoyer, who has been a backup for most of his four NFL seasons, was 2-1 as a starter for the Browns last season before tearing his ACL.
Browns training camp begins July 26 at the University of Akron.