The 62-year-old Brown earns about $5.4 million and is under contract until 2020.
Earlier Friday, Brown met with Roderick Bernard, a Sharpstown High wide receiver who has committed to Texas, in Houston and was to return to Austin to meet with several recruits, a source said. Brown said he has 17 recruits in town for visits Saturday.
Brown said Thursday he has no interest in discussing his job status or publicly campaigning for his job.
"There's been a little speculation about my job situation," Brown said. "... My situation has not changed."
Patterson took over for the retiring DeLoss Dodds in November, and Brown acknowledged the effect a new AD could have on his job.
"Any time the athletic director changes, it changes the game," Brown said. "I will sit down and talk to him and Bill and discuss the direction we're going. I'm looking forward to my meeting with Bill and Steve and then move forward."
Brown was one of the most successful coaches in the country from 2001-09 when he went 101-16, won two Big 12 titles and the 2005 national championship, the Longhorns' first undisputed national title in 36 years. The Longhorns returned to the national championship game in the 2009 season, but lost to Saban's Alabama.
Since that loss, Texas is 30-20 overall, with Brown's only losing season in 2010. Texas expected to return to national prominence this year but started 1-2 with blowout losses to BYU and Mississippi. The Longhorns rallied to start 6-0 in Big 12 play before losing two of its final three games, including a 30-10 decision to Baylor last Saturday in what had become a de facto conference title game.
Information from ESPN's Brett McMurphy, ESPN.com's Max Olson and The Associated Press contributed to this report.