While many sports teams have declined an offer in protest, including the Golden State Warriors and Philadelphia Eagles, Bennett blamed the blowoff on scheduling issues.
"We have received inquiries about a visit to the White House," Bennett said. "With several players either pursuing pro opportunities or moving on from UVA, it would be difficult, if not impossible to get everyone back together. We would have to respectfully decline an invitation."
It wasn't immediately clear whether the White House had made inquiries about a visit.
One person was killed in August 2017 when white nationalists clashed with counterprotesters following a "Unite the Right" rally held to oppose calls for Confederate statues to be removed from Charlottesville. Heather Heyer was killed when a person intentionally drove into a crowd of counterprotesters. James Alex Fields was convicted of first-degree murder last December.
Trump initially blamed both sides for the rioting, and said there were "some very fine people on both sides." On Friday, the president maintained he "answered perfectly" at the time.
"If you look at what I said, you will see that that question was answered perfectly," Trump said Friday in an exchange with ABC News' Terry Moran. "And I was talking about people that went because they felt very strongly about the monument to Robert E. Lee, a great general."
The president went on to defend the legacy of Lee, who led the Confederate Army in the Civil War in defense of states' rights to maintain slavery.
The president infamously feted the Clemson Tigers, who won this year's NCAA football title, earlier this year with a feast of fast food, including McDonald's, Burger King and Domino's.
Baylor's national champion women's basketball team, led by head coach Kim Mulkey, will visit the White House on Monday. They will become the first women's sports team to visit the White House under Trump.