Smart called the Golden Eagles on Thursday and turned down the job after the school offered him a lucrative contract shortly after Buzz Williams left for Virginia Tech last week, sources told ESPN.com.
Smart has spent the past five seasons with the Rams, taking them to the NCAA tournament each of the past four years. He took the program from the First Four to the Final Four in 2011.
VCU lost to Stephen F. Austin in the second round of the NCAA tournament last week.
Smart's current deal with VCU runs through 2023.
Sources told ESPN.com that Marquette -- which had focused its search on Smart the past few days -- now will move on to other candidates. Former UCLA coach Ben Howland, Green Bay's Brian Wardle (who played and was an assistant at Marquette) and Syracuse coach-in-waiting Mike Hopkins are all in the mix, sources said.