"I remember him saying, 'One day, boys, I'm going to get myself 20 horses,'" said Randy Sussman, another friend from the lean old days. "Now he's got 120. It's amazing to see what he's done."
But his meteoric rise has not come without controversy. In a sport rife with drug cheats, Dutrow is accused of being one of the worst.
This week, Dutrow talked about "one drug positive four, five, six years ago." But the Association of Racing Commissioners International database paints a much different picture.
Dutrow's ARCI rap sheet contains 72 entries, including fines and suspensions in Maryland, California, New York, Florida, Delaware and New Jersey.
Several of those are for his own personal conduct; many involve marijuana. Others include: an alleged attempt to pass forged checks in Maryland; an attempt to provide a false urine sample "by means of an apparatus concealed on his person"; and a failure to report on a New York license application a 1991 criminal conviction in Nevada.
And he's been fined or suspended at least once every year since 2000 for doping issues. In 2000, a barn search in New York produced "an injectible vitamin which is forbidden." In '01, a horse had excessive Lasix -- an anti-bleeding medication -- in its system. In '02, Dutrow "failed to follow Lasix procedures." In '03, a horse tested positive for Mepivacaine. From '04 through an '08 fine in Florida, there were citations regarding Lasix, Clenbuterol, Phenylbutazone and Oxyphenbutazone.
He served a 60-day suspension in 2005 after two of his horses tested positive for banned substances and for a claiming violation. Then, in 2007, he served an additional 14-day suspension and was fined $25,000 for violating conditions of his suspension by having contact with his stable.
No wonder Ray Paulick, former editor of the Blood-Horse magazine, a leading industry trade publication, wrote earlier in the year: "As a rule, a 2-year-old maiden race on turf at Saratoga (Big Brown's first race) is not the launchpad for the Kentucky Derby, but trainer Rick Dutrow has never been accused of being overly concerned with the rules."
Seemed like a natural question, so I asked it: Why have you broken so many rules?
"Why? I couldn't answer that," Dutrow said. "I know I wasn't working right."
But his horses are running fast these days. And that's all that seems to matter to his owners.
"Some of them were minor offenses," said Richard Schiavo, one of the owners of Big Brown. "I don't think it's something that really concerns us."
It concerns many others in racing. Drug issues have wounded the sport's credibility with its fan base.
Last year, trainer Steve Asmussen came off suspension to run third in the Derby with Curlin and then win the Preakness with him; he's back this year with Pyro. Todd Pletcher, who has two Derby entries, has served a suspension for drugged horses as well. Patrick Biancone, second in the 2004 Derby with Lion Heart, currently is serving a one-year suspension after racing authorities found cobra venom -- which is used to numb horses' feet -- in his Kentucky barn.
So now you have to wonder: Would Rick Dutrow's presence in the Churchill winner's circle Saturday be bad for racing?
"The trouble I've been in shouldn't take anything away from Big Brown," Dutrow said. "I don't see it. I haven't done anything bad around Big Brown ...
"You're here at the Derby. Who would want to do something that isn't right? This is all clean, all good. Anyone who would do something wrong at the Derby, that's not us. That would be despicable.
"We treat the game respectfully."
The Derby gods might get the final say on whether Rick Dutrow has shown the game the proper respect.
Pat Forde is a senior writer for ESPN.com. He can be reached at ESPN4D@aol.com.