"If you have high turnover of cells, if there's environmental toxins, that may just interfere with your body's ability to make healthy cells," he said.
Given the image of a macho man with cigar in hot tub as the pinnacle of manliness, Moalem said, what we know about sperm production shows that it's in fact "as far away from virility as you can get."
Avoid, But For Other Reasons
As debates about drug legalization arise, the science isn't entirely clear on how these substances might affect our national fertility rates.
"The data around marijuana is not definitive but highly suggestive. It can lower the quality of sperm as well as probably the count," said Morgentaler.
"There's not a lot of good data out there about the effect of marijuana on sperm or sperm production," said Williams. "There are a few studies about heavy marijuana use possibly impacting men's hormone levels and the quality of the sperm, but these are smaller studies."
Moalem notes that some risks can arise from cocaine use affecting circulation to the scrotal region.
"You could have a stroke in your testicle," he said.
Some of the harm might actually arise from legal narcotics, as men on narcotic pain medications can have very low testosterone.
"If men are taking certain pain medications, certain pain medications contain narcotics, narcotics can affect men's hormone levels," said Williams.
"The data on vitamins and supplements for men and sperm numbers -- I don't think there's any definitive answer around them," said Morgentaler.
That isn't to say that men shouldn't focus on getting vitamins and minerals into their diets naturally.
"Having a health lifestyle and eating a well-rounded, balanced diet is important for a variety of reasons," said Williams.
He noted that a diet rich in antioxidants can help when sperm quality.
"When you start talking about food or nutritional supplements, there are a number of labs that show improved sperm quality in the presence of antioxidants," he said.
"I think the take home message is just a well-rounded, balanced diet. Excess of anything is not necessarily a good thing," said Williams.
Moalem pointed out that adding more spinach to a diet for its folate can have added benefits, even if it's unclear if or how it helps men's sperm.
"It's healthy anyways and it's not something people think about," he said. "And it's good for women who are trying to conceive as well."
While the benefits of folate aren't clear for men, it has been shown to benefit pregnant women, and presumably a couple trying to conceive is also eating the bulk of their meals together.
"A lot of people have heard that red meat is supposed to be bad for you. The good news is it's not something bad for your sperm," said Morgentaler.
Because of the number of groups that oppose meat eating for moral reasons, any study showing harm from meat eating is likely to be promoted.
Last week a story in the British press on possible hazards of meat eating drew enough attention that the National Health Service (NHS) felt compelled to let people know that the study was not as significant as the promotion made it out to be.