Mitt Romney will likely struggle nationally, particularly with women, because his father was "born on a polygamy commune in Mexico," said Montana Gov. Brian Schweitzer.

Schweitzer, a Democrat, made the comment in an interview with the Daily Beast about Romney's chances in swing states.

The governor said Romney will also have trouble attracting Latino voters, adding that it was  "kind of ironic given that his family came from a polygamy commune in Mexico, but then he'd have to talk about his family coming from a polygamy commune in Mexico, given the gender discrepancy."

That part of Romney's family history will particularly irk women, Schweitzer said, because they are  "not great fans of polygamy, 86 percent were not great fans of polygamy." Schweitzer added that he did not believe Romney, raised in the Mormon Church and married to one woman for more than four decades, is a polygamist.

"I am not alleging by any stretch that Romney is a polygamist and approves of [the] polygamy lifestyle, but his father was born into [a] polygamy commune in Mexico," Schweitzer said. Romney's grandfather and other Mormons settled in Mexico following an 1890 ban on polygamy, a practice the church calls "plural marriage."

Many Mormons moved to Mexico in order to continue practicing polygamy, though Romney's grandfather was married to only one woman. Romney's father George, who served as governor of Michigan, was born in Mexico in 1907. Repeated calls to Schweitzer's office seeking comment were not returned.

The Romney campaign did not respond for comment, but Romney told Fox News in an interview "My dad's dad was not a polygamist."

ABC News' Emily Friedman contributed to this report