He said, "Well, first of all, my parents were both loving, and they still are. When they were divorced, I was one years old, but they were always there for me. When I referred to the violence in the home, it was with my mom's husband -- a couple of husbands. And I do remember getting up in the middle of the night and, you know, having to be the man of the family and come and rescue her and getting knocked around pretty good."
Brown said his experience has made him "appreciate my strong family."
The father of two credits his sense of humor as his saving grace for his now 23-year marriage to Boston-based television reporter Gail Huff.
"There's nothing I wouldn't do for them," he said.
To watch all of our exclusive interview with Scott Brown, go to ABCNews.com or click HERE for the full player.