Matt Damon is the envy of some of his famous friends, in particular, Brad Pitt.

"I remember telling him that I walk my kids to school, and his face just fell. He was very kind, but he was like, 'You b****rd.' Because he should be able to do that, too. And he can't," Damon told Esquire magazine in its August issue.

Damon said he's been able to keep his personal life mostly out of the public eye because he married "a civilian."

"I got lucky. I fell in love with a civilian," he said about his wife, Luciana, a former bartender. "Not an actress and not a famous actress at that. Because then the attention doesn't double - it grows exponentially. Because then suddenly everybody wants to be in your bedroom."

Esquire

"But I don't really give them anything. If I'm not jumping up and down on a bar, or lighting something on fire or cheating on my wife, there's not really any story to tell," said the 42-year-old father of four. "They can try to stake me out, but they're always going to get the same story - middle-aged married guy with four kids."

That could change, as Damon revealed he and his family are about to move to Los Angeles, to a home just down the street from his best pal Ben Affleck, with whom he's starting a production company.

Compared to Affleck, a childhood friend, and some of his other Hollywood peers, Damon got a late start on his career.

He said his mother, Nancy Carlsson-Paige, an early childhood education professor, believed putting a child onstage or in front of a camera was akin to "child abuse."

"My mother thought it was child abuse. She literally did," Damon told Esquire.

So while he acted in school plays and worked as an extra in Boston, Damon didn't get his first movie role until he was 18, in the movie "Mystic Pizza," starring Julia Roberts.

In his latest movie, "Elysium," playing opposite Jodie Foster, Damon got the chance to ask the former child star if his mother's assessment was right.

"I figured, if anyone's going to know, it should be her, right?" he said. "So I asked her. And she sort of smiled and said, 'It depends on the child.'"