Ron Tom/ABC

Simon Cowell says that he and Paula Abdul "didn't like each other" when they began together as judges on "American Idol."

"I mean, the truth was, on the first two seasons, we couldn't travel on the same plane together," he told Barbara Walters for "The 10 Most Fascinating People of 2011."  "We really disliked each other that much."

He attributes the friction between them to "sexual tension…on her part."

"You think…Paula Abdul wanted you?" Walters asked.

"A million percent," he laughed.  "I considered it, but then I thought, I don't think the after would be as good as the before."

Cowell joked that he wouldn't want to wake up in the morning with Abdul: " I'd let her stay for a few hours, but then it would have to be goodnight."

The truth is, Cowell, who has never married, might be hard to live with. He stays up all night tending to his businesses around the world and then gets right to work in the morning.

Cowell is feeling good these days, despite having walked away from the most popular show on television so that he could do his own show, "The X Factor." The ratings haven't lived up to his expectations, but "I will get there eventually," he said.

And if he doesn't?

"I'll be unhappy.  I like winning.  If I play Monopoly with you, and you win, I'm not happy."

When asked what kind of a child he was, Cowell described himself as "a brat."

"Anything that would get me into trouble, I would do," he said.

And now, he admits to being "vain."  "I mean anyone on TV's vain, come on," he said.

Cowell hired Abdul as a judge on "The X Factor," and says their relationship has changed for the better.

"We obviously like each other, and I think you can see that," he said.

Watch "Barbara Walters presents the 1o Most Fascinating People of the Year" Wednesday, Dec. 14 at 9:30 p.m. ET