"I've had a fantastic time over the last 12 years, both on 'The X Factor' and 'American Idol.' And apart from being lucky enough to find some amazing talent on the shows, I have always had an incredible welcome from the American public (most of the time!)," Cowell said in a statement. "America, I'll see you soon!"
Fox pinned the move, announced Friday, on Cowell's decision to return to the U.K. version of the series.
"Last year, for a number of reasons, I had to make a decision to return to the U.K. version of 'The X Factor' in 2014," Cowell said. "So for now, I'm back to the U.K."
Fox Entertainment Chair Kevin Reilly added, "Unfortunately, there is no 'X Factor' USA without Simon Cowell, but we understand and support his decision to focus on the international formats and on the next phase of his personal life."
But even if Cowell hadn't been ready to move on, it wasn't certain whether Fox would bring the singing competition show back for a fourth season. The series, an import from Cowell's original U.K. show, just never seemed to catch on with American audiences and never attracted the 20 million viewers Cowell once predicted.
"He just couldn't make it work in America," said M.J. Santilli, who writes about both shows on her widely read blog MJ's Big Blog. "It was like an altered 'American Idol' format but not as good."
It did reach a respectable 12 million its first season, but the number steadily declined over the next two seasons, with just 6 million tuning in for last December's third season finale.
The show also suffered when compared to its NBC rival "The Voice."
"'The Voice' brought something completely new and fresh," Santilli said.
So what's next for Cowell?
Santilli said the U.K. flagship show of "The X Factor" is showing some of the same signs of aging as "Idol."
"He's going to go back and sit on the panel and tend to his baby for sure," she said, adding that the brand is still popular in other parts of the world.
But Santilli has no doubt that he'll return to the U.S.
For one thing he still has one successful show on the air, "America's Got Talent," of which he's the executive producer. And Santilli wouldn't be surprised to see him come back with something new.
"This is not a guy unfamiliar with adversity or having to start over," Santilli said.
Of course if he took some time off, people would understand.
At 54, Cowell is about to become a father for the first time, to a baby boy.
"The baby is coming at a great time," Santilli said, adding that pictures of the new dad are sure to make headlines.
"I wouldn't be surprised if it doesn't change him in some way," she said. "I think underneath it all, he's sort of a softie. I think he'll be a ridiculously dotting dad."