In a dramatic move at today's Television Critic's Association panel for FOX, Simon Cowell signed a new contract for the network which will find him leaving "American Idol" as a judge after this season.  Cowell will produce and judge a new U.S. version of the hit British talent show "X-Factor" which will begin airing in the Fall of 2011 on FOX.

"We've always had a fantastic relationship with FOX. I've tried to be loyal to the people who gave me my break on television," Simon says, adding that there had been discussions about his staying on both shows, but he didn't think it was practical or functional.

"When the player retires, the football team will continue to be successful," Simon added, saying that he believes that "Idol" will be protected without him and that the show will continue to be No. 1 without him.

"Simon is irreplaceable, but it will be incumbent upon us to make sure that the show is vital," Fox TV head Peter Rice says.

Simon compares this to the uncertainty of the first year of "Idol."  "I genuinely didn't know if we were going to be here for six months or six days." He adds, "I loved that first year when I just didn't know if it was going to work or not and I feel that now about 'X Factor.'"

What fans really want to know is whether former "Idol" host Paula Abdul could have a home on "X-Factor." 

"Look. I've gone on the record. I adore Paula. Whatever happens, I will be working with her in some capacity, because I miss her. But like I say, you're just going to have to watch this space," Simon says.

As for "X-Factor" itself, Simon says the show involves the way people connect with audiences in live auditions and live performances. He calls it a "David vs. Goliath" format, that makes it more diverse and honest. "Right now? Who's got the biggest X-factor in the world?  Lady Gaga," Simon cracks

Like the current "American Idol" format there will be multiple weekly episodes of "X-Factor." The show is a bit different from its predecessor however.  Simon says the age range of a contestant could be as low as 14 or possibly lower and that there will be no upper limit.

"You can literally be 100," he says. Also, contestants can enroll as groups. Then, the judges are responsible as mentors as well, making a new level of responsible. "So essentially, the judges are judged as well as the singer," Simon says.

"I don't like rules," Simon says of the advantages for "X Factor" over "Idol." He references Susan Boyle as why he likes doing shows without an upper age limit. "Without a show like that, she couldn't have landed a recording contract," Simon says.

Simon notes that he'd always said he was leaving after this contract and people mistakenly just thought he was negotiating. "I want to leave 'Idol' this year bigger and better than it's been this year, genuinely I do," Simon says. 

Who does Simon think should fill his "American Idol" chair?

"I'm going to suggest that Ryan do it, because he can have another job then." His only recommendation is that "Idol" get somebody who knows what they're talking about. "This country, thank God, has still got thousands and thousands of talented people," Simon insists.

Look for more on this story and the future of Simon Cowell, "American Idol" and "X-Factor" on HitFix.