Netflix Prize contest closes; Ensemble wins

July 26th, 2009

Netflix has announced that the Netflix Prize contest is now closed. Presumably, The Ensemble is the winner, subject to final qualification.

“We are delighted to report that, after almost three years and more than 43,000 entries from over 5,100 teams in over 185 countries, the Netflix Prize Contest stopped accepting entries on 2009-07-26 18:42:37 UTC. The closing of the contest is in accordance with the Rules — thirty (30) days after a submitted prediction set achieved the Grand Prize qualifying RMSE on the quiz subset.

Qualified entries will be evaluated as described in the Rules. We look forward to awarding the Grand Prize, which we expect to announce in a few weeks. However if a Grand Prize cannot be awarded because no submission can be verified by the judges, the Contest will reopen. We will make an announcement on the Forum after the Contest judges reach a decision.”

So what’s left for the judges to do. The rules say that “a panel of senior Netflix engineers and qualified independent judges” need to “ensure that the provided algorithm description and source code could reasonably have generated the prediction sets submitted”. To do this, the candidate winner must produce the algorithm along with a description of who it works. And, of course, before receiving the prize the winner has to grant Netflix

“an irrevocable, royalty free, fully paid up, worldwide non-exclusive license under the Participants’ copyrights, patents or other intellectual property rights in the winning algorithm (“Winning Algorithm”) to reproduce, distribute, display, and create derivative works from the Winning Algorithm and also to make, have made, use, sell, offer for sale, and import products that would otherwise infringe the Winning Algorithm.”

The Netflix Prize was a great idea and generated a lot of interest around the world. It’s been good for the field of AI and its machine learning sub-field, especially. Congratulations to the Ensemble team and condolences to BellKor’s Pragmatic Chaos. I wish there could have been two winners.

UPDATE 2/27: Wait! The winner is still in doubt.


Ensemble leads Netflix Prize contest, besting BellKors Pragmatic Chaos

July 26th, 2009

The race for the Netflix Prize is still on.

With just one day left in the 30 day last call period before BellKor’s Pragmatic Chaos (BKPC) was awarded the $1M Netflix Prize for a better movie recommender system, another team has broken the 10% improvement threshold and taken the lead by one hundredth of one percent — The Ensemble.

The Ensemble was formed by the merger of two existing Netflix Prize teams that had been ranked second and third behind BKPC: ‘Grand Prize Team’ and ‘Opera Solutions and Vandelay United’. Here’s how The Ensemble describes it’s genesis.

The crowd is indeed wiser than the individual.

The 10% barrier once seemed distant and insurmountable. But when the contest’s “last call” heralded the heroic achievements of BellKor’s Pragmatic Chaos, the rest of the crowd pondered, and asked why the barrier couldn’t be broken twice.

And lo, as if powered by gravity, Grand Prize Team and Vandelay Industries! began to draw in more and more members. And Vandelay went on to join forces with Opera Solutions, and then Vandelay and Opera united with Grand Prize Team, and then … and then … well, things got so complex we decided just to call ourselves The Ensemble.

We can be sure that there will be a lot of Netflix Prize activity in the coming weeks and maybe months as these two teams compete and perhaps more mergers create super-teams. BKPC and Ensemble could even decide to merge and share the prize. Watch the Netflix Leaderboard for the latest ranking.

UPDATE: I had assumed the 30 day last call would reset with each new leader, like auctions on ebay. Not so. The prize will be won (and lost) today! Here’s the relevant section in the rules:

“To qualify for the Grand Prize the RMSE of a Participant’s submitted predictions on the test subset must be less than or equal to 90% of 0.9525, or 0.8572 (the “qualifying RMSE”). After three (3) months have elapsed from the start of the Contest, when the RMSE of a submitted prediction set on the quiz subset improves beyond the qualifying RMSE an electronic announcement will inform all registered Participants that they have thirty (30) days to submit additional candidate prediction sets to be considered for judging. At the end of this period, qualifying submissions will be judged (see Judging below) in order of the largest improvement over the qualifying RMSE on the test subset. In the case of tied RMSE values on the test subsets, the submission received earliest by the Site will be judged first.”

The August 2009 CACM has a short note, Just for You (pdf), on recommender systems and the Netflix prize by BKPC member Don Monroe that includes a visualization by Ensemble member Chris Hefele.

Spotted on Hacker News. See Techcrunch also.

UPDATE II: The Netflix Prize contest has closed.