Who won the Netflix Prize? Ensemble or BellKors Pragmatic Chaos?
Tim Finin, 6:43pm 27 July 2009Who won the Netflix Prize? Ensemble or BellKors Pragmatic Chaos
Who won the Netflix Prize? According to a post in the NYT Bits blog, Netflix Challenge Ends, But Winner Is In Doubt, it’s still very much up in the air.
” So The Ensemble won, right? Not necessarily. In an e-mail message Sunday night, Chris Volinsky, a scientist at AT&T Research and a leader of the BellKor’s team, said: “Our team is in first place as we were contacted by Netflix to validate our entry.” And in an online forum, another member of the BellKor team, Yehuda Koren, a researcher for Yahoo in Israel, said his team had “a better Test score than The Ensemble,” despite what the rival team submitted for the leaderboard.
So is BellKor the winner? Certainly not yet, according to a Netflix spokesman, Steve Swasey. “There is no winner,” he said.
A winner, Mr. Swasey said, will probably not be announced until sometime in September at an event hosted by Reed Hastings, Netflix’s chief executive. The movie rental company is not holding off for maximum P.R. effect, Mr. Swasey said, but because the winner has not yet been determined.
The Web leaderboard, he explained, is based on what the teams submit. Next, Netflix’s in-house researchers and outside experts have to validate the teams’ submissions, poring over the submitted code, design documents and other materials. “This is really complex stuff,” Mr. Swasey said.
A leading member of The Ensemble, Domonkos Tikk, a Hungarian computer scientist, did not sound too hopeful. “We didn’t get any notification from Netflix,” Mr. Tikk said in a phone interview from Hungary. “So I think the chances that we won are very slight. It was a nice try.”
It seems strange that Netflix called the Bellkor team first, since according to the Leaderboard the Ensemble team submitted the top entry.
UPDATE 2/28: Today’s NYT has a good article on the Netflix Prize and the role of teamwork for developing machine learning systems, Netflix Competitors Learn the Power of Teamwork.
Related posts:
Ebiquity Blog
July 27th, 2009 at 6:46 pm
[...] Ensemble wins « Ensemble leads Netflix Prize contest, besting BellKors Pragmatic Chaos Who won the Netflix Prize? Ensemble or BellKors Pragmatic Chaos? [...]
July 28th, 2009 at 8:45 am
Tim wrote: “It seems strange that Netflix called the Bellkor team first, since according to the Leaderboard the Ensemble team submitted the top entry.”.
Maybe I can explain a bit.
The competitors are judged on a so-called “Test set”, on which Netflix reveals no details, even not reflecting performance on it in the Leaderboard.
The reason for this secrecy is preventing gaming the competition, by building algorithms specifically tuned to that Test set.
It appears that “BellKor’s Prgamatic Chaos” has best performance on this important set.
On the other hand, the Leaderboard reflects performance on the so-called “Quiz set”, which serves as a good approximation to the “Test set”.
Apparently, “The Ensemble” leads on the “Quiz set”, but it isn’t the one that really counts.
July 28th, 2009 at 10:13 am
> It seems strange that Netflix called the Bellkor team first,
> since according to the Leaderboard the Ensemble team submitted the top entry.
The way it works is that Netflix splits the test set into two unspecified parts. They put the value of the first part on the leaderboard, but it is the value of the second that really counts. This was done to prevent using the leaderboard score to train to the test set.
It would appear the BellKor just did better on the true test set, despite what it says on the leaderboard.