Comments on: Who won the Netflix Prize? Ensemble or BellKors Pragmatic Chaos? http://ebiquity.umbc.edu/blogger/2009/07/27/who-won-the-netflix-prize-ensemble-or-bellkors-pragmatic-chaos/ EBB is the ebiquity research group\\\'s blog at the University of Maryland, Baltimore County (UMBC). We focus on technologies that facilitate the design, implementation and control of distributed, intelligent information systems -- mobile and pervasive computing, ad hoc networking, multiagent systems, knowledge representation and reasoning, and the semantic web. As the tides of technology ebb and flow, we hope the good ideas wash up on our beach and the bad ones drift back out to sea. Tue, 06 Dec 2011 19:29:16 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Craig Schmidt http://ebiquity.umbc.edu/blogger/2009/07/27/who-won-the-netflix-prize-ensemble-or-bellkors-pragmatic-chaos/comment-page-1/#comment-29982 Craig Schmidt Tue, 28 Jul 2009 15:13:29 +0000 http://ebiquity.umbc.edu/blogger/?p=2283#comment-29982 > It seems strange that Netflix called the Bellkor team first, > since according to the Leaderboard the Ensemble team submitted the top entry. The way it works is that Netflix splits the test set into two unspecified parts. They put the value of the first part on the leaderboard, but it is the value of the second that really counts. This was done to prevent using the leaderboard score to train to the test set. It would appear the BellKor just did better on the true test set, despite what it says on the leaderboard. > It seems strange that Netflix called the Bellkor team first,
> since according to the Leaderboard the Ensemble team submitted the top entry.

The way it works is that Netflix splits the test set into two unspecified parts. They put the value of the first part on the leaderboard, but it is the value of the second that really counts. This was done to prevent using the leaderboard score to train to the test set.

It would appear the BellKor just did better on the true test set, despite what it says on the leaderboard.

]]> By: Ric http://ebiquity.umbc.edu/blogger/2009/07/27/who-won-the-netflix-prize-ensemble-or-bellkors-pragmatic-chaos/comment-page-1/#comment-29981 Ric Tue, 28 Jul 2009 13:45:07 +0000 http://ebiquity.umbc.edu/blogger/?p=2283#comment-29981 Tim wrote: “It seems strange that Netflix called the Bellkor team first, since according to the Leaderboard the Ensemble team submitted the top entry.”. Maybe I can explain a bit. The competitors are judged on a so-called “Test set”, on which Netflix reveals no details, even not reflecting performance on it in the Leaderboard. The reason for this secrecy is preventing gaming the competition, by building algorithms specifically tuned to that Test set. It appears that “BellKor’s Prgamatic Chaos” has best performance on this important set. On the other hand, the Leaderboard reflects performance on the so-called “Quiz set”, which serves as a good approximation to the “Test set”. Apparently, “The Ensemble” leads on the “Quiz set”, but it isn’t the one that really counts. Tim wrote: “It seems strange that Netflix called the Bellkor team first, since according to the Leaderboard the Ensemble team submitted the top entry.”.
Maybe I can explain a bit.
The competitors are judged on a so-called “Test set”, on which Netflix reveals no details, even not reflecting performance on it in the Leaderboard.
The reason for this secrecy is preventing gaming the competition, by building algorithms specifically tuned to that Test set.
It appears that “BellKor’s Prgamatic Chaos” has best performance on this important set.
On the other hand, the Leaderboard reflects performance on the so-called “Quiz set”, which serves as a good approximation to the “Test set”.
Apparently, “The Ensemble” leads on the “Quiz set”, but it isn’t the one that really counts.

]]> By: Netflix Prize contest closes; Ensemble wins http://ebiquity.umbc.edu/blogger/2009/07/27/who-won-the-netflix-prize-ensemble-or-bellkors-pragmatic-chaos/comment-page-1/#comment-29975 Netflix Prize contest closes; Ensemble wins Mon, 27 Jul 2009 23:46:41 +0000 http://ebiquity.umbc.edu/blogger/?p=2283#comment-29975 [...] Ensemble wins « Ensemble leads Netflix Prize contest, besting BellKors Pragmatic Chaos Who won the Netflix Prize? Ensemble or BellKors Pragmatic Chaos? [...] [...] Ensemble wins « Ensemble leads Netflix Prize contest, besting BellKors Pragmatic Chaos Who won the Netflix Prize? Ensemble or BellKors Pragmatic Chaos? [...]

]]>