Who won the Netflix Prize? Ensemble or BellKors Pragmatic Chaos?

July 27th, 2009

Who won the Netflix Prize? Ensemble or BellKors Pragmatic Chaos

Who won the Netflix Prize? According to a post in the NYT Bits blog, Netflix Challenge Ends, But Winner Is In Doubt, it’s still very much up in the air.

” So The Ensemble won, right? Not necessarily. In an e-mail message Sunday night, Chris Volinsky, a scientist at AT&T Research and a leader of the BellKor’s team, said: “Our team is in first place as we were contacted by Netflix to validate our entry.” And in an online forum, another member of the BellKor team, Yehuda Koren, a researcher for Yahoo in Israel, said his team had “a better Test score than The Ensemble,” despite what the rival team submitted for the leaderboard.

So is BellKor the winner? Certainly not yet, according to a Netflix spokesman, Steve Swasey. “There is no winner,” he said.

A winner, Mr. Swasey said, will probably not be announced until sometime in September at an event hosted by Reed Hastings, Netflix’s chief executive. The movie rental company is not holding off for maximum P.R. effect, Mr. Swasey said, but because the winner has not yet been determined.

The Web leaderboard, he explained, is based on what the teams submit. Next, Netflix’s in-house researchers and outside experts have to validate the teams’ submissions, poring over the submitted code, design documents and other materials. “This is really complex stuff,” Mr. Swasey said.

A leading member of The Ensemble, Domonkos Tikk, a Hungarian computer scientist, did not sound too hopeful. “We didn’t get any notification from Netflix,” Mr. Tikk said in a phone interview from Hungary. “So I think the chances that we won are very slight. It was a nice try.”

It seems strange that Netflix called the Bellkor team first, since according to the Leaderboard the Ensemble team submitted the top entry.

UPDATE 2/28: Today’s NYT has a good article on the Netflix Prize and the role of teamwork for developing machine learning systems, Netflix Competitors Learn the Power of Teamwork.

Netflix Prize contest closes; Ensemble wins

July 26th, 2009

Netflix has announced that the Netflix Prize contest is now closed. Presumably, The Ensemble is the winner, subject to final qualification.

“We are delighted to report that, after almost three years and more than 43,000 entries from over 5,100 teams in over 185 countries, the Netflix Prize Contest stopped accepting entries on 2009-07-26 18:42:37 UTC. The closing of the contest is in accordance with the Rules — thirty (30) days after a submitted prediction set achieved the Grand Prize qualifying RMSE on the quiz subset.

Qualified entries will be evaluated as described in the Rules. We look forward to awarding the Grand Prize, which we expect to announce in a few weeks. However if a Grand Prize cannot be awarded because no submission can be verified by the judges, the Contest will reopen. We will make an announcement on the Forum after the Contest judges reach a decision.”

So what’s left for the judges to do. The rules say that “a panel of senior Netflix engineers and qualified independent judges” need to “ensure that the provided algorithm description and source code could reasonably have generated the prediction sets submitted”. To do this, the candidate winner must produce the algorithm along with a description of who it works. And, of course, before receiving the prize the winner has to grant Netflix

“an irrevocable, royalty free, fully paid up, worldwide non-exclusive license under the Participants’ copyrights, patents or other intellectual property rights in the winning algorithm (“Winning Algorithm”) to reproduce, distribute, display, and create derivative works from the Winning Algorithm and also to make, have made, use, sell, offer for sale, and import products that would otherwise infringe the Winning Algorithm.”

The Netflix Prize was a great idea and generated a lot of interest around the world. It’s been good for the field of AI and its machine learning sub-field, especially. Congratulations to the Ensemble team and condolences to BellKor’s Pragmatic Chaos. I wish there could have been two winners.

UPDATE 2/27: Wait! The winner is still in doubt.

Ensemble leads Netflix Prize contest, besting BellKors Pragmatic Chaos

July 26th, 2009

The race for the Netflix Prize is still on.

With just one day left in the 30 day last call period before BellKor’s Pragmatic Chaos (BKPC) was awarded the $1M Netflix Prize for a better movie recommender system, another team has broken the 10% improvement threshold and taken the lead by one hundredth of one percent — The Ensemble.

The Ensemble was formed by the merger of two existing Netflix Prize teams that had been ranked second and third behind BKPC: ‘Grand Prize Team’ and ‘Opera Solutions and Vandelay United’. Here’s how The Ensemble describes it’s genesis.

The crowd is indeed wiser than the individual.

The 10% barrier once seemed distant and insurmountable. But when the contest’s “last call” heralded the heroic achievements of BellKor’s Pragmatic Chaos, the rest of the crowd pondered, and asked why the barrier couldn’t be broken twice.

And lo, as if powered by gravity, Grand Prize Team and Vandelay Industries! began to draw in more and more members. And Vandelay went on to join forces with Opera Solutions, and then Vandelay and Opera united with Grand Prize Team, and then … and then … well, things got so complex we decided just to call ourselves The Ensemble.

We can be sure that there will be a lot of Netflix Prize activity in the coming weeks and maybe months as these two teams compete and perhaps more mergers create super-teams. BKPC and Ensemble could even decide to merge and share the prize. Watch the Netflix Leaderboard for the latest ranking.

UPDATE: I had assumed the 30 day last call would reset with each new leader, like auctions on ebay. Not so. The prize will be won (and lost) today! Here’s the relevant section in the rules:

“To qualify for the Grand Prize the RMSE of a Participant’s submitted predictions on the test subset must be less than or equal to 90% of 0.9525, or 0.8572 (the “qualifying RMSE”). After three (3) months have elapsed from the start of the Contest, when the RMSE of a submitted prediction set on the quiz subset improves beyond the qualifying RMSE an electronic announcement will inform all registered Participants that they have thirty (30) days to submit additional candidate prediction sets to be considered for judging. At the end of this period, qualifying submissions will be judged (see Judging below) in order of the largest improvement over the qualifying RMSE on the test subset. In the case of tied RMSE values on the test subsets, the submission received earliest by the Site will be judged first.”

The August 2009 CACM has a short note, Just for You (pdf), on recommender systems and the Netflix prize by BKPC member Don Monroe that includes a visualization by Ensemble member Chris Hefele.

Spotted on Hacker News. See Techcrunch also.

UPDATE II: The Netflix Prize contest has closed.

AAAI study examines long-term AI futures and impact on society

July 25th, 2009

John Markoff has an article for tomorrow’s New York Times, Scientists Worry Machines May Outsmart Man on a recent AAAI study on the future of AI.

“A robot that can open doors and find electrical outlets to recharge itself. Computer viruses that no one can stop. Predator drones, which, though still controlled remotely by humans, come close to a machine that can kill autonomously. Impressed and alarmed by advances in artificial intelligence, a group of computer scientists is debating whether there should be limits on research that might lead to loss of human control over computer-based systems that carry a growing share of society’s workload, from waging war to chatting with customers on the phone.”

The study was commissioned by AAAI to “to explore and address potential long-term societal influences of AI research and development”. Look for a report published by AAAI later this year. The study involved twenty-five participants who were divided into three subgroups: on concerns, control and guidelines, the nature and timing of disruptive advances, and ethical and legal issues.

There was a panel session earlier this month at IJCAI where some of the study participants discussed highlights from the study. Hopefully this was filmed and the results will be added to the videolectures.net IJCAI09 collection.

While I am generally skeptical of an impending technological singularity, which seems to sum up many of the concerns some have, there are aspects of the future that I do wonder about. At the top of my list is what will happen when virtually all of human knowledge is published on the Web (as it nearly is now) in a for that machines can understand. I’m pretty sure that this will happen in the next decade or two, either through the current Semantic Web approach (as a web of data) or by gradually improving techniques for machine understanding of human languages and images.

Canada: facebook violates privacy law

July 17th, 2009

APF and others report that Canada considers facebook’s practices to violate its privacy law.

“Canadian officials on Thursday said Facebook was breaking national privacy law by holding on to personal information from closed accounts at the social-networking service. A Canada privacy commission report expressed “an overarching concern” that privacy information Facebook provides its more than 250 million users is “often confusing or incomplete.” Facebook said it is working with the commission to resolve its concerns in ways that safeguard privacy without disrupting user-experiences at the world’s most popular online social-networking community.”

The Office of the Privacy Commissioner of Canada conducted an investigation into a wide-ranging complaint about facebook’s privacy practices filed by the Canadian Internet Policy and Public Interest Clinic (CIPPIC).

In a July 16 press release describes the highlights of the Report of Findings into the Complaint Filed by the Canadian Internet Policy and Public Interest Clinic (CIPPIC) against Facebook Inc.. These include the following:

“An overarching concern was that, although Facebook provides information about its privacy practices, it is often confusing or incomplete. For example, the “account settings” page describes how to deactivate accounts, but not how to delete them, which actually removes personal data from Facebook’s servers.

The investigation also raised significant concerns around the sharing of users’ personal information with third-party developers creating Facebook applications such as games and quizzes. (There are more than 950,000 developers in some 180 countries.) Facebook lacks adequate safeguards to effectively restrict these outside developers from accessing profile information, the investigation found.

The investigation also found that Facebook has a policy of indefinitely keeping the personal information of people who have deactivated their accounts – a violation of the Personal Information Protection and Electronic Documents Act (PIPEDA), Canada’s private-sector privacy law. The law is clear that organizations must retain personal information only for as long as is necessary to meet appropriate purposes.”

CFP: Journal of Web Semantics issue on Semantic Search

July 17th, 2009

Journal of Web SemanticsYong Yu and Rudi Studer are editing a special issue of the Journal of Web Semantics on Semantic Search that will appear in the summer 2010. Papers are due 20 January 2010 and decisions will will be sent two months later. Relevant topics include:

  • Information retrieval tasks on the Semantic Web
  • Incentives and interaction paradigms for resource annotation
  • Interaction paradigms for semantic search
  • Semantic technologies for query interpretation, refinement and routing
  • Modeling expressive resource descriptions
  • natural language processing and information extractions for the acquisition of resource descriptions
  • Scalable repositories and infrastructures for semantic search
  • Crawling, storing and indexing of expressive resource descriptions
  • fusion of semantic search results on the Semantic Web
  • Algorithms for matching expressive queries and resource descriptions
  • Algorithms and procedure to deal with vagueness, incompleteness and inconsistencies in semantic search
  • Evaluation methodologies for semantic search
  • Standard datasets and benchmarks for semantic search

See the full call for papers for more information.

Google Reader gets more social

July 16th, 2009

The most frequent complaint about facebook I’ve seen is that it provides a button to show you like an item, but not one for dislike. Google Reader recently added new social features the ability for users to mare a post as liked but it also doesn’t allow you to indicate your dislike. You can unlike an item that you previously had liked, but that just gets you back to a neutral stance.

Google Reader gets more social

It’s probably a prudent choice, aimed at keeping things civil. But there are two schools of thought about the old adage “If you don’t have anything nice to say about someone …”, one of which ends with “come sit next to me.”.

The first time you like a post on Google Reader it warns you that it’s a public act. Indeed, clicking on the “N people liked this” link at the top of a post in Reader shows you the Google names of readers who liked it. You can click through to their Google profiles or to see a list of other liked and shared items. Public indeed! At least on facebook your likes are visible only to people who can see the corresponding item.

I think Google Reader’s new social features look like they might be useful, but time will tell.

Tracking news memes: lipstick on Joe the Plumber

July 13th, 2009

Here’s a great graphic on the rise and fall of memes in the news from KDD 2009 paper, Meme-tracking and the Dynamics of the News Cycle by Leskovec, Backstrom and Kleinberg. Click on the image to see a larger version.

Tracking memes in news

Here’s the paper’s abstract.

“Tracking new topics, ideas, and “memes” across the Web has been an issue of considerable interest. Recent work has developed methods for tracking topic shifts over long time scales, as well as abrupt spikes in the appearance of particular named entities. However, these approaches are less well suited to the identification of content that spreads widely and then fades over time scales on the order of days —the time scale at which we perceive news and events. We develop a framework for tracking short, distinctive phrases that travel relatively intact through on-line text; developing scalable algorithms for clustering textual variants of such phrases, we identify a broad class of memes that exhibit wide spread and rich variation on a daily basis. As our principal domain of study, we show how such a meme-tracking approach can provide a coherent representation of the news cycle—the daily rhythms in the news media that have long been the subject of qualitative interpretation but have never been captured accurately enough to permit actual quantitative analysis. We tracked 1.6 million mainstream media sites and blogs over a period of three months with the total of 90 million articles and we find a set of novel and persistent temporal patterns in the news cycle. In particular, we observe a typical lag of 2.5 hours between the peaks of attention to a phrase in the news media and in blogs respectively, with divergent behavior around the overall peak and a “heartbeat”-like pattern in the handoff between news and blogs. We also develop and analyze a mathematical model for the kinds of temporal variation that the system exhibits.”

(via the NYT)

UK Teen on how teenagers consume media

July 13th, 2009

The Financial Times has an article, Note by ‘teenage scribbler’ causes sensation, on a research study written by a 15 year old Morgan Stanley intern on the new and old media habits of UK youth.

“Morgan Stanley’s European media analysts asked Matthew Robson, one of the bank’s interns from a London school, to describe his friends’ media habits.

“Teenagers do not use Twitter,” he pronounced. Updating the micro-blogging service from mobile phones costs valuable credit, he wrote, and “they realise that no one is viewing their profile, so their tweets are pointless”.

His peers find it hard to make time for regular television, and would rather listen to advert-free music on websites such as Last.fm than tune into traditional radio. Even online, teens find advertising “extremely annoying and pointless”.

Their time and money is spent instead on cinema, concerts and video game consoles which, he said, now double as a more attractive vehicle for chatting with friends than the phone.

Mr Robson had little comfort for struggling print publishers, saying no teenager he knew regularly reads a newspaper since most “cannot be bothered to read pages and pages of text” rather than see summaries online or on television.”

You can read his report on How Teenagers Consume Media online.

The Guardian also has a story today, Twitter is not for teens, on the intern’s report.

detexify: draw a symbol, get the LaTeX command

July 12th, 2009

The perfect document preparation system has yet to be invented, and I’ve tried many over the years, starting with TJ6. It’s surely impossible for any one system to be best, given the range of documents most of us have to produce: letters, memos, resumes, scientic papers, dissertations, books, etc. Microsoft Word is great for many of these, but like many, I’ve concluded that LaTeX is still the best for academic papers or large, complex documents. I think this graph attributed to Marko Pinteric says it elegantly.

Microsoft Word vs LaTeX

That LaTeX is so widely used is remarkable, given that it has been more that 25 years since it was first released and it was based on the somewhat arcane Tex. But LaTeX has its problems too, and one of them is remembering all of the commands to generate the many symbols that we like to use to make out papers seem more profound.

Detexify is a neat Web service that lets you draw a mathematical symbol with your mouse, interprets the result, and shows you what LaTeX command to use to generate it.

detexify readys your drawing and recommends a LaTeX command

It works pretty well! You can look at the source code — mostly in ruby — on github and contribute. Or you can volunteer to help train the system on new symbols.

(via Hacker News)

Tagged social networking site to be sued for privacy concerns

July 10th, 2009

New York state attorney general Andrew Cuomo announced he intends to sue social networking company tagged.com “for deceptive e-mail marketing practices and invasion of privacy”.

“Between April and June this year, Tagged sent tens of millions of misleading emails to unsuspecting recipients stating that Tagged members had posted private photos online for their friends to view. In reality, no such photos existed and the email was not from their friends. When recipients of these fraudulent emails tried to access the photos, they were forced to become a new member of Tagged. The company would then illegally gain access to their personal email contacts to send more fraudulent invitations.
     “This company stole the address books and identities of millions of people,” said Attorney General Cuomo. “Consumers had their privacy invaded and were forced into the embarrassing position of having to apologize to all their email contacts for Tagged’s unethical – and illegal – behavior. This very virulent form of spam is the online equivalent of breaking into a home, stealing address books, and sending phony mail to all of an individual’s personal contacts. We would never accept this behavior in the real world, and we cannot accept it online.”

See stories in the NYT and Independent.

(via AISL)

facebook demographics: biggest growth in older users

July 9th, 2009

SFGate.com reports that Most Facebook users are older, study finds.

“Long a hangout for college students, the social-networking giant has morphed into a virtual parlor for the middle-aged, according to a new study. People 35 to 54 are now the biggest age group on the Web site, accounting for 28.2 percent of all U.S. users as of July, according to iStrategyLabs, an online marketing firm. Following close behind are 24- to 34-year-olds, who represent 25.2 percent of users.”

The demographic data was extracted from facebook’s ad generation platform, which offers an estimate of the number of people matching your target audience description. For example, targeting your ad for people aged 55-60 living in Maryland shows that there are about 56,000 of them.

facebook users are old and in the way

Of course, there may be a downside to broadening the facebook community.

“Corbett said the influx of middle-age users raises the question of whether Facebook can retain its younger audience. How cool can a Web site be after Mom and Dad join? “Does this younger audience now leave Facebook and try to find their own place where they can be themselves?” Corbett asked.

And there are advantages to some in the change.

“If anything, the influx of older users makes Facebook a more attractive place for advertisers, said Corbett, the study’s author. There’s a lot of hype about the attractive 18-to-24 demographic, he said, but its people who are older who have more money to spend. “Do you want a massive population of wealthy Baby Boomers who have disposable income or a bunch of poor college kids?” Corbett said. “The audience that is growing now on Facebook is a really valuable one to have.”

This demographic data also shows dramatic declines in the number if facebook users in high school (-16.5%) and in college (-21.7%), but this if probably due to the fact that the school year just ended and the groups will shrink due to graduation before regaining their numbers in the fall.