The NYT has a story in today’s paper (Software Being Developed to Monitor Opinions of U.S.) on a recent DHS basic research contract to extract opinions from news articles.

“A consortium of major universities, using Homeland Security Department money, is developing software that would let the government monitor negative opinions of the United States or its leaders in newspapers and other publications overseas. Such a “sentiment analysis” is intended to identify potential threats to the nation, security officials said.”

In this case the DHS consortium (University Affiliate Center) consists of researchers from Cornell, Pittsburgh and Utah.

The article points out the potential for inappropriate use of such systems to monitor the opinions and beliefs of reporters, news sources and even ordinary citizens.

“Ultimately, the government could in a semiautomated way track a statement by specific individuals abroad or track reports by particular foreign news outlets or journalists, rating comments about American policies or officials.”

Marc Rotenberg, executive director of the Electronic Privacy Information Center in Washington, said the effort recalled the aborted 2002 push by a Defense Department agency to develop a tracking system called Total Information Awareness that was intended to detect terrorists by analyzing troves of information. “That is really chilling,” Mr. Rotenberg said. “And it seems far afield from the mission of homeland security.”

<opinion>
How worried should we be? I can see some real dangers for misuse of information extraction, especially in an age where so many ordinary citizens publish content on the web and in their blogs. However, there are many good and useful applications as well and the techniques are already being developed and extended for many applications.

Sentiment detection and opinion extraction are hot topics right now. A simple Google Scholar search turns up thousands of papers on the topic published in the past two years. One of the 2006 TREC (Text Retreival Conference) tracks is on opinion extraction from blog posts. Several companies offer a service to clients which tracks opinions expressed on blogs about the company’s products and services. And then there is Opinmind, a free web service that retrieves blog posts expressing an opinion about a topic described by a query string.

Two of the researchers involved in this DHS project have this to say about the issue.

Professors Cardie and Wiebe said they understood that there were legitimate questions about the ultimate use of their software. “There has to be guidelines and restrictions on the use of this kind of technology by the government,” Professor Wiebe said. “But it doesn’t mean it is not useful. It can just as easily help the government understand what is going on in places around the world.”

I agree with this position. We are not going to be able to control the problem by halting research on natural language understanding and information extraction. The technology is neither good nor bad — just its applications. Linguists and Computer Scientists have been working on improving the ability for machines to understand language for more than 50 years. The positive applications vastly outweigh the potential negative ones, even within the realm of intelligence and national security. Our government has an obligation to understand how its actions and policies are viewed by the press and the public around the world.

What is needed, IMHO, is a system of oversight and appropriate transparency, to ensure, as much as possible, that governments and corporations do not misuse the technology to do inappropriate things, like build dossiers of the beliefs and opinions of individual citizens.
</opinion>

Related posts:

1. Monitoring urban traffic with mobile phones