Improving Accuracy of Named Entity Recognition on Social Media Data

In recent years, social media outlets such as Twitter and Facebook have drawn attention from companies and researchers interested in detecting trends. The informal nature of status updates from these services leads to a higher volume of updates, because each update takes little care to generate, but each update is usually short and noisy (misspellings, lack of punctuation, non-standard abbreviations and capitalization). These shortcomings cause traditional Natural Language Processing (NLP) techniques to have substantially lower accuracy than is found with structured text such as newswire articles.

We present a system for improving the accuracy of one NLP technique, Named Entity Recognition or NER, on Twitter data by training a recognizer specifically for this type of data. NER is the process of automatically recognizing which words are names of people, places, or organizations. Prerequisite tasks already completed include cleaning and normalizing our collection of 150 million tweets from 1.5 million users (gathered over 20 months during 2007--2008), establishing a baseline entity detection rate with an off-the-shelf NER system, and identifying what language each update is in.

Committee Members:

• Dr. Tim Finin (Chair)
• Dr. Tim Oates
• Dr. Anupaum Joshi
• Dr. Charles Nicholas