Clustering short status messages: a topic model based approach

Recently, there has been an exponential rise in the use of online social media systems like Twitter and Facebook. Even more usage has been observed during events related to natural disasters, political turmoil or other such crises. Tweets or status messages are short and may not carry enough contextual clues. Hence, applying traditional natural language processing algorithms on such data is challenging. Topic model is a popular method for modeling term frequency occurrences for documents in a given corpus. A topic basically consists of set of words that co-occur frequently. Unsupervised nature allows topic models to be trained easily on datasets meant for specific domains.

We use the topic modeling feature of the MALLET machine learning tool kit to generate topic models from unlabelled data. We propose a way to cluster tweets by using the topic distributions in each tweet. We address the problem of determining which topic model is optimal for clustering tweets based on its clustering performances. We also demonstrate a use case wherein we cluster twitter users based on the content they tweet. We back our research with experimental results and evaluations.

• Dr. Tim Finin (Chair)
• Dr. Anupam Joshi
• Dr. Charles Nicholas