Twitter as the Web stream of consciousness

February 15th, 2009

TechCrunch has a post Mining The Thought Stream on why Twitter continues to be hot even thought it doesn’t yet have a business case. The argument is that Twitter has fond a niche that none of the search engines covers well — providing visibility over the stream of consciousness of the Web. The final graf caught my attention:

“An undifferentiated thought stream of the masses at some point becomes unwieldy. In order to truly mine that data, Twitter needs to figure out how to extract the common sentiments from the noise (something which Summize was originally designed to do, by the way, but it was putting the cart before the horse—you need to be able to do simple searches before you start looking for patterns). But what is the best way to rank real-time search results—by number of followers, retweets, some other variable? It is not exactly clear. But if Twitter doesn’t solve this problem, someone else will and they will make a lot of money if they do it right.”

Akshay looked at the problem of analyzing tweets back in 2007 (see Why We Twitter: Understanding Microblogging Usage and Communities). One difficulty is that tweets are necessarily short and telegraphic. This makes it hard to do any linguistic analysis with good accuracy. But, maybe if you can apply some back ground knowledge……