The University of Arizona’s AI Lab is engaged in an NSF funded project with the goal of “collecting and analyzing terrorism information, modeling terrorist behavior and terrorist networks, and disseminating information to the terrorized”. The most interesting aspect of the Dark Web Project is a collection of Web pages that are believed to be from terrorist affiliated groups.

“We have collected 500,000 Web pages created by 94 US domestic groups, 300,000 Web pages created by 41 Arabic-speaking groups, and 100,000 Web pages created by Spanish-speaking groups. The collection process is ongoing.” (link)

The methodology used to collect some of these pages is described in

Y. Zhou, J. Qin, G. Lai, E. Reid, and H. Chen, Exploring the Dark Side of the Web: Collection and Analysis of U.S. Extremist Online Forums, in Proc. Intelligence and Security Informatics: IEEE International Conference on Intelligence and Security Informatics, San Diego, 23-25 May 2006.

I sounds like a fascinating dataset that I hope can be made available in some form. I would like to try some of our sentiment detection and rust propagation techniques on this data, for example. The group has a demo system online, but it seems to be currently down.

Related posts:

1. The Dark Room
2. Proofiness: when mathematics turns to the dark side
3. Wikipedia original sin
4. New York Times publishes Linked Open Data
5. Grundke on the Dalhousie Bachelor of Informatics program