Why You Should Use N-grams for Multilingual Information Retrieval

Start: Wednesday, October 18, 2006, 11:00AM

End: Wednesday, October 18, 2006, 12:00PM

Location: 2120 A.V. Williams Building, UMCP

Abstract: While generally accepted for languages such as Chinese and Japanese, the use of character n-gram tokenization has not been widely adopted for information retrieval in alphabetic languages. However, n-grams are a simple representation for text that is surprisingly effective in diverse languages. In this talk I present empirical results in twelve European languages that have been studied in the Cross Language Evaluation Forum (CLEF) competitions. These results demonstrate that:

n-gram tokenization is very effective for monolingual retrieval;
morphologically complex languages benefit from n-gram use;
n-grams have performance disadvantages that can be overcome;
and, given parallel text, n-grams can be projected from one language to another, enabling highly accurate bilingual retrieval.

I will describe issues particular to n-gram indexing and retrieval such as increased disk space consumption and query times, make a case that n-grams are a synthetic form of morphological normalization, and argue that when linguistic and translation resources are scarce (as is the case with less-commonly studied languages), n-grams are an extremely attractive option for multilingual retrieval.


	UMBC eBiquity Events / Talks Why You Should Use N-grams for Multilingual Information Retrieval OWL metadata Update record		Why You Should Use N-grams for Multilingual Information Retrieval Tweet Speaker: Paul McNamee Start: Wednesday, October 18, 2006, 11:00AM End: Wednesday, October 18, 2006, 12:00PM Location: 2120 A.V. Williams Building, UMCP Abstract: While generally accepted for languages such as Chinese and Japanese, the use of character n-gram tokenization has not been widely adopted for information retrieval in alphabetic languages. However, n-grams are a simple representation for text that is surprisingly effective in diverse languages. In this talk I present empirical results in twelve European languages that have been studied in the Cross Language Evaluation Forum (CLEF) competitions. These results demonstrate that: n-gram tokenization is very effective for monolingual retrieval; morphologically complex languages benefit from n-gram use; n-grams have performance disadvantages that can be overcome; and, given parallel text, n-grams can be projected from one language to another, enabling highly accurate bilingual retrieval. I will describe issues particular to n-gram indexing and retrieval such as increased disk space consumption and query times, make a case that n-grams are a synthetic form of morphological normalization, and argue that when linguistic and translation resources are scarce (as is the case with less-commonly studied languages), n-grams are an extremely attractive option for multilingual retrieval. ,