Why You Should Use N-grams for Multilingual Information Retrieval

While generally accepted for languages such as Chinese and Japanese, the use of character n-gram tokenization has not been widely adopted for information retrieval in alphabetic languages. However, n-grams are a simple representation for text that is surprisingly effective in diverse languages. In this talk I present empirical results in twelve European languages that have been studied in the Cross Language Evaluation Forum (CLEF) competitions. These results demonstrate that:

• n-gram tokenization is very effective for monolingual retrieval;
• morphologically complex languages benefit from n-gram use;
• n-grams have performance disadvantages that can be overcome;
• and, given parallel text, n-grams can be projected from one language to another, enabling highly accurate bilingual retrieval.

I will describe issues particular to n-gram indexing and retrieval such as increased disk space consumption and query times, make a case that n-grams are a synthetic form of morphological normalization, and argue that when linguistic and translation resources are scarce (as is the case with less-commonly studied languages), n-grams are an extremely attractive option for multilingual retrieval.