How to answer in English to questions asked in French

Brigitte Grau, Anne-Laure Ligozat, Isabelle Robba, Anne Vilnat

Object

Open-domain Question-Answering (QA) is a growing area of research whose aim is to find precise answers to questions in natural language, unlike search engines that return whole documents. One challenge in this field consists in finding an answer in a target language different from the question language. This field aim at taking into account the fact that using the Web will be more effective if it is searched in English, as proved Figure 1.
The languages on the Web.
Figure 1 - The languages on the Web (source)

Thus, searching the French Web would not give as significant results as searching the English Web. Such a challenge is part of the CLEF evaluation since 2003.

Description

For our participation to CLEF evaluation, we developed MUSCLEF (Multilingual System for CLEF) which uses two strategies. The first one consists in analyzing the French question, translating "interesting parts", i.e. its relevant terms, and then using these translated terms to search the reference collection. MUSQAT which is our multilingual module follows this strategy. The second strategy consists in translating the question in English thanks to the help of a Machine Translation System, then, applying QALC [1], our existing monolingual system that answers to English question. Those strategies are the most commonly adopted, but to our knowledge, any other system except our own implements both.

Overview of MUSCLEF

The global architecture of MUSCLEF is illustrated Figure 2. First, its question analysis module aims at deducing characteristics which may help to find possible answers in selected passages. These characteristics are: the expected answer type, the question focus, the main verb and some syntactic characteristics. English questions were translated using Reverso.

Architecture of MUSCLEF .
Figure 2 - Architecture of MUSCLEF
For querying the CLEF collection and retrieving passages we used MG. Retrieved documents are then processed: they are re-indexed by the question terms and their linguistic variants, reordered according to the number and the kind of terms found in them, so as to select a subset of them. Named entity recognition processes are then applied. The answer extraction process relies on a weighting scheme of the sentences, followed by the answer extraction itself. We apply different processes according to the kind of expected answer, each of them leading to propose weighted answers.

The first run we submitted corresponds to the strategy implemented in MUSQAT: translation of selected terms. For the second run, we added a final step consisting in comparing the results issued from both strategies: the translated questions and the translated terms. This module named fusion in Figure 2, computes a final score for each potential answer, its principle is to boost an answer if both chains ranked it in the top 5 propositions, even with relatively low scores.

Term translation

Different methods can be used to achieve term translation and we considered the easiest one, which consists in using a bilingual dictionary to translate the terms from the source language to the target language. This simple method presents two drawbacks: it is impossible to directly disambiguate the various meanings of the words to be translated, and the two languages must be of equivalent lexical richness. Hence, we could not consider a dictionary giving only one meaning for a word. Among the GPL dictionaries, we chose Magic-Dic, because of its evolutivity: terms can be added by any user, but they are verified before being integrated. To prevent Magic-Dic incompleteness, and because it has been proved that the use of several dictionaries gives better results than a unique one, we used this year a second dictionaries FreeDict and merged their translations. FreeDict had added 424 different translations of the 690 words. However, these new translations are mainly other synonyms rather than new translations of unknown words. For example the French word mener is translated with Magic-Dic in to conduct, to guide, to lead, while accordis only translated in agreement. FreeDict added five translations for accord : accord, accordance, concurrence, chord and concord and gave translations for frontière that was missing in Magic-Dic.

Results and prospects

Table 1 gives the results that our system obtained at the CLEF04 and CLEF05 campaigns, with the different strategies: The evaluation was made by an automatic process that looks for the answer patterns in the system answers, applying regular expressions. These results were computed with 178 answer patterns that we built for the 200 questions of CLEF04 and 188 for the CLEF05 questions.

Table 1 - Results at CLEF04 and CLEF05
    MUSQAT04 QALC04 MUSQAT05 QALC05
Sentence 5 first ranks 56 (31%) 65 (37%) 78 (41%) 87(46%)
NE
answers		 Rank 1
5 first ranks		 17
32		 26
37		 16
24		 9
11
Non NE
answers		 Rank 1
5 first ranks		 7
12		 3
8		 16
22		 16
24
Total Rank 1
%
5 first ranks 		24
12%
44 		29
14.5%
45 		32
17%
46 		25
13%
35

Official results at CLEF04 and CLEF05
Fusion (official results) 38 (19%) 38 (19%)

The first line indicates the number of correct answers found in the 5 first sentences given by MUSQAT (using term translation) and QALC. The second line, NE answers, gives the number of correct answers on questions the system categorized as waiting for a Named Entity (the total is 107 in CLEF04 for MUSQAT and 97 for QALC and 91 in CLEF05 for MUSQAT and 66 for QALC). Our total number of questions of this category is far beyond the real number in CLEF05. The third line, non NE answers, concerns the other questions (the complement to 178 in CLEF04 and to 188 in CLEF05). Results are presented when the system just gives one answer and when it gives 5 answers. The last line indicates the best official result of our system on the 200 questions. The official score of MUSQAT was 22 (11%) in CLEF04 and 28 (14%) in CLEF05, thus we can observe that merging answers obtained by different strategies enables a significant gain.

These scores put our system MUSCLEF at the 4th place among 11 participants in the track consisting to answer in English to a question given in one of the different languages allowed at CLEF. We also can notice that if our CLEF05 system better selects sentences, it is less performant on extracting the answers, specially on named entity answers.

References

[1] O. Ferret, B. Grau, M. Hurault-Plantet, G. Illouz, C. Jacquemin, L. Monceaux, I. Robba, A.Vilnat (2002). How NLP Can Improve Question Answering , Knowledge Organization journal, Vol. 29, N3-4, 135-155.
[2] Brigitte Grau, Gabriel Illouz, Laura Monceaux, Isabelle Robba, Anne Vilnat, Guillaume Bourdil, Faïza Elkateb-Gara, Olivier Ferret And Benoît Mathieu (2005). Multilingual Information Access for Text, Speech and Images: 5th Workshop of the Cross-Language Evaluation Forum, CLEF 2004, Bath, UK, September 15-17, 2004, Revised Selected Papers, Ed. Carol Peters, Paul Clough, Julio Gonzalo, et al. , Answering French Questions in English by Exploiting Results from Several Sources of Information , Lecture Notes in Computer Science, Volume 3491 / 2005.
[3] Brigitte Grau, Anne-Laure Ligozat, Isabelle Robba, Madeleine Sialeu, Anne Vilnat, (2005). Term Translation Validation by Retrieving Bi-terms, Working Notes of CLEF Workshop, ECDL conference.