EASY : An Evaluation Protocol for Syntactic Parsers

Patrick Paroubek, Isabelle Robba, Anne Vilnat

Object

In natural language understanding, most of the complex systems use a syntactic parser as a basic functionality. The last decades have seen the emergence of a new class of systems: shallow parsers, while deep parsers have continued to be improved. Among parsers, some restrict themselves to chunk (basic syntactic constituent of a phrase) analysis, others compute the chunks and the functional relations that exists between the chunks, thus tackling problems which have also a semantic nature.

Facing this diversity, the need for a comparative evaluation framework is increasing: the EASY evaluation campaign has been launched to meet this need. EASY is part of a larger evaluation campaign named EVALDA, wich itself is part of the national Technolangue program (http://www.technolangue.net), supported by the French Research Ministry. EASY is a complete evaluation protocol including:


Description

The definition of the annotation formalism is the core element of the evaluation process. The chosen formalism indeed must have a coverage as broad as possible of the syntactical phenomena and it should not penalize any participant. Choices retained for annotation were made with the organizers, the participants and the corpus providers.

In EASY formalism, 6 types of chunks are annotated; moreover, chunks cannot be discontinuous nor embedded, and they must be as small as possible. Information which would not be expressed through chunks is expressed through functional relations: 14 relations have been identified. These relations can indifferently link constituents or word forms.

Chunks 1 verbal phrase (NV) 2 nominal phrase (GN) 3 adjectival phrase (GA) 4 adverbial phrase (GR) 5 prepositional group introducing a nominal phrase (GP) 6 prepositional group introducing a verbal phrase (PV)
Functional relations 1 subject-verb 2 auxiliary-verb 3 direct object-verb 4 complement-verb 5 modifier-verb 6 complement 7 attribute-subject/object 8 modifier-noun 9 modifier-adjective 10 modifier-adverb 11 modifier-preposition 12 coordination 13 apposition 14 juxtaposition

Five corpus providers participated in the corpus constitution and its annotation. The corpus which was intended to be highly heterogeneous contains:


An annotated example

From a practical viewpoint, annotation of the reference data is made in two steps using an HTML editor. The first table below shows an example of a constituent annotation: chunks of the same type have been colored using the same color. The second table (automatically built) visualizes the numerotation of chunks and words as well as the chunk labels; this table is used during relation annotation. At last, the figure 1 presents the final result with all the annotated relations, it was made with the visualization tool developed by Emmanuel Giguet and applied to the EASY annotation format in XML.

The original sentence is extracted from Le Monde newspaper:
In a corner of the room, four little blackboards covered with verses in Arabic drawn with chalk testify of their religious zeal.

Enoncé 1
Dans
un
coin
de
la
pièce
,
quatre
petits
tableaux
noirs
couverts
Enoncé 1
de
versets
en
arabe
tracés
à
la
craie
témoignent
de
leur
zèle
religieux
.

Table 1: Constituent annotation

Enoncé 1
GP 1 GP 2
GN 3 GA 4 NV 5 GP 6 GP 7
Dans un coin de la pièce , quatre petits tableaux noirs couverts de versets en arabe
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Enoncé 1
NV 8 GP 9 NV 10 GP 11 GA 12
tracés à la craie témoignent de leur zèle religieux .
17 18 19 20 21 22 23 24 25 26

Table 2: Constituent visualization
Functional relation visualization

Figure 1 : Functional relation visualization

Evaluation

Annotations collected in HTML are transcribed in an XML format and all the participants map their parses into this unique format on which evaluation measures are taken. The evaluation measures adopted in EASY are the standard recall and precision metrics computed both on constituents and dependencies. Nevertheless, since our copus is heterogeneous, it is relevant to decline the results according to the different types of corpus as well as to the different syntactic phenomena annotated.

Results and prospects

EASY was launched in january 2003, and should be completed at the end of march 2006. The constituent evaluation is now in the adjudication phase, and the evaluation of dependencies has started. The preliminary conclusions we can draw from EASY are very positive:

References

[1] P. Paroubek, L.G. Pouillot,I, Robba, A. Vilnat. (2005). EASy : campagne d'évaluation des analyseurs syntaxiques, Atelier EASy de TALN 05, Dourdan, France.
[2] V. Gendner, G. Illouz, M. Jardino, L. Monceaux, P. Paroubek, I. Robba, A. Vilnat (2003). PEAS, the first instantiation of a comparative framework for evaluating parsers of French, Proceedings of the Reasearch Note Sessions of the 10th Conference of the European Chapter of the Association for Computational Linguistics (EACL'03), Budapest, Hungary.