CLEF 2012, Rome QA4MRE, Question Answering for Machine Reading Evaluation Anselmo Peñas (UNED, Spain) Eduard Hovy (USC-ISI, USA) Pamela Forner (CELCT, Italy) Álvaro Rodrigo (UNED, Spain) Richard Sutcliffe (U. Limerick, Ireland) Roser Morante (U. Antwerp, Belgium) Walter Daelemans (U. Antwerp, Belgium) Caroline Sporleder (U. Saarland, Germany) Corina Forascu (UAIC, Romania) Yassine Benajiba (Philips, USA) Petya Osenova (Bulgarian Academy of Sciences) 1 Question Answering Track at CLEF 2003 2004 2005 2006 2007 2008 Multiple Language QA Main Task QA Tasks 2 Temporal restrictions and lists 2009 2010 ResPubliQA Answer Validation Exercise (AVE) Giki CLEF Real Time QA over Speech Transcriptions (QAST) WiQA WSD QA 2011 2012 QA4MRE Negation and Modality Biomedical Portrayal Question Question analysis Passage Retrieval 0.8 Answer Extraction x 0.8 Answer Answer Ranking x 1.0 = 0.64 Along the years, we learnt that the architecture is one of the main limitations for improving QA technology So we bet on a reformulation: 3 Hypothesis generation + validation Answer Searching space of candidate answers Hypothesis generation functions + Answer validation functions Question 4 We focus on validation … Is the candidate answer correct? QA4MRE setting: Multiple Choice Reading Comprehension Tests Measure progress in two reading abilities •Answer questions about a single text •Capture knowledge from text collections 5 … and knowledge Why capture knowledge from text collections? We need knowledge to understand language 6 The ability of making inferences about texts is correlated to the amount of knowledge considered Texts always omit information we need to recover • To build the complete story behind the document • And be sure about the answer Text as source of knowledge Text Collection (background collection) Set of documents that contextualize the one under reading (20,000-100,000 docs.) • We can imagine this done on the fly by the machine • Retrieval Big and diverse enough to acquire knowledge Define a scalable strategy: topic by topic Reference collection per topic Background Collections They must serve to acquire This is sensitive to occurrence in texts General facts (with categorization and relevant relations) Abstractions (such as Thus, also to the way we create the collection Key: Retrieve all relevant documents and only them Classical IR Interdependence with topic definition • The topic is defined by the set of queries that produce the collection 8 Example: Biomedical Alzheimer’s Disease Literature Corpus Search PubMed about Alzheimer Query: (((((("Alzheimer Disease"[Mesh] OR "Alzheimer's disease antigen"[Supplementary Concept] OR "APP protein, human"[Supplementary Concept] OR "PSEN2 protein, human"[Supplementary Concept] OR "PSEN1 protein, human"[Supplementary Concept]) OR "Amyloid beta-Peptides"[Mesh]) OR "donepezil"[Supplementary Concept]) OR ("gamma-secretase activating protein, human"[Supplementary Concept] OR "gamma-secretase activating protein, mouse"[Supplementary Concept])) OR "amyloid beta-protein (142)"[Supplementary Concept]) OR "Presenilins"[Mesh]) OR "Neurofibrillary Tangles"[Mesh] OR "Alzheimer's disease"[All Fields] OR "Alzheimer's Disease"[All Fields] OR "Alzheimer s disease"[All Fields] OR "Alzheimers disease"[All Fields] OR "Alzheimer's dementia"[All Fields] OR "Alzheimer dementia"[All Fields] OR "Alzheimer-type dementia"[All Fields] NOT "nonAlzheimer"[All Fields] NOT ("non-AD"[All Fields] AND "dementia"[All Fields]) AND (hasabstract[text] AND English[lang]) 9 66,222 abstracts Questions (Main Task) Distribution of question types 27 PURPOSE 30 METHOD 36 CAUSAL 36 FACTOID 31 WHICH-IS-TRUE Distribution of answer types 75 REQUIRE NO EXTRA KNOWLEDGE 46 REQUIRE BACKGROUND KNOWLEDGE 21 REQUIRE INFERENCE 20 REQUIRE GATHERING INFORMATION FROM DIFFERENT SENTENCES 10 Questions (Biomedical Task) Question types 1. Experimental evidence/qualifier 2. Protein-protein interaction 3. Gene synonymy relation 4. Organism source relation 5. Regulatory relation 6. Increase (higher expression) 7. Decrease (reduction) 8. Inhibition 11 Answer types Simple: The answer is found almost verbatim in the paper Medium: The answer is rephrased Complex: Require combining pieces of evidence and inference They involve a predefined set of entity types Main Task 16 test documents, 160 questions, 800 candidate answers 4 Topics 1. 2. 3. 4. AIDS Music and Society Climate Change Alzheimer (divulgative sources: blogs, web, news, …) 4 Reading tests per topic Document + 10 questions 5 choices per question 6 Languages new English, German, Spanish, Italian, Romanian, Arabic new Biomedical Task Same setting Scientific language Focus on one disease: Alzheimer Alzheimer's Disease Literature Corpus (ADLC) 66,222 abstracts from PubMed 9,500 full articles Most of them processed: • Dependency parser GDep (Sagae and Tsujii 2007) • UMLS-based NE tagger (CLiPS) • ABNER NE tagger (Settles 2005) Task on Modality and Negation Given an event in the text decide whether it is 1. 2. 3. 4. Asserted (NONE: no negation and no speculation) Negated (NEG: negation and no speculation) Speculated but negated (NEGMOD) Speculated and not negated (MOD) Is the event present as certain? Yes No Did it happen? Is it negated? Yes No Yes NONE NEG NEGMOD No MOD Participation Task Registered groups Participant groups Submitted Runs Main 25 11 43 Biomedical 23 7 43 Modality and Negation 3 3 6 Total 51 21 92 ~100% increase 15 100 80 60 40 20 0 Participants Runs 2011 2012 Evaluation and results QA perspective evaluation [email protected] over all questions (random 0.2) Best systems Main Best systems Biomedical 0.65 0.55 0.40 0.47 Reading perspective evaluation Aggregating results test by test (pass if [email protected] > 0.5) Best systems Main Best systems Biomedical Tests passed: 12 / 16 Tests passed: 3 / 4 Tests passed: 6 /16 16 More details during the workshop Monday 17th Sep. 17:00 - 18:00 Poster Session Tuesday 18th Sep. 10:40 – 12:40 Invited Talk + Overviews 14:10 – 16:10 Reports from participants (Main + Bio) 16:40 – 17:15 Reports from participants (Mod&Neg) 17:15 – 18:10 Breakout session 17 Thanks!