A Framework for Entailed Relation RecognitionDan Roth Mark Sammons V.G.Vinod Vydiswaran University of Illinois at Urbana-Champaign {danr|mssammon|vgvinodv}@illinois.edu Abstract We defin
Trang 1A Framework for Entailed Relation Recognition
Dan Roth Mark Sammons V.G.Vinod Vydiswaran
University of Illinois at Urbana-Champaign {danr|mssammon|vgvinodv}@illinois.edu
Abstract
We define the problem of recognizing entailed
re-lations – given an open set of rere-lations, find all
oc-currences of the relations of interest in a given
doc-ument set – and pose it as a challenge to scalable
information extraction and retrieval Existing
ap-proaches to relation recognition do not address well
problems with an open set of relations and a need
for high recall: supervised methods are not
eas-ily scaled, while unsupervised and semi-supervised
methods address a limited aspect of the problem, as
they are restricted to frequent, explicit, highly
lo-calized patterns We argue that textual entailment
(TE) is necessary to solve such problems, propose
a scalable TE architecture, and provide preliminary
results on an Entailed Relation Recognition task.
1 Introduction
In many information foraging tasks, there is a need
to find all text snippets relevant to a target concept
Patent search services spend significant resources
looking for prior art relevant to a specified patent
claim Before subpoenaed documents are used in
a court case or intelligence data is declassified, all
sensitive sections need to be redacted While there
may be a specific domain for a given application,
the set of target concepts is broad and may change
over time For these knowledge-intensive tasks,
we contend that feasible automated solutions
re-quire techniques which approximate an
appropri-ate level of natural language understanding
Such problems can be formulated as a relation
recognition task, where the information need is
ex-pressed as tuples of arguments and relations This
structure provides additional information which
can be exploited to precisely fulfill the
informa-tion need Our work introduces the Entailed
Rela-tion RecogniRela-tion paradigm, which leverages a
tex-tual entailment system to try to extract all relevant
passages for a given structured query without
re-quiring relation-specific training data This con-trasts with Open Information Extraction (Banko and Etzioni, 2008) and On-Demand Information Extraction (Sekine, 2006), which aim to extract large databases of open-ended facts, and with su-pervised relation extraction, which requires addi-tional supervised data to learn new relations Specifically, the contributions of this paper are:
1 Introduction of the entailed relation recognition framework; 2 Description of an architecture and a system which uses structured queries and an exist-ing entailment engine to perform relation extrac-tion; 3 Empirical assessment of the system on a corpus of entailed relations
2 Entailed Relation Recognition (ERR)
In the task of Entailed Relation Recognition, a cor-pus and an information need are specified The corpus comprises all text spans (e.g paragraphs) contained in a set of documents The information need is expressed as a set of tuples encoding rela-tions and entities of interest, where entities can be
of arbitrary type The objective is to retrieve all relevant text spans that a human would recognize
as containing a relation of interest For example:
Information Need: An organization acquires weapons Text 1: the recent theft of 500 assault rifles by FARC Text 2: the report on FARC activities made three main ob-servations First, their allies supplied them with the 3” mor-tars used in recent operations Second,
Text 3: Amnesty International objected to the use of artillery
to drive FARC militants from heavily populated areas.
An automated system should identify Texts 1 and
2 as containing the relation of interest, and Text 3
as irrelevant The system must therefore detect relation instances that cross sentence boundaries (“them” maps to “FARC”, Text 2), and that re-quire inference (“theft” implies “acre-quire”, Text 1)
It must also discern when sentence structure pre-cludes a match (“Amnesty International use artillery” does not imply “Amnesty International 57
Trang 2acquires artillery”, Text 3).
The problems posed by instances like Text 2
are beyond the scope of traditional
unsuper-vised and semi-superunsuper-vised relation-extraction
ap-proaches such as those used by Open IE and
On-Demand IE, which are constrained by their
de-pendency on limited, sentence-level structure and
high-frequency, highly local patterns, in which
relations are explicitly expressed as verbs and
nouns Supervised methods such as (Culotta and
Sorensen, 2004) and (Roth and Yih, 2004)
pro-vide only a partial solution, as there are many
pos-sible relations and entities of interest for a given
domain, and such approaches require new
anno-tated data each time a new relation or entity type is
needed Information Retrieval approaches are
op-timized for document-level performance, and
en-hancements like pseudo-feedback (Rocchio, 1971)
are less applicable to the localized text spans
needed in the tasks of interest; as such, it is
un-likely that they will reliably retrieve all correct
stances, and not return superficially similar but
in-correct instances (such as Text 3) with high rank
Attempts have been made to apply Textual
En-tailment in larger scale applications For the task
of Question Answering, (Harabagiu and Hickl,
2006) applied a TE component to rerank candidate
answers returned by a retrieval step However, QA
systems rely on redundancy in the same way Open
IE does: a large document set has so many
in-stances of a given relation that at least some will
be sufficiently explicit and simple that standard IR
approaches will retrieve them A single correct
in-stance suffices to complete the QA task, but does
not meet the needs of the task outlined here
Recognizing relation instances requiring
infer-ence steps, in the absinfer-ence of labeled training data,
requires a level of text understanding A
suit-able proxy for this would be a successful Textual
Entailment Recognition (TE) system (Dagan et
al., 2006) define the task of Recognizing Textual
Entailment (RTE) as: a directional relation
be-tween two text fragments, termed T – the entailing
text, and H – the entailed text T entails H if,
typ-ically, a human reading T would infer that H is
most likely true For relation recognition, the
rela-tion triple (e.g “Organizarela-tion acquires weapon”)
is the hypothesis, and a candidate text span that
might contain the relation is the text The
def-inition of RTE clearly accommodates the range
of phenomena described for the examples above
However, the more successful TE systems (e.g (Hickl and Bensley, 2007)) are typically resource intensive, and cannot scale to large retrieval tasks
if a brute force approach is used
We define the task of Entailed Relation Recog-nition thus: Given a text collection D, and an in-formation need specified in a set of [argument, re-lation, argument] triples S: for each triple s ∈ S, identify all texts d ∈ D such that d entails s The information need triples, or queries, encode relations between arbitrary entities (specifically, these are not constrained to be Named Entities) This problem is distinct from recent work in Textual Entailment as we constrain the structure
of the Hypothesis to be very simple, and we re-quire that the task be of a significantly larger scale than the RTE tasks to date (which are typically of the order of 800 Text-Hypothesis pairs)
3 Scalable ERR Algorithm
Our scalable ERR approach, SERR, consists of two stages: expanded lexical retrieval, and entail-ment recognition The SERR algorithm is pre-sented in Fig 1 The goal is to scale Textual Entailment up to a task involving large corpora, where hypotheses (queries) may be entailed by multiple texts The task is kept tractable by de-composing TE capabilities into two steps
The first step, Expanded Lexical Retrieval (ELR), uses shallow semantic resources and simi-larity measures, thereby incorporating some of the semantic processing used in typical TE systems This is required to retrieve, with high recall, se-mantically similar content that may not be lexi-cally similar to query terms, to ensure return of
a set of texts that are highly likely to contain the concept of interest
The second step applies a textual entailment system to this text set and the query in order to label the texts as ‘relevant’ or ‘irrelevant’, and re-quires deeper semantic resources in order to dis-cern texts containing the concept of interest from those that do not This step emphasizes higher pre-cision, as it filters irrelevant texts
3.1 Implementation of SERR
In the ELR stage, we use a structured query that allows more precise search and differential query expansion for each query element Semantic units
in the texts (e.g Named Entities, phrasal verbs) are indexed separately from words; each index is
Trang 3SERR Algorithm
S ETUP :
Input: Text set D
Output: Indices {I} over D
for all texts d ∈ D
Annotate d with local semantic content
Build Search Indices {I} over D
A PPLICATION :
Input: Information need S
E XPANDED L EXICAL R ETRIEVAL (ELR)(s):
R ← ∅
Expand s with semantically similar words
Build search query q s from s
R ← k top-ranked texts for q s using indices {I}
return R
SERR:
Answer set A ← ∅
for all queries s ∈ S
R ← ELR(s)
Answer set A s ← ∅
for all results r ∈ R
Annotate s, r with NLP resources
if r entails s
A s ← A s ∪ r
A ← A ∪ {A s }
return A
Figure 1 SERR algorithm
a hierarchical similarity structure based on a
type-specific metric (e.g WordNet-based for phrasal
verbs) Query structure is also used to selectively
expand query terms using similarity measures
re-lated to types of semantic units, including
distribu-tional similarity (Lin and Pantel, 2001), and
mea-sures based on WordNet (Fellbaum, 1998)
We assess three different Textual Entailment
components: LexPlus, a lexical-level system
that achieves relatively good performance on the
RTE challenges, and two variants of
Predicate-based Textual Entailment, strict and
PTE-relaxed, which use a predicate-argument
repre-sentation The former is constrained to select a
single predicate-argument structure from each
re-sult, which is compared to the query
component-by-component using similarity measures similar to
the LexPlus system PTE-relaxed drops the
single-predicate constraint, and can be thought of as a
‘bag-of-constituents’ model In both, features are
extracted based on the predicate-argument
compo-nents’ match scores and their connecting structure,
and the rank assigned by ELR These features are
used by a classifier that labels each result as
‘rel-evant’ or ‘irrel‘rel-evant’ Training examples are
se-lected from the top 7 results returned by ELR for
queries corresponding to entailment pair
hypothe-ses from the RTE development corpora; test exam-ples are similarly selected from results for queries from the RTE test corpora (see section 3.2) 3.2 Entailed Relation Recognition Corpus
To assess performance on the ERR task, we de-rive a corpus from the publicly available RTE data The corpus consists of a set S of informa-tion needs in the form of [argument, relainforma-tion, argu-ment] triples, and a set D of text spans (short para-graphs), half of which entail one or more s ∈ S while the other half are unrelated to S D com-prises all 1, 950 Texts from the IE and IR sub-tasks of the RTE Challenge 1–3 datasets The shorter hypotheses in these examples allow us to automatically induce their structured query form from their shallow semantic structure S was au-tomatically generated from the positive entailment pairs in D, by annotating their hypotheses with a publicly available SRL tagger (Punyakanok et al., 2008) and inferring the relation and two main ar-guments to form the equivalent queries
Since some Hypotheses and Texts appear mul-tiple times in the RTE corpora, we automatically extract mappings from positive Hypotheses to one
or more Texts by comparing hypotheses and texts from different examples This provides the label-ing needed for evaluation In the resultlabel-ing corpus,
a wide range of relations are sparsely represented; they exemplify many linguistic and semantic char-acteristics required to infer the presence of non-explicit relations
4 Results and Discussion Top # Basic ELR Rel.Impr Err.Redu.
1 48.1% 55.2% +14.8% 13.7%
2 68.1% 72.8% +6.9% 14.7%
3 75.2% 78.5% +4.4% 17.7%
Table 1 Change in relevant results retrieved in top 3
positions for basic and expanded lexical retrieval
System Acc Prec Rec F 1
Baseline 18.1 18.1 100.0 30.7 LexPlus 81.6 44.9 62.5 55.5 PTE-relax 71.9 37.7 72.0 49.0
(0.1) (5.5) (6.2) (4.1) PTE-strict 83.6 55.4 61.5 57.9
(1.3) (3.4) (7.9) (2.1)
Table 2 Comparison of performance of SERR with
different TE algorithms Numbers in parentheses are standard deviations.
Table 1 compares the results of SERR with and
Trang 4# System RTE 1 RTE 2 RTE 3 Avg Acc.
PTE-relaxed 54.5 (1.0) 68.7 (1.5) [3] 82.3 (2.0) [1] 71.2 (1.2)
PTE-strict 64.8 (2.3) [1] 71.2 (2.6) [3] 76.0 (3.2) [2] 71.8 (2.6)
Table 3 Performance (accuracy) of SERR system variants on RTE challenge
examples; numbers in parentheses are standard deviations, while numbers in
brackets indicate where systems would have ranked in the RTE evaluations.
Comparisons Standard TE 3,802,500
Table 4 Entailment
compar-isons needed for standard TE
vs SERR
without the ELR’s semantic enhancements For
each rank k, the entries represent the proportion of
queries for which the correct answer was returned
in the top k positions The semantic enhancements
improve the number of matched results at each of
the top 3 positions
Table 2 compares variants of the SERR
imple-mentation The baseline labels every result
re-turned by ELR as ‘relevant’, giving high recall
but low precision PTE-relaxed performs better
than baseline, but poorly compared to PTE-strict
and LexPlus Our analysis shows that LexPlus
has a relatively high threshold, and correctly labels
as negative some examples mislabeled by
PTE-relaxed, which may match two of the three
con-stituents in a hypothesis and label that result as
positive PTE-strict will correctly identify some
such examples as it will force some match edges to
be ignored, and will correctly identify some
neg-ative examples due to structural constraints even
when LexPlus finds matches for all query terms
PTE-strict strikes the best balance between
preci-sion and recall on positive examples
Table 3 shows the accuracy of SERR’s
clas-sification of the examples from each RTE
chal-lenge; results not returned in the top 7 ranks by
ELR are labeled ‘irrelevant’ strict and
relaxed perform comparably overall, though
PTE-strict has more uniform results over the different
challenges Both outperform the LexPlus system
overall, and perform well compared to the best
re-sults published for the RTE challenges
The significant computational gain of SERR is
shown in Table 4, exhibiting the much greater
number of comparisons required by a brute force
TE approach compared to SERR: SERR performs
well compared to published results for RTE
chal-lenges 1-3, but makes only 0.36% of the TE
com-parisons needed by standard approaches on our
ERR task
5 Conclusion
We have proposed an approach to solving the
En-tailed Relation Recognition task, based on
Tex-tual Entailment, and implemented a solution that shows that a Textual Entailment Recognition sys-tem can be scaled to a much larger IE problem than that represented by the RTE challenges Our preliminary results demonstrate the utility of the proposed architecture, which allows strong perfor-mance in the RTE task and efficient application to
a large corpus (table 4)
Acknowledgments
We thank Quang Do, Yuancheng Tu, and Kevin Small This work is funded by a grant from Boeing and by MIAS, a DHS-IDS Center for Multimodal Information Access and Synthesis at UIUC
References [Banko and Etzioni2008] M Banko and O Etzioni 2008 The Tradeoffs Between Open and Traditional Relation Ex-traction In ACL-HLT, pages 28–36.
[Culotta and Sorensen2004] A Culotta and J Sorensen.
2004 Dependency Tree Kernels for Relation Extraction.
In ACL, pages 423–429.
[Dagan et al.2006] I Dagan, O Glickman, and B Magnini, editors 2006 The PASCAL Recognising Textual Entail-ment Challenge., volume 3944 Springer-Verlag, Berlin [Fellbaum1998] C Fellbaum 1998 WordNet: An Electronic Lexical Database MIT Press.
[Harabagiu and Hickl2006] S Harabagiu and A Hickl 2006 Methods for Using Textual Entailment in Open-Domain Question Answering In ACL, pages 905–912.
[Hickl and Bensley2007] A Hickl and J Bensley 2007 A Discourse Commitment-Based Framework for Recogniz-ing Textual Entailment In ACL, pages 171–176 [Lin and Pantel2001] D Lin and P Pantel 2001 Induction of semantic classes from natural language text In SIGKDD, pages 317–322.
[Punyakanok et al.2008] V Punyakanok, D Roth, and
W Yih 2008 The Importance of Syntactic Parsing and Inference in Semantic Role Labeling CL, 34(2) [Rocchio1971] J Rocchio, 1971 Relevance feedback in In-formation Retrieval, pages 313–323 Prentice Hall [Roth and Yih2004] D Roth and W Yih 2004 A linear pro-gramming formulation for global inference in natural lan-guage tasks In CoNLL, pages 1–8.
[Sekine2006] S Sekine 2006 On-Demand Information Ex-traction In COLING/ACL, pages 731–738.