[not-yet-released

1、Subjectivity Annotation UpdateJosef RuppenhoferJan WiebeOutlineUpdate on our annotationsExploration of subjectivity and Discourse Treebank annotationsSubjectivity AnnotationBefore this year: MPQA annotation scheme and corpus /mpqaEnglish language versions of articles from the world press (187 news s

2、ources)535 Documents; 11,114 sentencesWiebe, Wilson, Cardie. Annotating Expressions of Opinions and Emotions in Language. LRE 2005.Subjectivity AnnotationCurrent work (goals have been expanded schemes, not high volume annotation) Extended MPQAAdditional data annotated2005 LRE scheme plus extensions

3、Theresa Wilsons PhD dissertation (2008)not-yet-released extensions added to the MPQA corpusDiscourse level relations between opinionsSubjectivity in health surveillance textsWord sense subjectivity and polarityExtended MPQA SchemeDocuments 85 Xbank files“Boyan” subset of ULA data1/3 completedexpecte

4、d completion: early summerMPQAExtended MPQA SchemeAnnotatorsJoseftwo undergraduatesTraining Time & EffortTrainingJosefs effort: 75 hours ( 2 weeks)10 preparing materials40 basic training25 extensionsAnnotators: 120 hours combined (1.5 weeks each)Problems with scheduling arose (annotators did not wor

5、k planned hours per week; redundant one-on-one meetings)With perfect scheduling, estimate 1 week to train two annotators (though Josef is involved in production annotation)Production annotationSingle annotator per documentAnnotator time per document (very rough est) 2 hours 45 mins 45 mins of which

6、is time spent on consultation, 15 with each other, 30 with JosefPeriodic Agreement Testing documents with known gold standardno consultationevery 5 documentspost-mortem meetings (one on one, group)Four annotations to compare (Theresa Wilson, Josef, two undergraduate annotators)Results of previous ag

7、reement studies in previous papersAgreement measurementSo far, average pair-wise agreement calculated per documentFull analysis forthcomingRelative label reliability:agent direct-subjective target attitude objective-speech-event expressive-subjective-elementGiven the interactions between the labels,

8、 errors are interrelatedAnnotation SchemesWhat is Subjectivity?The linguistic expression of somebodys opinions, sentiments, emotions, evaluations, beliefs, speculations (private states)Private state: state that is not open to objective observation or verification Quirk, Greenbaum, Leech, Svartvik (1

9、985). A Comprehensive Grammar of the English Language.OverviewFine-grained: expression-level rather than sentence or document levelAnnotate Subjective xpressionsmaterial attributed to a source, but presented objectivelyOverviewFocus on three ways private states are expressed in languageDirect Subjec

10、tive ExpressionsDirect mentions of private states The United States fears a spill-over from the anti-terrorist campaign.Private states expressed in speech events “We foresaw electoral fraud but not daylight robbery,” Tsvangirai said.Expressive Subjective Elements Banfield 1982“We foresaw electoral f

11、raud but not daylight robbery,” Tsvangirai saidThe part of the US human rights report about China is full of absurdities and fabricationsObjective Speech EventsMaterial attributed to a source, but presented as objective fact The government, it added, has amended the Pakistan Citizenship Act 10 of 19

12、51 to enable women of Pakistani descent to claim Pakistani nationality for their children born to foreign husbands.Nested Sources“The US fears a spill-over, said Xirao-Nima, a professorof foreign affairs at the Central University for Nationalities.(writer, Xirao-Nima, US) (writer, Xirao-Nima)(writer

13、)“The report is full of absurdities, he continued.(writer, Xirao-Nima)(writer, Xirao-Nima)(writer)“The report is full of absurdities,” Xirao-Nina said.Objective speech event anchor: the entire sentence source: implicit: trueDirect subjective anchor: said source: intensity: high expression intensity:

14、 neutral attitude type: negative target: reportExpressive subjective element anchor: full of absurdities source: intensity: high attitude type: negative Objective speech event anchor: the entire sentence source: implicit: trueObjective speech event anchor: said source: Direct subjective anchor: fear

15、s source: intensity: medium expression intensity: medium attitude type: negative target: new work “The US fears a spill-over, said Xirao-Nima, a professor of foreign affairs at the Central University for Nationalities.ExtensionsWilson 2008I think people are happy because Chavez has fallen.direct sub

16、jective span: are happy source: attitude:inferred attitude span: are happy because Chavez has fallen type: neg sentiment intensity: medium target: target span: Chavez has fallentarget span: Chavezattitude span: are happy type: pos sentiment intensity: medium target:direct subjective span: think sour

17、ce: attitude:attitude span: think type: positive arguing intensity: medium target:target span: people are happy because Chavez has fallenSubjectivity TypesWilson 2008Other (esp. general cognition)Discourse-Level Opinion Frames in Task-Oriented Dialogs (AMI)Frames are defined in terms of their compon

18、entsOpinion spansOpinion typeSentimentArguingOpinion PolarityTargetsSourcesRelationships between targetsSame or alternativeExample motivation: polarity and targets interactE.g. an argument for one design that is simultaneously an argument against an alternative designSubjectivity in Health Surveilla

19、nce TextsTypesSentimentBeliefBelief about what is the caseBelief about what should or should not be doneKnowledge/Awareness of factsAgreement/Disagreement between sources in the textSourcesWriterMediaNon-media organizationsMembers of the general publicTargetsOccurrence of a disease outbreakDanger/se

20、verity of an outbreakCause of a diseaseSymptomsExploring the relationship between PDTB (2) and Extended MPQAPenn Discourse TreeBankHierarchy of discourse relations with 4 top nodesTemporalContingencyComparisonExpansionOverviewRicher interpretations via combinationPotential disambiguation both ways A

21、 connective is marked as “Restatement” when it indicates that the semantics of Arg2 restates the semantics of Arg1. It is inferred that the situations described in Arg1 and Arg2 hold true at the same time.ExpansionRestatementGeneralizationSpecificationEquivalenceSubjectivity preserved in a restateme

22、nt“This means Nestle is now in the candybar business in a big way, said Lisbeth Echeandia, publisher of Orlando, Fla.-based Confectioner Magazine. “For them, it makes all kinds of sense.”Subjectivity preserved in a restatement: PDTB“This means Nestle is now in the candybar business in a big way ARG1

23、, said Lisbeth Echeandia, publisher of Orlando, Fla.-based Confectioner Magazine. “IMPLICIT_IN SHORT For them, it makes all kinds of sense ARG2.Subjectivity preserved in a restatementThis means Nestle is now in the candybar business in a big way ARGUING-POS, said Lisbeth Echeandia, publisher of Orla

24、ndo, Fla.-based Confectioner Magazine. “For them, it makes all kinds of sense ARGUING-POS.Related opinions; part of the same larger opinionSubjectivity preserved in a restatementThis means Nestle is now in the candybar business in a big way ARGUING-POS, said Lisbeth Echeandia, publisher of Orlando,

25、Fla.-based Confectioner Magazine. “For them, it makes all kinds of sense ARGUING-POS.Same polarity, type, source; Hyp: common pattern with restatementSubjectivity preserved in a restatementThis means Nestle is now in the candybar business in a big way ARGUING-POS, said Lisbeth Echeandia, publisher o

26、f Orlando, Fla.-based Confectioner Magazine. “For them, it makes all kinds of sense ARGUING-POS.Semantics of restatement: sameness includes subjectivityNote: Sentiment This means Nestle is now in the candybar business in a big way, said Lisbeth Echeandia, publisher of Orlando, Fla.-based Confectione

27、r Magazine. “For them, it makes all kinds of sense SENTIMENT-POS.”Not directly part of the restatement relationThe type “Cause” is used when the connective indicates that the situations described in Arg1 and Arg2 are causally influenced and the two are not in a conditional relation ContingencyCauseR

28、easonResultSubjectivity preserved across Reason relationBut Mr. Schwarz welcomes the competition in U.S. Trusts flagship businesses, calling it flattery. Mr. Schwarz says the competition broadens the base of opportunity for us. Other firms are dealing with the masses”Subjectivity preserved across Re

29、ason relation: PDTBBut Mr. Schwarz welcomes the competition in U.S. Trusts flagship businesses ARG1, calling it flattery SUP1. Mr. Schwarz says IMPLICIT_BECAUSE the competition broadens the base of opportunity for us ARG2. Other firms are dealing with the masses”ARG2 is a reason for ARG1Subjectivity

30、 preserved across Reason relation: subjectivityBut Mr. Schwarz welcomes SENTIMENT-POS the competition in U.S. Trusts flagship businesses, calling it flattery. Mr. Schwarz says the competition “broadens the base of opportunity for us SENTIMENT-POS. Other firms are dealing with the masses.Positive eva

31、luation which is a reason for a positive feeling; same overall opinion“I like it because it is so good”Subjectivity preserved across Reason relation: subjectivityBut Mr. Schwarz welcomes SENTIMENT-POS the competition in U.S. Trusts flagship businesses, calling it flattery. Mr. Schwarz says the compe

32、tition “broadens the base of opportunity for us SENTIMENT-POS. Other firms are dealing with the masses.Subjectivity: same source, target, polarity, type; Hyp: common with reason; Help with target recognition, for example.Subjectivity preserved across Reason relation: subjectivityBut Mr. Schwarz welc

33、omes SENTIMENT-POS the competition in U.S. Trusts flagship businesses, calling it flattery. Mr. Schwarz says the competition “broadens the base of opportunity for us SENTIMENT-POS. Other firms are dealing with the masses.Semantics of reason: specific subtype, where an evaluation is a reason for an a

34、ttitudeThe type “Cause” is used when the connective indicates that the situations described in Arg1 and Arg2 are causally influenced and the two are not in a conditional relation ContingencyCauseReasonResultPolarity preserved across Result relationOther firms are dealing with the masses. I dont beli

35、eve they have the culture to adequately service high-net-worth individuals, he adds.Polarity preserved across Result relation: PDTBOther firms are dealing with the masses ARG1. I dont believe IMPLICIT_SO they have the culture to adequately service high-net-worth individuals ARG2, he adds.ARG2 is a r

36、esult of ARG1Polarity preserved across Result relation: PDTBOther firms are dealing with the masses ARG1. I dont believe IMPLICIT_SO they have the culture to adequately service high-net-worth individuals ARG2, he adds.X said Y: “X said” Xs belief space“I dont believe” explicit in second sentence“Swa

37、rtz said” implicit in first sentenceARG spans: Dis. Rel within Swartzs belief spacePolarity preserved across Result relation: subjectivityOther firms “are dealing with the masses SENTIMENT-NEG. I dont believe they have the culture to adequately service high-net-worth individuals SENTIMENT-NEG, he ad

38、ds.Attitude span includes “dont believe”; schemes require different notions of spansPolarity preserved across Result relation: subjectivityOther firms “are dealing with the masses SENTIMENT-NEG. I dont believe they have the culture to adequately service high-net-worth individuals SENTIMENT-NEG, he a

39、dds.Two negative properties, where the second is a result of the firstPolarity preserved across Result relation: subjectivityOther firms “are dealing with the masses SENTIMENT-NEG. I dont believe they have the culture to adequately service high-net-worth individuals SENTIMENT-NEG, he adds.Dis Rel be

40、tween ARGS inside his belief spacePolarity preserved across Result relation: subjectivityOther firms “are dealing with the masses SENTIMENT-NEG. I dont believe they have the culture to adequately service high-net-worth individuals SENTIMENT-NEG, he adds.Semantics of result: specific subtype, where a

41、 negative state of affairs is the result of another oneThe class tag “COMPARISON” applies when the connective indicates that a discourse relation is established between Arg1 and Arg2 in order to highlight prominent differences between the two situations. Semantically, the truth of both arguments is

42、independent of the connective or the established relation.ComparisonContrastJuxtapositionOppositionConcessionexpectationContra-expectationIn that suit, the SEC accused Mr. Antar of engaging in a massive financial fraud to overstate the earnings of Crazy Eddie, Edison, N.J., over a three-year period.

43、Through his lawyers, Mr. Antar has denied allegations in the SEC suit and in civil suits previously filed by shareholders against Mr. Antar and others. PDTBIn that suit, the SEC accused Mr. Antar of engaging in a massive financial fraud to overstate the earnings of Crazy Eddie, Edison, N.J., over a

44、three-year period. ARG1IMPLICIT_HOWEVER Through his lawyers, Mr. Antar has denied allegations in the SEC suit and in civil suits previously filed by shareholders against Mr. Antar and others. ARG2Contrast between the SEC accusing Mr. Antar of something, and his denying the accusationSubjectivityIn t

45、hat suit, the SEC accused SENTIMENT-NEG Mr. Antar of engaging in a massive financial fraud to overstate the earnings of Crazy Eddie, Edison, N.J. ARGUING-POS, over a three-year period.Through his lawyers, Mr. Antar has denied AGREE-NEG allegations in the SEC suit and in civil suits previously filed

46、by shareholders against Mr. Antar and others. Two attitudes combined into one large disagreement between two partiesSubjectivityIn that suit, the SEC accused SENTIMENT-NEG Mr. Antar of engaging in a massive financial fraud to overstate the earnings of Crazy Eddie, Edison, N.J. ARGUING-POS, over a three-year period.Through his lawyers, Mr. Antar has denied AGREE-NEG allegations in the SEC suit and in civil suits previously filed by shareholders against Mr. Antar and others. Subjectivity: arguing-