自然语言处理介绍

1、Introduction to Introduction to NLPNLPWhat is Natural Language Processing?Dan JurafskyQuestion Answering: IBMs Watson Won Jeopardy on February 16, 2011!2WILLIAM WILKINSONS “AN ACCOUNT OF THE PRINCIPALITIES OFWALLACHIA AND MOLDOVIA”INSPIRED THIS AUTHORSMOST FAMOUS NOVELBram StokerDan JurafskyInformat

2、ion ExtractionSubject: curriculum meeting Date: January 15, 2012 To: Dan JurafskyHi Dan, weve now scheduled the curriculum meeting.It will be in Gates 159 tomorrow from 10:00-11:30.-Chris3 Create new Calendar entryEvent: Curriculum mtgDate: Jan-16-2012Start: 10:00amEnd: 11:30amWhere: Gates 159Dan Ju

3、rafskyInformation Extraction & Sentiment Analysis nice and compact to carry! since the camera is small and light, I wont need to carry around those heavy, bulky professional cameras either! the camera feels flimsy, is plastic and very light in weight you have to be very delicate in the handling

4、of this camera4Size and weightAttributes: zoom affordability size and weight flash ease of useDan JurafskyMachine Translation Fully automatic5 Helping human translatorsEnter Source Text:Translation from Stanfords Phrasal:这 不过 是 一 个 时间 的 问题 .This is only a matter of time.Dan JurafskyLanguage Technolo

5、gyCoreference resolutionQuestion answering (QA)Part-of-speech (POS) taggingWord sense disambiguation (WSD)ParaphraseNamed entity recognition (NER)ParsingSummarizationInformation extraction (IE)Machine translation (MT)DialogSentiment analysis mostly solvedmaking good progressstill really hardSpam det

6、ectionLets go to Agra!Buy V1AGRA Colorless green ideas sleep furiously. ADJ ADJ NOUN VERB ADVEinstein met with UN officials in PrincetonPERSON ORG LOCYoure invited to our dinner party, Friday May 27 at 8:30PartyMay 27addBest roast chicken in San Francisco!The waiter ignored us for 20 minutes.Carter

7、told Mubarak he shouldnt run again.I need new batteries for my mouse.The 13th Shanghai International Film Festival第13届上海国际电影节开幕The Dow Jones is upHousing prices roseEconomy is goodQ. How effective is ibuprofen in reducing fever in patients with acute febrile illness?I can see Alcatraz from the windo

8、w!XYZ acquired ABC yesterdayABC has been taken over by XYZWhere is Citizen Kane playing in SF? Castro Theatre at 7:30. Do you want a ticket?The S&P500 jumpedDan JurafskyAmbiguity makes NLP hard:Ambiguity makes NLP hard:“Crash blossoms”“Crash blossoms”Violinist Linked to JAL Crash BlossomsTeacher

9、 Strikes Idle KidsRed Tape Holds Up New BridgesHospitals Are Sued by 7 Foot DoctorsJuvenile Court to Try Shooting DefendantLocal High School Dropouts Cut in Half100%REALDan JurafskyAmbiguity is pervasiveAmbiguity is pervasiveFed raises interest ratesNew York Times headline (17 May 2000) Fed raises i

10、nterest ratesFed raises interest rates 0.5%Dan JurafskyIn-video quizzes! Most lectures will include a little quiz Just to check basic understanding Simple, multiple-choice. You can retake them if you get them wrong9Dan Jurafskynon-standard EnglishGreat job justinbieber! Were SOO PROUD of what youve

11、accomplished! U taught us 2 #neversaynever & you yourself should never give up eithersegmentation issuesidiomsdark horseget cold feetlose facethrow in the towelneologismsunfriendRetweetbromancetricky entity namesWhere is A Bugs Life playing Let It Be was recorded a mutation on the for gene world

12、 knowledgeMary and Sue are sisters.Mary and Sue are mothers.But thats what makes it fun!the New York-New Haven Railroadthe New York-New Haven RailroadWhy Why else is else is natural language natural language understanding difficult?understanding difficult?Dan JurafskyMaking progress on this problem

13、The task is difficult! What tools do we need? Knowledge about language Knowledge about the world A way to combine knowledge sources How we generally do this: probabilistic models built from language data P(“maison” “house”) high P(“Lavocat gnral” “the general avocado”) low Luckily, rough text featur

14、es can often do half the job.Dan JurafskyThis classTeaches key theory and methods for statistical NLP: Viterbi Nave Bayes, Maxent classifiers N-gram language modeling Statistical Parsing Inverted index, tf-idf, vector models of meaningFor practical, robust real-world applications Information extraction Spelling correction Information retrieval Sentiment analysisDan