英文培训课件：Towards a Natural User Interface

TowardsaNaturalUserInterface

Kai-FuLeeCorporateVicePresidentNaturalInteractiveServicesDivisionMicrosoftCorporationTalkOutlineTheNUIrevolutionThevision.Theingredientsandtechnologies.5challengestogettingNUIstarted10stepstoovercomechallengesConclusionGraphical

UserInterfaceUserInterfaceEvolutionCommandline1985PC1990GUIMultipleWindowsMenus1995InternetHyperlinksSearchEnginesUserInterfaceEvolution

GraphicalUserInterfaceGUIInteraction

Useristheexpertand

knowsWhatfunctionsexistWheretogoWhattoclick(andinwhatorder)WhattotypeGUITechnology

DirectmanipulationMouse/keyboardinputMouseinputmappedtoWindows,menus,etc.Keyboardinput(search)mappedtotextindexEnablesGraphical

UserInterfaceNatural

UserInterfaceUserInterfaceEvolutionCommandline1985PC1990GUIMultipleWindowsMenus1995InternetHyperlinksSearchEnginesXML

WebServices20002005Anyplace,Anytime,AnydeviceNaturalLanguageMultimodal(speech,ink…)PersonalAssistantNUI–“DoWhatIMean”UIUsersnaturallyarticulatewhattheymean,onanydevice,toanyapplicationorwebservice,andhavetheirintentioninterpretedandexecutedaccurately.WhyNUI?Expressive.Natural.Scalable.NaturalUserInterface

Motivatedbyhuman-humaninteraction“WhoknowsabouttravelingtoChicago?”Books,magazines,other

libraries…Presentchoices“InformationaboutChicago”“Chicagothecity,collegeormusicgroup?”“City”“Historyortravelinformation?”“Travel”InteractingwithaLibrarianUserasksnaturally

“Findinformationabout

Chicago”NUITechnology

NaturalUserInterface(NUI)TextNUITechnologyHearwhatyousayTyped,spoken,written2. Knowwhatyoumean

Analyzetext

ConsidercontextClarifyusingdialog

3. Dowhatyouwant

Brokerandexecuteby

findingcontent/services

Usersays/types:“Informationabout

Chicago”“Iwanttotravelto

Chicago”“BookmyflighttoChicagousing

Expedia”EnablesNUIInteractionNUI:HearWhatYouSayTextNUITechnologyHearwhatyousayTyped,spoken,written2. Knowwhatyoumean

Analyzetext

ConsidercontextClarifyusingdialog

3. Dowhatyouwant

Brokerandexecuteby

findingcontent/services

Usersays/types:“Informationabout

Chicago”“Iwanttotravelto

Chicago”“BookmyflighttoChicagousing

Expedia”EnablesNUIInteractionHearWhatYouSayHumanlanguage:“Invented”forinteraction“[Languageis]abiologicaladaptationtocommunicateinformation…Oneofnature’sengineeringmarvels”

–StevenPinker“Visionevolvedfromtheneedtosurvive;speechevolvedfromtheneedtocommunicate”

–MichaelDertouzos.

Speechiscentral……butalsotyping,handwriting,gestures…Bestformdependson:Habit/skills(typingtoPCs)Formfactor(speakingtophones)Situation(writingatmeetings)Eventually,multimodalNUI:KnowWhatYouMeanTextNUITechnologyHearwhatyousaidTyped,spoken,written2. Knowwhatyoumean

Analyzetext

ConsidercontextClarifyusingdialog

3. Dowhatyouwant

Brokerandexecuteby

findingcontent/services

Usersays/types:“Informationabout

Chicago”“Iwanttotravelto

Chicago”“BookmyflighttoChicagousing

Expedia”EnablesNUIInteractionSyntax(rulesofthehuman’slanguage)Nouns,verbs,etc.andhowtheycombine“BookaboutatriptoChicago”vs.“BookatriptoChicago”Normalizelinguisticvariations.SemanticsMeaningofthewordsBookmeansreserveaticket;requiresfrom-city,to-city,etc.Context(additionalhints)Domainknowledge:NotrainfromHawaiitoChicagoStatistics:Bookasanoun>Bookasaverb“BookChicago”PersonalPreferences:Whereyoulive,yourcalendar,howyoupay…Modeloftime,urgency,

presenceDialog(resolvingambiguity&determineintent)“Buyabookorbooktravel?”“Whatdatewouldyouliketotravel?”KnowWhatYouMean–bycombining:NUI:DoWhatYouWantTextNUITechnologyHearwhatyousaidTyped,spoken,written2. Knowwhatyoumean

Analyzetext

ConsidercontextClarifyusingdialog

3. Dowhatyouwant

Brokerandexecuteby

findingcontent/services

Usersays/types:“Informationabout

Chicago”“Iwanttotravelto

Chicago”“BookmyflighttoChicagousing

Expedia”EnablesNUIInteractionDoWhatYouWant

(bybrokering&combining)Nosystemcanknoweverything.Brokerintenttoexperts:Takeuserintentanddetermine“Whoarethepossibleexpertsforthisintent?”Eachexpert(webservice)registerswhatit

knowsEncartaknowsabouthistory,geography,…Expediaknowsabouttravel(tovariousplaces…)Amazonknowsaboutbooks,bookreviews…Combineallexperts’answers,Content(likeasearch)–FindhistoryofChicago.Action(likeaservice)–BuyatickettoChicago.NUIWillEnable…FindtheBillGatesbookonfutureWhenistheBritneySpearsconcert?HowdoIreplacemyprintercartridge?BuytheGladiatorDVDforlessthan$15SendflowerstomomonherbirthdayContinueworkingonmyannualreportChallengestoNUITechnologylimitationsUserreluctanceUnprovenbusinessvalueHighdevelopmenteffortMorethanUI–infrastructureneededSo,howdowemakeNUIwork?

10evolutionarysteps.ChallengestoNUITechnologylimitationsUserreluctanceUnprovenbusinessvalueHighdevelopmenteffortMorethanUI–infrastructureneeded1.Changetheworld,

onedomainatatimeGeneralunderstandingishard.Domainknowledgecan:Reducelinguisticambiguity:proceeds,IRA…Reducesemanticambiguity:Iwanttocheckin.Cangetbenefitevenfor“verybigdomains”Useofcontextualcueshelpsfurther.“ReasoningEngine”DomainConstraintsExplicitInputContext=implicitinput2.Don’ttrytosolvealltheproblemsatonce….ProbablySolvableNowLong-TermResearchProblemsAuthoringtoolthatlearnsfromuserbehaviorModelingofcontext/domainknowledgeBasicNL/disambiguationdialogOnemodelforspeechandNLHandlecomplexsentencesNaturaldialogsGeneralQ&AoverdocumentsGeneralContextModelMulti-domainreasoningPlanning,webservicescompositionRe-usabledomainandgenerallanguagelibrariesIntegratedsearch(documents,schema,database)Helpandcommands(slotfilling)Q&Afordatabases(e.g.,storedprocedures)3.Bereligiousaboutsolution,

butbepragmaticabouttechnologyDon’tfallinlovewithonetechnology!

Takewhatevertechnologyworks.Ifthereareseveralapproaches,Takethesimplest!Ruleof80-20couldbe99-1forhi-tech!Combinethemifthathelps.4.UseUItohidetechnologyimperfections.

Pushtheintelligence-requiredproblemsbacktothehuman.Request-and-chooseUINatural(insearch).TopNprecision,nottop1precision.OtherUItricks….ChallengestoNUITechnologylimitationsUserreluctanceUnprovenbusinessvalueHighdevelopmenteffortMorethanUI–infrastructureneeded5.NUIcomplementsandextendsGUIIt’sallaboutdiscoverability.Ifitfitsinonescreen,GUIworksgreat.AlwayswanttocombineGUIandNUI.Search&browseisagoodexampleNUIextendsGUI…

evenforsmalldevicesWhenthescreenisthere,useit!Example:SALTformultimodalandtelephony-onlyOneauthoringtoolforalldevices.Extendpresentationlayerdevelopersalreadyknow(e.g.,html,xhtml).6.NUIshouldstartwithuser’scomfortzoneDon’ttrytoaddrevolutionaryUIelements,andmakeuserslearn.Instead,findoutwhereuserstype/speak,andstartthere.Speaking:telephone(routing).Typing:searchengine(finding).Handwriting:annotation.Accommodatetheuserfirst;thenexpandnaturallyfromthere.TelephonePDAPC.Search

navigationhelpcommand.ChallengestoNUITechnologylimitationsUserreluctanceUnprovenbusinessvalueHighdevelopmenteffortMorethanUI–infrastructureneeded7.NUIneedsearlyadopterswithrealeconomicmotivationCooldemosorcheckboxitemswillnotleadtoadoption.Findanearlyadopterwiththeeconomicinventiveorend-userpain.Speechrecognitionforcallcenters,mobiledevices,automobiles.NLtranslationformanuals.SmartsearchforCRM,self-help,e-commerce.Useearlyadopterstoimprovetechnologyandtools.Thenlookforadjacentmarkets.ChallengestoNUITechnologylimitationsUserreluctanceUnprovenbusinessvalueHighdevelopmenteffortMorethanUI–infrastructureneeded8.NUINeedUnifiedAuthoringToolsThemostimportantenablerofNUIisgreatauthoringtools.Sameconceptneededforspeech,search,commands,help,orQ&A.Bridgethegapbetween“problemstatement”and“solutionstatement”:“Iwantsomethingthatpaysinterestandmaybeappreciatealittle,butnottoorisky”–problemstatement(thewayuserthinks)InvestmentArticles>Bonds>ZeroCoupons>Tutorial–solutionstatement(thewayauthorthinks)Organize“problems”withuserlogmining/analysis.Organize“solutions”withre-usabledata:Commontypes(date,time,synonyms…)Linguisticobjects(synonyms,morphology,linguisticnormalizationmodules…)Domainobjects(e.g.,financialterms…)Organizedontology?NeedtoavoidCycproblems.ChallengestoNUITechnologylimitationsUserreluctanceUnprovenbusinessvalueHighdevelopmenteffortMorethanUI–infrastructureneeded9.NUINeedsStructure.TextIntendedTaskTextAvailableTasksSearchEngine

(SiteSearchor

InternetSearch)LossLossLossResultsUserWebServiceUnstructureddocumentsearchTextText.F