Hierarchical-Multitream-Posterior-Based-Speech-recognition-…分层多流后语音识别…教学课件

IntroductiontothethesisExistingkeywordspottingapproachesareusuallybasedonspeechrecognitiontechniquesGrowingapartfromtheoriginalproblemcanleadtodrawbacks,likelackofgeneralityAnotherapproachispresentedandstudied,whereonlythetargetsoundsofthekeywordarelookedforTostudyandformulatethisapproachwasmyworkatIDIAPResearchInstitute,3/2019-8/2019Objectiveotthethesis:toseehowfarcanwegowithoutusinghiddenMarkovmodelsanddynamicprogrammingtechniques9/21/20191HierarchicalApproachforSpottingKeywordsOutlineIntroductiontokeywordspotting 4-7Motivationforthiswork 8

Stepsofhierarchicalprocessing 9-14Experiments 15-20Conclusions 219/21/20192HierarchicalApproachforSpottingKeywordsKeywordSpottingKeywordSpotting(KWS)aimsatfindingonlycertainwordswhilerejectingtherest(hypothesis–test)Findingonlycertain,rareandhigh-information-valuedwordsisfeasibleapproachinforexamplevoicecommanddrivenapplicationsormultimediaindexingPicturefrom[Jun96]9/21/20193HierarchicalApproachforSpottingKeywordsPerformancemeasuresforkeywordspottingThepossibleeventsinkeywordspottingarehit,falsealarmandmissTheperformanceisevaluatedbypresentingthedetectionrateasfunctionofthefalsealarmrateThisyieldsthereceiveroperatingcharasteristics(ROC)curveAveragedetectionratein0-10falsealarmsperhouriscalledfigureofmerit(FOM)[Roh89]FalseAlarms/HourKeywordsdetected/%9/21/20194HierarchicalApproachforSpottingKeywordsLVCSR/HMMbasedapproachesTypicallargevocabularycontinuousspeechrecognition(LVCSR)/hiddenMarkovmodel(HMM)basedKWSapproachesmodelbothkeywordsandnon-keywords(backgroundorgarbage)Keywordsaresearchedbyusingdynamicprogrammingtechniques

Keywordspottingnetworkfrom[Roh89].YXx1xNy1yM OptimalalignementbetweenXandYAnexampleofdynamicprogramming.9/21/20195HierarchicalApproachforSpottingKeywordsLVCSR/HMMbasedapproachesvs.hypothesistestapproachLABEL:um...okay,uh...please

open

the,uh...

window

Spot1:---------------------------1111-------------------

Spot2:--------------------------------------------111111Recog:garbagegarbagegarbage-OPEN–garbage–WINDOW

word1YesNotime9/21/20196HierarchicalApproachforSpottingKeywordsMotivationforthisworkTypicalLVCSR/HMMbasedapproachesrequiregarbagemodelforViterbidynamicprogrammingThebetterthegarbagemodel,thebetterthekeywordspottingperformance[Ros90]......andthecloserthesystemistoLVCSRUseofLVCSRtechniquescanintroducetaskdependency,lackofgeneralitycomputationalload,complexityneedfortrainingdataoff-lineoperatingmodecomplexitytoaddkeywords Howfarcanwegobylookingonlyatthekeysounds?9/21/20197HierarchicalApproachforSpottingKeywordsHierarchicalapproachfor

spottingkeywordsKeysounds(words)arespottedbylookingforthetargetsounds(phonemes)thatformthekeysound.STEP1:EstimateequallysampledphonemeposteriorsSTEP2:Derivephoneme-spacedposteriorestimatesSTEP3:Searchrightsequencesofhigh-confidencephonemesALARM9/21/20198HierarchicalApproachforSpottingKeywordsStep1:Fromacousticstream

tophonemeposteriorsTRAP-NNsystem:Featureextractionfrom2-Dfilteringofcriticalbandspectrogram,using1010mslongtemporalpatterns(TRAPs)Featuresarefedtoatrainedneuralnet(NN)vectorclassifierthatreturnsestimatesofphonemeposteriorprobabilitiesevery10msTRAP-NNwassuccesfullyusedin[Szö05]forphonemebasedkeywordspotting9/21/20199HierarchicalApproachforSpottingKeywordsStep2:Fromframe-basedphonemeposteriorstophoneme-spacedposteriorsPhonemesarefoundbyfilteringtheposteriogramwithabankofmatchedfiltersMatchedfiltersareobtainedbyaveraging0.5slongsegmentsofphonemetrajectoriesThepurposeoffilteringistohaveonepeakperphoneme9/21/201910HierarchicalApproachforSpottingKeywordsStep2:Fromframe-basedphonemeposteriorstophoneme-spacedposteriors(2)Thelocalmaxima(peaks)ofthefilteredposteriogramareextractedandtakenasestimatesofunderlyingphonemesbeingpresentTheplacesofthepeakscorrespondtothecenterframesoftheunderlyingphonemes:9/21/201911HierarchicalApproachforSpottingKeywordsStep2:Fromframe-basedphonemeposteriorstophoneme-spacedposteriors(3)Matchedfilterbank,estimatedfrom30,000phonemesofthetrainingdata(englishnumbers)Filterlengthsare41samples(210msprocessingdelay)9/21/201912HierarchicalApproachforSpottingKeywordsStep3:FromphonemeestimatestowordsMethod1:AposteriorthresholdisappliedforphonemeestimatesAnalarmissetforacorrectstreamofphonemesMinimumandmaximumintervalsbetweenphonemesaredefinedfromthetrainingdata OnlytheprimarylexicalformofeachwordissearchedThreshold9/21/201913HierarchicalApproachforSpottingKeywordsExperimentsTwotelephonecorporawereused[Col94,Col95]:TheMLPwastrainedtoestimatetheposteriorprobabilitiesof28Englishphonemes+silence(numbersfromzerotoninety-nine)Aseparatekeywordspotterwasimplementedforalldigitsfromzerotonine,withonlytheprimarylexicalformsResultswerecomparedtotime-alignedphonemiclabeling,andalllegalpronunciationsweretreatedastruealarms9/21/201914HierarchicalApproachforSpottingKeywordsResults–Experiment1

(phonemeestimatesonly) Keywordspottingresults(FOM)fromspottingdigitsinthestreamofotherdigits(OGI-Numbers95),experiment1(onlyphonemeestimates) Twomainreasonsfordifferenciesinperformance:

1.Somephonemesmoreprone toclassificationerrors 2.Theprobabilitythatakeywordismixedwithanotherwordisnotconstant9/21/201915HierarchicalApproachforSpottingKeywordsIntroductionofphonemetransitionprobabilityIntroductionofaconfidencemeasurethattells,arethereextraneousphonemesbetweentwophonemes

phonemetransitionprobability:Phonemetransitionprobabilityisestimatedusing:Strategy1:theheightofthecrossingpointofposteriortrajectoriesofthecorrespondingphonemesStrategy2:theheightofthecrossingpointoffilteredposteriortrajectoriesStrategy3:oneminustheminimumofthesumoftheposteriorsofthecorrespondingphonemes,betweenthephonemeestimatesNewmethodforStep3(withtransitionprobabilities):Theposteriorthresholdofappliedtotheproductofphonemeandtransitionestimates:9/21/201916HierarchicalApproachforSpottingKeywordsResults–Experiment2

(Phonemeandtransitionestimates) Keywordspottingresults(FOM)fromspottingdigitsinthestreamofotherdigits(OGI-Numbers95),experiment2(withphonemetransitionprobabilityestimates) TheaverageincreaseinFOMcomparedtofirstexperimetis5.6% Onlysmalldifferenciesbetweendifferentstrategiesofderivingthephonemetransitionestimates.9/21/201917HierarchicalApproachforSpottingKeywordsROCcurve–’zero’9/21/201918HierarchicalApproachforSpottingKeywordsROCcurve-’eight’9/21/201919HierarchicalApproachforSpottingKeywordsConclusionsAtheoreticalframeworkforkeysoundspottingwasintroducedandusedtospotdigits.Besideskeywordspotting,theproposedprocessingcanbeappliedin:Phonemedetection(experimentedinthethesis)EventspottingingeneralThisapproachhasnogarbagemodelandnodynamicprogrammingtechniquesorHMMsareusedBenefitsfromlookingonlyatthetargetsounds:IndependencefromvocabularySomeindependecefromlanguageLessneedfortrainingthemodelsSimpleandfastReliesonreliablephonemeestimatesQuiterobustforthechoiceofmatchedfilterandphonemesequencesearchtechniqueHighvarianceinresultsbetweendifferentwordsShortphonemesyieldweakerestimatesRoomtoimprovetheperformanceTreatclosureformsofplosivephonemesLookforallthepossiblepronunciationformsUsethenon-keywordphonemeestimatestoextractcomplementaryinformationIntroducepriorlexicalknowledge9/21/201920HierarchicalApproachforSpottingKeywordsQuestions?[Jun96] Junqua,J.C.,HatonJ.-P.:RobustnessinAutomaticSpeechRecognition,

FundamentalsandApplications.Dordrecht,TheNetherlands,KluwerAcademic Publishers,2019.[Roh89] Rohlicek.,J.,Russel,W.,Roukos,S.,Gish,H.:ContinuousHiddenMarkovModeling ForSpeaker-IndependentWord-Spotting.InICASSP89,pp.627-630,1989. [Ros90] Rose,R.,Paul,D.:AHiddenMarkovModelBasedKeywordRecognition System.InProceedingsofICASSP90,pp.129-132,Albuquerque,NewMexico, UnitedStates,1990.[Szö05] Szöke,I.,SchwarzP.,MatejkaP.,BurgetL.,FapsoM.,KarafiátM.,CernockýJ.: ComparisonofKeywordSpottingApproachesforInformalContinuousSpeech.In MLMI05,Edinburgh,UnitedKingdom,July2019.[Col94] Cole,R.etal.:TelephoneSpeechCorpusDevelopmentatCSLU.InProceedingsof ISCLP'94,pp.1815-1818,Yokohama,Japan,1994.[Col94] Cole,R.etal.:NewTelephoneSpeechCorporaatCSLU.InProceedingsof Eurospeech'95,pp.821-824,Madrid,Spain,2019.

Lehtonen,M.,Fousek,P.,Hermansky,H.:AHierarchicalApproachforSpotting Keywords.In2ndWorkshoponMultimodalInteractionandRelatedMachineLearning Algorithms–MLMI05,Edinburgh,UnitedKingdom,July2019.

9/21/201921HierarchicalApproachforSpottingKeywordsAppendix:ApplicationtophonemedetectionThephonemeestimatesofStep2wereusedinphonemedetectionThephonemestreamwasestimatedbycountingallthephonemeestimatesoverathreshold,withdifferentthresholdvaluesResultswereestimatedintermsofsubstitutios(S),insertionts(I)anddeletions(D)Forexample(N=Numberofphonemesinlabeling): Labeled:sehvahnfayvRecognized:silnehvnfayvOperation:ISD9/21/201922HierarchicalApproachforSpottingKeywordsAppendix:Applicationtophonemedetection(cont)Resultsfromphonemedetection:ThresholdAccuracy0.01-93.21%0.0528.57%0.1054.35%0.1564.37%0.2069.44%0.2571.24%0.3070.50%0.3568.12%0.4064.57%Takingintoaccountalsothetransitionprobabilitiesyielded73.15%accuracy.State-of-the-artphonemerecognitionaccuracyforunrestrictedspeech67%-77%.9/21/201923HierarchicalApproachforSpottingKeywordsAppendix:Systemdiagram9/21/201924HierarchicalApproachforSpottingKeywordsAppendix:Conclusions(table)Whataffects/determinestheperformancePlacesforimprovementStep1

(fromacousticstreamtophonemeposteriors)Phoneme’spronesstoclassificationerrorsPhoneme’sduration(longerphonemesyieldstrongerposteriors)TotreattheclosureformphonemesStep2

(fromframe-basedposteriorstophoneme-spacedposteriors)HowthematchedfiltermodelsthedurationofthephonemeToadaptthefilterlengthsmorepreciselytothephonemedurations(e.g.throughspeechrate)Ste