严彬-Unicorn：走向目标跟踪的大一统

OverviewOverviewPart1:ReviewofObjectTracking•SingleObjectTracking(SOT)•VideoObjectSegmentation(VOS)•MultipleObjectTracking(MOT)•Multi-ObjectTrackingandSegmentation(MOTS)•SummaryPart2:TowardsGrandUnificationofObjectTracking•GeneralVisionModels(GVM)•UnificationofObjectTracking•Unicorn•Experiments•FurtherAnalysis PartPart1:ReviewofObjectTracking•SingleObjectTracking(SOT)•VideoObjectSegmentation(VOS)•MultipleObjectTracking(MOT)•Multi-ObjectTrackingandSegmentation(MOTS) SingleSingleObjectTracking(SOT)TrackanarbitraryobjectinavideogivenitsinitiallocationSingle-object,Any-classOcclusion,LightChange,BackgroundClutter,etc. zCorrHead !Online !Head !TransformerzHeadxfffSingleObjectTrackingzCorrHead !Online !Head !TransformerzHeadxfffSingleObjectTracking(SOT)SiameseRPNf !fx•SiamRPN(CVPR18)•DaSiamRPN(ECCV18)•SiameRPN++(CVPR19)•Ocean(ECCV20)zDCFx•ATOM(CVPR19)•DiMP(ICCV19)•PrDiMP(CVPR20)•KYS(ECCV20)f !fTransf !f•TransT(CVPR21)•STARK(ICCV21)MostSOTmethodsarebasedonthesearchregion.Pros:Cons:•SavingcomputationV.S•Sensitivetotemporarytrackingfailure•Filteringoutdistractors•Time-consumingwhennumofobjectsislarge UnsupervisedVOSReferringUnsupervisedVOSReferringVOSVideoObjectSegmentation(VOS)nGoalnSegmentspecificobjectspreciselyinavideo.SegmentsalientmovingobjectSemi-supervisedVOSSegmentobjectsgiveninthe1stframebymasksSegmentobjectsgiveninthe1stframebylanguageSTM(ICCVSTM(ICCV19)CFBI(ECCV20)STCN(NeurIPS21)VideoObjectSegmentation(VOS)Semi-supervisedVOSisdominatedbySpace-TimeMemoryNetworkAlthoughachievinggreatperformance,STM-basedmethodssufferfromthefollowingdisadvantages:•Hugetimeandspacecomplexity,especiallyforhighspatialresolutionandthelongsequence.•Highlyrelyingonhigh-qualitymaskannotationsonthefirstframe.MultipleObjectMultipleObjectTracking(MOT)nGoalnTrackallobjectsofspecificclassesinavideo.MOTChallengeBDD100KVisdrone(1class:Person)(8classes:Car,pedestrian,etc)(10classes:Car,pedestrian,etc)ParadigmParadigmMultipleObjectTracking(MOT)RepresentativeMethodsuTrackingbyDetectionuTrackingbyDetection(SORT,DeepSORT,StrongSORT)uJointDetectionandTrackinguJointDetectionandTracking(JDE,FairMOT,CenterTrack,QDTrack)(TrackFormer,GTR)MOTmethodstakesthehigh-resolutionwholeimageastheinputtodetectobjectsascompletelyaspossible.Multi-ObjectTrackingandSegmentation(MOTS)nGoalnSegmentallobjectsofspecificclassesinavideo.MOTSChallengeBDD100KMOTS(1class:Person)(8classes:Car,pedestrian,etc)MOTScanbeseenasavariantofMOTbyreplacingboxeswithmasks.SummarSummaryReferenceOutputsClassTrackspervideoRepresentativeMethodsTypicalInputsSOTInitialboxBoxesagnosticOneOne-ShotDetectionSmallsearchregionVOSInitialmaskMasksagnosticSeveralSTMMedium-resolutionWholeImageMOTNOBoxesspecificTensorhundredsDetection+AssociationHigh-resolutionWholeImageMOTSNOMasksspecificTensorhundredsDetection+AssociationHigh-resolutionWholeImagettherearelargegapsbetweenthefourtrackingtasks•GeneralVisionModels(GVM)•UnificationofObjectTracking•Unicorn•Experiments•FurtherAnalysis entAIvsAGI–CurrentweakAIisdesignedforsolvingonespecifictask.–Artificialgeneralintelligence(AGI)isexpectedtounderstandorlearnanyintellectualtaskthatahumanbeingcan. •Pioneeringworksinthepastyear2021.082021.112021.112022.01ies Threeobstacleshinderingtheunification:(1)Thecharacteristicsoftrackedobjectsvary(onetargetofanyclassgiveninthereferenceframev.stensevenhundredsofinstancesofspecificcategories)(2)SOTandMOTrequiredifferenttypesofcorrespondence.(pixel-levelcorrespondencedistinguishingthetargetfromthebackgroundv.sinstance-levelcorrespondencematchingthecurrentlydetectedobjectswithprevioustrajectories)(3)DifferentInputs.(smallsearchregiontosavecomputationandfilterpotentialdistractorsv.shigh-resolutionfullimagefordetectinginstancesascompleteaspossible) •WeproposeUnicorn,aunifiedsolutionforSOT,MOT,VOSandMOTS.•Unicornaccomplishesthegreatunificationofthenetworkarchitectureandthelearningparadigmforfourtrackingtasks.•Unicornputsforwardsnewstate-of-the-artperformanceonmultiplechallengingtrackingbenchmarkswiththesamemodelparameters. Unifiedinputsandbackbone•Takingthefullimagesasinputsforalltasks.•Referenceframeisthe1stframeforSOT&VOSandthe(t-1)thframeforMOT&MOTS•Oneunifiedbackbone(ConvNeXtbydefault)ErefeRhwxcEcureRhwxcCpixeRhwxhwForMOT&MOTS,TheinstanceembeddingeisextractedfromtheframeembeddingE,wherethecenteroftheinstanceislocatederefeRMxc,ecureRNxcCinsteRNxMCinstisthesub-matrixofCpixLearninghighlydiscriminativeembedding{Eref,Ecur}isthekeytobuildingprecisecorrespondenceforalltrackingtasks.Aninteractionmoduleisusedtoenhancedtheoriginalimagefeature.Bydefaultweusethedeformableattentionblockforinteraction.LearningCorrespondencebyPropagation&LearningCorrespondencebyPropagation&Association.•ForSOT&VOS,Correspondencehelpstopropagatethetargetmapfromthereferenceframetothecurrentframe.•ForMOT&MOTS,Correspondencehelpstomatchthedetectionsonthecurrentframewiththetrajectoriesonthereferenceframe.Weintroducethetargetpriorastheswitchamongfourtrackingtasks.•ForSOT&VOS,thetargetpriorcanenhancetheoriginalFPNfeatureandmakesthenetworkfocusonthetrackedtarget.•ForMOT&MOTS,thefusedfeatureF′degeneratesbacktotheoriginalFPNfeatureFtodetectobjectsofspecificclasses.ObjectObjectdetectionheadbasedonYOLOXandCondInst•One-stage,anchor-free•NoRoIoperationssuchasRoI-AlignYOLOXHeadforobjectdetectionCondInstHeadforinstancesegmentationAddthemaskbranchandfreezeotherparametersStage1Target:Correspondence+DetectionLoss:Lstage1=Lcorr+LdetData:1:1fromSOT&MOTSOT:weuseCOCO,LaSOT,GOT-10KandTrackingNetMOT:•ForMOT17,weuseCrowdhuman,ETHZ,CityPerson,MOT17•ForBDD100K,weuseBDD100KStage2Target:MaskLoss:Lstage2=LmaskData:1:1fromVOS&MOTSVOS:weuseCOCO,DAVIS,Youtube-VOSMOTS:•ForMOTS,weuseCOCOandMOTS•ForBDD100K,weuseBDD100K•TrainingofVOS&MOTSwouldnotimpacttheperformanceofSOT&MOT.ForuserswhoareonlyinterestedintheSOT&MOT,runningStage1isenough.•Ineachstage,wetrainthemodel