TiDB+实践+HTAP+的架构进展和未来展望-韦万

InfoQInfoQTheHTAParchitecturedesignofTiDB,andtheimprovementinv6.2WeiWan@PingCAPArchSummit全球架构师峰会

AboutMeWeiWan,workatPingCAP,astheleaderofOLAPStorageteam.Over11yearsofexperienceingame,e-commerce,mobileapps,anddatabasedevelopment.ArchSummit构师峰会InfoQArchSummit构师峰会InfoQTiDBIntroductionTiDBisanopen-sourceNewSQLdatabasethatsupportsHTAPworkloads.ItisMySQLcompatibleandfeatureshorizontalscalability,strongconsistency,andhighavailability.ThegoalofTiDBistoprovideuserswithaone-stopdatabasesolutionthatcoversOLTP(OnlineTransactionalProcessing),OLAP(OnlineAnalyticalProcessing),andHTAPservices.ArchSummit全球架构师峰会ArchSummit ArchSummit InfoQArchSummit ArchSummit InfoQInfoQInfoQAgendaAtypicalusercaseThechallengestostoragemoduleonHTAPscenarioTheimprovementsinTiDBv6.2TiDB'sfuturearchitecturalevolutiondirectionArchSummit全球架构师峰会

TiDBCoreArchitectureAtypicalusercaseofHTAPworkloadsZTO-运单系统(ExadatatoTiDB)ExadataTOTiDB消息中间件，PingCAP应用ZTO-运单系统(ExadatatoTiDB) 丿Spo^fTiDB■ 丿Spo^fTiDB■ADistributedSQLDatabaseStreaming10Billions+Records丿multipledatasourcesArchSummit全力。而峰会InfoQZTO-运单系统(ExadatatoTiDB)UnlimitedScalabilityNotmoresharding“分库分表达到16000张表，业务上已经无法再继续扩展下去”GreatlyreducedtheworkloadofbusinessdevelopmentThelatencyofreportreducedfrom30mto2mReducedthecostofdatabaseservers.BigdatainfrastructureadaptionInfoQArchSummitInfoQArchSummitThechallengestostoragemoduleonHTAPscenarioIsolationbetweenOLTPandOLAPworkloadsIsolationbetweenOLTPandOLAPworkloadsArchSummit ArchSummit InfoQIsolationbetweenOLTPandOLAPworkloadsIsolationbetweenOLTPandOLAPworkloadsArchSummit ArchSummit InfoQIsolationisdifficultifwemixtheminthesamenodeTPandAPscaleseparatelyDifferenthardwarerequirementsDifferentbestdatastructures.RowbasedvscolumnbasedIndexIsolationbetweenOLTPandOLAPworkloadsIsolationbetweenOLTPandOLAPworkloadsIsolationbetweenOLTPandOLAPworkloadsIsolationbetweenOLTPandOLAPworkloadsPlacementDriverSQLLayerTiDB/TiSparkPD1PD1PD1Key-valuestoragelayerTiKVnode1Store1Region1Region3Region4•••RegionnTiKVnode21TiKVnode311Row-basedstorageforOLTPworkloadTiKVnode4TiFlashnode2Store4Region1Region2Region4RegionnTiFlashnode1Colunmn-basedstorageForOLAPworkloadInfoQInfoQInfoQInfoQIsolationbetweenOLTPandOLAPworkloads(HATtrickBench)(sdb)Andll6no」llJ_Fixed-Tlines.4.3.2,100 1 2 3 4 5Fixed-Alines0 1 2 3 4 5Frontier0 12 3 4(sdb)Andll6no」llJ_Fixed-Tlines.4.3.2,100 1 2 3 4 5Fixed-Alines0 1 2 3 4 5Frontier0 12 3 4AC=amaxXILSocnCILL50TC=0―8—TC=0.1*Tmax—TC=0.2^TmaxTC=0.5*Tmax—TC=0.8*Tmax°TC=Tmax—AC=011厶C=0.1*Qmax—AC=0.2^amax''厶。=°.5*(XmaxT—AC=0.8*amaxTransactinalThroughput(103tps)ArchSummit全球架构师峰会

RealtimesynchronizationandstrongconsistencyRaftLearnerReadArchSummit全球架构师峰会RealtimeupdateofcolumnarstoreRealtimeupdateofcolumnarstoreInfoQInfoQRealtimeupdateofcolumnarstoreRealtimeupdateofcolumnarstoreInfoQInfoQRealtimeupdateofcolumnarstoreRealtimeupdateofcolumnarstoreArchSummit ArchSummit ThebigissueisthebalancebetweenREADandWRITE.HighfrequencyupdatebringsfragmentationToomanyfilesslowsdownwritespeedColumnarstoreneedstosplitcolumnsintodifferentfilesToomanyIOPSTransactionsupportneedstostoremultipleversionsSortMergeisslowThesolutionofDeltaTree,thestorageengineofTiFlashIntroducesrowversionsTransformupdatesanddeletesintoappendoperationUseWALandMem-table,tobatchupdatesintosmallgroupsToavoidtoomanyIOPSandfilesAdoptsDelta+StableLayerarchitectureEachlayersusingdifferentstoragestrategiesAcceleratesSortMergebyDeltaindexArchSummitInfoQInfoQInfoQInfoQThewritepathofDeltaTreeBlockwrite()ColumnFilelnMemoryMemTableSetDeltaLayerMemTable(InMemory)FlushColumnFile

FlushTaskArchSummit]□PageStoragesegmentMergeDeltaDeltaLayer(InDisk)OrderedMergeDeltaBlockwrite()ColumnFilelnMemoryMemTableSetDeltaLayerMemTable(InMemory)FlushColumnFile

FlushTaskArchSummit]□PageStoragesegmentMergeDeltaDeltaLayer(InDisk)OrderedMergeDeltaStableLayer(InDisk)AdoptingDeltalndextoacceleratescanspeedThescanspeedofDeltaTreeis3xofClickHouseinSELECT…FINALArchSummit0址W ilcfi4ArchSummit ArchSummit InfoQArchSummit ArchSummit InfoQTheimprovementsinTiDBv6.2InfoQInfoQInfoQInfoQPageStoragebefore6.2V1ArchSummitV2构师峰zxPageStoragein6.2Instance1:50320-FitExact©2022-04-1105:38:06to2022-04-1111:15:39vondiskinmemorywritereada)createJtsnap少/ Xreturn Awriteflow >readflowOB/sWriteflow95.4MIB/s71.5MiB/s47.7MiB/s23.8MiB/s06:00 06:3007:00 07:3008:00 08:3009:00 09:30 10:0010:30 11:00—FileDescriptor■Page■AIOWriteflowmaxavgcurrent▼63.7MiB/s23.0MiB/s42.9M旧/s99.1MiB/s8.5MiB/sInstance1:50320-FitExact©2022-04-1105:38:06to2022-04-1111:15:39vondiskinmemorywritereada)createJtsnap少/ Xreturn Awriteflow >readflowOB/sWriteflow95.4MIB/s71.5MiB/s47.7MiB/s23.8MiB/s06:00 06:3007:00 07:3008:00 08:3009:00 09:30 10:0010:30 11:00—FileDescriptor■Page■AIOWriteflowmaxavgcurrent▼63.7MiB/s23.0MiB/s42.9M旧/s99.1MiB/s8.5MiB/s3.9M旧/sOB/sOB/sOB/sPageBackGround94.7MiB/s4.8MiB/sOB/sAIOOB/sOB/sPageBackGround93.4MiB/sOB/s2:501225.8MiB/sOB/s1:5032097.8MiB/sOB/sFileDescriptor97.5MiB/sOB/smaxcurrent▼V3WriteThroughputByInstance▼maxcurrent▼throughput_ingest-2:50122OB/sOB/sthroughput_ingest-1:50320OB/sOB/sthroughput_write-2:501229.0MiB/sOB/sthroughput_write-1:503207.9MiB/sOB/sArchSummitCPUUsage4000.0%3000.0%2000.0%1000.0%1:2129316:50 17:00 17:10 17:20 17:30 17:40 17:50 18:00 18:10 18:20 18:30 18:40 18:50 19:00 19:10 19:20 19:30 19:402:21293CPUUsage4000.0%3000.0%2000.0%1000.0%1:2129316:50 17:00 17:10 17:20 17:30 17:40 17:50 18:00 18:10 18:20 18:30 18:40 18:50 19:00 19:10 19:20 19:30 19:402:21293PageStoragein6.2InatypicalHTAPworkload,•APQPSimproves30%•PeakCPUusagedecreasesfrom3000%to2500%•PeakMemoryusagedecreasesfrom28GBto18GB•Peakwritethroughputdeceasesover30%DataSharingReadThreadandCache▼30.0%cache_hit_ratio15.0%2022-08-0915:44:30he_no_slot—get_cache_hit■get_cache_total74.1Kops/s\89.0Kops/s\\447ops/s4.25Kops/s442ops/s9.76ops/s50.3Kops/s183Kops/s15:46:0015:42:3015:43:0015:43:3015:44:00—add_cache_stale:—add_cache_succ:—sche_from_cache:—sche_new_task:—sche_no_pool:—sche_no_slot:—get_cache_hit:—aet_cache_total:—cache_hit_ratio: 27.5%.InfoQInfoQInfoQInfoQTPCHperformanceimprovementTPCH性能提升24%25 20 ■ I10 IIXjuilkjdiL^jL^jilLJLQ01Q02Q03Q04Q05Q06Q07Q08Q09Q10Q11Q12Q13Q14Q15Q16Q17Q18Q19Q20Q21Q22v5.4v6.2v5.4ArchSummit全球架构师峰会ArchSummit ArchSummit InfoQArchSummit ArchSummit InfoQTiDB'sfuturearchitecturalevolutiondirectionArchSummit ArchSummit InfoQArchSummit ArchSummit InfoQTiDBCoreArchitectureArchSummit ArchSummit InfoQArchSummit ArchSummit InfoQEvolutiontowardscloudnativeGoal:Bettercostefficienc