Cloud Native分布式数据库技术介绍

1、Cloud Native分布式数据库技术介绍技术创新，变革未来Calvin Sun2006.012016.022013.032017.102008.02AgendaCloud Computing in FintechCloud Native DatabaseHuawei Cloud Native Distributed Database SystemCloud Computing in Fintech金融行业云计算技术百花齐放春满园，ArkDB俏枝头MySQL金融级解决方案的百家争鸣时代周彦伟 极数云舟 CEO,05/11 15:30 16:10演讲简介：随着MySQL的广泛应用和深入人心，对

2、数据库技术要求最高 的金融领域也慢慢开始转型到MySQL，随之而来的就是业内 各种各样的MySQL金融级解决方案百花齐放，百家争鸣。本 次分享首先是对业内的一些做法做了调研，同时也详细讲述 了完全基于开源引擎Galera Cluster的MySQL金融级解决方案 的实现方式ArkDB，它最大的特点是:基于开源引擎Galera、 可自主控制、高一致性、高安全性、久经考验，在完全满足 金融系统要求的同时，也大大增加了系统自身的可控性，是 金融系统的又一个重要选择。金融行业云计算技术华为与金融行业云技术招商银行和华为成立分布式数据库联合创新实验室，实践云上科技变革【中国，深圳，2017年12月4日】

3、2017年11月27日，华为与招商银行股 份有限公司在招商银行研发中心举行了分布式数据库联合创新实验室 揭牌仪式。双方将共同应对“CloudFirst”的挑战，利用云、大数据、人 工智能先进技术，领先的金融业务实践和优秀资源，联接业务与技术， 联合进行分布式数据库技术的研发和产品应用，解决数据库应用上云 的问题。华为云助力招商银行加速数字化转型，成为“金融科技银行”，通过科技变革，为客户提供普惠、个性化、智能化的金融服务。05/11, 14:50-15:30Cloud Native DatabaseTraditional RDBMS with HABasic architecture is 3

4、0+ years oldDesigned for hardware at the time (single processor, small memory, slow hard disks)Master plus two secondaries for reliability (HA)Master handles all updates andships log to each secondaryEach secondary updates its owncopy of the databaseIf master fails, a secondary becomes the new maste

5、rSecondaries may handle read-only queriesLog shipping is only network loadPage reads (random)Master(read-write)Secondary(read-only)Secondary(read-only)DBMSRedo log writes (seq.)Page writes (random)Log shippingLocal diskLocaldiskLocal diskR/WR onlyR onlyQueryoptimizerQueryexecutionLock managerBuffer

6、poolDatabase systemRedo/undo logPage storeLocal disksTrends in Cloud DatabasesSeparation of compute and storageGartner: By 2019, 90% of cloud DBMS architectures will support separation of compute and storage, rendering those that do not as irrelevant in the overall market.How to leverage latest hard

7、ware advances:CPU: Multi-cores with NUMAStorage: Optane SSDs (Coldstream & AEP)Network: RDMASpecial hardware: GPU, FPGAAdvances in AI and MLRequirements of Cloud Native DatabaseHigh-PerformanceScalabilityEasy to MaintainHighReliabilityBig DataCloud Native DatabaseAWS Aurora HighlightsSeparation of s

8、torage: allows compute and storage to be scaled separatelyCompute layer is responsible for maintaining ACI(no D).Storage layer ensures Durability.Storage service receives redo logs; coalesce; and periodically apply the redologs to data pagesNear-instantaneous crash recoveryReplicate data locally (wi

9、thin the storage node)Leverage redo log across the distributed storage systemSignificantly reduces networktraffics.Efficient append-only logs.Improved lock management, query cache, etc.PolarDB HighlightsSeparation of compute and storageRedo-based physical replicationRDMA for fast redo transferRDMA b

10、ased optimization for parallel-raftUser-mode file system: optimized for redo and page IO3D XP + NVMeMany SQL node optimizations (MySQL 5.6 based)Huawei Cloud Native Database SystemHuawei Cloud Native Database Design Principles (1)Decoupling:Separate compute from storageDecouple master from replicaPu

11、shdown operations close to dataOffloading work to storage nodes, such as redo processing, pageconstructionExploit functionality provided by cloud storageBuilding storage as an independent fault-tolerant and self-healingserviceShared access (single writer, multiple readers)Exploit properties of SSDsA

12、void random writes to SSDs to minimize wearExploit good random read performance of SSDsHuawei Cloud Native Database Design Principles (2)Multi-tenant supportBelieve the central constraint in high throughput data processing has moved from compute and storage to the network.Reduce network trafficsTake

13、 advantages of new network technologies, e.g. RDMALeverage advances in AI and ML for autonomous systemAuto-scaling, self-tuningHigh-level ArchitectureSeparation of compute and storageEasy scale-out and load balancingStorage abstraction layer (SAL) isolates SQL front-end, transaction andquery executi

14、on from the way storage is organizedFoundation for additional database supportStorage layer provides fast, reliable, shared storageUse storage in a log structured manner; only sequential writes to minimize SSD wearStorageLayerSQLLayerSQL NodeSQL NodeOverview of Huawei Cloud Native DatabaseStorageNod

15、esSQL NodesMaster SQL node (read-write)Replica SQL node (read-only)Replica SQL node (read-only)CoherencetrafficCoherencetrafficRDMASliceSliceServSererverSAL ClientSAL ClientSAL ClientStorage Abstraction Layer (SAL)Storage NetworkSliceShared, reliable, scale-out storageRedo logPagesMaster database se

16、rverHandles all updatesWrites to the WAL logsRead Replica database serversCan handle read-only requestsEnable fast failoverCan be added at any timeDatabase data is partitioned across storage nodesPages are logically organized based on slice and distributed among slice serversEach slice is duplicated

17、 for reliabilityLog records for a page are sent to the corresponding sliceSlice ServerMaintain multiple slices fordifferent tenant databasesStore and process log recordsMaintain and construct pagesServe page read requestsSliceSliceSliceSlice SliceServerSliceSerSvliecerSliceSliceSlice ServerSQL Nodes

18、Managing client connections, parsing SQL requests, planning and executing queries and managing transaction isolationOne RW with multiple RO replicaLoose coupling between master and replicaLight traffic between master and replicaEnable fast failoverBuild on top of HWSQL with additionalenhancementsQue

19、ry result cacheQuery plan cacheOnline DDL05/11, 15:10-16:00Page readsMaster(read-write)Replica(read-only)Replica(read-only)Redo log writesPage writesCoherence trafficShared StorageR/WR onlyR onlySQL ReplicaReplica maintains multiple versions of page in its buffer poolReduce frequent page read from s

20、torage.Interactions with MasterFor MVCC: read replicas will receive the list of currently active transactions, (readview“), from the master periodically.For page invalidation: as updates happen on the master, read replicas also receive the list of pages updated during transaction commits.For purge:

21、read replicas need to feed their min LSNs to the master. The master conducts the purge operation based on info from all read replicas.For DDL operations: master coordinates with read replicas (via MDL)when performing DDL operations.Storage Abstraction Layer (SAL)Storage abstraction layer (SAL) is a

22、logical layerIsolates SQL front-end, transaction and query execution from the way storage is organizedConsists of a common log module executing on SQL nodes and slice stores executing on storage nodes.SAL operates on database pages and supports access to multiple versions of the same page.SAL divide

23、s all data pages into slices, based on spaceID, pageID.Scaling out as the database grows in size, resources (storage, memory) availableto grow proportionally as more slices are created.Data locality data intensive operations executed by slice servers running on thestorage nodes that store the data.Storage ServerSQL ServerPerformance EnhancementsContainer with Hi1882 ChipLower network latencyHigher network P