小米融合云架构

1、小米融合云架构技术创新，变革未来融合云介绍业务快速发展管理成本接入复杂隔离需求生态链需求公网访问融合云动机小米内部业务生态链业务融合云介绍服务化统身份认证权限管理多租户Web console基于开源功能扩展基于HBase做二级索引系统扩展基于HBase和HDFS做对象存储公网调用支持Http请求/无状态融合云设计融合云介绍融合云架构Web ConsoleSDK团队管理结构 化存 储对象 存储消息 队列流式 消息离线深度分析学习自动 部署Docker镜像HBase/HDFS/Kudu/TensorFlow/Spark/Docker团队管理Organization(Org)资源隶属Team用户身份

2、隶属于OrgAdmin/MemberOrg admin Team认证CAS签名认证生成秘钥秘钥轮换权限管理团队管理(Cloud-Manager)Cloud-ManagerServiceWeb consoleSDKCAS公司账号user: teamList resource: org签名签名认证Org(团队): notebookAdmin TeamDev_TeamTablesBucket/ ObjectsqueuesReport TeamAdminAdminR团队管理示例cc融合云存储/计算数据流结构化存储(SDS)离线分析消息队列(EMQ)对象存储(FDS)消息队列(EMQ)Snapshot

3、Export流式消息服务SDS StreamMR/ Spark结构化存储(SDS)分布式数据库基于HBase数据模型扩展支持EntityGroup/Primary Key支持数据类型多语言SDK支持ACL扩展功能扩展局部二级索引全局二级索引Stream实时恢复Structure Data Store(SDS)结构化存储(SDS)SDS架构ACLQuotaHBaseSDS CoreSDSServiceObject MappingFrontEndMulti-platform SDKThrift ProcessorHttpServer (TTransport)Thrift ProcessorHttp

4、Client (TTransport)flush/HttpRequest read/HttpResponseSDS Service认证与Cloud-Manager交互元信息管理UserTable = TableId = HBase TableSDKJava/php/python/go/c+/node.jsSDKService结构化存储(SDS)SDS CoreSDS Request HBase RequestObject MappingJava Object SDS RequestACL表级别权限控制EntityGroup间权限隔离QuotaTokenBucket算法Soft Limit引入集

5、群过载quota集群过载前允许超发SDS架构ACLQuotaHBaseSDS CoreSDSServiceObject MappingFrontEndMulti-platform SDK结构化存储(SDS)实现Entity Group(EG)功能业务特性: 以用户为中心定义Entity Group KeysEG在同个regionEG内跨行写原子性EG内局部二级索引强致Schema管理EntityGroup/Primary Index/Secondary Index/Column数据类型Bool/Int8/Int16/Float/Double/String/Binary, Set使用Ordere

6、dBinary编码确保顺序条件查询SDS Core主要功能结构化存储(SDS)局部二级索引数据Family/索引FamilyHBase coprocessor中更新索引读取主记录构造索引rowkey更新主记录/删除旧索引/写入新索引支持Eager/Lazy/Immutable IndexLazy invalid index定期清理Eager/Immutable Index支持属性投影全局二级索引使用单独的HBase table存储索引使用Themis提供HBase跨表更新原子性/xiaomi/themis全局单调递增时间戳: /xiaomi/chronos强致性二级索引EGRow01main_

7、row01Index_rowData FamilyIndex Family结构化存储(SDS)使用Java annotation和反射实现toRecord: Object = MapfromRecord: Map = ObjectObject Mapping对象映射SDS Object Mapping例Table Schema定义(SQL)SDS Object Mapping例Table Schema定义SDS Object Mapping例查询SDS Object Mapping例查询结构化存储(SDS)场景OLAP增量数据处理原理Stream Family存储image使用hbase co

8、processor build images 推送image到流式消息服务实时恢复StreamF1F2MRSnapshoterDataImagesnapshotsRecoverySnapshot just before Ts流式消息服务TsWeb console例结构化存储(SDS)SDS rest server 24 Intel(R) Xeon(R) E5620 2.4 GHz, 64G, 万兆网HBase region server Cpu/内存/网络同上，8 * 800G ssd，5台YCSB client性能测试Put Througput/Latency记录大小ThrougputAvg

9、(ms)P95(ms)P99(ms)0.1k337380.52790.5k300470.466111k256520.46610Get Througput/Latency记录大小ThrougputAvg(ms)P95(ms)P99(ms)0.1k302130.26470.5k274570.28391k269320.29312对象存储(FDS)KB到TB级别的文件基于HBase & HDFSBucket & ObjectRestful API多语言SDK权限管理事件通知Quota管理30+业务使用File Data Store(FDS)FrontEndMeta ManagerBlob StoreH

10、BaseHDFSCDN对象存储(FDS)Bucket TableBucket基本信息/ACL/QuotaObject TableObject基本信息/ACL/meta/通知信息Object name = blobUriBlobInfo TableBlobUri = FileId抽象Blob write/read/get接口，隐藏具体实现FileInfo TableFileId = Path, basicInfoMultiPart Upload TableuploadId = partInfo元数据管理对象存储(FDS)BlobManager/FileManagerBlob存储magic/dat

11、a/md5offset管理打开文件MaxOpenNum/MaxFileSizeIdle Files Busy FilesBlobClient创建Blob: BlobWriterWrapped OutputStream关闭Blob获取Blob: BlobReaderWrapped InputStreamBlobStoreBlobManager BlobInfo(BlobId-FileId)FileManager FileInfo(FileId-FilePath)Busy FilesIdle FilesHDFS FilesBlobClientcreateBlob/ getBlob closeBlo

12、b对象存储(FDS)创建对象BlobClient: createBlob将object信息写入hbase object table读取对象从hbase object table获取blobUriBlobClient: getBlob对象列表t=苗hbase object table删除对象删除hbase object table中对应记录对象橾作对象存储(FDS)Object Cleaner 民据object table删除blob tableFile Cleaner & Compactor基于BlobInfo Table和FileInfo Table民据BlobInfo Table统计Fil

13、e使用率归并低使用率文件Archive无Blob引用文件删除过期Archive文件Cleaner & CompactorobjNameblobUriblobUriFileIdFileIdFileInfoFDS性能消息队列(EMQ)应用间异步通信支持P2P, Pub/Sub两种模式基于HBase提供分布式能力At least once语义多读者消息优先级死信队列Short/Long PoolingElastic Message Queue(EMQ)EMQsend messagereceive messageack message消息队列(EMQ)QueueHBase TablePartition

14、sPartitionIdPartition分布致性hashEMQ架构Partition_0Partition_nRegion_0Region_nQueueHBase TableHBasePartition0Partition1Partition2Partition3Partition4Partition5EMQ Server消息队列(EMQ)Message MessageIdmessageId = hbaseRow = Timestamp = currentTime + delayTime Message BodyMessage状态消息状态NewInvisibleReadDeadDeleteRECVTIMEOUTRECVACKRECV LIMITCHANGE VISIBILITYSEND消息队列(EMQ)内存信息 Available set/Received set/Delete set 多读者Partition内存状态Available Set(Sort by MessageId)Received Set(Sort byT