基于RocketMQ Connect构建全新数据流转处理平台

1、基于RocketMQ Connect构建全新数据 流转处理平台技术创新，变革未来OpenMessaging Connect OverviewOpenMessaging Connect ArchitectureOpenMessaging Connect in practiceOpenMessaging Connect Now and Future目录ConceptRocketMQ Connect ：A simple understanding of RocketMQ Connect is the middleware that uses RocketMQ to copy data from a

2、source data storage system to a target data store.Mainly composed of source connector, sink connector, runtime.Source Connector：Responsible foringesting the data from source data and sending it to RocketMQSink Connector： Responsible for consuming messages from RocketMQ and writing the data to target

3、 storageRuntime： The runtime is the source, sink connector runtime environment, responsible for loading the Connector, providing the RESTfulinterface, starting the connector task, service discovery between cluster nodes, configuration synchronization, consumption schedule saving, failover, load bala

4、ncing, and so on.File Connect DemoRocketMQ Connect OverviewOLAPProducer ServiceSourcing DataConsumer ServiceTraditional stream data process modelProducer ServiceConsumer ServiceSink DataOpenMessaging Connect OverviewChallengeHow to copies data between so many different sourcesEfficiencyHigh Availabi

5、lity, failoverScalabilityOpenMessaging Connect OverviewOLAPOpenMessaging Connect OverviewOLAPOpenMessaging Connect OverviewBenefits of Messaging ConnectFocus on data copyingScales to applicationAccessible connector APIStreaming and batchEasy cluster managementOpenMessaging Connect OverviewWhats Mess

6、aging ConnectMessaging Connect is a distributed, scalable, fault-tolerant service designed to reliably stream data between messaging middleware andother data system.OpenMessaging Connect OverviewOLAPWhats OpenMessagingOpenMessaging is a standard for messaging middleware.OpenMessaging Connect Overvie

7、wRelationship of OpenMessaging Connect and RocketMQOpenMessaging Connect is a connect standard for messaging middleware.Runtime based on RocketMQ provide the ability of steaming data between other data system with Apache RocketMQOpenMessaging Connect ArchitectureConnect ModelWorker ModelConnect Runt

8、imeOpenMessaging Connect ArchitectureConnect ModelConnector: defines how developers create connector plugins which import and export data from another data systems.Two key components: connector, taskOpenMessaging Connect ArchitectureConnect ModelOpenMessaging Connect ArchitectureWorker ModelOpenMess

9、aging Connect ArchitectureConnect Runtime12345Config Management12345OpenMessaging Connect ArchitectureRebalance and Failover12345OpenMessaging Connect ArchitectureREST APIPOST /connectors/connector nameGET /connectors/connector name/configGET /connectors/connector name/statusPOST /connectors/connector name/stopOpenMessaging Connect ArchitectureOpenMessaging connect in PracticeMySQL connectorOpenMessaging connect in PracticeMySQL connectorOpenMessaging connect in PracticeMySQL connectorOpenMessaging connect in PracticeMySQL connectorOpenMessaging Connect Now a