2024从统⼀数据治理到下⼀代企业级 Data Agent 的实践与创新

统⼀数据治理到下⼀代企业级DataAgent的实践与创新目

AIAI201833ZB，增长到2025175ZB 数据AIAI 数据治理AIAI GarbageGarbagein,garbagehttps://\hwww.llamaindex.ai/blog/introducing-llamacloud-and-llamaparse-

GenAI 企业由单云单域架构向多云多域架构迈 数据合规的限

Business

DataEngineer Kafka+AWS-KafkaKafka+AWS-Kafka+Kafka+

AIBusiness

新业务和LLM MattersMost.Automateend-to-enddatarightsrequestsandreporting.Enrichtechnicalmetadatawithbusinessandoperationalmetadataforfullvisibility.

Technical&BusinessLegalHoldData3rdPartyDataPII&PIDataIP

Automaticallyfind,classify,andmapallofyourdata-everywhere.Automaticallyclassifymoretypesofdatainmoreplaces.Simplifyandautomatedatalifecyclemanagementfromcollectiontodestruction. Unified✔✔✔✔✔Unified✔✔✔✔✔✔✔Data DWAINext-GenDataCatalogistheCoreinNewOpenData

FunctionalitylayerInterface

Unified UnifiedUnifiedREST IcebergRESTCorewith

GravitinoVirtualArrowUnifiedNon-tabularSchema:GravitinoVirtualArrowUnifiedNon-tabularSchema:name:stringcomment:stringproperties:map<string,Fileset:name:stringstorageLocation:stringtype:TypeUnifiedTabularSchema:name:stringcomment:stringproperties:map<string,Table:name:stringcolumns:Colum[]partitioning:Transform[]distribution:DistributionsortOrder:SortOrder[]indexes:Index[] UnifiedAccessControlInternetofDataInternetofDataDataS3WriteInternetofDataScientistAIEngineerModelReadData

DataScientistAIEngineerData

Model

S3

Write

AccessControl

Metrics/ModelData

统一数据查找和访问；统一安全模型和数据访 RAGMulti-modularMulti-modular由决策引擎选择RAGAdvancedBasic 什么是Basic较为基本的数据准备（KB等较为简单的检索（基于语义的检索/ TextToSQL（或其它查询语法改写问题（通过Vector

内容总结（通过

Re-

KBPre-DecisionPre-DecisionHybridLLM NL2SQL/QL 更好的方案：统一结构化/非结构化数据接入的Demo：/apache/gravitino- 海量数据“资产化”DataAI ExternalTTL

AI PBHDFSTTL、 业务案例②：基于统一元数据的

LLMOps某业务部门的LLM微调工作流LLMOpsby

跨源数据Schema维护成本高 提供定制化接口：dropPartitiosByFilter//loadFilesBiliKafkaCatalog/BiliDatabusCatalog...... HiveMetaStore

FilesetOneMetaFilesetTTL