2022 数据峰会 数据湖 -现代数据湖要点

BuildingTheOpen

DataLakehouse

MarkLyons

Confidential-DoNotShareorDistribute

Mark

Product@Dremio

ProductManagement@

Vertica

ProductManager@

EnerNOC

Confidential-DoNotShareorDistribute

2

Agenda

●BriefHistoryofdataplatforms

●OpenDataArchitecture

●Tableformats

Scale&Performance

Askedtodotheimpossible

WhatYouWant

toAccomplish

DataDemocratization

SupportNewInitiatives

andProjects

FastTimetoValue

Security&Governance

ButIt’sGettingHarderbytheDay

DataIsRapidly

Increasing

AccessRequestsAreRapidlyIncreasing

YourBudgetIsFlat

DataTalentIsScarce

Whydatalakes&cloud?

Confidential-DoNotShareorDistribute

Past&Present

●Schemaonread

●Flexiblew/datafiles/types

●Costeffectivestorage

●Expensivetomaintain/upgrade

●Specializedtalent

●Schemaonwrite

●JSONdatatype

●Consumptionpricing

●Separatestorage&compute

●Managed

Confidential-DoNotShareorDistribute

OpenData

Architecture

Evolution

Confidential-DoNotShareorDistribute

OpenCompute

SQL:

Dremio,Presto,etc.

Streaming:

Databricks,Flink

…

OpenData

Metastore:HiveMetastore,AWSGlue

Storage

8

DataLake1.0

Spark:Databricks,EMR,etc.

FileFormat:ApacheParquet

S3,ADLS,GCS

●Thedatatierofthedatalakewasverybasic,requiringsignificantengineeringworkfromcustomers

●Analytics/Accessperformancesuffered

●Maintenanceandengineeringcostoverwhelmed

FileFormatsParquet,ORC,csv,json

S3

HDFS

ADLS

DataLakeStorage

OpenTableFormatsIceberg,DeltaLake,Hudi

MakingtheLakehouse

Users&Applications

Interfaces

ArrowF

light,ODB

C/JDBC

DataLakeEngines

Dremio

Spark

Hive

Athena

Confidential-DoNotShareorDistribute

ModernMetastoreNessie

New

OpenTableFormatsIcebergDeltaLake,Hudi

FileFormatsParquet,ORC,csv,json

S3

HDFS

ADLS

DataLakeStorage

,

GoingfurtherthantheEDW

Users&Applications

Interfaces

ArrowF

light,ODB

C/JDBC

DataLakeEngines

Dremio

Spark

Hive

Athena

Confidential-DoNotShareorDistribute

TransactionalTablesontheDataLake

Record-leveldatamutationswithSQLDML

INSERTINTOt1...

UPDATEt1SETcol1=...

DELETEFROMt1WHEREstate='CA'

Automaticpartitioning

CREATETABLEt1PARTITIONEDBY(month(date),

bucket[1000](user))

Instantschemaandpartitionevolution

ALTERTABLEt1ADD/DROP/MODIFY/RENAMECOLUMNc1...

ALTERTABLEt1ADD/DROPPARTITIONFIELD...

Timetravel

SELECT*FROMt1AT/BEFORE<timestamp>

●CreatedbyNetflix,Appleandotherbigtech

●INSERT/UPDATE/DELETEwithanyengine

●StrongmomentuminOSScommunity

●CreatedbyDatabricks

●INSERT/UPDATEwithSpark,SELECTwithanyengine

●PrimarilyusedinconjunctionwithDatabricks

Confidential-DoNotShareorDistribute

11

USEBRANCH'main'

SELECT*FROMt1//mainimplicit

SELECT*FROMt1@etl

SELECT*FROMt1AT'2020-10-26'

SELECT*FROMt1@etl'2020-10-26'

Nessie:Amodernmetastore

DataBranching

Transactionsonsteroids

–Multi-tableconsistency/transactions

–Experimentation(isolateddev/test)

–Pre-prodverification(stage→prod)

CREATEBRANCHetl

[Kafka]DataIngestion

[Spark]Transformation1

[Spark]Transformation2

[Dremio]ReflectionRefresh1

[Dremio]ReflectionRefresh2

USEBRANCHmain

MERGEBRANCHetl

DataVersionControl

Timetravelonsteroids

–Reproducibility

–Compliance

–Historicalcomparisons

USEBRANCH'main'

SELECT*FROMt1//mainimplicit

SELECT*FROMt1@etl

SELECT*FROMt1AT'2020-10-26'

SELECT*FROMt1@etlAT'2020-10-26'

Confidential-DoNotShareorDistribute

12

vs.

Real-WorldTransactions

Multiplesessions

MultipleusersMultipleengines/services

Builtfordataengineers

DataWarehouseTransactions

Onesession

Oneuser

SQL-only

Builtforapplicationdevelopers

CREATEBRANCHetl

[Kafka]Ingestdatasource1

[RDBMS]Ingestdatasource2

[Spark]Transformationstep1

[Dremio]Transformationstep2

[Spark]Insertintotable

[Dremio]Buildreflections

[Spark]Verifydata

USEBRANCHmain

MERGEBRANCHetl

ff

BEGINTRANSACTIONetl;

INSERTINTOt1...

UPDATEt1...

UPDATEt1...

COMMIT;

13

--ifchecksdon’tpass:

--don’tmerge,alert

Safe,real-worldtransactions

CREATEBRANCHetl_1897914;

USEBRANCHetl_1897914;

MERGEINTOordersoUSINGorders_stageo_s

ONo.order_id=o_s.order_id

WHENMATCHEDTHEN

UPDATE...

WHENNOTMATCHEDTHEN

INSERT*

MERGEINTOlineitemliUSINGli_stageli_s

ONli.order_id=li_s.order_id

ANDli.line_number=li_s.line_number

WHENMATCHEDTHEN

UPDATE...

WHENNOTMATCHEDTHEN

INSERT*

SELECTSUM(CASEWHENorder_amount<0THEN1ELSE0END)FROMorders;

SELECTSUM(CASEWHENquantity<0THEN1ELSE0END)FROMlineitem;

--additionaldataqualitychecksbynon-SQLtools

--ifcheckspass:

REFRESHREFLECTIONsales_dashboard_main;

--final

quality

checks

USEBRANCHproduction;

MERGEBRANCHetl_1897914;

Moreontable

formats

Confidential-DoNotShareorDistribute

16

Hivetableformat

Thede-factostandard

Atableisanlsof1ormoredirectories

Pros

●Single,centralanswerto“whatdataisinthistable”forthewholeecosystem

●Workswithbasicallyeveryenginesinceit’sbeenthede-factostandard

Cons

●Ifeverythinginadirectoryisatable’scontents,howdoIupdateanddeletedata?

●IfIneedtoaddmultiplefilesasasingleoperation,howdoImakesureaconsumerdoesn’tseeonlysomeofmyadditions?

●Allofthedirectorylistingsneededforlargetablestaketoolongfrom<startlisting>to<endlisting>

○Planningtime

○Consistencyproblemscanoccur-multiplecallstolistcontentsofthesetofpartitions

‘sgoals

●ACIDtransactionsonS3

●Tableevolution

●Tablecorrectness/consistency

●Fasterqueryplanningandexecution

●Decouplepartitioningfromlayout

●Accomplishalloftheseatscale

Howcanweresolvetheseissues?

→Weneedanewtableformat

Oldway

dir

F

F

F

Atableisan'ls'ofadirectory

New

way

Manifest(listoffiles)

F

F

F

Atableisacanonicallistoffiles

18

High-levelcomparisontoDatabricksDeltaLake

SIMILARITIES

DIFFERENCES

●ExtensiveintegrationinSpark

●Veryscalable

●Metadatastoredalongsidedatainthedatalake

●Immediatedatafreshness

●Fastplanning,datapruningviastats

Tableformat

Hierarchicalwithpointer-files;snapshotcreationonwrite

Non-hierarchical;sequentiallylogeachchangeonwrite,inlinesnapshotcreationevery10changes

Whocanreadandwrite

Anyonecanread,anyonecanwrite

Anyonecanread,Databrickschooseswhocanwrite(SparkandprobablySQLanalytics)

Writebehavior

V1:copyonwrite

V2:copyonwrite+mergeonread

Copyonwrite

Fileformats

Parquet,ORC,andAvro,andisintentionallyextensible

Parquetonly

OSS&governance

FullyOS,ApacheGovernance

MostlyOS,butkeycapab