2022 数据峰会 数据湖 -深入研究数据湖架构

©PythianServicesInc2022|1

FindingtheHiddenValueinDataLakes

JoeyJablonski

Pythian

VPAnalytics

2022-05-18

©PythianServicesInc2022|2

May18,2022

2:00PM-2:45PMEDT

B203:DrillingDownonDataLakeArchitecture

2:00PM-2:45PM

Newwaystostoredataandleverageitindifferentwaysarebeingutilizedbydata-drivenorganizationshungryforflexibilityandscale.

FindingtheHiddenValueinDataLakes

JoeyJablonski

LongassociatedwithHadoop,inacloudworldthedatalakeisoftenignoredinfavorofitsmorefashionablecousins—mesh,fabric,lakehouse,andwarehouse.Butignorethedatalakeatyourperil,asithasanimportantroletoplayinanymodernanalyticsstrategy.Jablonskifocusesontheevolvingroleofthedatalakewithaparticularemphasison:Whyadatalakeisacriticalcomponentofanycloudanalyticsproject;theroleofthedatalakeinthebattleoverETLvsELT;theimportanceofmetadataindataplatformdesign;andhowadatalakehelpsdeliverbusinessvalue,notjusttechnicalsuccess.

JoeyJablonski

VPAnalytics

Austin,TX

Data&Analytics

Applications

420+

ExpertsacrosseveryDataDomain&Technology

400+

GlobalCustomers

25

YearsinBusiness

CloudOps

©PythianServicesInc2022|3

©PythianServicesInc2022|4

Single

Repository

Enablethevisionofsinglelocationtolocate&analyzeallcorporatedata

Scalability

Ensurethattechnologyisneverthelimitingfactortocorporatestrategy.

WhyweBuildDataLakes

DataLakeValueMeasures

Data

Monetization

SupportbusinessobjectivestocreatenewrevenuestreamsandlowerOPEX.

Data

Quality

Simplifiedsetoftoolstomeasuredataquality,reportandimprove.

ModernCloudDataPlatforms

aretheenablerforinsights

(BI),predictions(ML)and

productactivation

(orchestration)andcreation

(AppDev)acrossALL

datasources.

Modern

Applications/SaaS

startwithmodern,oftencloud-nativedatabases.

Operational

Excellence

Howthedataestateisevolving

BusinessTransformation

DEFENSEOFFENSE

Dataisthedriverofinnovationtransformation

Datapowersthesoftwarethat

drivesthebusiness

Traditional

OnPremise

EnterpriseApps

i.e.Oracle,SAPetc

slowlymoving

toCloud,dragging

datawiththem.

and

TraditionalData

Warehousesare

beingreplacedwith

modernclouddata

platforms.

©PythianServicesInc2022|5

©PythianServicesInc2022|6

WhyDataLakeProjectsFail

1

©PythianServicesInc2022|7

Designyour

dataplatform

tosupportyourdata

strategy.

©PythianServicesInc2022|8

Buildyour

governance

programstoenable

yourdatalake.

2

●DataGovernanceensureswedonotcreateadataswamp

●DataGovernanceensuresriskismanagedwithintheplatform&consumerbase

●Governancecreatesstructureformanagingchange

©PythianServicesInc2022|9

DataPlatformFunctionalComponents

Ensureevery

feature&

investment

hasavocal

business

champion.

AdWords,

DoubleClick,

YouTube

GoogleTransferService

Support

●24/7monitoring

●Backup/restore

●DataServices

●Updates

PowerUsers

Tools:Looker,SQL

SAAS

MA,CRM,Social

Media,etc

Encryption

Enterprise

RDBMS,

File/CSV,

JSON

DataScientists

BigQueryML

Data

Marts

Data

Marts

IoTGateway

API

AppEngine

ML

Exploration

Datalab,

CloudML,

TensorFlow,

Keras

Machine

Learning

Operations

DataEngineer

Tools:Spark,Dataflow,SQL

ActivateviaotherSystems

MarketingAutomationsystems,GoogleAdManager,etc

DataScientists

Tools:Python,TensorFlow,R,Notebooks

3

GoogleBigQuery

Real-time/Streaming

Pub/Sub

DataManagement

Integration,

Deduplication,

Transformation,

Cleaning,Encryption,

Scheduling

GoogleDatafloworDataProc,DataFusion

Infrastructure

unstructured

data)

CurationLayer

DataCaptureStaging

GoogleCloudStorage

(structuredand

DataLakeStorage

GoogleCloudStorage

MarketingAnalyticsPlatformonGoogleCloud

Metadata/API

CloudDatastore,KubernetesEngine

CasualUsers

Reports,Dashboards,Excel

4

©PythianServices

Inc2022|10

Makeitmodular–

Iguaranteeyou’ll

bechangingout

componentswithin

ayear.

Avoidanalysis

paralysiswhen

pickingdatalake

components.

5

●Leveragerapidprototyping&pilotstoverifyfunctionalcapabilitiesoftools

●Acceptthatthelifespanoftoolsisgettingshorterandchangeisinevitable

●Thinkofthesedecisionsastwo-waydoors

©PythianServicesInc2022|11

Imagecredit;

AaronColcord

©PythianServicesInc2022|12

Decoupleprocessstepsfrom

technicalimplementations.

6

●AssumethatdifferentBUswillrequiredifferentdataconsumptionmethods

●Assumetechnologywillevolve

●Decouplebusinessprocessanddatatransformationfromunderlyingtechnologiestobetteracceleratefutureevolution

Acceptthatconsumersofdatawillchange&evolve

overtime

Decouplingtransformationfromconsumptionallows

differentconsumerstochoosedifferenttools

EDWshaveaplacefordataconsumptionandhigh

costsfortransformationworkloads

●

●

●

Matchyour

transformation

patternstoyour

flexibility&

operationalneeds.

7

©PythianServicesInc2022

|13

©PythianServicesInc2022|14

Metadata-storeit,

manageit,govern

it,useit!

8

DescriptiveStructural

Showdata

organizationand

relationships.

Usedfor

identification&

discoveryofdata

sets.

AdministrativeReferenceStatistical

Datadistribution,

outliersandrecord

counts.

Businessprocess

detailsabout

creation,destruction,

andSMEs.

Keyreferral

elementsaboutdata

quality&

consumption.

©PythianServicesInc2022|15

DeployMLOps

capabilityadjacent

todataplatformsto

enableexplorative

&productiondata

scienceworkloads.

9

©PythianServicesInc2022|16

DataQuality=

UserTrust=Data

Adoption.

10

©PythianServicesInc2022|17

ThepromiseofDataLakescanbeachieved.

Clos