虚拟化大数据的大规模应用

Date

|

City,

Country虚拟化⼤大数据的⼤大规模应⽤用

IT

at

theSpeed

of

Business*

Source:

Gartner,

2013:

“Hunting

and

Harvesting

in

a

Digital

World:

The

2013

CIO

Agenda”IT

at

the

speed

of

business

IT

Technology

ErasMainframeClient-ServerMobile-Cloud$$$$$$Steady

ITBudgets*

Business

ExpectationsAbility

of

IT

to

Deliver23Software-Defined

Data

CenterAgility,

Efficiency,

Control

and

Choice

in

the

Mobile-Cloud

EraAll

infrastructure

is

virtualized

and

deliveredas

a

service,

enabling

comprehensive

datacenter

management

through

software

and

extensibility

to

cloud

resources

AgilityControlEfficiency

Choice4Software-Defined

Data

CenterAgility,

Efficiency,

Control

and

Choice

in

the

Mobile-Cloud

EraAll

infrastructure

is

virtualized

and

deliveredas

a

service,

enabling

comprehensive

datacenter

management

through

software

and

extensibility

to

cloud

resources

Agility

EfficiencyControlChoice5Unified

Platform

for

Any

ApplicationScale-up,

Scale-out,

and

Everything

in

BetweenScale-UpScale-Outthan

running

bare-metalvSphere

6:

Great

for

Big

Data630

TB

TeraSort

Benchmark

(32-host

cluster,

4

VMs

per

host)Virtualized

Hadoopup

to

12%

faster11Basedon

VMware

internal

tests,

Jan

2015vSphere

6.0

Tests

–

Hardware

LayoutCONFIDENTIAL7CONFIDENTIAL8Virtualizing

Big

Data:

Value

Proposition

Agility• 

Infrastructure

on

demand• 

Rapid

scaling

of

clusters

Simplified

Management• 

Centralized

data

center

management• 

Automated

BD

workloads

configuration

and

deployment

with

best

practices

Efficiency• 

Resource

pooling• 

Server

and

cluster

consolidation

Performance• 

Up

to

12%

better

performance• 

No

significant

overhead9vSphere

+

Big

Data

ExtensionsAgility

Physical

Timeframe:

Weeks• 

Order

Hardware

(Servers,

Storage,

Etc)• 

Wait

for

them

to

arrive• 

Setup• 

Server

Preparation• 

OS

Installation• 

Disk

Configuration• 

Network

configuration• 

Hadoop

Install

and

Configuration

Virtual

Timeframe:

Minutes• 

Spin

up

a

new

virtual

machine• 

Add

virtual

machine

to

cluster1011

OS

Installation

Network

ConfigurationHadoop

Installation

and

ConfigurationHadoop

Cluster

Deployment

on

VMware

On

physical

machines

Server

PreparationOn

VMware

Big

Data

Extensions

for

VM

creation,

configuration,

start-upBig

Data

Extensions

orother

Hadoop

Management

ToolSimplified

Management12• 

Familiar

vSphere

Graphical

User

Interface• 

Setup

a

flexible

configuration

and

BDE

will

connect

the

dots

behind

the

scenesBDE

Allows

Flexible

Configurations

Storage

configuration

Choice

of

shared

or

localHigh

Availability

optionNumber

of

nodes

andresource

configurationVM

placement

policiesEnable

Hadoop

As

A

Service

On-premvCenter

PluginvCloud

Automation

CenterHadoop

as

a

ServiceEfficiency• 

Better

utilization

of

servers• 

Efficient

and

rapid

scaling

of

clusters

through

VM

cloning• 

Best

practices

for

VM

placement

onto

servers15Hadoop

1Hadoop

2

HBase• 

Consolidated

cluster

has

access

to

entire

pool

of

physical

resources• 

Take

advantage

of

multi-tenancy

to

increase

utilization

during

non-peak

hours• 

Reduce

latency

on

priority

jobs

on

consolidated

clusterIncrease

Utilization

to

Control

CostsVirtualizing

Big

Data:

Value

Proposition

Agility• 

Infrastructure

on

demand• 

Rapid

scaling

of

clusters

Simplified

Management• 

Centralized

data

center

management• 

Automated

BD

workloads

configuration

and

deployment

with

best

practices

Efficiency• 

Resource

pooling• 

Server

and

cluster

consolidation

Performance• 

Up

to

12%

better

performance• 

No

significant

overhead17vSphere

+

Big

Data

ExtensionsvCenter

Operations

ManagerIntegrated

Solution

for

Big

Data

Infrastructure

ManagementvCloud

Automation

Center

ElasticityConfiguration

IsolationAvailability

vSphere

vCenterBig

Data

Extensions

Deployment

Multi-tenancy

ESXi

ClusterHow

BDE

Works• • • BDE

is

a

downloadable

virtual

appliance

integrated

as

a

plug-

in

to

vCenter

server.BDE

requires

a

vSphere

5.0

or

later

license

and

an

Enterprise

or

Enterprise

Plus

license.BDE

clones

VMs

from

the

template

and

controls/configures

VMs

through

vCenterHostHostHostHostHostVirtualization

PlatformHadoop

NodeHadoop

NodevCenterManagement

ServerTemplateVirtual

ApplianceDeployment

Options

with

Big

Data

Extensions20

BDE

OnlyBDE

provisions

VMs

and

installs

the

Hadoop

software

BDE

2.0

BDE

provisions

emptyVMsHadoop

management

tool

installs

software

BDE

2.1

BDE

VMs

and

callsmanagement

tool

APIHadoop

management

tool

installs

software

under

the

hoodSkyscape

Cloud

Services• 

A

UK

company

that

provides

cloud

computing

services

to

the

UK

Government’s

G-Cloud

initiative.• 

Skyscape

offers

IaaS,

PaaS,

SaaS.• 

5

customers

lined

up

at

the

first

day

of

GA.• 

Expected

to

expand

to

140

servers

very

soon.• 

Skyscape

Hadoop

in

the

Cloud

is

built

on

top

of

BDE.• 

Used

our

API

quite

independently.• 

Public

Reference

with

case

studyAdobe

Case

Study• 

Digital

Marketing

business

unit• 

Hadoop

as

a

Service

using

vRA

solution• 

Goal

is

to

bring

Hadoop

on

AWS

in-house• 

Now

in

production• 

Currently

several

BDE

clusters

running

in

Oregon• 

Hundreds

nodes

across

6

data

centers

by

VMworld• 

Performance

meeting

and

exceeding

expectations• 

Public

reference

with

case

studyCONFIDENTIAL2223• Virtualized

• based

onvSphere6.0

and

VMware

Big

Data

Extensions

2.2• 

EMC

Isilon

7.2.0.2

with

two

patches

for

HDFS

Storage• 

Pivotal

Big

Data

Suite

3.0

for

Hadoop

2.6

andHAWQ

1.3

• 

Pivotal

Spring

XD

1.2

for

Data

Ingestion

to

Hadoop• 

Alpine

Data

Lab

5.4

for

running

• • Deeper

Analytic

FunctionsMachine

Learning

Algorithms

• 

HUE

2.6

for

GUI

based

HIVE/PIG

Query

execution

client

VMware

ITThis

Big

Data

Cluster

is

fully

At

VMware

IT,

we

have

established

the

fact

that

an

Enterprise

BigData

Analytics

Platform

can

be

successfully

built

and

run

on

top

of

VMware

Virtual

Infrastructure

with

EMC

Isilon

and

PHD

3.0

-with

great

performance

Take

Away

…Hybrid

storage

model

-

the

best

of

both

worlds

– Master

nodes– NameNode,

ResourceManager,

ZooKeeper

etc.

on

shared

storage

• Leverage

vSphere

vMotion,

HA

and

FT– Worker

nodes

• NodeManager/DataNode

on

local

storage

• Lower

cost,

scalable

bandwidth

• Temp

data

is

written

to

local

storage

for

best

performance

• Scale-out

NAS

storage

for

HDFS

data

is

a

good

alternative

to

local

Shared

Storage

Local

StorageNAS

(Isilon)

for

HDFSCombined

Model

–

Two

Virtual

Machines

on

a

HostVirtualization

HostHadoopVirtualNode

1DataNodeNodeManager

Ext4VMDKShared

storageSAN/NASLocal

disks

Ext4

OS

Image

–

VMDKOS

Image

–

VMDKExt4VMDKExt4

VMDKVMDKHadoopVirtualNode

2DataNode

Ext4VMDKNodeManager

Ext4VMDKExt4VMDKExt4

VMDK

Virtualiza*on

Host

VMDK

Hadoop

Virtual

Node

1

Ext4

Separated

Model

with

VMDK

Isolation

NodeManager

Shared

storage

SAN/NAS

OS

Image

–

VMDK

OS

Image

–

VMDK

Hadoop

Virtual

Node

2

Datanode

VMDK

VMDK

VMDK

Ext4

Ext4

Ext4

VMDK

VMDK

VMDK

Ext4

Ext4

Ext4

Local

disks

for

Temp

Data

JBOD

Local

disks

for

HDFS

Data

JBOD

Tenant

1

Tenant

2Compute

StorageCombined

Compute

andStorage

StorageVMVMVMVMVMVMUnmodified

Hadoop

node

in

a

VM

•