On-demand Grid Storage Using Scavenging

1、On-demand Grid Storage Using ScavengingSudharshan VazhkudaiNetwork and Cluster Computing, CSMDOak Ridge National Laboratory/Acknowledgments: ORNLCollaborators: Dr. Xiaosong Ma and Dr. Vincent Freeh (NCSU)OutlinelGrid Storage Fabric BackgroundlThe Evol

2、ving Computing LandscapeAn AnalogylStorage Scavenging of User Desktop WorkstationslUse Cases lRelated Work and Design ChoiceslArchitectureStorage LayerManagement LayerCurrent StatusGrid Storage Fabric BackgroundlScientific discoveries driven by analyses of massively distributed, bulk data.Proliferat

3、ion of high-end mass storage systems, SANs and datacentersProviders such as IBM, HP, Panasas, etc.lMerits:Excellent price/performance ratioGood storage speeds and access controlSupport intelligent parallel file systemsOptimized for wide-area, bulk transfersReliability!Successful demonstration in pro

4、duction Grids: DOE Science Grid, Earth System Grid, TeraGrid, etc.lDrawbacks:Increasing deployment/maintenance/administrative costsSpecialized software and central points of failureCosts and specialized features prohibit wider acceptability and limits to select few research labs & organizationsA

5、forementioned production Grids are hardly half-a-dozen sites!Meta-Message: If grids are to become prevalent and grow beyond the confines of a few organizations, exploiting commodity fabric features is absolutely essential! The Evolving HPC LandscapelComputing fabric for the Grid: Storage Fabric?Meta

6、-Message: Proprietary systems are being replaced with commodity clusters, delivering new levels of performance and availability at dramatically affordable price point. Tightly CoupledLoosely CoupledTime fliesSupercomputersBeowulf StyleAggregating idle CPU cycles from Commodity PCsLoosely CoupledTigh

7、tly CoupledTime flies againDatacentersRAID-like aggregationAggregating idle storage space from Commodity PCsVolatilityTrustPerformanceStorage Scavenging of User Desktop WorkstationslHarnessing collective storage potential of individual workstations Harnessing idle CPU cycleslWhy Storage Scavenging c

8、an be viable?Economics of buying gigabytes of storage is increasingly affordableSpace usage to Available storage ratio is significantly lowIncreasing numbers of workstations are online most of the timeEven a modest contribution (Contribution Available) can amass collective, staggering aggregate stor

9、age!lConcerns:Vagaries of volatilityQuestion of Trust: datasets on arbitrary user workstationsPerformance of such aggregate storage Meta-Message: Despite the high maintenance and administrative costs, a factor that attracts the Grid community to high-end storage and data centers is their ability to

10、deliver sustained high-throughput for data operations. Use CaseslStorage cloud as a:CacheIntermediate hopLocal, client-side scratchGrid replicaRAS for Terascale SupercomputersRelated Work and Design ChoiceslRelated Work:Network/Distributed File Systems (NFS, LOCUS)Parallel File Systems (PVFS, XFS)Se

11、rverless File Systems (FARSITE, xFS, GFS)Peer-to-Peer Storage (OceanStore, PAST, CFS)Grid Storage Services (LegionFS, SRB, IBP, SRM, GASS)lDesign Choices & Assumptions:Scalability: O(100) or O(1000)Commodity Components: Quality & QuantityUser AutonomyWell connected & SecureHeterogeneityL

12、arge, “write once read many” datasetsTransparentGrid AwareArchitecturePool nMorsel Access, Data Integrity, NonInvasivenessManagement LayerData Placement, Replication, Grid Awareness, Metadata ManagementPool ARegistrationStorage LayerPool mRegistrationGrid Data Access ToolsMeta-Message: Imagine “Cond

13、or” for Storage.Storage LayerlBenefactors:Morsels as a unit of contributionBasic morsel operations as RPC services new(), free(), get(), put()Space Reclaim:lUser withdrawallWhich morsels to relocate/evict?lWhich benefactor workstations to relocate to?Data Integrity through checksumsPerformance Trace

14、slPools:Benefactor registrations (soft state)Dataset distributionsMetadataSelection heuristicsFile 1:1 23File n:1a 2a 3a 4a2a1a214a3a232a1a3a1Management LayerlManager:Pool registrationsMetadata: datasets-to-pools; pools-to-benefactors, etc.Availability:lRedundant Array of Replicated MorselslMinimum

15、replication factor for morselslWhere to replicate?lWhich morsel replica to choose from in response to user file fetches?Grid Awareness:lInformation ProviderslSpace reservationslTransfer protocol agnosticTransparent Access:lNamespace Current StatusApplicationProxyManagerBenefactorOSBenefactorOSftp/Gr

16、idFTPrpc (A)rpc (C)rpc (B)rpc (D)reserve(): cancel() store() : open(); benefactorID.put() retrieve(): open(); benefactorID.get() delete()new() free() get() put()lrpc (A) services:Create/delete filesReserve.lrpc (B) services:File fetchesHintslrpc (C) services:ControlDataset distributionsBenefactor alerts, warnings, alarms to managerlrpc (D) services:Morsel relocationsStatus infoLoad balancinglrpc (E) services:Morsel relocations to different poolsUnder direction of managerPhilosophical Musings. lIts all about commod