IceCube Project Data Storage Requirements,2008 to 2013 IceCube 项目数据存储要求2008 年至 2013 年

IceCubeProjectDataStorageRequirements,2008to2013

Introduction

SincetheoriginaldesignestimatesfortheIceCubeproject,whicharecapturedinthe“PreliminaryDesignDocument”,therequirementsfordatastoragehaveevolveddramatically.Forinstance,thepresentvolumeofsimulationdata,tosimulateIC22andpriorconfigurations,heldondiskislargerthanthetotalestimatefor15yearsofexperimentalandsimulationdatainthePDD.

Thisincreaseindatavolumescanbetracedtonumeroussources,howeveritisdifficulttoquantifytheeffectofeachintermsoftheircontributiontotheresultingtotalincrease.MajorincreasescanbeattributedtotheDAQ,withtheaverageeventsizebeingsignificantlyhigherthanoriginallyestimated.Thisisthenimmediatelycompoundedbythechangeinemphasistowardsalowtriggerthreshold.

Itwasalsoexpectedthatthefilteringsystemwouldbemoreadvancedthanwhatitispresently.Whilethefilteringsystemiscertainlyreachingahighlevelofmaturity,itisstillatthepointwhereithasbeennecessarytore-filterlargevolumes(manytensofTB)ofrawdata,andlargevolumesofunfilteredsimulationdataisbeingheldonline.Itmayhavebeenthattheexpectationsofbeingabletoputthissoftwarequicklyintoproductionweretoohigh.

During2007itbecameobviousthatthestoragesystemwouldnotbeabletomeettheongoingneedsoftheexperiment.Manymeasureswereattemptedtominimizetheimpactofgrowingrequirements,suchasrunningatveryhighutilizationrates,andstretchinginfrastructuretoit’slimits.Thishaspredictablyresultedinincreasedfragilityandassociateddowntime.Ithasalsoresultedinthelossoftheabilitytorespondquicklytoexpansionneedswithnotonlythediskbeingfullyutilized,butinfrastructuresuchasserversandSANswitchesatcapacity.

InresponsetothissituationameetingwasheldinNovember2007,andthenamoredetailedfollowupinJanuary2008.FromtheJanuarymeetingrequirementsforeachpresentlyidentifiedareawerecaptured,andanewanalysisstorageareaproposed.ThisdocumentwillgiveanoverviewoftheIceCubestoragetypesandareas,andpresenttherequirementsascapturedattheJanuarymeeting.

Overview

ThetotaldatastoragerequirementsforIceCubearelargebutnotmassive.Originally,whentherequirementsweremoremodest,itwasexpectedthatitwouldbepossibletostorealldataononlinedisk.Howeverithasreachedthepointthatthisisnotpracticalwithinbudgetconsiderations.AlsolargeonlinediskvolumesresultincorrespondinglyhighDataCenterspacerequirements,electrical,andcoolingneeds.Inresponsetothisitwasdecidedtointroduceatapebasedfilesystemforlongtermstorageofpotentiallylargevolumesofdata.ThissystemwasoriginallydesignedtohavelimitedfunctionalityprimarilyaimedatrestoringSouthPoledatatapes.Itisproposedthissystembeexpandedtomeetotherstorageneeds.

PresentlytheIceCubeDataCenterhasapproximately250TB(usable)ofonlinedisk,mostlyonanonspecifichardwarevendorsoftwarebasedSAN(Ibrix),andtoalesserextentdirectattachedstorage.RecentlyanHSMtapebasedfilesystemwasaddedwithaninitiallicensedcapacityof80TB.Inadditiontouseraccessiblestoragethereisabackupsystemwhichdoesnightlyincrementalbackups,andperiodicoffsitebackups.ThebackupsystemusesAtempoTimeNavigatorsoftware,andaSpectraLogicT950tapelibrary.

Theonlinestorageispresentlydividedinto3areas,experimentaldata,simulationdata,anduserdata.Ithasbeenproposedthatanadditionalstorageareaforanalysisbeadded.Thepresentstatusofonlinestorageisasfollows.

/data/exp141TB,93%used

/data/sim69TB89%used

/net/user(total)33TBbrokendownto24TB(UW)13TBused,9TB(non-UW)7.1TBused

Theuserstorageareaisarelativelysmallstorageareawhicheverycollaboratorhasaccessto.Eachuseriscurrentlylimitedto100GB,unlesstheirinstitutionprovidesfundingtoincreasethislimit,suchasthecasewithUW.ItisapracticalnecessitytohaveastorageareacloselycoupledtotheDataWarehouseforeachactiveuser,whichisessentiallyanextensionoftheirhomedirectory,whilebeingphysicallyseparatetoavoidanyperformanceissuesassociatedwithdatastorage,whichcouldimpactroutineoperationsdependantonhomedirectories.

TheIceCubestoragetypes,andassociatedresponsiblepeople,areasfollows.Thisincludestheproposedareaofanalysisstorage.

PoleDAQ:KaelHanson

OnlineFiltering:ErikBlaufuss,chairofTFTBoard

OfflineFiltering:MartinMerck

Simulation:PaoloDesiati

Analysis:GaryHill,AnalysisCoordinator

TherequirementspresentedinthisdocumenthavemostlyoriginatedatthemeetingonJanuary28th2008inMadison,whichproducedthedocument“NotesoftheStorageRequirementsMeeting”.ThisdocumentisaccompaniedbyadetailedIceCubeDataStorageRequirementsspreadsheet.

Requirements

SouthPoleDAQandOnlineFiltering

KaelHansonandErikBlaufuss

Itisproposedthatforthecomingyear,IC40,thatthecurrentarchivingarrangementcontinue,with3keytypesofDAQdataoutputbemaintained.Theseare:

Rawdatastream

DataTransferredoverthesatellite,predominatelyfiltereddata

Filtereddatanottransferredoverthesatellite

ForIC40theexpectedsizeofthesedatastreamsare,raw500GBperday(610GBuncompressed),satellite30GBperday,andfilteredbutnotoversatellite15GBperday.

ItishopedthatduringtheIC40periodthatonlinefilteringwillreachalevelofmaturitythatarchivingoftherawdatacouldbediscontinued.Infutureyearsitisproposedthattheentirerawdatastreamnotbearchived,andthattherawdataistapedforalimitedperiodaftertheadditionofnewstringsduringasettlinginperiodofthenewfilter,lessthan2months.

TheexpecteddatavolumesforIC60are750GBperday,andfiltereddata80GBperday.ForIC80theexpecteddatavolumesare1TBperdayrawdata,and135GBperdayoffiltereddata.Thedatavolumeoffiltereddatatobetransferredoverthesatellite,andfortapingforlaterphysicaltransport,willdependonupgradestotheTDRStransfersystemandconflictswithotherSouthPoleusers.In2008/2009itisexpectedthatoperationswillmovetousingTDRSF3,allowingIceCubetotransfer60GBperdayresultingin20GBperdayoffiltereddatabeingtaped.

Alldatavolumesarecompresseddata.

Theaccompanyingspreadsheetshowsbothscenariosoftaping,andnottaping,rawdata.Italsoassumesatapetechnologyupgradein2010.MoredetailsaboutSouthPoletapingiscontainedinadocument“ProposalforArchivingofDataatSouthPole”,February2008.

SimulationData

PaoloDesiati

Thedatastoragerequirementsofsimulationdependonnumerousfactorswhicharesummarizedintheattachedspreadsheet.Itwasdeterminedthattheimmediateneedsofsimulationwasanadditional10TBtocompleteIC40simulation,andmidyearanadditional50TB,andanadditional30TBinthefalltodoIC60simulation.

NextyearforIC80simulationitisexpectedthat60TBofdatacouldbemovedtoanotherstoragemediasuchastheHSM,andthatanadditional60TBwouldberequired.Thiswouldbethesteadystateforan80stringdetector.

ExperimentalData

MartinMerck

ExperimentaldataisdefinedasanydatatransferredfromtheSouthPole,andtheoutputof“PrimaryDataProcessing”(a.k.a.OfflineFiltering).Thedistinctionbetween“PrimaryDataProcessing”andAnalysisDataisbasedontheprojectswelldefinedprocessinplaceforthetwoareasmorethananythingelse.

Thedatalifecycleoffiltereddataintheofflinefilteringsystemisthatdataundergoes3processesfromtheoriginaloutputoftheonlinefiltering.TheoutputoftheonlinefilteringsystemissaidtobePFFilt.TheofflinefilteringisdoneinthenorthernhemisphereandtheoutputofeachfilteringprocessisLevel0,Level1,andLevel2.ThesizeoftheLevel2dataisapproximately50%largerthanPFFilt,andintermediatelevelsaresomewhereinbetween.

DuringtheIC40processingitisexpectedthattheoriginalPFFiltfiles,andall3outputlevelswillbeheldondiscsimultaneously.With30GBperdayofPFFiltdataarrivingoverthesatellite,thetotalrequirementofofflinefilteringwillbeapproximately105GBperday.ThustosupportofflinefilteringforIC40approximately50TBofstoragewillberequired,whichincludesasmalloverheadmargin.

AtthestartoftheIC60yeartheIC40non-satellitefiltereddatawillarrivefromSouthPoleontapes.ThisdatawillneedprocessingconcurrentlyastheIC60offlinefilteringstarts.Atthispointtheoutputofatleast2oftheIC40processinglevelscouldbedeletedprovidingthespacefortheprocessingoftheextraIC40data.TheHSMstoragesystemwouldalsobeutilizedinrestoringandstoringthenewSouthPoledata.HoweveratthisstageadditionalonlinestorageneedstobeaddedfortheoutputofIC60processing.Itisanticipatedthatatthisstagethefilteringwillbematureenoughthattheoutputofalltheprocessinglevelswillnothavetobesavedsimultaneouslyandthatanadditional60TBofstoragewouldbeadequate.ForIC80itisanticipatedthat100TBadditionalstoragewouldagainberequired.Atthispointsteadystateisreachedandanadditional100TBeachyearwouldberequiredunlessolderdatastartstobemovedtodifferentstorageareassuchastheHSM.

AnalysisData

MartinMerckandGaryHill

Itisproposedthatanewstorageareabeaddedfororganizedanalysiswithinthecollaboration.Therelevantanalysisareaswouldbeanalysisbeyondlevel2inofflinefiltering,androughlybasedontheanalysisworkinggroups.AssuchitisproposedthattheAnalysisCoordinatorwouldberesponsibleforoversightoftheusageofthisstorage,andforfuturedefinitionofrequirementsinthisarea.

Forplanningpurposesitisproposedthatthecapacityofthisareaexpandatthesamerateasofflinefiltering.Thusitstartwithaninitial50TBthisyear,expand60TBin2009,and100TBeachyearthereafter.

Accesstothisstoragewouldbebasedontheinfrastructurealreadyinplaceforsimulationandexperimentaldata.Forinstance,aleadforananalysisworkinggroupwouldcreateadirectorywithinthisstorageareaviatheDataWarehousewebinterface.Thiswouldcaptureinformationsuchasthepurposefortherequiredstorage,andaresponsibleperson.Itwouldalsoallowforsubsequenteasyaccessviathewebandquerytools.RegularusagesummarieswouldbesenttotheAnalysisCoordinator(anddesignees)foroversightofutilization.

UserData

ThereisasignificantadvantageforallusersofthecollaborationtohavingstoragecloselyconnectedtothecoreDataWarehousestorageandCPUresources.Themainpurposeofthisareaisasatemporarystorageareafordatatransferofforindividualanalysis,andhasthedesiredtechnicalconsequenceofkeepingdataoutofaccounthomedirectories,whichcancausesignificantresourceissues.

Basedonthepresentusagepatternsthecapacityseemtobeonthesmallside.Presentlyeachuserhasadefaultsoftquotaof100GB.Howevercompellingargumentshavebeenmadebynumeroususerswhichhasresultedintemporaryincreasesofupto200GB.Thecapacitywouldbeexpandedtoabitover200TBforallusers,allowinganincreaseindefaultquotasto150GBsoft,and200GBhard.Allinstitutionswouldhavetheopportunitytoaddadditionalstoragebeyondthebase,asUWhas,byanadditionalcontributiontothecommonfund.Otherwiseitisuptoindividualstoworkwithintheallocatedstorage.

Todealwithorganizedanalysiswithinthecollaborationanewstoragetype,analysisstorage,isproposed.

HSM(TapeBasedFilesystem)

AnHSMisatapedbasedfile-system.Dataisstoredontapes,butafrontenddiskbufferandapplicationmakethesystemappearasonlinedisk.Ifanattemptismadetoaccessdatanotonthebufferdisktheapplicationautomaticallyloadsthedatafromtape.Thusitisonlinestoragewithahighlatency.TheprimarypurposeoftheIceCubeHSMsystemisforaccessingthedatastoredattheSouthPole,withtheintroductionofanHSMandtapelibrarythisseasonatPole.HoweveritpresentlyhaslimitedabilitytostoredataattheDataCenterthatisnotassociatedwithSouthPole.Atpresentthemainlimitationisthatthesystemisnotsimultaneouslyreadwrite.

ItisproposedtoupgradetheHSMsystemattheDataCentertoprovideacheaperstoragetechnologyforinfrequentlyaccesseddata,aswellasprovidingaccesstotheSouthPoledata.

Archive

Archivingisatypeofstoragewhichisavailabletotheproject.Howeveritisimportanttounderstandwhatarchivingis.InthecontextofthepresentIceCubeinfrastructure(andnochangetothisisplanned)archivingisfordatathatcanpotentiallybedeletedfromaccessiblestorage,butthereissomesmallassociatedriskindoingso.Thusanarchivecopyismadejustincasethedataisneededinthefutureforunforeseenreasons.Itisnotfortemporaryremovalofdataforrestorationinthefuture.Thustherestorationofarchiveddataisnotplannedorbudgetedandanyfutureaccesswouldneedtofundedwhenrequired.Thereisalsotheissueofdatamigrationastapestoragetechnologiesadvance.Atpres