软件工程专业英语-Big Data

我们毕业啦其实是答辩的标题地方软件工程专业英语SOFTWAREENGINEERINGESSENTIALSCOMPETENCIESAfteryouhavereadthischapter,youshouldbeableto:Explainwhatbigdatais.Explainthe4Vpropertiesofbigdata:volume,velocity,variety,andveracity.Distinguishthecategoriesofbigdata.Understandthebigvalueofbigdata.DescribeJimGray’sFourthParadigm.Understandtheevolutionofbigdata.Discussthefourkindsofchallengesforbigdata.2023/10/21..22023/10/21Weareexperiencingunprecedentedgrowthintheamountofdataavailableinnearlyeveryarea,rangingfromthephysicalworld,includingbiology,astronomy,remotesensingetc.,tohumanactivities,includingsocialnetworks,Internet,health,finance,economics,transportation,etc..2023/10/21..3Thesedataarecommonlycalledbigdata,andbelievedtocontaingreatvaluesandimplynewopportunities.Thischapterpresentsanoverviewofbigdata,includingitsdefinition,properties,andcategories,aswellaschallengesitbringstous.2023/10/212023/10/21..46.1

BIGDATAANDITSPROPERTIES6.2CATEGORIESOFBIGDATA6.3BIG

VALUEFORBIGDATA6.4JIMGRAY’SFOURTHPARADIGM6.5EVOLUTIONOFDATAMANAGEMENTSUMMARYCHPTER6

BIGDATA6.6BIGDATACHALLENGES2023/10/212023/10/21..56.1BIGDATAANDITSPROPERTIES2023/10/21Bigdataisabroadtermfordatasetssolargeorcomplexthattraditionaldataprocessingapplicationsareinadequate.Mostprofessionalsintheindustryconsidermultipleterabytesorpetabytestobethecurrentbigdatabenchmark.2023/10/21..6Others,however,arehesitanttocommittoaspecificquantity,astherapidpaceoftechnologicaldevelopmentmayrendertoday’sconceptofbigastomorrow’snorm.Inthemeantime,aconsensusisreachedonthepropertiesofbigdata.2023/10/212023/10/21..712WHATISBIGDATA?FOURDIMENSIONSOFBIGDATA2023/10/212023/10/21..8Inadynamicworld,organizationshavebeguntomoreheavilyrelyoninsightsderivedfromtheirdatatouncovernewfactsandopportunitiesforsciencediscoveryorrevenuegrowth.Intheprocessofdiscoveringanddeterminingtheseinsights,largecomplexdatasetsaregeneratedthatthenmustbemanaged,analyzedandmanipulatedbyskilledprofessionals.Thecompilationofthislargecollectionofdataiscollectivelyknownasbigdata.ThedefinitionofbigdatafromWikipediafollows.”Bigdatausuallyincludesdatasetswithsizesbeyondtheabilityofcommonlyusedsoftwaretoolstocapture,curate,manage,andprocessdatawithinatolerableelapsedtime.Bigdata“size”isaconstantlymovingtarget,asof2012rangingfromafewdozenterabytestomanypetabytesofdata.Bigdataisasetoftechniquesandtechnologiesthatrequirenewformsofintegrationtouncoverlargehiddenvaluesfromlargedatasetsthatarediverse,complex,andofamassivescale.”WHATISBIGDATA?2023/10/212023/10/21..9Ina2001researchreportandrelatedlectures,METAGroup

(nowGartner)analystDougLaneydefineddatagrowthchallengesandopportunitiesasbeingthree-dimensional,i.e.increasingvolume(amountofdata),velocity(speedofdatainandout),andvariety(rangeofdatatypesandsources).In2012,Gartnerupdateditsdefinitionasfollows:"Bigdataishighvolume,highvelocity,and/orhighvarietyinformationassetsthatrequirenewformsofprocessingtoenableenhanceddecisionmaking,insightdiscoveryandprocessoptimization."[4]Additionally,anewV"Veracity"isaddedbysomeorganizations.

FOURDIMENSIONSOFBIGDATA2023/10/212023/10/21..10VolumeThesizeofthedatadeterminesthevalueandpotentialofthedataunderconsideration,andwhetheritcanactuallybeconsideredbigdataornot.Bigdatadoesn‘tsample.Itjustobservesandtrackswhathappens.

VelocityVelocityisanindicationofhowquicklythedatacanbemadeavailableforanalysis.Bigdataisoftenavailableinreal-time.VarietyVarietyreferencesthedifferenttypesofstructuredandunstructureddatathatorganizationscancollect,suchastransactiondata,video,audio,textandlogfiles.Bigdatadrawsfromalltypesofdata.VeracityVeracityisanindicationofdataintegrityandtheabilityforanorganizationtotrustthedataandbeabletoconfidentlyuseittomakecrucialdecisions.2023/10/212023/10/21..11Differentkindofdatashowsdifferentproperties.Forexample,socialnetworkhasmorerequirementsondatavolumeandvelocityratherthanvarietyandveracity.2023/10/212023/10/21..126.2CATEGORIESOFBIGDATA2023/10/212023/10/21..1312DATAFROMTHEPHYSICALWORLDDATAFROMHUMANACTIVITIESTheamountofdatainourworldhasbeenexplodingwithourabilitytoacquiredataenhancing.Bigdataisubiquitousandcanbepartitionedintotwocategories:2023/10/212023/10/21..14Scientificexperimentsandobservationsproducemassivescientificdatasetsaboutthephysicalworld.Sensornetworksconsistingoflargenumberofcheapsensorshavebeenwidelyusedtoobtainednaturaldata.OceanObservatoriesInitiativeuseselectro-opticallycabledobservingsystemstomeasureoceanactivitiesinthenortheastPacificOcean.PanoramicSurveyTelescopeandRapidResponseSystem(Pan-STARRS)produce2.5PBdataeachyear.Over5000genomeprojectsin2010producedseveralEBgenomicdata.Remotesensingproducesevenlargernaturaldatasets.DATAFROMTHEPHYSICALWORLD2023/10/212023/10/21..152023/10/212023/10/21..16Bigdatasetsproducedbyhumanactivities,suchassocialnetworks,Internet,health,finance,economics,transportation,etc.,attractedmuchattentioninrecentyears.Togainabetterunderstandingonhowmanydataisbeinggenerated,considerthefollowingnoteworthyfacts:Facebookcurrentlyholdsmorethan45billionphotosinitsuserdatabase,anumberthatisgrowingdaily.AccordingtoIBM,userscreate2.5quintillionbytesofdataeveryday.Inpracticalterms,thismeansthat90%ofthedataintheworldtodayhasbeencreatedinthelasttwoyearsalone.AccordingtoFICO,thecreditcardfraud

detectionsystemcurrentlyinplacehelpsprotectovertwobillionaccountsallovertheglobe.

Walmartcontrolsmorethan1millioncustomertransactionseveryhour,whicharethentransferredintoadatabaseworkingwithover2.5petabytesofinformation.DATAFROMHUMANACTIVITIES2023/10/212023/10/21..176.3BIGVALUEFORBIGDATA2023/10/21GinniRometty,PresidentandChiefExecutiveOfficerofIBM,hassaidthat“Dataistheworld'sgreatnewnaturalresource.Whatsteampowerwastothe18thcentury,electromagnetismtothe19thandfossilfuelstothe20th…datawillbetothe21st.”Likeotheressentialfactorsofproductionsuchashardassetsandhumancapital,itisincreasinglythecasethatmuchofmoderneconomicactivity,innovation,andgrowthsimplycouldn’ttakeplacewithoutdata.2023/10/21..182023/10/21Whiledigitaldatamightoncehavebeenconcernedbyonlyafewdatageeks,bigdataisnowrelevantforleadersacrosseverysector,andconsumersofproductsandservicestostandtobenefitfromitsapplication.2023/10/21..19ThecombinationofdeepeninginvestmentsinbigdataandmanagerialinnovationtocreatecompetitiveadvantageandboostproductivityisverysimilartothewayITdevelopedfromthe1970sonward.TheexperienceofITstronglysuggeststhatwecouldbeonthecuspofanewwaveofproductivitygrowthenabledbytheuseofbigdata.2023/10/21Therearemanywaysthatbigdatacanbeusedtocreatevalue.Largecompaniesacrosstheglobehavescoredearlysuccessesintheiruseofbigdata.Therearenotableexamplesofcompaniesaroundtheglobethatarewell-knownfortheirextensiveandeffectiveuseofdata.Forinstance,Tesco’sloyaltyprogramgeneratesatremendousamountofcustomerdatathatthecompanyminestoinformdecisionsfrompromotionstostrategicsegmentationofcustomers.Amazonusescustomerdatatopoweritsrecommendationengine“youmayalsolike…”basedonatypeofpredictivemodelingtechniquecalledcollaborativefiltering.2023/10/21..202023/10/21Bymakingsupplyanddemandsignalsvisiblebetweenretailstoresandsuppliers,Wal-Martwasanearlyadopterofvendor-managedinventorytooptimizethesupplychain.Harrah’s,theUShotelsandcasinosgroup,compilesdetailedholisticprofilesofitscustomersandusesthemtotailormarketinginawaythathasincreasedcustomerloyalty.ProgressiveInsuranceandCapitalOnearebothknownforconductingexperimentstosegmenttheircustomerssystematicallyandeffectivelyandtotailorproductoffersaccordingly.Smart,aleadingwirelessplayerinthePhilippines,analyzesitspenetration,retailercoverage,andaveragerevenueperuseratthecityortownlevelinordertofocusonthemicromarketswiththemostpotential.2023/10/21..212023/10/21McKinsey&

Companyobservedhowbigdatacreatedvaluesafterin-depthresearchontheU.S.healthcare,theEUpublicsectoradministration,theU.S.retail,theglobalmanufacturing,andtheglobalpersonallocationdata.Throughresearchonthefivecoreindustriesthatrepresenttheglobaleconomy,theMcKinseyreportpointedoutthatbigdatamaygiveafullplaytotheeconomicfunction,improvetheproductivityandcompetitivenessofenterprisesandpublicsectors,andcreatehugebenefitsforconsumers.2023/10/21..222023/10/21McKinseysummarizedthevaluesthatbigdatacouldcreate:ifbigdatacouldbecreativelyandeffectivelyutilizedtoimproveefficiencyandquality,thepotentialvalueoftheU.S.medicalindustrygainedthroughdatamaysurpassUSD300billion,thusreducingtheU.S.healthcareexpenditurebyover8%;retailersthatfullyutilizebigdatamayimprovetheirprofitbymorethan60%;bigdatamayalsobeutilizedtoimprovetheefficiencyofgovernmentoperations,suchthatthedevelopedeconomiesinEuropecould

saveoverEUR100billion(whichexcludestheeffectofreducedfrauds,errors,andtaxdifference).

2023/10/21..232023/10/21TheMcKinseyreportisregardedasprospectiveandpredictive,whilethefollowingfactsmayvalidatethevaluesofbigdata.Duringthe2009flupandemic,Googleobtainedtimelyinformationbyanalyzingbigdata,whichevenprovidedmorevaluableinformationthanthatprovidedbydiseasepreventioncenters.Nearlyallcountriesrequiredhospitalsinformagencies,suchasdiseasepreventioncenters,ofnewtypeofinfluenzacases

.However,patientsusuallydidnotseedoctorsimmediatelywhentheygotinfected.Italsotooksometimetosendinformationfromhospitalstodiseasepreventioncenters,andfordiseasepreventioncenterstoanalyzeandsummarizesuchinformation.Therefore,whenthepublicisawareofthepandemicofanewtypeofinfluenza,thediseasemayhavealreadyspreadforonetotwoweekswithaserioushystereticnature.2023/10/21..242023/10/21Googlefoundthatduringthespreadingofinfluenza,entriesfrequentlysoughtatitssearchengineswouldbedifferentfromthoseatordinarytimes,andtheusagefrequenciesoftheentrieswerecorrelatedtotheinfluenzaspreadinginbothtimeandlocation.Googlefound45searchentrygroupsthatwerecloselyrelevanttotheoutbreakofinfluenzaandincorporatedtheminspecificmathematicmodelstoforecastthespreadingofinfluenzaandeventopredictplaceswhereinfluenzawillspreadfrom.TherelatedresearchresultshavebeenpublishedinNature.2023/10/21..252023/10/21In2008,MicrosoftpurchasedFarecast,asci-techventurecompanyintheU.S.Forecasthasanairlineticketforecastingsystemthatpredictsthetrendsandrising/droppingrangesofairlineticketprices.ThesystemhasbeenincorporatedintotheBingsearchengineofMicrosoft.By2012,thesystemhassavednearlyUSD50perticketperpassenger,withtheforecastaccuracyashighas75%.2023/10/21..262023/10/212023/10/21..276.4JIMGRAY’SFOURTHPARADIGM2023/10/212023/10/21..28FirstExperimentalScienceOriginally,therewasjustexperimentalsciencedescribingnaturalphenomena.SecondTheoreticalScienceTheninthelastfewhundredyears,therewastheoreticalscience,withKepler’sLaws,Newton’sLawsofMotion,Maxwell’sequations,andsoon.ThirdComputationalScienceInthelastfewdecades,formanyproblems,thetheoreticalmodelsgrewtoocomplicatedtosolveanalytically,andpeoplehadtostartsimulating.Thesesimulationshavecarriedusthroughmuchofthelasthalfofthelastmillennium.Atthispoint,thesesimulationsaregeneratingawholelotofdata,alongwithahugeincreaseindatafromtheexperimentalsciences.Peoplenowdonotactuallylookthroughtelescopes.Instead,theyare“looking”throughlarge-scale,complexinstrumentswhichrelaydatatodatacenters,andonlythendotheylookattheinformationontheircomputers.FourthData-intensiveScienceTheworldofsciencehaschanged,andthereisnoquestionaboutthis.Thenewmodelisforthedatatobecapturedbyinstrumentsorgeneratedbysimulationsbeforebeingprocessedbysoftwareandfortheresultinginformationorknowledgetobestoredincomputers.Scientistsonlygettolookattheirdatafairlylateinthispipeline.Thetechniquesandtechnologiesforsuchdata-intensivesciencearesodifferentthatitisworthdistinguishingdata-intensivesciencefromcomputationalscienceasanew,fourthparadigmforscientificexploration.

JimGraybelievesthatscientificdiscoveryhasexperiencedfourparadigms.2023/10/212023/10/21..296.5EVOLUTIONOFDATAMANAGEMENT2023/10/212023/10/21..30Peopleproposed“sharenothing”aparalleldatabasesystem,tomeetthedemandoftheincreasingdatavolume.Thesharenothingsystemarchitectureisbasedontheuseofclusterandeverymachinehasitsownprocessor,storage,anddisk.Teradatasystemwasthefirstsuccessfulcommercialparalleldatabasesystem.Suchdatabasebecameverypopularlately.Inthe1980sTheconceptof“databasemachine”emerged,whichisatechnologyspeciallyusedforstoringandanalyzingdata.Withtheincreaseofdatavolume,thestorageandprocessingcapacityofasinglemainframecomputersystemhasbecomeinadequate.Inlate1970s2023/10/212023/10/21..31Theadvantageoftheparalleldatabasewaswidelyrecognizedinthedatabasefield.Inlate1990sOnJune2,1986,amilestoneeventoccurred,whenTeradatadeliveredthefirstparalleldatabasesystemwithastoragecapacityof1TBtoKmarttohelpthelarge-scaleretailcompanyinNorthAmericatoexpanditsdatawarehouse.OnJune2,19862023/10/212023/10/21..32Anothermilestoneeventoccurred,whenEMC/IDCpublishedaresearchreporttitledExtractingValuesfromChaos,whichintroducedtheconceptandpotentialofbigdataforthefirsttime.Thisresearchreportarousedgreatinterestinbothindustryandacademiaonbigdata.InJune2011However,manychallengesonbigdataarose.Contentsgeneratedbyusers,sensors,andotherubiquitousdatasourcesdrivetheoverwhelmingdataflows,whichrequiredafundamentalchangeonthecomputingarchitectureandlarge-scaledataprocessingmechanism.JimGray,apioneerofdatabasesoftware,calledsuchtransformation“TheFourthParadigm”.Hethoughttheonlywaytocopewithsuchaparadigmwastodevelopanewgenerationofcomputingtoolstomanage,visualize,andanalyzemassivedata.InJanuary20072023/10/212023/10/21..33Overthepastfewyears,nearlyallmajorcompanies,includingEMC,Oracle,IBM,Microsoft,Google,Amazon,andFacebook,etc.,havestartedtheirbigdataprojects.TakingIBMasanexample,since2005,IBMhasinvestedUSD16billionon30acquisitionsrelatedtobigdata.Industry2023/10/212023/10/21..34Inacademia,bigdatawasalsounderthespotlight.In2008,Naturepublishedthebigdataspecialissue.In2011,Sciencealsolaunchedaspecialissueonthekeytechnologiesof“dataprocessing”inbigdata.In2012,EuropeanResearchConsortiumforInformaticsandMathematics(ERCIM)Newspublishedaspecialissueonbigdata.Inthebeginningof2012,areporttitledBigData,BigImpactpresentedattheDavosForuminSwitzerland,announcedthatbigdatahasbecomeanewkindofeconomicassets,justlikecurrencyofgold.Gartner,aninternationalresearchagency,issuedHypeCyclesfrom2012to2013,whichclassifiedbigdatacomputing,socialanalysis,andstoreddataanalysisinto48emergingtechnologiesthatdeservemostattention.Academia2023/10/212023/10/21..352023/10/212023/10/21..36ManynationalgovernmentssuchastheU.S.alsopaidgreatattentiontobigdata.InMarch2012,theObamaAdministrationannouncedaUSD200millioninvestmenttolaunchtheBigDataResearchandDevelopmentInitiative,whichwasasecondmajorscientificandtechnologicaldevelopmentinitiativeaftertheInformationHighwayInitiativein1993.InJuly2012,theJapan’sICTprojectissuedbyMinistryofInternalAffairsandCommunicationsindicatedthatthebigdatadevelopmentshouldbeanationalstrategyandapplicationtechnologiesshouldbethefocus

.InJuly2012,theUnitedNationsissuedBigDataforDevelopmentreport,whichsummarizedhowgovernmentsutilizedbigdatatobetterserveandprotecttheirpeople.Government2023/10/212023/10/21..376.6BIGDATACHALLENGES2023/10/212023/10/21..38VolumeDatavolumeposesthemostnoticeablechallengeduetolimitedhardwarecapacityandsoftwareefficiencyandeffectiveness.Hardwarecapacityincludessizeandspeedofstoragesystems,CPUfrequencyandparallelism,andcommunicationbandwidth.Softwareefficiencyandeffectivenesshavetodowitharchitectsandalgorithms.Theseissuesshouldberevisitedtoaddressthechallengeofbigdatavolume.VelocityThechallengeofbigdatavelocitycomeswiththeneedstotimelygenerateresultsandprocessarrivingdata.Bigdatavelocitybringschallengestoeverystackofadatamanagementplatform.Boththestoragelayerandthequery