spark大数据分析平台Spark快速计算讲师

Spark13DATAGURU专业数据分析Spark大数据快速计算平风 课程详 Spark大数据快速计算平风

DATAGURU专业数据分析RDD的编SparkSpark大数据快速计算平风

DATAGURU专业数据分析SparkSpark大数据快速计算平风

DATAGURU专业数据分析Driver

Ajarcontainingtheuser'sSparkapplication.Insomecasesuserswillwanttocreatean"uberjar"containingtheirapplicationalongwithitsdependencies.Theuser'sjarshouldneverincludeHadooporSparklibraries,however,thesewillbeaddedatTheprocessrunningthemain()functionoftheapplicationandcreatingtheCluster Anexternalserviceforacquiringresourcesonthecluster(e.g.standalonemanager,Mesos,Deploy Distinguisheswherethedriverprocessruns.In"cluster"mode,theframeworklaunchesthedriverinsideofthecluster."client"mode,thesubmitterlaunchesthedriveroutsideoftheWorker Anynodethatcanrunapplicationcodeinthe Aprocesslaunchedforanapplicationonaworkernode,thatrunstasksandkeepsdatainmemoryordiskstoragethem.Eachapplicationhasitsown Aunitofworkthatwillbesenttoone AparallelcomputationconsistingofmultipletasksthatgetsspawnedinresponsetoaSparkaction(e.g.save,collect);you'llseethistermusedinthedriver'slogs.EachjobgetsdividedintosmallerEachjobgetsdividedintosmallersetsoftaskscalledstagesthatdependoneachother(similartothemapandreducestagesinMapReduce);you'llseethistermusedinthedriver'slogs.DATAGURU专业数据分析SparkRDDSpark大数据快速计算平风

DATAGURU专业数据分析Spark编程模Spark大数据快速计算平风

DATAGURU专业数据分析RDDSpark大数据快速计算平风

DATAGURU专业数据分析描述scala>valmap1=rdd.map(_*3)res0:Array[Int]=Array(3,6,9,12,scala>valrddscala>valfiltered=res2:Array[String]=Array(jifeng,scala>valrdd=sc.parallelize(List("aaaddfdc","dcisfun"))scala>valfm=rdd.flatMap(str=>str.split(""))res3:Array[String]=Array(aa,ad,df,dc,dc,is,Spark大数据快速计算平风

DATAGURU专业数据分析描述reduceByKey(func,scala>valwd=fm.map(word=>(word,1))scala>wdnum.collect()Array[(String,Int)]=Array((aa,1),(is,1),(df,1),(dc,2),(ad,1),Array[(String,Int)]=Array((aa,1),(ad,1),(dc,2),(df,1),(fun,1),scala>valcntWrd=wdnum.map{case(word,count)=>(count,word)}scala>cntWrd.groupByKey().collect()Array[(Int,Iterable[String])]=Array(( pactBuffer(dc)),( pactBuffer(aa,is,df,ad,scala>fm.distinct().collect()Array[String]=Array(aa,is,df,dc,ad,Spark大数据快速计算平风

DATAGURU专业数据分析描述 scala>rdd.reduce(_+_)Int=scala>valrdd=rdd:Array[String]=Array(A,B,C,scala>rdd.count()Long=scala>rdd.take(3)Array[String]=Array(A,B,Spark大数据快速计算平风

DATAGURU专业数据分析描述scala>rdd.first()String=AScala>valrdd=scala>$ls//查看part-00000part-00001Spark大数据快速计算平风

DATAGURU专业数据分析描述Spark大数据快速计算平风

DATAGURU专业数据分析haRDhaRDRDaquet文件、Jh查询he注意Spark1.1使用registerTempTable代替1.0版本的Spark1.1在hiveContext中，hql()将被弃用，sql()将代替hql()来提交查询语句，统一了接口使用 表是一个临时表，生命周期只在所定义的sqlContext或hiveContext实例之中spark1.1提供了语法解析器选项spark.sql.dialect，目前有两种语法解析器，sql语法解析器和hiveql语法解析器sqlContext现在只支持sql语法解析器（SQL-92语法）hiveContext现在支持sql语法解析器和hivesql语法解析器，默认为hivesql语法解析器，用户可以通过配置切换成sql语法解析器Spark大数据快速计算平风

DATAGURU专业数据分析 [jifeng@feng03~]$mkdirhive[jifeng@feng03~]$cdhive/[jifeng@feng02hive]$tarzxfapache-hive-0.13.1-bin.tar.gz[jifeng@feng02hive]$lsSpark大数据快速计算平风

DATAGURU专业数据分析Hive[jifeng@feng03hive]$viexport2修改 下的文[jifeng@feng03conf]$perties.template[jifeng@feng02conf]$mvhive-env.sh.templatehive-env.sh[jifeng@feng02conf]$cphive-default.xml.templatehive-Spark大数据快速计算平风

DATAGURU专业数据分析Hive修改 下的文export[jifeng@feng02conf]$cd[jifeng@feng02bin]$vihive-#Defaulttouseexport"hive-config.sh"73L,2023CSpark大数据快速计算平风

DATAGURU专业数据分析Hive配置[jifeng@feng03conf]$vihive-<property><description>JDBCconnectstringforaJDBC <description>DriverclassnameforaJDBCSpark大数据快速计算平风

DATAGURU专业数据分析Hive<description>usernametouseagainstmetastore<description>passwordtouseagainstmetastoreSpark大数据快速计算平风

DATAGURU专业数据分析Hive启动[jifeng@feng02hadoop-2.6.0]$./sbin/start-启动[jifeng@feng03hive]$15/09/0516:31:10WARNconf.HiveConf:DEPRECATED:hive.metastore.ds.retry.*nolongerhasanyeffect.Usehive.hmshandler.retry.*insteadLogginginitializedusingconfigurationinjar:file:/home/jifeng/hive/apache-hive-0.13. hive>createdatabasesale;Timetaken:1.188Spark大数据快速计算平风

DATAGURU专业数据分析hive>createtableshop(locationid namestring)rowformatdelimitedfieldsterminatedbyexceptionwasthrownwhileadding/validatingclass(es):Specifiedkeywastoolong;maxkeylengthis767atatjava.lang.reflect.Constructor.newInstance(Constructor.java:526)atcom.mysql.jdbc.Util.getInstance(Util.java:360)mysql>alterdatabasehivecharactersetlatin1;QueryOK,1rowaffected(0.00sec)Spark大数据快速计算平风

DATAGURU专业数据分析hive>createtableshop(locationidint,name namestring)rowformatdelimitedfieldsterminatedby',';Timetaken:0.884hive>createtablesell(locationidint,fyearint,amountdouble,numdouble)rowformatdelimitedfieldsterminatedbyTimetaken:0.158hive>showtables;Timetaken:0.105seconds,Fetched:2row(s)Spark大数据快速计算平风

DATAGURU专业数据分析上传文件到jifeng@feng03hive]$hadoopfsputhome/jifeng/code/db/shop.txtuser/jifeng/fire/hive>loaddatainpath'/user/jifeng/fire/shop.txt'intotableshop;Loadingdatatotabledefault.shopTimetaken:0.657hive>loaddatainpath'/user/jifeng/fire/sell.txt'intotablesell;Loadingdatatotabledefault.sellTimetaken:0.38Spark大数据快速计算平风

DATAGURU专业数据分析Hivesupportisenabledbyaddingthe-Phiveand-Phive-thriftserverflagstoSpark’sConfigurationofHiveisdonebyplacingyourhive-site.xmlfileinspark-sql>select*fromshopwhere15/09/0516:54:52INFOscheduler.TaskSchedulerImpl:Addingtaskset2.0with2tasks15/09/0516:54:52INFOstorage.BlockManagerInfo:Addedbroadcast_3_piece0inmemoryon10:58340(size:35.2KB,free:267.2MB)15/09/0516:54:52INFOscheduler.TaskSetManager:Finishedtask0.0instage2.0(TID3)in574mson10(1/2)15/09/0516:54:53INFOscheduler.DAGScheduler:ResultStage2(processCmdatCliDriver.java:423)finishedin0.617s15/09/0516:54:53INFOscheduler.DAGScheduler:Job2finished:processCmdatCliDriver.java:423,took0.680182s Spark大数据快速计算平风

DATAGURU专业数据分析 importimportorg.apache.spark.sql.hive.HiveContextimportorg.apache.spark.mllib.linalg.VectorsvalsqlContext=newvalsqldata=sqlContext.sql("selectlocationid,sum(num)allnum,sum(amount)allamountfromsellgroupbylocationid")Spark大数据快速计算平风

DATAGURU专业数据分析valparsedData=sqldata.mapcaseRow(_,allnum,allamount)} valnumClusters=3valnumIterations=valmodel=KMeans.train(parsedData,numClusters,Spark大数据快速计算平风

DATAGURU专业数据分析valresult1=sqldata.map{caseRow(locationid,allnum,allamount)vallinevectore=Vectors.dense(features)valprediction=locationid+""+allnum+""+allamoun