大数据实验指导手册19-spark graphx

实验十 Spark实验Spark的图计算框架GraphXGraphX要求实验结束时，每位学生能完成正确运行SparkGraphX的示例程序，正确上传到集SparkGraphX是一个分布式图处理框架，SparkGraphXSpark平台提供对图计算社交网络中人与人之间有很多关系链，例如、、、，这些都机处理，SparkGraphXSpark来处理的，所以天然就是一个分布式的图处图的分布式或者并行处理其实是把这拆分成很多的子图，然后我们分别对这些子本实验主要可以分为在IDEA安装ScalaScala的程序、打包提交程序和在 lijIDEA中安装Scala的插注：InliJIDEA对Scala编程比较方便。如不习惯请参考网上Eclipse版本FilePlugins->输入”Scala”->Scala的项目->点击右侧绿色按钮Install->IDEA。新建Scala点击当前IDEA中的project然后new->module->选择Scala->next->输modulenamesparkgraphx”点击新建的sparkgraphx的module然后”AddFrameworkSupport”选择maven添加mavenpom.xml<artifactId>spark-graphx新建Scala打开项 然后右键新建package命名为test，然后新建Scalaclass将Kindt由packageimportorg.apache.log4j.{Level,packageimportorg.apache.log4j.{Level,Logger}importorg.apache.spark.graphx.importobjectGraphXExampledefmain(args:Array[String]){valconf=newSparkConf().setAppName("SimpleGraphX").setMaster("local")valsc=newSparkContext(conf)valvertexArray=Array((1L,("Alice",28)),(2L,("Bob",(3L, ",(4L,("David",(5L,("Ed",(6L,("Fran",)valedgeArray=Array(Edge(2L,1L,7),Edge(2L,4L,Edge(3L,2L,Edge(3L,6L,Edge(4L,1L,Edge(5L,2L,Edge(5L,3L,Edge(5L,6L,)valvertexRDD:RDD[(Long,(String,Int))]=sc.parallelize(vertexArray)valedgeRDD:RDD[Edge[Int]]=sc.parallelize(edgeArray)valgraph:Graph[(String,Int),Int]=Graph(vertexRDD, 大于30的顶点：")graph.verticesfilter{case(id,(name,age))=>age>30}.collectforeach{case(id,(name,age))=>println(s"$nameis$age")}graph.edgesfilter(e=>e.attr>5).collectforeach(e=>println(s"${e.srcId}to${e.dstId}att${e.attr}"))println("列出边属性>5的for(triplet<-graph.tripletsfilter(t=>t.attr>5).collect){}defmax(a:(VertexId,Int),b:(VertexId,Int)):(VertexId,Int)={if(a._2>b._2)aelseb}println("maxofoutDegrees:"+graph.outDegrees.reduce(max)+"maxofinDegrees:"+graph.inDegreesreduce(max)+"maxofDegrees:"+graph.degreesreduce(max))println("顶点的转换操作，顶点age+10：")graphmapVertices{case(id,(name,age))=>(id,(name,age+10))}.vertices.collect.foreach(v=>println(s"${v.2.1}is${v.2.2}"))graphmapEdges(e=>e.attr*2).edges.collectforeach(e=>println(s"${e.srcId}to${e.dstId}println("顶点年纪>30的子图：")valsubGraphgraph.subgraph(vpredid,vdvd._230)subGraph.vertices.collectforeach(v=>println(s"${v._2._1}is${v._2._2}"))subGraph.edges.collect.foreach(e=>println(s"${e.srcId}to${e.dstId}att${e.attr}"))valinDegrees:VertexRDD[Int]=graph.inDegreescaseclassUser(name:String,age:Int,inDeg:Int,outDeg:valinitialUserGraph:Graph[User,Int]=graph.mapVertices{case(id,(name,age))=>User(name,0,valuserGraph=initialUserGraph.outerJoinVertices(initialUserGraph.inDegrees){case(id,u,inDegOpt)=>User(,u.age,inDegOpt.getOrElse(0),u.outDeg)case(id,u,outDegOpt)=>User(uname,u.age,}userGraph.vertices.collectforeach(v=>println(s"${v.2name}inDeg:${v. }}else }(a,b)=>math)}}case(id,u)=>u.inDeg==case(id,property)=>println(property}println("找出5到各顶点的最短：")valsourceId:VertexId=valinitialGraph=graph.mapVertices((id,)=>if(id==sourceId)0.0elseDouble.PositiveInfinity)valsssp=initialGraph.pregel(Double.PositiveInfinity)((id,dist,newDist)=>math.min(dist,newDist),triplet=