K-means聚类试验

1、实验算法K-means聚类实验【实验名称】K-means聚类实验【实验要求】掌握K-means模型应用过程，根据模型要求进行数据预处理，建模，评价与应用；【背景描述】聚类算法是一种典型的无监督学习算法，在聚类算法中根据样本之间的相似性，将样本划分到不同的类别中，对于不同的相似度计算方法，会得到不同的聚类结果，常用的相似度计算方法有欧式距离法。【知识准备】了解K-means模型的使用场景，数据标准。了解Python/Spark数据处理一般方法。了解spark模型调用，训练以及应用方法【实验设备】Windows或Linux操作系统的计算机。部署Spark,Python,本实验提供centos6.8

2、环境。【实验说明】采用UCI机器学习库中的wine数据集作为算法数据，除去原来的类别号，把数据看做没有类别的样本，训练K-means模型，对样本进行聚类。【实验环境】Spark2.3.1,Pyrhon3.X,实验在命令行pyspark中进行，或者把代码写在py脚本，由于本次为实验，以学习模型为主，所以在命令行中逐步执行代码，以便更加清晰地了解整个建模流程。【实验步骤】第一步：启动pyspark命令行中键入pyspark-masterlocal4,本地模式启动spark与pythonGCC4.4.720129313作小Hat4.4.7-1)CftlihmTyp*p*.copyrighrt.cre

3、ditsorlicsnstformensiafanutinn.cUtepMhsruLnsmultipleSLFMJbindings.SLFU：FoundBindinginjdn/tpark-2412116.jdrifofo4j/ivDl/suticljOM0rttMtdM-.clKSSL=-4J：Foindtindingin|ar：fila：/o3：/hadQdp-j.l.i3/sharfi.-,hado；p/c&wndn/ib/sl.l4j-lojLi-i./.25.iriIcLoggerfilfider.cLassJSLT4LSettittp;/A*Mh5U4xrg/c&d

4、t5.hlml#aultiplc_birdrngi+crancxpUriation,SIF4T-ArtjaIbin-inqisrtft/peong.ilfdj.irpl.oj4jloggprfactor/2019-04-751S；41；16WMiNativeCodeLDader；62-Lhabletoloadnative-hadooplibraryfaryourplatfom.usingbuiltim-jdvaclassesiereapplicableS&itinqd#Fdult1唱levelto.TuJdjutU用irqlv#1u u馆”.、父1_叫1_门“皿4-白/).FflfEp

5、“4Rn、皿5吴“mwLmvfU蚓叽U/_/_f/_/_/_f/_yf_ryf_r_j_jjj/_/rsioi2.3UsUsinzPythianversionversion3.S.2(detault(detault, ,JulJul2201617:17:53:061 1SparkSess-ian-dvliable曰9spark.*.4 4I I第二步：导入用到的包，并读取数据：(1) .导入所需的包frompysparkimportSparkContext,SQLContext,SparkConffrommathimportsqrt(2) .读取数据源，数据源地址为：/opt/algorith

6、m/kmeans/wine.txtdf_wine=sc.textFile(ufile:/opt/algorithm/kmeans/wine.txt).map(lambdax:str(x).split(,).map(lambdax:float(z)forzinx)frompysparkuiportSparkContextrStJLCont日?d,SparkConffirofnmthimpartsqrtdfjwijie=sc.textFile(ufile：/opt/alganthffl/kmeans/wine,txt).niap(，lambdax：str(x).map(lambdax：z)forz

7、inx)a|(3) .数据转换为DataFramedf_wine_rdd=sqlContext.createDataFrame(df_wine)a，adfjrfinc_rdd-sqlContext.ersatftDatarramstdfinc)17rlS:?fiWARNflijortSto-Filtrg&t叮1独刮/4(1.returningNaSuchCbjtFxroption|(4) .展示数据df_wine_rdd.show()(ifjwine_r(ld.showOT-_Tr-r-L-r-4-l-4-4-T44=T-FIII2|314|5|6|7|8|9|LSIJl|INI13|

8、Ml+十一一 TH+T+*r+-+第三步：数据预处理(1) .去掉类别标号那一类df_wine_rdd=df_wine_rdd.drop(_1)(2) .构建向量importpyspark.ml.featureasml_featurecols=df_wine_rdd.columnsvectorAssembler=ml_feature.VectorAssembler().setInputCols(cols).setOutputCol(cols)|1*0|14,23111.013.2|1 。13.16|1.0114.2711.9113.24|1.9|14-2|1-0|14.3911.9114.06

9、I7I14.S3|1.013.|1.0|13.75I1.9IU.75|1.0|14,30111.013,63111.0143|1.013.831.712.43 口 5,51.732.1411173352,&718.61.952.516,8127,02.81W.S265101102.8113.03.353.062.763,243.490,Z80,260.30.242.592.&7I21.0I1L8,Q2.82.69|0.391.762-45|15.21112.33.；7|3.39|0.24l.ff7|2.45|14P6|96,012,5|2,52|0.321512.61|L7.6

10、|121,0|22.510JIl.42.17|14MQ1.352.27I1B.0中 02498.02.983295,54L043.921065-0128%领1.053.41050,02.815.6B|1.S33.171185.0|2AS|7.80.963.45L480.0I1.8214.32|1,Q4|2.93|735.0|1.97&.75|1.05|2.951450.0|1.我13,25l.Q2|3.5S|L290.0|1.2515,05|1.96|3,58|1295,G|2.161.4S1.731.711,87|1.81L921.572.318.02E16.82.41116.D2a

11、lidfwinerdd-dfwinerdd.drop()importpyspark,ml.featureasml_featurec&ls-dtwine_rdd,calumnsvectorAssembler-(nl_featdrerVectorAssmbl.er(TsetInputCols(c&ls).setOutputColfcols)wine_Vc=vectorAssenbler.transfom(df_wine_rdd I I(3).对数据进行标准化standardScaler=ml_feature.StandardScaler().setInputCol(cols).se

12、tOutputCol(cols_st).setWithMean(True).setWithStd(True).fit(wine_Vc)wine_Vc_St=standardScaler.transform(wine_Vc)nnd式口rfl_fmtu明文;)ndirdsc.sorinputcol(calsJ.otoutpuicoHJc.a工竹M刖1ru*Lt&irrtittiSTdIvc)wineVtSt=群taftd3rdscaTL&r.tm6年仃12楠白uc)!-一第四步：构建模型并应用，输出聚类后的分类.Kmeans模型，设置输入列，迭代次数，输出列，聚类数importpy

13、spark.ml.clusteringasml_clusteringclusters=ml_clustering.KMeans(featuresCol=cols_st,predictionCol=Pred,initMode=k-means|,initSteps=5,tol=1e-4,maxIter=20,seed=None).fit(wine_Vc_St)wine_Vc_St_clusters=clusters.transform(wine_Vc_St),5in|rtpyEfari.nl.tlMtaringKUjcluitarifq八、clusters gclueinq.Kjieint(tut

14、urco15O.-predictionCflil=Pre(l.initn)lelL-iadr)g|inLLSlyJ5.-5.-tol=lv-4,，MJuIgES,swdlitin?，).liTftfuie/c_St)17t22r03riAATdBLAS：61-Fdn回loICJJupl倒胡LhcinfJoniCDT.gj.thub.fonnil.rClibLlUtiueSyilMBl_AS2019-G4-2517t22;C3RAHM6L.SSLdFdil旬laIgJijipl#nldtiQAfFoniLQT.yjtKubonniJJ昭L4SwiiwVcSt_cltf8trs-cluitert.

15、trirnfoHI(mintmintvcSt)m mI I(2),打印模型wine_Vc_St_clusters.show()wwrke_/c_St_cluitenihMI2019-0i-3517：j3：0ffWAPW-TrunotMth*、tnn?ionVaplnline*it+aoTh*b备匚立nheAdjustedbysettingspjrkd-burniaxTcStringFipldl!i1inSpairkFnvfonif第五步：构建模型评估方法(1),输出聚类中心并合并到数据表importpyspark.mllib.linalgaslinalgcenter=zip(0,1,2,lin

16、alg.Vectors.dense(z)forzinclusters.clusterCenters()centers=sqlContext.createDataFrame(center).toDF(pred,center)wine_Vc_St_clusters_centers=wine_Vc_St_clusters.join(centers,on=pred)b“unpcrt分pdrk.nll必liklg曲liridlg4小center=zip(GGt t1,22t tliiulg.VectOE.densM/fori iinclusters.cljsterCenters)aacenters=5q

17、KontextcresteDataFrameicenter1,t(jDF(,prtd,fcenter)wineVcStclust&n_centers=winej/cStcljaters.joincenters,on=3r?dwineVcStclust&n_centers=winej/cStcljaters.joincenters,on=3r?d ) )4 4I I(2),计算出误差平方和WSSSE=wineVcStclusterscenters.select(center,colsst).rdd.map(lambdapl.2pl.216163724?3724?第曲的从仃13.1

18、4.13.14.0M0M1J.1J.1J.1J.14.14.1.7112.431.7112.431.7912.141.7912.142.3612.671 1.951.9512,5?2,5?W W|3B7l.7B|3B7l.7B|2.d51.0712.d51.0713.452.1513.452.1513.61l.M123.61l.M12.17.17SIT|,S|U7.9|2.813.16.28we.DRrbfi|2.7610.26JGLDI2.S3.2410.2113.DI3.BSI3.4910.24HDD|2E)|LS|0.M113D|3.Z7|3喇机3*90,9|25|J-52|0.JULa

19、I2e|J.51|e.3tVJ.VJ.0|3.BU.OS0.20叫112.2915.M|L94i.2Bi4aei，g2.9115.6SIL.Qr2.1617,0|0,95l.EDK32|1.0Al.V7|fi.75|1.051.58|5.35|1.031.25|5.96|1.001.965.211.081 114|COIE|工 用5内|1%2工1,口 工 .4次 . lC5O+0lll3,2J,7e.2.U,L+.lU0S.GI(13,Uf2.31r2,S7V.I1460,01114.37.1,95.2.5,1.I735dlim:比2,强.工仃一jlJ5).n|14.2.17fi,1芈1.|1

20、2M*i(14.33.J,ff7,2.|12Q5.a|：li.05,2.15,2.1,.IlC4S,0HU.&3Jl.Mr2.17f.|必更55%心731川10.19bJ2521lfal/(H.L5M791309B4215.1.4773000031713J51.394936427767112,25341490679220.BJIn45S!BJIn45S!都安界8585x:sqrt(linalg.Vectors.squared_distance(linalg.Vectors.dense(list(x.asDict()center),linalg.Vectors.

21、dense(list(x.asDict()cols_st).sum()/wine_Vc_St_clusters_centers.count()WMSlfc-wnw_7c_5t_clu9ters_centers.wltccenter*-1lcaU_at.rdd.iap(|lbdaKrSqrtnuialgnVMtQrarSquardjdistijricedinalg.VtoFSHden5c(lL5tXHa5Dict(H-centcruIJm虱9,丫箕如1评神*1心1以+口sOict：!.*cols_st-lH11.smI)wine_Vc_St_clu5Jter5_centers.cointlAAA

22、|第六步：输出模型效果(1).打印结果print(误差平方和=+str(WSSSE)intr 误若平方和=+strWSSSE)误爰 u 方和=2.921651410515S335aI第七步：可以通过以下命令执行python文件,查看最终结果spark-submit/opt/algorithm/kmeans/K-means.py【问题与回答】1、Q:K值怎么确定？A：对于K值的确定，是Kmean的一个最大缺点，往往需要经验判断，统计学上，遍历k1,样本量，step,刻画不同k的类样本与质心平均距离曲线确定k的取值。2013-M-2517:30:13IkFO2819-04-2517:30:13IN

23、FOriver(1/1EJCEIZLIIQIF154qFintshiedtdkGflin31D(T2D429)_1739bytesresultw-tlw-tliodriverT*k5*tNiiPdgeir：54-Finishi0.0inn胖第.0(TI042glin39monlothast(executordlWHB必？517:30:13INFO2019-04-2517:36:13INFO201S-a4-3517:36:13IWO唉兰干工和:2.525fl248BB642175TakScliPclLlerTii3l：54，RmovedT.skSei:31.0AwhoptakshaveallromipletedsftSSiheduler：54-ResultStage31(cDuntatNativEthcdAccessorImpI.java：Q：DJKJSihffduler;S4-Jo-b51finished:onuritfinished:onuritata