精通Spark集群搭建与测试

1、精通Spark集群搭建与测试电脑配置最好i5+内存最少8G.安装 VMware Workstation 软件推荐官方下载最新版，下载地址： HYPERLINK s:/my.vmware /cn/web/vmware/details?downloadGroup=WKST-1210-WIN&productld=52 s:/my.vmware /cn/web/vmware/details?downloadGroup=WKST-1210-WIN&productld=524&rPld=9763 VMware WorkstationFile Edit View VM Tabs Helpd 口口2.运行VM

2、ware Workstation,新建3台虚拟机，并安装Ubuntu操作系统Ubuntu 下载土也址: HYPERLINK :/ /download/alterTiative-downloads :/ /download/alterTiative-downloads我用的 ubuntu-14.04.5-desktop-amd64.iso需要配置虚拟机使之能够上网，在这里我们采用网络地址转换即NAT的方式，与宿主机共 享IP上网:注1：可以先装好一台机器，然后通过VMware的克隆功能生成另外两台。注2：安装完系统后，为了能从宿主机与虚拟机互相COPY文件，也为了能使虚拟机全屏 显示，推荐安装V

3、MwareTools,方法如下：tar -xzvf VMwareTools-9.6.0-1294478.tar.gzcd vmware-tools-distrib/c.sudo 7vmware-install.pld.然后一路回车即可e.由于版本不同操作可能不同，百度Ubuntu安装Tools即可3.为了简化后续操作中的权限问题，我们在这里配置为root账户登录系统，方法如下：a.终端进入root用户权限模式：命令sudo-sb.命令 gedit /etc/lightdm/lightdm.confc.最后一行添加：greeter-show-manual-login=trueallow-gues

4、t=falsehadoop.tmp.dir/usr/local/hadoop/hadoop-2.6.4/tmpA base for other temporary directorieshadoop.native.IibtrueShould native hadoop libraries, if present, be used.h.修改9皿11(55呢3(011,以下是最小配置，更详细的信息可以参考官网： :/hadoopapacheorg/docs/stable/hadoopproiectdist/hadoophdfs/hdfsdefa ult.xmlgedit hdfs-site.xml

5、:dfs.replication2.dir/usr/local/hadoop/hadoop-2.6.4/dfs/namedfs.datanode.data.dir/usr/local/hadoop/hadoop-2.6.4/dfs/data注：这里指定的 .dir 与 dfs.datanode.data.dir 假设不存在的话， 后续start-dfs时会报错:i.修改gedit mapre&site.xml,以下是最小配置，更详细的信息可以参考官网: : hadoop. apache. orq/docs/stable/hadoop-proiectdist/hadoop-hdfs/hdf s-

6、default.xml注：MRv1的Hadoop没有使用yam作为资源管理器，其配置如下:gedit mapred-site.xml : (without yarn) mapred.job.trackermaster:9001 *MRv2的hadoop使用yarn作为资源管理器,其配置如下:vim mapred-site.xml : (with yarn)yarnj.修改yarnsite.xml,以下是最小配置，更详细的信息可以参考官网: HYPERLINK :/docs/stable/hadoop-yarr/hadoop-yanvcommor/yarivd :/docs/stable/had

7、oop-yarr/hadoop-yanvcommor/yarivd efault, xmlgedit yarn-site.xml:yarn.resourcemanager.hostnamemasteryarn.nodemanager.aux-servicesmapreduce_shuffle注：Yarn是Hadoop推出整个分布式(大数据)集群的资源管理器，负责资源 的管理和分配，基于Yarn我们可以在同一个大数据集群上同时运行多个计算框架，例如 Spark、MapReduce、Stormo12.启动并验证hadoop集群：a. 格式化 hdfs 文件系统：hadoop namenode -f

8、ormat/hdfs namenode -formatrootmaster:/usr/local/hadoop/hadoop-2.6.0/bin# hadoop namenode -format DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it.16/03/03 14:38:15 INFO namenode.NameNode: STARTUP_MSG:STARTUP_MSG: Starting NameNodeSTARTUP_MSG:

9、host = master/30STARTUP_MSG:args = -formatSTARTUP_MSG:version = 2.6.Q该命令会启动，格式化，然后关闭namenode。实际上格式化后，在namenode上会生成以下文件:rootmaster:/usr/local/hadoop/hadoop-2.6.0/dfs/name/current# Is fsimage_O0O00O0000000000000 seen_txidfsimage_O0OOOOOOOOO00OOO0OO.md5 VERSIONrootmaster:/usr/local/hadoop/hadoop-2.6.O/

10、dfs/name/current#其中VERSION文件的内容如下:rootmaster:/usr/local/hadoop/hadoop-2.6.0/dfs/name# Is current rootmaster:/usr/local/hadoop/hadoop-2.6.0/dfs/name# cd current/ rootmaster:/usr/local/hadoop/hadoop-2.6.0/dfs/name/current# Is fsimage_00O0000O0O0000O0000 seen_txid fstmage_O000000O00000O00000 . md5 VERS

11、IONnamespaceID=1103891 clusterID=CID-69035837-rootmaster:/usr/local/hadoop/hadoop-2.6.0/dfs/name/current# more VERSION #Thu Mar 03 16:54:31 CST 2016.029a-45a3-b0b3-ld662751eb43cTime=0storageType=NAME NO blockpoolID=BP-996551254-192.168 85.130-1456995271763layoutVersion=-60rootmaster:/usr/local/hadoo

12、p/hadoop-2.6.0/dfs/name/current# ，该命令不会在datanode的dfs.datanode.data.dir对应的目录下生成任何文件:rootgworkerl:/usr/local/hadoop/hadoop-2.6.O/dfs/data# Is rootgworkerl:/usr/local/hadoop/hadoop-2.6.6/dfs/data# |有关该命令的细节请参考官方文档： : hadoop. aD/docs/stable/hadoop-Droject-dist/hadooD-hdfs/HDFSCo mmands.html#namenodeb.启动

13、 hdfs: start-dfs.shrootmaster:/usr/local/hadoop/hadoop-2.6.0/dfs# start-dfs.sh16/03/03 16:57:43 WARN util.NativeCodeLoader: Unable to load nattve-hadoop libra ry for your platform. using butltin-java classes where applicableStarting namenodes on mastermaster: starting namenode, logging to /usr/local

14、/hadoop/hadoop-26.0/logs/hadoop root-namenode-master.outworkerl: starting datanode, logging to /usr/local/hadoop/hadoop-2.6.0/logs/hadoo p-root-datanode-worker1.outworker2: starting datanode, logging to /usr/local/hadoop/hadoop-2.6.0/logs/hadoo p-root-datanode-worker2.outworker3: starting datanode,

15、logging to /usr/local/hadoop/hadoop-2.6.0/logs/hadoo p- root-datanode-worker3.outStarting secondary namenodes mastermaster: starting secondarynamenode, logging to /usr/local/hadoop/hadoop-2.6.0/10 gs/hadoop-root-secondarynamenode-master.out16/03/03 16:57:58 WARN util.NativeCodeLoader: Unable to load

16、 nattve-hadoop libra ry for your platform. using butltin-java classes where applicable rootmaster:/usr/local/hadoop/hadoop-2.6.0/dfs# |使用jps验证HDFS是否启动成功:rootmaster:/usr/local/hadoop/hadoop-2e6.O/bin# jps 3600 NameNode3926 Jps3815 SecondaryNameNode_通过webui检查HDFS是否启动成功 : master:50070 r.r-lh.H v YW a m

17、aster:50070/dfshealth.html#tab-overviewHadoop Overview Datanodes Snapshot Startup Progress UtilitiesOverview，maste匚900(y (active/LStarted:Thu Mar 03 16:57:44 CST 2016Version:2.6.0, re3496499ecb8d220fba99dc5ed4c99c8f9e33bblCompiled:2014-ll-13T21:10Z by jenkins from (detached from e349649)Cluster ID:C

18、ID-69035837-029a-45a3-b0b3-ld662751eb43Block Pool ID:BP-996551254-30-1456995271763Namenode information x6 master:50070/dfshealth.html#tab-overview CI Q SearchDFS Used:72 KBNon DFS Used:18.82 GBDFS Remaining:33.96 GBDFS Used%:0%DFS Remaining%:64.35%Block Pool Used:72 KBBlock Pool Used%:0%DataNodes us

19、ages% (Min/Median/Max/stdDev):0.00% / 0.00% / 0.00% / 0.00%Live NodesDead Nodes0 (Decommissioned： 0)3 (Decommissioned: 0)Decommissioning NodesNumber of Under-Replicated BlocksNumber of Blocks Pending Deletion注1：实际上第一次启动hdfs后，在datanode dfs.datanode.data.dir对应的目录下会生成current目录，该目录下的BP文件与namenode上.dir对应

20、的目录下的current子目录的VERSION文件中的 blockpoollD字段的值一致；在该目录下也会生成VERSION文件，该VERSION文 件中的 clusterlD 和 namenode 的 .dir 对应的目录下的 current 子目录的VERSION文件中的clusterlD 一致： O root0)worker3: /usr/local/hadoop/hadoop-2.6.0/dFs/data/currentrootworker3:/usr/local/hadoop/hadoop-2.6e0/dfs/data# Iscurrent in_uselockrootworker3

21、:/usr/local/hadoop/hadoop-2.6.0/dfs/data# cd current/rootworkefftjsr/*local/hadoop/hadoop-2.6.0/dfs/data/current# IsBP-996551254-30-1456995271763 VERSIONrootworker3:/usr/local/hadoop/hadoop-2.6.0/dfs/data/current# more VERSION#Thu Mar 03 16:57:50 CST 2016storageID=DS-773e81f4-39f9-4a20-9f36-b48952d0

22、6848clusterID=CID-69035837-029a-45a3-b0b3-ld662751eb43cTime=0datanodeUuid=db5rt1b7-6592-46ff-af4e-c99a0ee75b80storageType=DATA_NODElayoutVersion=-56rootworker3:/usr/local/hadoop/hadoop-2.6.0/dfs/data/current#实际上在后续执行了 hdfs namenode -format后，namenode的VERSION文件会 改变：000 root)masten /usr/local/hadoop/ha

23、doop-2.6.0/dfs/name/currentnamespaceID=2001999531clusterID=CID-d216d552-e79e-4d9c-8c6d-f9b412205090cTime=0storageType=NAME_NODEblockpoolID=BP-148499760630-1457136293776layoutVersion=-60而dananode的BP和VERSION文件都不会改变:rootujorker2:/usr/local/hadoop/hadoop-2.6.O/dfs/data/current# Is551254-192.1四二85130-145

24、6935271763 VERSIONrootuorker2:/usr/BatteadooiiZboop-Z,6, O/dfs/data/current# more VERSION#Fri Mar 04 19:03:10 EST 2ulb一_ _storageID=DS-a9f0dfd3-cdc0-4810-ab49-49579blee3b2clusterID=CID-69035837-029a-45a3-b0b3-ld662751eb43cTime=0datanodeUuid=f005a5B*e346fe-94fa-8061c8ac0fb0storageType=DATA_NODElayout

25、Version=-56rootiuorker2：/usr/local/hadoop/hadoop-2.6.O/dfs/data/current#再次start-dfs.sh时，namenode可以成功启动，但在datanode上，因为version 文件与namenode的不一致，datanode不能成功启动并成功注册到namenode! 所以：每次执行hdfs namenode -format前,必须清空datanode的data文件夹! （namenode的name文件夹不需要清空，namenode和datanode的tmp文件夹也 不需要空。）注2：注:有的朋友喜欢使用start-all

26、.sh,其实质是执行了 start-dfs.sh和start-yarn.sh, 如下列图可见，在提示中也可见，推荐分开使用start-dfs.sh和start-yarn.sh而不是 直接使用start-alLsh:# Start all hadoop daemons. Run this on master node.echo * T N s scr Is Depr(u ：dbtn= dtrname M$BASH_SOURCE-$Obtn= cd Sbin,; pwdDEFAULT_LIBEXEC_DIR=H$btn，7./ItbexecHADOOP_LIBEXEJD1R=$HADOOP_LIB

27、EXEJD1R：-$DEFAULT_L1BEXEC_DIR $HADOOP_LIBEXEC_DIR/hadoop-config.shstart hdfs daemons If hdfs is presentif -f 1 HADOOP HDFS HOME H /sbin/start-dfs.sh ; thenSfHADOOP_HDFS_HOMEjVsbin/start-dfs.sh -config $HADOOP_CONF_DIRflstart yarn daemons if yarn is presentif -f HADOOP YARN HOME/sbin/start-yarn.sh ;

28、thenStHADOOP-YARN-HOMEjVsbtn/start-yarn.sh -config $HADOOP_CONF_DIR代一一.38,1Botc.启动 yarn: start-yarn.shrootmaster:/usr/local/hadoop/hadoop-2.6.0/dfs/name/current# start-yarn.sh starting yarn daemonsstarting resourcemanager, logging to /usr/local/hadoop/hadoop-2.6.0/logs/yarn-roo t-resourcemanager-mas

29、ter.outworker3: starting nodemanager, logging to /usr/local/hadoop/hadoop-2.6.0/logs/yar n-root-nodemanager-worker3.outworker2: starting nodemanager, logging to /usr/local/hadoop/hadoop-2.6.0/logs/yar n-root- nodemanager-worker2.outworked: starting nodemanager, logging to /usr/local/hadoop/hadoop-2.

30、6.0/logs/yar n-root- nodemanager-workerl.outrootmaster:/usr/local/hadoop/hadoop-2.6.0/dfs/name/current# |使用jps验证yarn是否启动成功:rootmaster:/usr/local/hadoop/hadoop-2.6.0/dfs/name/current# jps9480 ResourceManagerr8908 NameNode9116 SecondaryNameNode9743 JpsrootQworkerl : -S jps通过webui检查yarn是否启动成功: :master:

31、8088/ : workerl:8042/Applicationsa master:8088/cluster ? Q Searcha master:8088/cluster 6.0/dfs/name/current# jps9878 JobHistoryServer9480 ResourceManager_8908 NameNode9116 SecondaryNameNode9948 JpsootQnaste：/us八ocal/hadoop/hadoop-2.6.e/dfs/namR/cuent# ，通过webui检查JobHistory Server是否启动成功: HYPERLINK :/m

32、aster:19888 :/master:19888也割:/亩可享J，JobHistory ApplicationAbout JobsRetired Jobs ToolsShow 20 entriesSearch:Submit Time 人Start Time 人 7Finish TimeJob ID人 YName人 VUser人 VQueue人 VState人 YMaps Total人 YMaps Completed 人 yNo data available in tableSubmitStartFinishJotNameUserQueueStateMapsMaps CompShowing

33、0 to 0 of 0 entriesJobHistory G | | Q Search华 净 master: 19888/jobhistorye.验证hadoop集群创立文件夹：hdfs dfs -mkdir -p /data/wordcounthdfs dfs -mkdir -p /output上传文件：hdfs dfs -put /usr/local/hadoop/hadoop-2.6.0/etc/hadoop/*.xml /data/wordcount查看上传文件是否成功：hdfs dfs -Is /data/wordcount保存d. 为root账号 设置密码：sudo passwd

34、 roote.重新启动系统后，即可用root账号登录：reboot注1：如果系统提示vim没有安装的话，可以通过apt-get install vim安装。注2：切换为root账户登录后，如果遇到以下问题：O Error found when loading /file：stdin: is not a ttyAs a result the session will not be configured correctly.Ydu should fix the problem as soon as feasible.OK.方法一：将/root/, profile 文件中的 mesg n替换成 tt

35、y -s & mesg n重启方法二：将非root账户目录中的.profile复制到/root/：例如：cp例ome/非root账户的名字/.profile /root/.重启在各个节点修改节点名称，并配置ip地址和hostname的对应关系:rootmaster:# hdfs dfs -Is /data/wordcount16/03/05 08:36:28 WARN util.NativeCodeLoader: Unable to load native-hadoop libra ry for your platform. using builtin-java classes where a

36、pplicableFound 10 items -rw-r-r-2 root(ity-scheduler.xml spark.deploy.history.Hi storyServer-1-master.out rootgmaster:/usr/local/spark/spark-1.6.0-bin-hadoop2.6/conf# jps 5826 NameNode 7107 Master 7636 Jps6046 SecondaryNameNoderootmaster:/usr/local/spark/spark-1.6.0-bin-hadoop2.6/conf#Jps可见，historys

37、erver没能启动成功。仔细观察日志可见，指定的日志目录不存在:rootmaster:/usr/local/spark/spark-l6.0-bin-hadoop2.6/conf# more /usr/local/spa rk/spark-1.6.0-bin-hadoop2.6/logs/spark-root-orgapache.spark.depl.oy.history.His toryServer-1-master.outSpark Command: /usr/lib/java/jdkl.8.0_60/bin/java -cp /usr/local/spark/spark-1.6 .0-b

38、in-hadoop2.6/conf/:/usr/local/spark/spark-1.6.0-bin-hadoop2.6/lib/spark-asse mbly-1.6.0-hadoop2.6.0.jar:/usr/local/spark/spark-1.6.0-bin-hadoop2.6/lib/datanu cleus-api-jdo-3.2.6.jar:/usr/local/spark/spark-1.6.0-bin-hadoop2.6/lib/datanucle us-core-3.2.10.jar:/usr/local/spark/spark-1.6.0-bin-hadocpZad

39、/ltb/datanucleus-rd bms-3.2.9.jar:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop/ -Xmslg -Xmxlg org.apach e.spark.deploy.history.HtstoryServer 16/03/05 16:47:15 INFO history.HistoryServer: Registered signal handlers for TE RM, HUP, INT16/03/05 16:47:15 WARN util.NativeCodeLoader: Unable to load native-ha

40、doop libra ry for your platform. using builtin-java classes where applicable16/03/05 16:47:15 INFO spark.SecurityManager: Changing view acls to: root 16/03/05 16:47:15 INFO spark.SecurityManager: Changing modify acts to: root 16/03/05 16:47:15 INFO spark.SecurityManager: SecurityManager: authenticat

41、ion di sabled; ui acls disabled; users with view permissions: Set(root); users with mod ify permissions: Set(root)Exception in thread “matn” java.lang.reflect.InvocationTargetExceptionat sun.reflect.NativeconstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeconstructorAccessorImp

42、l.newInstance(NativeConstruct orAccessorImpl. java:62)at sun.reflect.DelegattngConstructorAccessorImpl.newInstance(DelegattngC onstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newlnstance(Constructor.java:422)at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.sea la:

43、235)at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.seal a) TOC o 1-5 h z Caused by: java.lang.IllegalArgumentException: Log directory specified does not exist: hdfs:/master:9000/htstoryserverforspark.卜at org.apache.spark.deploy.history.FsHtstoryProvidorg$apache$spark$de ploy$his

44、tory$FsHistoryProvider$startPolling(FsHistoryProvi.scala:168)at org.apache.spark.deploy.history.FsHistoryProviderAQitialize(FsHistor yProvider.scala:120)at org.apache.spark.deploy.history.FsHtstoryProvider.(FsHistoryPro vtder.scala:116)at org.apache.spark.deploy.history.FsHistoryProvider.tntt(FsHtst

45、oryPro vider.scala:49)6 morerootmaster:/usr/local/spark/spark-1.6.0-bin-hadoop2.6/conf# |该目录是我们在spark-defaults.conf中指定的（事实上，假设不指定spark.eventLog.dir 和spark.history.fs.logDirectory,你也无法启动historyserver,因为系统不知道去哪里 存储你的history信息）contributor license agreements. See the NOTICE file distributed withthis wor

46、k for additional information regarding copyright ownership*The ASF licenses this file to You under the Apache License, Version 2.0(the License); you may not use this file except in compliance withthe License. You may obtain a copy of the License at # HYPERLINK :/ /licenses/LICENSE-2.0 :/ /licenses/L

47、ICENSE-2.0#Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an AS IS BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under th

48、e License. #Default system properties included when running spark-submit.This is useful for setting default environmental settings.Example:spark.masterspark:/master:7077spark.eventLog.enabledtruespark.eventLog.dirhdfs:/master:9000/historyserverforsparkspark.yarriehistoryserver.address Master: 1808。s

49、park.history . fs .logDirectorysparkeSerializer=org.apache.spark.serializer.KryoSeriaIizerspark.driver.memory5g3 spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers=none two three*31,1Bot我们只需使用hdfs命令hdfs dfs -mkdir -p /historyserverforspark来创立该目录，再启 动 historyserver, jps 可见多了 个

50、historyserver 进程：rootmaster:# hdfs dfs -mkdir p /historyserverforspark16/03/05 16:59:22 WARN util.NativeCodeLoader: Unable to load native-hadoop libra ry for your platform. using builtin-java classes where applicable rootmaster:#网络连接更改这3处地方，各虚拟机之间不能相同a. gedit /etc/hostname 将 4 台节点分别命名为 master,worker

51、1 ,worker2b.保存，重启虚拟机使更改生效c.验证hostname更改是否生效：用hostname命令 C Q Search“ master: 50070/explorer.html#/Browse DirectoryGo!Go!PermissionOwnerGroupdrwxr-xr-xrootsupergroupdrwxr-xr-xrootsupergroupdrwxr-xr-xrootsupergroupdrwxrwx-rootsupergrouproot)masten 0 B 00 B 00 B 00 B 0Size Replication Block Size Namedat

52、a0 B58267107784979006046rootmaster:# start-history-server.sh starting org.apache.spark.deploy.history.Historyserver, logging to /usr/local/sp| ark/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.history.Hi storyServer-1-master.out rootmaster:# jpsNameNodeMasterHtstoryServerJpsSecon

53、daryNameNode rootmaster:#f.使用 webUI 来查看 historyserver 是否成功启动 :master:8080Browsing HDFS xSpark Master at. xT Problem load. xWmaster: 18080/?page=1 &showlncomplete=fal C Q Search,History Server自 DHistory ServerEvent log directory: hdfs:/master:9000/historyserverforsparkNo completed applications found!

54、Did you specify the correct logging directory? Please verify your setting of spark.history.fsJogDirectory and whether you have the permissions to access it.It is also possible that your application did not run to completion or did not stop the SparkContext.Show incomplete applications注 1 ： start-all

55、.sh 其实质是先后执行了 start-master.sh 和 start-slaves.sh:Start all spark daemons.)Starts the master on this node.Starts a worker on each node specified in conf/slavesif -z ”$SPARK_HOME” ; thenexport SPARK_HOME=$(cd H dtrname $0 / pwd) ft一TACHYON_STR= |while ( $# ); do lease $1 in -with-tachyon) TACHYON_STR=/

56、start-masterash 二一 F /start-slaves.sh 5TACHYON_STR 9 9 esac shift done # Load the Spark configuration . $SPARK_HOME) # Start Master $SPARK_HOME # Start Workers $SPARK_HOME最正确实践是：在master节点上执行start-all.sh.注 2： start-master.sh 会首先尝试从 spark-env.sh 中获取 spark_master_ip ,获取到的话, 就在该参数对应的节点上（可以是当前节点也可以不是当期节点

57、）启动master进程；如果 没有狄取到的话，那么会在当前”点上启动master进程:rootmaster: /usr/local/spark/spark-1.6.0-bin-hadoop2.6/sbinif “$SPARK_MASTER_PORT” =; thenSPARK_MASTER_PORT=fiif SSPARK_MASTER_IP =; thenSPARK MASTER IP=hostnameftif ,$SPARK-MASTER_WEBUI_PORT =; thenSPARK-MASTER-WEBUI-PORTs 086fi: 卜_卜 1E/sparkdaemon.sh star

58、t ：CL-5S if ”$START_TACHYON” =; then*$ 5apK HOfIE /tachyon/bin/tachyon bootstrap-conf $ FK 卜二 EP_IF$SPARK HOZE八achyon/btn八achyon format -sHE E： /tachyon/bin/tachyon - start. sh master玳81,1Bot最正确实践是：在master节点上执行start-master.sh.注3： start-slaves.sh会在slaves文件中指定的每个节点上分别调用start-slave.sh来启动 worker进程，并尝试注册

59、到特定的master上。这个master通过以下方式获取：首先尝试 从spark-env.sh中获取spark_master_ip,获取到的话，该参数对应的节点（可以是当前 节点也可以不是当期节点）就是master节点；如果没有获取到的话，那么会视当前节点为 master节点。假设该master节点上的master进程没有启动，这些worker节点上的worker 进程会不断尝试注册到master上：最正确实践是： 在 master节点上start-slaves.sh.注4： start-slave.sh:该命令可以动态启动worker节点并注册到到master这样当我们已经启动了 spark

60、集群后，当后续有新的节点可用时，无需stop整个集群，只需要在新的 可用节点上执行该命令就可以动态启动并注册到master上。需要注意的是，当使用该命令 时，必须在命令行指定maste匚NOTE: This exact class name is matched downstream by SparkSubmit.Any changes need to be reflected there.CLASS=H j.apache.spark.deploy,worker”if $# t= *-help | M$M = *-h ; thenecho Usage: ./sbtn/start-slave.s