ClouderaManager大数据平台部署指南

部署指南ClouderaManagerCDH官方共给出了3中安装方式。第一种方法必需要求全部机器都能连网，由于CMyum1、ClouderaManagerCDH不支持在容器中部署。OracleJDK71.7u80，C5.3.x及以上版本也支持使用oracleJDK8CDHJDK版本必需全都。Python2.4Python3IPv4IPv6disabledtar压缩包的离线安装方式，会在6.0.0中停顿支持该方式。支持selinux，但假设不生疏使用方法，建议关闭。治理端口为7180，此外还有很多根底应用使用的端口，可以运行iptables，需要确认有恰当的授权掌握，例如至少要对同网段的访问放开全部端口。2、ClouderaManager部署的几个阶段和可选择的方式yumPATHC，RPMPATHB3、主机节点信息和根底配置主机信息NameNode1TB/dfs/dn1RAIDRAID掌握器，则直接对每块盘设置一个单独的RAID0，仍旧当作单盘使用。/dfs/dn2，更多的数据盘则按命名规章和挂盘规章扩展即可。RAID性。DataNodeRAID1CDH0cluster-701cluster-712cluster-723cluster-734cluster-745cluster-75CentOS6.9Minimal/etc/hosts文件中。免密登录cluster-70ssh：ssh-keygen将该密钥分发至其它各节点，以实现免密钥登录，如：ssh-copy-id cluster-71selinuxiptables全部节点配置为关闭selinux。全部节点的iptables设置为对同网段地址放开全部端口的访问权限。中即可。cat<<EOF>>/etc/sysctl.confvm.swappiness=0net.ipv6.conf.all.disable_ipv6=1net.ipv6.conf.default.disable_ipv6=1net.ipv4.ip_local_port_range=102465000EOFsysctl-pechonever>/sys/kernel/mm/transparent_huge/defragechonever>/sys/kernel/mm/transparent_huge/enabledcat<<EOF>>/etc/rc.localechonever>/sys/kernel/mm/transparent_huge/defragechonever>/sys/kernel/mm/transparent_huge/enabledEOF4JDK下载并分发OracleJDK安装包至全部节点，执行安装和配置命令。rpm-ivhjdk-7u80-linux-x64.rpm直接刷下面脚本完成配置：cat<<EOF>>/etc/profileJAVA_HOME=/usr/java/jdk1.7.0_80PATH=\$JAVA_HOME/bin:\$PATH:.CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexportJAVA_HOMEexportPATHexportCLASSPATHEOFsource/etc/profile以上JDK配置，需要在每个节点上都处理。5、安装和配置数据库主要有以下业务会使用到数据库效劳：ClouderaManager，存放各种效劳、配置数据，空间大约100MB，很重要，应常常OozieServer-ContainsOozieworkflow,coordinator,andbundledata.Cangrowverylarge.SqoopServer-Containsentitiessuchastheconnector,driver,linksandjobs.Relativelysmall.ActivityMonitor-Containsinformationaboutpastactivities.Inlargeclusters,thisdatabasecangrowlarge.ConfiguringanActivityMonitordatabaseisonlynecessaryifaMapReduceserviceisdeployed.Reports Manager - Tracks disk utilization and processing activities over time.Medium-sized.HiveMetastoreServer-ContainsHivemetadata.Relativelysmall.HueServer-Containsuseraccountinformation,jobsubmissions,andHivequeries.Relativelysmall.SentryServer-Containsauthorizationmetadata.Relativelysmall.ClouderaNavigatorAuditServer-Containsauditinginformation.Inlargeclusters,thisdatabasecangrowlarge.ClouderaNavigatorMetadataServer-Containsauthorization,policies,andauditreportmetadata.Relativelysmall.由于预备使用cluster-70作为NameNodeMaster，所以把mysql数据库安cluster-71，以分散下风险。可：yuminstallmysql-servermysqlchkconfigmysqldonservicemysqldstart完成数据库初始化：/usr/bin/mysql_secure_installation优化数据库配置servicemysqldstopInnoDBlog/var/lib/mysql/ib_logfile0and/var/lib/mysql/ib_logfile1outof/var/lib/mysql/参照下面调整f参数，以更适合CDH效劳使用：[mysqld]datadir=/var/lib/mysqlsocket=/var/lib/mysql/mysql.sockuser=mysql#Disablingsymbolic-linksisrecommendedtopreventassortedsecurityriskssymbolic-links=0transaction-isolation=READ-COMMITTEDkey_buffer_size=32Mmax_allowed_packet=32Mthread_stack=256Kthread_cache_size=64query_cache_limit=8Mquery_cache_size=64Mquery_cache_type=1max_connections=550#binlog_format=mixedread_buffer_size=2Mread_rnd_buffer_size=16Msort_buffer_size=8Mjoin_buffer_size=8M#InnoDBsettingsinnodb_file_per_table=1innodb_flush_log_at_trx_commit =2innodb_log_buffer_size=64Minnodb_buffer_pool_size=2Ginnodb_thread_concurrency=8innodb_flush_method=O_DIRECTinnodb_log_file_size=512M[mysqld_safe]log-error=/var/log/mysqld.logpid-file=/var/run/mysqld/mysqld.pidsql_mode=STRICT_ALL_TABLES启动mysql，假设有报错，可依据日志信息排查。创立需要使用的几个库createdatabasedatabaseDEFAULTCHARACTERSETutf8;grantallondatabase.*TO”user”@”%”IDENTIFIEDBY”password”;latin时日志中会报错创立索引时超长度限制。mysqljdbcdriver一下。Serverhost,ActivityMonitor,ReportsManager,HiveMetastoreServer,HueServer,SentryServer,ClouderaNavigatorAuditServer,andClouderaNavigatorMetadataServer从以下网址下载需要使用的驱动包：“://mysql/downloads/connector/j/5.1.html“://mysql/downloads/connector/j/5.1.htmltar-zxvf mysql-connector-java-5.1.43.tar.gzmkdir-p/usr/share/java/cpmysql-connector-java-5.1.43/mysql-connector-java-5.1.43-bin.jar/usr/share/java/mysql-connector-java.jar6NTP的系统时间严格同步。yum-yinstallntpservicentpdrestartchkconfigntpdon节点均配置/etc/ntp.confserver7ClouderaManagerServer软件ClouderaManagerServercluster-70下载和分发安装资源clouderamanagers://archive.cloudera/cm5/cm/5/cdhparcels下载地址：“://archive.cloudera/cdh5/parcels/“://archive.cloudera/cdh5/parcels/wget-cs://archive.cloudera/cm5/cm/5/cloudera-manager-el6-cm5.12.0_x86_64.tar.gzwget-c“://archive.cloudera/cdh5/parcels/5.12/CDH-5.12.0-1.cdh5.12.0“://archive.cloudera/cdh5/parcels/5.12/CDH-5.12.0-1.cdh5.12.0.p0.29-el6.parcelwget-c“://archive.cloudera/cdh5/parcels/5.12/CDH-5.12.0-1.cdh5.1“://archive.cloudera/cdh5/parcels/5.12/CDH-5.12.0-1.cdh5.12.0.p0.29-el6.parcel.sha1wget-c“://archive.cloudera/cdh5/parcels/5.12/manifest.json“://archive.cloudera/cdh5/parcels/5.12/manifest.jsonsha，否则上传到系统中后，CDHshacloudera-manager-el6-cm5.12.0_x86_64.tar.gz文件放到全部节点的/opt/cloudera-manager名目下。mkdir/opt/cloudera-managercd/opttarzxvfcloudera-manager-el6-cm5.12.0_x86_64.tar.gzCMServeruseradd--system--home=/opt/cloudera-manager/cm-5.12.0/run/cloudera-scm-server--no-create-home--shell=/bin/false--comment“ClouderaSCMUser“cloudera-scmCMServercluster-70：mkdir-p/var/lib/cloudera-scm-serverchowncloudera-scm:cloudera-scm/var/lib/cloudera-scm-serverClouderaManagerAgentssed-i”s/server_host=localhost/server_host=cluster-70/”/opt/cloudera-manager/cm-5.12.0/etc/cloudera-scm-agent/config.iniCMServercluster-71cluster-71：JDBCcp mysql-connector-java-5.1.43-bin.jar cm-5.12.0/share/cmf/lib初始化脚本语法如下：scm_prepare_database.shdatabase-type[options]database-nameusernamepassword/opt/cloudera-manager/cm-5.12.0/share/cmf/schema/scm_prepare_database.shmysql-hlocalhost-uroot-pHstest2017scmscmscm屏幕输出信息：JAVA_HOME=/usr/java/jdk1.7.0_80Verifyingthatwecanwriteto/opt/cloudera-manager/cm-5.12.0/etc/cloudera-scm-serverCreatingSCMconfigurationfilein/opt/cloudera-manager/cm-5.12.0/etc/cloudera-scm-serverExecuting: /usr/java/jdk1.7.0_80/bin/java-cp/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera-manager/cm-5.12.0/share/cmf/schema/../lib/*com.cloudera.enterprise.dbutil.DbCommandExecutor/opt/cloudera-manager/cm-5.12.0/etc/cloudera-scm-server/pertiescom.clouderaf.db.[main]DbCommandExecutor INFOSuccessfullyconnectedtodatabase.Alldone,yourSCMdatabaseisconfiguredcorrectly!mysql，增加一条对CMServergrantallonscm.*TO”scm”@”cluster-70”IDENTIFIEDBY”scm”;cluster-70CMServer/opt/cloudera-manager/cm-5.12.0/etc/cloudera-scm-server/perties需要依据实际状况更以下参数。com.clouderaf.db.host=cluster-71com.cloudera=scmcom.clouderaf.db.user=scmcom.clouderaf.db.password=scmcom.clouderaf.db.setupType=EXTERNALCDHParcelcluster-70mkdir-p/opt/cloudera/parcel-repo将以下已经预备好的parcel资源文件放入该名目：CDH-5.12.0-1.cdh5.12.0.p0.29-el6.parcelCDH-5.12.0-1.cdh5.12.0.p0.29-el6.parcel.sha1manifest.jsonchown-Rcloudera-scm:cloudera-scm/opt/cloudera/parcel-repo在其它全部节点上：mkdir-p/opt/cloudera/parcelschowncloudera-scm:cloudera-scm/opt/cloudera/parcelsclouderamanagerserverandagents直接使用root运行启动脚本即可，脚本中已经默认设定为会切换到cloudera-scm在全部节点上，先调整下文件属主权限：chown-Rcloudera-scm:cloudera-scm/opt/cloudera-managercluster-70CMServer：/opt/cloudera-manager/cm-5.12.0/etc/init.d/cloudera-scm-serverstartCMagent：/opt/cloudera-manager/cm-5.12.0/etc/init.d/cloudera-scm-agent tart检查../../log/下面的日志文件，分析错误缘由。8ClouderaManagerCDH访问以下地址：://0:7180/cmf初始治理账号信息为：admin/admin就选免费版，连续。66CDHparcels数配置上的优化建议，遵照处理后重做检验检查即可。”也可以。进入集群角色安排环节，这一步比较关键，留意要把NameNode我们设计的cluster-70节点，SecondaryNameNode安排到cluster-71节点上。留意HDFSDataNodecluster-72到cluster-75这4个cluster-70下一步，进入数据库设置。按实际配置信息填写，然后下一步。DataNode/dfs/dn；。这里要特别留意，假设给数据节点预备了“DataNode/dfs/dn2,盘时，按这样的规章挂载到系统中的。在进入下一步之前有一个细节格外重要，由于我们是使用的独立的数CDHHDFSchown-Rh