电商大数据分析项目部署运维手册

电商大数据分析项目部署运维手册XX科技股份有限公司编制

目录一、文章编写目的 5二、前置准备 52.1配置网络ip 52.2编写相关操作脚本 81.批量执行命令脚本 82.批量重命名脚本 93.批量拷贝脚本 104.cm_migrate.sh 125.format2.sh 126.mountDisk.sh 127.network.sh 138.node.list 149.node.txt 142.3hostname及hosts配置 141.配置每台节点的hostname 142.配置节点ip、hostname映射 152.4禁用SELinux 152.5关闭防火墙 162.6设置swappiness 172.7关闭透明大页面 182.8配置操作系统本地yum源 192.9安装http服务 202.10集群时钟同步 231.所有机器卸载chrony 232.所有机器安装ntp 233.node1机器配置时钟与自己同步 234.集群其它节点，配置找node1机器去同步 235.重启所有机器的ntp服务 246.验证始终同步，在所有节点执行ntpq-p命令，如下使用脚本批量执行 242.11安装MariaDB 24三、ClouderaManager安装 293.1配置本地CM源 293.2安装ClouderaManagerServer 32四、CDH安装 334.1CDH集群安装向导 334.2集群设置安装向导 40五、HBASE安装 445.1准备工作： 445.2安装： 441.下载hbase1.3.2 442.解压 443．配置环境变量~/.base_profile 444．hbase-env.sh配置java_home和ZK 455.hbase-site.xml配置参数 456．配置regionserver 467．scp-rhbase到其他节点 46六、Flume安装 476.1安装 47七、Kafka安装: 477.1安装 47

一、文章编写目的本文档主要描述Cloudera电商大数据分析平台的安装主要分为4个步骤：集群服务器配置，包括安装操作系统、关闭防火墙、同步服务器时钟等，又称为集群安装的前置准备，关于更多前置条件准备也可以参考之前的的文档《CDH安装前置准备》；安装ClouderaManager；通过ClouderaManager的WebUI向导式安装CDH集群；集群完整性检查，包括HDFS文件系统、MapReduce、Hive等是否可以正常运行。本文操作说明基于以下假设：操作系统版本：Redhat7.4CDH和CM版本均为5.16.1采用root对集群进行部署您已经下载CDH和CM的安装包二、前置准备2.1配置网络ip开启虚拟机，修改配置文件输入如下命令ll/etc/sysconfig/network-scripts/ifcfg-*会出现如下内容（ifcfg-ens后面的名字可能不同）-rw.1rootroot6168月1319:25/etc/sysconfig/network-scripts/ifcfg-ens16777736

-rw-r--r--.1rootroot2541月32018/etc/sysconfig/network-scripts/ifcfg-lo输入如下命令vi/etc/sysconfig/network-scripts/ifcfg-ens16777736出现下图界面[root@cm1network-scripts]#viifcfg-eno16777736

#网络类型（通常是Ethernet）

TYPE="Ethernet"

#IP的配置方法[none|static|bootp|dhcp]（引导时不使用协议|静态分配IP|BOOTP协议|DHCP协议）

BOOTPROTO="dhcp"

DEFROUTE="yes"

IPV4_FAILURE_FATAL="no"

IPV6INIT="yes"

IPV6_AUTOCONF="yes"

IPV6_DEFROUTE="yes"

IPV6_FAILURE_FATAL="no"

NAME="eno16777736"

#随机id

UUID="6d21f413-619d-4283-9ba2-e1a724c1c593"

DEVICE="eno16777736"

#系统启动的时候网络接口是否有效（yes/no）

ONBOOT="yes"

PEERDNS="yes"

PEERROUTES="yes"

IPV6_PEERDNS="yes"

IPV6_PEERROUTES="yes"

IPV6_PRIVACY="no"需要修改的地方有：BOOTPROTO=static#dhcp改为static（修改）

ONBOOT=yes#开机启用本配置，一般在最后一行（修改）

IPADDR=01#静态IP（增加）

GATEWAY=#默认网关，虚拟机安装的话，通常是2，也就是VMnet8的网关设置（增加）

NETMASK=#子网掩码（增加）

DNS1=#DNS配置，虚拟机安装的话，DNS就网关就行，多个DNS网址的话再增加（增加）这里注意，因为静态ip地址设置为01，因此默认网关和DNS地址前面部分,即192.168.139必须相同，不然会出现无法ping通的情况重启网络服务使ip生效servicenetworkrestart如果报错,reboot重启虚拟机三台节点配置的ip分别为:31,32,33注意：每台节点克隆后需要删除每台节点/etc/udev/rules.d/70-persistent-net.rules文件，清除mac地址。重启每台节点即可。节点克隆后还可以使用以下方式修改mac地址[root@hadoop101/]#vim/etc/udev/rules.d/70-persistent-net.rules进入如下页面，删除eth0该行；将eth1修改为eth0，同时复制物理ip地址，如图所示修改ip地址[root@hadoop101/]#vim/etc/sysconfig/network-scripts/ifcfg-eth0

（1）把复制的物理ip地址更新

HWADDR=00:0C:2x:6x:0x:xx#MAC地址

（2）修改成你想要的ip

IPADDR=32#IP地址2.2编写相关操作脚本脚本依赖expect程序，需要在执行这些脚本的服务器上安装expect,安装方式：yuminstallexpect1.批量执行命令脚本batch_cmd.sh#!/bin/sh

list_file=$1

cmd=$2

username=root

password=123456

cat$list_file|whilereadline

do

host_ip=`echo$line|awk'{print$1}'`

#username=`echo$line|awk'{print$2}'`

#password=`echo$line|awk'{print$3}'`

./expect_cmd$host_ip$username$password"$cmd"

doneexpect_cmd#!/usr/bin/expect

settimeout10

sethost[lindex$argv0]

setusername[lindex$argv1]

setpassword[lindex$argv2]

setcmd[lindex$argv3]

spawnssh-p22$username@$host"$cmd"

expect{

"(yes/no)?"

{

send"yes\n"

expect"*assword:"{send"$password\n"}

}

"*assword:"

{

send"$password\n"

}

}

expect"100%"

#expecteof

exit02.批量重命名脚本batchrenamecmd.sh#!/bin/sh

list_file=$1

username=root

password=123456

cat$list_file|whilereadline

do

host_ip=`echo$line|awk'{print$1}'`

host_name=`echo$line|awk'{print$2}'`

host_alias_name=`echo$line|awk'{print$3}'`

#username=`echo$line|awk'{print$2}'`

#password=`echo$line|awk'{print$3}'`

./expect_rename_cmd$host_ip$username$password$host_name

doneexpectrenamecmd#!/usr/bin/expect

settimeout10

sethost[lindex$argv0]

setusername[lindex$argv1]

setpassword[lindex$argv2]

sethostname[lindex$argv3]

spawnssh-p22$username@$host"sed-is/^HOST.*/HOSTNAME=$hostname//etc/sysconfig/network;hostname$hostname;hostnamectlset-hostname$hostname"

expect{

"(yes/no)?"

{

send"yes\n"

expect"*assword:"{send"$password\n"}

}

"*assword:"

{

send"$password\n"

}

}

expect"100%"

#expecteof

exit03.批量拷贝脚本batch_scp.sh#!/bin/sh

list_file=$1

src_file=$2

dest_file=$3

username=root

password=123456

cat$list_file|whilereadline

do

host_ip=`echo$line|awk'{print$1}'`

#username=`echo$line|awk'{print$2}'`

#password=`echo$line|awk'{print$3}'`

./expect_scp$host_ip$username$password$src_file$dest_file

doneexpect_scp#!/usr/bin/expect

settimeout10

sethost[lindex$argv0]

setusername[lindex$argv1]

setpassword[lindex$argv2]

setsrc_file[lindex$argv3]

setdest_file[lindex$argv4]

spawnscp-r$src_file$username@$host:$dest_file

expect{

"(yes/no)?"

{

send"yes\n"

expect"*assword:"{send"$password\n"}

}

"*assword:"

{

send"$password\n"

}

}

expect"100%"

expecteof4.cm_migrate.sh#!/bin/bash

#updateCMIP

if[$#!=1];then

echo"USAGE:$0<CM_IP>"

echo"e.g.:$0"

exit1;

fi

shbatch_cmd.shnode.list"sed-is/^server_host=.*/server_host=$1//etc/cloudera-scm-agent/config.ini"

#updatemysql

echo"updateCMHOSTID"

host_id=`mysql-uroot-p123456--execute="usecm;selectHOST_IDfromHOSTSwhereIP_ADDRESS='$1';"`

host_id2=$(echo${host_id}|awk-F'''{print$2}')

#echo${host_id}

#echo${host_id2}

mysql-uroot-p123456--execute="usecm;updateROLESsetHOST_ID=${host_id2}whereNAMElike'mgmt-%';"5.format2.sh#!/bin/sh

foriin{b..m}

do

parted-s/dev/sd$imklabelgpt

parted-s/dev/sd$imkpartprimary2048s100%

/usr/sbin/mkfs.xfs-f/dev/sd${i}1

done6.mountDisk.sh#!/bin/sh

#backup/etc/fstab

cp/etc/fstab/etc/fstab.bak

PARTITION_LIST="sdb1sdc1sdd1sde1sdf1sdg1sdh1sdi1sdj1sdk1sdl1sdm1"

i=1

forPARTITIONin$PARTITION_LIST

do

UUID=`blkid"/dev/""$PARTITION"|awk'{print$2}'|sed's/\"//g'`

echo$UUID

echo"add$PARTITIONto/etc/fstab"

MOUNTDIR="/data""$i"

i=$((i+1))

echo"mkdir-p$MOUNTDIR"

mkdir-p$MOUNTDIR

echo"appending\"$UUID$MOUNTDIRxfsdefaults00\"to/etc/fstab"

echo"$UUID$MOUNTDIRxfsdefaults00">>/etc/fstab

echo""

done

#mountallpartitions

mount-a

#showmountedpartitions

df-work.shvirshnet-destroydefault

virshnet-undefinedefault

systemctlrestartlibvirtd.service

yumremovelibvirt8.node.list31

32

339.node.txt31 node1

32 node2

33 node32.3hostname及hosts配置集群中各个节点之间能互相通信使用静态IP地址。IP地址和主机名通过/etc/hosts配置，主机名通过/etc/hostname进行配置。1.配置每台节点的hostname/etc/hostname文件如下：node1或者可以通过命令修改立即生效[root@ip-172-31-13-38~]$hostnamectlset-hostnamenode1可以使用脚本执行[root@localhostbin]#shbatch_rename_cmd.shnode.txt

spawnssh-p22root@31sed-is/^HOST.*/HOSTNAME=//etc/sysconfig/network;hostname;hostnamectlset-hostname

root@31'spassword:

spawnssh-p22root@32sed-is/^HOST.*/HOSTNAME=//etc/sysconfig/network;hostname;hostnamectlset-hostname

root@32'spassword:

spawnssh-p22root@33sed-is/^HOST.*/HOSTNAME=//etc/sysconfig/network;hostname;hostnamectlset-hostname

root@33'spassword:2.配置节点ip、hostname映射在每台节点/etc/hosts文件后追加如下内容，配置ip与hostname之间的映射31 node1

32 node2

33 node3可以将node.txt文件内容追加到/etc/hostscatnode.txt>>/etc/hosts脚本执行shbatch_scp.shnode.list/etc/hosts/etc/2.4禁用SELinuxSELinux就是Security-EnhancedLinux的简称，安全加强的linux。传统的linux权限是对文件和目录的owner,group和other的rwx进行控制，而SELinux采用的是委任式访问控制，也就是控制一个进程对具体文件系统上面的文件和目录的访问，SELinux规定了很多的规则，来决定哪个进程可以访问哪些文件和目录。虽然SELinux很好用，但是在多数情况我们还是将其关闭，因为在不了解其机制的情况下使用SELinux会导致软件安装或者应用部署失败。在所有节点执行setenforce0命令shbatch_cmd.shnode.list"setenforce0"集群所有节点修改/etc/selinux/config文件如下(重启才能生效,所有需要配置上面的临时关闭)：SELINUX=disabled

SELINUXTYPE=targeted

shbatch_cmd.shnode.list"sed-is/^SELINUX=.*/SELINUX=disabled//etc/selinux/config"批量拷贝到其他节点shbatch_scp.shnode.list/etc/selinux/config/etc/selinux/检查所有机器是否已修改成功：shbatch_cmd.shnode.list"cat/etc/selinux/config"|grepSELINUX2.5关闭防火墙集群所有节点执行systemctlstopfirewalld命令shbatch_cmd.shnode.list"systemctlstopfirewalld"

shbatch_cmd.shnode.list"systemctldisablefirewalld"

shbatch_cmd.shnode.list"systemctlstatusfirewalld"|grepActive2.6设置swappiness所有节点执行shbatch_cmd.shnode.list"sysctlvm.swappiness=1"

shbatch_cmd.shnode.list"echo1>/proc/sys/vm/swappiness"

shbatch_cmd.shnode.list"cat/proc/sys/vm/swappiness"2.7关闭透明大页面在管理节点使用脚本执行：shbatch_cmd.shnode.list"echonever>/sys/kernel/mm/transparent_hugepage/defrag"

shbatch_cmd.shnode.list"cat/sys/kernel/mm/transparent_hugepage/defrag"

shbatch_cmd.shnode.list"echonever>/sys/kernel/mm/transparent_hugepage/enabled"

shbatch_cmd.shnode.list"cat/sys/kernel/mm/transparent_hugepage/enabled"设置开机自关闭将如下脚本添加到/etc/rc.d/rc.local文件中iftest-f/sys/kernel/mm/transparent_hugepage/enabled;then

echonever>/sys/kernel/mm/transparent_hugepage/enabled

fi

iftest-f/sys/kernel/mm/transparent_hugepage/defrag;then

echonever>/sys/kernel/mm/transparent_hugepage/defrag

fi也可直接使用脚本shbatch_cmd.shnode.txt'echo"iftest-f/sys/kernel/mm/transparent_hugepage/enabled;then">>/etc/rc.d/rc.local;echo"echonever>/sys/kernel/mm/transparent_hugepage/enabled">>/etc/rc.d/rc.local;echo"fi">>/etc/rc.d/rc.local'

shbatch_cmd.shnode.txt'echo"iftest-f/sys/kernel/mm/transparent_hugepage/defrag;then">>/etc/rc.d/rc.local;echo"echonever>/sys/kernel/mm/transparent_hugepage/defrag">>/etc/rc.d/rc.local;echo"fi">>/etc/rc.d/rc.local'同步到所有节点#赋予rc.local脚本可执行权限

cat/etc/rc.d/rc.local

shbatch_scp.shnode.list/etc/rc.d/rc.local/etc/rc.d/

shbatch_cmd.shnode.list"chmod+x/etc/rc.d/rc.local"

shbatch_cmd.shnode.list"ls-l/etc/rc.d|greprc.local"2.8配置操作系统本地yum源挂载操作系统iso文件mkdir-p/media/DVD1

mount-oloopCentOS-7-x86_64-DVD-1511.iso/media/DVD1/

df-h配置操作系统repo[root@node1~]$sudovi/etc/yum.repos.d/local_os.repo

[local_iso]

name=CentOS-$releasever-Media

baseurl=file:///media/DVD1

gpgcheck=0

enabled=1

[root@node1~]$sudoyumrepolist以上内容中，baseurl是指Yum源的地址，该路径下有个repodata目录，就是yum安装源目录。file://表示yum源为文件。如果只想让local.repo生效，可以在每台节点的/etc/yum.repos.d目录下创建一个repobak目录，将其他的以“.repo”结尾的文件都移动到repobak目录中。shbatch_cmd.shnode.list"mkdir-p/etc/yum.repos.d/repo_bak"

shbatch_cmd.shnode.list"mv/etc/yum.repos.d/CentOS-*/etc/yum.repos.d/repo_bak/"在每台节点上执行以下命令，更新yum源shbatch_cmd.shnode.list"yumcleanall"

shbatch_cmd.shnode.list"yumrepolist"2.9安装http服务安装apache的httpd服务主要用于搭建OS、ClouderaManager、CDH的yum源。在集群服务器中选择一台服务器来安装httpd服务，命令如下：安装httpd服务yum-yinstallhttpd启动httpd服务systemctlstarthttpd

systemctlstatushttpd

systemctlenablehttpd安装完httpd后，重新制作操作系统repo，换成http的方式方便其它服务器也可以访问mkdir/var/www/html/iso

scp-r/media/DVD1/*/var/www/html/iso/

[root@node1~]$sudovi/etc/yum.repos.d/os.repo

[osrepo]

name=os_repo

baseurl=01/iso/

enabled=true

gpgcheck=false

[root@node1~]$sudoyumrepolist在浏览器输入01/iso看到如下截图则说明启动成功。注：需要将os.repo文件同步至集群所有服务器的/etc/yum.repo.d目录。shbatch_scp.shnode.list/etc/yum.repos.d/os.repo/etc/yum.repos.d/2.10集群时钟同步在Redhat7.x的操作系统上，已经默认的安装了chrony，我们这里先卸载chrony，然后安装ntp。使用ntp来配置各台机器的时钟同步，将node1（31）服务作为本地ntp服务器，其它2台服务器与其保持同步。1.所有机器卸载chronyshbatch_cmd.shnode.list"yum-yremovechrony"2.所有机器安装ntpshbatch_cmd.shnode.list"yum-yinstallntp"3.node1机器配置时钟与自己同步[root@node1~]#vi/etc/ntp.conf

#server0.iburst

#server1.iburst

#server2.iburst

#server3.iburst

server#localclock

fudgestratum104.集群其它节点，配置找node1机器去同步#server0.iburst

#server1.iburst

#server2.iburst

#server3.iburst

server015.重启所有机器的ntp服务shbatch_cmd.shnode.list"systemctlstartntpd"

shbatch_cmd.shnode.list"systemctlstatusntpd"|grepActive

shbatch_cmd.shnode.list"systemctlenablentpd"6.验证始终同步，在所有节点执行ntpq-p命令，如下使用脚本批量执行shbatch_cmd.shnode.list"ntpq-p"左边出现*号表示同步成功。2.11安装MariaDB安装MariaDByum-yinstallmariadb

yum-yinstallmariadb-server启动并配置MariaDBsystemctlstartmariadb

systemctlstatusmariadb

systemctlenablemariadb

[root@node1~]#/usr/bin/mysql_secure_installation

NOTE:RUNNINGALLPARTSOFTHISSCRIPTISRECOMMENDEDFORALLMariaDB

SERVERSINPRODUCTIONUSE!PLEASEREADEACHSTEPCAREFULLY!

InordertologintoMariaDBtosecureit,we'llneedthecurrent

passwordfortherootuser.Ifyou'vejustinstalledMariaDB,and

youhaven'tsettherootpasswordyet,thepasswordwillbeblank,

soyoushouldjustpressenterhere.

Entercurrentpasswordforroot(enterfornone):#回车

OK,successfullyusedpassword,movingon...

SettingtherootpasswordensuresthatnobodycanlogintotheMariaDB

rootuserwithouttheproperauthorisation.

Setrootpassword?[Y/n]Y

Newpassword:

Re-enternewpassword:

Passwordupdatedsuccessfully!

Reloadingprivilegetables..

...Success!

Bydefault,aMariaDBinstallationhasananonymoususer,allowinganyone

tologintoMariaDBwithouthavingtohaveauseraccountcreatedfor

them.Thisisintendedonlyfortesting,andtomaketheinstallation

goabitsmoother.Youshouldremovethembeforemovingintoa

productionenvironment.

Removeanonymoususers?[Y/n]Y

...Success!

Normally,rootshouldonlybeallowedtoconnectfrom'localhost'.This

ensuresthatsomeonecannotguessattherootpasswordfromthenetwork.

Disallowrootloginremotely?[Y/n]n

...skipping.

Bydefault,MariaDBcomeswithadatabasenamed'test'thatanyonecan

access.Thisisalsointendedonlyfortesting,andshouldberemoved

beforemovingintoaproductionenvironment.

Removetestdatabaseandaccesstoit?[Y/n]Y

-Droppingtestdatabase...

...Success!

-Removingprivilegesontestdatabase...

...Success!

Reloadingtheprivilegetableswillensurethatallchangesmadesofar

willtakeeffectimmediately.

Reloadprivilegetablesnow?[Y/n]Y

...Success!

Cleaningup...

Alldone!Ifyou'vecompletedalloftheabovesteps,yourMariaDB

installationshouldnowbesecure.

ThanksforusingMariaDB!建立CM，Hive等需要的表[root@node1~]#mysql-uroot-p

Enterpassword:

WelcometotheMariaDBmonitor.Commandsendwith;or\g.

YourMariaDBconnectionidis9

Serverversion:5.5.56-MariaDBMariaDBServer

Copyright(c)2000,2017,Oracle,MariaDBCorporationAbandothers.

Type'help;'or'\h'forhelp.Type'\c'toclearthecurrentinputstatement.

MariaDB[(none)]>

createdatabasemetastoredefaultcharactersetutf8;

CREATEUSER'hive'@'%'IDENTIFIEDBY'password';

GRANTALLPRIVILEGESONmetastore.*TO'hive'@'%';

FLUSHPRIVILEGES;

createdatabasecmdefaultcharactersetutf8;

CREATEUSER'cm'@'%'IDENTIFIEDBY'password';

GRANTALLPRIVILEGESONcm.*TO'cm'@'%';

FLUSHPRIVILEGES;

createdatabaseamdefaultcharactersetutf8;

CREATEUSER'am'@'%'IDENTIFIEDBY'password';

GRANTALLPRIVILEGESONam.*TO'am'@'%';

FLUSHPRIVILEGES;

createdatabasermdefaultcharactersetutf8;

CREATEUSER'rm'@'%'IDENTIFIEDBY'password';

GRANTALLPRIVILEGESONrm.*TO'rm'@'%';

FLUSHPRIVILEGES;

createdatabasehuedefaultcharactersetutf8;

CREATEUSER'hue'@'%'IDENTIFIEDBY'password';

GRANTALLPRIVILEGESONhue.*TO'hue'@'%';

FLUSHPRIVILEGES;

createdatabaseooziedefaultcharactersetutf8;

CREATEUSER'oozie'@'%'IDENTIFIEDBY'password';

GRANTALLPRIVILEGESONoozie.*TO'oozie'@'%';

FLUSHPRIVILEGES;

createdatabasesentrydefaultcharactersetutf8;

CREATEUSER'sentry'@'%'IDENTIFIEDBY'password';

GRANTALLPRIVILEGESONsentry.*TO'sentry'@'%';

FLUSHPRIVILEGES;

createdatabasenav_msdefaultcharactersetutf8;

CREATEUSER'nav_ms'@'%'IDENTIFIEDBY'password';

GRANTALLPRIVILEGESONnav_ms.*TO'nav_ms'@'%';

FLUSHPRIVILEGES;

createdatabasenav_asdefaultcharactersetutf8;

CREATEUSER'nav_as'@'%'IDENTIFIEDBY'password';

GRANTALLPRIVILEGESONnav_as.*TO'nav_as'@'%';

FLUSHPRIVILEGES;安装jdbc驱动mkdir-p/usr/share/java/

mvmysql-connector-java-5.1.34.jar/usr/share/java/

cd/usr/share/java

ln-smysql-connector-java-5.1.34.jarmysql-connector-java.jar

[root@node1java]#ll

total940

-rwxrwxr-x.1rootroot960372Feb108:31mysql-connector-java-5.1.34.jar

lrwxrwxrwx1rootroot31Feb200:52mysql-connector-java.jar->mysql-connector-java-5.1.34.jar三、ClouderaManager安装3.1配置本地CM源下载CM5.16.1的安装包，地址为：/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/cloudera-manager-agent-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm

/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm

/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm

/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm

/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/enterprise-debuginfo-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm

/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/jdk-6u31-linux-amd64.rpm

/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm下载CDH5.16.1的安装包，地址为：/cdh5/parcels/5.16.1/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel

/cdh5/parcels/5.16.1/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1

/cdh5/parcels/5.16.1/manifest.json将ClouderaManager安装需要的7个rpm包下载到本地，放在同一目录，执行createrepo命令生成rpm元数据。[root@node1cm5.16.1]#ll

total1019160

-rw-r--r--1rootroot9864584Nov2714:40cloudera-manager-agent-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm

-rw-r--r--1rootroot789872988Nov2714:40cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm

-rw-r--r--1rootroot8704Nov2714:40cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm

-rw-r--r--1rootroot10612Nov2714:40cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm

-rw-r--r--1rootroot30604172Nov2714:40enterprise-debuginfo-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm

-rw-r--r--1rootroot71204325Nov2714:40jdk-6u31-linux-amd64.rpm

-rw-r--r--1rootroot142039186Nov2714:40oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm

[root@node1cm5.16.1]#createrepo.

Spawningworker0with2pkgs

Spawningworker1with2pkgs

Spawningworker2with2pkgs

Spawningworker3with1pkgs

WorkersFinished

SavingPrimarymetadata

Savingfilelistsmetadata

Savingothermetadata

GeneratingsqliteDBs

SqliteDBscomplete如果createrepo命令没有,使用yum下载配置Web服务器将上述cdh5.16/cm5.16目录移动到/var/www/html目录下,使得用户可以通过HTTP访问这些rpm包。[root@node1~]#mvcm5.16/cdh5.16//var/www/html/验证浏览器能否正常访问制作ClouderaManager的repo源[root@node1~]#vi/etc/yum.repos.d/cm.repo

[cmrepo]

name=cm_repo

baseurl=01/cm5.16.2

enable=true

gpgcheck=false

[root@node1yum.repos.d]#yumrepolist

Loadedplugins:amazon-id,rhui-lb,search-disabled-repos

repoidreponamestatus

cmrepocm_repo7

rhui-REGION-client-config-server-7/x86_64RedHatUpdateInfrastructure2.0ClientC1

rhui-REGION-rhel-server-releases/7Server/x86_64RedHatEnterpriseLinuxServer7(RPMs)20,668

rhui-REGION-rhel-server-rh-common/7Server/x86_64RedHatEnterpriseLinuxServer7RHCommo233

repolist:20,909验证安装JDKyum-yinstalloracle-j2sdk1.7-1.7.0+update67-13.2安装ClouderaManagerServer通过yum安装ClouderaManagerServeryum-yinstallcloudera-manager-server初始化数据库[root@node1yum.repos.d]#/usr/share/cmf/schema/scm_prepare_database.shmysqlcmcmpassword

JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera

Verifyingthatwecanwriteto/etc/cloudera-scm-server

CreatingSCMconfigurationfilein/etc/cloudera-scm-server

Executing:/usr/java/jdk1.7.0_67-cloudera/bin/java-cp/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/usr/share/cmf/schema/../lib/*com.cloudera.enterprise.dbutil.DbCommandExecutor/etc/cloudera-scm-server/pertiescom.cloudera.cmf.db.

[main]DbCommandExecutorINFOSuccessfullyconnectedtodatabase.

Alldone,yourSCMdatabaseisconfiguredcorrectly!

6.2版本命令

/opt/cloudera/cm/schema/scm_prepare_database.shmysqlcmcmpassword启动ClouderaManagerServersystemctlstartcloudera-scm-server检查端口是否监听netstat-lnpt|grep7180出现以上情况,使用以下命令yum-yinstallnet-tools通过01:7180/cmf/login访问CM四、CDH安装4.1CDH集群安装向导admin/admin登录到CM同意license协议，点击继续选择60试用，点击继续点击“继续”输入主机ip或者名称，点击搜索找到主机后点击继续点击“继续”注意这里列出来的CDH版本都是系统最开始默认的，来自于Cloudera公网的下载源仓库，这里我们需要先将CDH的安装源修改一下。使用parcel选择，点击“更多选项”,点击“-”删除其它所有地址，输入01/cdh5.16.2，点击“保存更改”保存更改后，这时回到上个页面会看到我们之前准备好的http的CDH5.16.1的源，如果显示不出来，可能http源配置有问题，请参考前面步骤仔细进行检查。选择自定义存储库，输入cm的http地址点击“继续”，进入下一步安装jdk点击“继续”，进入下一步，默认多用户模式，不需要进行任何勾选点击“继续”，进入下一步配置ssh账号密码点击“继续”，进入下一步，安装ClouderaManager相关到各个节点如果安装速度很慢,注意检查本地yum源是否已经配置好点击“继续”，进入下一步安装cdh到各个节点点击“继续”，进入下一步主机检查，确保所有检查项均通过点击完成进入服务安装向导。4.2集群设置安装向导选择需要安装的服务点击“继续”，进入集群角色分配点击“继续”，进入下一步，测试数据库连接测试成功，点击“继续”，进入目录设置，此处使用默认默认目录，根据实际情况进行目录修改点击“继续”，进入各个服务启动安装成功后进入CM管理界面主页五、HBASE安装5.1准备工作：不同机器之间的时间同步要求每个节点子在30秒[root@hadoopNode5~]#yum-yinstallntp

#安装ntp软件

[root@hadoopNode5~]#ntpdate//指定与啊里云时间同步服务5.2安装：1.下载hbase1.3.22.解压[ambow@hadoopNode1~]$tar-xvzf~/soft/hbase-1.3.2-bin.tar.gz-C~/app/3．配置环境变量~/.base_profileHBASE_HOMEPATH4．hbase-env.sh配置java_home和ZK[ambow@hadoopNode1conf]$vi

$HBASE_HOME/conf/hbase-env.sh​

exportJAVA_HOME=/home/wl/app/jdk1.8.0_121

exportHADOOP_HOME=/home/wl/app/hadoop-2.7.3

​

exportHBASE_MANAGES_ZK=false

#禁用Hbase使用内置zookeper

​

exportHBASE_BACKUP_MASTERS=${HBASE_HOME}/conf/backup-masters#配置HA的第二个节HMaster节点

新建一个$HBASE_HOME/conf/backup-masters文件vi$HBASE_HOME/conf/backup-masters把备用的HMaster节点添加:hadoopNode25.hbase-site.xml配置参数​

<configuration>

<!--#指定hbase在HDFS中目录自动创建-->

<property>

<name>hbase.rootdir</name>

<value>hdfs://clusterldg/hbase</value>

</property>

<!--#true时，为集群模式u-->

<property>

<name>hbase.cluster.distributed</name>

<value>true</value>

</property>

<!--#设置自己的zookeeper用的那个几个节点-->

<property>

<name>hbase.zookeeper.quorum</name>

<value>hadoopNode1,hadoopNode2,hadoopNode3,hadoopNode4,hadoopNode5</value>

</property>

​

​

<!--#使用内置Zookeeper时要指定-->

<property>

<name>perty.dataDir</name>

<value>/home/ambow/zkdata/hdata</value>

</property>

</configuration>

​6．配置regionserver（配置每一个机器名子节点名不要配主节名）在hbase/conf/下新建regionserver文件,添加如入内容hadoopNode3

hadoopNode4

hadoopNode5

​7．scp-rhbase到其他节点[ambow@hadoopNode1conf]$scp-r~/app/hbase-1.3.2

ambow@hadoopNode5:~/app/

[ambow@hadoopNode1conf]$scp-r~/app/hbase-1.3.2

ambow@hadoopNode4:~/app/

[ambow@hadoopNode1conf]$scp-r~/app/hbase-1.3.2

ambow@had