cloudera 5 x部署手册Cloudera企业级数据中心安装主要分为4个步骤

操作系统版本：CentOS6/RHEL6SUSE11CM版本：CMCDH版本：CDH您已经CDH和CM的安装服务器配置如果不能连接互联网，先创建OS的repository，以便yum或zypper可以直接OS镜IP地址。如果使用DHCPDNS服务器，具体请参见对应软件的安装文档，此不赘述；IPIPhostname信息，并在/etc/hosts配置所有节点的静态DNS解析。若需要启用集群的Kerberos认证功能，则一定要配置集群的DNS，这可以是一个假域名。以RHEL为例：##$>$>serviceiptables$>chkconfigiptables$>sh/sbin/rcSuSEfirewall2$>sh/sbin/rcSuSEfirewall2$>chkconfigSuSEfirewall2_setup$>$>echo"SELINUX=disabled">/etc/sysconfig/selinux$>serviceboot.apparmor$>serviceboot.apparmor$>chkconfigboot.apparmor$>$>/etc/init.d/network$>yum$>yuminstall–y$>servicenscd$>chkconfignscd#在#在/etc/rc.local$>echonever>#在/etc/init.d/after.localif#在/etc/init.d/after.localiftest-f/sys/kernel/mm/transparent_hugepage/enabled;thenechonever>交换（可选CDHkernel$>vim$>vim$>sudosysctl（可选）修改/etc/security/limits.conf或者在/etc/security/limits.d下增加相应的配置文件，可以设置一些硬限制和软限制；ClouderaManager节点会为所有节点自动做这些修改。$$>cat/proc/<pid>/limits#$>$>yuminstall#createrepo$>yuminstall$>servicehttpd$>chkconfighttpd#pleasefind#pleasefindtheyum-metadata-parserandcreaterepopackageinosrepodirectory,usuallyyoucanfindthetheminx86_64ornoarchfolder.$>$>rpm–ivhyum-metadata-parser-$>rpm–ivhcreaterepo-$>rpm–ivhyast2-http-server-2.17.15-磁盘划分RAID推荐 HostMonitor（Astandalone 署ServiceMonitor（Astandalone 署EventServer（AserviceforCloudera整的最大数来控制空间占serviceforClouderaManager） NavigatorAuditServer（ANavigatorMetadataServer（ANameNode角色划分（样例NodeCM(andrelated),NodeDN,NM,RS,HS2,HMS,NodeDN,NM,RS,HM,Node4–DN,NM,NodeNN,RM,NodeDN,NM,RS,HS2,NodeDN,NM,RS,HM,Node14–DN,NM,NodeNN,RM,NodeDN,NM,RS,HS2,HMS,NodeDN,NM,RS,HM,Node44–DN,NM,NodeDN,RS,ZK,NodeDN,RS,Node63–DN,NodeDN,RS,ZK,Node72–DN,外部软件安装rpmrpm--e--nodepsmysql-MySQLServer MySQLConnectorJava:$>rpm$>rpm-ivhmysql-server-5.5.41-1.sles11.x86_64.rpmmysql- $>##Incaseyoumayneedtoinstalleddependencies,like$>chkconfigmysql$>servicemysql#MySQLServerClouderaManagerServerMySQL-shared-xxx.rpmswithfilefrompackageMySQL-server-5.5.41-1.el6.x86_64swithfilefrompackageMySQL-server-5.5.41-1.el6.x86_64# # root 方式登录数据Entercurrentpasswordforroot(enterfornone):OK,successfullyusedpassword,movingon...SetSetrootpassword?[Y/n]yNewpassword:123456Re-enternewRemoveanonymoususers?[Y/n]YDisallowrootloginremo y?[Y/n]NRemovetestdatabaseandaccesstoit[Y/n]YReloadprivilegetablesnow?[Y/n]YAlldone!mysql-urootmysql-uroot--password=<PWDforroot>-e'createdatabasemetastoredefaultcharactersetutf8;'mysql-uroot--password=<PWDforroot>-e"CREATEUSER'hive'@'%'IDENTIFIEDBY'123456';GRANTALLPRIVILEGESONmetastore.*TO'hive'@'%';FLUSHPRIVILEGES;"创建ClouderaEnterpriseHivemysql-uroot--password=<PWDforroot>-e"CREATEUSER'hive'@'%'IDENTIFIEDBY'123456';GRANTALLPRIVILEGESONmetastore.*TO'hive'@'%';FLUSHPRIVILEGES;"ReportsManagerSentryServerClouderaNavigatorAuditServerClouderaNavigatorMetadataServeryarnactivitymonitormysql-uroot--password='123456'-e"createuser'amon'@'%'identifiedbymysql-uroot--password='123456'-e'createdatabaseamondefaultcharactersetutf8'mysql-uroot--password='123456'-e"grantallprivilegesonamon.*tomysql-uroot--password='123456'-e"createuser'rman'@'%'identifiedbymysql-uroot--password='123456'-e'createdatabasermancharactersetmysql-uroot--password='123456'-"grantprivilegesonrman.*mysql-uroot--password='123456'-e"createuser'sentry'@'%'bymysql-uroot--password='123456'-e'createdatabasesentrydefaultcharactersetutf8'mysql-uroot--password='123456'-e"grantallprivilegesonsentry.*tomysql-uroot--password='123456'-e"createuser'nav'@'%'identifiedbymysql-uroot--password='123456'-e'createdatabasenavdefaultcharactersetutf8'mysql-uroot--password='123456'-e"grantallprivilegesonnav.*tomysql-uroot--password='123456'-e"createuser'navms'@'%'identifiedbymysql-uroot--password='123456'-e'createdatabasenavmsdefaultcharactersetutf8'mysql-uroot--password='123456'-e"grantallprivilegesonnavms.*tomysql-uroot--password='123456'-e"createuser'cm'@'%'identifiedbymysql-uroot--password='123456'-e'createdatabasecmdefaultcharactersetutf8'mysql-uroot--password='123456'-e"grantallprivilegesoncm.*toCDH软件与配置（Cloudera管理器节点 wgetwget-c-r-nd-np-k-L-Aparcel 2.Parcel包（包含了CDHparcel 的软件放置为如下结构（该步骤不是必须的，只是为了后续说明的方便 $>cd$>createrepoSpawning$>cd$>createrepoSpawningworker0with7pkgsWorkersFinishedGatheringworkerSavingPrimarymetadataSavingfilelistsmetadataSavingothermetadata DBs$> $>CDH-5.2.0-1.cdh5.2.0.p0.36-el6.parcelcm$>mkdir-p$>mvCDH-5.2.0-1.cdh5.2.0.p0.36-：$>mvmanifest.json$>mvcm od-Rugo+rX新建文件Cloudera管理器yumyuminstalloracle-zypperinstalloracle-zypperinstalloracle-yumyuminstallcloudera-manager-daemonscloudera-manager- zypperinstallcloudera-manager-daemonszypperinstallcloudera-manager-daemonscloudera-manager- cpmysql-connector-java-5.1.34.jarln–smysql-connector-java-5.1.34.jarcpmysql-connector-java-5.1.34.jarln–smysql-connector-java-5.1.34.jarmysql-connector-<DATABASE><USERNAME>-h<MYSQL_HOST>localhostDB_TYPE可以是mysql,oracleDATABASEClouderaManager

-h<MYSQL_HOST>servicecloudera-scm-serverservicecloudera-scm-server CDH集群 集群基本参数调整ClouderaManagementService群大小而定ServiceJava1GBJava2GBHostJava1GBJava2GBEventJavaReportJavaAlertJavaJava180000Java48GB或更高JavaJavaResourceJava11Javavcore数量相当的内存数量（32GB或者40GB80GBApplicationMaster容器内存1ApplicationMasterJava0.8倍于容器内存1Map/ReduceJavaImpalaImpalaDaemonHBaseMasterJavaHBaseRegionServerJava120000JavaHiveJavaHiveJava集群基准测试min.user.id（默认值是1000。A磁盘格式化与挂载$>$>catformatdisk.shDEVICE_LIST="/dev/sdb/dev/sdc"forDEVICEin$DEVICE_LISTecho"***createpartitionfor$DEVICEparted-s$DEVICEmklabelgptmkpartgpt2text20%100%echo"***formatting$PARTITION***"mkfs.ext4-Tlargefile$PARTITION$>sh$>catmountdisk.sh#backupcp/etc/fstab/etc/fstab.bakPARTITION_LIST="sdb1sdc1"forPARTITIONinUUID=`blkid"/dev/""$PARTITION"|awk'{print$2}'|sed's/\"//g'`echo"***UUID$UUID***"echo"***add$PARTITIONto/etc/fstab***"echo"mkdir-p$MOUNTDIR"mkdir-p$MOUNTDIRecho"***appending\"$UUID$MOUNTDIRext4defaults00\"to***echo"$UUID$MOUNTDIRext4defaults00">>/etc/fstabecho""#mountallpartitionsmount-a#showmountedpartitionsdf–h$>shB:ntpd时间同步#Permitallaccessovertheloopback#Permitallaccessovertheloopbackinterface.Thiscould#betightenedaswell,buttodosowouldeffectsomeof#theadministrativefunctions.restrictrestrict-6restrictmask$>servicentpd$>servicentpd启动后，ntp服务占用端123#Usepublicserversfrom#Uproject.#Pleaseconsiderjoiningthepool(#iburst#iburst##iburstserver13$>servicentpd$>servicentpd$>synchronisedtoNTP$>synchronisedtoNTPserver(13)atstratum3timecorrecttowithin379mspollingserverevery64C快速服务组件验证[root]$groupaddcloudera-dev#增加用户[root]$useradd-gcloudera-devcloudera-#查看用户组cloudera-dev[root]$lid-gcloudera-#查看用户cloudera-dev[root]$lidcloudera-#Hadoop[root]$sudo-uhdfshadoopfs-mkdir/user/cloudera-dev[root]$sudo-uhdfshadoopfs-chowncloudera-dev:cloudera-dev运行Hadoopwordcount示例程序[root]$sudosucloudera-[cloudera-dev]$echo" oWorldByeWorld">file0[cloudera-dev]$echo" oHadoopGoodbyeHadoop">file1[cloudera-dev]$hadoopfs-mkdir-p/user/cloudera-dev/wordcount/input[cloudera-dev]$hadoopfs-putfile*/user/cloudera-dev/wordcount/input[cloudera-dev]$hadoopjar/opt/cloudera/parcels/CDH/jars/hadoop-examples.jarwordcountwordcount/inputwordcount/output[cloudera-dev]$hadoopfs-getmergewordcount/outputoutput.txt[cloudera-dev]$catoutput.txt o World[root]$sudosu[root]$sudosucloudera-[cloudera-dev]$echo"Alex,Cloudera">file2[cloudera-dev]$hadoopfs-mkdir-phive/input[cloudera-dev]$hadoopfs-putfile2hive/input[cloudera-dev]$cattest.hqlcreateexternaltabletest(namestring,)rowformatdelimitedfieldsterminatedby','location'/user/cloudera-[cloudera-dev]$[cloudera-dev]$hive-f[cloudera-dev]$hive-e"select*fromtest"2>/dev/nullAlexCloudera[cloudera-dev]$hive-e"selectcount(*)fromtest"2>/dev/null[cloudera-dev]$hive-e"droptable[cloudera-dev]$HBase验证（可选[cloudera-dev]$cattest.hbasecreate'record',{NAME=>'user'}put'record','Alex',' get'record','Alex'[cloudera-dev]$hbases timestamp=1421390917686,1row(s)in0.0190seconds[cloudera-dev]$hbaseshbase(main):001:0>disable'record'0row(s)in2.7850secondshbase(main):002:0>drop'record'0row(s)in0.1880运行HiveOverHBase测试(Read&Write)[cloudera-dev]$c