Centos.离线安装CDH..生产环境详细教程(含Spark.)

1、CDH5.13.0集群安装文档版本编制人日期说明V1.0szdxxhb_2017.11.24Centos7.4安装CDH5.13.0一、 说明Cloudera 提供一个可扩展、灵活、集成的平台，可用来方便地管理您的企业中快速增长的多种多样的数据。业界领先的 Cloudera 产品和解决方案使您能够部署并管理 Apache Hadoop 和相关项目、操作和分析您的数据以及保护数据的安全。Cloudera 提供下列产品和工具：CDH Cloudera 分发的 Apache Hadoop 和其他相关开放源代码项目，包括 Impala 和 Cloudera Sea

2、rch。CDH 还提供安全保护以及与许多硬件和软件解决方案的集成。Cloudera Manager 一个复杂的应用程序，用于部署、管理、监控您的 CDH 部署并诊断问题。Cloudera Manager 提供 Admin Console，这是一种基于 Web 的用户界面，使您的企业数据管理简单而直接。它还包括 Cloudera Manager API，可用来获取群集运行状况信息和度量以及配置 Cloudera Manager。Cloudera Navigator CDH 平台的端到端数据管理工具。Cloudera Navigator 使管理员、数据经理和分析师能够了解 Hadoop 中的大量数

3、据。Cloudera Navigator 中强大的审核、数据管理、沿袭管理和生命周期管理使企业能够遵守严格的法规遵从性和法规要求。Cloudera Impala 一种大规模并行处理 SQL 引擎，用于交互式分析和商业智能。其高度优化的体系结构使它非常适合用于具有联接、聚合和子查询的传统 BI 样式的查询。它可以查询来自各种源的 Hadoop 数据文件，包括由 MapReduce 作业生成的数据文件或加载到 Hive 表中的数据文件。YARN 和 Llama 资源管理组件让 Impala 能够共存于使用 Impala SQL 查询并发运行批处理工作负载的群集上。您可以通过 Cloudera Ma

4、nager 用户界面管理 Impala 及其他 Hadoop 组件，并通过 Sentry 授权框架保护其数据。CDH5.13.01. 操作系统要求Operating SystemVersion (bold=new)Red Hat Enterprise Linux-compatibleRHEL / CentOS / Oracle Linux RHCKMax SE Linux support: 7.27.4, 7.3, 7.2, 7.16.9 , 6.8, 6.7, 6.6, 6.5, 6.45.11, 5.10, 5.7Oracle Linux (OL)7.4, 7.3, 7.2, 7.1 (U

5、EK default)6.9, 6.8 (UEK R2, R4)6.7, 6.6, 6.5 (UEK R2, R3)6.4 (UEK R2)5.11, 5.10, 5.7 (UEK R2)SUSE Linux Enterprise ServerSLES12 SP2, 12 SP111 SP4, 11 SP3, 11 SP2Ubuntu/DebianUbuntu16.04 LTS (Xenial)14.04 LTS (Trusty)12.04 LTS (Precise)Debian8.2, 8.4 (Jessie)7.0, 7.1, 7.8 (Wheezy)2. 数据库要求3. JDK版本要求O

6、nly 64 bit JDKs fromOracleare supported.Oracle JDK 7is supported across all versions of Cloudera Manager 5 and CDH 5.Oracle JDK 8is supported in C5.3.x and higher.Unless specifically excluded, support for a minor JDK release begins from the Cloudera major release in which support for themajor JDK re

7、leasewas added.For example, 8u102 was released in time for C5.9 but is actually supported from C5.3 because that is when support for JDK 1.8 was added. Cloudera excludes or removes support for select Java updates when security is jeopardized.Running CDH nodes within the same cluster on different JDK

8、 releases is not supported. JDK release across a cluster needs to match the patch level. All nodes in your cluster must run the same Oracle JDK version. All services must be deployed on the same Oracle JDK version.JDK 7All JDK 7 updates, from the minimum required version, are supported in CM/CDH 5.0

9、 and higher unless specifically excluded. Updates above the minimum that are not listed are supported but not tested.The Cloudera Manager repository containsOracle JDK 1.7.0_67(forexample) and can be automatically installed during a new installation or an upgrade.Warning:Kafka 3.0, which is included

10、 in CDH 5.13, requires JDK 8. Kafka 3.0 and higher do not support JDK 7.JDK 7 updates that are supported and testedJDK 7Supported in all C5.x1.7u80Recommended / Latest version tested1.7u75Recommended1.7u67Recommended1.7u55Minimum requiredJDK 8All JDK 8 updates, from the minimum required version, are

11、 supported in CM/CDH 5.3 and higher unless specifically excluded. Updates above the minimum that are not listed are supported but not tested.Warning: JDK 8u40, 8u45, and 8u60 are excluded from support due to a security risk:HTTP authentication can failfor web-based UI components such as HDFS, YARN,

12、SOLR, and Oozie. For CDH version 5.13.0 and higher, Sentry requires a Java version that hasJDK-8055949fixed. Kafka 3.0, which is included in CDH 5.13, requires JDK 8. Kafka 3.0 and higher do not support JDK 7. JDK 8u75 is supported but has a Known Issue:Oozie Web Console returns 500 error when Oozie

13、 server runs on JDK 8u75 or higher.JDK 8 updates that are supported and testedJDK 8Supported in C5.3 and Higher1.8u131Recommended / Latest version tested1.8u121Recommended1.8u111Recommended1.8u102Recommended1.8u91Recommended1.8u74Recommended1.8u31Minimum required4. 权限要求Important:Unless otherwise not

14、ed, when root orsudoaccess is required, using another system (such asPowerBroker) that provides root/sudo privileges is acceptable.Permission Requirements with Cloudera ManagerTaskPermissions RequiredInstall Cloudera Manager (usingcloudera-manager-installer.bin)root or sudo access on a single hostMa

15、nually start/stop/restart the Cloudera Manager Server (that is, log onto the host running Cloudera Manager and execute:service cloudera-scm-serveraction)root or sudoRun Cloudera Manager Server.cloudera-scmInstall CDH components through Cloudera Manager.One of the following, configured during initial

16、 installation of Cloudera Manager: Direct access to root user using the root password. Direct access to root user using a SSH key file. Passwordless sudo access for a specific user. This is the same requirement as the installation of CDH components on individual hosts, which is a requirement of the

17、UNIX system in general.Youcannotuse another system (such as PowerBroker) that provides root/sudo privileges.Install the Cloudera Manager Agent through Cloudera Manager.One of the following, configured during initial installation of Cloudera Manager: Direct access to root user using the root password

18、. Direct access to root user using a SSH key file. Passwordless sudo access for a specific user. This is the same requirement as the installation of CDH components on individual hosts, which is a requirement of the UNIX system in general.Youcannotuse another system (such as PowerBroker) that provide

19、s root/sudo privileges.Run the Cloudera Manager Agent.Ifsingle user modeis not enabled, access to the root account during runtime, through one of the following scenarios: During Cloudera Manager and CDH installation, the Agent is automatically started if installation is successful. It is then starte

20、d using one of the following, as configured during the initial installation of Cloudera Manager:o Direct access to root user using the root passwordo Direct access to root user using a SSH key fileo Passwordless sudo access for a specific userUsing another system (such as PowerBroker) that provides

21、root/sudo privileges isnotacceptable. Through automatic startup during system boot, using init.Manually start/stop/restart the Cloudera Manager Agent process.Ifsingle user modeis not enabled, root or sudo access.This permission requirement ensures that services managed by the Cloudera Manager Agent

22、assume the appropriate user (that is, the HDFS service assumes thehdfsuser) for correct privileges. Any action request for a CDH service managed within Cloudera Managerdoes notrequire root or sudo access, because the action is handled by the Cloudera Manager Agent, which is already running under the

23、 root user.Permission Requirements for Package-Based CDH Installation without Cloudera ManagerPermission Requirements without Cloudera ManagerTaskPermissions RequiredInstall CDH products.root or sudo access for the installation of any RPM-based package during the time of installation and service sta

24、rtup/shut down. Passwordless SSH under the root user is not required for the installation (SSH root keys).Upgrade a previously installed CDH package.root or sudo access. Passwordless SSH under the root user is not required for the upgrade process (SSH root keys).Manually install or upgrade hosts in

25、a CDH ready cluster.Passwordless SSH as root (SSH root keys), so that scripts can be used to help manage the CDH package and configuration across the cluster.Change the CDH package (for example: RPM upgrades, configuration changes the require CDH service restarts, addition of CDH services).root or s

26、udo access to restart any host impacted by this change, which could cause a restart of a given service on each host in the cluster.Start/stop/restart a CDH service.root or sudo according to UNIX standards.sudoCommands Run by Cloudera ManagerThesudocommands are: yum(RHEL/CentOS/Oracle) zypper(SLES) a

27、pt-get(Debian/Ubuntu) apt-key(Debian/Ubuntu) sed service /sbin/chkconfig(RHEL/CentOS/Oracle) /usr/sbin/update-rc.d(Debian/Ubuntu) id rm mv chown install5. 资源下载地址操作系统：Redhat7.4JDK：1.8.0+下载列表：CM包cloudera-manager.repo/cm5/redhat/7/x86_64/cm/cloudera-manager.repohttp:/archive.

28、/cm5/redhat/7/x86_64/cm/5.13.0/RPMS/x86_64/cloudera-manager-agent-5.13.0-1.cm5130.p0.55.el7.x86_64.rpm/cm5/redhat/7/x86_64/cm/5.13.0/RPMS/x86_64/cloudera-manager-daemons-5.13.0-1.cm5130.p0.55.el7.x86_64.rpm/cm5/redhat/7/x86_64/cm/5.13.0

29、/RPMS/x86_64/cloudera-manager-server-5.13.0-1.cm5130.p0.55.el7.x86_64.rpm/cm5/redhat/7/x86_64/cm/5.13.0/RPMS/x86_64/cloudera-manager-server-db-2-5.13.0-1.cm5130.p0.55.el7.x86_64.rpm/cm5/redhat/7/x86_64/cm/5.13.0/RPMS/x86_64/enterprise-debuginfo-5.1

30、3.0-1.cm5130.p0.55.el7.x86_64.rpm/cm5/redhat/7/x86_64/cm/5.13.0/RPMS/x86_64/jdk-6u31-linux-amd64.rpm/cm5/redhat/7/x86_64/cm/5.13.0/RPMS/x86_64/oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm/cdh5/repo-as-tarball/5.13.0/cdh5.13

31、.0-centos7.tar.gzCDH离线包/cdh5/parcels/9/CDH-5.13.0-1.cdh5.13.0.p0.29-el7.parcel/cdh5/parcels/9/CDH-5.13.0-1.cdh5.13.0.p0.29-el7.parcel.sha1/cdh5/parcels/9/manifest.jsonSpark2离线包http:/archive.cloud

32、/spark2/csd/SPARK2_ON_YARN-2.2.0.cloudera1.jar/spark2/parcels/2.2.0.cloudera1/manifest.json/spark2/parcels/2.2.0.cloudera1/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354-el7.parcel.sha1/spark2/parcels/2.2.0.cloudera1/

33、SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354-el7.parcel 二、 系统环境搭建6. 制作本地ISO1、 挂载ISO文件mkdir p /data/isomount -o loop -t iso9660 /home/bigdata/Downloads/CentOS-7-x86_64-Everything-1708.iso /data/iso拷贝至一个文件夹复制到下面供http挂载mkdir p /data/centos7.4cp -rf /data/iso/* /data/centos7.4umount /data/iso/7. 制作本地Yum

34、源roothadoop01-sit centos7.4# cd /etc/yum.repos.d/roothadoop01-sit yum.repos.d# lltotal 28-rw-r-r-. 1 root root 1664 Aug 30 23:53 CentOS-Base.repo-rw-r-r-. 1 root root 1309 Aug 30 23:53 CentOS-CR.repo-rw-r-r-. 1 root root 649 Aug 30 23:53 CentOS-Debuginfo.repo-rw-r-r-. 1 root root 314 Aug 30 23:53 Ce

35、ntOS-fasttrack.repo-rw-r-r-. 1 root root 630 Aug 30 23:53 CentOS-Media.repo-rw-r-r-. 1 root root 1331 Aug 30 23:53 CentOS-Sources.repo-rw-r-r-. 1 root root 3830 Aug 30 23:53 CentOS-Vault.reporoothadoop01-sit yum.repos.d# for file in ls ;do mv $file $filebak;done roothadoop01-sit yum.repos.d# lltotal

36、 28-rw-r-r-. 1 root root 1664 Aug 30 23:53 CentOS-Base.repobak-rw-r-r-. 1 root root 1309 Aug 30 23:53 CentOS-CR.repobak-rw-r-r-. 1 root root 649 Aug 30 23:53 CentOS-Debuginfo.repobak-rw-r-r-. 1 root root 314 Aug 30 23:53 CentOS-fasttrack.repobak-rw-r-r-. 1 root root 630 Aug 30 23:53 CentOS-Media.rep

37、obak-rw-r-r-. 1 root root 1331 Aug 30 23:53 CentOS-Sources.repobak-rw-r-r-. 1 root root 3830 Aug 30 23:53 CentOS-Vault.repobakroothadoop01-sit yum.repos.d# vi local.repocentos7_4name=centos7_4baseurl=file:/data/centos7.4enabled=1gpgcheck=0roothadoop01-sit yum.repos.d# yum clean allLoaded plugins: fa

38、stestmirror, langpacksCleaning repos: centos7_4Cleaning up everythingMaybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed reposCleaning up list of fastest mirrorsroothadoop01-sit yum.repos.d# yum makecacheLoaded plugins: fastestmirror, langpack

39、scentos7_4 | 3.6 kB 00:00:00 (1/4): centos7_4/group_gz | 156 kB 00:00:00 (2/4): centos7_4/primary_db | 5.7 MB 00:00:00 (3/4): centos7_4/filelists_db | 6.7 MB 00:00:00 (4/4): centos7_4/other_db | 2.5 MB 00:00:00 Determining fastest mirrorsMetadata Cache Created8. 安装http1、 检查服务是否存在roothadoop01-sit iso

40、# service httpd statusRedirecting to /bin/systemctl status httpd.serviceUnit httpd.service could not be found.roothadoop01-sit yum.repos.d# yum -y install httpdLoaded plugins: fastestmirror, langpacksLoading mirror speeds from cached hostfileResolving Dependencies- Running transaction check- Package

41、 httpd.x86_64 0:2.4.6-67.el7.centos will be installed- Processing Dependency: httpd-tools = 2.4.6-67.el7.centos for package: httpd-2.4.6-67.el7.centos.x86_64- Processing Dependency: /etc/mime.types for package: httpd-2.4.6-67.el7.centos.x86_64- Processing Dependency: libaprutil-1.so.0()(64bit) for p

42、ackage: httpd-2.4.6-67.el7.centos.x86_64- Processing Dependency: libapr-1.so.0()(64bit) for package: httpd-2.4.6-67.el7.centos.x86_64- Running transaction check- Package apr.x86_64 0:1.4.8-3.el7 will be installed- Package apr-util.x86_64 0:1.5.2-6.el7 will be installed- Package httpd-tools.x86_64 0:

43、2.4.6-67.el7.centos will be installed- Package mailcap.noarch 0:2.1.41-2.el7 will be installed- Finished Dependency ResolutionDependencies Resolved= Package Arch Version Repository Size=Installing: httpd x86_64 2.4.6-67.el7.centos centos7_4 2.7 MInstalling for dependencies: apr x86_64 1.4.8-3.el7 ce

44、ntos7_4 103 k apr-util x86_64 1.5.2-6.el7 centos7_4 92 k httpd-tools x86_64 2.4.6-67.el7.centos centos7_4 87 k mailcap noarch 2.1.41-2.el7 centos7_4 31 kTransaction Summary=Install 1 Package (+4 Dependent packages)Total download size: 3.0 MInstalled size: 10 MDownloading packages:-Total 74 MB/s | 3.

45、0 MB 00:00:00 Running transaction checkRunning transaction testTransaction test succeededRunning transaction Installing : apr-1.4.8-3.el7.x86_64 1/5 Installing : apr-util-1.5.2-6.el7.x86_64 2/5 Installing : httpd-tools-2.4.6-67.el7.centos.x86_64 3/5 Installing : mailcap-2.1.41-2.el7.noarch 4/5 Insta

46、lling : httpd-2.4.6-67.el7.centos.x86_64 5/5 Verifying : httpd-2.4.6-67.el7.centos.x86_64 1/5 Verifying : mailcap-2.1.41-2.el7.noarch 2/5 Verifying : apr-1.4.8-3.el7.x86_64 3/5 Verifying : httpd-tools-2.4.6-67.el7.centos.x86_64 4/5 Verifying : apr-util-1.5.2-6.el7.x86_64 5/5 Installed: httpd.x86_64

47、0:2.4.6-67.el7.centos Dependency Installed: apr.x86_64 0:1.4.8-3.el7 apr-util.x86_64 0:1.5.2-6.el7 httpd-tools.x86_64 0:2.4.6-67.el7.centos mailcap.noarch 0:2.1.41-2.el7 Complete!roothadoop01-sit html# service firewalld stopRedirecting to /bin/systemctl stop firewalld.serviceroothadoop01-sit html# s

48、ystemctl disable firewalld.serviceRemoved symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. roothadoop01-sit yum.repos.d# cd /var/www/htmlroothadoop01-sit html# lltotal 0roothadoop01-sit html# ln -s /d

49、ata/centos7.4/ centos7_4roothadoop01-sit html# lltotal 0lrwxrwxrwx. 1 root root 16 Nov 13 09:39 centos7_4 - /data/centos7.4/roothadoop01-sit html# service httpd startRedirecting to /bin/systemctl start httpd.service访问浏览器76/centos7_4/如果不能访问，请检查防火墙是否关闭，注意Centos7.4防火墙服务是firewall放置JDK服务包,版本：jdk-8u112-linux-x64.tar.gzroothadoop01-sit Documents# cd /var/www/html/roothadoop01-sit html# lltotal 0lrwxrwxrwx. 1 root root 16 Nov 13 09:39 cen