openstack与ceph整合安装指导文档

openstack与ceph整合安装指导文档目录TOC\o"1-5"\h\z\o"CurrentDocument"概述 3\o"CurrentDocument"版本配套表 3\o"CurrentDocument"系统架构图 3物理结构图 3逻辑结构图 4\o"CurrentDocument"openstack安装 5\o"CurrentDocument"ceph安装 5\o"CurrentDocument"ip规划 5\o"CurrentDocument"安装步骤 6\o"CurrentDocument"controller点和 compute 肖点安ceph客户端 7\o"CurrentDocument"controller 节点酉I置glance使用 ceph 8controller紡点配置cinder 使用 ceph 10\o"CurrentDocument"compute节点配宜 nova 使用 ceph 121概述木文档描述openstack在glance、cinder^nova组件后端如何配置使用ceph来进行存储。2版本配套表软件版本说明openstackicehouseceph集群0.80.5ceph客户端0.81Libvirt1.1.4qemu1.6.1ceph操作系统Centos6.5DVD大小：4Gopenstack操作系统Centos6.5Mini3系统架构图3.1物理结构图

3.2逻辑结构图

3.3openstack安装使用赵子顾的自动部署，3节点部署。3.4ceph安装3.4.1ip规划角色Ip配置说明controller40见自动部署说明虚拟机,cinder、glance、nova-apinetwork41见自动部署说明虚拟机，neutroncompute42见自动部署说明虚拟机，nova-computerCephnodd48/dev/sdb/dev/sdc虚拟机，deploy,mon,osdCephnode249/dev/sdb/dev/sdc虚拟机，deploy,mon,osdCephnode350/dev/sdb虚拟机，deploy,mon,osd/dev/sdc3.4.2安装步骤修改3台机器的主机名分别为:cephl48.cephl49、cephl50编辑3台机器/etc/hosts内容如下：cephl48cephl49cephl50将ceph.zip@录拷贝到/home/ceph目录下并且解压，生成ceph和deploy两个目录。编辑/etc/yum.repos.d/ceph.repo文件内容如卜:[ceph・noarch]name=Cephnoarchpackagesbaseurl=file:///home/ceph/cephenabled=lgpgcheck=O[ceph-deply]name=Cephdeploypackagesbaseurl=file:///home/ceph/deployenabled=lgpgcheck=O三个节点增加相互信任：cephl48上执行：ssh-keygenssh-copy-idcephl48ssh-copy-idcephl49ssh-copy-idcephl50cephl49上执行：ssh-keygenssh-copy-idcephl48ssh-copy-idcephl50cephl50上执行：ssh-keygenssh-copy-idcephl48ssh-copy-idcephl49三个节点均关闭selinux和防火墙：serviceiptablesstopchkconfigiptablesoff将/etc/sysconfig/seIinux中SELINUX=enforcing改为SELINUX=disabled重启机器reboot安装ceph,三台机器均执行如下命令：yuminstallceph-y

8.9.在cephl48上执行如卜命令安装ceph-deploy：8.9.执行如下命令：cd/etc/cephceph-deploynewcephl48cephl49cephl50部署mon节点，执行如下命令：ceph-deploymoncreatecephl48cephl49cephl50ceph-deploygatherkeyscephl48〃收集密钥部署osd节点，执行如下命令：ceph-deployosdpreparecephl48:/dev/sdbcephl48:/dev/sdccephl49:/dev/sdbcephl49:/dev/sdccephl50:/dev/sdbcephl50:/dev/sdc如果有需要，部署mds,执行如下命令：重启服务/etc/init.d/ceph重启服务/etc/init.d/ceph・arestart查看ceph状态是否正常：ceph-s13.24.显示如下：cluster4fa8cb32-feal-4d68-a341-ebddab2f3e0fhealthHEALTH_WARNclockskewdetectedonmon.cephl50monmap e2: 3 mons at{cephl48=48:6789/0/cephl49=49:6789/0,cephl50=50:6789/0},electionepoch8,quorum0,1/2cephl48,cephl49，cephl50osdmape41:6osds:6up,6inpgmapv76:192pgs,3pools,0bytesdata,0objects215MBused,91878MB/92093MBavail192active+clean15.配置148为ntp的server,貝他节点泄时向148同步时间3.5controller节点和compute节点安ceph客户端（不需要,在openstack上执行ceph-version能看到版本表示ceph已经安装）执行如下命令rpm・-import,/git/?p=ceph.git;a=blob_plain;f=keys/release.asc,rpm・-import,/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.ascl增加如下文件：vi/etc/yum・repos・d/ceph-extras内容如下：[=CephExtrasPackagesbaseurl=http://ceph・com/packages/ceph-extras/rpm/centos6/Sbasearchenabled=lpriority=2gpgcheck=ltype=rpm-mdgpgkey=https://ceph・com/git/?p=ceph・git;a=blob_plain;f=keys/release・asc[ceph-extras-noarch]name=CephExtrasnoarchbaseurl=http://ceph・com/packages/ceph-extras/rpm/centos6/noarchenabled=lpriority=2gpgcheck=ltype=rpm-mdgpgkey=https://ceph・com/git/?p=ceph・git;a=blob_plain:f=kejrs/release・asc[ceph-extras-source]name=CephExtrasSourcesbaseurl=http://ceph・com/packages/ceph-extras/rpm/centos6/SRPMSenabled=lpriority=2gpgcheck=ltype=rpm-mdgpgkey=https://ceph・com/git/?p=ceph・git;a=blob_plain:f=keys/release・asc添加ceph库rpm-Uvhhttp：//ceph・com/rpms/el6/noarch/ceph-release-l一0・el6・noarch・rpm添加epel库rpm~Uvhhttp：//d1・fedoraproject・org/pub/epel/6/x86_64/epel-release-6-8・noarch・rpm安装cephyumupdate-yyuminstallceph-y3.6controller节点配置glance使用ceph将cephl48点/etc/ceph0录下的两个文件拷贝到controller盯点和compute肖点cd/etc/ceph/scpceph・confceph・client・admin・keyring192・168・1・142:/etc/ceph/scpceph・confceph・client・admin.keyring192・168・1・140:/etc/ceph/修改ceph.client,admin,keyring的权限chmod+r/etc/ceph/ceph.client.admin.keyring在cephl48上创建glance的存储池radosmkpoolglance编辑140上glance的配置文件/etc/glance/glance-api.conf中如卜配置项rbd_store_ceph_conf=/etc/ceph/ceph.confdefaultstore=rbd

rbd_store_user=adminrbd_store_pool=glance重启glance-api进程/etc/init・d/openstack-glance-apirestart测试上传本地镜像，首先将测试镜像cirros-0.3.2-x86_64-disk・img放到140的/home/1然后执行如下上传命令：glanceimage-create一一name?，cirros-0・3・2-x86_64一10〃—disk-formatqcow2一-container-formatbare ―is-public True 一一progress </home/cirros-0・3・2-x86_64一disk・img显示如F：[ >]100%Property1Value |checksum64d7clcd2b6f60c92cl4662941cb7913 |container_formatbarecreated_atI2014-09-16T08:15:46 |deletedFalsedeleted_atINone |disk_formatqcow21id149a71de0-0842-4a7a-b756-edfcb0b86153is_publicITrue |minidisk101min_ram101namecirros-0.3.2-x86_64T0ownerI3636a6e92daf4991beb64643bcl45fab |protectedFalsesizeI13167616 |statusactiveupdated_atI2014-09-16T08:15:51 |virtual^size1None I查看上传的镜像glanceimage-list显示如下：TOC\o"1-5"\h\z + + F +ID Name DiskFormatContainerFormatSize Status» W49a71de0-0842-4a7a-b756-edfcb0b86153cirros-0.3.2-x86_64T0qcow2bare 13167616active

测试网页上传镜像，在网页上传一个镜像，然后查看镜像文件glanceimage-list显示如下：IDNameDiskFormatContainerFormatSizeStatusda28a635-2336-4603-a596-30879f4716f4asdadadabare13167616active49a71de0-0842-4a7a-b756-edfcb0b86153cirros-0.3.2-IDNameDiskFormatContainerFormatSizeStatusda28a635-2336-4603-a596-30879f4716f4asdadadabare13167616active49a71de0-0842-4a7a-b756-edfcb0b86153cirros-0.3.2-x86_64T0bare13167616activeqcow2qcow2查看ceph中glance查看ceph中glance池中的对象:rbdIsglance显示如下:49a71de0-0842-la7a-b756-edfcb0b86153da28a635-2336-4603-a596-30879f4716f43.7controller节点配「•••3.7controller节点配「•••l¥i[cinder使用ceph在cephl48在cephl48上创建cinder的存储池radosmkpoolcinder编辑140上cinder的配置文件/etc/cinder/cinder.conf中如卜配置项volume_driver=cinder・volume・drivers・rbd.RBDDriverrbd_pool=cinderrbd_user=adminrbd_ceph_conf=/etc/ceph/ceph・conf重启/etc/init・d/openstack-cinder-volume进程/etc/init・d/openstack-cinder-volumerestart命令行创建一个1G的磁盘cindercreate—display-namedevl1显示如F：cinder1ist"?- H Property I ValueH H attachmentsI []availability_zone novabootablecreatedatfalse2014-09-16T08:48:50.3679765.display_description

display_name

encryptedidmetadatasize

snapshot_id

sourcevolid5.display_description

display_name

encryptedidmetadatasize

snapshot_id

sourcevolidstatusvolume_type查看创建的磁盘状态Nonedevl |Falseld8f3416-fbl5-44a9-837f-7724a9034ble|{} I1 INone |None |creatingNonecinderlist显示如下：IDVolumeTypeBootableAttachedtoIDVolumeTypeBootableAttachedto+StatusDisplayNameSizeNone■ W» MW1d8f3416-fb15-44a9-837f-7724a9034b1ecreatingfalseNone■ W» MW1d8f3416-fb15-44a9-837f-7724a9034b1ecreatingfalsedevl一+ + 界面创建一个2G磁盘6.查看创建的磁盘状态6.查看创建的磁盘状态cinderlist显示如下：IDVolumeTypeBootableAttachedtoIDVolumeTypeBootableAttachedtoStatusDisplayNameSizeld8f3416-fbl5-44a9-837f-7724a9034ble eg订ableld8f3416-fbl5-44a9-837f-7724a9034ble eg订ableNone falsee53efe68-5d3b-438d-84cl-fa4c68bd9582availableNoneifalse idevldev210.查看10.查看ceph中cinder池中的对象:rbdIscinder显示如下:volume-ld8f3416-fbl5-44a9-837f-7724a9034ble

volume-e53efe68-5d3b-438d-84cl-fa4c68bd95823.8compute节点配置nova使用ceph升级libvirtl.1.0,参考《qemu-1ibvirt更新步骤・doct》在cephl48上创建nova的存储池radosmkpoolnova生成一个uuiduuidgen编译qemu-1.6・1,参考在cephl48上创建nova的存储池radosmkpoolnova生成一个uuiduuidgen3.4.显示如F：c245elef-d340-4d02-9dcf-fd091cdlfe47执行如下命令cat>secret・xml«E0F<secretephemeral二'no'private二'no'><uuid>c245elef-d340-4d02-9dcf-fd091cdlfe47</uuid><usagetype二'ceph'><name>client・cindersecret</name></usage></secret>EOFvirshsecret-define―filesecret・xml显示如F：Secretc245elef-d340-4d02-9dcf-fd091cdlfe47created执行如下命令：cat/etc/ceph/ceph・执行如下命令：cat/etc/ceph/ceph・client・admin・keyring显示如下：[client・admin.key=AQAXrRdU807uHRAAvYit51h4Dgiz6jkAtq8GLA=将“AQAXrRdU807uHRAAvYit51h4Dgiz6jkAtq8GLA二二”放到一个临时文件echo"AQAXrRdU807uHRAAvYit51h4Dgiz6jkAtq8GLA==，>key执行如下命令：7.8.virshsecret-set-value--secretc245elef-d340-4d02-9dcf-fd091cdlfe47--base64$(catkey)9.编辑142上nova的配置文件/etc/nova/nova,conf中如卜配置项images_type=rbdimages_rbd_pool=novaimages_rbd_ceph_conf=/etc/c