05-存储篇实验手册-存储篇

1、1分布式文件系统 GlusterFSGlusterFS 适合存储大文件，小文件性能较差。生产上可以使用 GlusterFS+Cinder 为Openstack 提供块存储，存储虚拟机镜像。7.1 GlusterFS 概述 GlusterFS 是 Scale-Out 存储解决方案 Gluster 的核心，它是一个开源的分布式文件系统，具有强大的横向扩展能力，通过扩展能够支持数 PB 存储容量和处理数千客户端。GlusterFS借助 TCP/IP 或 InfiniBand RDMA 网络将物理分布的存储资源聚集在一起，使用单一全局命名空间来管理数据。GlusterFS 基于可堆叠的用户空间设计，可

2、为各种不同的数据负载提供优异的性能。7.1.1 GlusterFS 架构7.1.2 GulsterFS 特性7.2 GlusterFS 部署 7.2.1 环境说明本文使用以下环境进行部署。主机名IP 地址角色备注1Server、C2Server、Client7.2.2 GlusterFS 安装目前 Gluster 的最新版本为 3.4.3 版本，直接使用官方提供的 repo 进行安装。rootlinux-node2 # yumtall cento

3、s-release-glusterrootlinux-node2 # yumtall glusterfs-server验证安装rootlinux-node1 yum.repos.d# glusterfs -Vglusterfs 3.4.3 built on Apr3 2014 16:02:46Repository revision: git://glusterfs.gitCopyright (c) 2006-2013 Red Hat, Inc. /启动 Glusterdrootlinux-node1 yum.repos.d#

4、 /etc/init.d/glusterd start请安装并启动 linux-node27.3 Trusted Storage Pools 创建 在开始创建 ClusterFS 卷之前，需要创建一个称之为 Trusted Storage 的池，是一个可信的网络存储服务器，可以理解为集群。为卷提供 bricks。gluster peer probe，建议是 IP 地址，或者内网用 DNS 做主机名解析，或者在/etc/hosts 做好绑定。7.3.1 增加 Server 到 Pool添加成员服务器到 Poll 中：例rootlinux-node1 # gluster peer probe 19

5、2peer probe: success7.3.2 从 Pool 中删除 Server删除集群中的成员服务器：例rootlinux-node1 # gluster peer detach peer detach: success注意不要添加本地的服务器。rootlinux-node1 # gluster peer probe peer probe: success: on localhost not neededrootlinux-node1 # gluster peer probe peer

6、probe: success7.3.3 查看 Pool 状态rootlinux-node1 # gluster peer statusNumber of Peers: 37.4 GlusterFS Volumes 创建 GlusterFS 的卷共有三种基本类型，可以组合共 7 种类型，其中 Distributed、Replicated、Strped、为基本类型。Distributed Striped、Distributed Replicated、Distributed Striped Replicated、Striped Replicated。为组合类型。本文为了测试 7 种卷类型，需要在各个

7、服务器分别创建 7 个目录。rootlinux-node1 # mkdir /data & cd /data & mkdir type1,2,3,4,5,6,7/exp1,2,3,4,5,6,7p创建 Volumes 使用以下命令进行：gluster volume create NEW-VOLNAME stripe COUNT | replica COUNTtransport tcp | rdma | tcp,rdma NEW-BRICK1 NEW-BRICK2 NEW-BRICK 创建分布式卷分布式卷架构示意图：rootlinux-node1 data# gluster vol

8、ume create test1-volume :/data/type1/exp:/data/type1/exp1 1:/data/type1/exp1 2:/data/type1/exp1 forcevolume create: test1-volume: success: please start the volume to access datarootlinux-node1 data# gluster volume info test1-volumeVolume Name: test1-volumeType: D

9、istributeVolume ID: 1ed9c3fe-3bc7-4525-b194-219a8e85f7fbStatus: CreatedNumber of Bricks: 4Transport-type: tcpBricks:Brick1: :/data/type1/exp1Brick2: :/data/type1/exp1Brick3: 1:/data/type1/exp1Brick4: 2:/data/type1/exp17.4.2 创建复制卷复制卷的架构示意图。rootlinux-node1 data# glu

10、ster volume create test2-volume replica 4 :/data/type2/exp:/data/type2/exp1 1:/data/type2/exp1 2:/data/type2/exp1 forcevolume create: test2-volume: success: please start the volume to access datarootlinux-node1 data# gluster volume info test2-volumeVolume Name: t

11、est2-volumeType: ReplicateVolume ID: 035b6768-2d07-4530-aa91-38571cacc90cStatus: CreatedNumber of Bricks: 1 x 4 = 4Transport-type: tcpBricks:Brick1: :/data/type2/exp1Brick2: :/data/type2/exp1Brick3: 1:/data/type2/exp1Brick4: 2:/data/type2/exp17.4.3 创建条带卷条带卷架构示意图：r

12、ootlinux-node1#glustervolumecreatetest3-volumestripe4:/data/type3/exp:/data/type3/exp1:/data/type3/exp2:/data/type3/exp1 forcevolume create: test3-volume: success: please start the volume to access datarootlinux-node1 # gluster volume info test3-volumeVolume Na

13、me: test3-volumeType: StripeVolume ID: 6d754a08-c0e9-4fb8-bc8e-54c8f1f3cbe4Status: CreatedNumber of Bricks: 1 x 4 = 4Transport-type: tcpBricks:Brick1: :/data/type3/exp1Brick2: :/data/type3/exp1Brick3: 1:/data/type3/exp1Brick4: 2:/data/type3/exp17.4.4 创建分布式+条带卷分布式条

14、带卷架构示意图：rootlinux-node1#glustervolumecreatetest4-volumestripe:/data/type4/exp:/data/type4/exp1:/data/type4/exp2:/data/type4/exp1 forcevolume create: test4-volume: success: please start the volume to access datarootlinux-node1 # gluster volume info test4-volume

15、Volume Name: test4-volumeType: Distributed-StripeVolume ID: 2b14574c-79c6-4329-91b9-8dc036969f2fStatus: CreatedNumber of Bricks: 2 x 2 = 4Transport-type: tcpBricks:Brick1: :/data/type4/exp1Brick2: :/data/type4/exp1Brick3: 1:/data/type4/exp1Brick4: 2:/data/type4/ex

16、p17.4.5 创建分布式+复制卷分布式复制卷架构示意图：rootlinux-node1#glustervolumecreatetest5-volumereplica:/data/type5/exp:/data/type5/exp1:/data/type5/exp2:/data/type5/exp1 forcevolume create: test5-volume: success: please start the volume to access datarootlinux-node1 # gluster vo

17、lume info test5-volumeVolume Name: test5-volumeType: Distributed-ReplicateVolume ID: fc94f6f7-a3e4-471b-82b0-4e74d5a529b9Status: CreatedNumber of Bricks: 2 x 2 = 4Transport-type: tcpBricks:Brick1: :/data/type5/exp1Brick2: :/data/type5/exp1Brick3: 1:/data/type5/exp1Brick4:

18、2:/data/type5/exp17.4.6 创建分布式+条带+复制卷rootlinux-node1#glustervolumecreatetest6-volumestripe2replica:/data/type6/exp:/data/type6/exp:/data/type6/exp:/data/type6/exp1:/data/type6/exp1:/data/type6/exp2:/data/type6/exp1 10.6.11

19、.32:/data/type6/exp2 forceMultiple bricks of a replicate volume are present on the same server. This setup is not optimal.Do you still want to continue creating the volume?(y/n) yvolume create: test6-volume: success: please start the volume to access datarootlinux-node1 # gluster volume info test6-v

20、olumeVolume Name: test6-volumeType: Distributed-Striped-ReplicateVolume ID: e6d681e3-b006-41ce-a466-c4c16cf03cceStatus: CreatedNumber of Bricks: 2 x 2 x 2 = 8Transport-type: tcpBricks:Brick1: :/data/type6/exp1Brick2: :/data/type6/exp2Brick3: :/data/type6/exp1Brick4: 10.6.1

21、6.2:/data/type6/exp2Brick5: 1:/data/type6/exp1Brick6: 1:/data/type6/exp2Brick7: 2:/data/type6/exp1Brick8: 2:/data/type6/exp27.4.7 创建条带+复制卷rootlinux-node1#glustervolumecreatetest7-volumestripe2replica:/data/type7/exp:/data/type7/exp1:/

22、data/type7/exp2:/data/type7/exp1 forcevolume create: test7-volume: success: please start the volume to access datarootlinux-node1 # gluster volume info test7-volumeVolume Name: test7-volumeType: Striped-ReplicateVolume ID: 3f007e9e-3c42-4dd3-935e-60da041659d1Status: CreatedNumber of Bricks

23、: 1 x 2 x 2 = 4Transport-type: tcpBricks:Brick1: :/data/type7/exp1Brick2: :/data/type7/exp1Brick3: 1:/data/type7/exp1Brick4: 2:/data/type7/exp17.4.8 启动卷在开始挂载使用卷之前，必需先启动卷。使用以下命令启动卷。rootlinux-node3 yum.repos.d# gluster volume startUsage: volume start force7.5 客户端挂载

24、GlusterFS 支持三种客户端类型。Gluster Native Client、NFS 和CIFS。官方推荐使用 NativeClient，可以使用 GlusterFS 的全部功能。7.5.1 Gluster Native ClientGluster Native Client 是基于 FUSE 的，所以需要保证客户端安装了 FUSE。这个是官方推荐的客户端，支持高并发和高效的写性能。rootlinux-node4 # yum -ytall glusterfs-clientrootlinux-node4 # mkdir /mnt/glusterfsrootlinux-node4 # mou

25、nt.glusterfs :/test1-volume /mnt/glusterfs/rootlinux-node4 # mount/dev/sda3 on / type ext4 (rw)proc on /proc type proc (rw)sysfs on /sys type sysfs (rw)devpts on /dev/pts type devpts (rw,gid=5,mode=620)tmpfs on /dev/shm type tmpfs (rw)/dev/sda1 on /boot type ext4 (rw)none on /proc/sys/fs/bi

26、nfmt_misc type binfmt_misc (rw):/test1-volumeon/mnt/glusterfstypefuse.glusterfs(rw,default_permissions,allow_other,max_read=131072)挂载参数当使用 mount t glusterfs 的使用，可以执行相关的参数如下：backupvolfile-server=server-namevolfile-max-fetch-attempts=number of attemptslog-level=loglevellog-file=logfiletranspo

27、rt=transport-typedirect-io-mode=enable|disableFor example:# mount -t glusterfs -o backupvolfile-server=volfile_server2 -volfile-max-fetch-attempts=2log-level=WARNING,log-file=/var/log/gluster.log server1:/test-volume /mnt/glusterfsIfbackupvolfile-serveroption is added while mounting fuse client, whe

28、n the first volfile serverfails, then the server specified inbackupvolfile-serveroption is used as volfile server to mountthe client.In-volfile-max-fetch-attempts=Xoption, specify the number of attempts to fetch volume fileswhile mounting a volume. This option is useful when you mount a server with

29、multiple IPaddresses or when round-robin DNS is configured for the server-name.7.5.2 NFS 挂载NFS 默认是使用的 UDP 进行通信，由于 GlusterFS 不支持 UDP。所以需要设置为 TCP 的方式挂载。mount -o mountproto=tcp -t nfs server1:/test-volume /mnt/glusterfs7.5.3 CIFS 挂载7.5.4 测试各个卷类型7.6 管理 GlusterFS 卷 7.6.1 停止卷rootlinux-node1 # gluster volu

30、me stop test1-volume7.6.2 删除卷rootlinux-node1 # gluster volume delete test1-volume7.6.3 扩展卷GlusterFS 支持在线进行卷的扩展。rootlinux-node3 data# gluster peer probe 1peer probe: successrootlinux-node3 data# gluster volume add-brick test1-volume1:/data/type1/exp1forcevolume add-brick: successroo

31、tlinux-node3 data# gluster volume info test1-volumeVolume Name: test1-volumeType: DistributeVolume ID: 1ed9c3fe-3bc7-4525-b194-219a8e85f7fbStatus: StoppedNumber of Bricks: 5Transport-type: tcpBricks:Brick1: :/data/type1/exp1Brick2: :/data/type1/exp1Brick3: 1:/data/type1/ex

32、p1Brick4: 2:/data/type1/exp1Brick5: 1:/data/type1/exp17.6.4 收缩卷收缩卷和扩展卷相似据以 Brick 为单位。rootlinux-node3 data# gluster volume remove-brick test1-volume 1:/data/type1/exp1startvolume remove-brick start: successID: 15764d05-5675-46b5-8467-b9eb12570e61rootlinux-node3 data# gluste

33、r volume remove-brick test1-volume 1:/data/type1/exp1statusrootlinux-node3 data# gluster volume remove-brick test1-volume 1:/data/type1/exp1commitRemoving brick(s) can result in data loss. Do you want to Continue? (y/n) yvolume remove-brick commit: successrootlinux-node3 data# glus

34、ter volume info test1-volumeVolume Name: test1-volumeType: DistributeVolume ID: 1ed9c3fe-3bc7-4525-b194-219a8e85f7fbStatus: StartedNumber of Bricks: 4Transport-type: tcpBricks:Brick1: :/data/type1/exp1Brick2: :/data/type1/exp1Brick3: 1:/data/type1/exp1Brick4: 2:/d

35、ata/type1/exp17.6.5 迁移卷进行卷迁移前需要确保迁移目标已经加入集群。或者迁移的 Brick 目录存在。迁移卷可以执行暂停和中止的操作。rootlinux-node3 data# gluster volume replace-brick test1-volume :/data/type1/exp:/data/type1/exp2 commit forceAll replace-brick commands except commit force are deprecated. Do you want to continue? (y/n)y

36、暂停迁移gluster volume replace-brick test1-volume :/data/type1/exp1 :/data/type1/exp2pause中止迁移gluster volume replace-brick test1-volume :/data/type1/exp1 :/data/type1/exp2abort 替换Brick#glustervolumereplace-brickVOLNAMEBRICKNEW-BRICKstart/pause/abort/status/comm

37、it#gluster volume replace-brick dht_vol server0:/mnt/sdb1 server0:/mnt/sdc1 start/如上，执行 replcace-brick 卷替换启动命令，使用 start 启动命令后，开始将原始 Brick 的数据迁移到即将需要替换的 Brick 上。#gluster volume replace-brick dht_vol server0:/mnt/sdb1 server0:/mnt/sdc1 status/在数据迁移的过程中，可以查看替换任务是否完成。#gluster volume replace-brick dht_vo

38、l server0:/mnt/sdb1 server0:/mnt/sdc1 abort/在数据迁移的过程中，可以执行 abort 命令终止 Brick 替换。#gluster volume replace-brick dht_vol server0:/mnt/sdb1 server0:/mnt/sdc1 commit/在数据迁移结束之后，执行 commit 命令结束任务，则进行 Brick 替换。使用 volumeinfo命令可以查看到 Brick 已经被替换。1.1.1系统配额开启/关闭系统配额# gluster volume quota VOLNAME enable/disable/在使用

39、系统配额功能时，需要使用 enable 将其开启；disable 为关闭配额功能命令。设置(重置)目录配额# gluster volume quota VOLNAME limit-usage /directory limit-value#gluster volume quota dht_vol limit-usage /quota 10GB/如上，设置dht_vol 卷下的 quota 子目录的限额为 10GB。PS：这个目录是以系统挂载目录为根目录”/”,所以/quota 即客户端挂载目录下的子目录 quota配额查看# gluster volume quota VOLNAME list# gluster volume quota VOLNAME list /directory name/可以使用如上两个命令进行系统卷的配额查看，第一个命令查看目的卷的所有配额设置，第二个命令则是执行目录进行查看。/可以显示配额大小及当前使用容量，若无使