多节点读写性能测试

1、测试环境：glustefs端：Glusterfs版本：glusterfs 3.6.1 built on Nov 7 2014 15:15:50系统版本：CentOS release 6.5 (Final)内核：硬件环境：CPU：Intel(R) Xeon(R) CPU E3-1230 V2 3.30GHz内存：32GB硬盘：4块128GSSD 11块2T SATA盘client端：Glusterfs版本：glusterfs 3.6.1 built on Nov 7 2014 15:15:50系统版本：CentOS release 6.5 (Final)内核：硬件环境：CPU：Intel(R)

2、Xeon(R) CPU E3-1230 V2 3.30GHz内存：8GB网络环境：gluster端：服务器网络配置为bridge，单根网线接到交换机上。data-node3ip设为，data-node4ip设为，data-node5ip设为，data-node6ip设为10.3.211.6。client端：eth0设置固定，eth1、eth2、eth3三个网口做bond0，ip设为 ，四个网口四根网线连接到交换机上。系统节点：IP 别名 Brick data-node3 /brick/brick1/ data-node4 /brick/brick1/ 10.3.211.5 data-node5

3、 /brick/brick1/ 10.3.211.6 data-node6 /brick/brick1/ 192.168.10.200 data-client测试目的： 测试不同节点数下使用glusterfs建立stripe卷 挂载到3网口做bond0的client端上，测试他的读写性能 测试方法测试工具： dd fio测试方法：不同节点数下建立stripe卷，挂载到client端使用dd命令测试他的吞吐量,fio测试iops测试过程测试步骤1、添加信任池gluster peer probe data-node3gluster peer probe data-node4gluster peer

4、 probe data-node5gluster peer probe data-node62、创建stripe卷 两个节点gluster volume create test stripe 2 data-node4:/brick/brick1/testw data-node3:/brick/brick1/testw 三个节点gluster volume create test1 stripe 3 data-node4:/brick/brick1/test data-node3:/brick/brick1/test data-node5:/brick/brick1/test四个节点gluste

5、r volume create test2 stripe 4 data-node4:/brick/brick1/teste data-node3:/brick/brick1/teste data-node5:/brick/brick1/teste data-node6:/brick/brick1/teste3、启动stripe卷gluster volume start testgluster volume start test1gluster volume start test25、挂载到client端mount.glusterfs data-node3:/test /mnt6、使用dd命令测

6、试吞吐率dd if=/dev/zero of=/mnt/dd.img bs=1M count=600007、使用fio测试iops测试脚本#!/bin/bashcd /mntsleep 10rm -rf test*sleep 30cd /sleep 10/usr/local/bin/fio -name=/mnt/test -direct=0 -iodepth=96 -rw=randread -ioengine=posixaio -bs=4k -size=16G -numjobs=20 -runtime=3600 -group_reporting >> /fiotest/fio.lo

7、gsleep 100cd /mntsleep 60rm -rf test*sleep 300cd /sleep 60umount testpoolsleep 100mount -t glusterfs data-node3:/test /mnt/sleep 60/usr/local/bin/fio -name=/mnt/test -direct=0 -iodepth=96 -rw=randwrite -ioengine=posixaio -bs=4k -size=16G -numjobs=20 -runtime=3600 -group_reporting >> /fiotest/f

8、io.logsleep 100cd /mntsleep 60rm -rf test*sleep 300cd /sleep 60umount testpoolsleep 100mount -t glusterfs data-node3:/test /mnt/sleep 60/usr/local/bin/fio -name=/mnt/test -direct=0 -iodepth=96 -rw=randread -ioengine=posixaio -bs=8k -size=16G -numjobs=20 -runtime=3600 -group_reporting >> /fiote

9、st/fio.logsleep 100cd /mntsleep 60rm -rf test*sleep 300cd /sleep 60umount testpoolsleep 100mount -t glusterfs data-node3:/test /mnt/sleep 60/usr/local/bin/fio -name=/mnt/test -direct=0 -iodepth=96 -rw=randwrite -ioengine=posixaio -bs=8k -size=16G -numjobs=20 -runtime=3600 -group_reporting >> /

10、fiotest/fio.logsleep 100cd /mntsleep 60rm -rf test*sleep 300cd /sleep 60umount testpoolsleep 100mount -t glusterfs data-node3:/test /mnt/sleep 60/usr/local/bin/fio -name=/mnt/test -direct=0 -iodepth=96 -rw=randread -ioengine=posixaio -bs=16k -size=16G -numjobs=20 -runtime=3600 -group_reporting >&

11、gt; /fiotest/fio.logsleep 100cd /mntsleep 60rm -rf test*sleep 300cd /sleep 60umount testpoolsleep 100mount -t glusterfs data-node3:/test /mnt/sleep 60/usr/local/bin/fio -name=/mnt/test -direct=0 -iodepth=96 -rw=randwrite -ioengine=posixaio -bs=16k -size=16G -numjobs=20 -runtime=3600 -group_reporting

12、 >> /fiotest/fio.logsleep 100cd /mntsleep 60rm -rf test*sleep 300cd /sleep 60umount testpoolsleep 100mount -t glusterfs data-node3:/test /mnt/sleep 60/usr/local/bin/fio -name=/mnt/test -direct=0 -iodepth=96 -rw=randread -ioengine=posixaio -bs=32k -size=16G -numjobs=20 -runtime=3600 -group_repo

13、rting >> /fiotest/fio.logsleep 100cd /mntsleep 60rm -rf test*sleep 300cd /sleep 60umount testpoolsleep 100mount -t glusterfs data-node3:/test /mnt/sleep 60/usr/local/bin/fio -name=/mnt/test -direct=0 -iodepth=96 -rw=randwrite -ioengine=posixaio -bs=32k -size=16G -numjobs=20 -runtime=3600 -grou

14、p_reporting >> /fiotest/fio.logsleep 100cd /mntsleep 60rm -rf test*sleep 300cd /sleep 60umount testpoolsleep 100mount -t glusterfs data-node3:/test /mnt/sleep 60/usr/local/bin/fio -name=/mnt/test -direct=0 -iodepth=96 -rw=randread -ioengine=posixaio -bs=64k -size=16G -numjobs=20 -runtime=3600

15、-group_reporting >> /fiotest/fio.logsleep 100cd /mntsleep 60rm -rf test*sleep 300cd /sleep 60umount testpoolsleep 100mount -t glusterfs data-node3:/test /mnt/sleep 60/usr/local/bin/fio -name=/mnt/test -direct=0 -iodepth=96 -rw=randwrite -ioengine=posixaio -bs=64k -size=16G -numjobs=20 -runtime

16、=3600 -group_reporting >> /fiotest/fio.logsleep 100cd /mntsleep 60rm -rf test*sleep 300cd /sleep 60umount testpoolsleep 100mount -t glusterfs data-node3:/test /mnt/sleep 60测试结果使用dd命令测试结果：dd if=/dev/zero of=/mnt/dd.img bs=1M count=60000节点数234吞吐率61.5 MB/s135 MB/s335 MB/s使用fio的测试结果：、1）随机读写的iops节点

17、数234读写类型块大小iops吞吐率iops吞吐率iops吞吐率randread4k2801122.3KB/s3601442.7KB/s7222888.1KB/s8k2181750.9 KB/s3532825.9KB/s6004802.6 KB/s16k2103369.5 KB/s3635818.6KB/s5598948.4 KB/s32k1936193.7 KB/s35011221KB/s54917581 KB/s64k19512483 KB/s35522782KB/s56536204KB/s节点数234读写类型块大小iops吞吐率iops吞吐率iops吞吐率randwrite4k41414

18、59.1 KB/s6742698.5KB/s13285314.7KB/s8k5564451.8 KB/s11048835.4KB/s206616532 KB/s16k3896234.7 KB/s74611945KB/s130320851 KB/s32k2788920.7 KB/s43914064KB/s69822346KB/s64k19812713 KB/s29819075KB/s42227039KB/siops randread 条形图iops randwrite 条形图2）顺序读写iops节点数234读写类型块大小iops吞吐率iops吞吐率iops吞吐率read4k1844273772KB/s1491659666KB/s26578106315KB/s8k844267539KB/s783662694KB/B/s16k421267394KB/s431569042KB/s6656106498KB/s32k213768396KB/s251280410KB/s3337106811KB/s64k766