SUSE-多路径Device-Mapper+Multipath

1、Device-Mapper Multipath一 DM-Multipath 概述DM-Multipath 能够使服务器与存储控制器间 multiple I/O路径变成一个单一的设备。I/O 路径是由线缆、交换机、控制器组成的物理SAN。 DM-Multipath能够创建一个由I/O路径聚集组成的新设备。在不配置DM-Multipath的情况下，盘阵的一个 LUN从控制器主机端口映射到服务器， 在操作系统里被识别成一个独立的设备，这样就会造成同一个LUN通过盘阵不同的主机端口映射到服务器被识别成不同的设备。作为一种解决方案，DM-Multipath 通过在物理设备上创建一个单独的多路径设备，提供

2、了一种在逻辑上管理I/O路径的机制，这样盘阵的LUN从控制器主机端口映射到服务器，在操作系统里被识别成一个多路径设备。每个多路径设备拥有一个唯一标识的World Wide Identifier （全球识别号，WWID）,默认情况下，多路径设备的名称被设置成它的WWID。通过修改multipath.conf文件中的user_friendly_names选项参数，可以设置多路径设备的别名为mpathn。例如如下的配置环境：一个拥有两个HBA的服务器通过一个未配置zone的FC交换机连接到有两个主机端口的盘阵控制器（盘阵仅有一个LUN ）上，在操作系统里能够看到四个设备：/dev/sda, /dev

3、/sdb,dev/sdc,和/dev/sdd。通过配置 multipath.conf 文件，DM-Multipath 就会创建一个拥有 WWID 的多路径设备，多路径设备受控于DM-Multipath，我们可以在三个不同目录查看多路径设备：/dev/目录；/dev/mapper/mpathn,/dev/mpath/mpathn ; /dev/dm-n.。使用 /dev/mapper 目录中 的设备名对多路径设备进行管理，如创建逻辑卷，创建文件系统等。/dev/mpath将所有的多路径设备放到该目录下，只是便于查看，不要使用该目录下的设备进行创建逻辑卷，创建文件系统等操作。/dev/dm-n只是

4、用于系统内部的使用，不要使用这里面的设备。二 DM-Multipath 功能DM-Multipath 能够提供：冗余在active/passive模式中，DM-Multipath提供失效转移功能。active/passive 环境中，在任何时间内只有一半的 I/O路径被使用。如果一个I/O路径内的元素例如线缆、交换机、控制器失效，DM-Multipath切换到轮换的路径。提高性能DM-Multipath 能够被配置在 active/ active模式下，在这种模式下，I/O 路径处于round-robin方式。DM-Multipath能够动态的平衡I/O负荷。三 DM-Multipath应用环

5、境示例 Active / Passive Multipath Configuration with One RAID Device该配置里服务器有两个 HBA，两个SAN交换机和两个控制器。从服务器到一个 RAID存在两个I/O路径。如下图：该配置中，第一条I/O路径通过hba1,SANl和controllerl 。第二条I/O路径通过hba2,SAN2和controller2。这个配置中存在多种潜在的失效因素：HBA failureFC cable failureSAN switch failureArray con troller port failure如果出现以上任何一个失效点，DM-

6、Multipath会切换到轮换的路径。 Active/Active Multipath Configuration with One RAID Device该配置里服务器有两个HBA，个SAN交换机和两个控制器，如下图：Serverhba 1hba 2SANcntrk 1cncrtr 2Storage该配置中，从服务器到一个存储存在四条I/O路径:hba1 to con troller1hbal to con troller2 hba2 to con trollerlhba2 to con troller2配置DM-Multipath后，DM-Multipath能够监测每条I/O路径的负载，动

7、态的平衡I/O负荷。四 Setting Up DM-Multipath在系统准备 DM-Multipath 之前，请确保系统包含 device-mapper-multipath包，操作系统安装 光盘内包含该rpm包。 编辑/etc/multipath.conf文件，将以下内容的注释移除，使 multipathing功能开启。 blacklist devnode *修改完应如下# blacklist # devnode * # 保存配置 modprobe dm-multipathservice multipathd startmultipath -/2/ multipath -/2命令输出多路径

8、设备，如果没有任何输出， 确保SAN正确连接。 chkconfig multipathd on / 确保 multipath daemon 开机自动启动 本地的硬盘设备不建议使用DM-Multipath，需要将本地硬盘从多路径设备中忽略。有两种方式，方式一：vi /etc/multipath.c onf将 blacklist devnode *修改成blacklist devnode Asda/扫描除本地硬盘的多路径设备方式二:/etc/multipath.conf 文件中添加blacklistwwid SIBM-ESXSST336732LCF3ET0EP0Q000072428BX1（该值为本

9、地硬盘的WWID）然后再运行multipath -multipath-2rootrh4cluster1 # multipath -F移除多路径设备rootrh4cluster1 #multipath -v2create: 3600a0b80001327d80000006d43621677 size=12 GBfeatures=0hwha ndler=0 _ roun d-rob in 0_ 2:0:0:0 sdb 8:16_ 3:0:0:0 sdf 8:80create: 3600a0b80001327510000009a436215ecsize=12 GBfeatures=0hwha ndl

10、er=0_ roun d-rob in 0_ 2:0:0:1 sdc 8:32_ 3:0:0:1 sdg 8:96create: 3600a0b80001327d800000070436216b3 size=12 GBfeatures=0hwha ndler=0_ roun d-rob in 0_ 2:0:0:2 sdd 8:48_ 3:0:0:2 sdh 8:112create: 3600a0b80001327510000009b4362163esize=12 GBfeatures=0hwha ndler=0_ roun d-rob in 0_ 2:0:0:3 sde 8:64_ 3:0:0

11、:3 sdi 8:128大多数的磁盘阵列型号支持DM-Multipath ,如果要添加的设备不是 /etc/multipath.conf文件里默认支持的设备，可以参照默认配置添加设备情况，例如添加一个如下的设备：devices device ven dor HPproduct OPEN-V.getuid_callout /sb in/scsi_id -g -u -p0x80 -s/block/% n附：北大某学院项目实际案例:项目实施过程中由于要实现光纤链路的多路径，由于DS6310FE没有提供LINUX下的多路径冗余软件，所以采用了LINUX系统下的device map per multip

12、ath解决方案。SAN组织结构如下：DS6310FE :双控制器，每个控制器2个4G FC主机端口，全部连接到FC交换机。4组RAID, 其中一个LUN同时映射给三个主机，其它三个LUN分别映射给各自的主机。Brocade 200E FC SwitchServer：每个配置一个 Qlogic QLE2460 HBA 卡，连接交换机这样LUN到主机的光纤链路存在多个。以下为配置步骤 # rpm -qa | grep device-mapper device-mapper-1.02.13-1.el5 device-mapper-multipath-0.4.7-8.el5如果显示如上说明系统已经安装

13、了device-mapper-multipath，如果未安装，在安装光盘可以找到该rpm包进行安装# rpm -ivh device-mapper-multipath-0.4.7-8.el5.i386.rpmPreparing. # 100% 1:device-mapper-multipath# 100%# 配置 /etc/multipath.conf 文件进行备份，cp /etc/multipath.c onf /etc/multipath.c on f.bakvi /etc/multipath.c onf将 devnode_blacklist devnode *修改成dev no de_b

14、lacklist devnode Asda/扫描除本地硬盘的多路径设备 执行 # multipath -v3load path ide ntifiers cache# all paths in cache : sda blacklisted222770001551df74d222f5000155f4aab4222770001551df74d222f5000155f4aab4222770001551df74d6:0:0:0 sdb 8:16 1 active Daw ning /Diskarray DS6310/0330 6:0:0:1 sdc 8:32 1 active Daw ning /D

15、iskarray DS6310/03306:0:1:0 sdd 8:48 1 active Daw ning /Diskarray DS6310/0330 6:0:1:1 sde 8:64 1 active Daw ning /Diskarray DS6310/03306:0:2:0 sdf 8:80 1 active Daw ning /Diskarray DS6310/0330222f5000155f4aab4222770001551df74d222f5000155f4aab46:0:2:1 sdg 8:96 1 active Daw ning /Diskarray DS6310/0330

16、6:0:3:0 sdh 8:112 1 active Daw ning /Diskarray DS6310/03306:0:3:1 sdi 8:128 1 active Daw ning /Diskarray DS6310/0330path info sdb (mask 0x1f) bus = 1 dev t = 8:16 size = 5847653376 ven dor = Dawningproduct = Diskarray DS6310rev = 0330h:b:t:l = 6:0:0:0tgt_node_n ame =serial = 495345200000000000000000

17、972AA3D84BF39EF4 path checker = readsectorO (internal default)state = 2getprio = (n ull) (internal default)prio = 1uid = 222770001551df74d (cache)= path info sdc (mask 0x1f) =bus = 1dev_t = 8:32size = 1949217792ven dor = Dawningproduct = Diskarray DS6310rev = 0330h:b:t:l = 6:0:0:1tgt_node_n ame =ser

18、ial = 4953452000000000000000006B1ED98B16A2FB26path checker = readsectorO (internal default)state = 2getprio = (n ull) (internal default)prio = 1uid = 222f5000155f4aab4 (cache)= path info sdd (mask 0x1f) =bus = 1dev_t = 8:48size = 5847653376ven dor = Dawningproduct = Diskarray DS6310rev = 0330h:b:t:l

19、 = 6:0:1:0tgt_node_n ame =serial = 495345200000000000000000972AA3D84BF39EF4path checker = readsector0 (internal default)state = 2getprio = (n ull) (internal default)prio = 1uid = 222770001551df74d (cache)= path info sde (mask 0x1f) =bus = 1dev_t = 8:64size = 1949217792ven dor = Dawningproduct = Disk

20、array DS6310rev = 0330h:b:t:l = 6:0:1:1tgt_node_n ame =serial = 4953452000000000000000006B1ED98B16A2FB26path checker = readsector0 (internal default)state = 2getprio = (n ull) (internal default)prio = 1uid = 222f5000155f4aab4 (cache)= path info sdf (mask 0x1f) =bus = 1dev_t = 8:80size = 5847653376ve

21、n dor = Dawningproduct = Diskarray DS6310rev = 0330h:b:t:l = 6:0:2:0tgt_node_n ame =serial = 495345200000000000000000972AA3D84BF39EF4path checker = readsector0 (internal default)state = 2getprio = (n ull) (internal default)prio = 1uid = 222770001551df74d (cache)= path info sdg (mask 0x1f) =bus = 1de

22、v_t = 8:96size = 1949217792ven dor = Dawningproduct = Diskarray DS6310rev = 0330h:b:t:l = 6:0:2:1tgt_node_n ame =serial = 4953452000000000000000006B1ED98B16A2FB26path checker = readsector0 (internal default)state = 2getprio = (n ull) (internal default)prio = 1uid = 222f5000155f4aab4 (cache)= path in

23、fo sdh (mask 0x1f) =bus = 1dev_t = 8:112size = 5847653376ven dor = Dawningproduct = Diskarray DS6310rev = 0330h:b:t:l = 6:0:3:0tgt_node_n ame =serial = 495345200000000000000000972AA3D84BF39EF4path checker = readsector0 (internal default)state = 2getprio = (n ull) (internal default)prio = 1uid = 2227

24、70001551df74d (cache)= path info sdi (mask 0x1f) =bus = 1dev_t = 8:128size = 1949217792ven dor = Dawningproduct = Diskarray DS6310rev = 0330h:b:t:l = 6:0:3:1tgt_node_n ame =serial = 4953452000000000000000006B1ED98B16A2FB26path checker = readsectorO (internal default)state = 2getprio = (n ull) (inter

25、nal default)prio = 1uid = 222f5000155f4aab4 (cache)# all paths :#222770001551df74d6:0:0:0 sdb 8:16 1 activeready Daw ning /Diskarray DS631222f5000155f4aab46:0:0:1 sdc 8:32 1 activeready Daw ning /Diskarray DS631222770001551df74d6:0:1:0 sdd 8:48 1 activeready Daw ning /Diskarray DS631222f5000155f4aab

26、46:0:1:1 sde 8:64 1 activeready Daw ning /Diskarray DS631222770001551df74d6:0:2:0 sdf 8:80 1 activeready Daw ning /Diskarray DS631222f5000155f4aab46:0:2:1 sdg 8:96 1 activeready Daw ning /Diskarray DS631222770001551df74d6:0:3:0 sdh 8:112 1 activeready Daw ning /Diskarray DS63222f5000155f4aab46:0:3:1

27、 sdi 8:128 1 activeready Daw ning /Diskarray DS63params = 0 0 4 1 rou nd-robin 0 1 1 8:32 1000 rou nd-robin 0 1 1 8:64 1000 roun d-robin 0 1 1 8:96 1000 rou nd-robin 0 1 1 8:128 1000status = 1 0 0 4 1 A 0 1 0 8:32 A 0 E 0 1 0 8:64 A 0 E 0 1 0 8:96 A 0 E 0 1 0 8:128 A 0params = 0 0 4 1 rou nd-robin 0

28、 1 1 8:16 1000 rou nd-robin 0 1 1 8:48 1000 roun d-robin 0 1 1 8:80 1000 rou nd-robin 0 1 1 8:112 1000 status = 1 0 0 4 1 A 0 1 0 8:16 A 0 E 0 1 0 8:48 A 0 E 0 1 0 8:80 A 0 E 0 1 0 8:112 A 0 Fou nd matchi ng wwid 222770001551df74d in bi ndi ngs file.Setti ng alias to mpath0pgpolicy = failover (inter

29、nal default)selector = roun d-rob in 0 (internal default)features = 0 (internal default)hwha ndler = 0 (internal default)rr_weight = 1 (internal default)rr_min_io = 1000 (config file default)no_path_retry = NONE (in ternal default)pg_timeout = NONE (in ternal default)0 5847653376 multipath 0 0 4 1 r

30、ound-robin 0 1 1 8:16 1000 round-robin 0 1 1 8:48 1000 rou nd-robin 0 1 1 8:80 1000 roun d-robin 0 1 1 8:112 1000set ACT_NOTHING: map un cha ngedFou nd matchi ng wwid 222f5000155f4aab4 in bi ndings file.Sett ing alias to mpath1pgpolicy = failover (internal default)selector = roun d-rob in 0 (interna

31、l default)features = 0 (internal default)hwha ndler = 0 (internal default)rr_weight = 1 (internal default)rr_m in_io = 1000 (config file default)no_path_retry = NONE (in ternal default) pg_timeout = NONE (in ternal default)0 1949217792 multipath 0 0 4 1 round-robin 0 1 1 8:32 1000 round-robin 0 1 1

32、8:64 1000 rou nd-robin 0 1 1 8:96 1000 roun d-robin 0 1 1 8:128 1000set ACT_NOTHING: map un cha nged显示出所有的多路径设备，从sdb到sdi共8个分区，以上的信息尤其是红色标明的是需要添加到/etc/multipath.conf的重要信息。 执行 # multipath -llmpath1 (222f5000155f4aab4)size=929 GBfeatures=0hwha ndler=0_ roun d-rob in 0 prio=1active_ 6:0:0:1 sdc 8:32 act

33、iveready_ roun d-rob in 0 prio=1e nabled_ 6:0:1:1 sde 8:64 activeready_ roun d-rob in 0 prio=1e nabled_ 6:0:2:1 sdg 8:96 activeready_ roun d-rob in 0 prio=1e nabled_ 6:0:3:1 sdi 8:128 activereadympath0 (222770001551df74d)size=2788 GBfeatures=0hwha ndler=0_ roun d-rob in 0 prio=1active_ 6:0:0:0 sdb 8

34、:16 activeready_ roun d-rob in 0 prio=1e nabled_ 6:0:1:0 sdd 8:48 activeready_ roun d-rob in 0 prio=1e nabled_ 6:0:2:0 sdf 8:80 activeready_ roun d-rob in 0 prio=1e nabled_ 6:0:3:0 sdh 8:112 activeready显示未作多路径冗余时的状态。 加载 dm_multipath 模块 modprobe dm_multipath打开守护进程 /etc/init.d/multipathd startchkconfi

35、g multipathd on 配置 /etc/multipath.conf 文件# This is an example con figurati on file for device mapper multipath.# For a complete list of the default con figurati on values, see# /usr/share/doc/device-mapper-multipath-0.4.5/multipath.c on f.defaults# For a list of con figurati on optio ns with descrip

36、ti ons, see# /usr/share/doc/device-mapper-multipath-0.4.5/multipath.c on f.a nno tated# Blacklist all devices by default. Remove this to en able multipath ing# on the default devices.dev no de_blacklist devnode Asda# Use user frien dly n ames, in stead of using WWIDs as n ames.defaults user_frie ndl

37、y_ names yes# By default, devices with ven dor = IBM and product = S/390.* are # blacklisted. To en able mulitpath ing on these devies, un comme nt the # followi ng lin es.#devices #device #vendorIBM#productS/390 DASD ECKD#path_group in g_policymultibus#getuid_callout/sbi n/dasdview -j -f /dev/% n#p

38、ath_checkerdirectio# This is a template multipath-tools con figurati on file# Un comme nt the lines releve nt to your environment# #defaults /dev#udev_dir#polli ngn terval10#selectorroun d-rob in#path_group in g_policymultibus#getuid_callout/sbi n/scsi_id#prio_callout/bin/true#path_checkerreadsector

39、O#rr_min_io100#rr_weightpriorities#failbackimmediate#no _path_retryfail#user_frie ndly_ nameyes0-g -u -s /block/% n# The wwid line in the followi ng blacklist sect ion is show n as an example # of how to blacklist devices by wwid.The 3 dev node lines are the# compiled in default blacklist. If you wa

40、nt to blacklist en tire types # of devices, such as all scsi devices, you should use a dev node line.# However, if you want to blacklist specific devices, you should use# a wwid line. Since there is no guara ntee that a specific device will # not cha nge n ames on reboot (from /dev/sda to /dev/sdb f

41、or example) # dev node lines are not recomme nded for blacklisti ng specific devices. # #dev no de_blacklist #wwid 26353900f02796769# devnode A(ram|raw|loop|fd|md|dm-|sr|scd|st)0-9*devnode Ahda-zmultipath wwid# multipaths #aliaspath_group in g_policy path_checker path_selector failback rr_weight no

42、_path_retrymultipath wwid alias3600508b4000156d700012000000b0000 yellowmultibusreadsectorOroun d-rob in 0manualpriorities51DEC321816758474redmultipath 222770001551df74ddatamultibustureroun d-rob in 0222f5000155f4aab4 scratchmultibustureroun d-rob in 0wwidalias path_group in g_policy path_checker pat

43、h_selectormultipath wwidalias path_group in g_policy path_checker path_selectordevices #device #vendorCOMPAQ #productHSV110 (C)COMPAQ#path_group in g_policymultibus#getuid_callout/sbi n/scsi_id -g -u -s /block/% n#path_checkerreadsector0#path_selectorro un d-rob in 0#hardware_ha ndler0#failback15#rr

44、_weightpriorities#no _path_retryqueue#device #vendorCOMPAQ #productMSA1000#path_group in g_policymultibus#device vendorDaw ning productDiskarray DS6310#path_group in g_policymultibuspath_group in g_policygroup_by_serialgetuid_callout path_checker path_selector hardware_ha ndler failback rr_weight n

45、o_path_retry/sbi n/scsi_id -g -u -s /block/% n readsectorOroun d-rob in 00immediateuniform12红色为改动部分执行 multipath -v3 load path ide ntifiers cache # all paths in cache :#222770001551df74d6:0:0:0 sdb 8:16 1 active Daw ning /Diskarray DS6310/0330222f5000155f4aab4222770001551df74d6:0:0:1 sdc 8:32 1 activ

46、e Daw ning /Diskarray DS6310/03306:0:1:0 sdd 8:48 1 active Daw ning /Diskarray DS6310/0330222f5000155f4aab46:0:1:1 sde 8:64 1 active Daw ning /Diskarray DS6310/0330222770001551df74d6:0:2:0 sdf 8:80 1 active Daw ning /Diskarray DS6310/0330222f5000155f4aab46:0:2:1 sdg 8:96 1 active Daw ning /Diskarray

47、 DS6310/0330222770001551df74d6:0:3:0 sdh 8:112 1 active Daw ning /Diskarray DS6310/0330222f5000155f4aab46:0:3:1 sdi 8:128 1 active Daw ning /Diskarray DS6310/0330sda blacklisted= path info sdb (mask 0x1f) =bus = 1dev_t = 8:16size = 5847653376ven dor = Dawningproduct = Diskarray DS6310rev = 0330h:b:t

48、:l = 6:0:0:0tgt_node_n ame =serial = 495345200000000000000000972AA3D84BF39EF4path checker = readsector0 (con troler sett ing)state = 2getprio = (n ull) (internal default)prio = 1uid = 222770001551df74d (cache)= path info sdc (mask 0x1f) =bus = 1dev_t = 8:32size = 1949217792ven dor = Dawningproduct =

49、 Diskarray DS6310rev = 0330h:b:t:l = 6:0:0:1tgt_node_n ame =serial = 4953452000000000000000006B1ED98B16A2FB26path checker = readsectorO (con troler sett ing)state = 2getprio = (n ull) (internal default)prio = 1uid = 222f5000155f4aab4 (cache)= path info sdd (mask 0x1f) =bus = 1dev_t = 8:48size = 5847

50、653376ven dor = Dawningproduct = Diskarray DS6310rev = 0330h:b:t:l = 6:0:1:0tgt_node_n ame =serial = 495345200000000000000000972AA3D84BF39EF4path checker = readsector0 (con troler sett ing)state = 2getprio = (n ull) (internal default)prio = 1uid = 222770001551df74d (cache)= path info sde (mask 0x1f)

51、 =bus = 1dev_t = 8:64size = 1949217792ven dor = Dawningproduct = Diskarray DS6310rev = 0330h:b:t:l = 6:0:1:1tgt_node_n ame =serial = 4953452000000000000000006B1ED98B16A2FB26path checker = readsector0 (con troler sett ing)state = 2getprio = (n ull) (internal default)prio = 1uid = 222f5000155f4aab4 (c

52、ache)= path info sdf (mask 0x1f) =bus = 1dev_t = 8:80size = 5847653376ven dor = Dawningproduct = Diskarray DS6310rev = 0330h:b:t:l = 6:0:2:0tgt_node_n ame =serial = 495345200000000000000000972AA3D84BF39EF4path checker = readsectorO (con troler sett ing)state = 2getprio = (n ull) (internal default)pr

53、io = 1uid = 222770001551df74d (cache)= path info sdg (mask 0x1f) =bus = 1dev_t = 8:96size = 1949217792ven dor = Dawningproduct = Diskarray DS6310rev = 0330h:b:t:l = 6:0:2:1tgt_node_n ame =serial = 4953452000000000000000006B1ED98B16A2FB26path checker = readsector0 (con troler sett ing)state = 2getprio = (n ull) (internal default)prio = 1uid = 222f5000155f4aab4 (cache)= path info sdh (mask 0x1f) =bus = 1dev_t = 8:112size =