附件a-sf存储故障后镜像重新同步

1、项目 SF故障后镜像重新同步日期：2012 年 3 月 10 日项目经理：主机工程师：、摘要本文档描述了复的过程。项目，在两套镜像后，有出现故障后，SF镜像重新同步数据库并恢1场景目前台州工商数据库是存放在电信机房HDS AMS 2300和台州工商IBM DS4700两套组成的逻辑镜像上，这样数据库空间有冗余的功能，当其中一套出现故障后，RAC 数据库还能正常工作，但是恢复后，需要对镜像盘做底层的重新同步。2处理步骤主要步骤1)1、 当磁盘阵列已恢复完毕后，先做重新扫盘操作：# cfgmgr v 2、 查看磁盘状态：#lsdev -Cc disk 或 vxdisk -e list2、SF 扫描

2、设备：# vxdctl enable3、将磁盘重新加入到 dg：# vxreattach4、执行完毕后，使用 vxdisk list 命令查看不在有 failed 的磁盘（应该为 online） 5、修复卷信息并做增量同步：# vxrecover -sb6、可以通过命令查看同步进度：# vxtask list2)执行过程roottzgs1:/#vxdctl enable roottzgs1:/#vxreattach roottzgs1:/#vxdisk listDEVICETYPEDISKGROUPSUSams_23000_0 auto:cdsdisk ams_23000_0 datadg o

3、nline shared ams_23000_1 auto:cdsdisk ams_23000_1 datadgonline shared ams_23000_2 auto:cdsdisk ams_23000_2 datadgonline shared ams_23000_3 auto:cdsdisk ams_23000_3 datadgonline shared ams_23000_4 auto:cdsdisk -onlineams_23000_5 auto:cdsdisk -onlineams_23000_6auto:cdsdisk-online ams_23000_7auto:cdsdi

4、sk ams_23000_7ocrvotedgonline shared disk_0auto:LVM- LVMdisk_1auto:LVM-LVM ds4700-0_0auto:cdsdiskds4700-0_0datadgonline shared ds4700-0_1auto:cdsdiskds4700-0_1datadgonline shared ds4700-0_4auto:cdsdiskds4700-0_4datadgonline shared ds4700-0_5auto:cdsdiskds4700-0_5datadgonline shared ds4700-0_8auto:cd

5、sdiskds4700-0_8ocrvotedgonline sharedroottzgs1:/#vxrecover -sb当 vxtask list 执行后没有输出内容就表示同步已经完成，否则可以看到同步进度。同步完成后，再查看磁盘状态，看到 sus 为 online 表示磁盘状态都正常。32012 年 3 月 21 日 故障故障现象1)小机维保公司 3.15 巡检的时候发现工商大楼机房的写入出错，列出磁盘状态如下：roottzgs1:/#vxdisk listDEVICETYPEDISKGROUPS USams_23000_0 auto:cdsdisk ams_23000_0 datadg

6、online shared ams_23000_1 auto:cdsdisk ams_23000_1 datadgonline shared ams_23000_2 auto:cdsdisk ams_23000_2 datadgonline shared ams_23000_3 auto:cdsdisk ams_23000_3 datadg online shared ams_23000_4 auto:cdsdisk -onlineams_23000_5 auto:cdsdisk -onlineams_23000_6auto:cdsdisk-online ams_23000_7auto:cds

7、disk ams_23000_7ocrvotedgonline shared disk_0auto:LVM- LVMdisk_1auto:LVM-LVMds4700-0_0 auto-error sharedroottzgs1:/#vxdisk listDEVICETYPEDISKGROUPS USams_23000_0 auto:cdsdisk ams_23000_0 datadgonline shared ams_23000_1 auto:cdsdisk ams_23000_1 datadg online shared ams_23000_2 auto:cdsdisk ams_23000_

8、2 datadgonline shared ams_23000_3 auto:cdsdisk ams_23000_3 datadgonline shared ams_23000_4 auto:cdsdisk -onlineams_23000_5 auto:cdsdisk -onlineams_23000_6auto:cdsdisk-online ams_23000_7auto:cdsdisk ams_23000_7ocrvotedgonline shared disk_0auto:LVM- LVMdisk_1auto:LVM-LVM ds4700-0_0auto:cdsdiskds4700-0

9、_0datadgonline shared ds4700-0_1auto:cdsdiskds4700-0_1datadgonline shared ds4700-0_4auto:cdsdiskds4700-0_4datadgonline shared ds4700-0_5auto:cdsdisk ds4700-0_5datadgonline sharedds4700-0_8auto:cdsdiskds4700-0_8ocrvotedgonline sharedroottzgs1:/#vxtask listTASKIDPTID TYPE/SEPCTPROGRESS168PARENT/R 50.0

10、0% 4/2(1) VXRECOVER171168ATCOPY/R 27.79% 0/798720000/221925376 PLXATT datavol datavol-02datadg smartmove2)故障原因分析经过分析尝试重新同步磁盘故障依旧，查看磁盘状态似乎都是正常的。确认出错的磁盘，测试内容状态：roottzgs1:/#vxdisk -e listDEVICETYPEDISKGROUPSUS OS_NATIVE_NAMEATTRams_23000_0auto:cdsdiskams_23000_0datadgonline sharedhdisk2 std fcams_23000

11、_1auto:cdsdiskams_23000_1datadgonline sharedhdisk3roottzgs1:/#lsdev -Cc diskhdisk0Available 03-08-00-5,0 16 Bit LVD SCSI Disk Drive hdisk1Available 03-08-00-8,0 16 Bit LVD SCSI Disk Drive hdisk2Available 00-08-01Hitachi Disk Array (Fibre) hdisk3Available 00-08-01Hitachi Disk Array (Fibre) hdisk4Avai

12、lable 00-08-01Hitachi Disk Array (Fibre) hdisk5Available 00-08-01Hitachi Disk Array (Fibre) hdisk6Available 00-08-01Hitachi Disk Array (Fibre) hdisk7Available 00-08-01Hitachi Disk Array (Fibre) hdisk8Available 00-08-01Hitachi Disk Array (Fibre) hdisk9Available 00-08-01Hitachi Disk Array (Fibre) hdis

13、k10 Available 0B-08-01 Hitachi Disk Array (Fibre) hdisk11 Available 0B-08-01 Hitachi Disk Array (Fibre) hdisk12 Available 0B-08-01 Hitachi Disk Array (Fibre) hdisk13 Available 0B-08-01 Hitachi Disk Array (Fibre) hdisk14 Available 0B-08-01 Hitachi Disk Array (Fibre) hdisk15 Available 0B-08-01 Hitachi

14、 Disk Array (Fibre) hdisk16 Available 0B-08-01 Hitachi Disk Array (Fibre) hdisk17 Available 0B-08-01 Hitachi Disk Array (Fibre) hdisk18 Available 00-08-01MPIO DS4700 Disk hdisk19 Available 00-08-01MPIO DS4700 Disk hdisk20 Available 00-08-01MPIO DS4700 Disk hdisk21 Available 00-08-01MPIO DS4700 Diskh

15、disk22 Available 00-08-01MPIO DS4700 Diskds4700-0_1 auto-error shared ds4700-0_4 auto-error shared ds4700-0_5 auto-error shared ds4700-0_8 auto-error shared-ds4700-0_0datadgfailed was:ds4700-0_0-ds4700-0_1datadgfailed was:ds4700-0_1-ds4700-0_4datadgfailed was:ds4700-0_4-ds4700-0_5datadgfailed was:ds

16、4700-0_5-ds4700-0_8ocrvotedgfailed was:ds4700-0_8可以看到 hdisk19-hdisk22 出现 error，使用 dd 命令错误磁盘内容，发现无法正常内容：初步判断，是电信大楼服务器无法向工商大楼 DS4700 中写入数据。3)故障处理结果现象处理时，发现级联 光纤不通，指示灯不亮，和电信沟通后，经过处理到 2012年 3 月 21 日下午 17:00 点左右， 光纤通过正常。roottzgs1:/#dd if=/dev/hdisk19 of=/hdisk19 bs=128K dd read error: No such device0+0 r

17、ecords in0+0 records outstd fcams_23000_2auto:cdsdiskams_23000_2datadgonline sharedhdisk4 std fcams_23000_3auto:cdsdiskams_23000_3datadgonline sharedhdisk5 std fcams_23000_4auto:cdsdisk-onlinehdisk6 std fcams_23000_5auto:cdsdisk-onlinehdisk7 std fcams_23000_6auto:cdsdisk-onlinehdisk8 std fcams_23000

18、_7auto:cdsdiskams_23000_7ocrvotedgonline sharedhdisk9 std fcdisk_0auto:LVM-LVMhdisk1-disk_1auto:LVM-LVMhdisk0-ds4700-0_0auto-error sharedhdisk19-ds4700-0_1auto-error sharedhdisk21-ds4700-0_4auto-error sharedhdisk18-ds4700-0_5auto-error sharedhdisk20-ds4700-0_8auto-error sharedhdisk22-ds4700-0_0datad

19、gfailed was:ds4700-0_0-ds4700-0_1datadgfailed was:ds4700-0_1-ds4700-0_4datadgfailed was:ds4700-0_4-ds4700-0_5datadgfailed was:ds4700-0_5-ds4700-0_8ocrvotedgfailed was:ds4700-0_8接下来重新扫描磁盘后，查看磁盘状态正常，重新开始同步数据，到 19:00 后再查看，数据同步已经完成，磁盘状态正常。roottzgs1:/#cfgmgr -v roottzgs1:/#vxdctl enable roottzgs1:/#vxrea

20、ttach roottzgs1:/#vxdisk listDEVICETYPEDISKGROUPSUSams_23000_0 auto:cdsdisk ams_23000_0 datadgonline shared ams_23000_1 auto:cdsdisk ams_23000_1 datadgonline shared ams_23000_2 auto:cdsdisk ams_23000_2 datadgonline shared ams_23000_3 auto:cdsdisk ams_23000_3 datadg online shared ams_23000_4 auto:cds

21、disk -onlineams_23000_5 auto:cdsdisk -onlineams_23000_6auto:cdsdisk-online ams_23000_7 auto:cdsdisk ams_23000_7ocrvotedgonline shared disk_0auto:LVM- LVMdisk_1auto:LVM-LVM ds4700-0_0auto:cdsdiskds4700-0_0datadgonline shared ds4700-0_1 auto:cdsdisk ds4700-0_1datadgonline shared ds4700-0_4auto:cdsdisk

22、ds4700-0_4datadgonline shared ds4700-0_5auto:cdsdiskds4700-0_5datadgonline shared ds4700-0_8auto:cdsdiskds4700-0_8ocrvotedgonline shared roottzgs1:/#vxrecover -sbroottzgs1:/#vxtask listTASKIDPTID TYPE/SEPCTPROGRESS /数据同步任务正在进行178PARENT/R 25.00% 4/1(2) VXRECOVER181178ATCOPY/R 00.07% 0/798720000/559104 PLXATT datavol datavol-02datadg smartmove roottzgs1:/ roottzgs1:/#vxtask listTASKIDPTID TYPE/SEPCTPROGRESS/到这里同步任务已经结束，同步完成