EMC Symmetrix 监控方法

1、EMC Symmetrix DMX 日常监控方法客户操作及代码说明EMC客户服务支持团队1/8/2010本文档仅供EMC客户使用，未经EMC公司授权，不得复制、转载、出版和传播EMC公司推荐客户使用EMC 远程监控及支持系统（包括call home，Webex、ESRS等多种模式），客户也可以通过本文中提及的方法，实现对EMC Symmetrix DMX系列产品的监控。一、 在主机上建立或更新存储设备信息：这个操作在监控主机上执行一次后，将会在监控主机上建立存储设备信息的数据库，以后只有在存储设备配置变更后，才需要重新执行，用来更新存储配置信息。登录到安装有Solutions Enabler的

2、主机，执行以下命令(需要一至二分钟)：symcfg discover Solutions Enabler安装目录UNIX： /usr/symcli/binWindows： C:Program FilesEMCSYMCLIbin二、使用以下命令列出该主机所连接的存储信息：symcfg list列出主机所连接的存储，包括local和remote，输出示例如下： S Y M M E T R I X Mcode Cache Num Phys Num Symm SymmID Attachment Model Version Size (MB) Devices Devices 000290xxxx26 L

3、ocal DMX4 5773 65536 0 1042 000290xxxx82 Remote DMX4 5773 65536 0 353三、查看存储设备的日志信息：symevent -sid <sn> -v list -start <start date> -end <end date> 收集序列号为SN的盘阵在start date到end date之间的日志信息。例如：symevent -sid 19 list -start 12/20/2009 此命令为收集序列号尾数为19的盘阵在2009年12月20日以来的日志信息。如果仅有以下输出：Symmetri

4、x ID: 000290xxxx19则表明存储系统在此期间没有Event出现。如果输出示例如下：Symmetrix ID: 000290xxxx19Time Zone : GMT Standard TimeEvent at Tue Feb 24 08:20:18 2009: Reporting Director : DF-1A Source : Service Processor Category : Environment Severity : Error Numeric Code : 0x006a Event Code Symbol : SYMAPI_SEVT_DAE_PSB_FAULTE

5、D Description : Environmental Error: Power Supply B faulted 需要关注以下几项：Detection time : 日志时间，为存储设备时间，可能与本地实际时间不同。Reporting Director : 事件是从哪个director报出来的Severity : 事件等级，分为information,warning,error,fatal。Event Code Symbol : 事件代码缩写，如上例，可用DAE_PSB_FAULTED在下面的ERROR CODE列表中查找Description : 事件描述，上例中可知道是连接到dire

6、ctor 1A的DAE的Power supply B故障。上面事件表明在此期间有DAE_PSB_FAULTED的Error Code出现，同一个事件代码可能会同时通过多个部件报出来，可以收集下面第四点所说的收集相关的命令输出，与EMC联系，进行进一步检查。常见的Event Code可查阅本文附件。四、发现Error Code / Event Code后的处理：请客户致电EMC服务支持热线800 819 0009（手机用户可拨打 400 670 0009），提供机器序列号， 报告故障现象并提供以下的命令输出配合故障诊断。（以下命令针对序列号尾数为19的存储）symevent -sid 19 li

7、st -v -start 10/30/2009:9:00 -end 10/30/2009:17:00symcfg -sid 19 v list -env_datasymcfg -sid 19 v list -env_data -service_state failedsymcfg -sid 19 v list -env_data -service_state degradedsymdev -sid 19 v list -hotspare symdisk -sid 19 -v -spare_info -hotspare listEMC公司收到客户的信息及资料后，会立即安排技术支持工程师进行分析处

8、理， 随后会安排工程师到客户现场进行进一步处理工作，现场工作可能包括数据采集、错误修复、部件更换等。本文附件列举了大量的Event Code及其对应的说明，常见的告警及错误都在列表中，如果发现有不在列表中的Event Code，请拨打800或者400热线向EMC技术支持中心了解详细信息。五、输出存储设备日志到文件部分客户系统环境中安装有客户自己开发的监控系统或者第三方公司的监控系统，客户如果需要让监控系统读到设备日志，可以利用以下这个命令：symevent -sid SID list -start mm/dd/yyyy:hh:mm:ss -end mm/dd/yyyy:hh:mm:ss -v

9、-warn > symevent_SID_yyyymmdd.log 这里举个例子，例如需要列举设备序列号尾数为1440的存储设备日志，时间范围是2009/12/24 22：00到第二天23：00  symevent -sid 1440 list -start 12/24/2009:22:00:00 -end 12/25/2009:23:00:00 -v -warn > symevent_1440_20091225.log那么就得到一个25小时范围的存储设备日志文件，文件名是symevent_1440_20091225.lo

10、g。文件名可以由客户在输入命令时自行设定。附件：常见Error Code / Event CodeHex Code Severity Alert Description 12_VOLTS_ONWarningOne of the Symmetrix subsystems is running in the abnormal 12-VoltsMode；某一子部件的12V电源不正常，需要进一步检查，可能同时会有其它代码产生AC_LINE_INTERRUPTEDWarningA Symmetrix power subsystem AC line interruption was detected；机柜

11、输入电源故障或电源模块故障。ACCESS_TO_NR_DEVICEWarningAccess was attempted to a Not Ready device；主机尝试访问Not ready设备ALARM_SIGNALWarningAn alarm signal was set but no alarm was found；有警告信号但没发现告警信息，需要手工检查。ALARM_SIGNAL_POWERWarningAn alarm signal was set indicating a power subsystem error；电源告警。ALL_DEVICES_MIGRATEDInfo

12、rmationalAll Symmetrix migration devices have completed the data migrationALL_RDF_LINKS_DOWNWarningNo RDF links in an RDF group are operational；RDFSRDF链路中断ALL_RDF_LINKS_NOW_UPInformationalAll RDF links in an RDF group are now operational after an 'All LinksDown' event；RDFSRDF链路中断后恢复BATTERY_F

13、AILED_TESTWarningAutomatic battery tests detected failures；自动电源测试失败BUS_PROBLEMInformationalBus Arbiter problem: primary arbiter has experienced a problemCANT_QUERY_MII_DIRWarningThe Service Processor could not query a director；Service Processor不能联系到directorCANT_READ_ENVIR_SENSORWarningThe Service Pr

14、ocessor failed to read an environmental sensor；Service Processor读不到环境传感器信息。COMM_BOARD_MISMATCHErrorThe Symmetrix communication board software data has a mismatch；通信卡信息不一致。DAE_PSA_M_FAN_FAULTErrorEnvironmental Error: Power Supply A multiple Fan fault；电源 A 多个风扇故障。DAE_PSA_S_FAN_FAULTErrorEnvironmental

15、Error: Power Supply A single Fan fault；电源 A 单个风扇故障。DAE_PSA_FAULTEDErrorEnvironmental Error: Power Supply A faulted；电源 A故障或者相应的电池故障。DAE_PSA_SHUTDOWNErrorEnvironmental Error: Power Supply A shutdown；电源 A故障，电源线松动或相应的电池故障DAE_PSB_M_FAN_FAULTErrorEnvironmental Error: Power Supply B multiple Fan fault；电源 B

16、多个风扇故障。DAE_PSB_S_FAN_FAULTErrorEnvironmental Error: Power Supply B single Fan fault；电源 B单个风扇故障。DAE_PSB_FAULTEDErrorEnvironmental Error: Power Supply B faulted；电源 B故障相应的电池故障。DAE_PSB_SHUTDOWNErrorEnvironmental Error: Power Supply B shutdown；电源 B故障，电源线松动或相应的电池故障DAE_LCC_A_TEMP_HIGHErrorEnvironmental Err

17、or: Link Card Controller A temperature high；LCC A温度高DAE_LCC_B_TEMP_HIGHErrorEnvironmental Error: Link Card Controller B temperature high；LCC B温度高DAE_SPS_INTRN_FAULTErrorEnvironmental Error: Supplemental Power Supply internal fault；电池SPS 故障DAE_SPS_BAT_ENDLINEErrorEnvironmental Error: Supplemental Pow

18、er Supply battery end of line；电池SPS 寿命到期，电池需要更换。DAE_SPS_LOW_VOLTAGEErrorEnvironmental Error: Supplemental Power Supply low input ACVoltage；电池SPS输入电源电压低DB_CHECKSUM_TRIGGERInformationalA Database Double Checksum detection event was triggeredDEVICE_RESYNC_STARTEDInformationalA Symmetrix device resynchr

19、onization process has startedDIAG_TRACE_TRIGInformationalA Symmetrix diagnostic event-trace was triggeredDIAG_TRACE_TRIG_REMOTEInformationalA diagnostic event-trace was triggered for a Symmetrixremotely-attached via RDF linksDIRECTOR_DEADFatalA Symmetrix Director is not responding；一块director没有响应DISA

20、BLED_MEMORY_BANKWarningA Symmetrix Director reported 'Disabled Memory Bank' to a host；director发现保留内存区DISK_ADAPTER_DEADFatalA Symmetrix Disk Director is not responding；一块disk director没有响应ENABLED_ENVIR_TESTINGInformationalAn event was detected to enable environmental testing in diagnosis ModeE

21、NVIR_READING_OUT_OF_LIMITWarningThe Service Processor found environmental readings to be out of Limits；Service Processor 发现环境指标将要超限EXCESS_TEMP_DETECTEDWarningThe Service Processor detected excessive temperature；Service Processor 发现温度过高告警FC_OPTICAL_MOD_ERRORWarningA Fibre Channel optical module has e

22、xperienced a problem；FA端口SFP有问题GEN_CHECKSUM_TRIGGERInformationalA Generic Double Checksum detection event was triggered.HIGH_CHARGE_MISSINGWarningThe Symmetrix battery system is not fully charged；电池SPS没有充满电HIGH_TEMP_DETECTEDWarningThe Service Processor detected high temperature；Service Processor发现温度

23、高告警INVALID_ENVIR_BITSWarningA validity problem was detected during an environmental test；环境测试发现问题LATCHED_ALARMSWarningA Symmetrix power subsystem discovered latched alarms；电源子系统有非临时告警，需要进一步检查M1_RESYNC_WITH_M2InformationalAn M1 mirror of a Symmetrix Device is resynchronizing with the M2 mirrorM2_RESY

24、NC_WITH_M1InformationalAn M2 mirror of a Symmetrix Device is resynchronizing with the M1 mirrorMEM_DISABLE_INVOKEDInformationalOne or more memory banks were disabled due to cache errors一个或者多个内存块由于错误而被禁止使用MIRROR_NRWarningA device has a mirror that is Not Ready；设备有一个份Mirror Not ready, 可能是硬盘故障或是热备盘脱离。M

25、IRROR_WDWarningA device has a mirror that is Write Disabled；设备有一个份Mirror write disable.NO_COMM_TO_MII_DIRWarningThe Service Processor could not communicate to a director; Service Processor和director通信有问题。OLD_BOARD_MISMATCHErrorThe Symmetrix communication board old information does not match current i

26、nformation；通信卡信息不一致ONE_RDF_LINK_DOWNWarningA single RDF link in an RDF group is not operational；RDF链路中有一条中断ONE_RDF_LINK_NOW_UPWarningA single RDF link in an RDF group is now operational after a 'Single Link Down' event；RDF链路中有一条中断后恢复PHONEHOME_TRIGGERInformationalA certain event triggered a C

27、all Home for servicePOWER_ON_TIME_FAILED_TESTErrorThe Symmetrix environment tests found inconsistencies in Power-on-Time；开机环境测试发现不一致POWER_ZONE_5_HOURS_BEFORE_SHUDOWNErrorPower zone 5 hours before shutdown, one of the power zones is down. 5 hours before Vault shutdown；输入电源出现故障，单路供电，5小时后机器开始关机POWER_ZO

28、NE_COUNTDOWN_STARTEDErrorPower zone count down started. One of the power zones is down, count down (20 hours) for Vault-Shutdown is started；输入电源出现故障，单路供电，20小时后机器开始关机POWER_ZONE_DOWN_FAILEDErrorPower zone down failed；输入电源故障POWER_ZONE_ILLEGAL_STATUSErrorPower zone down illegal status, found file with o

29、ld information；输入电源故障，发现文件包含旧信息POWER_ZONE_WAIT_TIME_CHANGED_ON_GUIErrorPower zone down - wait time changed on the GUI；输入电源单路故障，关机倒计时被更改RDF_CG_TRIGGERInformationalAn RDF CG trip event was triggeredRDF_ERRORErrorThe RDF subsystem has experienced an error；发生了RDF错误RDF_SIM_MESSAGEInformationalThe RDF sub

30、system initiated a SIM message to a Symmetrix remotely-attached via RDF linksRDF2_DEVICE_NRInformationalOne of the RDF2 devices was found to be Not ReadyRDFA_CONSISTENT_DEACTIVATE_ISSUEDWarningSRDF/A session transition from Asynchronous to Synchronous mode requested；SRDF/A进程被转成同步模式RDFA_CONSISTENT_DE

31、ACTIVATE_ISSUED_FROM_HOSTWarningSRDF/A session transition from Asynchronous to Synchronous mode requested (host software initiated)；主机发起SRDF/A进程转成同步模式RDFA_DEACTIVATE_ISSUEDWarningSRDF/A session transition out of Asynchronous mode requested；SRDF/A进程被停止RDFA_DEACTIVATE_ISSUED_FROM_HOSTWarningSRDF/A ses

32、sion transition out of Asynchronous mode requested (host software initiated) ；主机发起停止SRDF/A进程RDFA_DROP_ISSUEDWarningSRDF/A session drop requested；SRDF/A进程被请求中断RDFA_DROP_ISSUED_FROM_HOSTWarningSRDF/A session drop requested (host software initiated)；主机发起SRDF/A进程中断RDFA_PENDING_DROP_ISSUEDWarningSRDF/A s

33、ession drop at cycle boundary requested；SRDF/A进程被要求在CYCLE边界中断RDFA_PENDING_DROP_ISSUED_FROM_HOSTWarningSRDF/A session drop at cycle boundary requested (host software initiated)主机发起SRDF/A进程被要求在CYCLE边界中断RDFA_SESS_DROP_DEV_NR_CGErrorSRDF/A Session dropped, device not ready through consistency group.；SRD

34、F/A进程中断，Consistency Group中设备Not ReadyRDFA_SESS_DROP_DEV_NR_OFFErrorSRDF/A Session dropped, device not ready. Tolerance mode is off；SRDF/A中断，设备Not Ready，Tolerance模式关闭RDFA_SESS_DROP_NO_RDF_LNKErrorSRDF/A Session dropped, no RDF links operational.；SRDF/A中断，所有链路中断。RDFA_SESS_DROP_TIMEOUT_HAErrorSRDF/A Se

35、ssion dropped, time out on an HA.；SRDF/A进程中断，HA超时RDFA_SESS_DROP_TIMEOUT_MSCErrorSRDF/A Session dropped, time out in MSC mode.；SRDF/A中断，MSC mode超时RDFA_SESS_DROP_TIMEOUT_RAErrorSRDF/A Session dropped, time out on an RA.；SRDF/A中断，RA超时RDFA_SESS_DROP_WPL_DSBLErrorSRDF/A Session dropped, write pending lim

36、it reached. Host throttling disabled.；SRDF/A中断，Write pending达到限制，主机阀值停止。RDFA_SESS_DROP_WPL_ENBLErrorSRDF/A Session dropped, write pending limit reached. Host throttling enabled.；SRDF/A中断，Write pending达到限制，主机阀值打开。SAVEDEVS_FULLErrorThe SAVE device pool for TimeFinder/Snap sessions is full；SAVE设备满。SAVE

37、DEVS_NEAR_FULLWarningThe SAVE device pool for TimeFinder/Snap sessions is 90% fullSENSE_CABLE_MISSINGWarningA Symmetrix power subsystem Environment sense cable is missing；检测不到电源信号线。SMOKE_DETECT_ALERTWarningThe Service Processor detected a smoke detector alert；Service Processor发现烟感告警SMOKE_DETECT_MALF

38、UNCTIONWarningThe Service Processor detected a malfunction in the smoke detector；Service Processor发现烟雾探测器故障SP_COMM_TO_MII_DIRInformationalThe Service Processor is communicating via a local directorSP_CONNECT_TIME_NOT_FOUNDErrorThe Symmetrix has no records of the last Service Processorconnection time

39、；Service Processor通信故障。SP_CONNECT_VIA_SERIAL_LINEInformationalThe Service Processor is currently communicating via a serial lineSP_DISK_FULLWarningThe Service Processor disk is full；Service Processor磁盘满SP_EXCESS_MEMORY_USAGEWarningThe Service Processor software detected excessive memory usage；Servic

40、e Processor内存使用率超高SP_NOT_RESPONDINGWarningThe Symmetrix Service Processor is not communicating with theSymmetrix；Service Processor和Symmetrix通信不正常。SP_PHONEHOME_FAILWarningThe Symmetrix Service Processor could not complete a Call Home forService；Service Processor CALLHOME服务没能完成。SP_PHONEHOME_SUCCESSInformationalThe Symmetrix Service Processor completed a Call Home for serviceSP_REBOOT_SUCCESSInformationalThe Service Processor