HACMP Oracle Rac安装设置办法

1、PAGE HACMP Oracle Rac安装设置手册版本 v1.0二零一二年十月北京华胜天成科技股份有限公司文档操纵更改记录版本创建/修改时刻编制/修改者 文件/修改内容审批人V1.02012.10批阅姓名职位公布姓名职位目录 TOC o h z u HYPERLINK l _Toc225161693 目录 PAGEREF _Toc225161693 h 3 HYPERLINK l _Toc225161694 第1章 HACMP v5.4安装配置 PAGEREF _Toc225161694 h 4 HYPERLINK l _Toc225161695 1.1、 IP规划 PAGEREF _To

2、c225161695 h 4 HYPERLINK l _Toc225161696 1.2、 安装HACMP软件 PAGEREF _Toc225161696 h 4 HYPERLINK l _Toc225161697 1.2.1、 操作系统版本 PAGEREF _Toc225161697 h 4 HYPERLINK l _Toc225161698 1.2.2、 安装依靠的软件包 PAGEREF _Toc225161698 h 5 HYPERLINK l _Toc225161699 1.2.3、 安装HACMP v5.4 PAGEREF _Toc225161699 h 6 HYPERLINK l

3、_Toc225161700 1.3、 HACMP配置规划 PAGEREF _Toc225161700 h 8 HYPERLINK l _Toc225161701 1.3.1、 配置IP和网络 PAGEREF _Toc225161701 h 8 HYPERLINK l _Toc225161702 1.3.2、 编辑/etc/hosts文件 PAGEREF _Toc225161702 h 9 HYPERLINK l _Toc225161703 1.3.3、 创建共享卷组和文件系统 PAGEREF _Toc225161703 h 9 HYPERLINK l _Toc225161704 1.4、 HA

4、CMP Standard配置 PAGEREF _Toc225161704 h 10 HYPERLINK l _Toc225161705 1.4.1、 添加 cluster和节点 PAGEREF _Toc225161705 h 11 HYPERLINK l _Toc225161706 1.4.2、 创建并配置资源组 PAGEREF _Toc225161706 h 11 HYPERLINK l _Toc225161707 1.4.3、 同步HACMP的配置 PAGEREF _Toc225161707 h 13 HYPERLINK l _Toc225161708 1.5、 HACMP启动 PAGER

5、EF _Toc225161708 h 13 HYPERLINK l _Toc225161709 1.5.1、 启动HACMP PAGEREF _Toc225161709 h 13 HYPERLINK l _Toc225161710 第2章 Oracle RAC安装配置 PAGEREF _Toc225161710 h 13 HYPERLINK l _Toc225161711 2.1、 规划 PAGEREF _Toc225161711 h 13 HYPERLINK l _Toc225161712 2.2、 安装检查 PAGEREF _Toc225161712 h 15 HYPERLINK l _T

6、oc225161713 2.3、 安装前配置 PAGEREF _Toc225161713 h 17 HYPERLINK l _Toc225161714 2.4、 安装CRS PAGEREF _Toc225161714 h 20 HYPERLINK l _Toc225161715 2.5、 安装数据库 PAGEREF _Toc225161715 h 41 HYPERLINK l _Toc225161716 2.6、 创建数据库 PAGEREF _Toc225161716 h 47 HYPERLINK l _Toc225161717 2.7、 数据测试 PAGEREF _Toc225161717

7、h 59HACMP v5.4安装配置IP规划NodeInterface NameTypeIP AddressRegistered Inibm03ibm03Public8Hosts fileibm03ibm03-vipVirtual10Hosts fileibm03ibm03-privPrivate8Hosts fileibm04ibm04Public9Hosts fileibm04ibm04-vipVirtual11Hosts fileibm04ibm04-privPrivate9Hosts file安装HACMP软件完成HACMP安装配置规划后，能够开始安装HACMP软件，安装版本为HACM

8、P v5.4。操作系统版本# oslevel -r5300-06安装依靠的软件包检查系统的软件安装包，请参照IBM红皮书中High Availability Cluster Multi-Processing for AIX 5L Installation Guide。那个地点我们建议安装软件束APP-DEV和SERVER，其中包括的红皮书要求的大部分软件包。安装rsct软件包：pat.clientsrsct.opt.storagerm（可选）安装其它软件包：bos.clvm（使用并行资源组时需要，RAC下必须安装此包）bos.data#smitty install_latestInstall

9、SoftwareType or select values in entry fields.Press Enter AFTER making all desired changes.Entry FieldsINPUT device / directory for software /dev/cd0SOFTWARE to install _all_latestPREVIEW only? (install operation will NOT occur) noCOMMIT software updates? yesSAVE replaced files? noAUTOMATICALLY inst

10、all requisite software? yesEXTEND file systems if space needed? yesOVERWRITE same or newer versions? noVERIFY install and check file sizes? noInclude corresponding LANGUAGE filesets? yesDETAILED output? noProcess multiple volumes? yesACCEPT new license agreements? yesPREVIEW new LICENSE agreements?

11、no在“SOFTWARE to install”一栏中选择要安装的软件包，“ACCEPT new license agreements”中同意许可协议，即完成安装软件包。或参考以下脚本命令：#安装软件包/usr/lib/instl/sm_inst installp_cmd -a -Q -d /dev/cd0 -f bos.clvm ALL I:bos.clvm _all_filesets,bos.data ALL I:bos.data _all_filesets,rsct.basic ALL I:rsct.basic _all_filesets,pat.basic ALL I:pat.basi

12、c _all_filesets,pat.clients ALL I:pat.clients _all_filesets,rsct.opt.storagerm ALL I:rsct.opt.storagerm _all_filesets -c -N -g -X -G -Y #安装HACMP v5.4除了haview ，netwiew ( Tivoli),的 包以外，所有的HACMP的软件包都要安装，rsct.opt.storagerm为可选安装包，关于差不多包在安装光盘或Expansion光盘上能够查找到。#smitty install_latestInstall SoftwareType or

13、 select values in entry fields.Press Enter AFTER making all desired changes.Entry FieldsINPUT device / directory for software /dev/cd0SOFTWARE to install _all_latestPREVIEW only? (install operation will NOT occur) noCOMMIT software updates? yesSAVE replaced files? noAUTOMATICALLY install requisite s

14、oftware? yesEXTEND file systems if space needed? yesOVERWRITE same or newer versions? noVERIFY install and check file sizes? noInclude corresponding LANGUAGE filesets? yesDETAILED output? noProcess multiple volumes? yesACCEPT new license agreements? yesPREVIEW new LICENSE agreements? no在“SOFTWARE to

15、 install”一栏中选择要安装的软件包，“ACCEPT new license agreements”中同意许可协议，即完成安装软件包，安装过程大约在1030分钟完成。或参考以下脚本命令：#安装HACMP/usr/lib/instl/sm_inst installp_cmd -a -Q -d /dev/cd0 -f cluster.adt.es ALL I:cluster.adt.es _all_filesets,cluster.doc.en_US.es ALL I:cluster.doc.en_US.es _all_filesets,cluster.es ALL I:cluster.es

16、 _all_filesets,cluster.es.cfs ALL I:cluster.es.cfs _all_filesets,cluster.es.cspoc ALL I:cluster.es.cspoc _all_filesets,cluster.es.nfs ALL I:cluster.es.nfs _all_filesets,cluster.es.plugins ALL I:cluster.es.plugins _all_filesets,cluster.es.worksheets ALL I:cluster.es.worksheets _all_filesets,cluster.l

17、icense ALL I:cluster.license _all_filesets,cluster.man.en_US.es ALL I:cluster.man.en_US.es _all_filesets -c -N -g -X -G -Y系统提示成功后，完成HACMP补丁升级，将HACMP补丁上传至系统目录下，我们假设上传目录为/tmp/hav54，本地目录为d:hav54，上传命令如下：Windows：“开始”-“运行”，输入“cmd”，依照以下提示完成补丁上传：Microsoft Windows XP 版本 5.1.2600(C) 版权所有 1985-2001 Microsoft C

18、orp.C:Documents and SettingsAdministratorftp remote-host-ip-address输入用户名和密码（此处需要给用户设置密码，假如为空，AIX拒绝登录），成功登录后：ftpcd /tmp/hav54ftplcd d:hav54ftppromptftpbinftpmput *.bff上传完成后退出ftp程序ftpquit升级HACMP v5.4补丁#smitty update_all Update Installed Software to Latest Level (Update All)Type or select values in ent

19、ry fields.Press Enter AFTER making all desired changes.Entry FieldsINPUT device / directory for software /tmp/hav54SOFTWARE to update _update_allPREVIEW only? (update operation will NOT occur) noCOMMIT software updates? yesSAVE replaced files? noAUTOMATICALLY install requisite software? yesEXTEND fi

20、le systems if space needed? yesVERIFY install and check file sizes? noDETAILED output? noProcess multiple volumes? yesACCEPT new license agreements? yesPREVIEW new LICENSE agreements? no回车确认以上命令，即可完成升级工作。注意：假如安装程序提示“没有找到相关的fileset”等类似的提示，可在/tmp/hav54目录下执行命令“#inutoc .”，重新生成软件包安装列表。重新启动AIX安装完成HACMP后，为

21、幸免各种问题，要求将AIX系统重新启动，至此我们完成了所有的安装工作，能够开始配置HACMP双机集群了。#shutdown FrHACMP配置规划配置IP和网络AIX网卡IP地址配置：#smiitty mktcpip Available Network InterfacesMove cursor to desired item and press Enter.en0 08-08 Standard Ethernet Network Interfaceen1 08-09 Standard Ethernet Network Interface en2 0A-08 Standard Ethernet

22、Network Interface en3 0J-08 Standard Ethernet Network Interface选择想要配置IP地址的网卡，选择“en0”，回车确认，进入下一步配置：Minimum Configuration & StartupTo Delete existing configuration data, please use Further Configuration menusType or select values in entry fields.Press Enter AFTER making all desired changes.Entry Field

23、sHOSTNAME ora01Internet ADDRESS (dotted decimal) 2Network MASK (dotted decimal) Network INTERFACE en0NAMESERVERInternet ADDRESS (dotted decimal) DOMAIN Name Default GatewayAddress (dotted decimal or symbolic name) Cost 0Do Active Dead Gateway Detection? noYour CABLE Type N/ASTART TCP/IP daemons Now

24、no使用同样的方法配置完成网卡的IP地址。编辑/etc/hosts文件关于两个节点ibm03和ibm04要配置相同的/etc/hosts文件，使用vi配置hosts文件，建议首先对原有的hosts文件进行备份，配置如下：#cp /etc/hosts /etc/hosts.bak#vi /etc/hosts8 ibm039 ibm048 ibm03-priv9 ibm04-priv10 ibm03-vip11 ibm04-vip创建共享卷组和文件系统创建卷组#smitty _mkbvgAdd a Big Volume GroupType or select values in entry fie

25、lds.Press Enter AFTER making all desired changes.Entry FieldsVOLUME GROUP name vgdataPhysical partition SIZE in megabytesPHYSICAL VOLUME names hdisk2FORCE the creation of volume group? yesActivate volume group AUTOMATICALLY noat system restart?Volume group MAJOR NUMBER 100Create VG Concurrent Capabl

26、e? no我们在建立HACMP共享卷组时，要注意的参数 “Activate volume group AUTOMATICALLY at system restart? ”该项，设置为“no”，是使用HACMP的方式将卷组varyon，而不是系统启动时由操作系统varyon。Major NUMBER是指该设备（orvg）的主设备号，此处设置为系统唯一即可，可通过命令查询可用的major number：# lvlstmajor43.59,62.或可通过ls l /dev/命令查看当前所有设备的主设备号，第五列即是。创建逻辑卷和文件系统在ibm03节点上按照以下操作：#varyoffvg vgdat

27、a需将vgdata varyoff下来，在ora02节点上才能正确的导入。在ibm04节点上按照一下操作：#lspv hdisk0 00c2ef1fa38df28a rootvg activehdisk1 00c2ef1f38b81fac rootvg activehdisk2 00c2ecef3846c20e None 查看hdisk2是否为物理卷，否则需要使用命令将其属性更改：#chdev l hdisk2 a pv=yes导入卷组和文件系统的定义#smitty importvgImport a Volume GroupType or select values in entry fiel

28、ds.Press Enter AFTER making all desired changes.Entry FieldsVOLUME GROUP name vgdata* PHYSICAL VOLUME name hdisk2Volume group MAJOR NUMBER 100#varyonvg vgdata#varyoffvg vgdataHACMP Standard配置HACMP配置可采纳Standard和Extended两种方式，其中Standard配置能够配置Cluter名称，节点，差不多网络，应用服务器，各种资源和资源组等差不多的内容，假如要配置永久IP，串口心跳，对资源组进行

29、详细的配置等工作则需要在Extended中完成，能够讲Standard菜单是Extended菜单的一个子集，能够在Extended菜单中完成HACMP的所有配置工作。下面我们以Standard菜单完成HACMP的差不多配置工作：添加cluster和节点配置cluster资源创建并配置资源组同步HACMP的配置HACMP配置菜单如下：#smitty hacmp HACMP for AIXMove cursor to desired item and press Enter. Initialization and Standard Configuration Extended Configurat

30、ion System Management (C-SPOC) Problem Determination Tools选择“Initialization and Standard Configuration”，如下菜单：Initialization and Standard ConfigurationMove cursor to desired item and press Enter.Configuration AssistantsConfigure an HACMP Cluster and NodesConfigure Resources to Make Highly AvailableCo

31、nfigure HACMP Resource GroupsVerify and Synchronize HACMP ConfigurationDisplay HACMP ConfigurationHACMP Cluster Test Tool添加 cluster和节点#smitty hacmp-Initialization and Standard ConfigurationConfigure an HACMP Cluster and NodesConfigure Nodes to an HACMP Cluster (standard)Type or select values in entr

32、y fields.Press Enter AFTER making all desired changes.Entry Fields* Cluster Name ora_clusterNew Nodes (via selected communication paths) ibm03 ibm04Currently Configured Node(s)创建并配置资源组#smitty hacmp- Initialization and Standard ConfigurationConfigure HACMP Resource Groups Configure HACMP Resource Gro

33、upsMove cursor to desired item and press Enter. Add a Resource Group Change/Show a Resource Group Remove a Resource Group Change/Show Resources for a Resource Group (standard)Add a Resource GroupType or select values in entry fields.Press Enter AFTER making all desired changes.Entry Fields* Resource

34、 Group Name ora_rg* Participating Nodes (Default Node Priority) ibm03 ibm04Startup Policy Online On All AvailabFallover Policy Bring Offline (On ErrFallback Policy Never Fallback添加资源至资源组中#smitty hacmp- Initialization and Standard ConfigurationConfigure HACMP Resource Groups Configure HACMP Resource

35、GroupsMove cursor to desired item and press Enter. Add a Resource Group Change/Show a Resource Group Remove a Resource Group Change/Show Resources for a Resource Group (standard)Change/Show All Resources and Attributes for a Resource GroupType or select values in entry fields.Press Enter AFTER makin

36、g all desired changes.Entry FieldsResource Group Name ora_rgParticipating Nodes (Default Node Priority) ibm03 ibm04Startup Policy Online On Home Node OnlyFallover Policy Fallover To Next Priority Node In The ListFallback Policy Fallback To Higher Priority Node In The ListConcurrent Volume Groups vgd

37、ata +Use forced varyon of volume groups, if necessary false +Automatically Import Volume Groups false +Application Servers +Tape Resources + 至此，我们完成了资源和资源组的关联。同步HACMP的配置#smitty hacmp- Initialization and Standard ConfigurationVerify and Synchronize HACMP Configuration回车确认该操作，如此HACMP的配置差不多同步到两个节点中了，也确

38、实是ibm03和ibm04拥有同样的配置了。HACMP启动启动HACMP在节点ibm03上执行smitty clstart，回车确认即可启动HACMP* Start now, on system restart or both now + Start Cluster Services on these nodes ibm03 + BROADCAST message at startup? true + Startup Cluster Information Daemon? false + Reacquire resources after forced down ? false +Oracle

39、 RAC安装配置规划数据库配置规划ORACLE_BASE/dataCRS_HOME/data/product /10.2.0/crsORACLE_HOME /data/product /10.2.0/dbDB_NAMEorclORACLE_SIDorcl1，orcl2TNS_ADMIN$ORACLE_HOME/network/adminOracle治理帐户的口令 sys Systemoracleoracle数据库存放位置裸设备是否使用归档方式运行数据库Yes备份方式讲明RMAN/TSM裸设备规划设备名称数据库设施存储空间/dev/rorcl_ocrOcr信息存放500Mb/dev/rorcl_

40、vdiskvdisk信息存放500Mb/dev/rorcl_spfilespfile信息存放100Mb/dev/rorcl_srvconfigsrvconfig信息存放100Mb/dev/rorcl_systemSystem表空间10G/dev/rorcl_sysauxSysaux表空间10G/dev/rorcl_temp1系统临时表空间20G/dev/rorcl_temp2系统临时表空间20G/dev/rorcl_undo1撤销表空间20G/dev/rorcl_undo2撤销表空间20G/dev/rorcl_undo3撤销表空间20G/dev/rorcl_undo4撤销表空间20G/dev/

41、rorcl_usersUser表空间10G/dev/rorcl_toolsTools表空间10G/dev/rorcl_ctl1操纵文件1G/dev/rorcl_ctl2操纵文件1G/dev/rorcl_ctl3操纵文件1G/dev/rorcl_log1a日志文件500Mb/dev/rorcl_log1b日志文件500Mb/dev/rorcl_log2a日志文件500Mb/dev/rorcl_log2b日志文件500Mb/dev/rorcl_log3a日志文件500Mb/dev/rorcl_log3b日志文件500Mb/dev/rorcl_log4a日志文件500Mb/dev/rorcl_log

42、4b日志文件500Mb/dev/rorcl_data1用户数据表空间20G/dev/rorcl_data2用户数据表空间20G/dev/rorcl_data3用户数据表空间20G/dev/rorcl_data4用户数据表空间20G安装检查操作系统版本：# oslevel -r5300-06物理内存：# lsattr -El mem0goodsize 7744 Amount of usable physical memory in Mbytes Falsesize 7744 Total amount of physical memory in Mbytes False交换区：# lsps -aP

43、age Space Physical Volume Volume Group Size %Used Active Auto Typehd6 hdisk0 rootvg 512MB 3 yes yes lv系统空间：# df -gFilesystem GB blocks Free %Used Iused %Iused Mounted on/dev/hd4 0.50 0.44 13% 4788 5% /dev/hd2 5.00 3.10 38% 41240 6% /usr/dev/hd9var 1.00 0.95 6% 747 1% /var/dev/hd3 1.00 0.65 36% 1082

44、1% /tmp/dev/fwdump 0.38 0.37 1% 4 1% /var/adm/ras/platform/dev/hd1 0.50 0.50 1% 71 1% /home/proc - - - - - /proc/dev/hd10opt 2.00 1.86 7% 3461 1% /opt/dev/fslv00 15.00 4.42 71% 29243 3% /data/dev/lv00 0.12 0.12 4% 18 1% /var/adm/csd文件集：# lslpp -l bos.adt.base bos.adt.lib bos.adt.libm bos.perf.libper

45、fstat bos.perf.perfstat ctools rsct.basic.rte pat.clients.rte xlC.aix50.rte xlC.rte Fileset Level Path: /usr/lib/objrepos bos.adt.base COMMITTED Base Application Development Toolkit bos.adt.lib COMMITTED Base Application Development Libraries bos.adt.libm COMMITTED Base Application Development Math

46、Library bos.perf.libperfstat COMMITTED Performance Statistics Library Interface bos.perf.perfstat COMMITTED Performance Statistics Interface ctools COMMITTED Proc Filesystem Tools rsct.basic.rte COMMITTED RSCT Basic Function pat.clients.rte COMMITTED RSCT Event Management Client Function xlC.aix50.r

47、te COMMITTED XL C/C+ Runtime for AIX 5.2 xlC.rte COMMITTED XL C/C+ Runtime Path: /etc/objrepos bos.perf.libperfstat COMMITTED Performance Statistics Library Interface bos.perf.perfstat COMMITTED Performance Statistics Interface rsct.basic.rte COMMITTED RSCT Basic Function安装前配置为确保oracle rac安装成功，系统治理员

48、root用户需创建相应的系统用户、系统组，以便安装治理oracle软件。Root用户还要修改相关系统参数，打破默认值对实际应用的限制。调试rac节点之间的高级功能：rcp/rsh/rlogin，以确定两节点之间能够通信正常，能够在rac环境下进行相关同步操作和心跳监控。创建dba组，id为500# smitty mkgroup创建oracle用户，id为601# smitty mkuser修改oracle用户密码# passwd oracle在oracle用户下创建.rhosts文件，内容为“+”。# su oracle$ vi .rhosts+修改oracle用户限制# smitty chu

49、serSoft FILE size -1Soft CPU time -1Soft DATA segment -1Soft STACK size -1Soft CORE file size -1修改系统配置参数，增加系统进程数上限# smitty chgsysMaximum number of PROCESSES allowed per user 2048修改网络参数no -p -o udp_sendspace=65536no -p -o udp_recvspace=65536no -p -o tcp_sendspace=65536no -p -o tcp_recvspace=65536no -

50、p -o rfc1323=1no -p -o sb_max=1310720no -r -o ipqmaxlen=512创建裸设备mklv -y rac_system_raw_500m -T O -w n -s n -r n vgdata 8依次创建以下裸设备：/dev/rorcl_ocr/dev/rorcl_vdisk/dev/rorcl_spfile/dev/rorcl_srvconfig/dev/rorcl_system/dev/rorcl_sysaux/dev/rorcl_temp1/dev/rorcl_temp2/dev/rorcl_undo1/dev/rorcl_undo2/dev/

51、rorcl_undo3/dev/rorcl_undo4/dev/rorcl_users/dev/rorcl_tools/dev/rorcl_ctl1/dev/rorcl_ctl2/dev/rorcl_ctl3/dev/rorcl_log1a/dev/rorcl_log1b/dev/rorcl_log2a/dev/rorcl_log2b/dev/rorcl_log3a/dev/rorcl_log3b/dev/rorcl_log4a/dev/rorcl_log4b/dev/rorcl_data1/dev/rorcl_data2/dev/rorcl_data3/dev/rorcl_data4更改裸设

52、备权限# cd /dev# chown oracle:dba *orcl*修改oracle用户环境变量# vi /home/oracle/.profileumask 022export ORACLE_BASE=/dataexport ORACLE_HOME=$ORACLE_BASE/product/10.2.0/dbexport ORA_CRS_HOME=$ORACLE_BASE/product/10.2.0/crsexport LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/lib32export ORACLE_SID=orclexport PAT

53、H=$ORACLE_HOME/bin:$ORA_CRS_HOME/bin:$PATH安装CRS在ibm03节点上进入CRS安装盘，执行runInstaller启动CRS的安装；按提示在每个安装节点上以root用户执行rootpre.sh，该脚本在CRS安装盘上的rootpre目录下；# ./rootpre.sh./rootpre.sh output will be logged in /tmp/rootpre.out_06-04-13.12:33:04Saving the original files in /etc/ora_save_06-04-13.12:33:04Copying new

54、kernel extension to /etcLoading the kernel extension from /etc Oracle Kernel Extension Loader for AIX Copyright (c) 1998,1999 Oracle Corporation Successfully loaded /etc/pw-syscall.64bit_kernel with kmid: 0 x441e300 Successfully configured /etc/pw-syscall.64bit_kernel with kmid: 0 x441e300The kernel

55、 extension was successfuly loaded.Configuring Asynchronous I/OAsynchronous I/O is already definedConfiguring POSIX Asynchronous I/OChecking if group services should be configuredGroup hagsuser does not exist.Creating required group for group services: hagsuserPlease add your Oracle userid to the gro

56、up: hagsuserConfiguring HACMP group services socket for possible use by Oracle.The group or permissions of the group services socket have changed.Please stop and restart HACMP before trying to use Oracle.在两个节点都执行root.sh脚本：ibm03# ./root.shWARNING: directory /oracle/product/10.2.0 is not owned by root

57、WARNING: directory /oracle/product is not owned by rootChecking to see if Oracle CRS stack is already configuredChecking to see if any 9i GSD is upSetting the permissions on OCR backup directorySetting up NS directoriesOracle Cluster Registry configuration upgraded successfullyWARNING: directory /or

58、acle/product/10.2.0 is not owned by rootWARNING: directory /oracle/product is not owned by rootSuccessfully accumulated necessary OCR keys.Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.node : node 1: ibm03 ibm03-priv ibm03node 2: ibm04 ibm04-priv ibm04Creating OCR keys for user root, pr

59、ivgrp system.Operation successful.Now formatting voting device: /dev/rora_voteFormat of 1 voting devices complete.Startup will be queued to init within 30 seconds.Adding daemons to inittabAdding daemons to inittabExpecting the CRS daemons to be up within 600 seconds.CSS is active on these nodes. ibm03CSS is inactive on these nodes. ibm04Local node checking complete.Run root.sh on remaining nodes to start CRS daemons.ibm04 # ./root.shWARNING: directory /oracle/product/10.2.0 is not owned by roo