VMware Server RedHedServerbitidvdOracle g RAC

1、VMware Server 2.0.2+RedHed-Server-5.4-i386-dvd+Oracle 11g RAC作者：晴天雨2012-05-051、系统配置主机操作环境主机名称操作系统处理器内存(GB)磁盘(GB)网卡qty-pcWindows 7 旗舰版 64位 SP1 ( DirectX 11 )核心数: 2 / 线程数: 46500英特尔 82577LM Gigabit Network Connection客户端操作环境主机名称操作系统处理器内存(GB)本地磁盘(GB)racdb1rhel-server-5.4-i386-dvd1120racdb2rhel-server-5.4

2、-i386-dvd1120虚拟磁盘布局主机操作系统上的虚拟磁盘客户操作系统上的虚拟磁盘虚拟设备节点大小(GB)描述F:systemASM_DISKOCR_VOTE.vmdk/dev/sdbSCSI 1:02GBF:systemASM_DISKASM_DATA.vmdk/dev/sdcSCSI 1:24GBF:systemASM_DISKASM_FRA.vmdk/dev/sddSCSI 1:34GBRAC 数据库环境主机名称ASM 实例名RAC 实例名数据库名称数据库文件存储OCR与表和磁盘racdb1+ASM1Ora11g1Ora11gASMASMracdb2+ASM2Ora11g2Ora11

3、gASMASMRAC网络设置racdb1网络配置eth0将racdb1连接到公共网络eth1将racdb1连接到racdb2eth0:1Public Virtual IP (VIP)racdb2网络配置eth0将racdb1连接到公共网络eth1将racdb2连接到racdb1eth0:1Public Virtual IP (VIP)2、系统安装省略过.3、 补丁安装rpm -Uvh binutils-2.*rpm -Uvh compat-libstdc+-33*rpm -Uvh elfutils-libelf-0.*rpm -Uvh elfutils-libelf-devel-*rpm -U

4、vh gcc-4.*rpm -Uvh gcc-c+-4.*rpm -Uvh glibc-2.*rpm -Uvh glibc-common-2.*rpm -Uvh glibc-devel-2.*rpm -Uvh glibc-headers-2.*rpm -Uvh ksh-2*rpm -Uvh libaio-0.*rpm -Uvh libaio-devel-0.*rpm -Uvh libgcc-4.*rpm -Uvh libstdc+-4.*rpm -Uvh libstdc+-devel-4.*rpm -Uvh make-3.*rpm -Uvh sysstat-7.*rpm -Uvh unixOD

5、BC-2.*rpm -Uvh unixODBC-devel-2.*驱动程序支持文件用于默认内核用户空间库4、 系统参数配置在"/etc/sysctl.conf"文件中增加或修改如下参数： kernel.shmmni = 4096kernel.sem = 250 32000 100 128fs.file-max = 6815744net.ipv4.ip_local_port_range = 9000 65500net.core.rmem_default = 262144net.core.rmem_max = 4194304net.core.wmem_default = 262

6、144net.core.wmem_max = 1048576fs.aio-max-nr = 1048576在" /etc/security/limits.conf"文件中增加或修改如下参数： grid soft nproc 2047 grid hard nproc 16384 grid soft nofile 1024 grid hard nofile 65536 oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 1024 oracle hard nofile 65536 在" /e

7、tc/pam.d/login"文件中增加或修改如下参数：session required pam_limits.so注意：如果SELINUX在安装过程中打开，将会出现很多问题，解决方法请修改config文件；修改/etc/selinux/config文件修改参数为：SELINUX="" 为 disabled如果不想重启系统，使用命令setenforce 05、 创建RAC用户组和相关用户groupadd -g 1000 oinstallgroupadd -g 1100 asmadmingroupadd -g 1200 dbagroupadd -g 1300 asm

8、dbagroupadd -g 1400 asmoperuseradd -u 1100 -g oinstall -G asmadmin,asmdba,asmoper griduseradd -u 1101 -g oinstall -G dba,asmdba oracle设置oracle和grid用户的密码6、 创建相关目录mkdir -p /u01/app/gridchown -R grid:oinstall /u01mkdir -p /u01/app/oraclechown -R oracle:oinstall /u01/app/oraclechmod -R 775 /u017、 设置用户or

9、acle和grid的环境变量# Oracle Settingsexport TMP=/tmpexport TMPDIR=$TMPexport ORACLE_HOSTNAME=racdb1export ORACLE_UNQNAME=ora11gexport ORACLE_BASE=/u01/app/oracleexport ORACLE_SID=ora11g1export ORACLE_TERM=xtermexport PATH=$ORACLE_HOME/bin:$PATHexport LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/libexport CL

10、ASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlibexport DISPLAY=192.168.10.15:0.0if $USER = "oracle" ; then if $SHELL = "/bin/ksh" ; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi umask 022fi# grid Settingsexport TMPTMP=/tmpexport TMPDIRTMP

11、DIR=$TMPexport ORACLE_HOSTNAME=racdb1export ORACLE_BASE=/u01/app/gridexport ORACLE_SID=+ASM1export ORACLE_TERM=xtermexport PATH=$ORACLE_HOME/bin:$PATHexport DISPLAY=192.168.10.11:0.0 export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/libexport CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME

12、/rdbms/jlibif $USER = "grid" ; then if $SHELL = "/bin/ksh" ; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fiumask 022fi8、 创建共享存储关闭虚拟机，为虚拟机增加新硬盘，作为共享磁盘使用；在CDM命令行中进入安装vmware的目录，运行以下命令：vmware-vdiskmanager.exe -c -s 2048Mb -a ide -t 0 F:systemASM_DISKOCR_VOTE.vm

13、dkvmware-vdiskmanager.exe -c -s 4096Mb -a ide -t 0 F:systemASM_DISKASM_DATA.vmdkvmware-vdiskmanager.exe -c -s 4096Mb -a ide -t 0 F:systemASM_DISKASM_FRA.vmdk修改主机上的"racdb1.vmx"文件，添加以下内容disk.locking = "FALSE" diskLib.dataCacheMaxSize = "0" diskLib.dataCacheMaxReadAheadSiz

14、e = "0" diskLib.dataCacheMinReadAheadSize = "0" diskLib.dataCachePageSize = "4096" diskLib.maxUnsyncedWrites = "0" scsi1.present = "TRUE" scsi1.sharedBus = "VIRTUAL" scsi1.virtualDev = "lsilogic" scsi1:0.present = "TRUE"

15、 scsi1:0.fileName = "F:systemASM_DISKOCR_VOTE.vmdk" scsi1:0.writeThrough = "TRUE" scsi1:0.mode = "independent-persistent" scsi1:0.deviceType = "disk" scsi1:0.redo = "" scsi1:1.present = "TRUE" scsi1:1.fileName = "F:systemASM_DISKASM_DA

16、TA.vmdk" scsi1:1.writeThrough = "TRUE" scsi1:1.mode = "independent-persistent" scsi1:1.deviceType = "disk" scsi1:1.redo = "" scsi1:2.present = "TRUE" scsi1:2.fileName = "F:systemASM_DISKASM_FRA.vmdk" scsi1:2.writeThrough = "TRUE&q

17、uot; scsi1:2.mode = "independent-persistent" scsi1:2.deviceType = "disk" scsi1:2.redo = ""对新加入的磁盘进行分区fdisk /dev/sdb fdisk /dev/sdc fdisk /dev/sdd 输入的命令顺序是："n"，"p"，"1"，"回车"，"回车"，"p"和"w"。 9、 执行oracle AS

18、M配置rootracdb1 # oracleasm configureORACLEASM_ENABLED=falseORACLEASM_UID=ORACLEASM_GID=ORACLEASM_SCANBOOT=trueORACLEASM_SCANORDER=""ORACLEASM_SCANEXCLUDE=""rootracdb1 # oracleasm configure -iConfiguring the Oracle ASM library driver.This will configure the on-boot properties of th

19、e Oracle ASM librarydriver. The following questions will determine whether the driver isloaded on boot and what permissions it will have. The current valueswill be shown in brackets (''). Hitting <ENTER> without typing ananswer will keep that current value. Ctrl-C will abort.Default us

20、er to own the driver interface : gridDefault group to own the driver interface : asmadminStart Oracle ASM library driver on boot (y/n) n: yScan for Oracle ASM disks on boot (y/n) y: Writing Oracle ASM library driver configuration: donerootracdb1 # oracleasm initCreating /dev/oracleasm mount point: /

21、dev/oracleasmLoading module "oracleasm": oracleasmMounting ASMlib driver filesystem: /dev/oracleasmrootracdb1 # oracleasm createdisk OCR_VOTE /dev/sdb1Writing disk header: doneInstantiating disk: donerootracdb1 # oracleasm createdisk ASM_DATA /dev/sdc1Writing disk header: doneInstantiating

22、 disk: donerootracdb1 # oracleasm createdisk ASM_FRA /dev/sdd1Writing disk header: doneInstantiating disk: donerootracdb1 # oracleasm scandisksReloading disk partitions: doneCleaning any stale ASM disks.Scanning system for ASM disks.rootracdb1 # oracleasm listdisksASM_DATAASM_FRAOCR_VOTE10、 复制虚拟机rac

23、db2在VMware控制台中将racdb2复制过去修改racnode2虚拟机的网卡、主机名、oracle和grid用户.bash_profile文件中的ORACLE_SID以及ORACLE_HOSTNAME参数。修改linux主机的配置文件/etc/sysconfig/network 和 /etc/hosts 不重启机器下生效执行hostname yourname11、 修改racdb1和racdb2的hosts文件RACDB1# Public Network - (eth0) racdb1 racdb2# Private Interconnect - (eth1) 10.5.10.211 r

24、acdb1-priv 10.5.10.212 racdb2-priv # Public Virtual IP (VIP) addresses - (eth0:1) 192.168.10.213 racdb1-vip 192.168.10.214 racdb2-vip # Single Client Access Name (SCAN) 192.168.10.215 racdb-cluster-scan RACDB2# Public Network - (eth0) racdb1 racdb2# Private Interconnect - (eth1) 10.5.10.211 racdb1-p

25、riv 10.5.10.212 racdb2-priv # Public Virtual IP (VIP) addresses - (eth0:1) 192.168.10.213 racdb1-vip 192.168.10.214 racdb2-vip # Single Client Access Name (SCAN) 192.168.10.215 racdb-cluster-scan 12、 创建racdb1与racdb2直接的信任关系-bash-3.2$ mkdir .ssh-bash-3.2$ cd .ssh-bash-3.2$ ssh-keygen -t rsaGenerating

26、public/private rsa key pair.Enter file in which to save the key (/home/grid/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/grid/.ssh/id_rsa.Your public key has been saved in /home/grid/.ssh/id_rsa.pub.The key fingerp

27、rint is:b2:99:ae:7c:ee:75:f4:23:df:d3:97:fb:d6:52:14:c7 gridracdb1-bash-3.2$ touch authorized_keys13、 使用grid用户登录使用root用户登录执行xhost + （xhost + 是使所有用户都能访问Xserver. ）-bash-3.2$ su - oracle-bash-3.2$ ./runInstaller 正在启动 Oracle Universal Installer.检查临时空间: 必须大于 80 MB。 实际为 10671 MB 通过检查交换空间: 必须大于 150 MB。 实际为

28、 2047 MB 通过检查监视器: 监视器配置至少必须显示 256 种颜色。 实际为 16777216 通过准备从以下地址启动 Oracle Universal Installer /tmp/OraInstall2012-05-03_10-27-17AM. 请稍候.-bash-3.2$ Logfile Location : /tmp/OraInstall2012-05-03_10-27-17AM/sshsetup1_2012-05-03_10-27-17AM.logLogfile Location : /tmp/OraInstall2012-05-03_10-27-17AM/sshsetup1

29、_2012-05-03_10-27-17AM.log可以在以下位置找到本次安装会话的日志: /u01/app/oraInventory/logs/installActions2012-05-03_10-27-17AM.log由于我同时在racdb1和racdb2上运行了root.sh脚本，结果racdb2上报错如下：rootracdb2 Server#Running Oracle 11g root.sh script.The following environment variables are set as: ORACLE_OWNER= gridEnter the full pathname

30、 of the local bin directory: /usr/local/bin: Copying dbhome to /usr/local/bin . Copying oraenv to /usr/local/bin . Copying coraenv to /usr/local/bin .Creating /etc/oratab file.Entries will be added to the /etc/oratab file as needed byDatabase Configuration Assistant when a database is createdFinishe

31、d running generic part of root.sh script.Now product-specific root actions will be performed.2012-05-03 11:07:24: Parsing the host name2012-05-03 11:07:24: Checking for super user privileges2012-05-03 11:07:24: User has super user privilegesCreating trace directoryLOCAL ADD MODE Creating OCR keys fo

32、r user 'root', privgrp 'root'.Operation successful. root wallet root wallet cert root cert export peer wallet profile reader wallet pa wallet peer wallet keys pa wallet keys peer cert request pa cert request peer cert pa cert peer root cert TP profile reader root cert TP pa root cert

33、 TP peer pa cert TP pa peer cert TP profile reader pa cert TP profile reader peer cert TP peer user cert pa user certAdding daemon to inittabCRS-4123: Oracle High Availability Services has been started.ohasd is startingCRS-2672: 尝试启动 'ora.gipcd' (在 'racdb2' 上)CRS-2672: 尝试启动 'ora.

34、mdnsd' (在 'racdb2' 上)CRS-2676: 成功启动 'ora.mdnsd' (在 'racdb2' 上)CRS-2676: 成功启动 'ora.gipcd' (在 'racdb2' 上)CRS-2672: 尝试启动 'ora.gpnpd' (在 'racdb2' 上)CRS-2676: 成功启动 'ora.gpnpd' (在 'racdb2' 上)CRS-2672: 尝试启动 'ora.cssdmonitor'

35、; (在 'racdb2' 上)CRS-2676: 成功启动 'ora.cssdmonitor' (在 'racdb2' 上)CRS-2672: 尝试启动 'ora.cssd' (在 'racdb2' 上)CRS-2672: 尝试启动 'ora.diskmon' (在 'racdb2' 上)CRS-2676: 成功启动 'ora.diskmon' (在 'racdb2' 上)CRS-2676: 成功启动 'ora.cssd' (在 &#

36、39;racdb2' 上)CRS-2672: 尝试启动 'ora.ctssd' (在 'racdb2' 上)CRS-2676: 成功启动 'ora.ctssd' (在 'racdb2' 上)DiskGroup ORC_VOTE creation failed with the following message:Disk Group ORC_VOTE already exists. Cannot be created againConfiguration of ASM failed, see logs for detail

37、sDid not succssfully configure and start ASMCRS-2500: 由于资源 'ora.crsd' 未运行, 无法停止该资源CRS-4000: Command Stop failed, or completed with errors.Stop of resource "ora.crsd -init" failedFailed to stop CRSDCRS-2673: 尝试停止 'ora.asm' (在 'racdb2' 上)CRS-2677: 成功停止 'ora.asm

38、9; (在 'racdb2' 上)CRS-2673: 尝试停止 'ora.ctssd' (在 'racdb2' 上)CRS-2677: 成功停止 'ora.ctssd' (在 'racdb2' 上)CRS-2673: 尝试停止 'ora.cssdmonitor' (在 'racdb2' 上)CRS-2677: 成功停止 'ora.cssdmonitor' (在 'racdb2' 上)CRS-2673: 尝试停止 'ora.cssd' (

39、在 'racdb2' 上)CRS-2677: 成功停止 'ora.cssd' (在 'racdb2' 上)CRS-2673: 尝试停止 'ora.gpnpd' (在 'racdb2' 上)CRS-2677: 成功停止 'ora.gpnpd' (在 'racdb2' 上)CRS-2673: 尝试停止 'ora.gipcd' (在 'racdb2' 上)CRS-2677: 成功停止 'ora.gipcd' (在 'racdb2

40、9; 上)CRS-2673: 尝试停止 'ora.mdnsd' (在 'racdb2' 上)CRS-2677: 成功停止 'ora.mdnsd' (在 'racdb2' 上)解决方法如下：在db02上删除之前的crs配置信息，然后重新运行root.sh脚本即可rootracdb2 install# ./rootcrs.pl -verbose -deconfig -force 2012-05-03 11:49:53: Parsing the host name2012-05-03 11:49:53: Checking for sup

41、er user privileges2012-05-03 11:49:53: User has super user privilegesUsing configuration parameter file: ./crsconfig_paramsPRCR-1035 : Failed to look up CRS resource ora.cluster_vip.type for 1PRCR-1068 : Failed to query resourcesCannot communicate with crsdPRCR-1070 : Failed to check if resource ora

42、.gsd is registeredCannot communicate with crsdPRCR-1070 : Failed to check if resource ora.ons is registeredCannot communicate with crsdPRCR-1070 : Failed to check if resource ora.eons is registeredCannot communicate with crsdACFS-9200: SupportedCRS-4535: Cannot communicate with Cluster Ready Service

43、sCRS-4000: Command Stop failed, or completed with errors.CRS-2791: 正在启动用于关闭 'racdb2' 上 Oracle High Availability Services 管理的资源的操作CRS-2673: 尝试停止 'ora.drivers.acfs' (在 'racdb2' 上)CRS-2677: 成功停止 'ora.drivers.acfs' (在 'racdb2' 上)CRS-2793: 关闭 'racdb2' 上 Ora

44、cle High Availability Services 管理的资源的操作已完成CRS-4133: Oracle High Availability Services has been stopped.error: package cvuqdisk is not installedSuccessfully deconfigured Oracle clusterware stack on this noderootracdb2 install#Running Oracle 11g root.sh script.The following environment variables are s

45、et as: ORACLE_OWNER= gridEnter the full pathname of the local bin directory: /usr/local/bin: yCreating y directory. Copying dbhome to y . Copying oraenv to y . Copying coraenv to y .Entries will be added to the /etc/oratab file as needed byDatabase Configuration Assistant when a database is createdF

46、inished running generic part of root.sh script.Now product-specific root actions will be performed.2012-05-03 11:50:50: Parsing the host name2012-05-03 11:50:50: Checking for super user privileges2012-05-03 11:50:50: User has super user privilegesLOCAL ADD MODE Creating OCR keys for user 'root&#

47、39;, privgrp 'root'.Operation successful.Adding daemon to inittabCRS-4123: Oracle High Availability Services has been started.ohasd is startingCRS-4402: CSS 守护程序已在独占模式下启动, 但在节点 racdb1 (编号为 1) 上发现活动 CSS 守护程序, 因此正在终止An active cluster was found during exclusive startup, restarting to join the c

48、lusterCRS-2672: 尝试启动 'ora.mdnsd' (在 'racdb2' 上)CRS-2676: 成功启动 'ora.mdnsd' (在 'racdb2' 上)CRS-2672: 尝试启动 'ora.gipcd' (在 'racdb2' 上)CRS-2676: 成功启动 'ora.gipcd' (在 'racdb2' 上)CRS-2672: 尝试启动 'ora.gpnpd' (在 'racdb2' 上)CRS-2676:

49、 成功启动 'ora.gpnpd' (在 'racdb2' 上)CRS-2672: 尝试启动 'ora.cssdmonitor' (在 'racdb2' 上)CRS-2676: 成功启动 'ora.cssdmonitor' (在 'racdb2' 上)CRS-2672: 尝试启动 'ora.cssd' (在 'racdb2' 上)CRS-2672: 尝试启动 'ora.diskmon' (在 'racdb2' 上)CRS-2676: 成

50、功启动 'ora.diskmon' (在 'racdb2' 上)CRS-2676: 成功启动 'ora.cssd' (在 'racdb2' 上)CRS-2672: 尝试启动 'ora.ctssd' (在 'racdb2' 上)CRS-2676: 成功启动 'ora.ctssd' (在 'racdb2' 上)CRS-2672: 尝试启动 'ora.drivers.acfs' (在 'racdb2' 上)CRS-2676: 成功启动 

51、9;ora.drivers.acfs' (在 'racdb2' 上)CRS-2672: 尝试启动 'ora.asm' (在 'racdb2' 上)CRS-2676: 成功启动 'ora.asm' (在 'racdb2' 上)CRS-2672: 尝试启动 'ora.crsd' (在 'racdb2' 上)CRS-2676: 成功启动 'ora.crsd' (在 'racdb2' 上)CRS-2672: 尝试启动 'ora.evmd'

52、; (在 'racdb2' 上)CRS-2676: 成功启动 'ora.evmd' (在 'racdb2' 上)Preparing packages for installation.Configure Oracle Grid Infrastructure for a Cluster . succeededUpdating inventory properties for clusterwareStarting Oracle Universal Installer.Checking swap space: must be greater than

53、 500 MB. Actual 1178 MB PassedThe inventory pointer is located at /etc/oraInst.locThe inventory is located at /u01/app/oraInventory'UpdateNodeList' was successful.14、 创建ASM磁盘数据快速恢复-bash-3.2$ su - grid-bash-3.2$asmca15、 安装oracle数据库软件-bash-3.2$ su - oracle-bash-3.2$ ./runInstaller 正在启动 Oracle

54、Universal Installer.检查临时空间: 必须大于 80 MB。 实际为 4616 MB 通过检查交换空间: 必须大于 150 MB。 实际为 1089 MB 通过检查监视器: 监视器配置至少必须显示 256 种颜色。 实际为 16777216 通过准备从以下地址启动 Oracle Universal Installer /tmp/OraInstall2012-05-03_02-28-16PM. 请稍候.-bash-3.2$ Registering console handler with the logger.Logfile Location : /u01/app/oraInv

55、entory/logs/sshsetup1_2012-05-03_02-28-16PM.logRegistering console handler with the logger.Logfile Location : /u01/app/oraInventory/logs/sshsetup1_2012-05-03_02-28-16PM.log可以在以下位置找到本次安装会话的日志: /u01/app/oraInventory/logs/installActions2012-05-03_02-28-16PM.logRegistering console handler with the logge

56、r.Logfile Location : /u01/app/oraInventory/logs/sshsetup1_2012-05-03_02-28-16PM.log到这一步发现集群节点没有出来报INS-35354错误：INS-35354: The system on which you are attempting to install Oracle RAC is not part of a valid cluster.Cause: Before you can install Oracle RAC, you must install Oracle Grid Infrastructure on all servers (Oracle Clusterware and Oracle ASM) to create a cluster.Action: Oracle Grid Infrastructure is not installed. Install it either from the separate installation media included in your media pack, or install it by