We recently done with a RAC 2 node DB installation
Softwares used
1. ORACLE GRID software ( 11.2.0.3)
2.ORACLE DB software (11.2.0.3)
Storage used
NAS/SAN storage
We had faced some issues in storage configuration, finally it comes out with a misconfiguration in /etc/fstab file entry.
error description : due to this wrong mount point entry root.sh script when running from 2nd node it was not able to detect existing cluster so either it was finishing independently(without detecting 2nd node or it was failing with complaining about wrong fstab entry)
==============
Below is the symptoms of RAC installation failure if iptable service is on
Hi,
Please ignore the below error of node ocnnectivity. We have stopped iptable service , now cluster precheck passed... Now we are proceeding with installation with the /etc/fstab being modified as per ORACLE recommendation, would update further on it.
thanks
subhendra
Hi,
Together with the below also we are getting the below error in cluster pre-check
Check: TCP connectivity of subnet "10.184.132.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
SUB-233:10.184.134.233 SUB-234:10.184.134.234 failed
ERROR:
PRVF-7617 : Node connectivity between "SUB-233 : 10.184.134.233" and "SUB-234 : 10.184.134.234" failed
Result: TCP connectivity check failed for subnet "10.184.132.0"
Check: TCP connectivity of subnet "192.168.1.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
SUB-233:192.168.1.1 SUB-234:192.168.1.2 failed
ERROR:
PRVF-7617 : Node connectivity between "SUB-233 : 192.168.1.1" and "SUB-234 : 192.168.1.2" failed
Result: TCP connectivity check failed for subnet "192.168.1.0"
Hi,
After changing the fstab entry and rebooting server.The pre cluster verification fails with below error. Earlier before changing the fstab entry it was detecting fine.
runcluvfy.sh stage -pre crsinst -n SUB-233,SUB-234 -verbose
================
WARNING:
Could not find a suitable set of interfaces for the private interconnect
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "10.184.132.0".
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed.
Result: Node connectivity check failed
Checking multicast communication...
Checking subnet "10.184.132.0" for multicast communication with multicast group "230.0.1.0"...
PRVG-11134 : Interface "10.184.134.234" on node "SUB-234" is not able to communicate with interface "10.184.134.233" on node "SUB-233"
PRVG-11134 : Interface "10.184.134.233" on node "SUB-233" is not able to communicate with interface "10.184.134.234" on node "SUB-234"
PRVG-11134 : Interface "10.184.134.233" on node "SUB-233" is not able to communicate with interface "10.184.134.233" on node "SUB-233"
Checking subnet "10.184.132.0" for multicast communication with multicast group "224.0.0.251"...
PRVG-11134 : Interface "10.184.134.234" on node "SUB-234" is not able to communicate with interface "10.184.134.233" on node "SUB-233"
PRVG-11134 : Interface "10.184.134.233" on node "SUB-233" is not able to communicate with interface "10.184.134.234" on node "SUB-234"
PRVG-11134 : Interface "10.184.134.233" on node "SUB-233" is not able to communicate with interface "10.184.134.233" on node "SUB-233"
Checking subnet "192.168.1.0" for multicast communication with multicast group "230.0.1.0"...
PRVG-11134 : Interface "192.168.1.2" on node "SUB-234" is not able to communicate with interface "192.168.1.1" on node "SUB-233"
PRVG-11134 : Interface "192.168.1.1" on node "SUB-233" is not able to communicate with interface "192.168.1.2" on node "SUB-234"
PRVG-11134 : Interface "192.168.1.1" on node "SUB-233" is not able to communicate with interface "192.168.1.1" on node "SUB-233"
Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"...
PRVG-11134 : Interface "192.168.1.2" on node "SUB-234" is not able to communicate with interface "192.168.1.1" on node "SUB-233"
PRVG-11134 : Interface "192.168.1.1" on node "SUB-233" is not able to communicate with interface "192.168.1.2" on node "SUB-234"
PRVG-11134 : Interface "192.168.1.1" on node "SUB-233" is not able to communicate with interface "192.168.1.1" on node "SUB-233"
============================================================
So now iptable issue is resolved but still root.sh failed with ASM storage issue below is the details of it
=====================================================================
Hi
After changing the /etc/fstab entry as per ORACLE recommedation , again the root.sh failed showing the same error.
========================
[root@SUB-233 ~]# /oragrid/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /oragrid/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oragrid/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
OLR initialization - successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
Adding Clusterware entries to inittab
CRS-2672: Attempting to start 'ora.mdnsd' on 'SUB-233'
CRS-2676: Start of 'ora.mdnsd' on 'SUB-233' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'SUB-233'
CRS-2676: Start of 'ora.gpnpd' on 'SUB-233' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'SUB-233'
CRS-2672: Attempting to start 'ora.gipcd' on 'SUB-233'
CRS-2676: Start of 'ora.cssdmonitor' on 'SUB-233' succeeded
CRS-2676: Start of 'ora.gipcd' on 'SUB-233' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'SUB-233'
CRS-2672: Attempting to start 'ora.diskmon' on 'SUB-233'
CRS-2676: Start of 'ora.diskmon' on 'SUB-233' succeeded
CRS-2676: Start of 'ora.cssd' on 'SUB-233' succeeded
Disk Group OCRVOTE creation failed with the following message:
ORA-15018: diskgroup cannot be created
ORA-15031: disk specification '/asm_dat01-04/ocr0309' matches no disks
ORA-15025: could not open disk "/asm_dat01-04/ocr0309"
ORA-27054: NFS file system where the file is created or resides is not mounted with correct options
Configuration of ASM ... failed
see asmca logs at /oragrid/app/grid/cfgtoollogs/asmca for details
Did not succssfully configure and start ASM at /oragrid/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 6763.
/oragrid/app/11.2.0/grid/perl/bin/perl -I/oragrid/app/11.2.0/grid/perl/lib -I/oragrid/app/11.2.0/grid/crs/install /oragrid/app/11.2.0/grid/crs/install/rootcrs.pl execution failed
[root@SUB-233 ~]#
=========================
The above clearly says that the root.sh has failed due to incorrect mount options so next we see for possible error in /etc/fstab mount options
1. Update the fstab with below:
NAS.company.com:/export/asm_dat01-04 /asm_dat01-04 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0
2. Unmount and mount /asm_data01-04
3. Now perform the dd to check that the device is accessible
$dd if=/asm_dat01-04/ocr0309 of=/tmp/ocr.dd bs=1024 count=10
4. From GRID_HOME/bin
$kfod di=all asm_diskstring='/asm_data01-04/*'
5. $cd /asm_dat01-04/
$ls -ltr
existing /etc/fstab entry was like below
[root@SUB-234 ~]# cat /etc/fstab
LABEL=/1 / ext3 defaults 1 1
LABEL=/boot1 /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
LABEL=SWAP-sda2 swap swap defaults 0 0
NAS.company.com:/export/asm_crs01 /asm_crs01 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0 0 0
NAS.company.com:/export/asm_crs02 /asm_crs02 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0 0 0
NAS.company.com:/export/asm_dat01-04 /asm_dat01-04 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0 0 0
NAS.company.com:/export/asm_fra01-02 /asm_fra01-02 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0 0 0
/dev/mapper/vg00-lvol01 /home ext3 defaults 0 0
/dev/mapper/vg00-lvol2 /oracle ext3 defaults 0 0
/dev/mapper/vg00-lvol3 /oragrid ext3 defaults 0 0
======================================================
So after making the above changes our RAC installiton went smoothly without any issues
reference
http://www.oracle-base.com/articles/linux/using-nfs-with-asm.php
http://oracledba.org/11gR2/Pre_Install_11gR2.htm
