Oracle 11g RAC Installation on Linux
Prepare all nodes for Cluster Software Installation:
Change /etc/hosts file to contain name of all nodes or DNS should be able to provide service name if you are not using SCAN (For 11gR2 release only).
On both server run /usr/bin/ssh-keygen -t rsa from oracle account.
When prompted press enter (for now.)
all key files are located in ~/.ssh directory
1. id_rsa - is private key do not distribute.
2. id_rsa.pub is public key. (Copy this content to another server in authorzed_keys file.)
Similarly generate dsa key also and place in all nodes authorized_keys file.
Once copy is complete verify ssh is working from both server using
From node1 run ssh node2 date
From node2 run ssh node1 date
Verify that ntp server is runnin. Also make sure that both machine shows same time using following command:
date ;ssh anothernodename date
both should print same time.
If not use ntpdate command (check linux link).
Use RAID Software to create proper 3 LUNs (check linux link for more detail.)
Configure public and private networks using /usr/sbin/system-config-network for network adapters.
Verify those adapter are up and running using ifconfig command.
if particular adapter is not up then command ifup to restart those adapter and make sure it shows correct IP addresses and those IPs are pingable.
For private networks try to use network adapter that uses Interconnect/RDMA technology.
Make sure that firewall is not filtering UDP packet on private network (Because Oracle is using UDP packet for private network.)
verify that using /etc/rc.d/init.d/iptables status command.
The following command display current iptables settings.
Make backup of iptable files before you make any changes.
p>To take backup use /etc/init.d/iptables save command
then copy /etc/sysconfig/iptables BACKUPFILENAME
If you have Oracle Enterprise Linux installed then you can use Oracle public yum server to update:
To configure you can goto /etc/yum.repos.d and try following command
public-yum-el5.repo, public-yum-el4.report or public-yum-ovm2.repo depending on your configuration and set enabled=1 accordingly.
Once updated correctly you can see list of packages by issuying
If vncserver is installed on server and would like to use it then you have to start vncserver if it is not running.
first time it will ask you for password.
To enable desktop on vnc server change file /root/.vnc/xstartup.
#Uncomment the following two lines for normal desktop :
To generate kickstart file you can always use kickstart program.
To add/remote program on Linux with gui you can use pirut program.
You can copy all installation files on drive using following command.
mount -p loop disk1.iso /mnt/iso
copy all files to one location for example /root/binaries/linux/RPMS/
Repeat above instruction for all CDs and copy it in same directory.
or use DVD drive to mount DVD media
mount /dev/cdrom /mnt/cdrom
Once files are copied you can change directory to repodata under RPMS and run following command (optional)
createrepo -g comps-rhel5-server-core.xml .
Edit yum config file from /etc/yum.repos.d and add [oel5] tag and its content in file.
name=Enterprise Linux 5
Then run yum clean all and run pirut that should give you original installation screen.
Verify partition using cat /proc/partition command.
For EMC Fibre Channel storage make sure that EMC Navisphere agent is installed on each node. And on storage make sure that Navisphere software assign the correct storage group to each node.
If you have SCSI storage then those devices display as sdb, sdc and so on. The LUNs on Fibre Channel storage or SCSI enclosure should also be seen as SCSI devices.
With one RAID container it should show as sda and if you have two Logical Disk Groups then those will be display as sdb and sdc. If you have Fibre Channel Controller that should show as sda and RAID group on on storage will show as emcpowera and emcpowerb as pseudo devices.
Make sure that all nodes on cluster is able to see same number of LUNs. (Duplicate mount of same lun is also not allowed so if you use multiple device path to access same storage Oracle will error out.)
If you dont see any external storage devices for a Fibre Channel system thenstop services by following commands:
service naviagent stop
service PowerPath stop
Then sync HBA driver by reloading it using
Above example is for QLogic HBAs
For Emulex HBA
Then restart services using
service PowerPath start
service naviagent start
For SCSI enclosure reboot all nodes.
For iSCSI storage use following command to stop start services
service iscsi stop
service iscsi start
Before start installation set host equivalence for oracle user from all nodes otherwise it will give an error PRVF-4007.
exec /usr/bin/ssh-agent $SHELL
Create directory strcture for OFA on both nodes.
mkdir -p /u01/app/oracle
chown -R oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/app/oracle
To install Oracle ASM libraries for Linux, make sure that you are using version that match with your kernel, use uname -r command to find your kernel information. Install the following three packages using root user on all nodes.
- oracleasm-support - contains binary files (/usr/sbin/*) for supporting oracleasm
- oracleasm modules - which is kernel loadable module
- oracleasmlib - Library files needed to support oracleasm
rpm -ivh oracleasm-support-2.1.3-1.el5.i386.rpm
rpm -ivh oracleasm-2.6.18-194.el5-2.0.5-1.el5.i686.rpm
rpm -ivh oracleasmlib-2.0.4-1.el5.i386.rpm
Display configure oracle asm using
Configure oracle asm using oracleasm -i parameter
/usr/sbin/oracleasm configure -i
default user : oracle
default group : oinstall
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y] : y
This load oracle ASMLib driver filesystem, creates /etc/sysconfig/oracleasm file and creates /dev/oracleasm mount point.
If there is any problem loading module then check /var/log/oracleasm log file to find out more to resolve issue.
/usr/sbin/oracleasm init to load driver.
You should see message Mounting ASMLib driver filesystem: /dev/oracleasm
In case if it fails then it will log messages like Unable to load modules "oracleasm" then run up2date -i oracleasm-`uname -r`
To add package using rpm use
rpm -ivh packagename.rpm
and to remove package use
rpm -e packagename.rpm
In case if you want to find out which packages are loaded use rpm -qa |grep -in 'oracle'
Prepare ASM volumes
Use fdisk /dev/sde to partition the volume.
create new partition using n command when ask for partittion type select primary.
select first partition by selecting 1 and then create from 1st Cylinder by typing First Cylinder value : 1
Last Cylinder or +size or +sizeM or +sizeK (xxx) :512
similarly create another partition of same size but partition number 2. (from 512 to 1024)
Once all partition created run
By using partprobe command if you are using SCSI LUN then ownership will change for kernel 2.6 and above
Then assign those partition to raw device using raw command
raw /dev/raw/raw1 /dev/sdc1
raw /dev/raw/raw2 /dev/sdd2
Perform similar action on other nodes based on their mount point(sometime it may be different because their number of disk may be different.) so finally map it on /dev/raw/rawN correctly so from all node it can access.
Also add entry in /etc/sysconfig/rawdevices
So next time it boot it will autogenerate those devices.
If you have kernel 2.6 then create file /etc/udev/rules.d/99-raw.rules (for Oracle RHE5, OEL5, and SLES9)
KERNEL=="raw[1-2]*", OWNER="oracle",GROUP="oinstall", MODE="640"
First line will take raw1 and raw2 and change ownership to oracle:oinstall and set mode to 640.
now add those disk in asm by giving the following command (You may do this for 11gR2 because OCR and Voting Disk can be placed on ASM).
/usr/sbin/oracleasm createdisk DG_CRS1 /dev/raw/raw1
To display list use
/usr/sbin/oracleasm configure/init/exit/scandisk/status/listdisks/querydisk/createdisk/deletedisk/renamedisk/update-driver are options
In case if you have already existing asm disk then (in case of backup restored) use /usr/sbin/oracleasm scandisk (usually from other nodes.)
Try following command to start Oracle Universal Installer.
On Welcom click next
Specify Inventory directory and credentials
Enter Full path of the inventory directory : /u01/app/oraInventory
Specify Operating System group name :
You will see Specify Home Details
Path : /u01/crs
Product-specific Prerequisite Checks will verify all setup. if anything failed then correct that and come back on this screen.
Specify Cluster Configuration
Cluster Name : clustername_cluster
Cluster Nodes : node01.domain.com
Private Nodes : node01-priv.domain.com
Virtual Host Name : node01-vip.domain.com
Click on Add to add more nodes . for example this is two cluster nodes then
Cluster Nodes : node02.domain.com
Private Nodes : node02-priv.domain.com
Virtual Host Name : node02-vip.domain.com
Then click OK
Specify Network Interface Usage
Make sure that at least one adapter is public and another adapter is set to private.
If you have network adapter that are not in use or belongs to another purpose then select do not use by clicking on edit.
If Interface_type is wrong for given adapter then change it by clicking on Edit and select from Interface Type ( Public / Private / Do Not Use).
Then click Next
OCR Config : /dev/raw/raw1
Voting Disk Location : External Redundancy : /dev/raw/raw2 (Usually these should be on different set of disks)
Make sure that both nodes under Remote Nodes are checked before resuming further.
Provide OCR and CRS drive path /dev/raw/raw1 and /dev/raw/raw2 and click Next.
After installation it will ask you to run script file /u01/app/crs/root.sh
Oracle recommend that you install OCR, OCR Mirror and Voting Disks on separate set of disks. It will impact your performance if these devices are slow.
In case if you get Shared Librarires /u01/app/crs/lib/libnnz11.so: cannot restore segment prot after reloc: Permission denied
Failed to upgrade Oracle Clusterware Registry configuration
error then run the following command (Usually CentOS):
This will print as Enforcing.
After this set to permission
Run getenforce and it should display Permissive and /etc/sysconfig/selinux and change SELINUX=disabled or permissive.
If vip or any node address is set wrong then you can correct it.
Make sure everything shutdown except crs stack
srvctl stop database -d dbname
srvctl stop nodeapps -n node1
srvctl stop nodeapps -n node2
To display command run as oracle user :
oifcfg delif -global eth0
oifcfg setif -global eth0/10.0.0.1:public
To modify VIP address run as root
/u01/app/crs/bin/srvctl modify nodeapp -n node1 -A 10.0.0.101/255.255.255.0/eth0
Then shutdown crs using crsctl stop crs command and restart all nodes.
Create data disk if you have not create earlier (The following example create DATA1DG1 disk group on Coraid storage).
/etc/init.d/oracleasm createdisk DATA1DG1 /dev/etherd/e0.2p3
Change directory to oracle binaries directory cd binaries/database
Welcome screen click Next
Select Custom click Next
Select Oracle Base /u01/app/asm
Specify Hardware Cluster Installation Mode
click all nodes then next
On product-specifc Perequisite Checks click Next if pass.
On Available product Components screen please select the components you have license for then click Next.
Select Privileged Operating System Groups
Select database administrator (OSDBA) Group : dba
Select Databae Operator (OSOPER) Group : dba
Select ASM administrator (OSASM) Group : dba
On Create Database screen select Configure Automatic Storage Management (ASM) then click Next.
On summary page make sure that all Remote Nodes are checked then click Install
When binaries are installed on Configuraton Assistant page You will configure Listener using Oracle Net Configuration Assistant
Type LISTENER name as LISTENER clic Next
On next screen select No, I do not want to configure additional naming methods.
Database Configuration Assistant, Step 1 of 3 type SYS password and Create server parameter file (SPFILE) then clic Next
ASM Creation - It will create ASM Instance.
Run /u01/app/asm/product/11gr1/root.sh on both nodes
Click Exit on End of Installation.
ORACLE PRVF-5436 : The NTP daemon running on one or more nodes lacks he slewing option "-x"
To resolve this editing /etc/sysconfig/ntpd to
OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid"
OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -x"