Oracle Apps DBA: Oracle Real Application Clusters Pre-requisites for HP-UX Itanium 11.31

Overview of Groups and Users for Oracle Clusterware Installations

You must create the following group and user with same GID and UID on all the cluster members to install Oracle Clusterware:

The Oracle Inventory group (typically, oinstall)
You must create this group the first time that you install Oracle software on the system. In Oracle documentation, this group is referred to as oinstall.

Oracle clusterware software owner user (typically, oracle, if you intend to create a single software owner user for all Oracle software, or crs, if you intend to create separate Oracle software owners.)
You must create at least one software owner the first time you install Oracle software on the system. This user owns the Oracle binaries of the Oracle Clusterware software, and you can also make this user the owner of the binaries of Automatic Storage Management and Oracle Database or Oracle RAC.
On HP-UX, the owner of Oracle Clusterware software must have the RTPRIO, MLOCK, and RTSCHED privileges.

Understanding the Oracle Inventory Directory

The first time you install Oracle software on a system, Oracle Universal Installer checks to see if you have created an Optimal Flexible Architecture (OFA) compliant path in the format u[01-09]/app, such as /u01/app, and that the user running the installation has permissions to write to that path. If this is true, then Oracle Universal Installer creates the Oracle Inventory directory in the path /u[01-09]/app/oraInventory. For example:

/u01/app/oraInventory

If you have set the environment variable $ORACLE_BASE for the user performing the Oracle Clusterware installation, then OUI creates the Oracle Inventory directory in the path $ORACLE_BASE/../oraInventory. For example, if $ORACLE_BASE is set to /opt/oracle/11, then the Oracle Inventory directory is created in the path /opt/oracle/oraInventory.

Creating the Oracle Clusterware User

You must create a software owner for Oracle Clusterware in the following circumstances:

If an Oracle software owner user does not exist; for example, if this is the first installation of Oracle software on the system
If an Oracle software owner user exists, but you want to use a different operating system user, such as crs, with different group membership, to give separate clusterware and database administrative privileges to those groups in a new Oracle Clusterware and Oracle Database installation

Creating or Modifying an Oracle Software Owner User for Oracle Clusterware

If the Oracle software owner (oracle, crs) user does not exist, or if you require a new Oracle software owner user, then create it. The following procedure uses crs as the name of the Oracle software owner.

To create a user, enter a command similar to the following:

# /usr/sbin/useradd -u 501 -g oinstall crs

Example of Creating the Oracle Clusterware User and OraInventory Path

The following is an example of how to create the Oracle Clusterware software owner (in this case, crs), and a path compliant with OFA structure with correct permissions for the oraInventory directory.

# mkdir -p  /u01/app/crs

# chown -R crs:oinstall /u01/app

# mkdir  /u01/app/oracle

# chown oracle:oinstall /u01/app/oracle

# chmod 775 /u01/app/

At the end of this procedure, you will have the following:

/u01 owned by root.
/u01/app owned by crs:oinstall with 775 permissions. This ownership and permissions enables OUI to create the oraInventory directory, in the path /u01/app/oraInventory.
/u01/app/crs owned by crs:oinstall with 775 permissions. These permissions are required for installation, and are changed during the installation process.
/u01/app/oracle owned by oracle:oinstall with 775 permissions.

Checking the Hardware Requirements

Each system must meet the following minimum hardware requirements:

At least 2 GB of physical RAM
Swap space equivalent to the multiple of the available RAM, as indicated in the following table:

Available RAM	Swap Space Required
Between 1 GB and 2 GB	1.5 times the size of RAM
Between 2 GB and 8 GB	Equal to the size of RAM
More than 8 GB	.75 times the size of RAM

800 MB of disk space in the /tmp directory
4 GB of disk space for Oracle Clusterware files, in partitions on separate physical disks, assuming standard redundancy (2 Oracle Cluster Registry partitions and 3 voting disks)
1000 MB of disk space for the Oracle Clusterware home
If you intend to install Oracle Database, allocate 8 GB of disk space for the Oracle base
If you intend to install Oracle Database single instance, allocate between 3 and 6 GB of disk space for a preconfigured database that uses file system storage.

Network Hardware Requirements

The following is a list of requirements for network configuration:

Each node must have at least two ports: one for the public network interface, and one for the private network interface (the interconnect).
If you want to use more than one NIC for the public network or for the private network, then Oracle recommends that you use NIC bonding.
The public interface names associated with the network adapters for each network must be the same on all nodes, and the private interface names associated with the network adaptors should be the same on all nodes.
For example: With a two-node cluster, you cannot configure network adapters on node1 with eth0 as the public interface, but on node2 have eth1 as the public interface. Public interface names must be the same, so you must configure eth0 as public on both nodes. You should configure the private interfaces on the same network adapters as well. If eth1 is the private interface for node1, then eth1 should be the private interface for node2.
For the public network, each network adapter must support TCP/IP.
For the private network, the interconnect must support the user datagram protocol (UDP) using high-speed network adapters and switches that support TCP/IP (Gigabit Ethernet or better required).
Oracle does not support token-rings or crossover cables for the interconnect.
For the private network, the endpoints of all designated interconnect interfaces must be completely reachable on the network. There should be no node that is not connected to every private network interface. You can test whether an interconnect interface is reachable using a ping command.

IP Address Requirements

Before starting the installation, you must have the following IP addresses available for each node:

An IP address with an associated host name (or network name) registered in the DNS for the public interface. If you do not have an available DNS, then record the host name and IP address in the system hosts file, /etc/hosts.
One virtual IP (VIP) address with an associated host name registered in a DNS. If you do not have an available DNS, then record the host name and VIP address in the system hosts file, /etc/hosts. Select an address for your VIP that meets the following requirements:
- The IP address and host name are currently unused (it can be registered in a DNS, but should not be accessible by a ping command)
- The VIP is on the same subnet as your public interface
A private IP address with a host name for each private interface
Oracle recommends that you use private network IP addresses for these interfaces (for example: 10.*.*.* or 192.168.*.*). You can use DNS servers, or the /etc/hosts file, or both to register the private IP address. Note that if you use DNS servers alone, and the public network becomes unreachable due to NIC or cable failure, then the private IP addresses can fail to resolve.
For the private interconnects, because of Cache Fusion and other traffic between nodes, Oracle strongly recommends using a physically separate, private network. You should ensure that the private IP addresses are reachable only by the cluster member nodes.

For example, with a two node cluster where each node has one public and one private interface, you might have the configuration shown in the following table for your network interfaces, where the hosts file is /etc/hosts:

Node	Host Name	Type	IP Address	Registered In
node1	node1	Public	143.46.43.100	DNS (if available, else the hosts file)
node1	node1-vip	Virtual	143.46.43.104	DNS (if available, else the hosts file)
node1	node1-priv	Private	10.0.0.1	Hosts file
node2	node2	Public	143.46.43.101	DNS (if available, else the hosts file)
node2	node2-vip	Virtual	143.46.43.105	DNS (if available, else the hosts file)
node2	node2-priv	Private	10.0.0.2	Hosts file

Node Time Requirements

Before starting the installation, ensure that each member node of the cluster is set as closely as possible to the same date and time. Oracle strongly recommends using the Network Time Protocol feature of most operating systems for this purpose, with all nodes using the same reference Network Time Protocol server.

Configuring the Network Requirements

To verify that each node meets the requirements, follow these steps:

If necessary, install the network adapters for the public and private networks and configure them with either public or private IP addresses.
If you are using a domain name server (DNS), then for each node, register the host names and IP addresses for the public network interfaces in the DNS.
Even if you are using a DNS, Oracle recommends that you add lines to the /etc/hosts file on each node, specifying the private IP addresses and associated private host names. Oracle also recommends that you add public and virtual IP addresses. Configure the /etc/hosts file so that it is similar to as shown in the following example, with private interface eth1, and private hosts nodeint1 and nodeint2:, where xxx represents parts of a valid IP address.
```
#eth0 - PUBLIC
```

xxx.xxx.100.45   node1.example.com    node1

xxx.xxx.100.46   node2.example.com    node2

```
#eth1 - PRIVATE
```

10.0.0.1      nodeint1.example.com    nodeint1

10.0.0.2      nodeint2.example.com    nodeint2

```
#VIPs
```

xxx.xxx.100.47   pmvip1.example.com    nodevip1

xxx.xxx.100.48   pmvip2.example.com    nodevip2

To check network configuration, on each node, enter the following commands:
```
# hostname
							
```

# /usr/sbin/ifconfig interface_name

You can also check with the following command:
```
# usr/bin/netstat -in
```
Ensure that each server is properly identified, and that the interface name and IP address for all network adapters that you want to specify as public or private network interfaces are properly configured. In addition, use the ping command to ensure that each node can obtain a response for the public and private IP addresses from each other node in the cluster.

Configuring the Name Service Switch to Tolerate Public Network Failures

On HP-UX, to tolerate a complete public network failure, you should specify network addresses in /etc/nsswitch.conf, to avoid VIP failover or public address network failure response times being dependent on the network timeouts.

In the /etc/nsswitch.conf file, files must precede other entries for host, and preferably precede other entries in nsswitch.conf.

Identifying Software Requirements

HP-UX 11.31 packages and bundles	PHKL_35900 (operating system patch) PHKL_35936 (operating system patch) PHKL_36248 (operating system patch) PHKL_36249 (operating system patch)
Oracle JDBC/OCI Drivers	Java SDK 5.0 Note: HP JDK 1.5.0.06 is installed with this release.
Oracle Spatial	HP-UX Developer's Toolkit - X11 and Imake Note: This software is required only to build the sample programs.
Pro*COBOL complier	Micro Focus Server Express 5.0
Pro*Fortran compiler	HP-UX 11.31: HP FORTRAN/90 - Feb 2007 - release
Pro*C/C++, Oracle Call Interface, Oracle C++ Call Interface, Oracle XML Developer's Kit (XDK)	HP-UX 11.31 A.06.15 (September 2007 Application Release CD, swlist bundle C.11.31.03)
Oracle Clusterware	All HP-UX 11.31 installations No additional requirements for Oracle Clusterware. At the time of this release, Hyper Messaging Protocol (HMP) is not supported. Only HP Serviceguard A11.17.01 is supported with Oracle Clusterware 11g. Note: HP Serviceguard is optional. It is required only if you want to use shared logical volumes for Oracle Clusterware or database files.

Configuring Kernel Parameters

Note:

The kernel parameter values shown in this section are recommended values only. For production database systems, Oracle recommends that you tune these values to optimize the performance of the system. See your operating system documentation for more information about tuning kernel parameters.

On HP-UX 11.31 (version 3), the following parameters are not valid:

msgmap
ncallou

Parameter	Recommended Formula or Value
ksi_alloc_max	(nproc*8)
executable_stack	0
max_thread_proc	1024
maxdsiz	1073741824 (1 GB)
maxdsiz_64bit	2147483648 (2 GB)
maxssiz	134217728 (128 MB)
maxssiz_64bit	1073741824 (1 GB)
maxuprc	((nproc*9)/10)
msgmap	(msgtql+2)
msgmni	(nproc)
msgseg	32767
msgtql	(nproc)
ncsize	(ninode+1024)
nfile	(15*nproc+2048)
nflocks	(nproc)
ninode	(8*nproc+2048)
nkthread	(((nproc*7)/4)+16)
nproc	4096
semmni	(nproc)
semmns	(semmni*2)
semmnu	(nproc-4)
semvmx	32767
shmmax	The size of physical memory or 1073741824 (0X40000000), whichever is greater. Note: To avoid performance degradation, the value should be greater than or equal to the size of the SGA.
shmmni	4096
shmseg	512
vps_ceiling	64

Configuring SSH or RCP on All Cluster Nodes

Configuring RCP on Cluster Member Nodes

To enable remote copy:

On each node in the cluster, using a text editor, create or open the .rhosts file in the root account home directory. Add lines similar to the following, where node1 and node2 are member nodes in the cluster, and domain is the domain name for the cluster:
```
node1 root
```
```
node1.domain root
```
```
node2 root
```
```
node2.domain root
```
If you create the .rhosts file, then ensure that permissions for the .rhost file are set to 400. For example:
```
# chmod 400 .rhosts
```
For each Oracle installation owner account, open the .rhosts file in that user account home directory (in this example, crs). Add lines similar to the following, where node1 and node2 are member nodes in the cluster, and domain is the domain name for the cluster:
```
node1 crs
```
```
node1.domain crs
```
```
node2 crs
```
```
node2.domain crs
```
If you create the .rhosts file, then ensure that permissions for the .rhost file are set to 400. For example:
```
# chmod 400 .rhosts
```
Test the RCP configuration for root and for Oracle installation owner accounts. For example:
```
[root@node1] # remsh node2 11
```
```
[root@node1] # remsh node1 11
```
```
[root@node2] # remsh node1 11
```
```
[root@node2} # remsh node2 11
```
```
[crs@node1] $ remsh node2 11
```
```
[crs@node1] $ remsh node1 11
```
```
[crs@node2] $ remsh node1 11
```
```
[crs@node2} $ remsh node2 11
```

Configuring SSH on Cluster Member Nodes

To configure SSH, you must first create RSA or DSA keys on each cluster node, and then copy all the keys generated on all cluster node members into an authorized keys file that is identical on each node. Note that the SSH files must be readable only by root and by the software installation user (oracle, crs, asm), as SSH ignores a private key file if it is accessible by others. When this is done, then start the SSH agent to load keys into memory. In the examples that follow, the RSA key is used.

You must configure SSH separately for each Oracle software installation owner that you intend to use for installation.

To configure SSH, complete the following:

Create .SSH, and Create RSA Keys On Each Node

Complete the following steps on each node:

Log in as the software owner (in this example, the crs user).
To ensure that you are logged in as the Oracle user, and that the user ID matches the expected user ID you have assigned to the Oracle user, enter the commands id and id crs. Ensure that Oracle user group and user and the terminal window process group and user IDs are identical. For example:
```
$ id 
```

uid=502(crs) gid=501(oinstall) groups=501(oinstall),502(crs)

```
$ id crs
```

uid=502(crs) gid=501(oinstall) groups=501(oinstall),502(crs)

If necessary, create the .ssh directory in the crs user's home directory, and set permissions on it to ensure that only the oracle user has read and write permissions:
```
$ mkdir ~/.ssh
```
```
$ chmod 700 ~/.ssh
```
Enter the following command:
```
$ /usr/bin/ssh-keygen -t rsa
```
At the prompts:
- Accept the default location for the key file (press Enter).
- Enter and confirm a pass phrase unique for this installation user.
This command writes the RSA public key to the ~/.ssh/id_rsa.pub file and the private key to the ~/.ssh/id_rsa file.
Never distribute the private key to anyone not authorized to perform Oracle software installations.
Repeat steps 1 through 4 on each node that you intend to make a member of the cluster, using the RSA key.

Add All Keys to a Common authorized_keys File

Complete the following steps:

On the local node, change directories to the .ssh directory in the Oracle Clusterware owner's home directory (typically, either crs or oracle).
Then, add the RSA key to the authorized_keys file using the following commands:
```
$ cat id_rsa.pub >> authorized_keys
```
```
$ ls
```
In the .ssh directory, you should see the id_rsa.pub keys that you have created, and the file authorized_keys.
On the local node, use SCP (Secure Copy) or SFTP (Secure FTP) to copy the authorized_keys file to the oracle user .ssh directory on a remote node. The following example is with SCP, on a node called node2, with the Oracle Clusterware owner crs, where the crs user path is /home/crs:

[crs@node1 .ssh]$ scp authorized_keys node2:/home/crs/.ssh/

You are prompted to accept an RSA key. Enter Yes, and you see that the node you are copying to is added to the known_hosts file.

When prompted, provide the password for the crs user, which should be the same on all nodes in the cluster. The authorized_keys file is copied to the remote node.

Your output should be similar to the following, where xxx represents parts of a valid IP address:

[crs@node1 .ssh]$ scp authorized_keys node2:/home/crs/.ssh/

The authenticity of host 'node2 (xxx.xxx.173.152) can't be established.

RSA key fingerprint is 7e:60:60:ae:40:40:d1:a6:f7:4e:zz:me:a7:48:ae:f6:7e.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'node1,xxx.xxx.173.152' (RSA) to the list

of known hosts

crs@node2's password:

authorized_keys       100%             828             7.5MB/s      00:00

Using SSH, log in to the node where you copied the authorized_keys file, using the pass phrase you created. Then change to the .ssh directory, and using the cat command, add the RSA keys for the second node to the authorized_keys file:
```
[crs@node1 .ssh]$ ssh node2
```

The authenticity of host node2 (xxx.xxx.100.102) can't be established. RSA key fingerprint is z3:z3:33:z3:z3:33:zz:76:z3:z3:z3.

Are you sure you want to continue connecting? (yes/no)? yes

Enter passphrase for key '/home/oracle/.ssh/id_rsa':

```
[crs@node2 crs]$ cd .ssh
```
```
[crs@node2 ssh]$ cat id_rsa.pub  >> authorized_keys
```
Repeat steps 2 and 3 from each node to each other member node in the cluster.
When you have added keys from each cluster node member to the authorized_keys file on the last node you want to have as a cluster node member, then use scp to copy the authorized_keys file with the keys from all nodes back to each cluster node member, overwriting the existing version on the other nodes.
If you want to confirm that you have all nodes in the authorized_keys file, enter the command more authorized_keys, and check to see that there is an RSA key for each member node. The file lists the type of key (ssh-rsa), followed by the key, and then followed by the user and server. For example:
```
ssh-rsa AAAABBBB . . . = crs@node1
```

Enabling SSH User Equivalency on Cluster Member Nodes

To enable Oracle Universal Installer to use the ssh and scp commands without being prompted for a pass phrase, follow these steps:

On the system where you want to run Oracle Universal Installer, log in as the oracle user.
Enter the following commands:
```
$ exec /usr/bin/ssh-agent $SHELL
```
```
$ /usr/bin/ssh-add
```
At the prompts, enter the pass phrase for each key that you generated.
If you have configured SSH correctly, then you can now use the ssh or scp commands without being prompted for a password or a pass phrase.
If you are on a remote terminal, and the local node has only one visual (which is typical), then use the following syntax to set the DISPLAY environment variable:
Bourne or Korn shell:
```
$ export DISPLAY=hostname:0
```
C shell:
```
$ setenv DISPLAY hostname:0
```
For example, if you are using the Bourne shell, and if your hostname is node1, then enter the following command:
```
$ export DISPLAY=node1:0
```
To test the SSH configuration, enter the following commands from the same terminal session, testing the configuration of each cluster node, where nodename1, nodename2, and so on, are the names of nodes in the cluster:
```
$ ssh nodename1 date
```
```
$ ssh nodename2 date
```

These commands should display the date set on each node.

If any node prompts for a password or pass phrase, then verify that the ~/.ssh/authorized_keys file on that node contains the correct public keys.

If you are using a remote client to connect to the local node, and you see a message similar to "Warning: No xauth data; using fake authentication data for X11 forwarding," then this means that your authorized keys file is configured correctly, but your ssh configuration has X11 forwarding enabled. To correct this issue, proceed to step 6.

To ensure that X11 forwarding will not cause the installation to fail, create a user-level SSH client configuration file for the Oracle software owner user, as follows:
You must run Oracle Universal Installer from this session or remember to repeat steps 2 and 3 before you start Oracle Universal Installer from a different terminal session.

Configuring Software Owner User Environments

To set the Oracle software owners' environments, follow these steps, for each software owner (crs, oracle, asm):

Start a new terminal session; for example, start an X terminal (xterm).
Enter the following command to ensure that X Window applications can display on this system:
```
$ xhost + hostname
								
```
The hostname is the name of the local host.
If you are not already logged in to the system where you want to install the software, then log in to that system as the software owner user.
If you are not logged in as the user, then switch to the software owner user you are configuring. For example, with the crs user:
```
$ su - crs
```
To determine the default shell for the user, enter the following command:
```
$ echo $SHELL
```
Open the user's shell startup file in any text editor:
If the ORACLE_SID, ORACLE_HOME, or ORACLE_BASE environment variable is set in the file, then remove the appropriate lines from the file.
Save the file, and exit from the text editor.
To run the shell startup script, enter one of the following commands:
If you are not installing the software on the local system, then enter a command similar to the following to direct X applications to display on the local system:
In this example, local_host is the host name or IP address of the system that you want to use to display OUI (your workstation or PC).
If you determined that the /tmp directory has less than 1000 MB of free disk space, then identify a file system with at least 1000 MB of free space and set the TEMP and TMPDIR environment variables to specify a temporary directory on this file system:
Creating Required Symbolic Links

To enable you to successfully relink Oracle products after installing this software, enter the following commands to create required X library symbolic links in the /usr/lib directory:

# cd /usr/lib

# ln -s libX11.3 libX11.sl

# ln -s libXIE.2 libXIE.sl

# ln -s libXext.3 libXext.sl

# ln -s libXhp11.3 libXhp11.sl

# ln -s libXi.3 libXi.sl

# ln -s libXm.4 libXm.sl

# ln -s libXp.2 libXp.sl

# ln -s libXt.3 libXt.sl

# ln -s libXtst.2 libXtst.sl

Requirements for Creating an Oracle Clusterware Home Directory

During installation, you are prompted to provide a path to a home directory to store Oracle Clusterware binaries. Ensure that the directory path you provide meets the following requirements:

It should be created in a path separate from existing Oracle homes.
It should not be located in a user home directory.
It should be created either as a subdirectory in a path where all files can be owned by root, or in a unique path.
Before installation, it should be owned by the installation owner of Oracle Clusterware (typically oracle for a single installation owner for all Oracle software, or crs for role-based Oracle installation owners), and set to 750 permissions.

For installations with Oracle Clusterware only, Oracle recommends that you create a path compliant with Oracle Optimal Flexible Architecture (OFA) guidelines, so that Oracle Universal Installer (OUI) can select that directory during installation. For OUI to recognize the path as an Oracle software path, it must be in the form u0[1-9]/app.

When OUI finds an OFA-compliant path, it creates the Oracle Clusterware and Oracle Central Inventory (oraInventory) directories for you.

Create an Oracle Clusterware path. For example:

# mkdir -p  /u01/app

# chown -R crs:oinstall /u01

Alternatively, if you later intend to install Oracle Database software, then create an Oracle base path. OUI automatically creates an OFA-compliant path for Oracle Clusterware derived from the Oracle base path. The Optimal Flexible Architecture path for the Oracle Base is /u01/app/user, where user is the name of the user account that you want to own the Oracle Database software. For example:

# mkdir -p  /u01/app/oracle

# chown -R oracle:oinstall /u01/app/oracle

# chmod -R 775 /u01/app/oracle

Using CVU to Determine if Installation Prerequisites are Complete

You can use Cluster Verification Utility (CVU) to determine which system prerequisites for installation are already completed. Use this option if you are installing Oracle 11g release 1 (11.1) on a system with a pre-existing Oracle software installation. In using this option, note the following:

You must complete the prerequisites for using Cluster Verification Utility, notably configuring SSH between all nodes in the cluster, before you can complete a clusterwide status check.
Cluster Verification Utility can assist you by finding preinstallation steps that need to be completed, but it cannot perform preinstallation tasks

Use the following syntax to determine what preinstallation steps are completed, and what preinstallation steps must be performed

$ ./runcluvfy.sh stage -pre crsinst -n node_list

In the preceding syntax example, replace the variable node_list with the names of the nodes in your cluster, separated by commas.

For example, for a cluster with mountpoint /mnt/dvdrom/, and with nodes node1, node2, and node3, enter the following command:

$ cd /mnt/dvdrom/

$ ./runcluvfy.sh stage -pre crsinst -n node1,node2,node3

Oracle Apps DBA

Wednesday, June 22, 2011

Oracle Real Application Clusters Pre-requisites for HP-UX Itanium 11.31