3 Installing and Configuring Oracle Clusterware and Oracle RAC

Configuring the Operating System Environment

You run Oracle Universal Installer from the oracle user account. However, before you start Oracle Universal Installer you must configure the environment of the oracle user. You must set the ORACLE_SID and ORACLE_BASE environment variables to the desired values for your environment.

For example, if you want to create an Oracle database named sales on the mount point directory /opt/oracle, you would set ORACLE_SID to sales and ORACLE_BASE to the directory /opt/oracle/10gR2.

To modify the user environment on Red Hat Linux:

1. As the oracle user, modify the user profile in the /home/oracle directory on both nodes using the following commands:

   [oracle] $ cd $HOME
   [oracle] $ vi .bash_profile
   
   

   Add the following lines at the end of the file:

   export ORACLE_SID=sales
   export ORACLE_BASE=/opt/oracle/10gR2
   export ORACLE_HOME=/opt/oracle/crs
   export PATH=$PATH:$ORACLE_HOME/bin
   
   

   In the previous example, the ORACLE_HOME variable has been set to the location of the Oracle Clusterware home directory. After Oracle Clusterware has been installed the ORACLE_HOME environment variable will be modified to reflect the value of the Oracle Database home directory.

2. Read and execute the changes made to the .bash_profile file:

   source .bash_profile
   
   

Verifying the Configuration Using the Cluster Verification Utility

If you have not configured your nodes, network, and operating system correctly, your installation of the Oracle Clusterware or Oracle Database 10g software will not complete successfully.

As the oracle user, change directories to the staging directory for the Oracle Clusterware software, or to the mounted installation disk. Then, enter the following command to verify your hardware and operating system setup, where staging_area is the location of the installation media (for example, /home/oracle/downloads/10gR2/10.2.0 or /dev/dvdrom):

[oracle] $ cd /staging_area/clusterware/cluvfy
[oracle] $ ./runcluvfy.sh stage -pre crsinst -n docrac1,docrac2 -verbose

The preceding command instructs the CVU to verify that the system meets all the criteria for an Oracle Clusterware installation. It checks that all the nodes are reachable from the local nodes, proper user equivalence exists, connectivity exists between all the nodes through the public and private interconnects, the user has proper permissions to install the software, and that all system requirements (including kernel version, kernel parameters, memory, swap space, temporary directory space, required software packages) are met.

Using Oracle Universal Installer to Install Oracle Clusterware

As the oracle user on the docrac1 node, install Oracle Clusterware. Note that OUI uses Secure Shell (SSH) to copy the binary files from docrac1 to docrac2 during the installation.

To install Oracle Clusterware:

1. Use the following command to start Oracle Universal Installer, where staging_area is the location of the staging area on disk, or the location of the mounted installation disk:

   cd /staging_area/clusterware
   ./runInstaller
   
   

2. The OUI Welcome window appears. Click Next.

3. If you have not installed any Oracle software previously on this server, the Specify Inventory directory and credentials window appears. The path displayed for the inventory directory should be the oraInventory subdirectory of your Oracle base directory. For example, if you set the ORACLE_BASE environment variable to /opt/oracle/10gR2 before starting OUI, then the path displayed is /opt/oracle/10gR2/oraInventory. For the operating system group name, choose oinstall. Click Next.

   
   Description of the illustration install031.gif
   
   

4. The Specify Home Details window appears. Accept the default value for the Name field, which is the name of the Oracle home directory for this product. For the Path field, click Browse to go to and select the directory /opt/oracle/crs, if this path is not already displayed.

   After you have selected the path, click Next.

5. The next window, Product-Specific Prerequisite Checks, appears after a short period of time. When you see the message "Check complete. The overall result of this check is: Passed", as shown in the following screen shot, click Next.

   
   Description of the illustration install27.gif
   
   

6. The Specify Cluster Configuration window appears.

   Change the default cluster name from crs to a name that is unique throughout your entire enterprise network. For example, you might choose a name that is based on the node names' common prefix. This guide will use the cluster name docrac.

   The local node, docrac1, appears in the Cluster Nodes section. If the cluster node names include the domain name, click Edit and remove the domain name from the public, private, and virtual node names. For example, if the node name is docrac1, edit the entries so that they are displayed as docrac1, docrac1-priv, and docrac1-vip. When you have finished removing the domain names in the "Modify a node in the existing cluster" window, click OK.

   When you are returned to the Specify Cluster Configuration window, click Add.

   In the "Add a new node to the existing cluster" dialog window, enter the second node's public name (docrac2), private name (docrac2-priv), and virtual IP name (docrac2-vip), then click OK.

   The Specify Cluster Configuration window now displays both nodes in the Cluster Nodes section.

   
   Description of the illustration install003.gif
   
   

   Click Next.

7. The Specify Network Interface Usage window appears. Verify eth0 and eth1 are configured correctly (proper subnet and interface type displayed), then click Next.

   The Specify Oracle Cluster Registry (OCR) Location window appears.

8. Choose Normal Redundancy for the OCR Configuration. You will be prompted for two file locations. In the Specify OCR Location field enter the name of the device configured for the first OCR file. For example, /dev/raw/raw1. In the Specify OCR Mirror Location field, enter the name of the device configured for the OCR mirror file, for example /dev/raw/raw2. When finished, click Next. During installation, the OCR data will be written to the specified locations.

   
   Description of the illustration install028.gif
   
   

   The Specify Voting Disk Location window appears.

9. Select Normal Redundancy for the voting disk location. You will be prompted for three file locations. For the Voting Disk Location, enter the name of the device configured for the first voting disk file, for example, /dev/raw/raw3. Repeat this process for the other two Voting Disk Location fields. When finished, click Next.

   
   Description of the illustration install005.gif
   
   

10. The OUI Summary window appears. Review the contents of the Summary window and then click Install.

    OUI displays a progress indicator during the installation process.

11. During the installation process, the Execute Configuration Scripts window appears. Do not click OK until you have run the scripts.

    
    Description of the illustration install033.gif
    
    

    The Execute Configuration Scripts window shows configuration scripts, and the path where the configuration scripts are located. Run the scripts on all nodes as directed, in the order shown. For example, on Red Hat Linux you perform the following steps (note that for clarity, the examples show the current user, node and directory in the prompt):

    1. As the oracle user on docrac1, open a terminal window, and enter the following commands:

       [oracle@docrac1 oracle]$ cd /opt/oracle/10gR2/oraInventory
       [oracle@docrac1 oraInventory]$ su
       
       

    2. Enter the password for the root user, and then enter the following command to run the first script on docrac1:

       [root@docrac1 oraInventory]# ./orainstRoot.sh
       
       

    3. After the orainstRoot.sh script finishes on docrac1, open another terminal window, and as the oracle user, enter the following commands:

       [oracle@docrac1 oracle]$ ssh docrac2
       [oracle@docrac2 oracle]$ cd /opt/oracle/10gR2/oraInventory
       [oracle@docrac2 oraInventory]$ su
       
       

    4. Enter the password for the root user, and then enter the following command to run the first script on docrac2:

       [root@docrac2 oraInventory]# ./orainstRoot.sh
       
       

    5. After the orainstRoot.sh script finishes on docrac2, go to the terminal window you opened in step b. As the root user on docrac1, enter the following commands to run the second script, root.sh:

       [root@docrac1 oraInventory]# cd /opt/oracle/crs
       [root@docrac1 crs]# ./root.sh
       
       

       Note:

       Do not attempt to run the root.sh script on other nodes. Wait until the script finishes running on the local node.

       At the completion of this script, the following message is displayed:

       Local node checking complete.
       Run root.sh on remaining nodes to start CRS daemons.
       
       

       
       Description of the illustration install030.gif
       
       

    6. After the root.sh script finishes on docrac1, go to the terminal window you opened in step c. As the root user on docrac2, enter the following commands:

       [root@docrac2 oraInventory]# cd /opt/oracle/crs
       [root@docrac2 crs]# ./root.sh
       
       

    After the root.sh script completes, return to the OUI window where the Installer prompted you to run the orainstRoot.sh and root.sh scripts. Click OK.

12. The Configuration Assistants window appears. When the configuration assistants finish, OUI displays the End of Installation window. Click Exit to complete the installation process.

    If you encounter any problems, refer to the configuration log for information. The path to the configuration log is displayed on the Configuration Assistants window.

Completing the Oracle Clusterware Configuration

After you have installed Oracle Clusterware, verify that the node applications are running. Depending on which operating system you use, you may need to perform some postinstallation tasks to configure the Oracle Clusterware components properly.

To complete the Oracle Clusterware configuration on Red Hat Linux:

1. As the oracle user on docrac1, check the status of the clusterware targets by entering the following command:

   /opt/oracle/crs/bin/./crs_stat -t
   
   

   This command provides output showing if all the important cluster services, such as gsd, ons, and vip, are running on the nodes of your cluster.

   
   Description of the illustration install030b.gif
   
   

2. If you are using Red Hat Linux 3.0, then, for each raw device used to store files for Oracle Clusterware, you must add two entries in the /etc/rc.d/rc.local file.

   The following table shows examples of the entries you must add for each file type, where oracle is the Oracle software owner, oinstall is the Oracle install group, dba is the privileged Oracle user group, /dev/raw/raw# is an individual device file, and /dev/name is a raw device name:

   File Type	Entries to Add
   OCR	chown root:oinstall /dev/raw/raw# chmod 640 /dev/raw/raw#
   Voting disk	chown oracle:oinstall /dev/raw/raw# chmod 640 /dev/raw/raw#
   ASM disk	chown oracle:dba /dev/name chmod 660 /dev/name

   
   

   Using the example raw partitions and devices listed in this guide, you would log in as root and insert the following at the end of the /etc/rc.d/rc.local file on both nodes docrac1 and docrac2, so that the permissions are set correctly when the nodes are restarted:

   chown root:oinstall /dev/raw/raw1 
   chown root:oinstall /dev/raw/raw2
   chown oracle:oinstall /dev/raw/raw3
   chown oracle:oinstall /dev/raw/raw4
   chown oracle:oinstall /dev/raw/raw5
   chmod 640 /dev/raw/raw1
   chmod 640 /dev/raw/raw2
   chmod 640 /dev/raw/raw3
   chmod 640 /dev/raw/raw4
   chmod 640 /dev/raw/raw5
    
   chown oracle:dba /dev/sdg
   chown oracle:dba /dev/sdh
   chown oracle:dba /dev/sdi
   chmod 660 /dev/sdg
   chmod 660 /dev/sdh
   chmod 660 /dev/sdi
   
   

   If you are using Red Hat Enterprise Linux 4.0, then ownership of the raw devices after restart was configured in the previous chapter using the udev utility in the section titled "Configuring the Raw Storage Devices and Partitions".

Verifying Your ASM Installation

Verify that all the database services for ASM are up and running. For example, on the docrac1 node, change directories to the bin directory in the Oracle Clusterware home directory, and then run the following command as the oracle user:

cd /opt/oracle/crs/bin
./srvctl status asm -n docrac1
ASM instance +ASM1 is running on node docrac1.

The example output shows that there is one ASM instance running on the local node. Repeat the preceding command, substituting docrac2 for docrac1 to verify the successful installation on the other node in your cluster.

Verifying your Oracle RAC Database Installation

At this point, you should verify all the database services are up and running. To do this, log in as oracle on the docrac1 node, and run the following commands:

[oracle] $ cd /opt/oracle/crs/bin
[oracle] $ ./crs_stat –t

The output of the command should show that database processes are available for each host.

Configuring the Operating System Environment for Database Management

After you have installed the Oracle RAC software and created a cluster database, there are two additional tasks to perform to configure your operating system environment for easier database management:

• Update the oratab File

• Reconfigure the User Shell Profile

# This file is used by ORACLE utilities. It is created by root.sh
# and updated by the Database Configuration Assistant when creating
# a database.

# A colon, ':', is used as the field terminator. A new line terminates
# the entry. Lines beginning with a pound sign, '#', are comments.
#
# Entries are of the form:
# $ORACLE_SID:$ORACLE_HOME:<N|Y>:
#
# The first and second fields are the system identifier and home
# directory of the database respectively. The third field indicates
# to the dbstart utility that the database should, "Y", or should not, 
# "N", be brought up at system boot time.
#
# Multiple entries with the same $ORACLE_SID are not allowed.
#
#
+ASM1:/opt/oracle/10gR2/asm:N
sales:/opt/oracle/10gR2/db_1:N
sales1:/opt/oracle/10gR2/db_1:N

Verifying the Clusterware Installation

After the Oracle Clusterware installation is complete, OUI automatically runs the cluvfy utility as a Configuration Assistant to verify that the Clusterware installation has been completed successfully.

If the CVU reports problems with your configuration, correct these errors before proceeding.

Backing Up the Voting Disk

After your Oracle Database 10g with Oracle RAC installation is complete and after you are sure that your system is functioning properly, make a backup of the contents of the voting disk. Use the dd utility, as described in the section "Backing Up and Recovering Voting Disks" in Chapter 5 of this guide.

Also, make a backup copy of the voting disk contents after you complete any node additions or deletions, and after running any deinstallation procedures.

Downloading and Installing RDBMS Patches

Periodically, Oracle issues bug fixes for its software called patches. Patch sets are a collection of bug fixes that were produced up to the time of the patch set release. Patch sets are fully tested product fixes. Application of a patch set affects the software residing in your Oracle home only, with no upgrade or change to the database.Ensure that you are running the latest patch set of the installed software. You might also need to apply patches that are not included in a patch set. Information about downloading and installing patches and patch sets is covered in Chapter 10, " Managing Oracle Software and Applying Patches".

Verifying Oracle Enterprise Manager Operations

When you install the Oracle RAC Database software and choose Database Control for your database management, the Enterprise Manager Database Control utility is installed and configured automatically.

To verify Oracle Enterprise Manager Database Control has been started in your new Oracle RAC environment:

1. Go to the $ORACLE_HOME/bin directory.

2. Run the following command as the oracle user:

   ./emctl status dbconsole
   
   

   The EMCTL utility displays the current status of the Database Control console on the current node.

3. If the EMCTL utility reports that Database Control is not started, use the following command to start it:

   ./emctl start dbconsole
   
   

4. Repeat steps 1 through 3 for each node in the cluster.

Recommended Postinstallation Tasks

Oracle recommends that you complete the following tasks after installing Oracle RAC:

• Backing Up the root.sh Script

• Configuring User Accounts

Checking the Prerequisites

Before you start the process of converting your database to a cluster database, you must meet certain prerequisites:

• The existing database and the target Oracle RAC database must be on the same release of Oracle Database 10g and must be running on the same platform.

• The hardware and operating system software used to implement your Oracle RAC database must be certified for use with the version of the Oracle RAC software you are installing.

• You must configure shared storage for your Oracle RAC database.

• You must verify that any applications that will run against the Oracle RAC database do not need any additional configuration before they can be used successfully with the cluster database. This applies to both Oracle applications and database features, such as Oracle Streams, and applications and products that do not come from Oracle.

Making a Preconfigured Copy of the Single-Instance Database

As part of the database conversion process, you can use DBCA to create a preconfigured image of your database.

To create a preconfigured image of your single-instance database using DBCA:

1. Go to the bin directory in $ORACLE_HOME, and start DBCA.

2. At the Welcome window, click Next.

3. On the Operations window, select Manage Templates, and click Next.

4. On the Template Management window, select Create a database template and From an existing database (structure as well as data), and click Next.

5. On the Source Database window, enter the database name in the Database instance field, and click Next.

6. On the Template Properties window, enter a name for your template in the Name field. Oracle recommends that you use the database name, for example, sales.

   By default, the template files are generated in the directory $ORACLE_HOME/assistants/dbca/templates. If you choose to do so, you can enter a description of the file in the Description field, and change the template file location in the Template datafile field.

   When you have finished entering the information, click Next.

7. On the Location of Database Related Files window, select Maintain the file locations, so that you can restore the database to the current directory structure, and click Finish.

DBCA generates two files: a database structure file (template_name.dbc), and a database preconfigured image file (template_name.dfb).

Performing the Preinstallation Steps

Follow the steps documented in Chapter 2 of this guide, titled "Preparing Your Cluster". You must do the following:

• Configure the servers to act as nodes in your cluster.

• Configure shared storage for the nodes in your cluster.

• Configure the interconnect and network connectivity between the nodes in your cluster.

Validating the Cluster

Validate the cluster configuration using the CVU, as described previously in this chapter in the section "Verifying the Configuration Using the Cluster Verification Utility".

Copying the Preconfigured Database Files

Copy the database structure *.dbc file and the database preconfigured image *.dfb files that DBCA created in the previous section titled "Making a Preconfigured Copy of the Single-Instance Database" to a temporary location on the node in the cluster from which you plan to run DBCA.

Installing the Oracle Database 10g Software with Real Application Clusters

After you have copied the preconfigured database files to the new node, install the Oracle RAC software on the new node. During the installation process, you will use the template files you created previously to convert your single-instance database to an Oracle RAC database.

To install the Oracle RAC software and convert your single-instance database to a cluster database:

1. Start OUI to perform an Oracle Database installation with Oracle RAC.

2. Select Cluster Installation Mode in the Specify Hardware Cluster Installation window of OUI, and select the nodes to include in your Oracle RAC database.

3. In the Database Configuration Types window, select the Advanced installation type.

   After installing the Oracle Database software, OUI runs postinstallation configuration tools, such as Network Configuration Assistant (NETCA), DBCA, and so on.

4. In the DBCA Template Selection window, use the template that you copied to a temporary location in the section "Copying the Preconfigured Database Files". Use the browse option to select the template location.

5. If you selected raw storage in the Storage Options window, then select the DBCA File Locations Tab on the Initialization Parameters window. Replace the datafiles, control files, and log files, and so on, with the corresponding raw device files. You must do this only if you have not set up the DBCA_RAW_CONFIG environment variable. You must also replace the default database files with raw devices in the Storage window.

6. After creating the Oracle RAC database, DBCA displays the Password Management window in which you must change the passwords for database-privileged users who have SYSDBA and SYSOPER roles.

When DBCA exits, the conversion process is complete.