10 Adding and Deleting Nodes and Instances on UNIX-Based Systems

Adding an Oracle Clusterware Home to a New Node

Use one of the procedures described in this section to use OUI to add an Oracle Clusterware home to a node that is going to be part of your RAC cluster. These procedures assume that you are adding node2 and that you have already successfully installed Oracle Clusterware on node1 in a non-shared home, where CRS_home represents the successfully installed home. This section describes the following procedures:

• Adding an Oracle Clusterware Home to a New Node Using OUI in Interactive Mode

• Adding an Oracle Clusterware Home to a New Node Using OUI in Silent Mode

  Note:

  When you extend a RAC database, you must first extend the Oracle Clusterware home to the new nodes and then extend the Oracle home. In other words, you extend the software onto the new nodes in the same order as you installed the clusterware and Oracle database software components on the existing nodes.

  The following sections are based on the addition of a node named node2 to an existing cluster that has a node named node1.

Adding an Oracle Home with RAC to a New Node

Use one of the following procedures to add an Oracle home to a new node using OUI. These procedures assume that you want to extend an existing Oracle home with RAC on node1 to node2. Node2 should already be a member node of the cluster to which node1 belongs. The contents of these procedures assume that you have successfully installed RAC on node1 in a non-shared home and that Oracle_home represents the successfully installed Oracle home. This section describes the following procedures:

• Adding an Oracle Home with RAC to a New Node Using OUI in Interactive Mode

• Adding an Oracle Home with RAC to a New Node Using OUI in Silent Mode

Deleting an Oracle Home with RAC from an Existing Node

Use the following procedures to delete nodes at the database Oracle home layer on Windows-based systems. These procedures assume that a RAC database home is installed on both node1 and node2 and that you want to delete the database home on node2. Also assume that Oracle_home represents the location of this home on the nodes. The procedures for deleting an Oracle home assume that you have successfully installed RAC on the node from which you want to delete the Oracle home. You can use either of the following procedures to delete an Oracle home from a node:

• Deleting an Oracle Home with RAC Using OUI in Interactive Mode

• Deleting an Oracle Home with RAC Using OUI in Silent Mode

  Note:

  When you remove Oracle Clusterware and RAC database software, you must first remove the Oracle database software and then remove Oracle Clusterware from the nodes that you are deleting. In other words, you remove the software components from the nodes that you are deleting in the reverse order that you originally installed the clusterware and Oracle database software components.

Deleting an Oracle Clusterware Home from an Existing Node

The procedures for deleting an Oracle Clusterware home assume that you have successfully installed the clusterware on the node from which you want to delete the Oracle Clusterware home. You can use either of the following procedures to delete an Oracle Clusterware home from a node:

• Deleting an Oracle Clusterware Home Using OUI in Interactive Mode

• Deleting an Oracle Clusterware Home Using OUI in Silent Mode

Making Physical Connections

Connect the new nodes' hardware to the network infrastructure of your cluster. This includes establishing electrical connections, configuring network interconnects, configuring shared disk subsystem connections, and so on. Refer to your hardware vendor documentation for details about this step.

Installing the Operating System

Install a cloned image of the operating system that matches the operating system on the other nodes in your cluster. This includes installing required service patches and drivers. Refer to your hardware vendor documentation for details about this process.

Creating Oracle Users

As root user, create the Oracle users and groups using the same user ID and group ID as on the existing nodes.

Verifying the Installation with the Cluster Verification Utility

Verify your installation using the Cluster Verification Utility (CVU) as in the following steps:

1. From the directory CRS_home/bin on the existing nodes, run the CVU command to verify your installation at the post hardware installation stage as shown in the following example, where node_list is a comma-delimited list of nodes you want in your cluster:

   cluvfy stage -post hwos -n node_list|all [-verbose]
   
   

   This command causes CVU to verify your hardware and operating system environment at the post-hardware setup stage. After you have configured the hardware and operating systems on the new nodes, you can use this command to verify node reachability, for example, to all of the nodes from the local node. You can also use this command to verify user equivalence to all given nodes the local node, node connectivity among all of the given nodes, accessibility to shared storage from all of the given nodes, and so on.

   Note:

   You can only use the all option with the -n argument if you have set the CV_NODELIST variable to represent the list of nodes on which you want to perform the CVU operation.

   See Also:

   "Using the Cluster Verification Utility" for more information about enabling and using the CVU

Checking the Installation

To verify that your installation is configured correctly, perform the following steps:

1. Ensure that the new nodes can access the private interconnect. This interconnect must be properly configured before you can complete the procedures in "Step 2: Extending Clusterware and Oracle Software to New Nodes".

2. If you are not using a cluster file system, then determine the location on which your cluster software was installed on the existing nodes. Make sure that you have at least 250MB of free space on the same location on each of the new nodes to install Oracle Clusterware. In addition, ensure you have enough free space on each new node to install the Oracle binaries.

3. Ensure that the Oracle Cluster Registry (OCR) and the voting disk are accessible by the new nodes using the same path as the other nodes use. In addition, the OCR and voting disk devices must have the same permissions as on the existing nodes.

4. Verify user equivalence to and from an existing node to the new nodes using rsh or ssh.

After completing the procedures in this section, your new nodes are connected to the cluster and configured with the required software to make them visible to the clusterware. Configure the new nodes as members of the cluster by extending the cluster software to the new nodes as described in "Step 2: Extending Clusterware and Oracle Software to New Nodes".

Adding Nodes at the Vendor Clusterware Layer

Add the new nodes at the clusterware layer according to the vendor clusterware documentation. For systems using shared storage for the Oracle Clusterware home, ensure that the existing clusterware is accessible by the new nodes. Also ensure that the new nodes can be brought online as part of the existing cluster. Proceed to the next section to add the nodes at the clusterware layer.

Adding Nodes at the Oracle Clusterware Layer

Before beginning this procedure, ensure that your existing nodes have the $CRS_HOME environment variable set correctly. The OUI requires access to the private interconnect that you verified as part of the installation validation in Step 1. If OUI cannot make the required connections, then you will not be able to complete the following steps to add nodes.

1. On one of the existing nodes go to the CRS_home/oui/bin directory and run the addNode.sh script to start OUI.

2. The OUI runs in add node mode and the OUI Welcome page appears. Click Next and the Specify Cluster Nodes for Node Addition page appears.

3. If you are using vendor clusterware, then the public node names automatically appear in the lower table. Click Next and OUI verifies connectivity on the existing nodes and on the new nodes. The verifications that OUI performs include determining whether:

   • The nodes are up

   • The nodes and private node names are accessible by way of the network

     Note:

     If any of the existing nodes are down, you can proceed with the procedure. However, once the nodes are up, you must run the following command on each of those nodes:

     runInstaller -updateNodeList -local 
     "CLUSTER_NODES={available_node_list}" 
     ORACLE_HOME=CRS_home
     

     This operation should be run from the CRS_home/oui/bin directory and the available_node_list values is comma-delimited list of all of nodes currently in the cluster and CRS_home defines the Oracle Clusterware home directory.

   • The virtual hostnames are not already in use on the network.

4. If any verifications fail, then OUI re-displays the Specify Cluster Nodes for Node Addition page with a Status column in both tables indicating errors. Correct the errors or deselect the nodes that have errors and proceed. However, you cannot deselect existing nodes; you must correct problems on nodes that are already part of your cluster before you can proceed with node addition. If all the checks succeed, then OUI displays the Node Addition Summary page.

   Note:

   Oracle strongly recommends that you install Oracle Clusterware on every node in the cluster on which you have installed vendor clusterware.

5. The Node Addition Summary page displays the following information showing the products that are installed in the Oracle Clusterware home that you are extending to the new nodes:

   • The source for the add node process, which in this case is the Oracle Clusterware home

   • The private node names that you entered for the new nodes

   • The new nodes that you entered

   • The required and available space on the new nodes

   • The installed products listing the products that are already installed on the existing Oracle Clusterware home

   Click Next and OUI displays the Cluster Node Addition Progress page.

6. The Cluster Node Addition Progress page shows the status of the cluster node addition process. The table on this page has two columns showing the four phases of the node addition process and the phases' statuses as follows:

   • Instantiate Root Scripts—Instantiates rootaddNode.sh with the public nodes, private node names, and virtual hostnames that you entered on the Cluster Node Addition page.

   • Copy the Oracle Clusterware home to the New Nodes—Copies the Oracle Clusterware home to the new nodes unless the Oracle Clusterware home is on a cluster file system.

   • Save Cluster Inventory—Updates the node list associated with the Oracle Clusterware home and its inventory.

   • Run rootaddNode.sh and root.sh—Displays a dialog prompting you to run the rootaddNode.sh script from the local node (the node on which you are running OUI) and to run the root.sh script on the new nodes. If OUI detects that the new nodes do not have an inventory location, then OUI instructs you to run orainstRoot.sh on those nodes. The central inventory location is the same as that of the local node. The addNodeActionstimestamp.log file, where timestamp shows the session start date and time, contains information about which scripts you need to run and on which nodes you need to run them.

   The Cluster Node Addition Progress page's Status column displays In Progress while the phase is in progress, Suspended when the phase is pending execution, and Succeeded after the phase completes. On completion, click Exit to end the OUI session. After OUI displays the End of Node Addition page, click Exit to end the OUI session.

7. Obtain the remote port number, which you will use in the next step, using the following command from the CRS_home/opmn/conf directory:

   cat ons.config
   
   

8. Run the racgons utility from the bin subdirectory of the Oracle Clusterware home to configure the Oracle Notification Services (ONS) port number. Use the following command, supplying the name of the node that you are adding for new_node_name and the remote port number obtained in the previous step:

   racgons add_config new_node_name:remote_port
   
   

9. Check that your cluster is integrated and that the cluster is not divided into partitions by completing the following operations:

   • Run the following CVU command to obtain detailed output for verifying cluster manager integrity on all of the nodes that are part of your RAC environment:

     cluvfy comp clumgr -n all [-verbose]
     
     

   • Use the following CVU command to obtain detailed output for verifying cluster integrity on all of the nodes that are part of your RAC environment:

     cluvfy comp clu [-verbose]
     
     

   • Use the following command to perform an integrated validation of the Oracle Clusterware setup on all of the configured nodes, both the pre-existing nodes and the nodes that you have added:

     cluvfy comp stage -post crinst -n all [-verbose]
     
     

   See Also:

   "Using the Cluster Verification Utility" for more information about enabling and using the CVU

You can optionally run addNode.sh in silent mode, replacing steps 1 through 7, as follows where nodeI, nodeI+1, and so on are the new nodes that you are adding:

addNode.sh -silent "CLUSTER_NEW_NODES={nodeI, nodeI+1, … nodeI+n}" 
"CLUSTER_NEW_PRIVATE_NODE_NAMES={pnI, pnI+1, … pnI+n}" 
"CLUSTER_NEW_VIRTUAL_HOSTNAMES={vipI, vipI+1,…,vipI+n}"

You can alternatively specify the variable=value entries in a response file and run the addNode script as follows:

addNode.sh -silent -responseFile filename OR addNode.bat -silent -responseFile filename

Run rootaddNode.sh on the local node, or the node on which you are performing this procedure, and run root.sh on the new nodes. If OUI detects that the new nodes do not have an inventory location, then OUI instructs you to run orainstRoot.sh on those nodes. The central inventory location is the same as that of the local node. The addNodeActionstimestamp.log file, where timestamp shows the session start date and time, contains the information about which scripts you need to run and on which nodes you need to run them.

After you have completed the procedures in this section for adding nodes at the Oracle Clusterware layer, you have successfully extended the Oracle Clusterware home from your existing the Oracle Clusterware home to the new nodes. Proceed to "Step 3: Preparing Storage on New Nodes" to prepare storage for RAC on the new nodes.

Raw Device Storage Preparation for New Nodes

To prepare raw device storage on the new nodes, you need at least two new disk partitions to accommodate the redo logs for each new instance. Make these disk partitions the same size as the redo log partitions that you configured for the existing nodes' instances. Also create an additional logical partition for the undo tablespace for automatic undo management.

On applicable operating systems, you can create symbolic links to your raw devices. Optionally, you can create a raw device mapping file and set the DBCA_RAW_CONFIG environment variable so that it points to the raw device mapping file. Use your vendor-supplied tools to configure the required raw storage.

Run the following command to verify that the prepared raw device storage is accessible from all of the configured cluster nodes where node_list includes both the pre-existing nodes and the newly added nodes and storageID_list is a comma-delimited list of storage identifiers:

cluvfy comp ssa [ -n node_list ] [ -s storageID_list ] [-verbose]

After completing the procedures in this section, you have configured your shared storage for use by the instances to be added on the new nodes so that the new nodes can access the Oracle software. Additionally, the existing nodes can access the new nodes and instances. Use OUI as described in the procedures in "Step 4: Adding Nodes at the Oracle RAC Database Layer" to configure the new nodes at the RAC database layer.

Using Enterprise Manager to Add Database Instances to New Nodes

To add a database instance to a new node with Enterprise Manager, perform the following procedure:

1. From the Cluster Database Home page, click the Maintenance tab.

2. Under the Deployments section, click Add Instance. This initiates a wizard to guide you through the instance addition process.

3. Perform the following tasks for each wizard step:

   • If the database uses ASM, then provide host and ASM credentials.

   • Specify the hosts for which you want to add instances.

   • Review and submit the job.

   After you submit the job, the wizard provides a summary page that shows whether the job succeeded and the wizard also provides other detailed information about the job, such as the elapsed time and an output log.

After adding the instances to the new nodes using the steps described in this section, perform any needed service configuration procedures as described in Chapter 6, "Introduction to Workload Management".

Using DBCA in Interactive Mode to Add Database Instances to New Nodes

To add a database instance to a new node with DBCA in interactive mode, perform the following procedure:

1. Start the Database Configuration Assistant (DBCA) by entering dbca at the system prompt from the bin directory in the Oracle_home directory.

   The DBCA displays the Welcome page for RAC. Click Help on any DBCA page for additional information.

2. Select Real Application Clusters database, click Next, and DBCA displays the Operations page.

3. Select Instance Management, click Next, and DBCA displays the Instance Management page.

4. Select Add Instance and click Next. The DBCA displays the List of Cluster Databases page that shows the databases and their current status, such as ACTIVE, or INACTIVE.

5. From the List of Cluster Databases page, select the active RAC database to which you want to add an instance. Enter user name and password for the database user that has SYSDBA privileges. Click Next and DBCA displays the List of Cluster Database Instances page showing the names of the existing instances for the RAC database that you selected.

6. Click Next to add a new instance and DBCA displays the Adding an Instance page.

7. On the Adding an Instance page, enter the instance name in the field at the top of this page if the instance name that DBCA provides does not match your existing instance naming scheme. Then select the new node name from the list, click Next, and DBCA displays the Services Page.

8. Enter the services information for the new node's instance, click Next, and DBCA displays the Instance Storage page.

9. If you are using raw devices or raw partitions, then on the Instance Storage page select the Tablespaces folder and expand it. Select the undo tablespace storage object and a dialog appears on the right-hand side. Change the default datafile name to the raw device name for the tablespace.

10. If you are using raw devices or raw partitions or if you want to change the default redo log group file name, then on the Instance Storage page select and expand the Redo Log Groups folder. For each redo log group number that you select, DBCA displays another dialog box. Enter the raw device name that you created in the section "Raw Device Storage Preparation for New Nodes" in the File Name field.

11. If you are using a cluster file system, then click Finish on the Instance Storage page. If you are using raw devices, then repeat step 10 for all of the other redo log groups, click Finish, and DBCA displays a Summary dialog.

12. Review the information on the Summary dialog and click OK or click Cancel to end the instance addition operation. The DBCA displays a progress dialog showing DBCA performing the instance addition operation. When DBCA completes the instance addition operation, DBCA displays a dialog asking whether you want to perform another operation.

13. After you terminate your DBCA session, run the following command to verify the administrative privileges on the new node and obtain detailed information about these privileges where nodelist consists of the newly added nodes:

    cluvfy comp admprv -o db_config -d oracle_home -n nodelist [-verbose]
    
    

After adding the instances to the new nodes using the steps described in this section, perform any needed service configuration procedures as described in Chapter 6, "Introduction to Workload Management".

Using DBCA in Silent Mode to Add Database Instances to New Nodes

You can use the Database Configuration Assistant (DBCA) in silent mode to add instances to nodes onto which you have extended an Oracle Clusterware home and an Oracle home. Use the following syntax to perform this operation where node is the node onto which you want to add the instance, gdbname is the global database name, instname is the name of the new instance, sysdba is the name of an Oracle user with SYSDBA privileges, and password is the password for the user name in sysdba:

dbca -silent -addInstance -nodeList node -gdbName gdbname [-instanceName instname] -sysDBAUserName sysdba -sysDBAPassword password

Note that you only need to provide an instance name if you want to override the Oracle naming convention for RAC instance names.

After you have completed either of the DBCA procedures in this section, DBCA has successfully added the new instance to the new node and completed the following steps:

• Created and started an ASM instance on each new node if the existing instances were using ASM

• Created a new database instance on each new node

• Created and configured high availability components

• Created the Oracle Net configuration

• Started the new instance

• Created and started services if you entered services information on the Services Configuration page

After adding the instances to the new nodes using the steps described in this section, perform any needed service configuration procedures as described in Chapter 6, "Introduction to Workload Management".

Using Enterprise Manager to Delete Database Instances from Existing Nodes

To delete an instance with Enterprise Manager from an existing node, perform the following steps:

1. From the Cluster Database Home page, click the Maintenance tab.

2. Under the Deployments section, click Delete Instance. This action initiates a wizard to guide you through the deletion process.

Using DBCA in Interactive Mode to Delete Database Instances from Existing Nodes

To delete an instance using DBCA in interactive mode, perform the following steps:

1. Start DBCA on a node other than the node that hosts the instance that you want to delete. On the DBCA Welcome page select Oracle Real Application Clusters Database, click Next, and DBCA displays the Operations page.

2. On the DBCA Operations page, select Instance Management, click Next, and DBCA displays the Instance Management page.

3. On the Instance Management page, Select Delete Instance, click Next, and DBCA displays the List of Cluster Databases page.

4. Select a RAC database from which to delete an instance. Enter a user name and password for the database user that has SYSDBA privileges. Click Next and DBCA displays the List of Cluster Database Instances page. The List of Cluster Database Instances page shows the instances that are associated with the RAC database that you selected and the status of each instance.

5. Select an instance to delete and click Finish.

6. If you have services assigned to this instance, then the DBCA Services Management page appears. Use this feature to reassign services from this instance to other instances in the cluster database.

7. Review the information about the instance deletion operation on the Summary page and click OK. Otherwise, click Cancel to cancel the instance deletion operation. If you click OK, then DBCA displays a Confirmation dialog.

8. Click OK on the Confirmation dialog to proceed with the instance deletion operation and DBCA displays a progress dialog showing that DBCA is performing the instance deletion operation. During this operation, DBCA removes the instance and the instance's Oracle Net configuration. When DBCA completes this operation, DBCA displays a dialog asking whether you want to perform another operation.

9. Click No and exit DBCA or click Yes to perform another operation. If you click Yes, then DBCA displays the Operations page.

Using DBCA in Silent Mode to Delete Instance from Existing Nodes

Use DBCA to delete a database instance from a node as follows, where the variables are the same as those in the preceding add instance command:

dbca -silent -deleteInstance [-nodeList node] -gdbName gdbname -instanceName instname -sysDBAUserName sysdba -sysDBAPassword password

You only need to provide a node name if you are deleting an instance from a node other than the one on which you are running DBCA.

At this point, you have accomplished the following:

• De-registered the selected instance from its associated Oracle Net Services listeners

• Deleted the selected database instance from the instance's configured node

• Removed the Oracle Net configuration

• Deleted the Oracle Flexible Architecture directory structure from the instance's configured node.