Oracle® Enterprise Manager Advanced Configuration 10g Release 3 (10.2.0.3.0) Part Number B40002-02 |
|
|
View PDF |
The notification system allows you to notify Enterprise Manager administrators of alerts, policy violations, and the status changes of job executions. In addition to notifying administrators, the notification system can perform actions such as executing operating system commands (including scripts) and PL/SQL procedures when an alert is triggered. This capability allows you to implement automatically specific IT practices under particular alert conditions. For example, if an alert is generated when monitoring the operational (up/down) status of a database, you may want the notification system to automatically open an in-house trouble-ticket using an OS script so that the appropriate IT staff can respond in a timely manner.
By using Simple Network Management Protocol (SNMP) traps, the Enterprise Manager notification system also allows you to send traps to SNMP-enabled third-party applications such as HP OpenView. Some administrators may want to send third-party applications a notification when a certain metric has exceeded a threshold.
This chapter covers the following:
All Enterprise Manager administrators can set up e-mail notifications for themselves. Super Administrators also have the ability to set up notifications for other Enterprise Manager administrators.
Before Enterprise Manager can send e-mail notifications, you must first specify the Outgoing Mail (SMTP) servers to be used by the notification system. Once set, you can then define e-mail notifications for yourself or, if you have Super Administrator privileges, other Enterprise Manager administrators.
You specify the Outgoing Mail (SMTP) server on the Notification Methods page (Figure 13-1). Display the Notification Methods page by clicking Setup on any page in the Grid Control Console and clicking Notification Methods in the vertical navigation bar.
Note:
You must have Super Administrator privileges in order to set up SMTP servers.Specify one or more outgoing mail server names, the mail server authentication credentials (User Name, Password, and Confirm Password), if required, the name you want to appear as the sender of the notification messages, and the e-mail address you want to use to send your e-mail notifications. This address, called the Sender's Mail Address, must be a valid address on each mail server that you specify. A message will be sent to this e-mail address if any problem is encountered during the sending of an e-mail notification. Example 13-1 shows sample notification method entries.
Example 13-1 Mail Server Settings
Outgoing Mail (SMTP) Server - smtp01.mycorp.com:587, smtp02.mycorp.com
User Name - myadmin
Password - ******
Confirm Password - ******
Identify Sender As - Enterprise Manager
Sender's E-mail Address - mgmt_rep@mycorp.com
Note:
The e-mail address you specify on this page is not the e-mail address to which the notification is sent. You will have to specify the e-mail address (where notifications will be sent) from the Preferences General page.After configuring the e-mail server, click Test Mail Servers to verify your e-mail setup. You should verify that an e-mail message was received by the e-mail account specified in the Sender's E-mail Address field.
Defining multiple mail servers will improve the reliability of e-mail notification delivery and spread the load across multiple systems. The Management Service makes use of each mail server to send e-mails and the behavior is controlled by the following parameters found in the $ORACLE_HOME/sysman/config/emoms.properties
file.
Example 13-2 Management Service Parameters
# The maximum number of emails that can be sent in a single connection to an # email server # em.notification.emails_per_connection=20 # # The maximum number of emails that can be sent in a minute # em.notification.emails_per_minute=250
Based on the defaults in Example 13-2, the first mail server is used to send 20 e-mails before the Management Service switches to the next mail server which is used to send another 20 e-mails before switching to the next mail server. This prevents one mail server from becoming overloaded and should improve overall reliability and throughput.
If you want to receive notifications by e-mail, you will need to specify your e-mail address(s) in the General page under the Preferences link in the Grid Control console. In addition to defining notification e-mail addresses, you associate the notification message format (long or short) to be used for your e-mail address.
Setting up e-mail involves three steps:
Step 1: Define e-mail addresses.
Step 2: Set up a Notification Schedule.
Step 3: Subscribe to receive e-mail for notification rules.
An e-mail address can have up to 128 characters. There is no upper limit with the number of e-mail addresses.
To add an e-mail address:
From the Grid Control Console, click Preferences. By default the General page is selected.
Click Add Another Row to create a new e-mail entry field in the E-mail Addresses table.
Specify the e-mail associated with your Enterprise Manager account. All e-mail notifications you receive from Enterprise Manager will be sent to the e-mail addresses you specify.
For example, user1@oracle.com
Select the message format for your e-mail address. The Long Format sends a HTML formatted e-mail that contains detailed information. Example 13-3 shows a typical notification that uses the long format.
The Short Format (Example 13-4) sends a concise, text e-mail that is limited to a configurable number of characters, thereby allowing the e-mail be received as an SMS message or page. The content of the message can be sent entirely in the subject, entirely in the body or split across the subject and body. For example, in the last case, the subject could contain the severity type (for example, Critical) and the target name. The body could contain the time the severity occurred and the severity message. Since the message length is limited, some of this information may be truncated. If truncation has occurred there will be an ellipsis end of the message.
Click Apply to save your e-mail address.
Example 13-3 Long E-mail Notification for Alerts
Name=myhost.com Type=Host Host=myhost.com Metric=Filesystem Space Available (%) Mount Point =/usr Timestamp=06-OCT-2006 16:27:05 US/Pacific Severity=Warning Message=Filesystem / has only 76.07% available space Rule Name=Host Availability and Critical States Rule Owner=SYSMAN
Example 13-4 Short E-mail Notification for Alerts
Subject is : EM:Unreachable Start:myhost Body is : Nov 16, 2006 2:02:19 PM EST:Agent is Unreachable (REASON = Connection refused) but the host is UP
More about Short E-mail Format
Enterprise Manager does not directly support paging, SMS, etc., but instead relies on external gateways to, for example, perform the conversion from e-mail to page.
Entries in the emoms.properties file define the size and format of the short e-mail.
You must establish the maximum size your device can support and whether the message is sent in subject, body or both
emoms.properties Entries for a Short E-mail Format
# The maximum size of a short format email # em.notification.short_format_length=155 # The format of the short email. It can be set to subject, body or both. # # When set to subject the entire message is sent in the subject i.e. # EM:<severity>:<target>:<message>:<timestamp> # When set to body the entire message is sent in the body i.e. # EM:<severity>:<target>:<message>:<timestamp> # When set to both the message is split i.e. the subject contains # EM:<severity>:<target> # and the body contains # <message>:<timestamp> # In all cases the message is truncated to the length specified in the # em.notification.short_format_length parameter # em.notification.short_format=both
Once you have defined your e-mail notification addresses, you will need to define a notification schedule. For example, if your e-mail addresses are user1@oracle.com, user2@oracle.com, user3@oracle.com, you can choose to use one or more of these e-mail addresses for each time period in your notification schedule.
Note:
When you enter e-mail addresses for the first time, a 24x7 weekly notification schedule is set automatically. You can then review and modify the schedule to suit your monitoring needs.A notification schedule is a repeating schedule used to specify your on-call schedule—the days and time periods and e-mail addresses that should be used by Enterprise Manager to send notifications to you. Each administrator has exactly one notification schedule. When a notification needs to be sent to an administrator, Enterprise Manager consults that administrator's notification schedule to determine the e-mail address to be used. Depending on whether you are Super Administrator or a regular Enterprise Manager administrator, the process of defining a notification schedule differs slightly.
If you are a regular Enterprise Manager administrator and are defining your own notification schedule:
From the Enterprise Manager Grid Control, click Preferences at the top of the page. By default the General page is selected.
Click Notification Schedule in the vertical navigation bar. Your Notification Schedule page appears.
Follow the directions on the Notification Schedule page to specify when you want to receive e-mails.
A notification rule is a user-defined rule that defines the criteria by which notifications should be sent for alerts, policy violations, corrective action execution status, and job execution status. Specifically, in each rule, you can specify the criteria you are interested in and the notification methods (such as e-mail) that should be used for sending these notifications. For example, you can set up a rule that when any database goes down or any database backup job fails, e-mail should be sent and the "log trouble ticket" notification method should be called. Or you can define another rule such that when the CPU or Memory Utilization of any host reach critical severities, SNMP traps should be sent to another management console. During notification rule creation, you specify criteria such as the targets you are interested in, their monitored metrics, associated alert severity conditions (clear, warning, critical), policy violations, corrective action execution status, or job execution status, and the associated notification method.
To subscribe to a notification rule you create, while creating the rule, go to the Methods page and check the "Send Me E-mail" option.
Out-of-Box Notification Rules
Enterprise Manager Grid control comes with out-of-box notification rules that cover the most common alert conditions. When you install the Oracle Management Service, you are given the option to receive e-mail notifications for critical alerts. If you choose this option, and if an e-mail address for the SYSMAN user was specified, then some default notification rules are created that cover the availability and critical states for common target types and would also be configured to send e-mail notifications to the SYSMAN e-mail address for the conditions defined in the notification rules.
You can access the out-of-box notification rules by clicking on Preferences on any page in the Enterprise Manager console and clicking Public Rules in the vertical navigation bar. If the conditions defined in the out-of-box notification rules meet your requirements, you can simply subscribe to receive e-mail notifications for the conditions defined in the rule by clicking on Subscribe column in the row of the Public Rules table that corresponds to the notification rule that you are interested in. Click Apply to save your changes.
Table 13-1 lists all the default notification rules. These are all owned by the SYSMAN user and are public rules.
Table 13-1 Default Notification Rules
Name | Description | Applies to Targets of the Type | Send Notification on the Following Availability States | Send Notification if Any of the Metrics is at CRITICAL Alert Severity |
---|---|---|---|---|
System-generated notification rule for monitoring Agents that may have problems uploading data to the Management Service. |
Oracle Management Service and Repository |
N/A |
Count of targets not uploading data |
|
System-generated notification rule for monitoring Agents that lose contact with the Management Service due to network problems, host problems or Agents going down. |
Agents |
Agent UnreachableAgent Unreachable Resolved |
N/A |
|
System-generated notification rule for monitoring Application Servers' availability, and critical metric statuses. |
Application Servers |
Down |
CPU Usage (%) |
|
System-generated notification rule for monitoring Databases' availability, and critical metric statuses. |
Databases (single instance only) |
Down |
Process Limit Usage (%) Session Limit Usage (%) Blocking Session Count All Objects Archiver Hung Alert Log Error Status Data Block Corruption Alert Log Error Status Generic Alert Log Error Status Media Failure Alert Log Error Status Session Terminated Alert Log Error Status Archive Area Used (%) All Objects Segments Not Able to Extend Count All Objects Segments Approaching Maximum Extents Count All Objects Tablespace Space Used (%) All Objects Wait Time (%) |
|
System-generated notification rule for monitoring HTTP Server's availability, and critical metric statuses. |
Oracle HTTP Server |
Down |
CPU Usage (%) Percentage of Busy Processes Active HTTP Connections Request Processing Time (seconds) |
|
System-generated notification rule for monitoring Hosts' availability, and critical metric statuses. |
Hosts |
Agent Unreachable Agent Unreachable Resolved |
Average Disk I/O Service Time (ms) Disk Device Busy (%) Filesystem Space Available (%) CPU in I/O Wait (%) Run Queue Length (5 minute average) CPU Utilization (%) Memory Utilization (%) Memory Page Scan Rate (per second) Swap Utilization (%) Network Interface Combined Utilization (%) |
|
System-generated notification rule for monitoring database Listeners' availability, and critical metric statuses. |
Listeners |
Down |
N/A |
|
System-generated notification rule for monitoring OC4J instance's availability, and critical metric statuses. |
OC4J |
Down |
CPU Usage (%) OC4J Instance - Request Processing Time (seconds) OC4J Instance - Active Sessions |
|
System-generated notification rule for monitoring the availability of the DBMS jobs that are part of the Management Repository. |
Oracle Management Service and Repository |
Critical |
DBMS Job UpDown |
|
Violation Notification for Database Security Policies |
System-generated notification rule for monitoring the secureness of the database configuration. |
Databases |
Critical |
N/A |
System-generated notification rule for monitoring Web Cache's instance's availability, and critical metric statuses. |
Oracle Web Cache |
Down |
Hits (% of requests) Web Cache CPU Usage (%) |
Creating Your Own Notification Rules
If you find that the default notification rules do not meet your needs, you can define your own custom rules. The following procedure documents the process of notification rule creation for non-Super Administrators.
To create your own notification rule:
From the Enterprise Manager Grid Control, click Preferences.
Click My Rules in the vertical navigation bar.
If you are not logged in as an administrator with Super Administrator privileges, you will see a link for My Rules instead of Rules as in the case of an administrator with Super Administrator privileges.
Click Create.
Enterprise Manager displays the Create Notification Rule pages. Enter the requisite information on each page to create your notification rule.
When you specify the notification rule properties, check Make Public in the General page if you want other non-privileged users to be able to view and share that rule. For example, it allows other administrators to later specify that they should receive e-mail for this rule.
When you specify the notification rule, you will only be able to choose from e-mail and SNMP traps. Specifying custom commands and PL/SQL procedures is an option that is only available to Super Administrators. To receive e-mail notifications for conditions defined in the rule, go to the Methods page and select the Send Me E-Mail option.
eIf you have Super Administrator privileges, you can set up e-mail notifications for other Enterprise Manager administrators. To set up e-mail notifications for other Enterprise Manager administrators, you must need to:
Step 1: Ensure Each Administrator Account has an Associated E-mail Address
Each administrator to which you want to send e-mail notifications must have a valid e-mail address.
Click Setup.
Click Administrators from the vertical navigation bar.
For each administrator, define an e-mail address. This sets up a 24x7 notification schedule for this user that uses all the e-mail addresses specified.
Enterprise Manager also allows you to specify an administrator address when editing an administrator's notification schedule.
Step 2: Define Administrators' Notification Schedules
Once you have defined e-mail notification addresses for each administrator, you will need to define their respective notification schedules. Although a default 24x7 notification schedule is created when you specified the e-mail addresses for the first time, you should review and edit the notification schedule as needed.
Click Setup.
From the vertical navigation bar, click Schedules (under Notification). The Notification Schedule page appears.
Specify the administrator who's notification schedule you wish to edit and click Change.
Click Edit Schedule Definition. The Edit Schedule Definition: Time Period page appears. If necessary, modify the rotation schedule.
Click Continue. The Edit Schedule Definition: E-mail Addresses page appears.
Follow the directions on the Edit Schedule Definition: E-mail Addresses page to modify the notification schedule.
Click Finish when you are done.
Repeat steps three through seven for each administrator.
Step 3: Assign Notification Rules to Administrators
With the notification schedules set, you now need to assign the appropriate notification rules for each designated administrator.
Click Setup.
From the vertical navigation bar, click Administrators.
Select the desire administrator.
Click Subscribe to Rules. The Subscribe admin to Public Notification Rules page appears.
Select the desired notification rules and click Subscribe.
Click OK when you are finished.
Repeat steps three through six for each administrator.
Notification Methods are the mechanisms by which alerts are sent. Enterprise Manager Super Administrators can set up e-mail notifications by configuring the 'e-mail' notification method. Most likely this would already have been setup as part of the Oracle Management Service installation.Enterprise Manager Super Administrators can also define other custom notification methods. For example, alerts may need to be forwarded to a 3rd party trouble-ticketing system. Assuming APIs to the third-party trouble-ticketing system are available, a custom notification method can be created to call a custom OS script that has the appropriate APIs. The custom notification method can be named in a user-friendly fashion, for example, "Log trouble ticket". Once that is defined, any time an administrator needs to send alerts to the trouble-ticketing system, he merely needs to invoke the now globally available notification method called "Log trouble ticket". Custom notification methods can be defined based on any custom OS script, any custom PL/SQL procedure, or by sending SNMP traps.
Only Super Administrators can define OS Command, PL/SQL, and SNMP Trap notification methods. However, any Enterprise Manager administrator can add these notification methods (once defined by the Super Administrator) to their notification rules.
Through the Notification Methods page, you can:
Set up the outgoing mail servers if you plan to send e-mail notifications through notification rules
Create other custom notification methods using OS and PL/SQL scripts and SNMP traps.
You can create other custom notification methods based on OS scripts, PL/SQL procedures, or SNMP traps. Any administrator can then use these methods in Notification Rules.
Complete the following four steps to define a notification method based on an OS command or script.
Note:
Notification methods based on OS commands must be configured by an administrator with Super Administrator privileges.Step 1: Define your OS command or script.
You can specify an OS command or script that will be called by the notification system. You can use target and alert or policy violation context information, corrective action execution status and job execution status within the body of the script. Passing metric severity attributes (severity level, type, notification rule, rule owner, or rule owner, and so on) or policy violation information to OS commands/scripts allows you to customize automated responses to alerts or policy violations. For example, if an OS script opens a trouble ticket for an in-house support trouble ticket system, you will want to pass severity levels (critical, warning, and so on) to the script to open a trouble ticket with the appropriate details and escalate the problem. For more information on passing specific types of information to OS Command or Scripts, see
"Passing Alert and Policy Violation Information to an OS Command or Script"
"Passing Corrective Action Execution Status to an OS Command or Script"
Step 2: Deploy the script on each Management Service host.
You must deploy the OS Command or Script on each Management Service host machine that connects to the Management Repository. The OS Command is run as the user who started the Management Service.
The OS Command or Script should be deployed on the same location on each Management Service host machine. The OS Command should be an absolute path, for example, /u1/bin/logSeverity.sh. The command is run by the user who started the Management Service. If an error is encountered during the running of the OS Command, the Notification System can be instructed to retry the sending of the notification to the OS Command by returning an exit code of 100. The procedure is initially retried after one minute, then two minutes, then three minutes and so on, until the notification is a day old, at which point it will be purged.
Example 13-5 shows the parameter in emoms.properties that controls how long the OS Command can execute without being killed by the Management Service. This is to prevent OS Commands from running for an inordinate length of time and blocking the delivery of other notifications. By default the command is allowed to run for 30 seconds before it is killed.
Example 13-5 Parameter in emoms.properties File
# The amount of time in seconds after which an OS Command started by the # Notification System will be killed if it has not exited # em.notification.os_cmd_timeout=30
Step 3: Register your OS Command or Script as a new Notification Method.
Add this OS command as a notification method that can be called in Notification Rules. Log in as a Super Administrator, click Setup and then Notification Methods from the vertical navigation bar. From this page, you can define a new notification based on the 'OS Command' type. See "Adding a Notification Method based on an OS Command or Script" .
The following information is required for each OS command notification method:
Name
Description
Both Name and Description should be clear and intuitive so that the function of the method is clear to other administrators.
OS Command
You must enter the full path of the OS command or script in the OS command field (for example, /u1/bin/myscript.sh
). For environments with multiple Management Services, the path must be exactly the same on each machine that has a Management Service. Command line parameters can be included after the full path (for example, /u1/bin/myscript.sh arg1 arg2).
Example 13-6 shows information required for the notification method.
Example 13-6 OS Command Notification Method
Name Trouble Ticketing Description Notification method to log trouble ticket for a severity occurrence OS Command /private/mozart/bin/logTicket.sh
Note:
There can be more than one OS Command configured per system.Step 4: Assign the notification method to a rule.
You can edit an existing rule (or create a new notification rule), then go to the Methods page. In the list of Advanced Notification Methods, select your notification method and click 'Assign Method to Rule'. To assign multiple rules to a method or methods to a single rule, see "Assigning Rules to Methods" or "Assigning Methods to Rules".
Passing Alert and Policy Violation Information to an OS Command or Script
The notification system passes severity information to an OS script or executable using system environment variables.
Conventions used to access environmental variables vary depending on the operating system:
UNIX: $ENV_VARIABLE
Windows:%ENV_VARIABLE%
The notification system sets the following environment variables before calling the script. The script can then use any or all of these variables within the logic of the script.
Table 13-2 Environment Variables
Environment Variable | Description |
---|---|
TARGET_NAME |
Name of the target on which the severity occurred. |
TARGET_TYPE |
Type of target on which the severity occurred. Targets are defined as any monitorable entity, such as Host, Database, Listener, or Oracle HTTP Server. You can view the type of a monitored target on the All Targets page. |
HOST |
Name of the machine on which the target resides. |
METRIC |
Metric generating the severity. This variable is not set for policy violations. |
METRIC_VALUE |
The value of the metric when the threshold was exceeded. Not set for policy violations |
POLICY_RULE |
The name of the policy when the threshold was exceeded. Not set for metric severities |
KEY_VALUE |
For metrics that monitor a set of objects, the KEY_VALUE indicates the specific object that triggered the severity. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE is 'USERS' if the USERS tablespace triggered at warning or critical severity. |
KEY_VALUE_NAME |
For metrics that monitor a set of objects, the KEY_VALUE_NAME indicates the type of object monitored. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE_NAME is 'Tablespace Name'. |
VIOLATION_CONTEXT |
A comma-separated list of name-value pairs that shows the alert context for a policy violation. |
TIMESTAMP |
Time when the severity occurred. |
SEVERITY |
Type of severity. For example, severity for a target's (availability) status metric are:
Other metrics can have any of the following severities:
|
MESSAGE |
Message for the alert that provides details about what triggered the condition. |
RULE_NAME |
Name of the notification rule to which the OS Command notification method was assigned. |
RULE_OWNER |
Name of the Enterprise Manager administrator who owns the notification rule. |
Your script may reference some or all of these variables.
The sample OS script shown in Example 13-7 appends environment variable entries to a log file. In this example, the script logs a severity occurrence to a file server. If the file server is unreachable then an exit code of 100 is returned to force the Oracle Management Service Notification System to retry the notification
Example 13-7 Sample OS Command Script
#!/bin/ksh LOG_FILE=/net/myhost/logs/severity.log if test -f $LOG_FILE then echo $TARGET_NAME $MESSAGE $TIMESTAMP >> $LOG_FILE else exit 100 fi
Example 13-8 shows an OS script that logs alert information to the file 'alertmsg.txt'. The file is saved to the /u1/results directory.
Example 13-8 Alert Logging Script
#!/usr/bin/sh echo "Alert logged:" > /u1/results/alertmsg.txt echo "\n" >> /u1/results/alertmsg.txt echo "target name is " $TARGET_NAME >> /u1/results/alertmsg.txt echo "target type is " $TARGET_TYPE >> /u1/results/alertmsg.txt echo "target is on host " $HOST >> /u1/results/alertmsg.txt echo "metric in alert is " $METRIC >> /u1/results/alertmsg.txt echo "metric index is " $KEY_VALUE >> /u1/results/alertmsg.txt echo "timestamp is " $TIMESTAMP >> /u1/results/alertmsg.txt echo "severity is " $SEVERITY >> /u1/results/alertmsg.txt echo "message is " $MESSAGE >> /u1/results/alertmsg.txt echo "notification rule is " $RULE_NAME >> /u1/results/alertmsg.txt echo "rule owner is " $RULE_OWNER >> /u1/results/alertmsg.txt exit 0
Example 13-9 shows a script that sends an alert to an HP OpenView console from Enterprise Manager Grid Control. When a metric alert is triggered, the Enterprise Manager Grid Control displays the alert. The HP OpenView script is then called, invoking opcmsg and forwarding the information to the HP OpenView management server.
Complete the following four steps to define a notification method based on a PL/SQL procedure.
Step 1: Define the PL/SQL procedure.
The procedure must have one of the following signatures depending on the type of notification that will be received.
For alerts and policy violations:
PROCEDURE p(severity IN MGMT_NOTIFY_SEVERITY)
For job execution status changes:
PROCEDURE p(job_status_change IN MGMT_NOTIFY_JOB)
For corrective action status changes:
PROCEDURE p(ca_status_change IN MGMT_NOTIFY_CORRECTIVE_ACTION)
Note:
The notification method based on a PL/SQL procedure must be configured by an administrator with Super Administrator privileges before a user can select it while creating/editing a notification rule.For more information on passing specific types of information to scripts or PL/SQL procedures, see the following sections:
"Passing Alert and Policy Violation Information to a PL/SQL Procedure"
"Passing Corrective Action Status Change Information"
"Passing Job Execution Status Information"
Step 2: Create the PL/SQL procedure on the Management Repository.
Create the PL/SQL procedure on the repository database using one of the following procedure specification:
PROCEDURE p(severity IN MGMT_NOTIFY_SEVERITY) PROCEDURE p(job_status_change IN MGMT_NOTIFY_JOB) PROCEDURE p(ca_status_change IN MGMT_NOTIFY_CORRECTIVE_ACTION)
The PL/SQL procedure must be created on the repository database using the database account of the repository owner (such as SYSMAN)
If an error is encountered during the running of the procedure, the Notification System can be instructed to retry the sending of the notification to the procedure by raising a user defined exception that uses the error code -20000. See Example 13-11, "PL/SQL Procedure Using a Severity Code". The procedure initially retried after one minute, then two minutes, then three minutes and so on, until the notification is a day old, at which point it will be purged.
Step 3: Register your PL/SQL procedure as a new notification method.
Log in as a Super Administrator, click Setup and then Notification Methods from the vertical navigation bar. From this page, you can define a new notification based on 'PL/SQL Procedure'. See "Adding a Notification Method Based on a PL/SQL Procedure".
Make sure to use a fully qualified name that includes the schema owner, package name and procedure name. The procedure will be executed by the repository owner and so the repository owner must have execute permission on the procedure.
Create a notification method based on your PL/SQL procedure. The following information is required when defining the method:
Name
Description
PL/SQL Procedure
You must enter a fully qualified procedure name (for example, OWNER.PKGNAME.PROCNAME) and ensure that the owner of the Management Repository has execute privilege on the procedure.
An example of the required information is shown in Example 13-10.
Example 13-10 PL/SQL Procedure Required Information
Name Open trouble ticket Description Notification method to open a trouble ticket in the event PLSQL Procedure ticket_sys.ticket_ops.open_ticket
Step 4: Assign the notification method to a rule.
You can edit an existing rule (or create a new notification rule), then go to the Methods page. In the list of Advanced Notification Methods, select your notification method and click 'Assign Method to Rule'. To assign multiple rules to a method or methods to a single rule, see "Assigning Rules to Methods" or "Assigning Methods to Rules".
There can be more than one PL/SQL-based method configured for your Enterprise Manager environment.
Information about the severity types that relate to a target's availability, and how metric severity and policy violation information is passed to the PLSQL procedure is covered in the next section.
Passing Alert and Policy Violation Information to a PL/SQL Procedure
Passing metric severity attributes (severity level, type, notification rule, rule owner, or rule owner, and so on) or policy violation information to PL/SQL procedures allows you to customize automated responses to alerts or policy violations.
The notification system passes information about metric severities or policy violations to a PL/SQL procedure using the MGMT_NOTIFY_SEVERITY object. An instance of this object is created for every alert or policy violation. When an alert or policy violation occurs, the notification system calls the PL/SQL procedure associated with the notification rule and passes the populated object to the procedure. The procedure is then able to access the fields of the MGMT_NOTIFY_SEVERITY object that has been passed to it.
The following table lists all metric severity attributes that can be passed:
Table 13-3 Metric Severity Attributes
Attribute | Datatype | Additional Information |
---|---|---|
TARGET_NAME |
VARCHAR2(256) |
Name of the target on which the severity occurred. |
TARGET_TYPE |
VARCHAR2(64) |
Type of target on which the severity occurred. Targets are defined as any monitorable service. |
TIMEZONE |
VARCHAR2(64) |
The target's regional timezone |
HOST_NAME |
VARCHAR2(128) |
Name of the machine on which the target resides. |
METRIC_NAME |
VARCHAR2(64) |
Metric or policy generating the severity. |
METRIC_DESCRIPTION |
VARCHAR2(128) |
Meaningful description of the metric that can be understood by other administrators. |
METRIC_COLUMN |
VARCHAR2(64) |
For table metrics, the metric column contains the name of the column in the table that is being defined. If the metric that is being defined is not a table metric, the value in this column is a single space. This attribute is not used for policy violations. |
METRIC_VALUE |
VARCHAR2(1024) |
The value of the metric. |
KEY_VALUE |
VARCHAR2(1290) |
For metrics that monitor a set of objects, the KEY_VALUE indicates the specific object that triggered the severity. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE is 'USERS' if the USERS tablespace triggered at warning or critical severity. |
KEY_VALUE_NAME |
VARCHAR2(512) |
For metrics that monitor a set of objects, the KEY_VALUE_NAME indicates the type of object monitored. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE_NAME is 'Tablespace Name'. |
KEY_VALUE_GUID |
VARCHAR2(256) |
GUID associated with a composite key value name. |
CTXT_LIST |
MGMT_NOTIFY_COLUMNS |
Details of the alert context. |
COLLECTION_TIMESTAMP |
DATE |
The time when the target status change was last detected and logged in the management repository. |
SEVERITY_CODE |
NUMBER |
Numeric code identifying the severity level. See Severity Code table below. |
MESSAGE |
VARCHAR2(4000) |
An optional message that is generated when the alert is created that provides additional information about the alert condition. |
SEVERITY_GUID |
RAW(16) |
Severity global unique identifier. |
METRIC_GUID |
RAW(16) |
Metric global unique identifier. |
TARGET_GUID |
RAW(16) |
Target global unique identifier. |
RULE_OWNER |
VARCHAR2(64) |
Name of the Enterprise Manager administrator who owns the rule. |
RULE_NAME |
VARCHAR2(132) |
Name of the notification rule that resulted in the severity. |
When a severity occurs for the target, the notification system creates an instance of the MGMT_NOTIFY_SEVERITY object and populates it with values from the severity. The severity codes in Table 13-4 have been defined as constants in the MGMT_GLOBAL package and can be used to determine the type of severity in the severity_code field of the MGMT_NOTIFY_SEVERITY object.
Table 13-4 Severity Codes
Name | Datatype | Value |
---|---|---|
G_SEVERITY_COMMENT |
NUMBER(2) |
10 |
G_SEVERITY_CLEAR |
NUMBER(2) |
15 |
G_SEVERITY_WARNING |
NUMBER(2) |
20 |
G_SEVERITY_CRITICAL |
NUMBER(2) |
25 |
G_SEVERITY_UNREACHABLE_CLEAR |
NUMBER(3) |
115 |
G_SEVERITY_UNREACHABLE_START |
NUMBER(3) |
125 |
G_SEVERITY_BLACKOUT_END |
NUMBER(3) |
215 |
G_SEVERITY_BLACKOUT_START |
NUMBER(3) |
225 |
G_SEVERITY_ERROR_END |
NUMBER(3) |
315 |
G_SEVERITY_ERROR_START |
NUMBER(3) |
325 |
G_SEVERITY_NO_BEACONS |
NUMBER(3) |
425 |
G_SEVERITY_UNKNOWN |
NUMBER(3) |
515 |
Example 13-11 PL/SQL Procedure Using a Severity Code
CREATE TABLE alert_log (target_name VARCHAR2(64), alert_msg VARCHAR2(4000), occured DATE); PROCEDURE LOG_CRITICAL_ALERTS(severity IN MGMT_NOTIFY_SEVERITY) IS BEGIN -- Log all critical severities IF severity.severity_code = MGMT_GLOBAL.G_SEVERITY_CRITICAL THEN BEGIN INSERT INTO alert_log (target_name, alert_msg, occured) VALUES (severity.target_name, severity.message, severity.collection_timestamp); EXCEPTION WHEN OTHERS THEN -- If there are any problems then get the notification retried RAISE_APPLICATION_ERROR(-20000, 'Please retry'); END; COMMIT; END IF; END LOG_CRITICAL_ALERTS;
Enterprise Manager supports integration with third-party management tools through the SNMP. For example, you can use SNMP to notify a third-party application that a selected metric has exceeded its threshold.
The trap is an SNMP Version 1 trap and is described by the MIB definition shown at the end of this chapter. See "Management Information Base (MIB)".
For more comprehensive configuration information, see the documentation specific to your platform; SNMP configuration differs from platform to platform.
Note:
Notification methods based on SNMP traps must be configured by an administrator with Super Administrator privileges before any user can then choose to select one or more of these SNMP trap methods while creating/editing a notification rule.Step 1: Define a new notification method based on an SNMP trap.
Log in to Enterprise Manager as a Super Administrator. Click Setup and then click Notification Method from the vertical navigation bar to access the Notification Methods page. From this page you can add a new method based on an SNMP trap.
You must provide the name of the host (machine) on which the SNMP master agent is running and other details as shown in the following example. In Example 13-12, the SNMP host will receive your SNMP traps.
Example 13-12 SNMP Trap Required Information
Name HP OpenView Console Description Notification method to send trap to HP OpenView console SNMP Trap Host Name machine1.us.oracle.com SNMP Host Port 162 SNMP Community public This SNMP host will receive your SNMP traps.
Note:
A Test Trap button exists for you to test your setup.Metric severity information will be passed as a series of variables in the SNMP trap.
An example SNMP Trap is shown in Example 13-13. Each piece of information is sent as a variable embedded in the SNMP Trap.
Tue Oct 28 05:00:02 2006 Command: 4 Enterprise: 1.3.6.1.4.1.111.15.2 Agent: 138.1.6.200 Generic Trap: 6 Specific Trap: 1 Time Stamp: 8464:39.99 Count: 11 Name: 1.3.6.1.4.1.111.15.1.1.1.2.1 Kind: OctetString Value: "mydatabase" Name: 1.3.6.1.4.1.111.15.1.1.1.3.1 Kind: OctetString Value: "Database" Name: 1.3.6.1.4.1.111.15.1.1.1.4.1 Kind: OctetString Value: "myhost.com" Name: 1.3.6.1.4.1.111.15.1.1.1.5.1 Kind: OctetString Value: "Owner's Invalid Object Count" Name: 1.3.6.1.4.1.111.15.1.1.1.6.1 Kind: OctetString Value: "Invalid Object Owner" Name: 1.3.6.1.4.1.111.15.1.1.1.7.1 Kind: OctetString Value: "SYS" Name: 1.3.6.1.4.1.111.15.1.1.1.8.1 Kind: OctetString Value: "28-OCT-2006 04:59:10 (US/Eastern GMT)" Name: 1.3.6.1.4.1.111.15.1.1.1.9.1 Kind: OctetString Value: "Warning" Name: 1.3.6.1.4.1.111.15.1.1.1.10.1 Kind: OctetString Value: "12 object(s) are invalid in the SYS schema." Name: 1.3.6.1.4.1.111.15.1.1.1.11.1 Kind: OctetString Value: "Database Metrics" Name: 1.3.6.1.4.1.111.15.1.1.1.12.1 Kind: OctetString Value: "SYSMAN"
Step 2: Assign the notification method to a rule.
You can edit an existing rule (or create a new notification rule), then go to the Methods page. In the list of Advanced Notification Methods, select your notification method and click 'Assign Method to Rule'. To assign multiple rules to a method or methods to a rule, see "Assigning Rules to Methods" or "Assigning Methods to Rules".
Passing corrective action status change attributes (new status, job name, job type, notification rule, or rule owner, etc.) to PL/SQL procedures or OS commands/scripts allows you to customize automated responses to status changes. For example, you many want to call an OS script to open a trouble ticket for an in-house support trouble ticket system if a critical corrective action fails to run. In this case you will want to pass status (Problems, Aborted, etc.) to the script to open a trouble ticket and escalate the problem.
The notification system passes information to an OS script or executable via system environment variables. Conventions used to access environmental variables vary depending on the operating system:
UNIX: $ENV_VARIABLE
MS Windows: %ENV_VARIABLE%
The notification system sets the following environment variables before calling the script. The script can then use any or all of these variables within the logic of the script.
Table 13-5 Environment Variables
Environment Variable | Description |
---|---|
JOB_NAME |
The name of the corrective action. |
JOB_OWNER |
The owner of the corrective action. |
JOB_TYPE |
The type of corrective action. |
JOB_STATUS |
The corrective action status. |
TIMESTAMP |
Time when the severity occurred. |
NUM_TARGETS |
The number of targets. |
TARGET_NAMEn |
The name of the nth target on which the corrective action ran. Example: TARGET_NAME1, TARGET_NAME2. |
METRIC |
The name of the metric in the alert that caused the corrective action to run. Not set for policy violations. |
POLICY_RULE |
The name of the policy rule in the alert that caused the corrective action to run. Not set for metric severities. |
METRIC_VALUE |
The value of the metric column in the alert that caused the corrective action to run. |
VIOLATION_CONTEXT |
A comma-separated list of name-value pairs that show the policy violation context. |
KEY_VALUE_NAME |
For metrics that monitor a set of objects, the KEY_VALUE_NAME indicates the type of object monitored. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE_NAME is 'Tablespace Name'. |
KEY_VALUE |
For metrics that monitor a set of objects, the KEY_VALUE indicates the specific object that triggered the severity. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE is 'USERS' if the USERS tablespace triggered at warning or critical severity. |
SEVERITY |
Type of alert severity. For example, severity types that relate to a target's availability are:
Other metrics can have any of the following severities:
|
RULE_NAME |
Name of the notification rule that resulted in the execution of the corrective action. |
RULE_OWNER |
Name of the Enterprise Manager administrator who owns the notification rule. |
The notification system passes corrective action status change information to a PL/SQL procedure via the MGMT_NOTIFY_CORRECTIVE_ACTION object. An instance of this object is created for every status change. When a corrective action executes, the notification system calls the PL/SQL procedure associated with the notification rule and passes the populated object to the procedure. The procedure is then able to access the fields of the MGMT_NOTIFY_CORRECTIVE_ACTION object that has been passed to it.
Table 13-6 lists all corrective action status change attributes that can be passed:
Table 13-6 Corrective Action Status Attributes
Attribute | Datatype | Additional Information |
---|---|---|
JOB_NAME |
VARCHAR2(128) |
The corrective action name. |
JOB_OWNER |
VARCHAR(256) |
The owner of the corrective action. |
JOB_TYPE |
VARCHAR2(32) |
The type of the corrective action. |
JOB_STATUS |
NUMBER |
The new status of the corrective action. See Table 13-7, "Corrective Action Status Codes" for a list of possible status conditions. |
STATE_CHANGE_GUID |
RAW(16) |
The GUID of the state change record. |
JOB_GUID |
RAW(16) |
The unique id of the corrective action. |
EXECUTION_ID |
RAW(16) |
The unique id of the corrective action execution. |
TARGETS |
SMP_EMD_NVPAIR_ARRAY |
An array of the target name/target type pairs that the corrective action runs on. |
METRIC_NAME |
VARCHAR2(256) |
The name of the metric/policy rule in the alert that caused the corrective action to run. |
METRIC_COLUMN |
VARCHAR2(64) |
The name of the metric in the alert that caused the corrective action to run. This is not used for policy violations. |
METRIC_VALUE |
VARCHAR2(1024) |
The value of the metric in the alert that caused the corrective action to run. |
SEVERITY_CODE |
NUMBER |
The severity code of the alert that caused the corrective action to run. See Table 13-4, "Severity Codes". |
KEY_VALUE_NAME |
VARCHAR2(512) |
For metrics that monitor a set of objects, the KEY_VALUE_NAME indicates the type of object monitored. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE_NAME is 'Tablespace Name'. |
KEY_VALUE |
VARCHAR2(1290) |
For table metrics, this column contains the value of the key column for the row in the table whose thresholds are being defined. If the thresholds are not for a table metric, or the thresholds apply for all rows in the metric column, then the value in this column will contain a single space. |
KEY_VALUE_GUID |
RAW(16) |
GUID associated with a composite key value name. |
CTXT_LIST |
MGMT_NOTIFY_COLUMNS |
Details of the corrective action status change context. |
RULE_OWNER |
VARCHAR2(64) |
The owner of the notification rule that caused the PL/SQL notification to be sent. |
RULE_NAME |
VARCHAR2(132) |
The name of the notification rule that caused the PL/SQL notification method to be invoked. |
OCCURRED_DATE |
DATE |
The time and date when the status change happened. |
The following status codes are possible values for the job_status field of the MGMT_NOTIFY_CORRECTIVE_ACTION object.
Table 13-7 Corrective Action Status Codes
Name | Datatype | Value |
---|---|---|
SCHEDULED_STATUS |
NUMBER(2) |
1 |
EXECUTING_STATUS |
NUMBER(2) |
2 |
ABORTED_STATUS |
NUMBER(2) |
3 |
FAILED_STATUS |
NUMBER(2) |
4 |
COMPLETED_STATUS |
NUMBER(2) |
5 |
SUSPENDED_STATUS |
NUMBER(2) |
6 |
AGENTDOWN_STATUS |
NUMBER(2) |
7 |
STOPPED_STATUS |
NUMBER(2) |
8 |
SUSPENDED_LOCK_STATUS |
NUMBER(2) |
9 |
SUSPENDED_EVENT_STATUS |
NUMBER(2) |
10 |
SUSPENDED_BLACKOUT_STATUS |
NUMBER(2) |
11 |
STOP_PENDING_STATUS |
NUMBER(2) |
12 |
SUSPEND_PENDING_STATUS |
NUMBER(2) |
13 |
INACTIVE_STATUS |
NUMBER(2) |
14 |
QUEUED_STATUS |
NUMBER(2) |
15 |
FAILED_RETRIED_STATUS |
NUMBER(2) |
16 |
WAITING_STATUS |
NUMBER(2) |
17 |
SKIPPED_STATUS |
NUMBER(2) |
18 |
REASSIGNED_STATUS |
NUMBER(2) |
20 |
Example 13-14 PL/SQL Procedure Using a Status Code
CREATE TABLE ca_log (jobid RAW(16), occured DATE); CREATE OR REPLACE PROCEDURE LOG_PROBLEM_CAS(status_change IN MGMT_NOTIFY_CORRECTIVE_ACTION) IS BEGIN -- Log all failed corrective actions IF status_change.job_status = MGMT_JOBS.FAILED_STATUS THEN BEGIN INSERT INTO ca_log (jobid, occured) VALUES (status_change.job_guid, SYSDATE); EXCEPTION WHEN OTHERS THEN -- If there are any problems then get the notification retried RAISE_APPLICATION_ERROR(-20000, 'Please retry'); END; COMMIT; END IF; END LOG_PROBLEM_CAS;
Passing job status change attributes (new status, job name, job type, notification rule, or rule owner, etc.) to PL/SQL procedures or OS commands/scripts allows you to customize automated responses to status changes. For example, you many want to call an OS script to open a trouble ticket for an in-house support trouble ticket system if a critical job fails to run. In this case you will want to pass status (Problems, Aborted, etc.) to the script to open a trouble ticket and escalate the problem.
The notification system passes job status change information to a PL/SQL procedure via the MGMT_NOTIFY_JOB object. An instance of this object is created for every status change. When a job changes status, the notification system calls the PL/SQL procedure associated with the notification rule and passes the populated object to the procedure. The procedure is then able to access the fields of the MGMT_NOTIFY_JOB object that has been passed to it.
Table 13-8 lists all corrective action status change attributes that can be passed:
Table 13-8 Job Status Attributes
Attribute | Datatype | Additional Information |
---|---|---|
job_name |
VARCHAR2(128) |
The job name. |
job_owner |
VARCHAR2(256) |
The owner of the job. |
job_type |
VARCHAR2(32) |
The type of the job. |
job_status |
NUMBER |
The new status of the job. |
state_change_guid |
RAW(16) |
The GUID of the state change record. |
job_guid |
RAW(16) |
The unique id of the job. |
execution_id |
RAW(16) |
The unique id of the execution. |
targets |
SMP_EMD_NVPAIR_ARRAY |
An array of the target name/target type pairs that the job runs on. |
rule_owner |
VARCHAR2(64) |
The name of the notification rule that cause the notification to be sent. |
rule_name |
VARCHAR2(132) |
The owner of the notification rule that cause the notification to be sent. |
occurred_date |
DATE |
The time and date when the status change happened. |
When a job status change occurs for the job, the notification system creates an instance of the MGMT_NOTIFY_JOB object and populates it with values from the status change. The following status codes have been defined as constants in the MGMT_JOBS package and can be used to determine the type of status in the job_status field of the MGMT_NOTIFY_JOB object.
Table 13-9 Job Status Codes
Name | Datatype | Value |
---|---|---|
SCHEDULED_STATUS |
NUMBER(2) |
1 |
EXECUTING_STATUS |
NUMBER(2) |
2 |
ABORTED_STATUS |
NUMBER(2) |
3 |
FAILED_STATUS |
NUMBER(2) |
4 |
COMPLETED_STATUS |
NUMBER(2) |
5 |
SUSPENDED_STATUS |
NUMBER(2) |
6 |
AGENTDOWN_STATUS |
NUMBER(2) |
7 |
STOPPED_STATUS |
NUMBER(2) |
8 |
SUSPENDED_LOCK_STATUS |
NUMBER(2) |
9 |
SUSPENDED_EVENT_STATUS |
NUMBER(2) |
10 |
SUSPENDED_BLACKOUT_STATUS |
NUMBER(2) |
11 |
STOP_PENDING_STATUS |
NUMBER(2) |
12 |
SUSPEND_PENDING_STATUS |
NUMBER(2) |
13 |
INACTIVE_STATUS |
NUMBER(2) |
14 |
QUEUED_STATUS |
NUMBER(2) |
15 |
FAILED_RETRIED_STATUS |
NUMBER(2) |
16 |
WAITING_STATUS |
NUMBER(2) |
17 |
SKIPPED_STATUS |
NUMBER(2) |
18 |
REASSIGNED_STATUS |
NUMBER(2) |
20 |
Example 13-15 PL/SQL Procedure Using a Status Code (Job)
CREATE TABLE job_log (jobid RAW(16), occured DATE); CREATE OR REPLACE PROCEDURE LOG_PROBLEM_JOBS(status_change IN MGMT_NOTIFY_JOB) IS BEGIN -- Log all failed jobs IF status_change.job_status = MGMT_JOBS.FAILED_STATUS THEN BEGIN INSERT INTO job_log (jobid, occured) VALUES (status_change.job_guid, SYSDATE); EXCEPTION WHEN OTHERS THEN -- If there are any problems then get the notification retried RAISE_APPLICATION_ERROR(-20000, 'Please retry'); END; COMMIT; END IF; END LOG_PROBLEM_JOBS;
The notification system passes job execution status information to an OS script or executable via system environment variables. Conventions used to access environmental variables vary depending on the operating system:
UNIX: $ENV_VARIABLE
MS Windows: %ENV_VARIABLE%
The notification system sets the following environment variables before calling the script. The script can then use any or all of these variables within the logic of the script.
Table 13-10 Environment Variables
Environment Variable | Description |
---|---|
JOB_NAME |
The name of the job. |
JOB_OWNER |
The owner of the job. |
JOB_TYPE |
The type of job. |
JOB_STATUS |
The job status. |
TIMESTAMP |
Time when the severity occurred. |
NUM_TARGETS |
The number of targets. |
TARGET_NAMEn |
The name of the nth target. For example, TARGET_NAME1, TARGET_NAME2. |
TARGET_TYPEn |
The type of the nth target. For example TARGET_TYPE1, TARGET_TYPE2. |
RULE_NAME |
Name of the notification rule that resulted in the severity. |
RULE_OWNER |
Name of the Enterprise Manager administrator who owns the notification rule. |
For each notification rule, you can assign one or more notification methods to be called when any of the criteria in the notification rule is met.
From the Enterprise Manager Grid Control, click Preferences at the top of the page.
Click Notification Rules in the vertical navigation bar.
The Enterprise Manager Grid Control displays the Notification Rules page. Any notification rules already created are listed in the Notification Rules table.
Click Assign Methods to Multiple Rules.
Perform your assignments.
For each notification method, you can associate one or more notification rules that will use that method to send notifications.
From the Enterprise Manager Grid Control, click Preferences at the top of the page.
Click Notification Rules in the vertical navigation bar.
The Enterprise Manager Grid Control displays the Notification Rules page. Any notification rules already created are listed in the Notification Rules table.
Click Assign Methods to Multiple Rules.
From the View menu, select By Method.
Perform your assignments.
Enterprise Manager Grid Control can send SNMP Traps to third-party, SNMP-enabled applications. Details of the trap contents can be obtained from the management information base (MIB) variables. The following sections discuss Enterprise Manager MIB variables in detail.
A MIB is a text file, written in ASN.1 notation, which describes the variables containing the information that SNMP can access. The variables described in a MIB, which are also called MIB objects, are the items that can be monitored using SNMP. There is one MIB for each element being monitored. Each monolithic or subagent consults its respective MIB in order to learn the variables it can retrieve and their characteristics. The encapsulation of this information in the MIB is what enables master agents to register new subagents dynamically — everything the master agent needs to know about the subagent is contained in its MIB. The management framework and management applications also consult these MIBs for the same purpose. MIBs can be either standard (also called public) or proprietary (also called private or vendor).
The actual values of the variables are not part of the MIB, but are retrieved through a platform-dependent process called "instrumentation". The concept of the MIB is very important because all SNMP communications refer to one or more MIB objects. What is transmitted to the framework is, essentially, MIB variables and their current values.
This section covers the format used to describe MIB variables. Note that the STATUS element of SNMP MIB definition, Version 2, is not included in these MIB variable descriptions. Since Oracle has implemented all MIB variables as CURRENT, this value does not vary.
Maps to the SYNTAX element of SNMP MIB definition, Version 2.
Maps to the MAX-ACCESS element of SNMP MIB definition, Version 2.
Maps to the STATUS element of SNMP MIB definition, Version 2.
Describes the function, use and precise derivation of the variable. (For example, a variable might be derived from a particular configuration file parameter or performance table field.) When appropriate, incorporates the DESCRIPTION part of the MIB definition, Version 2.
Describes the typical, rather than theoretical, range of the variable. For example, while integer values for many MIB variables can theoretically range up to 4294967295, a typical range in an actual installation will vary to a lesser extent. On the other hand, some variable values for a large database can actually exceed this "theoretical" limit (a "wraparound"). Specifying that a variable value typically ranges from 0 to 1,000 or 1,000 to 3 billion will help the third-party developer to develop the most useful graphical display for the variable.
Describes the significance of the variable when monitoring a typical installation. Alternative ratings are Very Important, Important, Less Important, or Not Normally Used. Clearly, the DBA will want to monitor some variables more closely than others. However, which variables fall into this category can vary from installation to installation, depending on the application, the size of the database, and on the DBA's objectives. Nevertheless, assessing a variable's significance relative to the other variables in the MIB can help third-party developers focus their efforts on those variables of most interest to the most DBAs.
Lists other variables in this MIB, or other MIBs implemented by Oracle, that relate in some way to this variable. For example, the value of this variable might derive from that of another MIB variable. Or perhaps the value of this variable varies inversely to that of another variable. Knowing this information, third-party developers can develop useful graphic displays of related MIB variables.
Suggests how this variable can be presented most usefully to the DBA using the management application: as a simple value, as a gauge, or as an alarm, for example.
Example 13-16 shows a typical MIB definition used by Enterprise Manager.
Example 13-16 MIB Definition
ORACLE-ENTERPRISE-MANAGER-4-MIB DEFINITIONS ::= BEGIN IMPORTS TRAP-TYPE FROM RFC-1215 DisplayString FROM RFC1213-MIB OBJECT-TYPE FROM RFC-1212 enterprises FROM RFC1155-SMI; oracle OBJECT IDENTIFIER ::= { enterprises 111 } oraEM4 OBJECT IDENTIFIER ::= { oracle 15 } oraEM4Objects OBJECT IDENTIFIER ::= { oraEM4 1 } oraEM4AlertTable OBJECT-TYPE SYNTAX SEQUENCE OF OraEM4AlertEntry ACCESS not-accessible STATUS mandatory DESCRIPTION "Information on alerts generated by Oracle Enterprise Manager. This table is not queryable; it exists only to document the variables included in the oraEM4Alert trap. Each trap contains a single instance of each variable in the table." ::= { oraEM4Objects 1 } oraEM4AlertEntry OBJECT-TYPE SYNTAX OraEM4AlertEntry ACCESS not-accessible STATUS mandatory DESCRIPTION "Information about a particular Oracle Enterprise Manager alert." INDEX { oraEM4AlertIndex } ::= { oraEM4AlertTable 1 } OraEM4AlertEntry ::= SEQUENCE { oraEM4AlertIndex INTEGER, oraEM4AlertTargetName DisplayString, oraEM4AlertTargetType DisplayString, oraEM4AlertHostName DisplayString, oraEM4AlertMetricName DisplayString, oraEM4AlertKeyName DisplayString, oraEM4AlertKeyValue DisplayString, oraEM4AlertTimeStamp DisplayString, oraEM4AlertSeverity DisplayString, oraEM4AlertMessage DisplayString, oraEM4AlertRuleName DisplayString oraEM4AlertRuleOwner DisplayString oraEM4AlertMetricValue DisplayString, oraEM4AlertContext DisplayString } oraEM4AlertIndex OBJECT-TYPE SYNTAX INTEGER ACCESS read-only STATUS mandatory DESCRIPTION "Index of a particular alert, unique only at the moment an alert is generated." ::= { oraEM4AlertEntry 1 } oraEM4AlertTargetName OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The name of the target to which this alert applies." ::= { oraEM4AlertEntry 2 } oraEM4AlertTargetType OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The type of the target to which this alert applies." ::= { oraEM4AlertEntry 3 } oraEM4AlertHostName OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The name of the host on which this alert originated." ::= { oraEM4AlertEntry 4 } oraEM4AlertMetricName OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The name of the metric or policy which generated this alert." ::= { oraEM4AlertEntry 5 } oraEM4AlertKeyName OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The name of the key-column, if present, for the metric which generated this alert." ::= { oraEM4AlertEntry 6 } oraEM4AlertKeyValue OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The value of the key-column, if present, for the metric which generated this alert." ::= { oraEM4AlertEntry 7 } oraEM4AlertTimeStamp OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The time at which this alert was generated." ::= { oraEM4AlertEntry 8 } oraEM4AlertSeverity OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The severity of the alert e.g. Critical." ::= { oraEM4AlertEntry 9 } oraEM4AlertMessage OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The message associated with the alert." ::= { oraEM4AlertEntry 10 } oraEM4AlertRuleName OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The name of the notification rule that caused this notification." ::= { oraEM4AlertEntry 11 } oraEM4AlertRuleOwner OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The owner of the notification rule that caused this notification." ::= { oraEM4AlertEntry 12 } oraEM4AlertMetricValue OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The value of the metric which caused this alert to be generated." ::= { oraEM4AlertEntry 13 } oraEM4AlertContext OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "A comma separated list of metric column names and values associated with the metric that caused this alert to be generated." ::= { oraEM4AlertEntry 14 } oraEM4Traps OBJECT IDENTIFIER ::= { oraEM4 2 } oraEM4Alert TRAP-TYPE ENTERPRISE oraEM4Traps VARIABLES { oraEM4AlertTargetName, oraEM4AlertTargetType, oraEM4AlertHostName, oraEM4AlertMetricName, oraEM4AlertKeyName, oraEM4AlertKeyValue, oraEM4AlertTimeStamp, oraEM4AlertSeverity, oraEM4AlertMessage, oraEM4AlertRuleName, oraEM4AlertRuleOwner, oraEM4AlertMetricValue, oraEM4AlertContext } DESCRIPTION "The variables included in the oraEM4Alert trap." ::= 1 oraEM4JobAlertTable OBJECT-TYPE SYNTAX SEQUENCE OF OraEM4JobAlertEntry ACCESS not-accessible STATUS mandatory DESCRIPTION "Information on alerts generated by Oracle Enterprise Manager. This table is not queryable; it exists only to document the variables included in the oraEM4JobAlert trap. Each trap contains a single instance of each variable in the table." ::= { oraEM4Objects 2 } oraEM4JobAlertEntry OBJECT-TYPE SYNTAX OraEM4AlertEntry ACCESS not-accessible STATUS mandatory DESCRIPTION "Information about a particular Oracle Enterprise Manager alert." INDEX { oraEM4JobAlertIndex } ::= { oraEM4JobAlertTable 1 } OraEM4JobAlertEntry ::= SEQUENCE { oraEM4JobAlertIndex INTEGER, oraEM4JobAlertJobName DisplayString, oraEM4JobAlertJobOwner DisplayString, oraEM4JobAlertJobType DisplayString, oraEM4JobAlertJobStatus DisplayString, oraEM4JobAlertTargets DisplayString, oraEM4JobAlertTimeStamp DisplayString, oraEM4JobAlertRuleName DisplayString oraEM4JobAlertRuleOwner DisplayString, oraEM4JobAlertMetricName DisplayString, oraEM4JobAlertMetricValue DisplayString, oraEM4JobAlertContext DisplayString, oraEM4JobAlertKeyName DisplayString, oraEM4JobAlertKeyValue DisplayString, oraEM4JobAlertSeverity DisplayString } oraEM4JobAlertIndex OBJECT-TYPE SYNTAX INTEGER ACCESS read-only STATUS mandatory DESCRIPTION "Index of a particular alert, unique only at the moment an alert is generated." ::= { oraEM4JobAlertEntry 1 } oraEM4JobAlertJobName OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The name of the job to which this alert applies." ::= { oraEM4JobAlertEntry 2 } oraEM4JobAlertJobOwner OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The owner of the job to which this alert applies." ::= { oraEM4JobAlertEntry 3 } oraEM4JobAlertJobType OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The type of the job to which this alert applies." ::= { oraEM4JobAlertEntry 4 } oraEM4JobAlertJobStatus OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The status of the job to which this alert applies." ::= { oraEM4JobAlertEntry 5 } oraEM4JobAlertTargets OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "A comma separated list of target to which this alert applies." ::= { oraEM4JobAlertEntry 6 } oraEM4JobAlertTimeStamp OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The time at which this job status changed causing this alert." ::= { oraEM4JobAlertEntry 7 } oraEM4JobAlertRuleName OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The name of the notification rule that caused this notification." ::= { oraEM4JobAlertEntry 8 } oraEM4JobAlertRuleOwner OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The owner of the notification rule that caused this notification." ::= { oraEM4JobAlertEntry 9 } oraEM4JobAlertMetricName OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The name of the metric or policy which caused the Corrective Action to run that caused this alert." ::= { oraEM4JobAlertEntry 10 } oraEM4JobAlertMetricValue OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The value of the metric which caused the Corrective Action to run that caused this alert." ::= { oraEM4JobAlertEntry 11 } oraEM4JobAlertContext OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "A comma separated list of metric column names and values associated with the metric which caused the Corrective Action to run that caused this alert." ::= { oraEM4JobAlertEntry 12 } oraEM4JobAlertKeyName OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The name of the key-column, if present, for the metric which caused the Corrective Action to run that generated this alert." ::= { oraEM4JobAlertEntry 13 } oraEM4JobAlertKeyValue OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The value of the key-column, if present, for the metric which caused the Corrective Action to run that generated this alert." ::= { oraEM4JobAlertEntry 14 } oraEM4JobAlertSeverity OBJECT-TYPE SYNTAX DisplayString ACCESS read-only STATUS mandatory DESCRIPTION "The severity of the metric which caused the Corrective Action to run that generated this alert e.g. Critical." ::= { oraEM4JobAlertEntry 15 } oraEM4JobAlert TRAP-TYPE ENTERPRISE oraEM4Traps VARIABLES { oraEM4JobAlertJobName, oraEM4JobAlertJobOwner, oraEM4JobAlertJobType, oraEM4JobAlertJobStatus, oraEM4JobAlertTargets, oraEM4JobAlertTimeStamp, oraEM4JobAlertRuleName, oraEM4JobAlertRuleOwner, oraEM4JobAlertMetricName, oraEM4JobAlertMetricValue, oraEM4JobAlertContext, oraEM4JobAlertKeyName, oraEM4JobAlertKeyValue, oraEM4JobAlertSeverity } DESCRIPTION "The variables included in the oraEM4JobAlert trap." ::= 2 END
To function properly, the notification system relies on various components of Enterprise Manager and your IT infrastructure. For this reason, there can be many causes of notification failure. The following guidelines and suggestions can help you isolate potential problems with the notification system.
The first step in diagnosing notification issues is to ensure that you have properly configured and defined your notification environment.
OS Command, PL/SQL and SNMP Trap Notifications
Make sure all OS Command, PLSQL and SNMP Trap Notification Methods are valid by clicking the Test button. This will send a test notification and show any problems the OMS has in contacting the method. Make sure that your method was called, for example, if the OS Command notification is supposed to write information to a log file, check that it has written information to its log file.
E-mail Notifications
Make sure an e-mail gateway is set up under the Notification Methods page of Setup. The Sender's e-mail address should be valid. Clicking the Test button will send an e-mail to the Sender's e-mail address. Make sure this e-mail is received. Note that the Test button ignores any Notification Schedule.
Make sure an e-mail address is setup under General page of Preferences. Clicking the Test button will send an e-mail to specified address and you should make sure this e-mail is received. Note that the Test button ignores any Notification Schedule.
Make sure an e-mail schedule is defined under the Schedule page of Preferences. No e-mails will be sent unless a Notification Schedule has been defined.
Make sure a Notification Rule is defined to match the target, metric, severity and availability states you are interested and make sure e-mail and notification methods are assigned to the rule. A summary of the notification rule can be checked by going to the Rules page under Setup and clicking the rule name.
For any alerts involving problems with notifications, check the following for notification errors.
Any serious errors in the Notification System are logged as system errors in the MGMT_SYSTEM_ERROR_LOG table. These errors can be seen in the Errors page under Management Services and Repository under Setup.
Check for any delivery errors. From the Alerts section of a target home page, click on the alert message to access the metric details page. In the Alert History section, click on the Details icon for more information about the alert. The details will give the reason why the notification was not delivered. Delivery errors are stored in MGMT_NOTIFICATION_LOG with the DELIVERED column set to 'N'.
Severities will not be displayed in the Grid Control console if no metric values have been loaded for the metric associated with the severity.
The Notification System can produce trace messages in sysman/log/emoms.trc file.
Tracing is configured by setting the following flag in sysman/config/emomslogging.properties file. You can set the trace level to INFO, WARN, DEBUG. For example,
log4j.em.notification=DEBUG
Trace messages contain the string "em.notification". If you are working in a UNIX environment, you can search for messages in the emoms.trc file using the grep command. For example,
grep em.notification emoms.trc
What to look for in the trace file.
The following entries in the emoms.trc file are relevant to notifications.
Normal Startup Messages
When the OMS starts, you should see these types of messages.
2006-11-08 03:18:45,385 [Orion Launcher] INFO em.notification init.1279 - Short format maximum length is 155 2006-11-08 03:18:45,386 [Orion Launcher] INFO em.notification init.1297 - Short format is set to both subject and body 2006-11-08 03:18:45,387 [NotificationMgrThread] INFO em.notification run.1010 - Waiting for connection to EM Repository... 2006-11-08 03:18:46,006 [NotificationMgrThread] INFO em.notification run.1041 - Registering for Administrative Queue Name... 2006-11-08 03:18:46,250 [NotificationMgrThread] INFO em.notification run.1078 - Administrative Queue is ADM21 2006-11-08 03:18:46,250 [NotificationMgrThread] INFO em.notification run.1089 - Creating thread pool: min = 6 max = 24 2006-11-08 03:18:48,206 [NotificationMgrThread] INFO em.notification handleAdminNotification.655 - Handling notifications for EMAIL1
Notification Delivery Messages
2006-11-08 03:18:45,387 [NotificationMgrThread] INFO em.notification run.682 - Notification ready on EMAIL1 2006-11-08 03:18:46,006 [DeliveryThread-EMAIL1] INFO em.notification run.114 - Deliver to SYSMAN/admin@oracle.com 2006-11-08 03:18:47,006 [DeliveryThread-EMAIL1] INFO em.notification run.227 - Notification handled for SYSMAN/admin@oracle.com
Notification System Error Messages
2006-11-08 07:26:30,242 [NotificationMgrThread] ERROR em.notification getConnection.237 - Failed to get a connection Io exception: The Network Adapter could not establish the connection
The SMTP gateway is not set up correctly:
Failed to send e-mail to my.admin@oracle.com: For e-mail notifications to be sent, your Super Administrator must configure an Outgoing Mail (SMTP) Server within Enterprise Manager. (SYSMAN, myrule)
Invalid host name:
Failed to connect to gateway: badhost.us.oracle.com: Sending failed; nested exception is: javax.mail.MessagingException: Unknown SMTP host: badhost.us.oracle.com;
Invalid e-mail address:
Failed to connect to gateway: rgmemeasmtp.oraclecorp.com: Sending failed; nested exception is: javax.mail.MessagingException: 550 5.7.1 <smpemailtest_ie@oracle.com>... Access denied
Always use the Test button to make sure the e-mail gateway configuration is valid. Check that an e-mail is received at the sender's e-mail address
When attempting to execute an OS command or script, the following errors may occur. Use the Test button to make sure OS Command configuration is valid. If there are any errors, they will appear in the console.
Invalid path or no read permissions on file:
Could not find /bin/myscript (stacb10.us.oracle.com_Management_Service) (SYSMAN, myrule )
No execute permission on executable:
Error calling /bin/myscript: java.io.IOException: /bin/myscript: cannot execute (stacb10.us.oracle.com_Management_Service) (SYSMAN, myrule )
Timeout because OS Command ran too long:
Timeout occurred running /bin/myscript (stacb10.us.oracle.com_Management_Service) (SYSMAN, myrule )
Any errors such as out of memory or too many processes running on OMS machine will be logged as appropriate.
Always use the Test button to make sure OS Command configuration is valid.
Use the Test button to make sure SNMP Trap configuration is valid.
Other possible SNMP trap problems include: invalid host name, port, or community for a machine running an SNMP Console.
When attempting to execute an PL/SQL procedure, the following errors may occur. Use the Test button to make sure the procedure is valid. If there are any errors, they will appear in the console.
Procedure name is invalid or is not fully qualified. Example: SCOTT.PKG.PROC
Error calling PL/SQL procedure plsql_proc: ORA-06576: not a valid function or procedure name (SYSMAN, myrule)
Procedure is not the correct signature. Example: PROCEDURE p(s IN MGMT_NOTIFY_SEVERITY)
Error calling PL/SQL procedure plsql_proc: ORA-06553: PLS-306: wrong number or types of arguments in call to 'PLSQL_PROC' (SYSMAN, myrule)
Procedure has bug and is raising an exception.
Error calling PL/SQL procedure plsql_proc: ORA-06531: Reference to uninitialized collection (SYSMAN, myrule)
Care should be taken to avoid leaking cursors in your PL/SQL. Any exception due to this condition will result in delivery failure with the message being displayed in the Details section of the alert in the Grid Control console.
Always use the Test button to make sure the PL/SQL configuration is valid.