2010
02.10

Failed to shutdown DBConsole Gracefully

Category: IT / Tag: , , ,  / Add Comment
1 person likes this post.
Unlike
Share

Pretty average Oracle RAC Cluster (10g, two nodes). All of a sudden, trying to stop dbconsole results in an error:

[oracle@racnode1 log]$ emctl stop dbconsole
TZ set to Europe/Vatican
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
https://racnode1:1158/em/console/aboutApplication
Stopping Oracle Enterprise Manager 10g Database Control ...
--- Failed to shutdown DBConsole Gracefully ---
 failed.

Similar behaviour when attempting to start it:

[oracle@racnode1 log]$ emctl start dbconsole
TZ set to Europe/Vatican
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
https://racnode1:1158/em/console/aboutApplication
Agent Version     : 10.1.0.6.0
OMS Version       : 10.1.0.6.0
Protocol Version  : 10.1.0.2.0
Agent Home        : /opt/oracle/product/10.2.0/db_1/racnode1_DBSID1
Agent binaries    : /opt/oracle/product/10.2.0/db_1
Agent Process ID  : 24756
Parent Process ID : 24753
Agent URL         : https://racnode1:3938/emd/main
Started at        : 2010-02-09 13:48:34
Started by user   : oracle
Last Reload       : 2010-02-09 13:48:34
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far :     0.00
Number of XML files pending upload           :     3971
Size of XML files pending upload(MB)         :    50.11
Available disk space on upload filesystem    :    59.30%
Agent is already started. Will restart the agent
Stopping agent ... stopped.
Starting Oracle Enterprise Manager 10g Database Control ............................................................................................. failed.
------------------------------------------------------------------
Logs are generated in directory /opt/oracle/product/10.2.0/db_1/racnode1_DBSID1/sysman/log

Peeking into the emdctl.trc logfile, I found something that definitely smelled of expired certificates:

2010-02-09 13:54:14 Thread-4134193952 ERROR http: 6: Unable to initialize ssl connection with server, aborting connection attempt
2010-02-09 13:54:16 Thread-4133477152 WARN  http: snmehl_connect: connect failed to (racnode1:3938): Connection refused (error = 111)
2010-02-09 13:54:41 Thread-4134140704 ERROR ssl: nzos_Handshake failed, ret=29024

This thread on Oracle forums seems to confirm my suspicion.

And here’s what you do to fix the issue:

Ready your environment ($ORACLE_SID, $ORACLE_HOME, …). I “source” a script for that, each instance has its own.

[oracle@racnode1 ~]$ cat envDBSID.sh
export ORACLE_SID=DBSID1
export ORACLE_HOME=/opt/oracle/product/10.2.0/db_1
export PATH=/usr/local/bin:/bin:/usr/bin:/home/oracle/bin:"$ORACLE_HOME"/bin

[oracle@racnode1 ~]$ . envDBSID.sh

As I said, if the certificate is expired, dbconsole won’t shutdown cleanly. Fetch its pid and kill it manually.

[oracle@racnode1 ~]$ cat /opt/oracle/product/10.2.0/db_1/racnode1_DBSID1/emctl.pid  25608

[oracle@racnode1 ~]$ ps axo pid,command | grep 25608
25608 /opt/oracle/product/10.2.0/db_1/jdk/bin/java -server -Xmx256M -XX [..]

[oracle@racnode1 ~]$ kill 25608

[oracle@racnode1 ~]$ ps axo pid,command | grep 25608
[oracle@racnode1 ~]$

Run emctl secure dbconsole, it will generate the new certificates. Provide Oracle SYSMAN’s password and the hostname you’ll use (without domain name, in my case).
The URL displayed (the port number) will also tell you if you’re on the right instance and got the intended environment.

[oracle@racnode1 ~]$ emctl secure dbconsole
TZ set to Europe/Vatican
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
https://racnode1:5500/em/console/aboutApplication
Enter Enterprise Manager Root password :
Enter a Hostname for this OMS : racnode1

DBCONSOLE already stopped...   Done.
Agent is already stopped...   Done.
Securing dbconsole...   Started.
Checking Repository...   Done.
Checking Em Key...   Done.
Checking Repository for an existing Enterprise Manager Root Key...   Done.
Fetching Root Certificate from the Repository...   Done.
Updating HTTPS port in emoms.properties file...   Done.
Generating Java Keystore...   Done.
Securing OMS ...   Done.
Generating Oracle Wallet Password for Agent....   Done.
Generating wallet for Agent ...    Done.
Copying the wallet for agent use...    Done.
Storing agent key in repository...   Done.
Storing agent key for agent ...   Done.
Configuring Agent...
Configuring Agent for HTTPS in DBCONSOLE mode...   Done.
EMD_URL set in /opt/oracle/product/10.2.0/db_1/racnode1_RDS1/sysman/config/emd.properties
   Done.
Configuring Key store..   Done.
Securing dbconsole...   Sucessful.

Try and start dbconsole.

[oracle@racnode1 ~]$ emctl start dbconsole
TZ set to Europe/Vatican
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
https://racnode1:5500/em/console/aboutApplication
Starting Oracle Enterprise Manager 10g Database Control ............................. started.
------------------------------------------------------------------
Logs are generated in directory /opt/oracle/product/10.2.0/db_1/racnode1_RDS1/sysman/log

[oracle@racnode1 ~]$ emctl status dbconsole
TZ set to Europe/Vatican
Oracle Enterprise Manager 10g Database Control Release 10.2.0.4.0
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
https://racnode1:5500/em/console/aboutApplication
Oracle Enterprise Manager 10g is running.
------------------------------------------------------------------
Logs are generated in directory /opt/oracle/product/10.2.0/db_1/racnode1_RDS1/sysman/log

Be prepared to handle the same situation in a few months. To see the new certificate expiry date, open any HTTPS URL served by dbconsole (i.e.: https://racnode1:5500/em/console/aboutApplication) and click on the lock icon your web browser should show somewhere.

Share