Node Eviction - Where to look for information

Node evictions are usually a result of the following processes

OCSSD (Oracle CSS Daemon)
CSSDAGENT
CSSDMONITOR

We need to determine which process was responsible for the eviction.
Details can be found in the following locations/files.

1) Check the Clusterware alert log

Set the oraenv to ASM

echo $ORACLE_HOME
cd $ORACLE_HOME/log/nodename/

There should be a file called "alert(nodename).log"

Also Check the O/S messages file

linux - /var/log/messages
Sun - /var/adm/messages


2) Check the OCSSD log

From the Oracle docs..

"ocssd (Cluster Synchronization Service daemon):
  Manages cluster node membership and runs as the oracle user; failure of this process results in a node restart."

cd $ORACLE_HOME/log/[nodename]/cssd

There should be a file called "ocssd.log"

Use ls -latr to find the latest logs


3) CSSDAGENT and CSSDMONITOR

From the Oracle docs these process are responsible for the following.

"cssdagent: Starts, stops, and checks the status of the CSS daemon, ocssd.
   In addition, the cssdagent and cssdmonitor provide the following services to guarantee data integrity:
Monitors the CSS daemon; if the CSS daemon stops, then it shuts down the node
Monitors the node scheduling to verify that the node is not hung, and shuts down the node on recovery from a hang."

What we are probably not going to see here is database instance related crashes. Only Cluster related information.

cd $ORACLE_HOME/log/[nodename]/agent/ohasd/oracssdagent_root

There should be a file called "oracssdagent_root.log"


CSSDMONITOR logs

cd $ORACLE_HOME/log/[nodename]/agent/ohasd/oracssdmonitor_root

There should be a file called "oracssdmonitor_root.log"


4) Lastgasp!

cd /etc/oracle/lastgasp/

Check any files in this directory.

******************************************
keywords: cluster node eviction
******************************************
rdbms version: 11g r2
******************************************