Hi Team,
Many times we face resource faults alerts on our AIX server,but when we check it shows all are running fine.
Also, when checked logs it shows not (initiated by VCS) alerts.
Our main concern is to troubleshoot, why we get these alerts and if we get these alerts then why cluster resource is not showing faulty.
Below are the logs,
-- SYSTEM STATE
-- System State Frozen
A xxxibm012 RUNNING 0
A xxxibm014 RUNNING 0
-- GROUP STATE
-- Group System Probed AutoDisabled State
B ClusterService xxxibm012 Y N ONLINE
B ClusterService xxxibm014 Y N OFFLINE
B DB_INSIGHT_STAGE xxxibm012 Y N ONLINE
B DB_INSIGHT_STAGE xxxibm014 Y N OFFLINE
=============================================================
2015/04/21 10:14:53 VCS INFO V-16-1-53504 VCS Engine Alive message!!
2015/04/21 12:57:32 VCS WARNING V-16-10011-5611 (clnibm014) NIC:csgnic:monitor:Second PingTest failed for Virtual Interface en4. Resource is OFFLINE
2015/04/21 12:57:32 VCS INFO V-16-1-50135 User root fired command: hares -modify csgnic ConfidenceMsg from localhost
2015/04/21 12:57:33 VCS ERROR V-16-1-54031 Resource csgnic (Owner: Unspecified, Group: ClusterService) is FAULTED on sys clnibm014
2015/04/21 12:57:33 VCS INFO V-16-6-0 (clnibm014) resfault:(resfault) Invoked with arg0=clnibm014, arg1=csgnic, arg2=ONLINE
2015/04/21 12:57:49 VCS INFO V-16-6-15002 (clnibm014) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault clnibm014 csgnic ONLINE successfully
2015/04/21 12:58:18 VCS ERROR V-16-1-54031 Resource proxy_DB_INSPRD (Owner: Unspecified, Group: DB_INSIGHT_STAGE) is FAULTED on sys clnibm014
2015/04/21 12:58:18 VCS INFO V-16-6-0 (clnibm014) resfault:(resfault) Invoked with arg0=clnibm014, arg1=proxy_DB_INSPRD, arg2=ONLINE
2015/04/21 12:58:29 VCS INFO V-16-6-15002 (clnibm014) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/resfault clnibm014 proxy_DB_INSPRD ONLINE successfully
2015/04/21 12:58:33 VCS INFO V-16-1-50135 User root fired command: hares -modify csgnic ConfidenceMsg Primary test to confirm Online status succeeded. from localhost
2015/04/21 12:58:34 VCS INFO V-16-1-10299 Resource csgnic (Owner: Unspecified, Group: ClusterService) is online on clnibm014 (Not initiated by VCS)
2015/04/21 12:58:34 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes
2015/04/21 12:58:34 VCS NOTICE V-16-1-51034 Failover group ClusterService is already active. Ignoring Restart
2015/04/21 12:59:18 VCS INFO V-16-1-10299 Resource proxy_DB_INSPRD (Owner: Unspecified, Group: DB_INSIGHT_STAGE) is online on clnibm014 (Not initiated by VCS)
2015/04/21 12:59:18 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group DB_INSIGHT_STAGE on all nodes
2015/04/21 12:59:18 VCS NOTICE V-16-1-51034 Failover group DB_INSIGHT_STAGE is already active. Ignoring Restart
2015/04/21 12:59:34 VCS INFO V-16-1-50135 User root fired command: hares -modify csgnic ConfidenceMsg Primary test to confirm Online status succeeded. from localhost
2015/04/21 13:18:53 VCS INFO V-16-1-50135 User root fired command: hares -modify csgnic ConfidenceMsg Relying on secondary test to confirm Online status. from localhost
2015/04/21 13:19:34 VCS INFO V-16-1-50135 User root fired command: hares -modify csgnic ConfidenceMsg Primary test to confirm Online status succeeded. from localhost
2015/04/21 13:44:49 VCS INFO V-16-1-50135 User root fired command: hares -modify csgnic ConfidenceMsg Relying on secondary test to confirm Online status. from localhost
2015/04/21 13:45:34 VCS INFO V-16-1-50135 User root fired command: hares -modify csgnic ConfidenceMsg Primary test to confirm Online status succeeded. from localhost
2015/04/21 14:14:54 VCS INFO V-16-1-53504 VCS Engine Alive message!!
2015/04/21 16:48:59 VCS INFO V-16-1-50135 User root fired command: hares -modify csgnic ConfidenceMsg Relying on secondary test to confirm Online status. from localhost
2015/04/21 16:49:34 VCS INFO V-16-1-50135 User root fired command: hares -modify csgnic ConfidenceMsg Primary test to confirm Online status succeeded.