At 16:45 on 26th April oss service group tried to failover but failed and hence outage happened.
status at that time was
root@yginmate> hastatus -sum
-- SYSTEM STATE
-- System State Frozen
A yginmaster RUNNING 0
A yginmate RUNNING 0
-- GROUP STATE
-- Group System Probed AutoDisabled State
B Oss yginmaster Y N OFFLINE
B Oss yginmate Y N STARTING|PARTIAL
B Ossfs yginmaster Y N OFFLINE|FAULTED
B Ossfs yginmate Y N ONLINE
B PrivLan yginmaster Y N ONLINE
B PrivLan yginmate Y N ONLINE
B PubLan yginmaster Y N ONLINE
B PubLan yginmate Y N ONLINE
B Sybase1 yginmaster Y N OFFLINE
B Sybase1 yginmate Y N ONLINE
On checking engine logs on uginmaster
2013/04/26 16:42:11 VCS WARNING V-16-10001-0 (yginmaster) Application:mount_lofs:mount_lofs.sh:Mount error (/ossrc/3pp/opt/adobe on /opt/adobe). Still mounted but directories moved or deleted
2013/04/26 16:42:11 VCS WARNING V-16-10001-1063 (yginmaster) DiskGroup:ossdg:monitor:Disk Group: ossdg is disabled on system: yginmaster. Not failing-over to protect data from potential corruption.
2013/04/26 16:42:11 VCS WARNING V-16-10001-0 (yginmaster) Application:mount_lofs:mount_lofs.sh:Mount error (/ossrc/3pp/opt/adventnet on /opt/adventnet). Still mounted but directories moved or deleted
2013/04/26 16:42:11 VCS WARNING V-16-10001-0 (yginmaster) Application:mount_lofs:mount_lofs.sh:Mount error (/ossrc/3pp/opt/doc on /opt/doc). Still mounted but directories moved or deleted
2013/04/26 16:42:11 VCS WARNING V-16-10001-0 (yginmaster) Application:mount_lofs:mount_lofs.sh:Mount error (/ossrc/3pp/opt/htdocs oI can see
2013/04/26 16:43:14 VCS WARNING V-16-10001-0 (yginmaster) Application:mount_lofs:mount_lofs.sh:Mount error (/export/opt/ericsson on /opt/ericsson). Still mounted but directories moved or deleted
2013/04/26 16:43:14 VCS ERROR V-16-2-13067 (yginmaster) Agent is calling clean for resource(eba_ebsw_mount) because the resource became OFFLINE unexpectedly, on its own.
2013/04/26 16:43:14 VCS ERROR V-16-2-13067 (yginmaster) Agent is calling clean for resource(eniq_pm_vol_mount) because the resource became OFFLINE unexpectedly, on its own.
2013/04/26 16:43:14 VCS INFO V-16-2-13068 (yginmaster) Resource(sgwcg_share) - clean completed successfully.
2013/04/26 16:43:14 VCS ERROR V-16-2-13073 (yginmaster) Resource(sgwcg_share) became OFFLINE unexpectedly on its own. Agent is restarting (attempt number 1 of 99) the resource.
2013/04/26 16:43:14 VCS ERROR V-16-2-13067 (yginmaster) Agent is calling clean for resource(eba_ebsw_share) because the resource became OFFLINE unexpectedly, on its own.
2013/04/26 16:43:14 VCS ERROR V-16-2-13067 (yginmaster) Agent is calling clean for resource(eniq_pm_vol_share) because the resource became OFFLINE unexpectedly, on its own.
013/04/26 16:43:20 VCS INFO V-16-2-13716 (yginmaster) Resource(cluster_maint): Output of the completed operation (offline)
==============================================
df: cannot statvfs /export: I/O error
df: cannot statvfs /ossrc/ericsson: I/O error
df: cannot statvfs /ossrc/data/pms/segment1: I/O error
df: cannot statvfs /ossrc/3pp: I/O error
df: cannot statvfs /opt/adobe: I/O error
df: cannot statvfs /opt/adventnet: I/O error
df: cannot statvfs /opt/doc: I/O error
df: cannot statvfs /opt/htdocs: I/O error
df: cannot statvfs /opt/borland: I/O error
df: cannot statvfs /opt/prismtech: I/O error
df: cannot statvfs /opt/jakarta: I/O error
df: cannot statvfs /opt/python: I/O error
df: cannot statvfs /opt/sun: I/O error
df: cannot statvfs /opt/sybase: I/O error
df: cannot statvfs /opt/uab: I/O error
df: cannot statvfs /opt/versant: I/O error
df: cannot statvfs /opt/misc3pp: I/O error
df: cannot statvfs /opt/maverick: I/O error
df: cannot statvfs /opt/Sentinel: I/O error
df: cannot statvfs /opt/activemq: I/O error
df: cannot statvfs /opt/glassfish: I/O error
df: cannot statvfs /opt/glassfish3: I/O error
df: cannot statvfs /opt/OpenDJ: I/O error
df: cannot statvfs /opt/miscOSGi3pp: I/O error
df: cannot statvfs /opt/misc3ppsparc: I/O error
df: cannot statvfs /etc/opt/adventnet: I/O error
df: cannot statvfs /etc/opt/borland: I/O error
df: cannot statvfs /etc/opt/prismtech: I/O error
df: cannot statvfs /etc/opt/python: I/O error
df: cannot statvfs /etc/opt/sun: I/O error
df: cannot statvfs /etc/opt/sybase: I/O error
df: cannot statvfs /etc/opt/uab: I/O error
df: cannot statvfs /etc/opt/versant: I/O error
df: cannot statvfs /opt/ericsson: I/O error
df: cannot statvfs /etc/opt/ericsson: I/O error
df: cannot statvfs /var/opt/ericsson: I/O error
df: cannot statvfs /var/opt/ericsson/nms_umts_pms_seg/segment1: I/O error
df: cannot statvfs /var/opt/ericsson/lge/k-count: I/O error
df: cannot statvfs /var/opt/ericsson/lge/sybase: I/O error
df: cannot statvfs /export: I/O error
df: cannot statvfs /ossrc/ericsson: I/O error
df: cannot statvfs /ossrc/data/pms/segment1: I/O error
df: cannot statvfs /ossrc/3pp: I/O error
df: cannot statvfs /opt/adobe: I/O error
df: cannot statvfs /opt/adventnet: I/
==============================================
2013/04/26 16:43:20 VCS INFO V-16-2-13716 (yginmaster) Resource(smrs_nfs): Output of the completed operation (offline)
==============================================
svcadm: Instance "svc:/ericsson/smrs/smrs_nfs:default" is in maintenance state.
==============================================
2013/04/26 16:43:20 VCS INFO V-16-2-13075 (yginmaster) Resource(oad) has reported unexpected OFFLINE 1 times, which is still within the ToleranceLimit(2).
2013/04/26 16:43:20 VCS INFO V-16-2-13068 (yginmaster) Resource(eba_ebss_mount) - clean completed successfully.
2013/04/26 16:43:20 VCS INFO V-16-1-10307 Resource eba_ebss_mount (Owner: Unspecified, Group: Ossfs) is offline on yginmaster (Not initiated by VCS)
2013/04/26 16:43:20 VCS ERROR V-16-2-13073 (yginmaster) Resource(eba_rsdm_share) became OFFLINE unexpectedly on its own. Agent is restarting (attempt number 1 of 99) the resource.
2013/04/26 16:43:40 VCS INFO V-16-1-10305 Resource supervisor (Owner: Unspecified, Group: Oss) is offline on yginmaster (VCS initiated)
2013/04/26 16:43:56 VCS INFO V-16-2-13075 (yginmaster) Resource(tomcat) has reported unexpected OFFLINE 2 times, which is still within the ToleranceLimit(2).
2013/04/26 16:43:56 VCS INFO V-16-2-13075 (yginmaster) Resource(sentinel) has reported unexpected OFFLINE 1 times, which is still within the ToleranceLimit(2).
2013/04/26 16:43:56 VCS INFO V-16-2-13075 (yginmaster) Resource(opendj) has reported unexpected OFFLINE 2 times, which is still within the ToleranceLimit(2).
2013/04/26 16:43:56 VCS INFO V-16-2-13075 (yginmaster) Resource(rmi_reg_ext) has reported unexpected OFFLINE 2 times, which is still within the ToleranceLimit(2).
2013/04/26 16:43:56 VCS INFO V-16-2-13075 (yginmaster) Resource(osagent) has reported unexpected OFFLINE 1 times, which is still within the ToleranceLimit(2).
2013/04/26 16:43:56 VCS INFO V-16-2-13075 (yginmaster) Resource(nsa) has reported unexpected OFFLINE 1 times, which is still within the ToleranceLimit(2).
2013/04/26 16:43:58 VCS INFO V-16-2-13075 (yginmaster) Resource(notif) has reported unexpected OFFLINE 1 times, which is still within the ToleranceLimit(2).
2013/04/26 16:43:59 VCS INFO V-16-2-13075 (yginmaster) Resource(ext_notif) has reported unexpected OFFLINE 1 times, which is still within the ToleranceLimit(2).
2013/04/26 16:43:59 VCS INFO V-16-2-13075 (yginmaster) Resource(ext_nsa) has reported unexpected OFFLINE 1 times, which is still within the ToleranceLimit(2).
2013/04/26 16:43:59 VCS INFO V-16-2-13075 (yginmaster) Resource(gui_nsa) has reported unexpected OFFLINE 1 times, which is still within the ToleranceLimit(2).
2013/04/26 16:43:59 VCS INFO V-16-2-13075 (yginmaster) Resource(sb_nsa) has reported unexpected OFFLINE 1 times, which is still within the ToleranceLimit(2).
2013/04/26 16:44:00 VCS WARNING V-16-10001-1063 (yginmaster) DiskGroup:ossdg:monitor:Disk Group: ossdg is disabled on system: yginmaster. Not failing-over to protect data from potential corruption.
13/04/26 16:44:17 VCS INFO V-16-2-13068 (yginmaster) Resource(var_share) - clean completed successfully.
2013/04/26 16:44:17 VCS ERROR V-16-2-13066 (yginmaster) Agent is calling clean for resource(mail_share) because the resource is not up even after online completed.
2013/04/26 16:44:17 VCS INFO V-16-2-13071 (yginmaster) Resource(var_share): reached OnlineRetryLimit(0).
2013/04/26 16:44:17 VCS ERROR V-16-2-13066 (yginmaster) Agent is calling clean for resource(a3pp_share) because the resource is not up even after online completed.
2013/04/26 16:44:17 VCS INFO V-16-2-13071 (yginmaster) Resource(segment1_share): reached OnlineRetryLimit(0).
2013/04/26 16:44:17 VCS ERROR V-16-2-13066 (yginmaster) Agent is calling clean for resource(ericsson_share) because the resource is not up even after online completed.
2013/04/26 16:44:17 VCS ERROR V-16-2-13066 (yginmaster) Agent is calling clean for resource(home_share) because the resource is not up even after online completed.
2013/04/26 16:44:17 VCS ERROR V-16-2-13066 (yginmaster) Agent is calling clean for resource(etc_share) because the resource is not up even after online completed.
2013/04/26 16:44:17 VCS ERROR V-16-1-10303 Resource segment1_share (Owner: Unspecified, Group: Ossfs) is FAULTED (timed out) on sys yginmaster
Can you guys please have a looka t above logs and suggest. I highlighted the line above which might be helpful.