need to know root cause for node fault
Jan 1 19:08:51 kyorns051 AgentFramework[19516]: [ID 702911 daemon.notice] VCS ERROR V-16-2-13027 Thread(4) Resource(private_lan) - monitor proc
edure did not complete within the expected time.
Jan 1 19:08:51 kyorns051 Had[19421]: [ID 702911 daemon.notice] VCS ERROR V-16-2-13027 (kyorns051) Resource(private_lan) - monitor procedure did
not complete within the expected time.
Jan 1 19:13:02 kyorns051 genunix: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port h gen 2a150b membership 0
Jan 1 19:13:02 kyorns051 genunix: [ID 674723 kern.notice] GAB INFO V-15-1-20038 Port h gen 2a150b k_jeopardy ;1
Jan 1 19:13:02 kyorns051 genunix: [ID 513393 kern.notice] GAB INFO V-15-1-20040 Port h gen 2a150b visible ;1
Jan 1 19:13:02 kyorns051 Had[19421]: [ID 702911 daemon.notice] VCS INFO V-16-1-10077 Received new cluster membership
Jan 1 19:13:02 kyorns051 Had[19421]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10113 System kyorns052 (Node '1') is in DDNA Membership - Membe
rship: 0x1, Visible: 0x0
Jan 1 19:13:02 kyorns051 Had[19421]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10322 System kyorns052 (Node '1') changed state from RUNNING to
FAULTED
Jan 1 19:13:22 kyorns051 AgentFramework[19516]: [ID 702911 daemon.notice] VCS ERROR V-16-2-13210 Thread(7) Agent is calling clean for resource(
private_lan) because 4 successive invocations of the monitor procedure did not complete within the expected time.
Jan 1 19:13:22 kyorns051 Had[19421]: [ID 702911 daemon.notice] VCS ERROR V-16-2-13210 (kyorns051) Agent is calling clean for resource(private_l
an) because 4 successive invocations of the monitor procedure did not complete within the expected time.
Jan 1 19:13:22 kyorns051 AgentFramework[19516]: [ID 702911 daemon.notice] VCS ERROR V-16-2-13068 Thread(7) Resource(private_lan) - clean comple
ted successfully.
Jan 1 19:13:22 kyorns051 AgentFramework[19516]: [ID 702911 daemon.notice] VCS ERROR V-16-2-13074 Thread(7) The monitoring program for resource(
private_lan) has consistently failed to determine the resource status within the expected time. Agent is restarting (attempt number 1 of 1) the
resource.
Jan 1 19:13:22 kyorns051 Had[19421]: [ID 702911 daemon.notice] VCS ERROR V-16-2-13074 (kyorns051) The monitoring program for resource(private_l
an) has consistently failed to determine the resource status within the expected time. Agent is restarting (attempt number 1 of 1) the resource.
Jan 1 19:13:22 kyorns051 Had[19421]: [ID 702911 daemon.notice] VCS ERROR V-16-10001-87 (kyorns051) Monitor:private_lan:online:Failed to execute
[/ericsson/core/cluster/scripts/private_lan.sh start].
Jan 1 19:14:59 kyorns051 genunix: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 0 (oce1) node 1 in trouble
Jan 1 19:14:59 kyorns051 genunix: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 1 (oce8) node 1 in trouble
Jan 1 19:15:01 kyorns051 genunix: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 2 (oce0) node 1 in trouble
Jan 1 19:15:05 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 1 inactive 8 sec (4110347)
Jan 1 19:15:05 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 1 inactive 8 sec (8723290)
Jan 1 19:15:05 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (oce8) node 1 inactive 8 sec (8723059)
Jan 1 19:15:06 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 1 inactive 9 sec (4110347)
Jan 1 19:15:06 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 1 inactive 9 sec (8723290)
Jan 1 19:15:06 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (oce8) node 1 inactive 9 sec (8723059)
Jan 1 19:15:07 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 1 inactive 10 sec (4110347)
Jan 1 19:15:07 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 1 inactive 10 sec (8723291)
Jan 1 19:15:07 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (oce8) node 1 inactive 10 sec (8723060)
Jan 1 19:15:08 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 1 inactive 11 sec (4110347)
Jan 1 19:15:08 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 1 inactive 11 sec (8723291)
Jan 1 19:15:08 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (oce8) node 1 inactive 11 sec (8723060)
Jan 1 19:15:09 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 1 inactive 12 sec (4110347)
Jan 1 19:15:09 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 1 inactive 12 sec (8723292)
Jan 1 19:15:09 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (oce8) node 1 inactive 12 sec (8723061)
Jan 1 19:15:15 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 1 inactive 18 sec (4110347)
Jan 1 19:15:15 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 1 inactive 18 sec (8723295)
Jan 1 19:15:15 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (oce8) node 1 inactive 18 sec (8723064)
Jan 1 19:15:25 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 1 inactive 28 sec (4110347)
Jan 1 19:15:25 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 1 inactive 28 sec (8723301)
Jan 1 19:15:25 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (oce8) node 1 inactive 28 sec (8723070)
Jan 1 19:15:35 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 1 inactive 38 sec (4110349)
Jan 1 19:15:35 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 1 inactive 38 sec (8723306)
Jan 1 19:15:35 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (oce8) node 1 inactive 38 sec (8723075)
Jan 1 19:15:45 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 1 inactive 48 sec (4110349)
Jan 1 19:15:45 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 1 inactive 48 sec (8723311)
Jan 1 19:15:45 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (oce8) node 1 inactive 48 sec (8723080)
Jan 1 19:15:55 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 1 inactive 58 sec (4110350)
Jan 1 19:15:55 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 1 inactive 58 sec (8723317)
Jan 1 19:15:55 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (oce8) node 1 inactive 58 sec (8723086)
Jan 1 19:16:05 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 1 inactive 68 sec (4110351)
Jan 1 19:16:05 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 1 inactive 68 sec (8723322)
Jan 1 19:16:05 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (oce8) node 1 inactive 68 sec (8723091)
Jan 1 19:16:15 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 1 inactive 78 sec (4110351)
Jan 1 19:16:15 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 1 inactive 78 sec (8723327)
Jan 1 19:16:15 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (oce8) node 1 inactive 78 sec (8723096)
Jan 1 19:16:25 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 1 inactive 88 sec (4110352)
Jan 1 19:16:25 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 1 inactive 88 sec (8723333)
Jan 1 19:16:25 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (oce8) node 1 inactive 88 sec (8723102)
Jan 1 19:16:35 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 1 inactive 98 sec (4110353)
Jan 1 19:16:35 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 1 inactive 98 sec (8723338)
Jan 1 19:16:35 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (oce8) node 1 inactive 98 sec (8723107)
Jan 1 19:16:45 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 1 inactive 108 sec (4110353)
Jan 1 19:16:46 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 1 inactive 108 sec (8723343)
Jan 1 19:16:46 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (oce8) node 1 inactive 108 sec (8723112)
Jan 1 19:16:53 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 1 inactive 116 sec (4110353)
Jan 1 19:16:54 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 1 inactive 116 sec (8723348)
Jan 1 19:16:54 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (oce8) node 1 inactive 116 sec (8723117)
Jan 1 19:16:54 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 1 inactive 117 sec (4110353)
Jan 1 19:16:55 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 1 inactive 117 sec (8723349)
Jan 1 19:16:55 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (oce8) node 1 inactive 117 sec (8723118)
Jan 1 19:16:55 kyorns051 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 2 (oce0) node 1. 4 more to go.
Jan 1 19:16:55 kyorns051 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (oce1) node 1. 4 more to go.
Jan 1 19:16:55 kyorns051 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 1 (oce8) node 1. 4 more to go.
Jan 1 19:16:55 kyorns051 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 2 (oce0) node 1. 3 more to go.
Jan 1 19:16:55 kyorns051 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (oce1) node 1. 3 more to go.
Jan 1 19:16:55 kyorns051 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 1 (oce8) node 1. 3 more to go.
Jan 1 19:16:55 kyorns051 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 2 (oce0) node 1. 2 more to go.
Jan 1 19:16:55 kyorns051 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (oce1) node 1. 2 more to go.
Jan 1 19:16:55 kyorns051 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 1 (oce8) node 1. 2 more to go.
Jan 1 19:16:55 kyorns051 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 2 (oce0) node 1. 1 more to go.
Jan 1 19:16:55 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 2 (oce0) node 1 inactive 118 sec (4110354)
Jan 1 19:16:56 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 0 (oce1) node 1 inactive 118 sec (8723349)
Jan 1 19:16:56 kyorns051 genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (oce8) node 1 inactive 118 sec (8723118)
Jan 1 19:16:56 kyorns051 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (oce1) node 1. 1 more to go.
Jan 1 19:16:56 kyorns051 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 1 (oce8) node 1. 1 more to go.
Jan 1 19:16:56 kyorns051 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 2 (oce0) node 1. 0 more to go.
Jan 1 19:16:56 kyorns051 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (oce1) node 1. 0 more to go.
Jan 1 19:16:56 kyorns051 genunix: [ID 592107 kern.notice] LLT INFO V-14-1-10510 sent hbreq (NULL) on link 1 (oce8) node 1. 0 more to go.
Jan 1 19:16:56 kyorns051 genunix: [ID 205468 kern.notice] LLT INFO V-14-1-10509 link 2 (oce0) node 1 expired
Jan 1 19:16:56 kyorns051 genunix: [ID 205468 kern.notice] LLT INFO V-14-1-10509 link 0 (oce1) node 1 expired
Jan 1 19:16:56 kyorns051 genunix: [ID 205468 kern.notice] LLT INFO V-14-1-10509 link 1 (oce8) node 1 expired
Jan 1 19:17:01 kyorns051 genunix: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port a gen 2a1503 membership 0
Jan 1 19:17:01 kyorns051 genunix: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port b gen 2a1505 membership 0
Jan 1 19:17:01 kyorns051 genunix: [ID 316943 kern.notice] GAB INFO V-15-1-20036 Port h gen 2a150b membership 0
Jan 1 19:17:01 kyorns051 Had[19421]: [ID 702911 daemon.notice] VCS INFO V-16-1-10077 Received new cluster membership
Jan 1 19:17:01 kyorns051 Had[19421]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10079 System kyorns052 (Node '1') is in Down State - Membership
: 0x1
Jan 1 19:17:01 kyorns051 genunix: [ID 330349 kern.notice] NOTICE: VXFEN INFO V-11-1-88 RACER Node won the race for the coordination points
Jan 1 19:17:01 kyorns051 genunix: [ID 776598 kern.notice] NOTICE: VXFEN INFO V-11-1-90 Sending WON_RACE
Jan 1 19:17:01 kyorns051 genunix: [ID 979517 kern.notice] NOTICE: VXFEN INFO V-11-1-67 call to VM ioctl VOL_CLEAR_PR returned non-zero
Jan 1 19:17:01 kyorns051 Had[19421]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10205 Group BkupLan is faulted on system kyorns052
Jan 1 19:17:01 kyorns051 Had[19421]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10205 Group DDCMon is faulted on system kyorns052
Jan 1 19:17:01 kyorns051 Had[19421]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10205 Group PrivLan is faulted on system kyorns052
Jan 1 19:17:01 kyorns051 Had[19421]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10205 Group PubLan is faulted on system kyorns052
Jan 1 19:17:01 kyorns051 Had[19421]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10205 Group StorLan is faulted on system kyorns052
Jan 1 19:17:01 kyorns051 Had[19421]: [ID 702911 daemon.notice] VCS ERROR V-16-1-10205 Group Sybase1 is faulted on system kyorns052
Jan 1 19:17:06 kyorns051 vxvm:vxconfigd: [ID 702911 daemon.notice] V-5-1-11401 : dg import with I/O fence enabled
Jan 1 19:17:06 kyorns051 vxvm:vxconfigd: [ID 702911 daemon.notice] V-5-1-11401 sybasedg: dg import with I/O fence enabled
Jan 1 19:17:06 kyorns051 vxvm:vxconfigd: [ID 702911 daemon.notice] V-5-1-0 Selecting configuration database copy from c0t5006016446E07160d18s2
from disks: c0t5006016446E07160d18s2 c0t5006016446E07160d16s2
Jan 1 19:17:06 kyorns051 vxvm:vxconfigd: [ID 702911 daemon.notice] V-5-1-0 Trying to import the disk group sybasedg using configuration databas
e copy from c0t5006016446E07160d18s2
Jan 1 19:17:07 kyorns051 vxvm:vxconfigd: [ID 702911 daemon.notice] V-5-1-0 Disk group import of sybasedg succeeded.
=================================================================================
2014/01/01 19:08:51 VCS ERROR V-16-2-13027 (kyorns051) Resource(private_lan) - monitor procedure did not complete within the expected time.
2014/01/01 19:13:02 VCS INFO V-16-1-10077 Received new cluster membership
2014/01/01 19:13:02 VCS NOTICE V-16-1-10112 System (kyorns051) - Membership: 0x1, DDNA: 0x2
2014/01/01 19:13:02 VCS ERROR V-16-1-10113 System kyorns052 (Node '1') is in DDNA Membership - Membership: 0x1, Visible: 0x0
2014/01/01 19:13:02 VCS ERROR V-16-1-10322 System kyorns052 (Node '1') changed state from RUNNING to FAULTED
2014/01/01 19:13:02 VCS NOTICE V-16-1-10449 Group BkupLan autodisabled on node kyorns052 until it is probed
2014/01/01 19:13:02 VCS NOTICE V-16-1-10449 Group DDCMon autodisabled on node kyorns052 until it is probed
2014/01/01 19:13:02 VCS NOTICE V-16-1-10449 Group Oss autodisabled on node kyorns052 until it is probed
2014/01/01 19:13:02 VCS NOTICE V-16-1-10449 Group Ossfs autodisabled on node kyorns052 until it is probed
2014/01/01 19:13:02 VCS NOTICE V-16-1-10449 Group PrivLan autodisabled on node kyorns052 until it is probed
2014/01/01 19:13:02 VCS NOTICE V-16-1-10449 Group PubLan autodisabled on node kyorns052 until it is probed
2014/01/01 19:13:02 VCS NOTICE V-16-1-10449 Group StorLan autodisabled on node kyorns052 until it is probed
2014/01/01 19:13:02 VCS NOTICE V-16-1-10449 Group Sybase1 autodisabled on node kyorns052 until it is probed
2014/01/01 19:13:02 VCS NOTICE V-16-1-10449 Group VCShmg autodisabled on node kyorns052 until it is probed
2014/01/01 19:13:02 VCS NOTICE V-16-1-10446 Group BkupLan is offline on system kyorns052
2014/01/01 19:13:02 VCS NOTICE V-16-1-10446 Group DDCMon is offline on system kyorns052
2014/01/01 19:13:02 VCS NOTICE V-16-1-10446 Group PrivLan is offline on system kyorns052
2014/01/01 19:13:02 VCS NOTICE V-16-1-10446 Group PubLan is offline on system kyorns052
2014/01/01 19:13:02 VCS NOTICE V-16-1-10446 Group StorLan is offline on system kyorns052
2014/01/01 19:13:02 VCS NOTICE V-16-1-10446 Group Sybase1 is offline on system kyorns052
2014/01/01 19:13:22 VCS ERROR V-16-2-13210 (kyorns051) Agent is calling clean for resource(private_lan) because 4 successive invocations of the
monitor procedure did not complete within the expected time.
2014/01/01 19:13:22 VCS INFO V-16-2-13068 (kyorns051) Resource(private_lan) - clean completed successfully.
2014/01/01 19:13:22 VCS ERROR V-16-2-13074 (kyorns051) The monitoring program for resource(private_lan) has consistently failed to determine the
resource status within the expected time. Agent is restarting (attempt number 1 of 1) the resource.
2014/01/01 19:13:22 VCS ERROR V-16-10001-87 (kyorns051) Monitor:private_lan:online:Failed to execute [/ericsson/core/cluster/scripts/private_lan
.sh start].
2014/01/01 19:13:23 VCS INFO V-16-2-13716 (kyorns051) Resource(private_lan): Output of the completed operation (online)
==============================================
[ "$1" != "monitor" ] && exit 100
+ '[' start '!=' monitor ']'
+ exit 100
==============================================
2014/01/01 19:16:28 VCS INFO V-16-2-13026 (kyorns051) Resource(private_lan) - monitor procedure finished successfully after failing to complete
within the expected time for (6) consecutive times.
2014/01/01 19:16:28 VCS NOTICE V-16-2-13076 (kyorns051) Agent has successfully restarted resource(private_lan).
2014/01/01 19:16:28 VCS INFO V-16-1-55031 Resource private_lan in online state received recurring online message on system kyorns051
2014/01/01 19:17:01 VCS INFO V-16-1-10077 Received new cluster membership
2014/01/01 19:17:01 VCS NOTICE V-16-1-10112 System (kyorns051) - Membership: 0x1, DDNA: 0x0
2014/01/01 19:17:01 VCS NOTICE V-16-1-10034 RECONFIG received. VCS waiting for I/O fencing to be completed
2014/01/01 19:17:01 VCS NOTICE V-16-1-10036 I/O fencing completed
2014/01/01 19:17:01 VCS ERROR V-16-1-10079 System kyorns052 (Node '1') is in Down State - Membership: 0x1
2014/01/01 19:17:01 VCS NOTICE V-16-1-10451 Cleared attribute-'autodisabled' for Group BkupLan on node kyorns052
2014/01/01 19:17:01 VCS NOTICE V-16-1-10451 Cleared attribute-'autodisabled' for Group DDCMon on node kyorns052
2014/01/01 19:17:01 VCS NOTICE V-16-1-10451 Cleared attribute-'autodisabled' for Group Oss on node kyorns052
2014/01/01 19:17:01 VCS NOTICE V-16-1-10451 Cleared attribute-'autodisabled' for Group Ossfs on node kyorns052
2014/01/01 19:17:01 VCS NOTICE V-16-1-10451 Cleared attribute-'autodisabled' for Group PrivLan on node kyorns052
2014/01/01 19:17:01 VCS NOTICE V-16-1-10451 Cleared attribute-'autodisabled' for Group PubLan on node kyorns052
2014/01/01 19:17:01 VCS NOTICE V-16-1-10451 Cleared attribute-'autodisabled' for Group StorLan on node kyorns052
2014/01/01 19:17:01 VCS NOTICE V-16-1-10451 Cleared attribute-'autodisabled' for Group Sybase1 on node kyorns052
2014/01/01 19:17:01 VCS NOTICE V-16-1-10451 Cleared attribute-'autodisabled' for Group VCShmg on node kyorns052
2014/01/01 19:17:01 VCS ERROR V-16-1-10205 Group BkupLan is faulted on system kyorns052
2014/01/01 19:17:01 VCS ERROR V-16-1-10205 Group DDCMon is faulted on system kyorns052
2014/01/01 19:17:01 VCS ERROR V-16-1-10205 Group PrivLan is faulted on system kyorns052
2014/01/01 19:17:01 VCS ERROR V-16-1-10205 Group PubLan is faulted on system kyorns052
2014/01/01 19:17:01 VCS ERROR V-16-1-10205 Group StorLan is faulted on system kyorns052
2014/01/01 19:17:01 VCS ERROR V-16-1-10205 Group Sybase1 is faulted on system kyorns052
2014/01/01 19:17:01 VCS NOTICE V-16-1-10446 Group BkupLan is offline on system kyorns052
2014/01/01 19:17:01 VCS NOTICE V-16-1-10446 Group DDCMon is offline on system kyorns052
2014/01/01 19:17:01 VCS NOTICE V-16-1-10446 Group PrivLan is offline on system kyorns052
2014/01/01 19:17:01 VCS NOTICE V-16-1-10446 Group PubLan is offline on system kyorns052
2014/01/01 19:17:01 VCS NOTICE V-16-1-10446 Group StorLan is offline on system kyorns052
2014/01/01 19:17:01 VCS NOTICE V-16-1-10446 Group Sybase1 is offline on system kyorns052
2014/01/01 19:17:01 VCS INFO V-16-1-10493 Evaluating kyorns051 as potential target node for group BkupLan
2014/01/01 19:17:01 VCS INFO V-16-1-50010 Group BkupLan is online or faulted on system kyorns051
2014/01/01 19:17:01 VCS INFO V-16-1-10493 Evaluating kyorns052 as potential target node for group BkupLan
2014/01/01 19:17:01 VCS INFO V-16-1-10494 System kyorns052 not in RUNNING state
2014/01/01 19:17:01 VCS INFO V-16-1-10493 Evaluating kyorns051 as potential target node for group DDCMon
2014/01/01 19:17:01 VCS INFO V-16-1-50010 Group DDCMon is online or faulted on system kyorns051
2014/01/01 19:17:01 VCS INFO V-16-1-10493 Evaluating kyorns052 as potential target node for group DDCMon
2014/01/01 19:17:01 VCS INFO V-16-1-10494 System kyorns052 not in RUNNING state
2014/01/01 19:17:01 VCS INFO V-16-1-10493 Evaluating kyorns051 as potential target node for group PrivLan
2014/01/01 19:17:01 VCS INFO V-16-1-50010 Group PrivLan is online or faulted on system kyorns051
2014/01/01 19:17:01 VCS INFO V-16-1-10493 Evaluating kyorns052 as potential target node for group PrivLan
2014/01/01 19:17:01 VCS INFO V-16-1-10494 System kyorns052 not in RUNNING state
2014/01/01 19:17:01 VCS INFO V-16-1-10493 Evaluating kyorns051 as potential target node for group PubLan
2014/01/01 19:17:01 VCS INFO V-16-1-50010 Group PubLan is online or faulted on system kyorns051
2014/01/01 19:17:01 VCS INFO V-16-1-10493 Evaluating kyorns052 as potential target node for group PubLan
2014/01/01 19:17:01 VCS INFO V-16-1-10494 System kyorns052 not in RUNNING state
2014/01/01 19:17:01 VCS INFO V-16-1-10493 Evaluating kyorns051 as potential target node for group StorLan
2014/01/01 19:17:01 VCS INFO V-16-1-50010 Group StorLan is online or faulted on system kyorns051
2014/01/01 19:17:01 VCS INFO V-16-1-10493 Evaluating kyorns052 as potential target node for group StorLan
2014/01/01 19:17:01 VCS INFO V-16-1-10494 System kyorns052 not in RUNNING state
2014/01/01 19:17:01 VCS INFO V-16-1-10493 Evaluating kyorns051 as potential target node for group Sybase1
2014/01/01 19:17:01 VCS INFO V-16-1-10493 Evaluating kyorns052 as potential target node for group Sybase1
2014/01/01 19:17:01 VCS INFO V-16-1-10494 System kyorns052 not in RUNNING state
2014/01/01 19:17:01 VCS INFO V-16-6-15025 (kyorns051) hatrigger:invoking nfs_preonline
2014/01/01 19:17:01 VCS INFO V-16-6-15076 (kyorns051) hatrigger:invoking regular preonline trigger if it exists
2014/01/01 19:17:01 VCS INFO V-16-6-0 (kyorns051) preonline:(preonline) Invoked with arg0=kyorns051, arg1=Sybase1
2014/01/01 19:17:01 VCS INFO V-16-6-0 (kyorns051) preonline:Executing /ericsson/core/cluster/scripts/preonline.sh with arg0=kyorns051, arg1=Syba
se1
2014/01/01 19:17:01 VCS INFO V-16-6-0 (kyorns051) preonline.sh:Sybase1:Nothing done
2014/01/01 19:17:01 VCS INFO V-16-6-0 (kyorns051) preonline:Completed execution of /ericsson/core/cluster/scripts/preonline.sh for group Sybase1
. Onlining group Sybase1 with -nopre option
2014/01/01 19:17:01 VCS INFO V-16-1-50135 User root fired command: hagrp -online Sybase1 kyorns051 from localhost
2014/01/01 19:17:01 VCS NOTICE V-16-1-10166 Initiating manual online of group Sybase1 on system kyorns051
2014/01/01 19:17:01 VCS NOTICE V-16-1-10460 Clearing start attribute for resource sybasedg of group Sybase1 on node kyorns052
2014/01/01 19:17:01 VCS NOTICE V-16-1-10460 Clearing start attribute for resource sybmaster_mount of group Sybase1 on node kyorns052
2014/01/01 19:17:01 VCS NOTICE V-16-1-10460 Clearing start attribute for resource syblog_mount of group Sybase1 on node kyorns052
2014/01/01 19:17:01 VCS NOTICE V-16-1-10460 Clearing start attribute for resource sybdata_mount of group Sybase1 on node kyorns052
2014/01/01 19:17:01 VCS NOTICE V-16-1-10460 Clearing start attribute for resource pmsyblog_mount of group Sybase1 on node kyorns052
2014/01/01 19:17:01 VCS NOTICE V-16-1-10460 Clearing start attribute for resource pmsybdata_mount of group