hi,

i wish to ask a couple of this all related.

i could have failover and parallel service groups in the same vcs.right?

in my opinion would be better if i setup a different vcs only for failover services and another for parallel services rather that a single vcs that comprises both failover and patrallel service groups.what is your opinion?

in an vcs spread accross 2 sites within same vlan,the failover service group should has the data in sync between sites and if i have an parallel service group with a node in a site and another in other site,than the dg will be made of mirror volumes (a submirror in one site and other in the other site).right?

if vcs start an a certain node and normally an certain service group start on this node,but in this scenario the service group runs on other vcs node,than the new node that joins will not start the service group.right?

tnx a lot.

hi,

if i have node a and b in site 1 and node c and d in site 2,then if i wish to configure them into global vcs in this way-nod a and node c serve a failover group and node b and d serve a parallel group.

i saw if i wish to setup gco first in site 1 i setup gco and then in site 2 i setup  the vcs cluster  and it will be joining next to the gco.

so in site 1 could i setup gco with node a and node c in server list of the failover group,even at this time the vcs in site 2 is not setup yet?

then could i setup node b and node d to server parallel group?it means that the disk group shared will be formed up by volumes mirrored accross the sites.or only i could have parallel service on 2 nodes within site 1?

tnx a lot.

hi,

i think that vxfencing could be setup only within evey cluster from every site.i am not aware if vxfencing could be setup as global fencing.suppose i have a node in site 1 and other node in site 2,and a failover group uses node1 or node2.if the heartbeat link between the node1 and node2 fails,then a split brain happens.then how this issue is solved not to corrupt data?

in my opinion the advantage of gco over classic disaster recovery is the speed of recovery,i mean the failover groups will failover fast on the other site if a site goes down.right?

tnx a lot.

hi,

i understand that a gco is a virtual(global) cluster formed from 2 vcs clusters each one in a separate site.the heartbeat between nodes from the sites goes over the wan?the gab traffic that updates all the nodes between the 2 sites,goes over this wan between sites?

Hi Guy.
I have a global cluster formed by 2 mini-cluster of one node only each one synchronized in asynchronous mode.

I wanted to simulate a fault of primary node with solaris command 'halt'

root@MILWB02S # hagrp -state AppService
#Group Attribute System Value
AppService State MILWB03SCluster:MILWB03S |OFFLINE|
AppService State localclus:MILWB02S |ONLINE|

after 'halt' on primary node MILWB02S, we have:

root@MILWB03S # hagrp -state AppService
#Group Attribute System Value
AppService State MILWB02SCluster:MILWB02S |OFFLINE|
AppService State localclus:MILWB03S |OFFLINE|

root@MILWB03S # hasys -state
#System Attribute Value
MILWB02SCluster:MILWB02S SysState EXITED
localclus:MILWB03S SysState RUNNING

root@MILWB03S # vradmin -g datadg repstatus datarvg
VxVM VVR vradmin INFO V-5-52-1205 Primary is unreachable or RDS has configuration error. Displayed status information is from Secondary and can be out-of-date.
Replicated Data Set: datarvg
Primary:
Host name: 10.66.28.53
RVG name: datarvg
DG name: datadg
RVG state: enabled for I/O
Data volumes: 1
VSets: 0
SRL name: srl_vol
SRL size: 1.00 G
Total secondaries: 1

Secondary:
Host name: 10.66.28.54
RVG name: datarvg
DG name: datadg
Data status: consistent, up-to-date
Replication status: paused due to network disconnection
Current mode: asynchronous
Logging to: SRL (0 updates behind, last update ID 5730.50511)
Timestamp Information: behind by 0h 0m 0s
Last Update on Primary: May 29 13:32:06
Secondary up-to-date as of: May 29 13:32:06

Config Errors:
10.66.28.53: Pri or Sec IP not available or vradmind not running, stale information

is this situation correct ?

I decided to manually start the service (AppService) on secondary node, because MILWB02S is down

root@MILWB03S # hagrp -online -force AppService -sys MILWB03S

root@MILWB03S # hagrp -state AppService
#Group Attribute System Value
AppService State MILWB02SCluster:MILWB02S |OFFLINE|
AppService State localclus:MILWB03S |ONLINE|

root@MILWB03S # vradmin -g datadg repstatus datarvg
Replicated Data Set: datarvg
Primary:
Host name: 10.66.28.54
RVG name: datarvg
DG name: datadg
RVG state: enabled for I/O
Data volumes: 1
VSets: 0
SRL name: srl_vol
SRL size: 1.00 G
Total secondaries: 1

Config Errors:
10.66.28.53: Pri or Sec IP not available or vradmind not running

after a lot of time, I booted the server down, and I noted a automatic switch of service from MILWB03S to MILWB02S

root@MILWB02S # hagrp -state AppService
#Group Attribute System Value
AppService State MILWB03SCluster:MILWB03S |OFFLINE|
AppService State localclus:MILWB02S |ONLINE|

root@MILWB02S # vradmin -g datadg repstatus datarvg
Replicated Data Set: datarvg
Primary:
Host name: 10.66.28.53
RVG name: datarvg
DG name: datadg
RVG state: enabled for I/O
Data volumes: 1
VSets: 0
SRL name: srl_vol
SRL size: 1.00 G
Total secondaries: 1

Config Errors:
10.66.28.54: Primary-Primary configuration

Is thi situation correct ? Why the cluster switched the service ?

Environment

OS = Linux 6x/7x

SFHA = Suppose latest i.e 6x

Query:

We have a two nodes local HA. A customized application running on it(HA) fine. The customized application send a tcp echo response (800) on the tcp echo request of (810) from a machine exist on LAN. I want to monitor this echo cycle under HA.

Hi ,

Can any one check this and help here to understand the root cause.

Please let me know if you need any other logs apart from below.

OS is : Suse linux

i tried clearing the resources but no luck.

ENGINE logs :

reason: user initiated abort
2015/06/16 10:17:33 VCS ERROR V-16-20006-1005 (atsfsx2526_01) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: user initiated abort
2015/06/16 10:22:33 VCS ERROR V-16-20006-1005 (atsfsx2526_01) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: user initiated abort
2015/06/16 10:27:32 VCS ERROR V-16-20006-1005 (atsfsx2526_01) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: user initiated abort
2015/06/16 10:32:33 VCS ERROR V-16-20006-1005 (atsfsx2526_01) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: user initiated abort
2015/06/16 10:37:33 VCS ERROR V-16-20006-1005 (atsfsx2526_01) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: user initiated abort
2015/06/16 10:42:33 VCS ERROR V-16-20006-1005 (atsfsx2526_01) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: user initiated abort
2015/06/16 10:47:33 VCS ERROR V-16-20006-1005 (atsfsx2526_01) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: user initiated abort
2015/06/16 10:52:32 VCS ERROR V-16-20006-1005 (atsfsx2526_01) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: user initiated abort
2015/06/16 10:57:33 VCS ERROR V-16-20006-1005 (atsfsx2526_01) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: user initiated abort
2015/06/16 11:02:33 VCS ERROR V-16-20006-1005 (atsfsx2526_01) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: user initiated abort
2015/06/16 11:07:33 VCS ERROR V-16-20006-1005 (atsfsx2526_01) CVMCluster:cvm_clus:monitor:node - state: out of cluster
reason: user initiated abort
"/var/VRTSvcs/log/engine_A.log" 67543L, 8334165C   

 vxclustadm nidmap
Name                             CVM Nid    CM Nid     State
atsfsx2526_01                    0          0          Out of Cluster
atsfsx2526_02                    1          1          Joined: Master

-- RESOURCES FAILED
-- Group           Type                 Resource             System

D  NFSgroup105     Share                ishare_105           atsfsx2526_02
D  NFSgroup105     Share                share_105            atsfsx2526_02
D  NFSgroup109     Share                ishare_109           atsfsx2526_02
D  NFSgroup109     Share                share_109            atsfsx2526_02
D  NFSgroup110     Share                ishare_110           atsfsx2526_02
D  NFSgroup110     Share                share_110            atsfsx2526_02
D  NFSgroup116     Share                share_116            atsfsx2526_02
D  NFSgroup120     Share                share_120            atsfsx2526_02
D  NFSgroup121     Share                share_121            atsfsx2526_02
D  NFSgroup231     Share                share_231            atsfsx2526_02
D  NFSgroup237     Share                share_237            atsfsx2526_02
D  NFSgroup241     Share                share_241            atsfsx2526_02
D  NFSgroup242     Share                share_242            atsfsx2526_02
D  cvm             CVMCluster           cvm_clus             atsfsx2526_01
D  vrts_vea_cfs_int_cfsmount11 CFSMount             cfsmount11           atsfsx2526_02
D  vrts_vea_cfs_int_cfsmount12 CFSMount             cfsmount12           atsfsx2526_02
D  vrts_vea_cfs_int_cfsmount7 CFSMount             cfsmount7            atsfsx2526_02

CVM logs :

2015/06/11 16:21:07 VCS ERROR V-16-2-13066 Thread(4141861744) Agent is calling clean for resource(cvmvoldg17) because the resource is not up even after online completed.
2015/06/11 16:21:07 VCS ERROR V-16-2-13066 Thread(4137651056) Agent is calling clean for resource(cvmvoldg19) because the resource is not up even after online completed.
2015/06/11 16:21:07 VCS ERROR V-16-2-13066 Thread(4139756400) Agent is calling clean for resource(cvmvoldg9) because the resource is not up even after online completed.
2015/06/11 16:21:07 VCS ERROR V-16-2-13066 Thread(4142914416) Agent is calling clean for resource(cvmvoldg1) because the resource is not up even after online completed.
2015/06/11 16:21:07 VCS ERROR V-16-2-13066 Thread(4143967088) Agent is calling clean for resource(cvmvoldg16) because the resource is not up even after online completed.
2015/06/11 16:21:07 VCS ERROR V-16-2-13066 Thread(4138703728) Agent is calling clean for resource(cvmvoldg4) because the resource is not up even after online completed.
2015/06/11 16:21:07 VCS ERROR V-16-2-13066 Thread(4140809072) Agent is calling clean for resource(cvmvoldg5) because the resource is not up even after online completed.

Regards ,

Satish

Hi,

I am following the veritas cluster server administrators guide for linux and trying to trigger the resnotoff script. From the documentation it is my understanding that is a resource faults and the clean command returns 1, resnotoff should be triggered.

To begin my service group is in an ONLINE state:

[root@node1 ~]# hastatus -sum | grep test                                       
B  Grp_CS_c1_testservice node1                Y          N               ONLINE
B  Grp_CS_c1_testservice node2                Y          N               ONLINE

I have the clean limit set to 1 and the clean script set to /bin/false to force this to return an error exit code.

Res_App_c1_fmmed1_testapplication ArgListValues         node1      User 1       root    StartProgram    1       "/usr/share/litp/vcs
_lsb_start vmservice 5" StopProgram     1       "/usr/share/litp/vcs_lsb_stop vmservice 5"      CleanProgram    1       /bin/false M
onitorProgram   1       "/usr/share/litp/vcs_lsb_status vmservice"      PidFiles        0       MonitorProcesses        0       EnvF
ile     1       ""      UseSUDash       1       0       State   1       2       IState  1       0                                   
Res_App_c1_fmmed1_testapplication ArgListValues         node2      User 1       root    StartProgram    1       "/usr/share/litp/vcs
_lsb_start vmservice 5" StopProgram     1       "/usr/share/litp/vcs_lsb_stop vmservice 5"      CleanProgram    1       /bin/false M
onitorProgram   1       "/usr/share/litp/vcs_lsb_status vmservice"      PidFiles        0       MonitorProcesses        0       EnvF
ile     1       ""      UseSUDash       1       0       State   1       2       IState  1       0                                   
Res_App_c1_fmmed1_testapplication CleanProgram          global     /bin/false                                                       
Res_App_c1_fmmed1_testapplication CleanRetryLimit       global     1                      
                                         

The resnotoff is enables for this resource

Res_App_c1_fmmed1_testapplication TriggersEnabled       global     RESNOTOFF

Now I manually kill the service Grp_CS_c1_testservice on node 1 and see the following in the /var/log/messages

Jun 16 17:02:33 node1 AgentFramework[10323]: VCS ERROR V-16-2-13067 Thread(4147325808) Agent is calling clean for resource(Res_App_c
1_fmmed1_testapplication) because the resource became OFFLINE unexpectedly, on its own.                       

Jun 16 17:02:33 node1 Had[9975]: VCS ERROR V-16-2-13067 (node1) Agent is calling clean for resource(Res_App_c1_fmmed1_testapplicatio
n) because the resource became OFFLINE unexpectedly, on its own.                                                                    
Jun 16 17:02:34 node1 AgentFramework[10323]: VCS ERROR V-16-2-13069 Thread(4147325808) Resource(Res_App_c1_fmmed1_testapplication) -
 clean failed.                                                                                                                      

and in the engine_A.log

2015/06/16 17:02:33 VCS ERROR V-16-2-13067 (node1) Agent is calling clean for resource(Res_App_c1_fmmed1_testapplication) because the resourc
e became OFFLINE unexpectedly, on its own.                                                                                                   
2015/06/16 17:02:34 VCS INFO V-16-10031-504 (node1) Application:Res_App_c1_fmmed1_testapplication:clean:Executed /bin/false as user root     
2015/06/16 17:02:35 VCS ERROR V-16-2-13069 (node1) Resource(Res_App_c1_fmmed1_testapplication) - clean failed.     

2015/06/16 17:03:35 VCS ERROR V-16-1-50148 ADMIN_WAIT flag set for resource Res_App_c1_fmmed1_testapplication on system node1 with the reason
 4                                                                                                                                           
2015/06/16 17:03:35 VCS INFO V-16-10031-504 (node1) Application:Res_App_c1_fmmed1_testapplication:clean:Executed /bin/false as user root     

From my understanding of the VCS adminisrator guide section titles 'VCS behavior when an online resource faults' the resnotoff should be triggered however it is not and the resource goes to an ADMIN WAIT state.

group           resource             system          message                  
--------------- -------------------- --------------- --------------------     
                Res_App_c1_fmmed1_testapplication node1           |ADMIN WAIT|

Is it possible to get the resnotoff triggered for a cluster in this state or do I need to use the resadminwait trigger (contrary to the documentation).

Thanks,

Under assessments => Install and Upgrade :

When I select SFHA as a product and select 6.2.1 as product version you are installing to , I cannot see RHEL7.

But

When I select SFHA as a product , select 6.2 as product version you are installing to and platform RHEL7 then under Platform Configuration I can see 6.2.1 supported RHEL7 GA. what is this ?

hi,

if an application hangs and it is part of an service group,how i troubleshoot things?

first should i stop the service group hagrp -offline?if this not works then should i try to stop each resource hares -offline?if this not wotks then i kill -9 every process of this application.

should i take out the floating ip?and unmount if trhe above not work?

then i start everything with hagrp -online?

normally the application resides on SAN luns not on local disk from server.right?

in this case if there are some issues with dg,volumes or plexes and i freeze this group,then the application could hang.right?

tnx a lot.

Hi,

I want to create MySQL cluster using storage foundation cluster file system in RHEL environment. Is there anybody who has worked on Mysql clustering using SFCFS. I have below queries regarding mysql clustering.

- Which version should we use of MySQL i.e. Mysql community server or Mysql cluster server

- Did anybody has any installation of Mysql and cluster configuration steps.

hi,

if i have 7 nodes in vcs,node a to node e serve 4 service failover groups and node f and g serve an orac.If suddenly i have a split brain case,all the nodes lose the llt conection,what is the fastest and secure way to deal with this?

sure i should to shutdown nodes but i need the sequence.

i will start with shutdown node f or g.then i will shutdown 4 from the nodes a to e.let`s say i shutdown node b to e.i start then  node b and see what service group in online on it.then node,then node e.i make sure on every node a to e there is only a single group online.

tnbx a lot.

Hi there,

I have a system where the cleanup script can fail/timeout and I want to execute another script if this happens. And I was wondering which can be the best way of doing this.

In the veritas cluster server administrators guide for Linux I found the trigger RESNOTOFF.

From the documentation it is my understanding that this trigger will be triggered in the following cases:

• A resource fails going offline (started by VCS) and the clean up fails.
• A resource goes offline unexpectedly and the clean up fails.

I have tested this and the RESNOTOFF is working in the first scenario but not in the second.

For testing the second scenario I kill the service and I can see the following message in the engine_A.log:

VCS ERROR V-16-2-13067 (node1) Agent is calling clean for resource(service1) because the resource became OFFLINE unexpectedly, on its own.

When the cleanup fails I would expect the resource to became UNABLE TO OFFLINE. However, the status of the resource is still ONLINE:

# hares -state service1
#Resource                    Attribute             System     Value
service1                         State                 node1      ONLINE
service1                         State                 node2      OFFLINE
 

So the resource is ONLINE and VCS keeps running the cleanup command indefinitely (which is failing).

I was wondering if I need to configure something else to make the RESNOTOFF to work in this particular scenario.

Thanks,

Hi,

I want to manage a CIFS mount within a Service Group and cannot find the appropriate resource.

I just one to mount a CIFS share in my VCS cluster, acting as a client.

Can anyone let me know which Resource type should I use?

Thanks & Regards,

JL

Hello,

We have configured the sybase database  instance under VCS,  but binaries are installed on shared drive as well.  When it comes to sybase licence, database team generated using hostid.  If we want to share the sybase licence between both nodes,  is there an option?

Hi all,

Our environment as the following:
OS: redhat 6.5
VCS: VCS 6.2.1

Our server have two physical network port, namely eth0 and eth1.
We do create tagged vlan, vlan515, vlan516, vlan518, vlan520 based on eth0 and eth1.

We are able to create resource IP on vlan518 and failover between two nodes.

However, when we create resource IP on vlan515, it is not able to bring it online.

According to the link, https://support.symantec.com/en_US/article.TECH214469.html,
It knows that duplicate mac address would cause the problem.
However, it can't figure out where "MACAddress" attribute in VCS Java Console as mentioned in the solution.

I did manually add "MACAddress" attribute on main.cf on either NIC or IP resource, it come with not support with haconf -verify command.

Any hints or solution for the problem when configure the IP agent resource on same mac address?

Thanks,

Xentar

Hi,

We have 2 Storage Foundation for Oracle RAC 5.1SP1 servers running on AIX 5.3 Servers. We have Application Service Group and Database Service Group  which is both active active. I encountered a problem when i initiated manual offline of the Service group. Application Service Group is ok. But in the Database Service Group it failed and the 2 servers unpectedly restarted and i encountered split-brain condition.

Here's the engineA.log:

ServiceGroup_PRD Application successfull offiline

2015/06/24 01:17:14 VCS INFO V-16-1-50135 User root fired command: hagrp -offline ServiceGroup_PRD  Server1  from localhost

2015/06/24 01:17:14 VCS NOTICE V-16-1-10167 Initiating manual offline of group SERVICEGROUP_PRD on system SERVER1

2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource archive2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource backup2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource redolog5_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource redolog6_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u12_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u13_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u14_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:14 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u15_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:17 VCS INFO V-16-1-10305 Resource redolog5_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:17 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgredolog5 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:19 VCS INFO V-16-1-10305 Resource u12_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:19 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu12 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:19 VCS INFO V-16-1-10305 Resource redolog6_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:19 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgredolog6 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:20 VCS INFO V-16-1-10305 Resource oradgredolog5 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:20 VCS INFO V-16-1-10305 Resource u14_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:20 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu14 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:21 VCS INFO V-16-1-10305 Resource oradgu12 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:21 VCS INFO V-16-1-10305 Resource u15_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:21 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu15 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:21 VCS INFO V-16-1-10305 Resource oradgredolog6 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:21 VCS INFO V-16-1-10305 Resource oradgu14 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:21 VCS INFO V-16-1-10305 Resource u13_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:21 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu13 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:24 VCS INFO V-16-1-10305 Resource oradgu15 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:24 VCS INFO V-16-1-10305 Resource oradgu13 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:32 VCS INFO V-16-1-10305 Resource archive2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:32 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgarchive2 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:32 VCS INFO V-16-1-10305 Resource backup2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:32 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgbackup2 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER1

2015/06/24 01:17:33 VCS INFO V-16-1-10305 Resource oradgarchive2 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:34 VCS INFO V-16-1-10305 Resource oradgbackup2 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:17:35 VCS NOTICE V-16-1-10446 Group SERVICEGROUP_PRD is offline on system SERVER1

2015/06/24 01:17:35 VCS INFO V-16-6-15002 (SERVER1) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/lvmvg_postoffline SERVER1 SERVICEGROUP_PRD   successfully

2015/06/24 01:17:35 VCS INFO V-16-6-0 (SERVER1) postoffline:Invoked with arg0=SERVER1, arg1=SERVICEGROUP_PRD

2015/06/24 01:17:36 VCS INFO V-16-6-15002 (SERVER1) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/postoffline SERVER1 SERVICEGROUP_PRD   successfully

2015/06/24 01:17:41 VCS NOTICE V-16-1-10167 Initiating manual offline of group SERVICEGROUP_PRD on system SERVER2

2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource archive2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource backup2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource redolog5_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource redolog6_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u12_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u13_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u14_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:41 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u15_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:43 VCS INFO V-16-1-10305 Resource redolog5_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:43 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgredolog5 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:43 VCS INFO V-16-1-10305 Resource archive2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:43 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgarchive2 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:43 VCS INFO V-16-1-10305 Resource backup2_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:43 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgbackup2 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:43 VCS INFO V-16-1-10305 Resource u12_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:43 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu12 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:44 VCS INFO V-16-1-10305 Resource u13_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:44 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu13 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:44 VCS INFO V-16-1-10305 Resource oradgredolog5 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:44 VCS INFO V-16-1-10305 Resource u14_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:44 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu14 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:44 VCS INFO V-16-1-10305 Resource u15_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:44 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgu15 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:44 VCS INFO V-16-1-10305 Resource redolog6_mnt (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:44 VCS NOTICE V-16-1-10300 Initiating Offline of Resource oradgredolog6 (Owner: unknown, Group: SERVICEGROUP_PRD) on System SERVER2

2015/06/24 01:17:45 VCS INFO V-16-1-10305 Resource oradgbackup2 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:45 VCS INFO V-16-1-10305 Resource oradgarchive2 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:46 VCS INFO V-16-1-10305 Resource oradgu12 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:46 VCS INFO V-16-1-10305 Resource oradgu14 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:46 VCS INFO V-16-1-10305 Resource oradgu13 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:47 VCS INFO V-16-1-10305 Resource oradgu15 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:47 VCS INFO V-16-1-10305 Resource oradgredolog6 (Owner: unknown, Group: SERVICEGROUP_PRD) is offline on SERVER2 (VCS initiated)

2015/06/24 01:17:47 VCS NOTICE V-16-1-10446 Group SERVICEGROUP_PRD is offline on system SERVER2

2015/06/24 01:17:47 VCS INFO V-16-6-15002 (SERVER2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/lvmvg_postoffline SERVER2 SERVICEGROUP_PRD   successfully

2015/06/24 01:17:47 VCS INFO V-16-6-0 (SERVER2) postoffline:Invoked with arg0=SERVER2, arg1=SERVICEGROUP_PRD2015/06/24 01:17:48 VCS INFO V-16-6-15002 (SERVER2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/postoffline SERVER2 SERVICEGROUP_PRD successfully

ServiceGroup_DBPRD Database fails to offline

2015/06/24 01:18:51 VCS INFO V-16-1-50135 User root fired command: hagrp -offline SERVICEGROUP_DBPRD  SERVER1  from localhost

2015/06/24 01:18:51 VCS NOTICE V-16-1-10167 Initiating manual offline of group SERVICEGROUP_DBPRD on system SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource archive_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource backup_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource ocr_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource quorum_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource redolog1_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource redolog2_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u02_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u03_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u04_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u05_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u06_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource u07_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:51 VCS NOTICE V-16-1-10300 Initiating Offline of Resource voting_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:52 VCS ERROR V-16-20011-5503 (SERVER1) CFSMount:redolog1_mnt:offline:Umount Failed : Mount Point : /redolog1

2015/06/24 01:18:52 VCS ERROR V-16-20011-5503 (SERVER1) CFSMount:quorum_mnt:offline:Umount Failed : Mount Point : /quorum

2015/06/24 01:18:52 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:redolog1_mnt:offline:Attempting fuser TERM : Mount Point : /redolog1

2015/06/24 01:18:52 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:quorum_mnt:offline:Attempting fuser TERM : Mount Point : /quorum

2015/06/24 01:18:53 VCS INFO V-16-1-10305 Resource ocr_mnt (Owner: unknown, Group: SERVICEGROUP_DBPRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:18:53 VCS NOTICE V-16-1-10300 Initiating Offline of Resource cvmocr (Owner: unknown, Group: SERVICEGROUP_DBPRD) on System SERVER1

2015/06/24 01:18:53 VCS ERROR V-16-20011-5503 (SERVER1) CFSMount:redolog2_mnt:offline:Umount Failed : Mount Point : /redolog2

2015/06/24 01:18:53 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:redolog2_mnt:offline:Attempting fuser TERM : Mount Point : /redolog2

2015/06/24 01:18:55 VCS INFO V-16-1-10305 Resource cvmocr (Owner: unknown, Group: SERVICEGROUP_DBPRD) is offline on SERVER1 (VCS initiated)

2015/06/24 01:18:55 VCS ERROR V-16-20011-5503 (SERVER1) CFSMount:u03_mnt:offline:Umount Failed : Mount Point : /u03

2015/06/24 01:18:55 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:u03_mnt:offline:Attempting fuser TERM : Mount Point : /u03

2015/06/24 01:18:57 VCS ERROR V-16-20011-5503 (SERVER1) CFSMount:u02_mnt:offline:Umount Failed : Mount Point : /u02

2015/06/24 01:18:57 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:u02_mnt:offline:Attempting fuser TERM : Mount Point : /u02

2015/06/24 01:18:58 VCS ERROR V-16-20011-5503 (SERVER1) CFSMount:u04_mnt:offline:Umount Failed : Mount Point : /u04

2015/06/24 01:18:58 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:u04_mnt:offline:Attempting fuser TERM : Mount Point : /u04

2015/06/24 01:19:03 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:redolog1_mnt:offline:Attempting fuser TERM : Mount Point : /redolog1

2015/06/24 01:19:03 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:redolog2_mnt:offline:Attempting fuser TERM : Mount Point : /redolog2

2015/06/24 01:19:03 VCS NOTICE V-16-20011-5510 (SERVER1) CFSMount:quorum_mnt:offline:Attempting fuser TERM : Mount Point : /quorum

2015/06/24 01:40:15 VCS NOTICE V-16-1-11022 VCS engine (had) started

2015/06/24 01:40:15 VCS INFO V-16-1-10196 Cluster logger started

2015/06/24 01:40:15 VCS NOTICE V-16-1-11050 VCS engine version=5.1

2015/06/24 01:40:15 VCS NOTICE V-16-1-11051 VCS engine join version=5.1.00.0

2015/06/24 01:40:15 VCS NOTICE V-16-1-11052 VCS engine pstamp=Veritas-5.1-10/06/09-14:37:00

2015/06/24 01:40:15 VCS NOTICE V-16-1-10114 Opening GAB library

2015/06/24 01:40:16 VCS NOTICE V-16-1-10619 'HAD' starting on: SERVER1

2015/06/24 01:40:16 VCS INFO V-16-1-10125 GAB timeout set to 30000 ms

2015/06/24 01:40:16 VCS NOTICE V-16-1-11057 GAB registration monitoring timeout set to 200000 ms

2015/06/24 01:40:16 VCS NOTICE V-16-1-11059 GAB registration monitoring action set to log system message

2015/06/24 01:40:26 VCS INFO V-16-1-10077 Received new cluster membership

2015/06/24 01:40:26 VCS NOTICE V-16-1-10112 System (SERVER1) - Membership: 0x3, DDNA: 0x0

2015/06/24 01:40:26 VCS NOTICE V-16-1-10322 System  (Node '1') changed state from UNKNOWN to INITING

2015/06/24 01:40:26 VCS NOTICE V-16-1-10086 System SERVER1 (Node '0') is in Regular Membership - Membership: 0x3

2015/06/24 01:40:26 VCS NOTICE V-16-1-10086 System  (Node '1') is in Regular Membership - Membership: 0x3

2015/06/24 01:40:26 VCS NOTICE V-16-1-10453 Node: 1 changed name from: '' to: 'SERVER2'

2015/06/24 01:40:26 VCS NOTICE V-16-1-10322 System SERVER2 (Node '1') changed state from INITING to CURRENT_DISCOVER_WAIT

2015/06/24 01:40:26 VCS NOTICE V-16-1-10322 System SERVER1 (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD

2015/06/24 01:40:26 VCS NOTICE V-16-1-10322 System SERVER2 (Node '1') changed state from CURRENT_DISCOVER_WAIT to CURRENT_PEER_WAIT

2015/06/24 01:40:28 VCS NOTICE V-16-1-52006 UseFence=SCSI3. Fencing is enabled

2015/06/24 01:40:28 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...

2015/06/24 01:40:43 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...

2015/06/24 01:40:58 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...

2015/06/24 01:41:13 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...

2015/06/24 01:41:28 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...

2015/06/24 01:41:43 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...

2015/06/24 01:41:58 VCS CRITICAL V-16-1-10031 VxFEN driver not configured. VCS Stopping. Manually restart VCS after configuring fencing

The solution i did here was to run vxfenclearpre.

Is there any logs i need to check why the servers faults and restarted unexpectedly after service group offline?

Any help will appreciate.

Thanks.

heartbeat MAC address not identical after VCS fresh install on solaris Logical domain virtual server(T4-5),.Does any body has idea ?

I was able to set the MAC addressin LLT , but its getting changed after every server reboot,

Node A

 # lltstat -nvv | head -10

LLT node information:
    Node                 State    Link  Status  Address
   *  0 node a            OPEN
                                  vnet2   UP      00:14:4F:F8:3B:B7
                                  vnet3   UP      00:14:4F:F9:73:DC   --
                                  vnet1   UP      00:14:4F:F8:F5:AC   ---
    1 node b          OPEN
                                  vnet2   UP      00:14:4F:FA:2B:77
                                  vnet3   UP      00:14:4F:FB:5E:07  --
                                  vnet1   UP      00:14:4F:F9:E0:17   --

Node B

 # lltstat -nvv | head -10

LLT node information:
    Node                 State    Link  Status  Address
    0 node a        OPEN
                                  vnet2   UP      00:14:4F:F8:3B:B7
                                  vnet3   UP      00:14:4F:F9:73:DC   --
                                  vnet1   UP      00:14:4F:F8:F5:AC    ---
     * 1 node b         OPEN
                                  vnet2   UP      00:14:4F:FA:2B:77
                                  vnet3   UP      00:14:4F:F9:E0:17   ---
                                  vnet1   UP      00:14:4F:FB:5E:07  ----

I got above post VCS instllation, So I followed below steps to change the MAC address for Vnet1 and Vnet3

gabconfig -U
svcadm disable svc:/system/gab:default

lltconfig -k disable
svcadm disable svc:/system/llt:default

svcadm enable svc:/system/llt:default
lltconfig -k enable

svcadm enable svc:/system/gab:default
/sbin/gabconfig -c -n2

It has changed here, 

Node A: 

 # lltstat -nvv | head -10
LLT node information:
    Node                 State    Link  Status  Address
   * 0 node a     OPEN
                                  vnet2   UP      00:14:4F:F8:3B:B7
                                  vnet3   UP      00:14:4F:F9:73:DC
                                  vnet1   UP      00:14:4F:F8:F5:AC
     1 node b    OPEN
                                  vnet2   UP      00:14:4F:FA:2B:77
                                  vnet3   UP      00:14:4F:FB:5E:07
                                  vnet1   UP      00:14:4F:F9:E0:17
Node B

 # lltstat -nvv | head -10

LLT node information:
    Node                 State    Link  Status  Address
     0 node a          OPEN
                                  vnet2   UP      00:14:4F:F8:3B:B7
                                  vnet3   UP      00:14:4F:F9:73:DC
                                  vnet1   UP      00:14:4F:F8:F5:AC
   * 1 node b      OPEN
                                  vnet2   UP      00:14:4F:FA:2B:77
                                  vnet3   UP      00:14:4F:FB:5E:07
                                  vnet1   UP      00:14:4F:F9:E0:17

but after server reboot went to orginal state of non-sync mode. where Vnet1's MAC and Vnet3's MAC changed.

Any body know how the setting happens? how it is fetching and from where? how to make it permanent?

Hi Guys,

I configured VCS Clustered on single node and my requirement if resaouce going offline or terminated VCS should try to online it.

i set the resfault file but no luck please help.