We are running VCS 6.0.2 on RHEL 6.5, below is our cluster configuration:

Heartbeat link: eth3, eth4
Low-priority heartbeat link: not be enabled
Fencing: not be enabled
Cluster contains 2 servers: jarry-crf1, jarry-crf2.

Server groups:
   TestGrp1 contains a "FileOnOff" resource and Parallel mode is enabled.
   TestGrp2 contains a "FileOnOff" resource, depends on TestGrp1 and Failover mode is enabled.

Test steps:

1. Take TestGrp1 online on both server, take TestGrp2 online on server "jarry-crf2"
2. Stop both heartbeat links on server "jarry-crf1" by command "ifdown eth3; sleep 60; ifdown eth4"
3. Recover heartbeat links by command "ifup eth3; ifup eth4"

Then we found the "had" process is restared on server "jarry-crf2", and we found below logs in engine_A.log

2015/02/10 00:48:43 VCS NOTICE V-16-1-10433 Group TestGrp2 will not start automatically on System jarry-crf2 as the system is in restart mode.
2015/02/10 00:48:43 VCS NOTICE V-16-1-10433 Group TestGrp1 will not start automatically on System jarry-crf2 as the system is in restart mode.
2015/02/10 00:48:43 VCS NOTICE V-16-1-10445 Group TestGrp1 will not start automatically as atleast one system in the SystemList attribute of the group is in restart mode.
2015/02/10 00:48:47 VCS NOTICE V-16-1-10433 Group VCShmg will not start automatically on System jarry-crf2 as the system is in restart mode.
2015/02/10 00:48:47 VCS NOTICE V-16-1-10445 Group VCShmg will not start automatically as atleast one system in the SystemList attribute of the group is in restart mode.

My Questions:

1. Why "had" process is restarted after heartbeat being recovered.
2. What's means of "restart mode", how to bring service group leave "restart mode" and start automatically.

Thanks in advance!

Hello Experts,

we are experincing issue in which one of the cluster nodes hangs .

we need to poweroff and poweron the server to recover it.

However we noticed, there is no failover on one of the nodes geeting hand and this impacts the service.

Please suggest what needs to be corrected so that there should be a failover if one node hangs

Regards

S

Hello,

on 2 servers OS Linux Red Hat 6.3, I've got a VRTS 6.0

For on Application, when I put it offline and then online on the same server (with all other resources online on this server) it start well.

But when I test a 'switch to' of all the service group on the other node it doesn't start properly (and all the other resources started )

The application is link with 3 Mount ressources and 1 IP ressource

we set attributes critical to false, and we set UseSUDash to true.

The StartProgram script is supposed to start several processes, with a offline, online action all the processes are started, with a 'switch to' action, only half of them are started.

No interesting log in the Application side.

Any suggestion to debug will be appreciated.

Hi All,

We have a veritas cluster server 5.7 running on SLES 11 OS, here i am facing an issue while giving cluster reboot to node 1, the NFSgroup does not come online automatically after reboot. But after running nfs server start. all the NFSgroups are online. Logs attached here:

2015/02/10 11:51:15 VCS INFO V-16-1-10077 Received new cluster membership
2015/02/10 11:51:15 VCS NOTICE V-16-1-10112 System (vdel2sfs_01) - Membership: 0
x1, DDNA: 0x2
2015/02/10 11:51:15 VCS ERROR V-16-1-10113 System vdel2sfs_02 (Node '1') is in D
DNA Membership - Membership: 0x1, Visible: 0x0
2015/02/10 11:51:22 VCS WARNING V-16-1-11141 LLT heartbeat link status changed. 
Previous status = priveth0 DOWN priveth1 UP; Current status = priveth0 UP privet
h1 UP.
2015/02/10 11:52:23 VCS INFO V-16-1-10077 Received new cluster membership
2015/02/10 11:52:23 VCS NOTICE V-16-1-10112 System (vdel2sfs_01) - Membership: 0
x3, DDNA: 0x0
2015/02/10 11:52:23 VCS NOTICE V-16-1-10322 System  (Node '1') changed state fro
m UNKNOWN to INITING
2015/02/10 11:52:23 VCS NOTICE V-16-1-10086 System vdel2sfs_02 (Node '1') is in 
Regular Membership - Membership: 0x3
--More--(85%)
2015/02/10 11:52:23 VCS NOTICE V-16-1-10453 Node: 1 changed name from: 'vdel2sfs
_02' to: 'vdel2sfs_02'
2015/02/10 11:52:23 VCS NOTICE V-16-1-10322 System vdel2sfs_02 (Node '1') change
d state from EXITED to INITING
2015/02/10 11:52:23 VCS NOTICE V-16-1-10322 System vdel2sfs_02 (Node '1') change
d state from INITING to CURRENT_DISCOVER_WAIT
2015/02/10 11:52:23 VCS NOTICE V-16-1-10322 System vdel2sfs_02 (Node '1') change
d state from CURRENT_DISCOVER_WAIT to REMOTE_BUILD
2015/02/10 11:52:23 VCS INFO V-16-1-10455 Sending snapshot to node membership: 0
x2
2015/02/10 11:52:25 VCS NOTICE V-16-1-10322 System vdel2sfs_02 (Node '1') change
d state from REMOTE_BUILD to RUNNING
2015/02/10 11:52:27 VCS INFO V-16-6-15023 (vdel2sfs_02) dump_tunables:
########## VCS Environment Variables ##########
VCS_CONF=/etc/VRTSvcs
VCS_DIAG=/var/VRTSvcs
VCS_GAB_RMACTION=panic
VCS_HOME=/opt/VRTSvcs
VCS_LOG=/var/VRTSvcs
VCS_LOG_AGENT_NAME=
VCS_LOG_CATEGORY=6
VCS_LOG_SCRIPT_NAME=hatrigger
########## Other Environment Variables ##########
--More--(85%)
COLUMNS=96
CONSOLE=/dev/console
DO_CONFIRM=
EAT_CATALOG_DIR=/opt/VRTSvcs
EAT_HOME_DIR=/opt/VRTSvcs
HOME=/
INIT_VERSION=sysvinit-2.86
LANG=
LC_ALL=POSIX
LD_LIBRARY_PATH=/opt/VRTSvcs/lib:
LINES=33
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/bin:/sbin:/usr
/bin:/usr/sbin:/opt/VRTSvcs/bin
PREVLEVEL=N
PWD=/var/VRTSvcs/diag/had
REDIRECT=/dev/tty1
ROOTFS_BLKDEV=/dev/disk/by-label/root18593
ROOTFS_FSCK=0
ROOTFS_FSTYPE=ext3
RUNLEVEL=3
SELINUX_INIT=YES
SHELL=/bin/sh
SHLVL=5
--More--(85%)
SPLASH=yes
SPLASHCFG=/etc/bootsplash/themes/SLES/config/bootsplash-800x600.cfg
TERM=linux
VCSONE_LOG_AGENT_NAME=
VCSONE_LOG_CATEGORY=6
VCSONE_LOG_SCRIPT_NAME=hatrigger
_=/usr/bin/env
crashkernel=1G-8G:256M,8G-:512M
splash=silent
vga=0x314

2015/02/10 11:52:27 VCS INFO V-16-6-15002 (vdel2sfs_02) hatrigger:hatrigger exec
uted /opt/VRTSvcs/bin/triggers/dump_tunables vdel2sfs_02 1   successfully
2015/02/10 11:52:27 VCS WARNING V-16-10031-7017 (vdel2sfs_02) NFS:nasgw_nfs:moni
tor:nfsd filesystem not mounted, returning offline
2015/02/10 11:52:27 VCS ERROR V-16-20006-1005 (vdel2sfs_02) CVMCluster:cvm_clus:
monitor:node - state: out of cluster
2015/02/10 11:52:27 VCS INFO V-16-10031-20902 (vdel2sfs_02) CVMVxconfigd:CVMVxco
nfigd:imf_init:pid is 2257
2015/02/10 11:52:27 VCS INFO V-16-10031-20902 (vdel2sfs_02) CFSfsckd:CFSfsckd:im
f_init:pid is 1625
2015/02/10 11:52:27 VCS INFO V-16-10031-20903 (vdel2sfs_02) CFSfsckd:CFSfsckd:im
f_init:/opt/VRTSamf/bin/amfinit -i -rCFSfsckd 1625
--More--(85%)
2015/02/10 11:52:27 VCS INFO V-16-10031-20903 (vdel2sfs_02) CVMVxconfigd:CVMVxco
nfigd:imf_init:/opt/VRTSamf/bin/amfinit -i -rCVMVxconfigd 2257

...

2015/02/10 11:52:27 VCS INFO V-16-1-10304 Resource VIP10 (Owner: Unspecified, Gr
oup: VIPgroup10) is offline on vdel2sfs_02 (First probe)
2015/02/10 11:52:27 VCS INFO V-16-1-10304 Resource phantomproc_canHostConsole (O
wner: Unspecified, Group: CanHostConsole) is offline on vdel2sfs_02 (First probe
)

...

2015/02/10 11:52:28 VCS INFO V-16-1-10304 Resource cvm_clus (Owner: Unspecified,
 Group: cvm) is offline on vdel2sfs_02 (First probe)
2015/02/10 11:52:28 VCS INFO V-16-1-10304 Resource nasgw_nfs (Owner: Unspecified
, Group: NFS) is offline on vdel2sfs_02 (First probe)
2015/02/10 11:52:28 VCS INFO V-16-1-10304 Resource Samba_netbios (Owner: Unspeci
fied, Group: CIFS) is offline on vdel2sfs_02 (First probe)
2015/02/10 11:52:28 VCS INFO V-16-1-10304 Resource SambaServerResource (Owner: U
nspecified, Group: CIFS) is offline on vdel2sfs_02 (First probe)
2015/02/10 11:52:28 VCS NOTICE V-16-1-10438 Group CIFS has been probed on system
 vdel2sfs_02
2015/02/10 11:52:28 VCS NOTICE V-16-1-10435 Group CIFS will not start automatica
lly on System vdel2sfs_02 as the system is not a part of AutoStartList attribute
 of the group.
2015/02/10 11:52:28 VCS INFO V-16-1-10304 Resource nasgw_nfsrestart (Owner: Unsp
ecified, Group: NFS) is offline on vdel2sfs_02 (First probe)

..

2015/02/10 11:52:36 VCS INFO V-16-1-10304 Resource share_247 (Owner: Unspecified
--More--(86%)
, Group: NFSgroup247) is offline on vdel2sfs_02 (First probe)
2015/02/10 11:52:36 VCS INFO V-16-1-10304 Resource ishare_247 (Owner: Unspecifie
d, Group: NFSgroup247) is offline on vdel2sfs_02 (First probe)
2015/02/10 11:52:36 VCS NOTICE V-16-1-10438 Group NFSgroup247 has been probed on
 system vdel2sfs_02
2015/02/10 11:52:36 VCS NOTICE V-16-1-10435 Group NFSgroup247 will not start aut
omatically on System vdel2sfs_02 as the system is not a part of AutoStartList at
tribute of the group.

..

2015/02/10 11:53:30 VCS ERROR V-16-10031-11511 (vdel2sfs_02) Share:share_324:onl
ine:/vx/smrs-gran_vdel2ss1x is not a directory, cannot online the resource.
2015/02/10 11:53:31 VCS INFO V-16-1-10298 Resource nasgw_nfsrestart (Owner: Unsp
ecified, Group: NFS) is online on vdel2sfs_02 (VCS initiated)
2015/02/10 11:53:31 VCS WARNING V-16-10031-0 (vdel2sfs_02) NFS:nasgw_nfs:online:
FILESTORE: NFS modules will not be unloaded.

English follows Japanese.

■問い合わせ内容
Server2台でクラスタを構成している。
仮にServer2をOS shutdownした状態ではVCSサービスは継続する。
この状況下でServer1がもしOS不具合でOSリスタートもしくは手動でOS停止してしまった場合に
Server1がOS起動完了してもVCSのサービスが起動しない。

この状況下でServer1のみでも各サービスグループを起動する(ONLINE化)方法、手順を知りたい。
※Server1でhastart -onenodeコマンドを実行して見たがサービス起動せず。

- Question about Verital server
Cluster server1 and server2 are running. I would like to know how to contiure VCS service in case of one server.

1. When one of server2 is down, but VCS service is still continue.
2. The rest of server1 restaerted, or rebooted when 1.
3. After Server1 boot up and OS restarted. But VCS service cnannot start.

We tried to execute "hastart -onenode" on Server1. However, it does not effect.

Hi,

I have rebooted a cluster with CPS vxfencing. 

there is no SAN disks mapped to the cluster. However, "vxfenadm -d" shows 2 out of 4 nodes is in (replaying state) for a long time. i had to do

hastop -all (properly on the nodes with the replay status would do)

then hastart again

it seems like a timing. has anyone out there seen this before?

Thanks in advance.

Hi

My intention is to remove a SG and it’s resources associated with that, but when I try hagrp delete <service group> it’s giving below warning, but same I could remove as a cluster admin account in GUI. Any difference between CLI and GUI?

hagrp -delete <SG name>

VCS WARNING V-16-1-10137 Resources exist; cannot delete group

I am planning to setup alerts for my application to notify users upon faults.

I understand that i need to configure a NotifierMngr resourse under ClusterService. Please answer following

questions that i have for my understanding.

1) Should I configure a new ClusterService group or use the existing one. Currently I have following resources in my existing ClusterService group.

# hagrp -resources ClusterService
webip
csgnic

Wouldnt be a risk if Notifier resource is configured in this group itself?

If i need to configure a seperate a ClusterService, could you please guide me how do i do that using command line.

2)How to do give multiple severity to a single recipient. What i mean is that if i want error, severe error and warning alerts to reach my email ID, how do i configure? is below command fine

hares -modify ntfr SmtpRecipients parakh.agarwal@abc.com Error Warning SevereError

3) How to i give multiple email ID with their respective severity labels. Do i need to modify main.cf file and give email IDs semi colon seperated or is it possible through command line as well?

4) Are below commands to add Notifier resource OK?

 haconf -makerw
hares -add ntfr NotifierMngr ClusterService
hares -modify ntfr SmtpServer mailhost
hares -modify ntfr SmtpRecipients parakh.agarwal@abc.com Error Warning
hares -modify ntfr Enabled 1
haconf -dump -makero

5) is there anything that needs to be done apart from this.

Hi Forum Friends,

I need some help installing SFHA with all patches included. I wanted to know if there is a way to install all of these below in the same command?

The Sort installation report shows I need these patches

To install in a single shot sfha-rhel6.6_x86_64-6.0.5.200 says to run something like this.

You can also install this patch together with 6.0.1 GA release and 6.0.5 Patch release
    # ./installSFHA605P2 -base_path [<601 path>] -mr_path [<605 path>] [<host1> <host2>...]

So how do I get the rest installed is there any extended commands I can use to include in this to get them all done in one shot?

I am open to using just installmr with -base_path and other switches as well. What ever would do it.

I plan on using a response file btw.

Thanks Bros

I have a quick question regarding service group offline. We only want to bring up/down services when we execute hagrp  -online/-offline manually.

When VCS shutdown, we don't want to bring up the service offline. I know if we freeze it won't go to offline entry. Do we have any other option? For an exmaple, for only we can disable the autostart. Do we have any thing for offline?

Hi all,

I'm in the midst of planning a heartbeat switch migration and will need to iron out a few kinks. My environment consists of multiple VCS cluster groups, running on Windows 2003/2008. Apparently one of the network switch connecting all the heartbeats in need of replacement(it is still online, it just needs to be replaced due to end of support).

My headache is on how to perform the migration without interrupting the clustered applications. Each of the cluster nodes are running on 2-3HB adapters, so initially I thought that replacing them one-by-one will work flawlessly. But when I tested it on a test environment, apparently after I removed and reconnect a HB connection from each pair of nodes, the nodes' state still remain as "Jeopardy" even after reconnection of HB cables. 

Do you have any idea on what's the best method to migrate all HB connections to a different switch online? 

What I've did during testing :

1. Freeze all service groups

2. Unplug connections for HB1 for both NodeA and NodeB from oldSwitch and connect them to newSwitch. Both NodeA and NodeB turns to "Jeopardy" at this stage and didn't return to normal state even after reconnection to new switch.

3. I didn't proceed to perform the same on HB2 connections for both node due to my concern with that "Jeopardy" state, afraid that if I were to proceed, it will make two mini-clusters instead.

4. What I did to return both node from Jeopardy state is to run Cluster Configuration Wizard on NodeA and reconfigure LLT connection to reset communication between the HB adapters.

Appreciate any advise. Thanks in advance.

Environment

SFHA = 6.2

Cluster nodes = 2  (local cluster)

OS = Solaris 10 SPARC

SAN storage connectivity = Dual Fiber cables (DMP installed)

Application = Custom made application with Start/Stop and Monitor process in placed (configured the attributes)under the Java console. Complete application is copied under Mount point (Mount resource)

WE ARE ASUMING THAT CLUSTER SENCE FIRST THE APPLICATION FAILURE INSTEAD OF DISKGROUP AND EXECUTE FAILURE

Query

We are performing a test case in which we are plugging out both fiber cables for SAN connectivity. Instead of Diskgroup or Mount resources crash, we are noticing our Application resource getting crash

Postline trigger is getting failed to execute on onlining a resource. Getting a below error.

2015/02/27 01:25:47 VCS INFO V-16-6-15003 (dsacl2ip) hatrigger:Failed to execute /opt/VRTSvcs/bin/triggers/postonline dsacl2ip SG-APP-TM-003
2015/02/27 01:29:27 VCS INFO V-16-6-15003 (dsacl2ip) hatrigger:Failed to execute /opt/VRTSvcs/bin/triggers/postonline dsacl2ip SG-APP-TM-003

i manually executed the command hatrigger -postonline 0 dsacl2ip SG-APP-TM-003 but again same error.

Please let me know what can be the issue.

Below is my script

ccmId="@(#)%name: postonline % %version: 5 %, %date_modified: Thu Feb 19 12:13:59 2015 %"

#####################################################
# Configure required IP addresses for TM in this section
#####################################################
export TMVIRTIP=10.225.230.57
export TMIPADDRS="
10.15.14.139
10.15.15.137
10.215.17.0

"

#####################################################
# Configure required IP addresses for APPin this section
#####################################################
export RMVIRTIP=10.225.238.58
export RMIPADDRS="
192.168.17.4
"

#####################################################
#
# DO NOT CHANGE ANYTHING BELOW THIS LINE
#
#####################################################

#####################################################
if [ -z "$VCS_HOME" ]; then
    VCS_HOME=/opt/VRTSvcs
fi

#####################################################
# Source VCS helper script
if [ -r $VCS_HOME/bin/ag_i18n_inc.sh ]; then
    . $VCS_HOME/bin/ag_i18n_inc.sh
else
function VCSAG_LOG_MSG {
    msgtype=$1
    msgtxt="$2"
    category="$3"
    case $msgtype in
        E) msgtype="ERROR:";;
        W) msgtype="WARNING:";;
        I) msgtype="INFO:";;
    esac
    echo "`date +'%m/%d %T'`: $msgtype $msgtxt ($category)"
}
fi

#####################################################
# Check if relevant arguments were provided
system=$1
group=$2
if [ -z "$system" ]; then
    VCSAG_LOG_MSG "W""Failed to continue; undefined system name" 15028
    exit
elif [ -z "$group" ]; then
    VCSAG_LOG_MSG "W""Failed to continue; undefined group name" 15031
    exit
fi

#####################################################
# Determine the interface that the APP virtual IP is bound to
function getInterface {
    /bin/netstat -ie | grep -B1 "$1" | sed -ne "s/\(^$2.*\) *Link.*/\1/p"
}

#####################################################
function addRoutes {
    typeset device=$1
    typeset gwAddr=$2
    shift 2

    typeset maskAddr
    for ipAddr in $*; do
        if [ "${ipAddr##*.}" = "0" ];then
            maskAddr=255.255.255.0
        else
            maskAddr=255.255.255.255
        fi

        echo /sbin/route add -net $ipAddr netmask $maskAddr gw $gwAddr dev $device
    done
}

#####################################################
case $group in
    SG-APP-TM-003)
        virtInterface=$(getInterface $TMVIRTIP bond)
        if [ -z "$virtInterface" ]; then
            VCSAG_LOG_MSG "E""Could not determine interface for virtual IP" 15032
        else
            VCSAG_LOG_MSG "I""Adding routes for interface $virtInterface" 15032
            addRoutes $virtInterface 10.225.238.2 $TMIPADDRS
        fi
        ;;
    SG-APP-DS_RM-004)
        virtInterface=$(getInterface $RMVIRTIP bond)
        if [ -z "$virtInterface" ]; then
            VCSAG_LOG_MSG "E""Could not determine interface for virtual IP" 15032
        else
            VCSAG_LOG_MSG "I""Adding routes for interface $virtInterface" 15032
            addRoutes $virtInterface 10.225.238.5 $RMIPADDRS
        fi
        ;;
    *)  ;;
esac

Hello  Guys,

I am in the process of setting up 2 service group and runs independently, but one of the mount needs to be shared between both service groups.

How can I do that?

Refer this url http://www.symantec.com/connect/forums/can-we-make-cluster-filesystem-available-both-nodes-time. I want to know wowza software can support CFS.

Hi All,

We have recently built a 4-node Oracle single-instance CFS cluster, with more to on the way.

The cluster will host many databases, and all nodes will be active. The databases will migrate from non-clustered to CFS clustered servers, and we are concerned about what effect CFS may have on performance.

Concerns:

  - Suitability of our LLT links to handle intra-node communication generated by CFS ?

  - Location of CFS primary, for each filesytem, and how to determine if it is optimal ?

  - When multiple nodes write to the same filesystem simultaeously, would that be detrimental to performance ?

  - I am aware of GLM lock/release as one source of CFS intra-node traffic, but wondering if there are other significant sources, such as VxFS cache coherency ?

  - How to best monitor CFS overhead, intra-node traffic, and overall health ?

Thank your for reading !

Ken W

---------------------Configuration---------------------

Red Hat RHEL 6.5

SFCFSHA 6.1.1 with P200 patch

2 x 1 Gbit ethernet as LLT links, on dedicated VLANs

Oracle 11.2.0.3 and 12.1.0.2

Oracle code local to each node (outside of cluster control)

Oracle data files on CFS filesystem dedicated to only datases running on that node.

Oracle Redo on a single shared filesytem for all nodes.

-------------------------------------------------------------

Failover for VCS Global Cluster

Setup:

Veritas Storage foundation HA for Windows v6.1

Single node Local cluster

Single node Remote cluster

Data replicated between clusters via VVR

A Global cluster will only failover to the remote cluster if the entire local cluster dies

However if a Service Group fails on the local cluster it will not initiate a failover to the remote cluster.

Is there a way around this since we have single node clusters? Basically I need the cluster to automatically failover to the remote cluster if a service group fails on the local cluster.

Hi, 

Just trying to tap into the experience out there. Have anyone upgraded CPS servers to the latest 6.2? and what is your procedure?

Much appreciated.

Regards,

Ida

Hello Guys,

We do configure many VCS cluster and we would like to verify all the confiugraion before we hand over to customer. We would like to automate the verification. here is my plan to do basic verificaiton

- Verify LLT config

-Verify  GAB config

- Check main.cf , and see volume group or VIP is not configured in boot up.

Anything else I need to check ?  feell free to share any idea

Environment

SFHA version = 6.1

OS = 2008R2

Cluster nodes = two(Local HA)

Shared storage = yes

Volume created on shared storage = yes

Query

When I am going to trigger the SQL Server service group wizar, at the time of resource creation, I am getting the below ERROR message.