vxfen module cause sles 11 sp1 kernel panic

Hi all: 

I use three share disks on iscsi(target using LIO) as the I/O fence disk in VCS 6.0 on SLES11 SP1. After configuration, the kernel panic when starting vxfen. Log is here:

[   83.640266] BUG: unable to handle kernel NULL pointer dereference at 0000000000000060

[   83.641248] IP: [<ffffffff81396113>] down_read+0x3/0x10

[   83.641933] PGD 37b62067 PUD 3c7b2067 PMD 0 

[   83.642512] Oops: 0002 [#1] SMP 

[   83.643016] last sysfs file: /sys/devices/platform/host2/iscsi_host/host2/initiatorname

[   83.644009] CPU 0 

[   83.644054] Modules linked in: vxfen(PN) dmpalua(PN) vxspec(PN) vxio(PN) vxdmp(PN) snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc gab(PN) ipv6 crc32c iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi af_packet llt(PN) amf(PN) microcode fuse loop fdd(PN) exportfs vxportal(PN) vxfs(PN) dm_mod virtio_blk virtio_balloon virtio_net sg rtc_cmos rtc_core rtc_lib tpm_tis button tpm tpm_bios floppy i2c_piix4 virtio_pci virtio_ring pcspkr i2c_core virtio uhci_hcd ehci_hcd sd_mod crc_t10dif usbcore edd ext3 mbcache jbd fan processor ide_pci_generic piix ide_core ata_generic ata_piix libata scsi_mod thermal thermal_sys hwmon

[   83.644054] Supported: Yes, External

[   83.644054] Pid: 4670, comm: vxfen Tainted: P             2.6.32.12-0.7-default #1 Bochs

[   83.644054] RIP: 0010:[<ffffffff81396113>]  [<ffffffff81396113>] down_read+0x3/0x10

[   83.644054] RSP: 0018:ffff88002e831638  EFLAGS: 00010286

[   83.644054] RAX: 0000000000000060 RBX: 0000000000000000 RCX: ffff88003c4e2480

[   83.644054] RDX: 0000000000000001 RSI: 0000000000002000 RDI: 0000000000000060

[   83.644054] RBP: ffff88002c9aa000 R08: 0000000000000000 R09: ffff88003c4e2480

[   83.644054] R10: ffff88003d6da140 R11: 00000000000000d0 R12: 0000000000000060

[   83.644054] R13: ffff88002c9a8000 R14: 0000000000000000 R15: ffff88002c9a8001

[   83.644054] FS:  0000000000000000(0000) GS:ffff880006200000(0000) knlGS:0000000000000000

[   83.644054] CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b

[   83.644054] CR2: 0000000000000060 CR3: 000000003c72a000 CR4: 00000000000006f0

[   83.644054] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000

[   83.644054] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

[   83.644054] Process vxfen (pid: 4670, threadinfo ffff88002e830000, task ffff88002c8e8140)

[   83.644054] Stack:

[   83.644054]  ffffffff810325ab 0000000000000000 0000001000000000 ffff88003ca9d690

[   83.644054] <0> ffff88003c4e2480 000000013d0181c0 0000000000000002 0000000000000002

[   83.644054] <0> 0000000000000002 0000000000001010 0000000000000000 0000000000000000

[   83.644054] Call Trace:

[   83.644054]  [<ffffffff810325ab>] get_user_pages_fast+0x11b/0x1a0

[   83.644054]  [<ffffffff81127c27>] __bio_map_user_iov+0x167/0x2a0

[   83.644054]  [<ffffffff81127d69>] bio_map_user_iov+0x9/0x30

[   83.644054]  [<ffffffff81127dac>] bio_map_user+0x1c/0x30

[   83.644054]  [<ffffffff811bab91>] __blk_rq_map_user+0x111/0x140

[   83.644054]  [<ffffffff811bacc5>] blk_rq_map_user+0x105/0x190

[   83.644054]  [<ffffffff811bebe7>] sg_io+0x3c7/0x3e0

[   83.644054]  [<ffffffff811bf1ec>] scsi_cmd_ioctl+0x2ac/0x470

[   83.644054]  [<ffffffffa01b75b1>] sd_ioctl+0xa1/0x120 [sd_mod]

[   83.644054]  [<ffffffffa0ccaa93>] vxfen_ioctl_by_bdev+0xc3/0xd0 [vxfen]

[   83.644054]  [<ffffffffa0ccb6ac>] vxfen_ioc_kernel_scsi_ioctl+0xec/0x3c0 [vxfen]

[   83.644054]  [<ffffffffa0ccbe1f>] vxfen_lnx_pgr_in+0xff/0x380 [vxfen]

[   83.644054]  [<ffffffffa0ccc11a>] vxfen_plat_pgr_in+0x7a/0x1c0 [vxfen]

[   83.644054]  [<ffffffffa0cd26c3>] vxfen_readkeys+0xa3/0x380 [vxfen]

[   83.644054]  [<ffffffffa0cd3514>] vxfen_membreg+0x84/0xae0 [vxfen]

[   83.644054]  [<ffffffffa0cce6d6>] vxfen_preexist_split_brain_scsi3+0x96/0x2d0 [vxfen]

[   83.644054]  [<ffffffffa0ccf96d>] vxfen_reg_coord_disk+0x7d/0x660 [vxfen]

[   83.644054]  [<ffffffffa0ca5e0b>] vxfen_reg_coord_pt+0xfb/0x250 [vxfen]

[   83.644054]  [<ffffffffa0cb849c>] vxfen_handle_local_config_done+0x14c/0x8d0 [vxfen]

[   83.644054]  [<ffffffffa0cbad57>] vxfen_vrfsm_cback+0xad7/0x17b0 [vxfen]

[   83.644054]  [<ffffffffa0cd5b20>] vrfsm_step+0x1b0/0x3b0 [vxfen]

[   83.644054]  [<ffffffffa0cd7e1c>] vrfsm_recv_thread+0x32c/0x970 [vxfen]

[   83.644054]  [<ffffffffa0cd85b4>] vxplat_lx_thread_base+0xa4/0x100 [vxfen]

[   83.644054]  [<ffffffff81003fba>] child_rip+0xa/0x20

[   83.644054] Code: 48 85 f6 74 0f 48 89 e7 e8 5b 09 cd ff 85 c0 48 63 d0 7e 07 48 c7 c2 fc fd ff ff 48 83 c4 78 48 89 d0 5b 5d c3 00 00 00 48 89 f8 <3e> 48 ff 00 79 05 e8 52 fa e4 ff c3 90 48 89 f8 48 ba 01 00 00 

[   83.644054] RIP  [<ffffffff81396113>] down_read+0x3/0x10

[   83.644054]  RSP <ffff88002e831638>

[   83.644054] CR2: 0000000000000060

 

 

vcs1:~ # vxdisk  list

DEVICE       TYPE            DISK         GROUP        STATUS

aluadisk0_1  auto:cdsdisk    -            -            online

aluadisk0_2  auto:cdsdisk    -            -            online

aluadisk0_3  auto:cdsdisk    -            -            online

sda          auto:none       -            -            online invalid

vda          simple          vda          data_dg      online

vdb          simple          vdb          data_dg      online

 

vcs1:~ # cat /proc/partitions 

major minor  #blocks  name

 

   8        0    8388608 sda

   8        1    7333641 sda1

   8        2    1052257 sda2

 253        0   10485760 vda

 253       16    4194304 vdb

   8       16    1048576 sdb

   8       19    1046528 sdb3

   8       24    1046528 sdb8

   8       32    1048576 sdc

   8       35    1046528 sdc3

   8       40    1046528 sdc8

   8       48    1048576 sdd

   8       51    1046528 sdd3

   8       56    1046528 sdd8

 201        0    8388608 VxDMP1

 201        1    7333641 VxDMP1p1

 201        2    1052257 VxDMP1p2

 201       16    1048576 VxDMP2

 201       19    1046528 VxDMP2p3

 201       24    1046528 VxDMP2p8

 201       32    1048576 VxDMP3

 201       35    1046528 VxDMP3p3

 201       40    1046528 VxDMP3p8

 201       48    1048576 VxDMP4

 201       51    1046528 VxDMP4p3

 201       56    1046528 VxDMP4p8

 199     6000    8388608 VxVM6000

 199     6001     153600 VxVM6001

 

 

 

LIO Target server message:

 

 

[60529.780169] br0: port 3(vnet2) entered forwarding state

[60675.372132] TARGET_CORE[iSCSI]: Unsupported SCSI Opcode 0x24, sending CHECK_CONDITION.

[60675.372217] TARGET_CORE[iSCSI]: Unsupported SCSI Opcode 0x24, sending CHECK_CONDITION.

[60675.373337] TARGET_CORE[iSCSI]: Unsupported SCSI Opcode 0x24, sending CHECK_CONDITION.

[60675.373426] TARGET_CORE[iSCSI]: Unsupported SCSI Opcode 0x24, sending CHECK_CONDITION.

[60675.374304] TARGET_CORE[iSCSI]: Unsupported SCSI Opcode 0x24, sending CHECK_CONDITION.

[60675.374374] TARGET_CORE[iSCSI]: Unsupported SCSI Opcode 0x24, sending CHECK_CONDITION.

 

 

All test have passed by 

vcs1:~ # vxfentsthdw -m 

 

Veritas vxfentsthdw version 6.0.000.000-GA Linux

 

 

The utility vxfentsthdw works on the two nodes of the cluster.

The utility verifies that the shared storage one intends to use is

configured to support I/O fencing.  It issues a series of vxfenadm

commands to setup SCSI-3 registrations on the disk, verifies the

registrations on the disk, and removes the registrations from the disk.

 

 

******** WARNING!!!!!!!! ********

 

THIS UTILITY WILL DESTROY THE DATA ON THE DISK!! 

 

Do you still want to continue : [y/n] (default: n) y

The logfile generated for vxfentsthdw is /var/VRTSvcs/log/vxfen/vxfentsthdw.log.9431

 

Enter the first node of the cluster:

vcs1

Enter the second node of the cluster:

vcs2

 

Enter the disk name to be checked for SCSI-3 PGR on node vcs1 in the format:

for dmp: /dev/vx/rdmp/sdx

for raw: /dev/sdx

Make sure it is the same disk as seen by nodes vcs1 and vcs2

/dev/sdb

 

Enter the disk name to be checked for SCSI-3 PGR on node vcs2 in the format:

for dmp: /dev/vx/rdmp/sdx

for raw: /dev/sdx

Make sure it is the same disk as seen by nodes vcs1 and vcs2

/dev/sdb

 

***************************************************************************

 

Testing vcs1 /dev/sdb vcs2 /dev/sdb

 

Evaluate the disk before testing  ........................ No Pre-existing keys

RegisterIgnoreKeys on disk /dev/sdb from node vcs1 ..................... Passed

Verify registrations for disk /dev/sdb on node vcs1 .................... Passed

RegisterIgnoreKeys on disk /dev/sdb from node vcs2 ..................... Passed

Verify registrations for disk /dev/sdb on node vcs2 .................... Passed

Unregister keys on disk /dev/sdb from node vcs1 ........................ Passed

Verify registrations for disk /dev/sdb on node vcs2 .................... Passed

Unregister keys on disk /dev/sdb from node vcs2 ........................ Passed

Check to verify there are no keys from node vcs1 ....................... Passed

Check to verify there are no keys from node vcs2 ....................... Passed

RegisterIgnoreKeys on disk /dev/sdb from node vcs1 ..................... Passed

Verify registrations for disk /dev/sdb on node vcs1 .................... Passed

Read from disk /dev/sdb on node vcs1 ................................... Passed

Write to disk /dev/sdb from node vcs1 .................................. Passed

Read from disk /dev/sdb on node vcs2 ................................... Passed

Write to disk /dev/sdb from node vcs2 .................................. Passed

Reserve disk /dev/sdb from node vcs1 ................................... Passed

Verify reservation for disk /dev/sdb on node vcs1 ...................... Passed

Read from disk /dev/sdb on node vcs1 ................................... Passed

Read from disk /dev/sdb on node vcs2 ................................... Passed

Write to disk /dev/sdb from node vcs1 .................................. Passed

Expect no writes for disk /dev/sdb on node vcs2 ........................ Passed

RegisterIgnoreKeys on disk /dev/sdb from node vcs2 ..................... Passed

Verify registrations for disk /dev/sdb on node vcs1 .................... Passed

Verify registrations for disk /dev/sdb on node vcs2 .................... Passed

Write to disk /dev/sdb from node vcs1 .................................. Passed

Write to disk /dev/sdb from node vcs2 .................................. Passed

Preempt and abort key KeyA using key KeyB on node vcs2 ................. Passed

Test to see if I/O on node vcs1 terminated ............................. Passed

RegisterIgnoreKeys on disk /dev/sdb from node vcs1 ..................... Passed

Verify registrations for disk /dev/sdb on node vcs1 .................... Passed

Preempt key KeyC using key KeyB on node vcs2 ........................... Passed

Test to see if I/O on node vcs1 terminated ............................. Passed

Verify registrations for disk /dev/sdb on node vcs1 .................... Passed

Verify registrations for disk /dev/sdb on node vcs2 .................... Passed

Verify reservation for disk /dev/sdb on node vcs1 ...................... Passed

Verify reservation for disk /dev/sdb on node vcs2 ...................... Passed

Remove key KeyB on node vcs2 ........................................... Passed

Check to verify there are no keys from node vcs1 ....................... Passed

Check to verify there are no keys from node vcs2 ....................... Passed

Check to verify there are no reservations on disk /dev/sdb from node vcs1  Passed

Check to verify there are no reservations on disk /dev/sdb from node vcs2  Passed

RegisterIgnoreKeys on disk /dev/sdb from node vcs1 ..................... Passed

Verify registrations for disk /dev/sdb on node vcs1 .................... Passed

RegisterIgnoreKeys on disk /dev/sdb from node vcs1 ..................... Passed

Verify registrations for disk /dev/sdb on node vcs1 .................... Passed

Clear PGR on node vcs1 ................................................. Passed

Check to verify there are no keys from node vcs1 ....................... Passed

 

ALL tests on the disk /dev/sdb have PASSED.

The disk is now ready to be configured for I/O Fencing on node vcs1.

 

ALL tests on the disk /dev/sdb have PASSED.

The disk is now ready to be configured for I/O Fencing on node vcs2.

 

Removing test keys and temporary files, if any...