The XenServer host has recently been updated with a hotfix that updates the kernel.
After patching the XenServer host is in a boot loop. During startup there is a message "Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)" followed by a call trace:


The initrd file size is approximately 2MB:

# ls -ltr /boot/initrd*
...
-rw------- 1 root root 2172312 Jan 12 16:39 /boot/initrd-3.10.0+2.img

On a working server with root disk multipathing the correct file size should be approximately 4MB:

# ls -ltr /boot/initrd*
...
-rw------- 1 root root 4408001 Dec 23 11:01 /boot/initrd-3.10.0+2.img

The initrd file refers to the incorrect root device:

# zcat /boot/initrd-3.10.0+2.img |cpio -i --to-stdout init |grep mkrootdev
mkrootdev -t ext3 -o defaults,ro /dev/mapper/3​600601234567890abcdef11234567890p1

Find the SCSI ID of the root disk:

# scsi_id -gus  /block/sda
​3600601603a9038006aad798e5b7de511

Update the rootdev line in /etc/sysconfig/mkinitrd.latches based on the new SCSI ID:

--rootdev=/dev/mapper/​3600601603a9038006aad798e5b7de511p1

Generate a new initrd file while root disk multipathing is active (the fallback kernel does not have root disk multipathing enabled). See How to chroot to the XenServer root disk from the Install CD to get a working multipathed root disk if needed.

# . /boot/initrd-3.10.0+2.img.cmd

Verify initrd contains the correct multipath entry:

# zcat /boot/initrd-3.10.0+2.img |cpio -i --to-stdout init |grep mkrootdev
mkrootdev -t ext3 -o defaults,ro /dev/mapper/3​600601603a9038006aad798e5b7de511p1

---

Problem Cause

The LUNs were migrated between storage arrays resulting in new Unique IDs being given to the LUNs by EMC:


The /etc/sysconfig/mkinitrd.latches file on XenServer hosts with multipath boot from SAN contain a reference to the disk by its SCSI ID. This SCSI ID is returned by EMC based on the Unique ID.
mkinitrd generates the initrd with the device listed in the latches file.
When scanning for that disk it does not find the device, this results in the fnci and multipath kernel modules not being added into the initrd.
Then when booting there is no fnic module loaded so the disk in inaccessible through the HBA.

After updating XenServer with a hotfix that updates the kernel, the server fails to start correctly when rebooting.