This article contains information about the Intel 55x0 chipset errata - Interrupt remapping issue. Intel 5500/5520/X58 chipset revision 0x13 and 0x22 have an errata (#47 and #53) which makes the IOMMU interrupt remapping unit unreliable. This erratum causes interruptions and the interrupt remapping invalidations become unresponsive.
XenServer versions newer than XenServer 6.0 are affected because this is when the IOMMU support was enabled by default in the Xen hypervisor.
The recommended workaround of Intel for the errata is to disable the Interrupt remapping feature itself. Unfortunately, many BIOS vendors have not implemented this workaround.
This erratum has the following symptoms:
Intermittent host crashes, restarts, and becoming unresponsive.
Storage controllers reset or stop responding
Network Controllers reset or stop responding
Error messages are seen in logs as a side effect of the interrupt remapping issue:
NMI: IOCK error (debug interrupt?)
NMI - PCI system error (SERR)
BUG: soft lockup - CPU#0 stuck for 61s!
Rejecting I/O to offline device
Mailbox command timeout occurred. Issuing ISP abort
Determine if the host has the affected chipset:
# lspci -n | grep "8086:342e (rev 13)" or lspci -n | grep "8086:342e (rev 22)"
If the preceding command returns no results, then the system is not affected.
If the preceding command returns a result, then the xen-dmesg should be examined to see if interrupt remapping is enabled using the following commands:
# xl debug-keys V
# xl dmesg | grep "Interrupt Remapping”
If remapping is enabled in the result, complete the workaround steps to disable the feature.
You can either prevent the BIOS advertising the interrupt remapping feature or prevent Xen from using the feature.
Turn off any BIOS feature mentioning Interrupt remapping or Turn off IOMMU support. This is vendor specific and might not be available in all BIOS implementations.
Example of edited /boot/extlinux.conf
append /boot/xen.gz iommu=no-intremap dom0_mem=752M lowmem_emergency_pool=1M crashkernel=128M@32M console= vga=mode-0x0311 --- /boot/vmlinuz-2.6-xen root=LABEL=root-abcdef ro xencons=hvc console=hvc0 console=tty0 quiet vga=785 splash --- /boot/initrd-2.6-xen.img
The following command adds it to the Xen boot parameters for you without having to manually edit the /boot/extlinux.conf file:
/opt/xensource/libexec/xen-cmdline --set-xen iommu=no-intremap
Intel errata document - http://www.intel.co.uk/content/dam/doc/specification-update/5520-and-5500-chipset-ioh-specification-update.pdf
Wikipedia article on IOMMU - http://en.wikipedia.org/wiki/IOMMU