A VM with 2 VBD, 8 vCPUs, 7 VIFs may consume over 90 grant tables, but ONLY consume 26 grant tables when running with xen PV driver version 9.1.2.

It will lead to virtual NIC initialization failure issue after Windows VM startup.

Device manager shows NICs are not initialized:



Setupapi.log:

Device not started: Device has problem: 0x1f (CM_PROB_FAILED_ADD), problem status: 0xc0000495.

Fix identified in XenBus driver, which will be included in the next tools release.

Workaround:

1. Run below command on each Dom0 to increase gnttab_max_frames from default value 64 to 128

/opt/xensource/libexec/xen-cmdline --set-xen gnttab_max_frames=128

2. Reboot host

3. Configure parameter max_grant_frames per Virtual Machine:

xe vm-param-set uuid=$UUID platform:max_grant_frames=128

Note : 
Please ensure to increase dom0 memory as well, extend dom0 memory to 16GB or higher depending on total VM numbers and physical memory of the system. 

 

---

Problem Cause

Per VBDs = 2 (ring-grantrefs + overhead for traffic)
Per VIFs = (1 [receiver-ring] + 256 [receive-slots] + 1 [transmit-ring] (+ traffic) + 1 [controller]) x NumberOfQueues

In actual fact, each granter will round up to the nearest CACHE slab size (253 grants), so each VBD will consume at-least 253 grants (+ more for traffic) and each VIF will consume at-least 1012 grants (+ possibly more for transmit traffic) per Queue (vCPUs up-to-8).
 

<What is Grant Table Frame>
Grant table frames contain grant references which point to the shared (granted) pages, the grant table doesn't contain the shared pages themselves. Those are maintained separately.
A frame is a memory page and is 4096 bytes in size. A grant reference is a pointer and is 8 bytes in size. So grant references per frame is 4096/8.