4 production VMs show Gray screen when booted up with vGPU attached, error message shows in Hypervisor status report:

kern.log:
May 14 20:33:16 citrix-srv-172 kernel: [ 1545.091820] nvrm-nvlog: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
May 14 20:33:16 citrix-srv-172 kernel: [ 1545.091820] nvrm-nvlog: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
May 14 20:33:16 citrix-srv-172 kernel: [ 1545.091821] nvrm-nvlog: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
May 14 20:33:16 citrix-srv-172 kernel: [ 1545.091822] nvrm-nvlog: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
May 14 20:33:16 citrix-srv-172 kernel: [ 1545.091823] nvrm-nvlog: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
May 14 20:33:16 citrix-srv-172 kernel: [ 1545.091823] nvrm-nvlog: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
May 14 20:33:16 citrix-srv-172 kernel: [ 1545.091823] nvrm-nvlog: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
May 14 20:33:16 citrix-srv-172 kernel: [ 1545.091824] nvrm-nvlog: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
May 14 20:33:16 citrix-srv-172 kernel: [ 1545.091824] nvrm-nvlog: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
May 14 20:33:16 citrix-srv-172 kernel: [ 1545.091825] nvrm-nvlog: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
May 14 20:33:16 citrix-srv-172 kernel: [ 1545.091825] nvrm-nvlog: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
May 14 20:33:16 citrix-srv-172 kernel: [ 1545.091827] nvrm-nvlog: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
May 14 20:33:16 citrix-srv-172 kernel: [ 1545.091827] nvrm-nvlog: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
May 14 20:33:16 citrix-srv-172 kernel: [ 1545.091828] nvrm-nvlog: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
May 14 20:33:16 citrix-srv-172 kernel: [ 1545.091828] nvrm-nvlog: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
May 14 20:33:16 citrix-srv-172 kernel: [ 1545.091829] nvrm-nvlog: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
May 14 20:33:16 citrix-srv-172 kernel: [ 1545.091829] nvrm-nvlog: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
May 14 20:33:16 citrix-srv-172 kernel: [ 1545.091830] nvrm-nvlog: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
May 14 20:33:16 citrix-srv-172 kernel: [ 1545.092286] NVRM: Xid (PCI:0000:5e:00): 43, pid=10669, Ch 00000043

daemon.log:
May 14 20:33:15 citrix-srv-172 vgpu-7[22130]: error: vmiop_log: (0x0): pte blit resource initialization failed with error 7
May 14 20:33:15 citrix-srv-172 vgpu-7[22130]: error: vmiop_log: (0x0): Failed to setup push buffer: 0x7
May 14 20:33:15 citrix-srv-172 vgpu-7[22130]: error: vmiop_log: (0x0): Failed to init PTE blit resources
May 14 20:33:15 citrix-srv-172 vgpu-7[22130]: error: vmiop_log: (0x0): VGPU message 56 failed, result code: 0xff000007
May 14 20:33:15 citrix-srv-172 vgpu-7[22130]: error: vmiop_log: (0x0):         0x1019, 0x200, 0xc1d00016, 0xff020000,
May 14 20:33:15 citrix-srv-172 vgpu-7[22130]: error: vmiop_log: (0x0):         0x46400000, 0x7000, 0x0, 0x0, 0x1000,
May 14 20:33:15 citrix-srv-172 vgpu-7[22130]: error: vmiop_log: (0x0):         0xdeadcafe, 0x0, 0x0, (Not logged)
May 14 20:33:15 citrix-srv-172 vgpu-7[22130]: error: vmiop_log: vmiopd_set_fill_pte_mem_params_v14_01: BAR1 VA Space handle is invalid
May 14 20:33:15 citrix-srv-172 vgpu-7[22130]: error: vmiop_log: vmiopd_set_fill_pte_mem_params_v14_01: BAR1 VA Space handle is invalid
May 14 20:33:15 citrix-srv-172 vgpu-6[18861]: error: vmiop_log: (0x0): pte blit resource initialization failed with error 7
May 14 20:33:15 citrix-srv-172 vgpu-6[18861]: error: vmiop_log: (0x0): Failed to setup push buffer: 0x7
May 14 20:33:15 citrix-srv-172 vgpu-6[18861]: error: vmiop_log: (0x0): Failed to init PTE blit resources
May 14 20:33:15 citrix-srv-172 vgpu-6[18861]: error: vmiop_log: (0x0): VGPU message 56 failed, result code: 0xff000007
May 14 20:33:15 citrix-srv-172 vgpu-6[18861]: error: vmiop_log: (0x0):         0x1019, 0x200, 0xc1d00016, 0xff020000,
May 14 20:33:15 citrix-srv-172 vgpu-6[18861]: error: vmiop_log: (0x0):         0x45800000, 0x7c000, 0x0, 0x0, 0x1000,
May 14 20:33:15 citrix-srv-172 vgpu-6[18861]: error: vmiop_log: (0x0):         0xdeadcafe, 0x0, 0x0, (Not logged)
May 14 20:33:15 citrix-srv-172 vgpu-2[10423]: error: vmiop_log: (0x0): pte blit resource initialization failed with error 7
May 14 20:33:15 citrix-srv-172 vgpu-2[10423]: error: vmiop_log: (0x0): Failed to setup push buffer: 0x7
May 14 20:33:15 citrix-srv-172 vgpu-2[10423]: error: vmiop_log: (0x0): Failed to init PTE blit resources
May 14 20:33:15 citrix-srv-172 vgpu-2[10423]: error: vmiop_log: (0x0): VGPU message 56 failed, result code: 0xff000007
May 14 20:33:15 citrix-srv-172 vgpu-2[10423]: error: vmiop_log: (0x0):         0x1019, 0x200, 0xc1d00016, 0xff020000,
May 14 20:33:15 citrix-srv-172 vgpu-2[10423]: error: vmiop_log: (0x0):         0x44400000, 0x82000, 0x0, 0x0, 0x1000,
May 14 20:33:15 citrix-srv-172 vgpu-2[10423]: error: vmiop_log: (0x0):         0xdeadcafe, 0x0, 0x0, (Not logged)
May 14 20:33:16 citrix-srv-172 vgpu-1[10669]: error: vmiop_log: (0x0): Timeout occurred, reset initiated.
May 14 20:33:16 citrix-srv-172 vgpu-1[10669]: error: vmiop_log: (0x0): TDR_DUMP:0x52445456 0x008607e8 0x000001cc 0x00000001
May 14 20:33:16 citrix-srv-172 vgpu-1[10669]: error: vmiop_log: (0x0): TDR_DUMP:0x00989680 0x00000000 0x000001bb 0x0000000f
May 14 20:33:16 citrix-srv-172 vgpu-1[10669]: error: vmiop_log: (0x0): TDR_DUMP:0x00000100 0x00000000 0xbad0acac 0x00000000
May 14 20:33:16 citrix-srv-172 vgpu-1[10669]: error: vmiop_log: (0x0): TDR_DUMP:0x00000000 0x00000000 0x00001600 0x00000000
May 14 20:33:16 citrix-srv-172 vgpu-1[10669]: error: vmiop_log: (0x0): TDR_DUMP:0xbad0acad 0x00000000 0x00000000 0x00000000

Problem Cause

"Nvidia PhysX System Software 9.21.0731" is installed in the VMs:

Nvidia PhysX is essentially an open-source engine/software which allows you to simulate the physics of fixed objects that are colliding, overlapping, or intersecting with each other in order to produce a more realistic result.

This software is now downloadable through the Nvidia website where everyone can have access to it.

Ref: https://c4re.gr/what-is-nvidia-physx-system-software/