NVIDIA GPU accelerated hardware encode (NVENC) in XenDesktop 7.11 is enabled only when the framebuffer profile is 1GB or more. It is not enabled by NVIDIA for the 512MB framebuffer profile.
Application GPU acceleration remains fully supported and available with all profiles including 512MB.
This issue affects NVIDIA GRID GPUs used for vGPU including the M60, M10 and M6 cards.

Solution or Workaround

This is a known limitation and a resolution is being actively investigated by NVIDIA and Citrix engineers. At the moment NVENC is not enabled on 512MB profiles by NVIDIA and as such this restriction applies to all hypervisors including VMware ESXi and Citrix XenServer. At the time of writing, NVENC is only available when using GRID 4.0 or higher with 1GB and higher frame buffer profiles.
 
If you are an NVIDIA customer, and wish to be provide feedback and/or receive notification when the fix is available, please raise a support ticket via the enterprise support portal: https://nvidia-esp.custhelp.com

Problem Cause

NVIDIA have disabled NVENC on 512MB profiles with the GRID 4.0 (August 2016 release) to minimize the risk of users encountering the Memory exhaustion issue documented in the GRID vGPU release notes - see Section 5.1. “Memory exhaustion can occur with vGPU profiles that have 512 Mbytes or less of framebuffer.
The following vGPU profiles have 512 Mbytes or less of frame buffer:

• Tesla M6-0B, M6-0Q
• Tesla M10-0B, M10-0Q
• Tesla M60-0B, M60-0Q
• GRID K100, K120Q • GRID K200, K220Q

NVIDIA and Citrix engineering are actively working to resolve the underlying issues associated with memory exhaustion.

• NVIDIA Blog

•  NVIDIA Forums

•  Citrix Documentation - GPU acceleration for Windows Desktop OS