What are Experimental Features?

This article covers a product configuration or procedure for which Citrix does not currently offer support.  Use of this configuration should only be used in a lab or test environment and not with production deployments.  Experimental features are occasionally released as a mechanism to actively seek feedback on the potential of implementing support for technologies, but the form any level of support takes has yet to be determined and Citrix makes no commitment to productise experimental features in the future and may withdraw them based on feedback or technological challenges at any point. 

Experimental Support for XenApp and Multiple GPUs

XenServer
Citrix XenServer does not officially support >1 GPU per virtual machine, nor does NVIDIA validate >1 GPU per VM. However XenServer does provide the capability to try multiple pass-through GPUs and it is possible to assign >1 GPU into a VM. If any customer tries this, they should understand that it’s not a supported configuration; read more at Assigning Multiple GPUs to a VM with Citrix XenServer.
 
XenApp
Additionally the HDX team added experimental support for XenApp session allocation to use more than one GPU. Again this experimental feature is not and has never been supported. On 2008 R2 this added OpenGL, DX9 and DX11 support at a time when many GPU vendor (e.g. NVIDIA) drivers did not offer hardware OpenGL support in XenApp environments. This also worked on physical servers.

Details of the Experimental Support for Multiple GPUs on XenApp

The feature was experimental and required the GPUs to be identical both in terms of GPU model and firmware version. This was because the feature was designed to use the “best” GPU available with highest level of support for OpenGL/DirectX etc. and therefore the highest chance of a session starting on a GPU with the capabilities to support an application.
The feature also used a formally unsupported mechanism to interact with the Microsoft Server OS. This is one of the reasons that the feature was experimental as we ensure formally supported features use fully supported and tested interfaces jointly agreed with Microsoft. At the time of implementation we were trying to assess the value of the feature and its potential use cases.
The allocation of sessions to the GPUs was by simple round-robin, so with 3 GPUs. The first, fourth, seventh … sessions started would be allocated to one GPU and the second, fifth, eighth… sessions to the second GPU, and so on.  

Feedback on the Experimental Support for multiple GPUs on XenApp

Some users chose to evaluate the feature for us. In particular Tobias Kreidl from the Northern Arizona University published a whitepaper on his results on the benefits to scalability of using multiple GPUs, exactly the kind of data and information from real customer hardware that we were seeking by releasing the experimental feature. 
 
Additionally we did more testing internally ourselves on the session allocation scheme and the results were confirmed by customers who were also trialling the feature. The session allocation was sufficient for scalability investigations such as Tobias’, but we felt it was not robust enough to be supported for production systems. The allocation scheme did not account for:

• Real time load on the GPU and what the sessions were actually doing: Users could be allocated to GPUs heavily loaded while others had more capacity
• Deployments where users were logging on and off frequently: They statistically would end up with different number of users on GPUs leading to different loads

What has changed for DirectX on Windows Server 2012 R2?

Recently we have had users reporting that the behavior on 2012 R2 has changed. We have investigated and found that DirectX9 (DX9) is still working on 2012 R2. However, with applications using DirectX11 (DX11), a single GPU will be allocated to all sessions. This is because DirectX11 has changed one of those unsupported Microsoft interfaces on which the feature was developed. This is only an issue with the version of the DirectX11 runtime used on Windows Server 2012. The round-robin allocation of GPUs to sessions still works for both DirectX9 and DirectX11 on Windows Server 2008 R2.

Additional changes for OpenGL with NVIDIA GPUs on all Server OS

Allocation of sessions across multiple GPUs is also not working with OpenGL and a NVIDIA driver that supports their (recently introduced) OpenGL hardware acceleration on RDS. In this case NVIDIA overrides the experimental feature at the driver level resulting in support from NVIDIA’s single GPU per VM implementation instead.

Does Citrix Intend to Fix the Experimental Support?

We currently have no plans to resurrect/fix the existing prototype. We may in the future consider a new implementation but the issues with third-party drivers (particularly that GPU vendors are supporting single GPU per server OS VM implementations) and changing interfaces mean that we would approach it differently.
We are actually getting fewer requests for this feature as GPU vendors are releasing increasingly more powerful GPUs suited to supporting Windows Server VMs.

Disclaimer

This article covers a product configuration or procedure which Citrix does not currently offer support for.  Use of this configuration should only be used in a lab or test environment and not with production deployments.  Experimental features are occasionally released as a mechanism to actively seeking feedback on the potential of implementing support for technologies, but the form any level of support takes has yet to be determined and Citrix makes no commitment to productise experimental features in the future and may withdraw them based on feedback or technological challenges at any point.