Citrix Virtual Apps and Desktops 2402 LTSR and 2507 LTSR with XenServer hosting connections.
When many machines are shut down or restarted at once, for example at the end of the working day or during an Autoscale schedule change:
The Delivery Controller allows up to 20 connections to each XenServer host by default. A shutdown or restart keeps one of those connections busy until it finishes, which can take several minutes. By default, a hosting connection allows up to 100 power actions at once.
When more power actions are running than there are connections, the Delivery Controller can no longer receive power state updates from XenServer, and it marks the hosting connection as lost. The connection test uses a different service with its own connections, so it still succeeds.
Upgrade to Citrix Virtual Apps and Desktops 2607 LTSR.
Raise the connection limit above the number of simultaneous power actions allowed on the hosting connection.
The option name is case sensitive.
To make the same change with PowerShell:
Set-HypHypervisorConnectionMetadata -HypervisorConnectionName "<connection name>" -Name "Citrix_MachineManagement_Options" -Value "DefaultConnectionLimit=150"
This command replaces any existing connection options, so include them in the value if there are any.
When many machines are shut down or restarted at once, for example at the end of the working day or during an Autoscale schedule change:
Confirming the issue. During a bulk shutdown or restart, run this command on the Delivery Controller, replacing <coordinator> with the IP address of the XenServer pool coordinator. Use port 80 if the connection uses HTTP.
Get-NetTCPConnection -RemoteAddress <coordinator> -RemotePort 443 -State Established | Where-Object OwningProcess -eq (Get-Process BrokerService).Id | Measure-Object
A count that repeatedly reaches 20 confirms this issue. After the workaround, the count can rise above 20.
Power actions still take a long time. The workaround stops the connection loss but does not make power actions faster. How quickly they complete depends on the XenServer pool. Raising Simultaneous actions or Maximum new actions per minute does not speed up a large batch. It makes more machines wait in a queue on the XenServer pool at once. If machines wait a long time in XenCenter with a pending operation, lower these values instead.
Wrong power states after recovery. Restart the Citrix Broker Service on the Delivery Controllers.