XenServer hosting connection shows HypervisorConnectionLost during bulk power actions

book

Article ID: CTX697171

calendar_today

Updated On:

Description

Citrix Virtual Apps and Desktops 2402 LTSR and 2507 LTSR with XenServer hosting connections.

When many machines are shut down or restarted at once, for example at the end of the working day or during an Autoscale schedule change:

  • The XenServer hosting connection shows HypervisorConnectionLost and is reported as unavailable.
  • All machines on that connection show a power state of Unknown.
  • A connection test on the hosting connection still succeeds.
  • The power actions still complete, and the connection recovers once they finish.
  • Some machines can still show the wrong power state after the connection recovers.

Cause

The Delivery Controller allows up to 20 connections to each XenServer host by default. A shutdown or restart keeps one of those connections busy until it finishes, which can take several minutes. By default, a hosting connection allows up to 100 power actions at once.

When more power actions are running than there are connections, the Delivery Controller can no longer receive power state updates from XenServer, and it marks the hosting connection as lost. The connection test uses a different service with its own connections, so it still succeeds.

Resolution

Upgrade to Citrix Virtual Apps and Desktops 2607 LTSR.

Workaround

Raise the connection limit above the number of simultaneous power actions allowed on the hosting connection.

  1. In Studio, go to Hosting, select the XenServer connection, and choose Edit Connection.
  2. On the Advanced page, note the Simultaneous actions (all types) value.
  3. In Connection options, enter DefaultConnectionLimit=150. Use a value about 50 higher than the Simultaneous actions value. If the field already has options, separate them with a comma.
  4. Save the connection. The change takes effect automatically within a minute or two.

The option name is case sensitive.

To make the same change with PowerShell:

Set-HypHypervisorConnectionMetadata -HypervisorConnectionName "<connection name>" -Name "Citrix_MachineManagement_Options" -Value "DefaultConnectionLimit=150"

This command replaces any existing connection options, so include them in the value if there are any.

Issue/Introduction

When many machines are shut down or restarted at once, for example at the end of the working day or during an Autoscale schedule change:

  • The XenServer hosting connection shows HypervisorConnectionLost and is reported as unavailable.
  • All machines on that connection show a power state of Unknown.
  • A connection test on the hosting connection still succeeds.
  • The power actions still complete, and the connection recovers once they finish.
  • Some machines can still show the wrong power state after the connection recovers.

Additional Information

Confirming the issue. During a bulk shutdown or restart, run this command on the Delivery Controller, replacing <coordinator> with the IP address of the XenServer pool coordinator. Use port 80 if the connection uses HTTP.

Get-NetTCPConnection -RemoteAddress <coordinator> -RemotePort 443 -State Established | Where-Object OwningProcess -eq (Get-Process BrokerService).Id | Measure-Object

A count that repeatedly reaches 20 confirms this issue. After the workaround, the count can rise above 20.

Power actions still take a long time. The workaround stops the connection loss but does not make power actions faster. How quickly they complete depends on the XenServer pool. Raising Simultaneous actions or Maximum new actions per minute does not speed up a large batch. It makes more machines wait in a queue on the XenServer pool at once. If machines wait a long time in XenCenter with a pending operation, lower these values instead.

Wrong power states after recovery. Restart the Citrix Broker Service on the Delivery Controllers.