The power state of several virtual machines is changing to "unknown," resulting in users being unable to connect.

Manually starting the affected Virtual Machines from the Hypervisor will sometimes resolve the issue.

This problem is happening multiple times a day across different machines, impacting multiple catalogs.

The broker connection throws the following error:

'A problem occurred while updating a broker host connection ('******') with information from the Citrix Host Service. Verify that the Citrix Host Service is running and configured. Verify that the specified host is reachable with the URL and credentials specified in the host connection. Error details: Exception 'Failed to create HCL plugin : Request Url = https://******/PrismGateway/services/rest/v1/cluster/ Request Body = , Request status : SecureChannelFailure, Exception :System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel. at System.Net.HttpWebRequest.GetResponse() at ******.PrismConnection.ResultFromHttpReques 'PluginUtilities.Exceptions.HostingInfrastructureCommunicationsFailureException'.

Following: https://support.citrix.com/external/article?articleUrl=CTX464102-citrix-hosting-connection-error-after-upgrading-nutanix-aos-cluster-to-52046-lts&language=en_US does not resolve the issue as the issue is sporadic and does not affect all VMs at once.

"Citrix is not responsible for and does not endorse or accept any responsibility for the contents or your use of these third party Web sites. Citrix is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement by Citrix of the linked Web site. It is your responsibility to take precautions to ensure that whatever Web site you use is free of viruses or other harmful items." 

On the DDCs we see Event ID 3032 for Citrix Broker Service

A problem occurred while updating a broker host connection ('Nutanix_Cluster_Name') with information from the Citrix Host Service.

Verify that the Citrix Host Service is running and configured. Verify that the specified host is reachable with the URL and credentials specified in the host connection.

Error details:
Exception 'Acropolis_Management :: GetMachineDetails() :: Exception: System.Exception: Request Url = https://Nutanix_Cluster_FQDN:9440/PrismGateway/services/rest/v1/hosts/
Request Body =,
Request status : ProtocolError,
Http Status:500,
Response: {"message": "Cookie authentication failed due to IDF lookup failure"],
Exception :System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
at System.Net.HttpWebRequest.GetResponse(
at NTNXVMProvisioning.PrismConnection.ResultFromHttpRequest(String requestUrl, String requestHeaderString, String jsonData, String clusterlp, HttpRequestType requestMethod, IDictionary 2
requestHeaders, String contentType, Int32 httpRequestTimeout)
at NTNXVMProvisioning.PrismConnection.ResultFromHttpRequest(String requestUrl, String requestHeaderString, String jsonData, String clusterlp, HttpRequestType requestMethod, IDictionary 2
requestHeaders, String contentType, Int32 httpRequestTimeout)
at NTNXVMProvisioning.PrismConnection.ConnectNutanixGETAsString(String requestUrl, String requestHeaderString, String clusterlp)
at NTNXVMProvisioning.AHVPrismVMInterfaces.GetHosts(PrismConnection prismConnection, Credentials credentials)
at NTNXVMProvisioning.AHVPrismVMInterfaces.GetHostsUuidAndName(PrismConnection prismConnection, Credentials credentials, Int32 threadld, Int32 requestld)
at AcropolisProvisioningPlugin.CommonOperations.getManagedMachine(Credentials credentials, String machineld, Int32 requestld, Logger logger)
at AcropolisProvisioningPlugin.AcropolisService.GetMachineDetails(ConnectionSettings connectionSettings, String machineld) ' of type 'PluginUtilities.Exceptionk.WrappedPluginException'.

Citrix CDF tracing shows this exception is repeated multiple times when the Broker queries the power state of the affected VM to ascertain if the VM is powered On or Off, until it eventually times out and marks the power state of the VM as "Unknown".

In the same trace, we can see the same type of query returning successfully either side of the error, which does not imply an invalid certificate or bad credentials

Toggling Maintenance Mode on the affected hosting connection temporarily resolved the issue, enabling end users to connect to affected VMs while the customer's IT staff were investigating the main issue.

The customer engaged Nutanix support to understand why the Nutanix Cluster was giving 503 errors when querying the power status of some VMs, whilst queries worked using the same credentials within a few minutes either side of the error.

The issue was caused by a performance issue on certain nodes of the Nutanix cluster that was resolved by the customer's IT staff working with Nutanix support. 

Citrix Virtual Apps and Desktops - The Power state of a VM in studio can randomly show as 'Unknown' when using Nutanix.