Randomly some of the Power Managed Server VDA's receive a shutdown request approximately 20 mins after the scheduled reboot and do not power on automatically and remain Turned Off.

Scout logs detect the following:

Some Power Actions are marked as failures by Broker possibly because VDAs are not registering within the timeout. XD Controller may shutdown these VDAs due to multiple registration failures. MaxRegistrationDelayMin timeout may need to be tweaked for some environments. Refer to http://support.citrix.com/article/CTX138738 for the registry keys that can be tweaked.For other Registration failures refer to https://support.citrix.com/article/CTX126992

As per the event logs on Delivery Controllers, the Scheduled shutdown shows the origin:Schedule. The second shutdown that is triggered due to registration timeout shows origin:Policy.

Below is the sequence of events logged on the Delivery Controllers.

04/12/2017 04:05:31    xxxxxxx    3013    Citrix Broker Service    The Citrix Broker Service successfully performed power action "Shutdown" (origin: Schedule) on virtual machine "yyyyyyy".    Information

04/12/2017 04:06:11    xxxxxxx    3013    Citrix Broker Service    The Citrix Broker Service successfully performed power action "TurnOn" (origin: Schedule) on virtual machine "yyyyyyy".    Information


2017-04-12 04:30:16    xxxxxxx    3015    Citrix Broker Service    The Citrix Broker Service marked a power action as failed because virtual machine "yyyyyyy" failed to register with the site after being started, restarted, or resumed.    Information


04/12/2017 04:30:18    xxxxxxx    3012    Citrix Broker Service    The Citrix Broker Service detected that power action "Shutdown" (origin: Policy) on virtual machine "yyyyyyy" failed. 
 
This problem is most likely due to a host issue. Check that the configuration of the virtual machine on the host does not prohibit the requested power action. Verify that there are no storage problems on the host. Check that the host connection information is correct. 
 
Error details: 
Exception "Operation not available due to missing tools" of type "PluginUtilities.Exceptions.OperationNotAvailableException".    Warning


04/12/2017 04:30:25    xxxxxxx    3013    Citrix Broker Service    The Citrix Broker Service successfully performed power action "TurnOff" (origin: Policy) on virtual machine "yyyyyyy".    Information

CDF Logs show below

04:23:26:40117,4972,4460,-1,FailOverdueRegistration(xxxxxxxxxx): Enqueue shutdown action result: Success, 1 failed registrations",""

1. We can increase the value of MaxRegistrationDelayMin. The default value is 20 mins.

 

 Or

2. Change the Restart Duration from "Restart all machines at the same time" to 30 mins or more depending on the number of VDA's in the Delivery Group.



For example, assume a Delivery Group has five servers, a Restart first group at time of 13:00 (1:00 pm), and a Restart additional groups every selection of 1 hour. That duration (60 minutes) is divided by the number of machines (five), which yields a restart interval of 12 minutes. So, the restart times are 1:00 pm, 1:12 pm, 1:24 pm, 1:36 pm, and 1:48 pm. This gives all five machines the chance to complete their restart at the end of the specified interval (1 hour)

Or

3. To find the root cause, investigate the reason for Registration timeout by collecting CDF logs from VDA's as well as Delivery Controllers.

 

Problem Cause

• https://docs.citrix.com/en-us/xenapp-and-xendesktop/7-6/xad-build-new-enviroment/xad-dg-create/xad-dg-manage-machines.html
• https://support.citrix.com/article/CTX137376
• https://support.citrix.com/article/CTX126704