Old controller names in the Events for Broker Agent Service while making registration calls.

This software application is provided to you as is with no representations, warranties or conditions of any kind. You may use and distribute it at your own risk. CITRIX DISCLAIMS ALL WARRANTIES WHATSOEVER, EXPRESS, IMPLIED, WRITTEN, ORAL OR STATUTORY, INCLUDING WITHOUT LIMITATION WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NONINFRINGEMENT. Without limiting the generality of the foregoing, you acknowledge and agree that: (a) the software application may exhibit errors, design flaws or other problems, possibly resulting in loss of data or damage to property; (b) it may not be possible to make the software application fully functional; and (c) Citrix may, without notice or liability to you, cease to make available the current version and/or any future versions of the software application. In no event should the software application be used to support ultra-hazardous activities, including but not limited to life support or blasting activities. NEITHER CITRIX NOR ITS AFFILIATES OR AGENTS WILL BE LIABLE, UNDER BREACH OF CONTRACT OR ANY OTHER THEORY OF LIABILITY, FOR ANY DAMAGES WHATSOEVER ARISING FROM USE OF THE SOFTWARE APPLICATION, INCLUDING WITHOUT LIMITATION DIRECT, SPECIAL, INCIDENTAL, PUNITIVE, CONSEQUENTIAL OR OTHER DAMAGES, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. You agree to indemnify and defend Citrix against any and all claims arising from your use, modification or distribution of the software application.

We examined the affected VDA using following steps:

- Since in the traces we could see the oldDDc details are showing up as hardcoded on VDA.
  Verified ListofDDcs registry on the VDA was pointing to right set of controllers.
  
- There is a file located in C:\Program Data that contains historical data for the Broker Agent.
This files is located in:
C:\ProgramData\Citrix\PvsAgent\Locallypersisteddata\BrokerAgentInfo\SavedListOfDDCsSids.xml

This location , by default is only accessible by the SYSTEM account in windows. In order to make any changes the file, the admin must change the NTFS permission on the objects. 
First, the admin must take ownership of the C:\ProgramData\Citrix\PvsAgent\Locallypersisteddata folder and all sub-containers then allow read and write permission for the admin user.

Once access is granted to edit the file, view the file in a text editor in order to see if the old DDC name is in the file.
If the old DDC is found rename the file to SavedListOfDDCsSids.xml.old and restart the Citrix Desktop Service.
 OR
-Rename the file SavedListOfDdcsSids.xml in that folder : ren SavedListOfDdcsSids.xml oldSavedList
   
  - Restart Citrix Desktop Service and it should pick up the right controllers.

In order to resolve the issue on all affected machines if there are multiple in the environment, we can use te following script:

============================================
1 Create the following batch file with the name Ren.bat in c: drive.

Ren "c:\Program Files\Citrix\PvsVm\Service\PersistedData\BrokeragentInfo\SavedListOfDdcsSids.xml" oldSavedList
net stop brokeragent
net start brokeragent


2 Create a txt file (computer_list.txt) with names of VDAs in following format:
VDA1
VDA2
..

To copy the batch file to all affected computers: (use the following powershell)

$computers = Get-Content "C:\computer_list.txt"
$fileToCopy = "C:\ren.bat"
ForEach($computer in $Computers)
{
    Copy-Item -Path $fileToCopy -Destination "\\$computer\C$"
}


Post that run the following command from where Psexec is placed:

psexec @c:\computer_list.txt -s -d cmd.exe /c "C:\ren.bat"
===============================================================

---

Problem Cause

Persistent Data which is managed by PVSvm Service and Auto Update Controllers policy containing SIDs of old Delivery Controllers.

VDA is unable to get registered and Broker Agent events suggests that they are contacting wrong controllers which were probably older controllers and are decommissioned now. On trying to rename the file SavedListOfDDCsSids.xml under C:\Program Files\Citrix\PvsVm\Service\PersistedData\BrokerAgentInfo, we face permission issues since it is owned by System account. At times, we would not be able to take the ownership or change permissions to access and rename the suggested file.

Download link for PSEXEC tool:
https://docs.microsoft.com/en-us/sysinternals/downloads/psexec