Unable to manage PVS farm, we get ‘Server connection timeout’ error.

To deal with AD group search too slow or timeout issue, below regkey allow customers to select the domain search approach,See below description:

create a DWORD registry "DomainSelectOption" under HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\ProvisioningServices. Set it to:

1)     "DomainSelectOption",  Value:

0= use simple search in user domain only (default).

1= use advanced cross-domain search (Use only when default setting cannot find the correct PVS admin membership. Note: Un-restricted cross domain search may take time.

2= use advanced search in "white-listed" domains. Only the white-listed domains will be used for cross-domain group enumeration.  The white-list domains can be configured in a "whitlelist.json” file.

     if no file is provided, we will create default list with computer domain and user domain by default. The domains listed in the blacklist json file will not be used in domain enumeration nor group enumeration.

The white-listed domains are the domains the user knows that the PVS Admin groups (and their nested groups) are located. These are provided in a JSON file named "whitelist", which should be put into the "C:\ProgramData\Citrix\Provisioning Services" folder.  Data in the file should have the format as show below (for example):

{

 "Domains":

 [

  "sub.xs.local",

  "sb.xs.local"

 ]

}

Problem Cause

From PVS AOT tracing we see that PVS server is unable to fetch the group membership from all domains as PVS is unable to lookup few domains.

295,8,2020/05/21 04:57:22:99267,8056,4360,1,PVS_Dll_ADSupport_AO,,0,,3,Warning,"Could not find domain DOMAIN1.COM, Exception = System.Runtime.InteropServices.COMException (0x8007203A): The server is not operational.

   at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail)

   at System.DirectoryServices.DirectoryEntry.Exists(String path)

   at ADSupport.PublicFunctions.FindUnreachableDomains(List`1& unreachabledDomainList)    [StartupADThread.cs(558): FindUnreachableDomains]",""

296,10,2020/05/21 04:57:22:99526,8056,4360,1,PVS_Dll_ADSupport_AO,,0,,3,Warning,"Could not find domain DOMAIN2.COM, Exception = System.Runtime.InteropServices.COMException (0x8007203A): The server is not operational.

   at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail)

   at System.DirectoryServices.DirectoryEntry.Exists(String path)

   at ADSupport.PublicFunctions.FindUnreachableDomains(List`1& unreachabledDomainList)    [StartupADThread.cs(558): FindUnreachableDomains]",""

297,6,2020/05/21 04:57:49:30918,3900,4360,1,PVS_Dll_ADSupport_AO,,0,,3,Information,"Error looking up GUID in DOMAIN3.COM domain. Restarting background AD thread.    [ADSupport.cs(2908): SearchForADCanonicalFromGuid]",""

298,10,2020/05/21 04:57:49:31826,3900,4360,1,PVS_Dll_ADSupport_AO,,0,,3,Information,"Error looking up GUID in DOMAIN4.COM domain. Restarting background AD thread.    [ADSupport.cs(2908): SearchForADCanonicalFromGuid]",""

299,1,2020/05/21 04:57:54:69212,7220,3760,1,PVS_Dll_EnterpriseAccess_AO,,0,,1,Error,"CommunicationObjectFaultedException: The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.",""

300,0,2020/05/21 04:57:54:69303,8548,3760,1,PVS_Dll_EnterpriseAccess_AO,,0,,1,Error,"TimeoutException: This request operation sent to net.tcp://localhost:54322/pvs/mapi/commandset did not receive a reply within the configured timeout (00:02:00).  The time allotted to this operation may have been a portion of a longer timeout.  This may be because the service is still processing the operation or because the service was unable to send a reply message.  Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client.",""

301,0,2020/05/21 04:57:54:69309,8548,3760,1,PVS_Dll_EnterpriseAccess_AO,,0,,1,Error,"TimeoutExceptionHandler    [ErrorHandlers.cs(121): dumpToLog]",""