DaaS | Cloud Connector often have Event ID 3001 - Citrix Remote Broker Provider (Websocket unavailable or Closed by Server)

book

Article ID: CTX695810

calendar_today

Updated On:

Description

If your seeing [EventID: 3001] & [Event ID: 3003 – “NewStatePendingHA”] on your Cloud Connector as a once off (no repeat event over a space of 5 minutes across any other CC) – This will not trigger the issue
 
It would indicate at that time a CC did have a network issue (and it recovered on its own) – but it would elect a new CC as a primary and communication would still be stable across your environment
 

If you are seeing:

" The underlying connection was closed: A connection that was expected to be kept alive was closed by the server"
" Unable to read data from the transport connection"

- These errors indicate Network interruption
 
 
 
Anytime there's an issue with WebSockets not being able to establish connectivity or continued connectivity, it comes down to a customer-owned networking component like Firewall or some rule/proxy configuration that doesn't allow that specific traffic or does not allow ongoing WebSocket connectivity. 

- Our logs will at the maximum tell us the WebSocket connections are disconnected but won't get into Network level details to outline why the WebSockets are disconnected
- The connector to initiate a WebSocket connection and then wait for some time before sending the Application data. 
- However, The WebSocket connection should remain persistent and the Connector sends a Keep Alive message every 60 seconds to ensure the WebSocket connection stays alive. 
- If the TCP Timeout at the customer firewall or networking components is less than 60 seconds, the WebSocket connection gets broken.
 
 

Cause

This event can occur gracefully (as a once off event), when the Cloud DDCs are rebalancing (Electing a new Primary changing the previous DDC to a Secondary)

- This would close the existing Websockets with the existing Cloud Connectors & they would need to establish a new WebSocket connections

- However if this occurs frequently throughout the day (Network Team would need to investigate why the communication / packets are being dropped on your network to Target Url: https://<CustID>.xendesktop.net/scripts/wpnbr.dll

- Be it due to Proxy configurations, SSL inspection enabled or due to any malware or monitoring software running.

Resolution


 These Event 3001 from Citrix Remote Broker Provider would get log when there are minor Network glitch.
 
- If this event is logged very frequently, you  would need to check the status of Cloud connector.
- If cloud connector is not healthy or end users are facing issues, your Network team should investigate the issue with Network and any Network level intermediate devices by capturing Wireshark traces.
- If there are any pointers towards Citrix components and need assistance, we can collaborate with them and assist you further

Issue/Introduction

If your seeing [EventID: 3001] & [Event ID: 3003 – “NewStatePendingHA”] on your Cloud Connector as a once off (no repeat event over a space of 5 minutes across any other CC)

- This will not trigger any outage.



As per our Docs: https://docs.citrix.com/en-us/citrix-daas/manage-deployment/local-host-cache.html

"The 3001 events on your Cloud Connectors periodically throughout the day are typically not a cause for concern. However, if they’re occurring multiple times an hour, then it indicates a network issue and might warrant further investigation."