This is a Live article - updated frequently with the latest info and known issues

-------------------------------------------------------------------------------------------------------------------------------------------------

Recommended Teams version : latest version. Update history for Microsoft Teams App Version here.

The following article provides guidelines for troubleshooting the HDX optimization for Microsoft Teams in Citrix Virtual Apps and Desktops.

Citrix strongly recommends the latest current release Workspace app version. Most of the new features require an upgrade of the Workspace app, hence the Current Release model is the preferred delivery mechanism.

-------------------------------------------------------------------------------------------------------------------------------------------------
Live updates on known issues

• 12/19/23 Citrix has published a support article covering Teams 2.1 support including APM and App Layering https://support.citrix.com/article/CTX585013/microsoft-teams-21-supported-for-vdidaas .
• 12/13/23   Microsoft has released Teams 2.1 for VDI in early December 2023.  This version supports Optimization with the Citrix HDX Optimization engine (WebRTC).   See the following Microsoft support article for more information https://learn.microsoft.com/en-us/microsoftteams/new-teams-vdi-requirements-deploy.
• 9/25/23 Mac users running CWA 2308 who have the Secondary Ringer configured may occasionally encounter an issue where sounds for messages or ringtones may play continuously without stopping. To stop the sound from playing the user must disconnect the desktop session. To completely prevent this issue from happening users should not use Secondary Ringer in 2308 Mac CWA.
• 9/12/2023 On January 31, 2024, Microsoft will retire the Microsoft Teams support for Single-window UI when using VDI Teams optimization and support only the Multi-Window experience. Microsoft gave notice of this deprecation on 9/8/2023 in the M365s Admin Center (Post ID: MC674419).  Customers must upgrade their VDA and CWA to supported versions in order to retain Optimized video calling and screen sharing.

  Public details about the Multi-Window feature can be found in the Tech Community article: New Meeting and Calling Experience in Microsoft Teams!

• 9/8/2023 Microsoft has enabled Secondary Ringtone 100% for CWA release 2307.1 and greater. 
• 8/20/2023   Microsoft has enabled Simulcast 100% for CWA release 2305 and greater.
• 4/19/2023 - Microsoft has reverted changes it made to the Teams software which we jointly believe have mitigated the frequency of occurrence of the "floating" video or black square issue.  If this is still happening  please try restarting the Teams client.  If it is still occurring please file a support ticket with Citrix.
• 4/11/2023 – Several customers have reported problems with video in calls where the video is “floating” on the desktop window and / or may be a black square.  They are also reporting crashes in HDXRTCEngine.exe associated with this occurring.  Citrix and Microsoft are working together to get to the root of this issue and fix it ASAP. 
• 4/4/2023 - Desktop Sharing in older VDAs (1912 CU5 or below) that do not support multi-window mode may no longer allow sharing of more than one Desktop due to changes in Teams support for Single-Window mode.   Citrix recommends upgrading to at least 1912 CU6+ to get or retain this functionality.
• 4/4/2023 - Citrix has received reports that some desktop applications are not sharable in Optimized Teams.  If you have any Windows applications that are not shareable, please contact Citrix Support to troubleshoot this issue. 
• 4/3/2023 - The issue with CWA on Mac documented here impacts Screen Sharing and Video Conferencing on the Mac platform.  Please follow the instructions to upgrade your CWA to the fixed version to regain functionality.
• 3/16/2023 - Citrix has received new reports of HDXRTCMediaEngine.exe crashes during meetings with video and screen sharing enabled which we are investigating.  We suspect that this is related to the Intel Driver previously mentioned in this article.
• 3/16/2023 - Citrix has fixed a problem in the Javascript “Shim” that runs inside of Teams where some Point-to-Point calls were failing to connect sporadically.  This bug is fixed in the latest version of Microsoft Teams (Shim version 1.13.0.2).  To get the latest Shim update you must log out and log back into Teams and it will be automatically downloaded.
• 1/26/2023 - The issue with the Intel graphics device driver has been fixed.  Please contact your device manufacturer to download the latest version of the driver. See HDX Microsoft Teams Optimization Crash or Video Freeze on newer Intel-based clients for more information.
• 1/26/2023 - Background blurring and image effects are available in release 2301 for Mac.  This requires MultiWindow as a prerequisite  which is available in VDA 2112 or higher and Teams 1.5.00.11865 or higher.
• 1/26/2023 - Background blurring is available in release 2212 for Linux.  This requires MultiWindow as a prerequisite  which is available in VDA 2112 or higher and Teams 1.5.00.11865 or higher. Image effects will be delivered in a subsequent Linux release.
• 11/18/2022- Citrix is aware of an issue with graphics device drivers for Windows endpoints that have the Intel I5 1245U 1.6GHZ processor using the UHD Graphics 770 adapter. Endpoints using this driver can experience call freezes or disconnections in meetings or calls. See HDX Microsoft Teams Optimization Crash or Video Freeze on newer Intel-based clients for more information.
• 8/31/2022: Background Blurring and effects (default image templates) are now enabled in Ring 4 (GA) with CWA for Windows 2207. This requires MultiWindow as a pre-requisite (VDA 2112 or higher, and Teams 1.5.00.11865 or higher). 
• 7/27/2022: CWA 2207 for Windows is GA. This version will support Background Blurring and Effects once Microsoft rolls out a feature flag (check for updates here). Please note that MultiWindow is a pre-requisite (VDA upgrade to 2112 or higher) for Background Blurring and Effects.
• 6/17/2022: critical hotfix for 1912 LTSR, 2203 LTSR and CR VDAs (single session or multisession). In these VDAs, WebSocketService.exe can crash and cause Teams call failures ("Still connecting to remote devices. Calling is not available yet"). See CTX460173. [Internal reference: CVADHELP-17758]
• 6/7/2022: Simultaneous outgoing video and outgoing screensharing are now enabled in Ring 4 (GA). CWA 2109 or higher for Windows/Linux/Mac required. No VDA dependency.
• 6/6/2022: Background blurring and effects tech preview is now available with CWA for Windows 2206 EAR. No VDA or Teams versions dependency. The feature is controlled via client-side regkeys.
• 6/3/2022: MultiWindow (a.k.a pop-out chat, a.k.a new meeting experience) is now enabled in Ring 4 (GA). CWA for Windows 2112.1 or higher, CWA for Linux/Mac 2203 or higher, VDA 2112 or higher and Teams 1.5.00.11865 or higher required. A Teams sign out/sign in and restart will be required.
• 5/25/2022: Simultaneous outgoing video and outgoing screensharing are now enabled in Teams Public Preview Channel (a.k.a Ring 3.6, check here for how to enable your account). This feature requires CWA for Windows/Linux/Mac 2109 or higher. Thanks to this feature, users can share their webcam and screen at the same time. In the past, the webcam would have been disabled the moment a user also started to screenshare.
• 5/9/2022: Give and Take Control of shared content is now enabled in Ring 4 (G). CWA 2112.1 for Windows (or higher) and CWA 2203 for MAC (or higher) required.
• 5/2/2022: Live Captions are now enabled in Ring 4 (GA). CWA 2109 for Windows/Linux/Mac or higher required.
• 4/11/2022: WebRTC 1.0 is now enabled in GCC. See note from 3/7/2022.
• 3/22/2022: VDA 2103 can cause Teams users to fail to connect to a meeting (HDX-34394). Issue is fixed in VDA 2106 or higher.
• 3/7/2022: Gallery View performance improvements (WebRTC 1.0) are now enabled in Ring4 (GA). CWA 2109 or higher can now provide better user experience in conference calls with Gallery View (ability to switch resolutions more gracefully by the sender, which in itself smoothens bandwidth estimation drops and video freezes on the receiver side; and also faster ramp-up time to a desired bitrate/resolution at the beginning of the video call or after packet loss).
• 2/22/2022: Dynamic e911 is now available in Ring4 (GA). See edocs 'Microsoft Phone System' section for more details
• 2/10/2022: App Sharing is now available in Ring4 (GA). See edocs 'App Sharing' section for more details
• 2/10/2022: The Citrix API (colloquially known as shim) was auto-updated in Teams today to version 1.8.0.12. No action required by the IT Admins in either VDA, CWA or Teams client msi versions. Users might need to sign out/sign back in. This release introduced numerous fixes:
  • Self-preview video disappears after changing to different audio device in Teams UI/Settings/Devices [CVADHELP-17416]
  • Can't put PSTN call on hold for a second time [HDX-24447] [MS174365]
  • Toggling “Audio off” --> "Audio on" in Teams UI when joining a meeting does not work [HDX-35200]
  • “Audio off” doesn’t work when user is invited to join a meeting, so user still hears incoming audio [HDX-29206]
  • Performance improvements for enumerateDevice issues that could spike endpoint’s CPU [HDX-36752] [HDX-36494]
  • UI Performance improvements during video calls (sluggish Teams UI) [CVADHELP-18844] [HDX-37058]
• 11/05/2021 : VDA 1912 CU4 released, including critical fixes for CtxSvcHost.exe (Citrix HDX Teams Redirection Service) exiting unexpectedly because of a faulting CtxTeamsSvc.dll module, and also for CtxHdxWebSocketService.exe (HTML5 Video Redirection Service) exiting unexpectedly. As a result, Microsoft Teams might fail to load optimized ("Citrix HDX Not Connected") or optimized calls drop suddenly while in progress. These fixes are also available in VDA 2106 or higher [CVADHELP-16918] [CVADHELP-17341] [CVADHELP-17146] [CVADHELP-17505]
• 11/1/2021 : Citrix and Microsoft strongly recommend to update to a Teams version higher than 1.3.00.28778. Any version equal or lower will not be able to run new features for VDI. Public Microsoft Teams roadmap is available here.
• 9/29/2021: Workspace app 2109 for Windows, Linux and Mac is now GA. These releases include improvements in conference call performance, but the availability of this feature depends on an upcoming Teams update.
• 9/24/2021 : for Remote PC scenarios, Citrix strongly recommends Teams version 1.4.00.22472 or higher, as this version includes improved detection logic for HDX or local sessions to the client machine. Older Teams versions might disable certain features/UI elements otherwise for local sessions. Check edocs for further details.
• 9//9/2021: when optimized, drag and drop of files in the VDA into Teams chat fails and Teams might crash. Microsoft working on a fix (Microsoft case ref. 267297731 and 240731800).
• 6/25/2021: Call Quality Dashboard (CQD) in Teams Admin Center is now pulling up HDX optimized endpoint data
• 5/20/2021: New blog on Single Server Scalability considerations and testing. See here.
• 5/12/2021: VDA 1912 CU3 released, including important fix for WebSocketAgent.exe (Citrix HDX HTML5 Video Redirection Agent Service) stopping intermittently. As a result, incoming calls are not displayed in Microsoft Teams toaster (called party does not receive any notifications). [CVADHELP-15611]
• 5/11/2021: Large Gallery View 7x7 and Together Mode are now supported. Please sign out and sign back into Teams, no CWA nor VDA upgrade required. [Please note that Gallery and Large Gallery are very different - see edocs, The former is a 2x2 layout where the endpoint decodes 4 individual video streams, with maximum quality 360p. The latter is a 7x7 layout (i.e up to 49 participants) where the endpoint decodes a single composite video feed created by the Conference Server, with maximum quality 720p)
• 12/8/2020: Microsoft introduced a new public tech preview methodology for Teams. Citrix recommends you enroll.
• 11/15/2020: VDA 1912 LTSR CU2 released, including important fixes for bugs causing Teams to load unoptimized . See Release notes (HDX-25074)
• In some scenarios, even though "Citrix HDX Not Connected" is displayed, attempting to make a call will trigger Teams to connect to the Citrix VDA services and optimization will succeed. This is caused by a delay in the Teams UI refresh.
• If Teams is configured to Auto-start, the issue in the bullet above might also manifest. We recommend disabling auto-start by deleting the Teams regkeys in HKLM\Software\Microsoft\Windows\CurrentVersion\Run, HKLM\Software\WOW6432Node\Microsoft\Windows\CurrentVersion\Run and HKCU\Software\Microsoft\Windows\CurrentVersion\Run
• Note that currently it is not possible to disable Auto-start even if using the command line flag OPTIONS="noAutoStart=true" with the machine-wide MSI and the ALLUSER=1 property. Deleting the regkey described in the bullet above should fix this.
• If the Virtual Desktop does not have a GPU/vGPU, we recommend "Disable GPU hardware acceleration" in Teams/Settings in order to improve performance and scalability, otherwise Teams will use the VDA's CPU to render UI elements/animations. This setting ("disableGpu":true") is stored in %Appdata%\Microsoft\Teams inside desktop-config.json, so you could also use a logon script to edit that file and set the value to true.
• In some scenarios, when the VM's NTP time sync server is updated via GPOs, this can create a clock drift when compared to the time when the Citrix HTML5 Video Redirection service is started. This drift then translates into invalid/expired Citrix certificates and Teams will fail to load in optimized mode. We recommend changing the HTML5 Video Redirection Service ("CtxHdxWebSocketService") Startup Type to "Automatic (Delayed Start)" instead of the default "Automatic". Citrix fixed this problem in CVAD 2006 VDA. [HDX-24350]
• When installing Teams in a machine that has a VDA (either VM or Remote PC), Teams will disable background effects.
• Background blurring and live events are known limitations in VDI. Please contact Microsoft for timelines and more information.

-------------------------------------------------------------------------------------------------------------------------------------------------

Instructions

Please note that, generally, fully quitting Teams (Systray icon / right click / quit) and restarting it clears all the components involved and can fix whatever issue you were experiencing.

Also, keep in mind that Teams feature set is different when comparing optimized vs non-optimized. When Teams is not optimized, it runs on the VDA with its native media engine and all the features are supported (although multimedia rendering happens on the VDA, incurring in a CPU tax)

[Warning: If your VDA is 7.15 (any CU) and you still want to use Teams in unoptimized mode (the only mode supported in these VDA versions), scroll here. Teams HDX optimization will not be backported to the LTSR 7.15].

Step-by-step:

Microsoft Teams optimization mechanisms can be divided into 5 main areas

5. Monitoring
More information on how to leverage HDX Monitor to monitor Teams optimization can be found here.
 

Proper troubleshooting requires the Admin to verify each area progressively, moving down the list.

Please note that the presence of a Citrix Gateway as an HDX proxy should have no impact, so you can discard that from your troubleshooting. If you are also using the Gateway for VPN, make sure you are allowing the client machine to reach the O365 Teams servers.


#1: Media Engine instantiation

The quickest way to determine if the user is running in optimized mode is to look for HdxRtcEngine.exe running on the user’s client machine:



For Linux, run:
#ps -ef | grep -i hdxrtcengine

If the process is not found, then point #1.1 should be investigated first.




#1.1: Teams must launch in optimized mode

Click on the avatar / About / Version. “Citrix HDX Optimized" should be displayed. This means Teams is ready for an optimized call.



If the legend is not there, and instead you see "Citrix HDX Not Connected" then Teams did not load in VDI mode correctly.



Citrix HDX Not Connected means Teams loaded the Citrix API correctly, but it could not complete an end-to-end check because there was an error in subsequent parts of the stack. Most likely, VDA Services (Citrix HDX HTML5 Video Redirection, Citrix HDX Teams Redirection) or in Workspace app.

If there is no legend with the word Citrix, then Teams did not load in VDI mode at all:



The VDA regkey MsTeamsRedir is set by a Citrix service (CtxSvcHost.exe) when the user logs on or reconnects to the virtual desktop. Teams will read this key when starting, and if the value is 1, it will load in VDI mode.



The value is set to 1 automatically if and only if:

1. Microsoft Teams Redirection policy is enabled in Studio (and by default is Allowed)
2. CWA has loaded the Teams Virtual Channel (VDTeamsn.dll) and the CWA version is 1907 or higher

If = 0 or missing – trouble!

Action: Quit Teams from the systray itself (right click on the icon / Quit).
Sometimes, if Teams auto-updated while the session was in disconnected state, it can fail to read the key properly upon reconnections.

Lastly,  if the regkey value is 1 but Teams still fails to display "Citrix HDX Optimized" or "Citrix HDX Not Connected", there is likely a problem with Teams itself. Common issues could be erroneous cached items (settings.json and storage.json) in AppData\Roaming\Microsoft\Teams , or even dual Teams installations (under %appdata% and Program Files). Customers are advised to contact Microsoft support in this case.


#1.2: WebSocket phase

The VDA Service “Citrix HDX HTML5 Video Redirection” [WebSocketService.exe] is the point of entry for Teams into the VDA. In other words, Teams must open a secure websocket connection to this service, which listens at 127.0.0.1:9002 in the VDA.
If WebSocketService.exe is not actively listening, optimization will fail. In some cases, the Service will show as “Running” in the services.msc console, but when you run netstat you will not see the socket.
Therefore, you must make sure it is running and LISTENING :

`


After a successful connection, you will see the state change to “ESTABLISHED”:



[Note: WebSocketService.exe listens also at 9001 – this socket is used for other HDX features (Browser Content Redirection and HTML5 Video Redirection, and they are independent of Teams)]

-Proxies: If an explicit proxy server is configured in the VDA, make sure connections to localhost are not routed through it. Otherwise, redirection will fail. You must "bypass proxy servers for local address" 127.0.0.1:9002

-PAC files: make sure your VDA pac file script is returning DIRECT for wss://127.0.0.1:9002. If not, optimization will fail. One way to do this:

 shExpMatch(url, "wss://127.0.0.1:9002/*")

-Anti-Virus: Some security software products interfere with the proper functioning of WebSocketService.exe and its certificates. While the actual service (Citrix HDX HTML5 Video Redirection) might be running in the services.msc console, the localhost 127.0.0.1:9002 TCP socket is never in listening mode as seen in netstat. Trying to restart the service causes it to hang (“Stopping…”). Ensure you apply the proper exclusions for the WebScoketService.exe process.


Once there is a successful secure WebSocket (wss) connection, WebSocketService.exe will create a new process in the user’s session called WebSocketAgent.exe:



This process can also be spotted in Director:



[Note: WebSocketAgent.exe is also used for BCR and HTML5 Video Redirection]

Action: if WebSocketAgent.exe is not being created while 127.0.0.1:9002 is ESTABLISHED, restart the WebSocketService.exe service. Relaunching Teams might be required. Also, in some rare cases, you might see WebSocketAgent.exe coming up but closing within a second, and in this case this could be related to a crash in HdxRtcEngine.exe in the client machine (hence make sure to collect EventViewer entries on the client).

The last part of this second phase is the establishment of the Teams virtual channel (CTXMTOP), and this process is ultimately handled by the “Citrix HDX Teams Redirection” Service (CtxSvcHost.exe in the VDA).
CTXMTOP is a Command & Control virtual channel only – no media is exchanged between CWA and VDA. All the media is handled by CWA in a Client-Fetch-Client-Render manner.

Action: If WebSocketAgent.exe is running on the VDA but HdxRtcEngine.exe is not running on the client, restart “Citrix HDX Teams Redirection”.

#1.3: HdxRtcEngine.exe

Once the virtual channel is established, the Citrix HDX Engine in CWA (wfica32.exe) will spawn the media engine on the client:



At this point, Teams is ready to handle calls.

If HdxRtcEngine.exe launches, then the respective webrpc log will be created.

-Windows Logs:
They can be found on the user’s machine in %userprofile%\AppData\Local\Temp\HdxRtcEngine folder.
Each HDX session will create its own separate folder.

-Mac Logs:
1.    vdwebrtc log -- records the execution of Virtual Channel vdwebrtc
Location：  /Users/<User Name>/Library/Logs/Citrix Workspace/CitrixViewer_<Y_M_D_H_M_S>.txt

2.    HdxRtcEngine log -- records the execution of process HdxRtcEngine
Location: /Users/<User Name>/Library/Logs/HdxRtcEngine/hdxrtcengine/<W_M_D_H_M_S_Y>/hdxrtcengine.log
Note that  Hdxrtcengine log is enabled by default.

-Linux Logs:
The /tmp/webrpc/<current date>/ and /tmp/hdxrtcengine/<current date>/ directory will have log files.


#2:Media Engine peripheral acquisition

In Teams, go to Settings à Devices. All the peripherals should now be mapped, and the video self-preview visible. You should be able to toggle between the cameras too.
The "Make a test call" button is a very handy tool to assess peripheral acquisition and performance. Go to Settings/Devices/Audio devices.



Note: when Teams enumerates Playback/Recording peripherals, it will select the ones that are defaults at the endpoint. Open mmsys.cpl on the endpoint, and you can see what the default devices are in the user's client machine. These are the ones that will be auto-selected by Teams.
You can toggle these devices in the middle of a call - in that case, a 1 sec audio glitch might be experienced while the new devices are remapped.




The peripheral acquisition process is captured in the webrpc.txt logs.
All device enumeration entries will have “webrtcapi.RPCStubs” in the respective lines.

• worldaz.relay.teams.microsoft.com
• usaz.relay.teams.microsoft.com
• euaz.relay.teams.microsoft.com
• inaz.relay.teams.microsoft.com
• uaeaz.relay.teams.microsoft.com
• turn.gov.teams.microsoft.us
• turn.dod.teams.microsoft.us

• a local one describing itself ("webrtcapi.RTCPeerConnection Info: setLocalDescription”)

• a remote one describing the other peer (setRemoteDescription)

The VDI user caller will register remote ICE candidate and begin connectivity checks (STUN Bind request/response). Sometimes, the calling peer can be behind a Firewall that blocks STUN/UDP packets, and in such a case there will no answer to the STUN Bind requests:



[Warning: STUN Bind requests are essential for a healthy call setup, as this allows candidate discovery with optimal call routing (especially in p2p calls). If your branch office Firewalls are blocking STUN and UDP destination ports 3478-81, then a transport relay in O365 must be contacted using TCP which adds additional latency and degrades call quality experience].

Since the Firewall is blocking UDP, the calling peer must attempt a TCP connection to the other candidate:2
If that also fails, it will have to use his transport relay (“relay raddr”) seen in the Trickle ice snippet above (52.114.188.57).



Important note for branch-office proxies: HdxRtcEngine.exe cannot read explicit proxy configurations on the endpoint.

This was fixed in:

• CWA for Windows 2012 (Negotiate:Kerberos, NTLM, Basic/Digest)
• CWA for MAC 2104 (anonymous authentication only)
• CWA for Linux 2101 (anonymous authentication only)

or higher versions. In those releases, the HDX media engine will read the system proxy settings and honor the configuration (e.g. explicit proxies or pac files).


#3.3 RTP media flows
If the connection was established using TCP through a TURN server, you will not be able to decode the traffic in Wireshark as RTP since it is encrypted using TLS.

In case of a peer-to-peer connection or conference via UDP, then Wireshark will show more detailed info that can be used to understand the traffic flow between the participants.



Wireshark recommendations:

RTP cannot be automatically analyzed by Wireshark. Applying the Display filter ‘rtp’ won’t result in any RTP packets displayed.
Instead, first set the Display Filter to ‘udp’ to spot UDP traffic. After the connection is established, there will be UDP traffic sent regularly between the peers. RTP over UDP traffic will begin with 90 hexa (or 80 hexa if no RTP header extensions are present) in Data, the first octet of the RTP header.



By selecting one of these UDP packets and selecting "Analyze/Decode As" … then selecting RTP, Wireshark will process the packets as RTP.

The WebRTC media engine in Workspace app (HdxRtcEngine.exe) uses the Secure RTP and DTLS protocols for multimedia streams that are offloaded to the client. SRTP provides confidentiality and authentication to RTP, using symmetric keys to encrypt media and control messages, and uses the AES encryption cipher.


4. Media Engine multimedia stream processing

Once the call set up is finalized and the ICE connectivity checks are completed, the media will be exchanged between the parties.
Currently we support video resolutions up to 720p@30fps, with 360p being common [H264 and VP9].

-The video streams are captured in the webrp.txt logs as "DxVideoRenderer ":
webrtcapi.DxVideoRenderer Info: configure. window = 657170 frameWidth = 640 frameHeight = 360
-The webcam captures entries in webrp.txt are"CameraCapture":
webrtcapi.CameraCapture Info: start. capabilities: width = 640 height = 360 fps = 30 fmt = 9

Lip sync issues might be caused by high CPU usage on the endpoint. The encoding process (camera capture and subsequent encoding) is more expensive in CPU terms than decoding. Some endpoints might not have enough processing power (see here for system requirements), resulting in 100% CPU utilization and causing audio/video to be out of sync.

The RTP protocol can be decoded in Wireshark to understand what type of codecs are being used and how much bandwidth is consumed.

#4.1 As an example we will investigate a p2p audio call between two VDI users. HdxRtcEngine.exe will select the Opus codec for this scenario, which can be seen clearly in the SDP Offer/Answer negotiation.
The calling party will state all his supported codecs, and the callee will then reply with his preferred choice in the SDP answer.

[Note: The notification audio (ringtone) is played by Teams out of the VDA’s default playout device.]



The call bandwidth usage can then be plotted in Wireshark using the Statisticsà IO Graph option.
Apply the proper filters to display only the traffic associated with the call.
We can see that the typical bandwidth for a voice call is ~40-70 kbps (each way).


Green: outgoing | Blue: incoming | Red: outgoing + incoming


Screensharing

In the Skype for Business optimization pack (RTOP), screensharing was not optimized. In other words, there was no offloading of such streams and therefore they were rendered server-side natively by Skype for Business.

But in terms of HDX optimization for Microsoft Teams, incoming screensharing is no different than incoming webcam video streams, since Teams relies on video-based-screen-sharing. Hence it is optimized by default.

However, a VDI user sharing his monitor (i.e. outgoing screensharing) will result in HdxRtcEngine.exe performing display captures at the endpoint (with H.264). Currently it is only possible to share the primary (default) monitor with CWA 2105 or older.
HdxRtcEngine.exe crops the sharing to the Desktop Viewer window only (if CWA's Desktop Viewer is in Windowed mode).



If Desktop Viewer (cdviewer.exe) is in full screen mode, then of course the entire monitor is shared.
In multi-monitor setups, only the primary monitor is shared, and the screen picker button in Teams will label the preview as 'Screen #1'.



Multimonitor setups are now supported with CWA 2106 for Win/Linux/Mac or higher.
In the Windows case, Desktop Viewer must be enabled in StoreFront/ICA file and in full screen mode in order for the picker to display all available monitors.

Troubleshooting screensharing issues:

Common issues with screensharing, as perceived by the other peer, are:
- low resolution or low fps
- black window instead of the actual screen
- screen freeze from the sender's side

The webrpc.txt logs (AppData\Local\Temp\HdxRtcEngine in the user's device, not the VDA) are the best source for troubleshooting such issues.
The outgoing screensharing flow is as follows:

[CreateDisplayTrack --> VideoTrackSource (id = ########) --> startVideoTrackSource --> DesktopCapture.start]

Oct 01 10:06:24.222 webrtcapi.WebRTCEngine Info: createDisplayTrack. id = DV-9d1651420da4 sourceId = 
Oct 01 10:06:24.223 webrtcapi.VideoTrackSource Info: VideoTrackSource. id = 03B28CB8 source-id = 0 width = 1920 height = 1080 fps = 15
Oct 01 10:06:24.224 webrtcapi.MediaStreamTrack Trace: MediaStreamTrack. video track-id = DV-9d1651420da4 label = display-track state = 0
Oct 01 10:06:24.224 webrtcapi.WebRTCEngine Trace: startVideoTrackSource. first capture consumer, source-id = 0
Oct 01 10:06:24.224 webrtcapi.DesktopCapture Info: start.

HdxRtcEngine.exe contains a bandwidth estimator that will allow Teams to pick the best resolution that can be sustained by the network, and sometimes it can take a few seconds for it to converge to the right value. So screensharing might actually start with a low resolution before it jumps to a higher one.

In the webrpc logs, look for "webrtcapi.VideoTrackSource Info: setOptions" to spot what resolution and fps is being applied:
 

In the logs, also the parameter "webrtcapi.DesktopCapture" can be used for quick finds, and if there are errors they would be logged as "webrtcapi.DesktopCapture Error:". 
Black windows can be caused by issues on the sender's side (user capturing) or the receiver's side (user rendering the stream).
For example, if the error is on the sender's side the following entry could be seen:
webrtcapi.DesktopCapture Error: OnCaptureResult. invalid cropping rectangle (x = 0 y = 0 width = 0 height = 0)



Monitoring Teams with HDX Monitor

On 1912 VDAs (or higher), you can monitor an optimized active call using HDX Monitor (3.5.2 or higher).
Important : make sure you are matching the HDX monitor version with the right VDA version (HDX Monitor is packaged with the CVAD .iso)
To turn this on, create the following WebrtcDirectorIntegration regkey on the VDA:

HKLM\Software\Citrix\HDXMediaStream\
Or
HKLM\SOFTWARE\WOW6432Node\Citrix\HdxMediaStream
- reg key value:
name: WebrtcDirectorIntegration
type: DWORD
value: enable(1), disable(0)


The information is displayed under the WebRTC icon :



The following data content will be populated when an active Teams redirection is going on.
Please see below for the detailed field description:

• Call Device Used Audio: audio device used for the last active call
• Call Device Used Video: video device used for the last active call
• Call Direction: incoming or outgoing
• Call Duration Audio: last active audio call duration in seconds
• Call Duration Video: last active video call duration in seconds
• Call Duration Conference: last active conference call duration in seconds
• Call Duration Screenshare: last active desktop sharing call duration in seconds
• Call Establishment Duration Incoming: last active incoming call establishment in sec
• Call Establishment Duration Outgoing: last active outgoing call establishment in sec
• Call State: Active or Idle
• Call Type: Audio,  Video, Conference, Screenshare
• Version - Endpoint OS: not filled currently
• Version - Receiver: CWA version
• Version - Typescript: API version
• Version - Vda: not filled currently
• Version - Webrpc: webrpc version
• Version - WebrtcCodec: webrtc codec version

Guidelines for 7.15 LTSR

If you are using 7.15 LTSR (with any CU), please note that the new HDX Optimization for Microsoft Teams is not (and will not be) available in this version and the only way to use Teams is through the legacy HDX technologies:

• Client Audio Redirection (we recommend using the RTP/UDP policy, which requires the Audio quality to be set to Medium)
• Client Microphone Redirection
• Multimedia Conferencing (for Webcam redirection, a.k.a HDX RealTime Webcam Video Compression)
• Thinwire (Server-side rendering of Video feeds)

Known Issues with Microsoft Teams and 7.15 VDA:

• Webcam redirection will not work with 64-bit Teams, only with 32-bit. This is known limitation in 7.15 (any CU) - see here for more details.
• In Windows Server, Teams.exe might crash, or the Teams logon screen might become blank and no input fields are presented to the user. Please add Teams.exe to the VDA registry under SfrHook, and make sure the Flag value REG_DWORD is set to 204 Hexadecimal  
• After you add the SfrHook key, you must reboot your VDA.
• As a last resort, you can disable ALL hooking for Teams by following CTX107825

Limiting Video Resolutions 

Windows

• 3/23/2022: CWA for Windows 2203 LTSR and 2204 CR are now GA. These versions allow App Protection users to be able to see incoming video and screensharing, and also introduce support for Secondary Ringtone (pending update from Microsoft, refer to edocs).
• 3/15/2022: A fix for the issue below has been rolled out by Microsoft. Users are required to sign out / sign back in or Exit Teams.
• 3/14/2022: Customers in Public GCC or GCC High Office 365 Rings might be experiencing intermittent webcam failures (Teams detects the camera but not able to acquire video) if the Workspace app is equal or higher to 2109 (on any OS platform). This issue has been reported recently in March. Microsoft is investigating [Microsoft IcM Reference #294654482] [Citrix Reference CVADHELP-19623]
• 3/8/2022: Fix for P2P call failure is now completely rolled out by Microsoft. No Teams client/Workspace app/VDA upgrade required (the fix was applied to the Teams backend services)
• 2/25/2022: P2P calls (not conference) between an HDX optimized Teams user and a user on native Teams desktop client 1.5.00.2164 or higher can intermittently fail a few seconds after establishment. User would see "Hold on, looks like something went wrong – we are trying to get you back on the Call", the call would resume and subsequently interrupt. The issue is seen irrespective of the CWA or VDA versions. The problem is not seen in P2P calls between two optimized users. Citrix and Microsoft are investigating. References for Microsoft IcM# cases (279646315 and 291199411) and Citrix SR# cases (81036409 and 81046791)
• 2/4/2022: App Sharing (i.e. sharing a single app and not a full desktop) is now available in Ring 3.6 (Public Preview Channel). This is an opt-in program, controlled via IT admin policy. Ring 4 (GA) is not yet available.
• 12/9/2020: Gallery View (2x2) is now on public tech preview. See here for more details.
• 12/8/2020: Microsoft introduced a new public tech preview methodology for Teams. Citrix recommends you enroll.
• 12/8/2020: Citrix Workspace app for Windows 2012 is GA, adding support for proxy servers and mouse pointer capture. See Release notes.
• 12/2/2020: Citrix Workspace app for MAC 2012 is GA, adding support for screensharing. See Release notes.
• 11/15/2020: VDA 1912 LTSR CU2 released, including important fixes for bugs causing Teams to load unoptimized . See Release notes.
• 11/11/2020 : "Make a test call" is now available under Teams/Settings/Devices/Audio devices. Sign out / sign back into Teams required.
• 11/2/2020 : Citrix Workspace app for Windows 2010 is GA. See fixed issues (around screensharing)
• 10/29/2020 : Citrix Workspace app for Linux 2010 is GA. See fixed issued
• 10/15/2020 : support for DTMF has been enabled in Teams. Sign out / sign back into Teams required. (Workspace app for Windows 2006.1 or higher)
• 10/8/2020 : a bug has been reported in Workspace app for Windows 2009.6 for outgoing screensharing failures (in setups where the monitor has high DPI or multi-monitor with high resolutions). Fixed in 2010.
• 9/24/2020 : Citrix Workspace app for Windows 2009.6 GA. See Release notes.
• 9/23/2020 : Citrix Workspace app for MAC 2009 supports Teams optimization
• 9/8/2020 : In Windows Server (any version, with any VDA 1906.2 or higher), session disconnect/reconnect might result in Teams failing to load in optimized mode ("Citrix HDX Not Connected" in About/Version). Citrix and Microsoft rolled out the fix on Sept-8-2020, deployed already in GA (a.k.a Ring4). This did not require any component upgrade (Teams, VDA or Workspace app) and was rolled out via feature flags (only sign out/sign in required). [RFWIN-15624] [Microsoft Teams Bug ID: 873138].
• Teams version 1.3.0.13565 now includes a feature that can control the fallback mechanism from optimized to non-optimized (e.g. disabling Webcams or disabling both audio and webcams so there is no impact to server-side resources). See "fallback" section in edocs.
• 8/22/2020 : Timezone Redirection mismatch when Teams is running in VDI: please upgrade Teams to get the fix (to version 1.3.00.21759)
• 8/16/2021: multimonitor screensharing is now available with CWA for Win/Mac/Linux 2106 or higher. Log out/log back into Teams required. Edocs link here.
• 8/12/2021: Teams Keyboard Shortcuts such as Ctrl+E (Global Search box) , Ctrl+F (Find in existing context), etc. are now working in VDI. Log out/log back into Teams required.
• 8/11/2021: CWA 1912 CU5 is now available. Customers using CWA 1912 are encouraged to test and update their clients, since CU5 contains numerous bug fixes and performance improvements not available in any previous CU. Release notes here.
• 8/11/2020 : Optimization is now available for Microsoft Teams in GCC High
• 8/10/2021: CWA 2108 for MAC is now available. This release fixes a bug where an optimized user from MAC is seen by other peers in the conference call in slow motion (10x speed reduction).
• 8/10/2021: DTMF (CWA for Windows 2102, Linux/Mac 2101 or higher and CWA for Windows 1912 CU5) is now available. Please sign out and sign back into Teams.
• 8/9/2021: Hidden chat channels are now visible/clickable in optimized Teams when running in a virtual desktop - Microsoft issue (incident ref 241949910)
• 7/21/2020 : Screensharing from Chat window is now available in GA (Ring4). Sign out and sign back into Teams to enable the feature.
• 6/16/2021: Citrix Workspace app for Windows 2106 is available, addressing a bug where incoming video feeds might flicker [CVADHELP-17398]
• 5/25/2021: Bandwidth estimation improvements rolled out in Teams for better performance/video quality in Gallery View 2x2 (sign out/sign in to Teams)
• 5/24/2021: Gallery View 2x2 is now also available in Microsoft 365 Government - GCC High
• 5/23/2021: Hidden chat channels are not visible in Teams when running in a virtual desktop - Microsoft issue (incident reference 241949910) ; currently working on the fix)
• 5/10/2021: Citrix Workspace app 2105 for Windows is GA (release notes here). This release fixes incoming video/screensharing seen as black screens, failure to receive video feeds when joining a meeting from Outlook, video rendering flickers and HdxRtcEngine crashes in p2p calls (HDX-30176).
• 4/01/2021: endpoint/client devices with multiple NICs (e.g. VPN, Wireless, Ethernet) might experience optimized webrtc traffic not being routed through the proper interface, hence degrading call quality because of routing metrics and different Layer 3 performances (CVADHELP-16480). This was fixed in CWA 2105 for Windows.
• 03/23/2021: Citrix Workspace app (Windows 2103.1, Linux 2103, Mac 2102) does not support VP9 anymore for p2p video calls. H264 will be used instead for every type of call now (meetings or p2p), preserving client CPU.
• 03/17/2021: Endpoints with quad-channel microphones (e.g. HP EliteBook laptops) might fail to capture audio with CWA for Windows 2009.6 or higher, up to 2103 (HDX-29725). This was fixed in CWA for Windows 2105 or higher, and it requires the creation of a client-side REG_DWORD regkey with Value 1 in [HKCU\Software\Citrix\HDXMediaStream ; Key: DisableADM2].
• In Windows 10 VDAs, disconnecting from a session by using the tsdiscon.exe Microsoft utility (as described in CTX225970 option#2) will cause Teams to load un-optimized when reconnecting to the session. We recommend using the Desktop Viewer toolbar for disconnecting.
• In P2P calls, VDI users might experience a 'black screen' when another peer shares his screen. If the user then minimizes the Teams UI (e.g. by clicking on the chat button) so it is positioned in the upper left corner, the screen share becomes visible. Maximizing it shows the black screen again. Sign out/Sign in back into Teams fixes the issue. [CVADHELP-14716]
• In some scenarios, even though "Citrix HDX Not Connected" is displayed, attempting to make a call will trigger Teams to connect to the Citrix VDA services and optimization will succeed. This is caused by a delay in the Teams UI refresh.
• If Teams is configured to Auto-start, the issue in the bullet above might also manifest. We recommend disabling auto-start by deleting the Teams regkeys in HKLM\Software\Microsoft\Windows\CurrentVersion\Run, HKLM\Software\WOW6432Node\Microsoft\Windows\CurrentVersion\Run and HKCU\Software\Microsoft\Windows\CurrentVersion\Run
• Note that currently it is not possible to disable Auto-start even if using the command line flag OPTIONS="noAutoStart=true" with the machine-wide MSI and the ALLUSER=1 property. Deleting the regkey described in the bullet above should fix this.
• With Workspace app 1912 or 2002, peer-to-peer calls (not conferences) might result in lower encoding resolutions for the VDI user sharing his webcam, depending on the endpoint CPU. You can try either disabling VP9, or forcing the resolution to medium. See here for instructions. Citrix fixed this issue in Workspace app for Windows 2006.1 [RFWIN-17890]
• Endpoints with high-end sound cards (Studio quality) might fail to load in optimized mode. See section #2.1 below. Clicking on Teams/Settings/Devices will show "None". Citrix is aware of this issue and working on a fix in Workspace app. [RFWIN-17892] [Issue fixed in Workspace app for Windows 2009.6]
• If the Virtual Desktop does not have a GPU/vGPU, we recommend "Disable GPU hardware acceleration" in Teams/Settings in order to improve performance and scalability, otherwise Teams will use the VDA's CPU to render UI elements/animations. This setting ("disableGpu":true") is stored in %Appdata%\Microsoft\Teams inside desktop-config.json, so you could also use a logon script to edit that file and set the value to true.
• In some scenarios, when the VM's NTP time sync server is updated via GPOs, this can create a clock drift when compared to the time when the Citrix HTML5 Video Redirection service is started. This drift then translates into invalid/expired Citrix certificates and Teams will fail to load in optimized mode. We recommend changing the HTML5 Video Redirection Service ("CtxHdxWebSocketService") Startup Type to "Automatic (Delayed Start)" instead of the default "Automatic". Citrix fixed this problem in CVAD 2006 VDA. [HDX-24350]
• When installing Teams in a machine that has a VDA (either VM or Remote PC), Teams will disable background effects.

********************************************************************************************