PVS server SMB Client-Side Caching of PVS vdisk reads intermittently fails

book

Article ID: CTX695986

calendar_today

Updated On:

Description

When a PVS farm vdisk stores are configured to use SMB shares, PVS server operating system memory can be inspected to view SMB Client-Side Caching of PVS vdisks (using a tool like RAMMap):

  • Immediately after the PVS server boots, the operating system memory will not have any PVS vdisk files being cached by the SMB Client-Side Caching
    • PVS target devices booting at this time may take longer than during normal production time.
  • After some time, with PVS targets booting and streaming from the PVS server, the operating system memory will have PVS vdisk files cached in memory by the SMB Client-Side caching.
    • PVS target device vdisk reads will have lower latency, and PVS target devices may boot faster than earlier, as vdisk reads as satisfied by the SMB Client and now coming from SMB Client Side Cache, so from operating system memory, rather than across the network from the SMB server.

 

In a customer’s environment, it was observed that random PVS servers would spontaneously have target devices taking longer times to boot.

  • The PVS servers had not been recently rebooted and had on earlier inspection shown to have SMB Client-Side caching operating as expected, storing vdisk reads in local system memory.
  • Once the behavior occurred, inspection of PVS server operating system memory showed that the SMB Client-Side caching was using no memory for PVS vdisks.
  • This resulted in longer boot times for target devices and reduced target device performance for users.
  • Rebooting the PVS server enabled SMB Client-Side caching to work again as expected.

 

To create this behavior manually, a robocopy job can be used to copy a vdisk, a vdisk which has been cached in SMB Client-Side cache from earlier reads.

  • It will be observed on inspection of memory that this file copy job will disable the SMB Client-Side caching of this file until the operating system is rebooted.

Environment

"Citrix is not responsible for and does not endorse or accept any responsibility for the contents or your use of these third party Web sites. Citrix is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement by Citrix of the linked Web site. It is your responsibility to take precautions to ensure that whatever Web site you use is free of viruses or other harmful items." 

Cause

When a file is copied to/from remote SMB share, the Copy Engine sends an IOCTL_LMR_DISABLE_LOCAL_BUFFERING, which causes RDR (the client-side SMB redirector) to purge its local Client-Side Cache.

As a result, while the file will remain cached on the SMB server-side, but the SMB client side will end up purging its cache, and impact on the latency of subsequent reads of the file.

  1. The Client-Side Cache will be purged when copying a remote file that was cached by earlier reads.
  2. Then the subsequent reads after starting the copy must be satisfied from the source, which is the SMB server from across the network
  3. So all subsequent reads of that file stored on the SMB share will have higher latency than if they were satisfied using the local SMB Client-Side cache (satisfied from operating system memory).

Resolution

When it is observed that PVS servers SMB client is not caching reads from SMB shares, rebooting the PVS server operating system will immediately restore expected caching operation.

 

Should the behavior occur spontaneously but reliably sometime after each reboot of the PVS server operating system, an attempt should be made to identify and eliminate file copy jobs which send IOCTL_LMR_DISABLE_LOCAL_BUFFERING commands.

  • An example is eliminating robocopy jobs on PVS servers, where vdisk files are copied from the SMB share, as described above in the manual method mentioned for creating the behavior.
  • Ensuring configuration of Security Software exclusions on PVS vdisk files, to ensure Security Software is not a source of the issue.

 

Local vdisk stores are not affected by this behavior

  • Using local storage to the operating system for PVS vdisk storage eliminates the SMB Client-Side caching behavior from affecting PVS server vdisk read performance

Issue/Introduction

When PVS vdisks are stored on SMB shares, PVS relies on underlying SMB client operation to satisfy vdisk reads.

  • PVS vdisk reads may be accelerated by the SMB Client-Side Caching of PVS vdisk reads.
  • In some circumstances the SMB client may purge its cache, and vdisk reads will have higher latency than when SMB Client-Side Caching of PVS vdisk reads is active.

Additional Information

https://learn.microsoft.com/en-us/sysinternals/downloads/rammap