Introduced in XenServer 5.6 SP1, IntelliCache is designed to reduce the volume of storage I/O hitting the network and the storage repository (SR) shared with other hosts. Introduced in XenServer 6.5, in-memory read-caching is designed to reduce the volume of storage I/O hitting an SR and reduce I/O latency. When used together, in-memory read-caching and IntelliCache combine to provide a powerful solution that can dramatically reduce the rate of both network and local disk I/O to deliver low-latency storage to VMs. This article explains how these two technologies relate to each other as complementary caching technologies.

What kinds of I/O are cached?

In the context of a collection of VMs where some VMs are cloned from a common source, the VMs' disks share common base image(s). Data that is common to all VMs resides in the base image; all other data resides in a disk image specific to that VM.

IntelliCache caches both reads and writes of data in both the base image and the particular disk images. This means it can cache both data that is unique to a particular VM and data that is shared between several VMs.

By contrast, in-memory read-caching only caches reads of data in the base image (i.e. it does not cache any writes, nor any reads of data specific to a particular VM).

IntelliCache has two modes of operation depending on whether the VM's disks are persistent or not (corresponding to XenDesktop's "static/private image" and "random/shared/pooled image" Desktop Experience modes respectively). In the case of persistent VM disks, data is always written through to the remote SR as well as being cached on the local physical disk.

What kinds of VM can take advantage of caching?

The in-memory read cache is used for VM disks in NFS (or Local EXT) SRs. Only data in read-only disk images is cached. This means that:

• all the disk data that is shared between VMs cloned from a single base image can be cached;

• a VM with at least one snapshot will use read-caching on the data that has not changed since the snapshot was created; and

• a VM created from a template with the "fast clone" operation will use read-caching on the data that has not deviated from what was in the template.

For XenDesktop, MCS-provisioned VMs can use the cache if they are in an NFS SR, since all VMs are cloned from a single base image, which is marked as read-only. PVS-provisioned VMs use the network rather than a XenServer SR for their base disks so can only use PVS-specific caching solutions.

Non-XenDesktop VMs will automatically take advantage of read-caching when you have more than one clone of a VM, with only the data that the clones have in common being cached.

Similarly, IntelliCache only works for VMs in NFS (or Local EXT) SRs. However, it is trickier to set up correctly so is normally only used when enabled by XenDesktop with MCS provisioning.

How much data can be cached?

The in-memory read cache automatically uses available dom0 RAM to cache reads. Allocation of more memory to dom0 means there is the potential to cache more data. However, there is a trade-off because a larger dom0 memory allocation means a lower amount of available host memory in which to run VMs. Article CTX134951  - How to Configure dom0 Memory in XenServer 6.1 and Later describes how to change the amount of memory allocated to dom0

By contrast, the IntelliCache cache uses a local SR, so can be as large as any individual disk or block device presented by a hardware RAID controller.

While IntelliCache does not actively evict data cached on local disk, in-memory read-caching will actively expire cached data when there is memory pressure inside dom0, favouring the retention of the most frequently accessed data. (Data cached by IntelliCache is removed when the virtual disk is deleted or reset.)

If both IntelliCache and in-memory read-caching are enabled, which cache takes precedence?

Data read for the first time by the VM is read from shared storage and a copy is retained in local storage for the lifetime of the VM if there is space on the disk. A copy is also retained in the in-memory cache if it is read from a read-only base image.

When the same data is re-read, first the in-memory cache is checked. If the data is present, that is used, and no I/O is issued to either local or shared storage. If the data is absent, because it has been evicted to make space for other data, the local disk is used. If the data is present, that is used and again it is retained in the in-memory cache, and no I/O is issued to shared storage. If it is absent, because the local disk was full, I/O is issued to shared storage.

When data is written, any corresponding entry in the in-memory cache is invalidated. If it's a write to a non-persistent disk, the I/O is issued to the local disk only. If it's a write to a persistent disk, the I/O is issued to both the local disk and the shared storage.

These data paths can be summarized in the following table and shown pictorially in the following figures.

Figure 1: Write to a persistent VDI.

Figure 2: Write to a non-persistent VDI

Figure 3: Read of data cached in memory.

Figure 4: Read of data cached by IntelliCache.

Figure 5: Read of data that is presently uncached.