Summary

The Session Recording feature from Citrix is a highly scalable session recording system. Competitors’ products define scalability in terms of hundreds of sessions but Session Recording handles thousands or tens of thousands of sessions. This makes it the only true enterprise-scalable recording solution available, and Striped in most cases, this is achieved without any additional hardware or administration costs.

This article explains how Session Recording achieves high scalability and how an IT manager or administrator can get the most out of their recording system at the lowest cost.

Why Session Recording scales so well

There are a number of reasons Session Recording scales so well compared with competitive products. The main reason is file size. A recorded session file made with Session Recording is highly compact. It is many orders of magnitude smaller than an equivalent video recording made with solutions that “screen scrape”. The network bandwidth, disk space, and disk IOPS required to transport and store each Session Recording file is typically at least ten times less than an equivalent video file.

The other major reason is the low processing required to generate files. A Session Recording file contains the ICA protocol data for a session that is extracted virtually in its native format. This means the file captures the ICA protocol data stream that is used to communicate with Citrix Receiver. There is no need to run expensive transcoding or encoding software components to change the format of data in real time. This low amount of processing is also very important for XenApp Server OS VDA scalability and ensures the end-user experience is maintained when many sessions are recorded from the same VDA.

These scalability benefits do not compromise functionality. In fact, in many cases functionality is improved. The small size of Session Recording files means faster and smoother rendering of video frames. The recordings are also completely lossless and have no pixilation common in most compact video formats. This makes the text in recordings as easy to read during playback as in the original session. To maintain small file sizes, Session Recording does not record key-frames within the file. 

Only those ICA virtual channels that are capable of being played back are recorded, resulting in a further optimization. For example, the printer and client drive mapping channels are not recorded because they can generate high volumes of data without any benefit in video playback.

Recording sessions

The Session Recording Server is the central collection point for recorded session files. Each machine running the XenApp Server OS VDA with Session Recording enabled sends recorded session data here. Session Recording is designed to handle high volumes of data and can tolerate bursts and faults, but there are physical limits on how much data any one server can handle.

The most frequently asked questions regarding Session Recording are how large are recorded session files, and how many session recordings can one Session Recording Server handle? The answer to the second question depends on the first. If session recording generates a small file with a low data rate, the number of sessions that can be recorded by one Session Recording Server is high.

The amount of data per session varies greatly depending on what is being recorded, while other factors like the screen resolution, color depth and the graphics mode also have impacts. An ICA session in which the end user sends and receives email in Microsoft Outlook will likely generate about 5 MB recording data per hour, with 1024*768 32bit color display settings and default XenApp 7.6 configurations. 20MB over an eight-hour work day period is not unusual. A session running a CAD package where graphics activity is constantly high will likely generate a much larger recording (typically, many hundreds of megabytes of data over the same period).

Many companies have tested the network bandwidth consumed by ICA sessions. Because each Session Recording file essentially contains the ICA protocol data from a session, the data sent to the Session Recording Server for storage will closely resemble the data generated using Citrix Receiver.

The number recordings each Session Recording Server can handle depends on the speed at which it can process and store the incoming data. The rate at which data is processed and stored must be higher than the input rate. For example, recording 5,000 simultaneous sessions running Outlook over an eight-hour work day equates to 100,000MB in total (that is, 5,000 x 20MB). As the data rate per second, this is approximately 3.5MB/s (that is, 100,000 divided by eight hours, divided by 3600 seconds). This is the input rate. A typical Session Recording Server connected to a 100Mbps LAN with sufficient disk space to store the recorded data would be capable of processing data at around 5.0MB/s based on the physical limits imposed by disk and network IOPS. This is the processing rate. In the example, this rate (5.0MB/s) is higher than the input rate (3.5MB/s), so recording the 5,000 Outlook sessions is feasible. However, recording the same number of CAD sessions would generate an extremely high input rate and would require the use of more Session Recording Servers.

Bursts and faults

The previous example assumes a very simple uniform throughput of data but does not explain how the system deals with short periods of higher activity, known as bursts. A burst might occur when all users log on at the same time in the morning, known as the 9 o’clock rush, or when they receive the same email in their Outlook inbox at once. The 5.0MB/s processing rate of the Session Recording Server is highly inadequate at dealing with this sudden demand.

The Session Recording Agent running on each XenApp Server OS VDA uses Microsoft Message Queuing (MSMQ) to send recorded data to the Storage Manager running on the central Session Recording Server. The data is sent in a store-and-forward manner similar to how email is delivered between sender, mail servers, and receiver. If the Session Recording Server or the network cannot handle the high rate of data in a burst, the recorded session data is temporarily stored until the backlog of data messages is cleared. The data message might be temporarily stored in the outgoing queue on the XenApp Server OS VDA if the network is congested, or stored on the Session Recording Server’s receiving queue if the data has traversed the network but the Storage Manager is still busy processing other messages.

MSMQ also serves as a fault tolerance mechanism. If the Session Recording Server goes down or the link is broken, recorded data is held in the outgoing queue on each XenApp Server OS VDA. When the fault is rectified, all queued data is sent together.  The use of MSMQ also allows administrators to take a Session Recording Server offline for upgrade or maintenance without interrupting the recording of existing sessions and losing data. 

The main limitation of MSMQ is that disk space for the temporary storage of data messages is finite. This limits how long a burst, fault, or maintenance event can last before data is eventually lost. The overall system can continue after data loss, but in this situation individual recordings have chunks of data missing. A file with missing data is still playable but only up to the point where data was first lost. Note the following:

• Adding more disk space to each server, especially the Session Recording Server, and making it available to MSMQ can increase the tolerance to bursts and faults.

• It is important to configure the Message Life setting for each Session Recording Agent to an appropriate level (on the Connections tab in the Session Recording Agent Properties application). The default value of 7,200 seconds (two hours) means that each recorded data message has two hours to reach the Storage Manager before it is discarded and recording files are damaged. With more disk space available (or less sessions to record), the administrator may choose to increase this value. The maximum value is 365 days.

The other limitation with MSMQ is that when data backlogs, there is extra disk IOPS in the queue to read and write data messages. Under normal conditions, the Storage Manager receives and processes data from the network directly without the data message ever being written to disk. Storing the data involves a single write operation to disk that appends the recorded session file. When data is backlogged the disk IOPS is tripled; each message must be written to disk, read from disk, and written to file. As the Storage Manager is heavily IOPS bound, the processing rate of the Session Recording Server drops until the backlog of messages is cleared. To mitigate the effects of this extra IOPS, adopt these recommendations:

• Ensure that the disk on which MSMQ stores messages is different from the recording file storage directories. Even though IOPS bus traffic is tripled, the drop in the true processing rate is never as severe. 

• Have planned outages at off-peak times only. Depending on budget constraints, follow recognized approaches to building high availability servers. This includes the use of UPS, dual NICs, redundant switches, and hot swappable memory and disks.

Design for spare capacity

The data rate of recorded session data is unlikely to be uniform, bursts and faults will occur, and the clearing of message backlogs is expensive in IOPS. For this reason, design each Session Recording Server with plenty of spare capacity. Adding more servers or improving the specification of existing servers, as described in later sections, always gains you extra capacity. The general rule of thumb is to run each Session Recording server at a maximum of 50% of its total capacity. In the earlier example, if the server is capable of processing 5.0MB/s, target the system to run only at 2.5MB/s. Instead of recording 5,000 Outlook sessions that generate 3.5MB/s on one Session Recording Server, reduce this to 3,500 sessions that generate only about 2.5MB/s.

Backlogs and live playback

Live playback is when a reviewer opens a session recording for playback while session itself is still active. During live playback the Session Recording Agent responsible for the session switches into a streaming mode for that session, and recording data is sent immediately to the Storage Manager without internal buffering. Because the recording file is constantly updated, the player can continue to be fed with the latest data from the live session. However, data sent from the Agent to the Storage Manager is through MSMQ so the queuing rules described earlier apply. A problem can occur in this scenario. When MSMQ is backlogged, the new recorded data available for live playback is queued like all other data messages. The reviewer can still play the file but viewing latest live recorded data is delayed. If live playback is an important feature for reviewers, ensure a low probability of backlog by designing spare capacity and fault tolerance into your deployment.

XenApp scalability

Session Recording never reduces XenApp session performance and never stops sessions in response to recorded data backlogs. Maintaining the end-user experience and single-server scalability is paramount in the design of the Session Recording feature. If the recording system becomes irreversibly overloaded, recorded session data is simply discarded. Extensive scalability testing by Citrix reveals that the impact of recording ICA sessions on XenApp Server performance and scalability is very low. The size of the impact depends on the platform, memory available, and the graphical nature of the sessions being recording. With the following configuration, you can expect a single-server scalability impact of between 1% and 5%. In other words, if a server can host 100 users without Session Recording installed, it can host 95 - 99 users after installation:

• 64-bit server with 8GB RAM running XenApp Server OS VDA

• All sessions running Office productivity applications, such as Outlook or Excel

• The use of applications is active and sustained

• All sessions are recorded as configured by the Session Recording policies

If fewer sessions are recorded or session activity is less sustained and more sporadic, the impact is less. In many cases, the scalability impact is negligible and user density per server remains the same. As mentioned earlier, the low impact is due to the very simple processing requirements of the Session Recording components installed on each XenApp Server OS VDA. Recorded data is simply extracted from the ICA session stack and sent as-is to the Session Recording Server through MSMQ. There is no expensive encoding of data.

There is a very minor overhead of using Session Recording even when no sessions are recorded. Although the impact is low, if you are sure that no sessions will ever be recorded from a particular server, you can disable recording on that server. Removing Session Recording is one way of doing this. A less invasive approach is to deselect the Enable session recording for this XenApp Servercheck box on the Recording tab in the Session Recording Agent Properties application. If session recording is required in future, reselect this option.

Measuring throughput

There are a number of ways to measure throughput of recorded session data from the sending XenApp Server OS VDA and the receiving Session Recording Server. One of the simplest and most effective approaches is to observe the size of files that are recorded, and the rate at which disk space on the Session Recording Server is being consumed. The volume of data written to disk very closely reflects the volume of network traffic being generated. The Windows Performance Monitor tool (perfmon.exe) has a range of standard system counters that can observed as well as some counters provided by Session Recording. Counters can be used to measure throughput, and identify bottlenecks and system problems. The following table outlines some of the most useful performance counters.

Session Recording Server hardware

You can increase the capacity of your deployment by carefully selecting the hardware used for the Session Recording Server. You have two choices: scaling up (by increasing the capacity of each server) or scaling out (by adding more servers). In choosing one of these, your aim should be to increase scalability at the lowest cost. 

Scaling up

When examining a single Session Recording Server, consider the following best practices to ensure optimal performance for available budget. The system is very dependent on IOPS. This ensures a high throughput of recorded data from the network on to the disk. So it’s important to invest in appropriate network and disk hardware. For a high-performance Session Recording Server, a dual CPU or dual core CPU is recommended but little is gained from any higher specification. 64-bit processor architecture is recommended but an x86 processor type is also suitable. 2 to 4GB of RAM is recommended but again there is little benefit from adding more.

Network

From a network perspective, a 100Mbps link is suitable but some gains can be made by using a gigabit Ethernet connection. However, note that although “gigabit” suggests a tenfold improvement over 100Mbps, in practice the gain in throughput is significantly less. 

Ensure network switches are not shared with third-party applications that might compete for available network bandwidth. Ideally, switches should be dedicated to the Session Recording Server. If network congestion proves to be the bottleneck, a network upgrade is a relatively inexpensive way to increase the scalability of the system.

Storage

Investment in disk and storage hardware is the single most important factor in server scalability. The faster that data can be written to disk, the higher the performance of the overall system. When selecting a storage solution, take more note of the write performance than the read performance. 

Store data on a set of local disks controlled either as RAID by a local disk controller or as a Storage Area Network (SAN). Storing data on a Network Attached Storage (NAS) based on file-based protocols such as SMB, CIFS, or NFS has serious performance and security implications. Never use this configuration in a production deployment of Session Recording. 

For a local drive setup, aim for a disk controller with built-in cache memory. Caching allows the controller to use elevator sorting during write-back, which minimizes disk head movement and ensures write operations are completed without waiting for the physical disk operation to complete. This can improve write performance significantly at minimal extra cost. Caching does however raise the problem of data loss after a power failure. To ensure the integrity of data and the file system, the caching disk controller must have a battery backup facility, which ensures that, if power is lost, the cache is maintained and data is written to disk when power is eventually restored.

Consider using a suitable RAID storage solution. There are a number RAID levels available depending on performance and redundancy requirements. The following table specifies each of the RAID levels and how applicable each standard is to Session Recording.

RAID 0 and RAID 3 are the most recommended RAID levels that should be considered. RAID 1 and RAID 5 are popular standards but are not recommended for Session Recording. RAID 10 does provide some performance benefits but is too expensive for the additional gain.

Decide on the type and specification of disk drives. IDE/ATA drives and external USB or Firewire drives are not suitable for use in Session Recording. The main choice is between SATA and SCSI. SATA drives provide reasonably high transfer rates at a reduced cost per Megabyte compared with SCSI drives. However, SCSI drives provide better performance and are more common in server deployments. Server RAID solutions mostly support SCSI drives but some SATA RAID products are now available. When evaluating the specifications of disk drive products consider the rotational speed of disk, seek performance, and other write performance characteristics. 

Because the recording of thousands of sessions per day can consume significant amounts of disk space, you must choose between overall capacity and performance. From the earlier example, recording 5,000 Outlook sessions over an eight-hour work day consumes about 100GB of storage space. To store 10 days’ worth of recordings (that is, 50,000 recorded session files) 1000GB (1TB) is required. This pressure on disk space can be eased by shortening the retention period before archiving or deleting old recordings. If 1TB of disk space is available, a seven-day retention period is reasonable, ensuring disk space usage remains around 700GB, with 300GB remaining as a buffer for busy days. In Session Recording, the archiving and deleting of files is supported with the icldb command-line utility and has a minimum retention period of two days. You can schedule this to run as a background task once a day at some off-peak time. For more information, on the icldb command and archiving see the product documentation.

The alternative to using local drive and controllers is to use a SAN storage solution based on block-level disk access. To the Session Recording Server, the disk array appears as a local drive. SANs are more expensive to set up, but as the disk array is shared, SANs do have the advantage of simplified and centralized management. There are two main types of SAN: fibre-channel and iSCSI. iSCSI is essentially SCSI over TCP/IP and is gaining popularity over Fibre Channel since the introduction of gigabit Ethernet.

Scaling out

Even with the best scaling up practices, there are limits to performance and scalability that can be reached with a single Session Recording Server when recording a very large number of sessions. It may be necessary to add extra servers to meet the load. Each server must have its own dedicated storage, network switches, and database. Load balancing is achieved by splitting the computers running XenApp Server OS VDA with the Session Recording Agent into groups that point at different Servers.

Database scalability

The Session Recording Database is hosted on Microsoft SQL Server 2014 (Enterprise, Standard and Express editions), Microsoft SQL Server 2012 (Enterprise, Standard and Express editions) with Service Pack 2, or Microsoft SQL Server 2008 R2 (Enterprise,  Standard and Express editions) with Service Pack 3. Because the physical recording files are written to separate disk files, the volume of data sent to the database on the SQL Server is very small. The database only stores metadata about recorded sessions that is typically only about 1KB of data per recording. Recording 5,000 sessions per eight-hour day only consumes about 5MB of database space. SQL Server 2005 Express Edition imposes a database size limit of 4GB. At 1KB per recording, this means the database is capable of cataloguing up to about four million recordings. The other editions of SQL Server, such as the Enterprise Editions, do not have this size restriction so available disk space is the only limitation. The performance of the database and the search speed only degrade at a negligible rate as the number of records increases.

Recordings are larger than 1KB if the Session Recording Event API is used to inject searchable events into recording files. The space required to store event metadata in the database depends on the frequency and size of each event. Non-searchable events have no effect on the database as the event metadata is only recorded in the recording file itself.

If the Session Recording Event API is not used, each recording file generates four database transactions: two when the recording starts, one when the user logs on to the session being recorded, and one when the recording ends. For 5,000 sessions this equates to around 20,000 transactions over an eight-hour day. If the API is used, each searchable event recorded will generate one transaction. Since even the most basic database deployments can typically handle hundreds of transactions per second, the processing load on the database is never likely to be stressed. As the impact is so low, the Session Recording database can co-exist with other databases (including the XenApp Server Data Store database) on the same SQL Server instance.

For larger databases where many millions of recordings are stored, follow the Microsoft guidelines on setting up and configuring SQL Server for scalability.