When publishing an image, the task fails with the message, "A failure occurred while publishing the Layered Image:  An error occurred while compositing the layer or image.  Please check the available disk space on the local storage or the size of the target image."

Also, in the ELM console, you might see messages like this immediately before the task fails.

Aug 30 19:44:08 localhost ntfs-3g[13416]: allocated extent inode 46
Aug 30 19:44:13 localhost ntfs-3g[13416]: extending $MFT bitmap
Aug 30 19:44:13 localhost ntfs-3g[13416]: allocated extent inode 47
Aug 30 19:44:15 localhost ntfs-3g[13416]: allocated extent inode 58
Aug 30 19:44:22 localhost ntfs-3g[13416]: No free mft record for $MFT: No space left on device
Aug 30 19:44:22 localhost ntfs-3g[13416]: No free mft record for $MFT: No space left on device
Aug 30 19:44:23 localhost ntfs-3g[13416]: No free mft record for $MFT: No space left on device


This solution also applies if the OS layer is reported as too fragmented.

First Check:

- Is your image template size large enough for the layers you have selected? Edit the template and select the "Layered Image Disk" tab. The UI will show you what the maximum expected size of the image will be. Please increase the image size if this is set too low and try to publish again.

- Do you have enough free space in the ELM's layer repository? To check, select System -> Manage Appliance -> check how much free space is available for the Layering Service. If free space is less than the maximum image size, then you will need to free up space by deleting older / unused layer versions or expand the layer repository. If you cannot delete any layers and need to expand, see here: https://docs.citrix.com/en-us/citrix-app-layering/4/manage/storage.html


If Free Space Has Been Verified:

This could be a problem handling the NTFS Master File Table ($MFT), which sometimes causes the $MFT to run out of space.  The $MFT is where NTFS stores file information, including the cluster maps, file attributes, and other meta-data.  If a file is small enough, its entire contents could potentially be stored in the $MFT instead if allocating cluster space for it. The $MFT should always dynamically resize, but sometimes it does not, leading to a spurious out-of-space issue. File fragmentation can also lead to the same situation.

When publishing an image, the appliance will clone the OS layer and resize it to the size of the image. All layers are then copied in to this disk while composing the image. We are only able to safely expand a disk but we cannot shrink it. If the disk needs to be smaller, the appliance will create a new disk, format it, and copy in the files from the selected OS layer. Since this process creates a new disk and file system, this will remove any file system problems we see on the existing layer.

Solution:

Option 1:
Add a version to your OS layer, and de-fragment the OS disk. Finalize the OS layer, update your image, and publish again. If this still does not help, try option 2 or 3.

Option 2:
Switch to a connector using the Offload Compositing engine. This option moves the compositing work to a temporary Windows PE virtual appliance. Since the NTFS file system on the layers are being handled natively by Windows, it should be able to manage a fragmented file system more gracefully than the Linux based ELM is capable of. See here for information on the Offload Engine:

https://docs.citrix.com/en-us/citrix-app-layering/4/connect.html#about-offloading-layer-compositing-hyper-v-vmware-vsphere-only

Option 3:
Using a Connector with the Offload Compositing engine disabled, add a version to your OS layer. Version Details will be shown in the new version dialog. Set the "Max Layer Size" to an amount smaller than the current layer size. For example, if the OS layer size is 60GB. Set the new Max Layer Size to 55GB. As explained above, this will force the appliance to create a new disk and file system for the OS layer. Increasing the layer size will NOT have the same effect since the existing disk will be expanded instead of being recreated.


Clean Up and Notes:

This problem can arise again after enough updates and activity on your OS layer. In the option 3 example the OS layer was reduced to 55GB. On the next occurrence you could reduce the size to 50GB. Obviously this is not sustainable and there is only so small you can make the layer. After making a version at 55GB, add another version and bring it back up to 60GB (or higher as needed). This can wait till your next set of OS updates or done proactively.

If file fragmentation is the problem, Option 3 will still correct the issue as the file system is recreated in the process. However, running defrag on your OS layer from time to time will prevent those occurrences. Note: Defrag can only be run when editing an OS layer and will not work for app or platform layers. The layering driver will interfere with Windows defrag. OS layers do not have our driver running and will not be negatively effected by defrag.