Troubleshoot failures when creating or updating a machine catalog by keeping the image preparation VM

book

Article ID: CTX697166

calendar_today

Updated On:

Description

Requirements

  • The -KeepPreparationVMOnFailure parameter on New-ProvScheme and Publish-ProvMasterVMImage is available in Citrix DaaS, and in CVAD on-prem 2411 CR and later.
  • The Get-ProvImagePrepVM and Remove-ProvImagePrepVM cmdlets are available in Citrix DaaS; for CVAD on-prem they are planned for 2611 CR (and later).
  • On releases that support -KeepPreparationVMOnFailure but not the Get-ProvImagePrepVM / Remove-ProvImagePrepVM cmdlets, you can still keep the image preparation VM on failure, but you must locate and remove it manually from the hypervisor.
  • Permissions. Running Get-ProvImagePrepVM requires the View Machine Catalogs delegated-administration permission — the same read-level permission as Get-ProvVM and Get-ProvScheme. Running Remove-ProvImagePrepVM requires the Delete Machine Catalog permission — the same as Remove-ProvVM and Remove-ProvScheme. Both permissions are included in the built-in Machine Catalog Administrator and Full Administrator roles, are the same in Citrix DaaS and CVAD on-prem, and can be scoped to specific catalogs. You will also need the corresponding hypervisor-side access to inspect the retained VM — for example, rights to detach disks and to boot the VM or download/attach its OS disk.

Frequently Asked Questions

What is image preparation?

Image preparation is an important step as part of creating a machine catalog or updating the master image used by a machine catalog. It involves making a copy of the master image and performing a series of operations needed for Machine Creation Services (MCS) to be able to use that prepared image to create additional VMs. In a typical scenario, the results of image preparation are sent back to MCS. This allows MCS to interpret the results, and in failure cases, an appropriate error message can be shown in Studio (or Powershell). The image preparation VM and associated resources are typically removed after the image preparation process is completed.

How does this feature work?

This feature is available in Powershell as an optional parameter (-KeepPreparationVMOnFailure) to the New-ProvScheme and Publish-ProvMasterVMImage cmdlets.

The behavior is as follows:

  • Whether this feature is enabled or not, the image preparation VM and resources are automatically removed when image preparation completes successfully
  • If this feature is not enabled, the image preparation VM and resources will also be removed after image preparation process fails
  • When enabling the feature, the image preparation VM and resources will be left on the hypervisor if the image preparation process fails

How do I enable it?

Add the -KeepPreparationVMOnFailure parameter when creating a catalog or updating its master image.

Create a machine catalog:New-ProvScheme -ProvisioningSchemeName "MyCatalog" -MasterImageVM <snapshot> -KeepPreparationVMOnFailure

Update the master image of an existing catalog:Publish-ProvMasterVMImage -ProvisioningSchemeName "MyCatalog" -MasterImageVM <snapshot> -KeepPreparationVMOnFailure

For complete, ready-to-run examples, see the Citrix MCS SDK samples on GitHub for VMware and XenServer. The -KeepPreparationVMOnFailure parameter and the Get-ProvImagePrepVM / Remove-ProvImagePrepVM cmdlets work the same way on every hypervisor, so these samples apply to the other hypervisors too — only the master-image and hosting-specific parameters differ.

When is the image preparation VM kept? The VM is retained only when image preparation itself fails and the feature is enabled:

  • Kept: the feature is enabled and image preparation runs but fails (for example, the "No Image Preparation results found…" error).
  • Not kept: image preparation succeeds — the VM is always removed on success, regardless of the parameter.
  • Not kept: the failure occurs before the image preparation VM is created or boots (for example, a failure copying the master image or configuring networking) — there is no preparation VM to retain.
  • Not kept: the feature is not enabled.

Why would I want to keep the image preparation VM?

Image preparation includes running many steps on a temporary virtual machine with a VDA installed on it. There are many different options that can be specified during machine catalog creation, and as a result, a lot of different types of issues can occur. Sometimes, when failures occur, the image preparation results reported back to MCS are able to be shown in Studio (or Powershell). Other times, the failure may simply state that no VDA is installed. This happens when MCS is unable to retrieve the image preparation results from the image preparation VM. When failures occur, it may not be clear what happened, and sometimes the only way to determine the root cause is to inspect the logs on the image preparation VM. This feature introduces the ability to keep the image preparation VM on the hypervisor so that these logs can be inspected. If image preparation is successful, then the resources are removed automatically.

What is left on the hypervisor when using this feature?

At minimum, the image preparation virtual machine will be there. Other resources that may be present will vary from one hypervisor to the next. The list of resources may include one or more of the following: virtual machine, network interface, and OS disk. Optional features may also impact what resources may be present. For example, when using MCS I/O (write back cache), there will be an additional disk on the hypervisor. For some cloud-based hypervisors, the specific SKU may also determine the resources kept on the hypervisor as well.

Caution: the retained image preparation VM is a full copy of your master image, and its OS disk (and the identity disk) may contain sensitive contents. Treat these resources as sensitive and remove them promptly once you have finished collecting the logs (see "I am finished with the image preparation VM, now what?"). On cloud hypervisors, leaving them in place also incurs ongoing cost.

How do I find the image preparation VM on the hypervisor?

Powershell (where available; see Requirements)

Use the Get-ProvImagePrepVM Powershell cmdlet. You can provide the ProvisioningSchemeName or ProvisioningSchemeUid to retrieve details for one target provisioning scheme, or not provide them to see image preparation VM information for all provisioning schemes that have it.

The output looks like this:C:\PS>Get-ProvImagePrepVM ImagePrepVMId : test-rg/Preparati-iylir MasterImageId : test-rg/win10-novda_OsDisk_1_976b0f797e04481390049e443fa2f989 MasterImageVM : /East US.region/image.folder/test-rg.resourcegroup/win10-novda_OsDisk_1_976b0f797e04481390049e 443fa2f989.manageddisk PluginId : AzureRmFactory PreparationImageName : Preparation - test-azure-fail ProvisioningSchemeName : test-azure-fail ProvisioningSchemeUid : a68bec0f-ab81-424b-87d6-83154a35232c State : ErrorCreating

Depending on the hypervisor, the field indicating exactly how to find the image preparation VM may vary. The PluginId field in the output above tells you which hypervisor the provisioning scheme uses. Use the table below to match it to the right field and to know what the VM looks like in that hypervisor's own console:

Hypervisor (PluginId)

 

Field to match in Get-ProvImagePrepVM

 

Notes — what you see in the hypervisor console

 

Azure (AzureRmFactory)

ImagePrepVMId

Format ResourceGroup/VMName. The VM name is shortened/normalized (for example Preparati-nv7g6) and the VM is placed in an MCS-managed resource group named citrix-xd-<schemeUid>-<suffix>; the catalog name is not visible in the portal. Match on ImagePrepVMId.

AWS (AWSMachineManagerFactory)

ImagePrepVMId or PreparationImageName

ImagePrepVMId is the EC2 instance-id (i-…). The catalog name is not in the instance-id — it is written to the EC2 Name tag as Preparation - <CatalogName> (this equals PreparationImageName). Find the instance by its Name tag, or by the exact instance-id.

GCP (GcpPluginFactory)

ImagePrepVMId

Format projectId:region:instance-name. The instance name is sanitized for GCP (lowercased, spaces/punctuation become -; for example Preparation - MyCatalog → preparation-mycatalog, truncated with a short unique suffix). PreparationImageName will not match the console name — match the last (:-separated) segment of ImagePrepVMId.

VMware (VmwareFactory)

PreparationImageName

Format Preparation - CatalogName. Matches the vSphere VM name exactly for normal names.

XenServer (XenFactory)

PreparationImageName

Format Preparation - CatalogName. Matches the XenCenter VM name exactly.

Nutanix AHV (AcropolisHypervisorPCFactory)

PreparationImageName

Format Preparation - CatalogName. Matches the Prism VM name exactly. (Legacy Prism Element connections use PluginId AcropolisFactory.)

SCVMM (MicrosoftPSFactory)

PreparationImageName

Format Preparation - CatalogName. Matches the SCVMM / Hyper-V Manager VM name exactly.

 
 
 

How do I use the image preparation VM which has been left on the hypervisor after a failure?

At a high level, follow the below steps, but note that the exact steps are different across hypervisors. See the Hypervisor Reference section for instructions for specific hypervisors.

  1. Locate the image preparation virtual machine on the hypervisor. See the "How do I find the image preparation VM on the hypervisor?" section for details.
  2. Detach the identity disk from the VM (required for on-prem hypervisors; not needed for cloud hypervisors — see the Hypervisor Reference section)
    1. Detaching this disk prevents the image preparation process from running again when booting the machine, which will cause the logs to be removed.
    2. Once detached, this disk will need to be cleaned up manually from the hypervisor console, but do not do this yet.
    3. The procedure for doing this will be different across hypervisors. See the section on how to do this for your hypervisor.
  3. Access the image preparation VM. The method depends on your hypervisor type (see the Hypervisor Reference section): for on-prem hypervisors, you typically boot the VM and use the hypervisor console; for cloud hypervisors, you work with the OS disk instead of booting the VM. This can be done a couple of different ways:
    1. Connect directly to the console of the image preparation VM via the hypervisor interface
    2. Detach the OS disk of the image preparation VM and attach it as a data disk to another VM on the hypervisor and mount it as a new drive in that VM
    3. Export the OS disk of the image preparation VM and either mount it directly or attach as data disk to another machine
  4. Retrieve the logs from the image preparation VM.
    1. Diagnostic logs are generated during image preparation. Look in the folder C:\ProgramData\Citrix\ImagePreparation for the image preparation log — named image-prep-<timestamp>.log (older VDA versions in the master image write a static image-prep.log); there may be more than one, so use the most recent or collect them all.
    2. The image preparation results. The identity disk detached in step 2 may be attached as a data disk to another VM and mounted as a new drive to that VM. This allows for retrieval of the PreparationStatus.xml file which may also provide troubleshooting information.
  5. Review the collected logs and review or provide to a support engineer for assistance.

What to collect for a support case:

  1. The image preparation log — image-prep-<timestamp>.log (older VDA versions in the master image write a static image-prep.log), from C:\ProgramData\Citrix\ImagePreparation; collect the newest, or all of them if there are several
  2. PreparationStatus.xml (from the identity disk)
  3. The ProvisioningScheme name and Uid (from Get-ProvImagePrepVM)
  4. The hypervisor type and the exact error message shown

PreparationStatus.xml is the structured results file that MCS reads back from the image preparation VM's identity disk. Alongside the free-form image preparation log, it records the overall preparation status and a per-step status showing which preparation steps ran and how each one finished — usually the quickest way to pinpoint the failing step. It also includes results from any third-party preparation steps and an embedded operation log. To read it, attach the identity disk (detached earlier) to another VM as a data disk.

I am finished with the image preparation VM, now what?

There are multiple options for removing the image preparation VM from the hypervisor:

  • If the failure occurred during machine catalog creation, you can simply delete the machine catalog and this will automatically remove the associated image preparation VM from the hypervisor as well.
  • If you don't want to remove the entire machine catalog, you can use the Remove-ProvImagePrepVM Powershell cmdlet (where available; see Requirements).
    • The Remove-ProvImagePrepVM cmdlet can take the ProvisioningSchemeName or ProvisioningSchemeUid parameter to specify which provisioning scheme will have the associated image preparation VM removed
    • You can also use piped output, e.g.: Get-ProvScheme -ProvisioningSchemeName myProvScheme | Remove-ProvImagePrepVM
  • You can manually remove the VM from the hypervisor using the hypervisor console.
  • If the failure occurred while updating the master image of an existing machine catalog, you can simply try again, and the existing image preparation VM from the previous failed update will be removed automatically.
    • This can be done using Studio or Powershell using the Publish-ProvMasterVMImage cmdlet
    • Be sure to use the -KeepPreparationVMOnFailure optional parameter to preserve the image preparation VM to help troubleshoot if the new attempt also fails
  • Note that if the identity disk was detached from the image preparation VM and not deleted, it must be manually deleted from the hypervisor console.

How does this feature interact with Orphaned Resource Detection when using Azure hypervisor?

The image preparation VM and associated resources are considered temporary. As a result, when using this feature to preserve the image preparation VM and associated resources to assist in troubleshooting the reason for the failure, these resources will be flagged by Orphaned Resource Detection. This is an intended behavior with the goal of ensuring that customers do not forget to remove the image prep VM and resources after debugging, as forgetting to do so may incur additional monthly charges.

Hypervisor Reference

The exact steps for detaching the identity disk and viewing the image preparation logs depend on the type of hypervisor, which falls into one of two categories:

  • On-prem hypervisors — for example XenServer, VMware, Nutanix, and SCVMM. You inspect the logs by booting the image preparation VM and reading them from the hypervisor console (or, alternatively, by exporting and mounting its OS disk).
  • Cloud hypervisors — Azure, AWS, and GCP. You inspect the logs by exporting or attaching the OS disk to another machine, without booting the image preparation VM.

The steps below cover the most common hypervisors in each category. Other hypervisors of the same category follow the same approach — the concepts are identical; only the specific console actions (menu names, button labels) differ. Consult your hypervisor's own documentation for the exact console steps.

Detaching Identity Disk from Image Preparation VM

Detaching the identity disk (a small ~16 MB disk named prepare-identity) prevents the image preparation process from running again the next time the VM boots, which would overwrite the logs you are trying to collect.

  • On-prem hypervisors: this step is required before you boot the VM to view the logs. Locate the VM, find the ~16 MB prepare-identity disk, and detach it (steps for XenServer and VMware are below).
  • Cloud hypervisors (Azure, AWS, GCP): this step is not required. For cloud, you inspect the OS disk without booting the image preparation VM, so there is no risk of the image preparation process running again and overwriting the logs. Skip to Viewing the image preparation logs.

On-prem hypervisors

XenServer

  1. Using XenCenter, locate the image preparation VM
    1. Use the Get-ProvImagePrepVM Powershell cmdlet
    2. In XenCenter, check the UUID to verify that it matches the ImagePrepVMId
  2. Select the VM and go to Storage tab
  3. Locate the disk named "prepare-identity"
  4. Right click > Choose Delete Disk or Detach Disk
    1. If using the Detach option, the disk must be manually deleted
    2. You may rename the disk before detaching to make it easier to find later
VMware
  1. Using vSphere, locate the image preparation VM
    1. Use the Get-ProvImagePrepVM Powershell cmdlet
    2. Match the VM name to the PreparationImageName field from Get-ProvImagePrepVM (format: Preparation - <CatalogName>).
  2. Select the VM and Right click > Choose Edit Settings
  3. Under Virtual Hardware, find the identity disk
    1. It will be 16 MB
    2. The "Disk File" will end with "prepare-identity.vmdk"
  4. Click the three vertical dots next to the identity disk > Choose "Remove device and data"
  5. Press OK on the Edit Settings dialog
Other on-prem hypervisors (Nutanix, SCVMM, etc.)

Follow the same approach in your hypervisor's own console: locate the image preparation VM (see How do I find the image preparation VM? section), find the ~16 MB prepare-identity disk attached to it, and detach it (then delete it manually once you are finished).

Viewing the image preparation logs on the image preparation VM

The image preparation log lives in the folder C:\ProgramData\Citrix\ImagePreparation inside the image preparation VM's OS disk. The log file is named image-prep-<timestamp>.log; if the VM has been booted more than once there may be several timestamped files, so collect the most recent one (or all of them). The filename format is determined by the VDA installed in the master image — older VDA versions write a single static image-prep.log instead. How you reach the folder depends on the hypervisor category.

On-prem hypervisors

Make sure to follow the correct steps to detach the identity disk first (see above). Then either boot the VM and read the log from the hypervisor console, or export/mount the OS disk and read it from the filesystem. Steps for XenServer and VMware follow; other on-prem hypervisors use the same approach in their own console.

XenServer

There are two options:

Retrieve directly from VM running on hypervisor
  1. Start the image preparation VM in XenCenter
  2. Go to the Console tab and login to the machine if needed
  3. Open Windows Explorer and Browse to %programdata%\Citrix\ImagePreparation. If prompted with a security prompt, press Continue
  4. View the image preparation log file or copy and provide to support for troubleshooting
Export the OS disk
  1. Shut down the image preparation VM
  2. In XenCenter, select the host where the image preparation VM is hosted
  3. In the right pane, select the Search tab. In the Overview, scroll to the image preparation VM
  4. Right click > Export
  5. Ensure the correct VM is selected, press Next
  6. For Format, choose OVF/OVA Package
  7. Press Next (no Advanced Options are required) and then Finish
  8. Once downloaded, mount the .vhd file
    1. On the new drive (assume the new drive letter is D:), navigate to D:\ProgramData\Citrix. If prompted with a security prompt, press Continue
    2. Then open the ImagePreparation folder. From there, view the image preparation log file or copy and provide to support for troubleshooting
VMware
  1. Power On the image preparation VM in vSphere console
  2. Use Launch Console or Launch Web Console to connect to the VM, and login to the machine if needed
  3. Open Windows Explorer and Browse to %programdata%\Citrix\ImagePreparation. If prompted with a security prompt, press Continue
  4. View the image preparation log file or copy and provide to support for troubleshooting

Cloud hypervisors (Azure, AWS, GCP)

For cloud hypervisors, you do not boot the image preparation VM. Instead, export, attach, or download its OS disk to another VM (or to your local machine), mount it, and read the image preparation log from …\ProgramData\Citrix\ImagePreparation\ on the mounted drive. The steps below are for Azure; AWS and GCP follow the same pattern using their own tooling (for example, attach the OS disk to another EC2 or Compute Engine instance, or download it), then read the same path.

Azure

There are two options for retrieving image preparation logs from an image preparation operation running in Azure.

Attach the OS disk directly to a VM already running in Azure
  1. The OS disk may be able to be attached directly to a running VM in Azure and then mounted as a new disk drive
  2. On the new drive (assume the new drive letter is D:), navigate to D:\ProgramData\Citrix. If prompted with a security prompt, press Continue
  3. Then open the ImagePreparation folder. From there, view the image preparation log file or copy and provide to support for troubleshooting.
Download the OS disk from Azure and retrieve that way
  1. Using Microsoft Azure Storage Explorer, find the resource group where the image preparation resides
  2. Then find the OS disk corresponding to the image preparation VM and download it.
  3. There are multiple options for viewing the image preparation log using the downloaded .vhd file:
    1. Open it directly in Windows
      1. Double click the downloaded .vhd file, which will map its contents as a new drive
      2. On the new drive (assume the new drive letter is D:), navigate to D:\ProgramData\Citrix. If prompted with a security prompt, press Continue
      3. Then open the ImagePreparation folder. From there, view the image preparation log file or copy and provide to support for troubleshooting
    2. Import it to a local hypervisor
      1. If using a local XenServer hypervisor, use Import and select the downloaded .vhd file
      2. For boot options, be sure to use UEFI
      3. After import completes, start the VM
      4. Open Windows Explorer and Browse to %programdata%\Citrix\ImagePreparation. If prompted with a security prompt, press Continue
      5. View the image preparation log file or copy and provide to support for troubleshooting

Common Failures

This feature has been used to diagnose the following issues in the past. In each case, inspecting the logs on the retained image preparation VM (as described above) is what identified the root cause and led to the resolution.

Encrypted master image

  • This results in the image preparation VM being unable to boot and as a result, image prep cannot run at all
  • The error is No Image Preparation results found. There may be no suitable VDA installed, or some other serious failure in the Master VM. Image preparation failed.
  • There is no image preparation log because the image preparation VM cannot boot
  • The resolution is to decrypt the master image and take a new snapshot

Snapshot taken too early before shutdown completes

  • This results in the snapshot containing the pending shutdown request. The image preparation process will start with the VM having a shutdown operation in queue and will be unable to complete.
  • The error is No Image Preparation results found. There may be no suitable VDA installed, or some other serious failure in the Master VM. Image preparation failed.
  • The image preparation log shows Unable to proceed due to shutdown in progress.
  • The resolution is to allow the master image to fully shutdown before taking the snapshot

Read-only instruction disk

  • This may happen due to a policy where write access is denied to removable drives not protected by BitLocker
  • The error is No Image Preparation results found. There may be no suitable VDA installed, or some other serious failure in the Master VM. Image preparation failed.
  • The image preparation log shows The media is write protected.
  • The resolution is to remove the values FDVDenyWriteAccess and RDVDenyWriteAccess (or set them to 0). Depending on how the policy was applied, these values may be located under either HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\FVE or HKEY_LOCAL_MACHINE\System\CurrentControlSet\Policies\Microsoft\FVE

Incorrect SAN policy configuration

  • Somehow the Diskpart SAN policy in the master image is set to disable auto-mount of new disks. VMW machine version 7 (ESX 3.5 and newer) presents drives to Windows as SAN storage. By not allowing auto-mount of new disks the image-preparation-identity.vmdk was not accessible and PvsVm service would not run the ImagePreparation.exe.
  • The error is No Image Preparation results found. There may be no suitable VDA installed, or some other serious failure in the Master VM. Image preparation failed.
  • See link for resolution instructions: https://support.citrix.com/external/article/CTX202229

Issue/Introduction

When errors occur while creating or updating a machine catalog, it can sometimes be difficult to understand what happened and what to do to address the issue. The issue most commonly happens during the image preparation step. Usually image preparation results are shown in Web Studio (or Powershell) when performing the operation, but occasionally, the results are not able to be shown and the error is No Image Preparation results found. There may be no suitable VDA installed, or some other serious failure in the Master VM. Image preparation failed. When this happens, the next course of action is to inspect the logs on the image preparation virtual machine. However, the default behavior is that the image preparation VM is always deleted at the conclusion of provisioning scheme preparation, regardless of whether the operation was successful. A new feature is now available that introduces the ability to keep the image preparation VM if there was a failure, so that the administrator can use it to help troubleshoot image preparation failures.

Additional Information

https://support.citrix.com/external/article/CTX140734

https://support.citrix.com/external/article/CTX695041/citrix-daas-mcs-catalog-creation-failed.html

https://support.citrix.com/external/article/CTX202229

https://support.citrix.com/external/article/CTX214572

https://support.citrix.com/external/article/CTX202656

https://support.citrix.com/external/article/CTX476119