VDA Boot Failure (“No Boot Media”) After MCS Catalog Update Due to Secure Boot 2023 Certificate

book

Article ID: CTX696696

calendar_today

Updated On:

Description

After updating with the MCS Machine Catalogs with a golden image that has it's bootloader signed by the Windows UEFI CA 2023, VDAs can fail to boot with “No Boot Media”.

Applies to:

  • Any hypervisor where VMs were originally created with only the Windows UEFI CA 2011 certificate in the Secure Boot database
  • Citrix Virtual Apps and Desktops (CVAD) — all versions, on-premises and Citrix Cloud

Cause

This happens because the updated master image contains a 2023-signed bootloader, but the VM’s Secure Boot database still only trusts the 2011 certificate. The diagram below illustrates the root cause of the problem.

image.png

  1. Existing VMs on a hypervisor only include the Windows UEFI CA 2011 certificate in the Secure Boot database (stored in NVRAM).
  2. The administrator updates the master image. Windows Update installs a cumulative update that includes a 2023-signed bootloader.
  3. An MCS catalog update is performed. MCS replaces the OS disk (including the bootloader) with the updated master image.
  4. However, MCS does not update the VM’s NVRAM — it retains the original Secure Boot certificates from when the VM was first created.
  5. On next boot, the VM’s firmware rejects the bootloader because the Secure Boot database does not trust the 2023 certificate. The VM fails to boot.
  6. This creates a mismatch: - Bootloader on disk: signed with the 2023 certificate (from the updated master image) - Secure Boot DB in NVRAM: only trusts the 2011 certificate (from original VM creation)

Example: VMware ESXi

  1. VMware ESXi 8.0.0 creates VMs with only the 2011 certificate in the Secure Boot database.
  2. After upgrading ESXi to 8.0.3, the hypervisor generates NVRAM with both 2011 and 2023 certificates for new VMs
  3. But existing VMs’ NVRAM is not updated.
  4. When a catalog update applies a 2023-signed bootloader from the new master image, existing VMs fail to boot.

Resolution

Follow your hypervisor below for resolution steps.

VMware

Recommendation:

For VMware, administrators can follow the steps below to address the Windows UEFI CA 2023 issue.

  • Step 1: Upgrade the hypervisor to VMware ESXi 8.0 U3j or later.

  • Step 2: If VMs have already failed to boot, reset the NVRAM files (refer to Fix Option 1 below).

  • Step 3: Deploy the Windows UEFI CA 2023 certificate to the Secure Boot DB.

  • Step 4: Verify the deployment of the Windows UEFI CA 2023 certificate.

The details of each step are described below.

Fix VMs That Failed to Boot

Among the fixes option below, we recommend the option 1.

  • Fix Option 1: Reset NVRAM Files (Recommended)

  • Fix Option 2: Roll Back the Master Image

    • Revert the MCS catalog to the previous master image snapshot that uses a 2011-signed boot loader.

  • Fix Option 3: Disable Secure Boot

Deploy the 2023 certificate to VDAs

  • Step 1: Complete Windows updates by running the PowerShell cmdlets below. Windows Updates can be automated via Group Policy (GPO).

  • Start-Process "usoclient.exe" -ArgumentList "StartScan"   -Wait -NoNewWindow
    Start-Process "usoclient.exe" -ArgumentList "StartInstall" -Wait -NoNewWindow

  • Step 2: Deploy the Windows UEFI CA 2023 to the VMs by running the PowerShell cmdlet below.

  • Start-ScheduledTask -TaskName "\Microsoft\Windows\PI\Secure-Boot-Update"

  • Step 3: Restart the VM twice from inside the guest OS, not from Citrix Web Studio.

  • To automate this, refer to the example script (Prevent Boot Failure - Deploy Certificate)

Verification

Run the following cmdlet in PowerShell:

[System.Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI db).bytes) -match 'Windows UEFI CA 2023'

Returns True if the Windows UEFI CA 2023 certificate has been deployed to the Secure Boot DB.

XenServe

For XenServer, administrators can currently either disable Secure Boot or roll back the image update. Further updates and releases are underway.

SCVMM / Hyper-V

Recommendation:

For SCVMM, administrators can follow the steps below to address the Windows UEFI CA 2023 issue.

  • Step 1: If VMs have already failed to boot, roll back the master image or disable secure boot.

  • Step 2: Deploy the Windows UEFI CA 2023 certificate to the Secure Boot DB.

  • Step 3: Verify the deployment of the Windows UEFI CA 2023 certificate.

The details of each step are described below.

Fix VMs That Failed to Boot

  • Fix Option 1: Roll Back the Master Image

    • Revert the MCS catalog to the previous master image snapshot that uses a 2011-signed boot loader.

  • Fix Option 2: Disable Secure Boot

    • Disable Secure Boot on affected VMs so the boot loader signature is not checked.

Deploy the 2023 certificate to VDAs

  • Step 1: Complete Windows updates by running the PowerShell cmdlets below. Windows Updates can be automated via Group Policy (GPO).

    Start-Process "usoclient.exe" -ArgumentList "StartScan"   -Wait -NoNewWindow
    Start-Process "usoclient.exe" -ArgumentList "StartInstall" -Wait -NoNewWindow

  • Step 2: Deploy the Windows UEFI CA 2023 to the VMs by running the PowerShell cmdlet below.

    Start-ScheduledTask -TaskName "\Microsoft\Windows\PI\Secure-Boot-Update"

  • Step 3: Restart the VM twice from inside the guest OS, not from Citrix Web Studio.

  • To automate this, refer to the example script (Prevent Boot Failure - Deploy Certificate)

Verification

Run the following cmdlet in PowerShell:

[System.Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI db).bytes) -match 'Windows UEFI CA 2023'

Returns True if the Windows UEFI CA 2023 certificate has been deployed to the Secure Boot DB.

OpenShift

OpenShift will not have the Windows UEFI CA 2023 certificate issue. This is because the OpenShift versions 4.18+ that Citrix supports already include 2023 certificate.

Nutanix AHV

Recommendation:

For Nutanix, administrators can follow the steps below to address the Windows UEFI CA 2023 issue.

  • Step 1: Upgrade the hypervisor to AHV 10.3.1 (AOS 7.3.1)

  • Step 2: If VMs have already failed to boot, roll back the master image or disable secure boot.

  • Step 3: Deploy the Windows UEFI CA 2023 certificate to the Secure Boot DB.

  • Step 4: Verify the deployment of the Windows UEFI CA 2023 certificate.

The details of each step are described below.

Fix VMs That Failed to Boot

  • Fix Option 1: Roll Back the Master Image

    • Revert the MCS catalog to the previous master image snapshot that uses a 2011-signed boot loader.

  • Fix Option 2: Disable Secure Boot

    • Disable Secure Boot on affected VMs so the boot loader signature is not checked.

Deploy the 2023 certificate to VDAs

  • Step 1: Complete Windows updates by running the PowerShell cmdlets below. Windows Updates can be automated via Group Policy (GPO).

    Start-Process "usoclient.exe" -ArgumentList "StartScan"   -Wait -NoNewWindow
    Start-Process "usoclient.exe" -ArgumentList "StartInstall" -Wait -NoNewWindow

  • Step 2: Deploy the Windows UEFI CA 2023 to the VMs by running the PowerShell cmdlet below.

    Start-ScheduledTask -TaskName "\Microsoft\Windows\PI\Secure-Boot-Update"

  • Step 3: Restart the VM twice from inside the guest OS, not from Citrix Web Studio.

  • To automate this, refer to the example script (Prevent Boot Failure - Deploy Certificate)

Verification

Run the following cmdlet in PowerShell:

[System.Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI db).bytes) -match 'Windows UEFI CA 2023'

Returns True if the Windows UEFI CA 2023 certificate has been deployed to the Secure Boot DB.

Azure

Recommendation:

For Azure, VMs created after March 2024 are not affected by the Windows UEFI CA 2011 certificate issue. VMs created prior to March 2024 do not include the Secure Boot 2023 certificates.

  • Step 1: If VMs have already failed to boot, roll back the master image or disable secure boot.

  • Step 2: Deploy the Windows UEFI CA 2023 certificate to the Secure Boot DB.

  • Step 3: Verify the deployment of the Windows UEFI CA 2023 certificate.

For Azure Local,

  • Step 1: Upgrade the hypervisor to 2603.

  • Step 2: If VMs have already failed to boot, roll back the master image or disable secure boot.

  • Step 3: Deploy the Windows UEFI CA 2023 certificate to the Secure Boot DB.

  • Step 4: Verify the deployment of the Windows UEFI CA 2023 certificate.

The details of each step are described below.

Fix VMs That Failed to Boot

  • Fix Option 1: Roll Back the Master Image

    • Revert the MCS catalog to the previous master image snapshot that uses a 2011-signed boot loader.

  • Fix Option 2: Disable Secure Boot

    • Disable Secure Boot on affected VMs so the boot loader signature is not checked.

Deploy the 2023 certificate to VDAs

  • Step 1: Complete Windows updates by running the PowerShell cmdlets below. Windows Updates can be automated via Group Policy (GPO).

    Start-Process "usoclient.exe" -ArgumentList "StartScan"   -Wait -NoNewWindow
    Start-Process "usoclient.exe" -ArgumentList "StartInstall" -Wait -NoNewWindow

  • Step 2: Deploy the Windows UEFI CA 2023 to the VMs by running the PowerShell cmdlet below.

    Start-ScheduledTask -TaskName "\Microsoft\Windows\PI\Secure-Boot-Update"

  • Step 3: Restart the VM twice from inside the guest OS, not from Citrix Web Studio.

  • To automate this, refer to the example script (Prevent Boot Failure - Deploy Certificate)

Verification

Run the following cmdlet in PowerShell:

[System.Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI db).bytes) -match 'Windows UEFI CA 2023'

Returns True if the Windows UEFI CA 2023 certificate has been deployed to the Secure Boot DB.

GCP

Recommendation:

For GCP, administrators can follow the steps below to address the Windows UEFI CA 2023 issue.

  • Step 1: If VMs have already failed to boot, roll back the master image or disable secure boot.

  • Step 2: Deploy the Windows UEFI CA 2023 certificate to the Secure Boot DB.

  • Step 3: Verify the deployment of the Windows UEFI CA 2023 certificate.

The details of each step are described below.

Fix VMs That Failed to Boot

  • Fix Option 1: Roll Back the Master Image

    • Revert the MCS catalog to the previous master image snapshot that uses a 2011-signed boot loader.

  • Fix Option 2: Disable Secure Boot

    • Disable Secure Boot on affected VMs so the boot loader signature is not checked.

Deploy the 2023 certificate to VDAs

  • Step 1: Complete Windows updates by running the PowerShell cmdlets below. Windows Updates can be automated via Group Policy (GPO).

    Start-Process "usoclient.exe" -ArgumentList "StartScan"   -Wait -NoNewWindow
    Start-Process "usoclient.exe" -ArgumentList "StartInstall" -Wait -NoNewWindow

  • Step 2: Deploy the Windows UEFI CA 2023 to the VMs by running the PowerShell cmdlet below.

    Start-ScheduledTask -TaskName "\Microsoft\Windows\PI\Secure-Boot-Update"

  • Step 3: Restart the VM twice from inside the guest OS, not from Citrix Web Studio.

  • To automate this, refer to the example script (Prevent Boot Failure - Deploy Certificate)

Verification

Run the following cmdlet in PowerShell:

[System.Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI db).bytes) -match 'Windows UEFI CA 2023'

Returns True if the Windows UEFI CA 2023 certificate has been deployed to the Secure Boot DB.

AWS

Recommendation:

For AWS, administrators can follow the steps below to address the Windows UEFI CA 2023 issue.

  • Step 1: If VMs have already failed to boot, roll back the master image or disable secure boot.

  • Step 2: Deploy the Windows UEFI CA 2023 certificate to the Secure Boot DB.

  • Step 3: Verify the deployment of the Windows UEFI CA 2023 certificate.

The details of each step are described below.

Fix VMs That Failed to Boot

  • Fix Option 1: Roll Back the Master Image

    • Revert the MCS catalog to the previous master image snapshot that uses a 2011-signed boot loader.

  • Fix Option 2: Disable Secure Boot

    • Disable Secure Boot on affected VMs so the boot loader signature is not checked.

Deploy the 2023 certificate to VDAs

  • Step 1: Complete Windows updates by running the PowerShell cmdlets below. Windows Updates can be automated via Group Policy (GPO).

    Start-Process "usoclient.exe" -ArgumentList "StartScan"   -Wait -NoNewWindow
    Start-Process "usoclient.exe" -ArgumentList "StartInstall" -Wait -NoNewWindow

  • Step 2: Deploy the Windows UEFI CA 2023 to the VMs by running the PowerShell cmdlet below.

    Start-ScheduledTask -TaskName "\Microsoft\Windows\PI\Secure-Boot-Update"

  • Step 3: Restart the VM twice from inside the guest OS, not from Citrix Web Studio.

  • To automate this, refer to the example script (Prevent Boot Failure - Deploy Certificate)

Verification

Run the following cmdlet in PowerShell:

[System.Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI db).bytes) -match 'Windows UEFI CA 2023'

Returns True if the Windows UEFI CA 2023 certificate has been deployed to the Secure Boot DB.

AWS Workspace Core

Recommendation:

For AWS Workspace Core, administrators can follow the steps below to address the Windows UEFI CA 2023 issue.

  • Step 1: If VMs have already failed to boot, roll back the master image or disable secure boot.

  • Step 2: Deploy the Windows UEFI CA 2023 certificate to the Secure Boot DB.

  • Step 3: Verify the deployment of the Windows UEFI CA 2023 certificate.

The details of each step are described below.

Fix VMs That Failed to Boot

  • Fix Option 1: Roll Back the Master Image

    • Revert the MCS catalog to the previous master image snapshot that uses a 2011-signed boot loader.

  • Fix Option 2: Disable Secure Boot

    • Disable Secure Boot on affected VMs so the boot loader signature is not checked.

Deploy the 2023 certificate to VDAs

  • Step 1: Complete Windows updates by running the PowerShell cmdlets below. Windows Updates can be automated via Group Policy (GPO).

    Start-Process "usoclient.exe" -ArgumentList "StartScan"   -Wait -NoNewWindow
    Start-Process "usoclient.exe" -ArgumentList "StartInstall" -Wait -NoNewWindow

  • Step 2: Deploy the Windows UEFI CA 2023 to the VMs by running the PowerShell cmdlet below.

    Start-ScheduledTask -TaskName "\Microsoft\Windows\PI\Secure-Boot-Update"

  • Step 3: Restart the VM twice from inside the guest OS, not from Citrix Web Studio.

  • To automate this, refer to the example script (Prevent Boot Failure - Deploy Certificate)

Verification

Run the following cmdlet in PowerShell:

[System.Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI db).bytes) -match 'Windows UEFI CA 2023'

Returns True if the Windows UEFI CA 2023 certificate has been deployed to the Secure Boot DB.

Issue/Introduction

Microsoft is transitioning from the Windows UEFI CA 2011 certificate to the Windows UEFI CA 2023 certificate for signing Windows bootloaders. The 2011 certificate expires in 2026.

Additional Information

Powered by Wolken Software