Debugging Layer Integrity Problems in Citrix App Layering
Article ID: CTX222099
Updated On:
Resolution
In App Layering, when you're ready to shutdown and finalize a layer, you run the Shutdown for Finalize icon on the desktop (As Administrator). It makes a call to uniservice.exe to get the current Layer Integrity state. Uniservice is tracking all the same things it always has for Layer Integrity: NGEN or MSCORLIB is still running, a reboot is pending, a domain operation is still pending, or a RunOnce script is still waiting.
Shutdown for Finalize is checking to see if anything is still pending that should happen in the layer rather than happen in the image later. If something is, it does not shut down, and instead puts up a statement about the pending issue. Fix the issue (for instance, reboot) and try again. It also writes this information into two log files:
C:\Program Files\Unidesk\Uniservice\Log\LayerIntegrity.txt
C:\Program Files\Unidesk\Uniservice\Log\UniBilcLogs_X.txt
You can't know exactly which UniBilcLogs file it's using, so look for the one with the latest timestamp. That will be for the current boot. Search for "Integrity".
You might think you could bypass the Layer Integrity check by just shutting down the machine normally and finalizing that. But if you try, you will find the ELM will stop the task and return you to the Packaging Machine, because it knows that the Layer Integrity Checks either failed or never ran. You must successfully run that Shutdown for Finalize script to finalize a layer.
The registry key, noted at the end of this article, to bypass or ignore integrity problems still works, and you should be just as reluctant to use it as ever. But it's still a valid way to give up and bypass it .
There are 9 Layer Integrity warnings you can see:
"A RunOnce script is outstanding" is telling you that there is a key in either of these two locations:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\RunOnce
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\RunOnce
Windows normally deletes those keys on reboot, but we have seen circumstances (especially with our own script, envsetup.cmd) where that doesn't happen. You can manually run the referenced script and delete the key, or just delete the key if the script file no longer exists.
"A post-installation reboot is pending" is looking at six different registry keys. Your first course of action should be to reboot, more than once(in some cases it has taken 3+ reboots), just to make sure that it isn't a real reboot being requested by some software. It may also be helpful, if the problem is NetLogon, to restart the Unidesk Service for Message Management.
First we check for the existence of any of these three:
HKLM\System\CurrentControlSet\Control\Session Manager\PendingFileRenameOperations
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\RebootPending
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired
You can manually modify any of these to suit your needs, including just deleting them.
Then we look for changes in the NetLogon key (if the current value is now different from what it was at bootup), and to see if the computer name doesn't match the active computer name. This is how we determine that a domain-join operation is still waiting for a reboot.
HKLM\SYSTEM\CURRENTCONTROLSET\SERVICES\NETLOGON\Start
HKLM\SYSTEM\CURRENTCONTROLSET\CONTROL\COMPUTERNAME\ACTIVECOMPUTERNAME
HKLM\SYSTEM\CURRENTCONTROLSET\CONTROL\COMPUTERNAME\COMPUTERNAME
Generally you cannot modify these. I've seen some software modify the NETLOGON\Start key on every reboot, so maybe that's happening. If, after cleaning out the top three, you still get the prompt on reboot, you will need to use the flag to ignore layer integrity checks.
"A Microsoft NGen operation is in progress in the background" is telling you that a foreground or background NGEN operation (where .Net assemblies are compiled into native images) is still in progress. Generally this is simply true: the ngen rebuild is still running. To watch it in the foreground, run "ngen update /force". Or you can wait it out, and run "ngen queue status" periodically to see how it's doing, but that will slow it down because the background process pauses every time you check its status in the foreground. Don't reboot or you might cause it to have to start over.
It's important to let NGEN finish. If you kill the process or reboot in the middle, you might wind up with partially written .Net assemblies that crash programs when they show up in an image. You should be patient. However, sometimes we have seen background MSCORSVW.EXE processes, clearly doing nothing, that just don't finish even after hours. A reboot might help those.
We are looking for the following services in the running process: ngen.exe, ngentask.exe, mscorsvw.exe.
"An MSI install operation is in progress" is very specific: it is saying that a system mutex (mutual exclusion object) named precisely Global\_MSIExecute exists. The MSI installer uses that to ensure that only one installer can run at a time. I don't know of anything you can do about this manually, if you are certain that no MSI installations are happening.
(Note, there was in App Layering 4.2 a bug with upgrading an existing Windows 10 layer from 1611 to 1703 where this flag could be set and not cleared.)
"A reboot is pending to update drivers on the boot disk" is telling you that a service or driver that is set to start at system boot time (the START= value in the registry is 0) was modified or installed, and App Layering wants to make sure the modified driver can boot successfully. Normally you just need to reboot once, and the driver will work fine. We have on some occasions seen software (like Microsoft Defender) attempt to modify its driver file on every single boot, triggering this integrity check every time, so no number of reboots is sufficient to clear it.
"a Microsoft NGen operation is needed" is telling you that an application was installed on the packaging machine and that it scheduled items to be updated at a priority level of 3. That means that the ngen will run when idle and that it is simply waiting until there is no more activity. We are blocking because the ngen needs to create the binaries now instead of on every machine that the application will be deployed to in order to ensure that the application will run in the most optimal way. You should run ngen eqi 3 in both the c:\windows\microsoft.net\framework\v4.0.30319 directory and the c:\windows\microsoft.net\framework64\v4.0.30319 dirctory to have the ngen complete the operations that are needed. You can also wait, as the ngen will typically pick up and run on its own after 15 minutes of idle time.
The values that is being examined are HKLM\SOFTWARE\Microsoft\.NETFramework\v2.0.50727\NGenService\Roots\WorkPending and HKLM\SOFTWARE\WOW6432Node\Microsoft\.NETFramework\v2.0.50727\NGenService\Roots\WorkPending. A value of 1 means that there are work items queued up to be processed.
"Software Center Client is configured to run, but the SMSCFG.INI is still present...." is telling you that we have seen that this machine has ccmexec.exe configured as a service and that it is not configured as disabled. Since we know that any layers created on a packaging machine need to be sealed properly in order to deploy correctly in a VDI environment, we are checking to make sure the SMSCFG.ini is not present. See the web page indicated to get an understanding of why the software center client needs to be sealed. We have provided the commands to run in a batch command file that you can use to seal the layer (run c:\windows\setup\scripts\SEALSCCMCLIENT.cmd for an administrator command window).
"FSLogix driver is configured to run but the altitude is not set to 138010..." is telling you that the FSLogix product is installed and is currently configured to start up, but it is not set to the altiude that is required to run when running with application layering (which is 138010). This is to allow FSLogix to run at a higher altitude than application layer such that FSLogix will be able process the profile data for the users correctly. To do, open regedit and navigate to HKLM\System\CurrentControlSet\Services\frxdrvvt\Instances\frxdrvvt and change the Altitude to 138010. The setting will be noticed by application layering in less than 30 seconds and you should be allowed to then shutdown for finalize. For details about why this needs to be set you can see: https://www.citrix.com/blogs/2020/01/07/citrix-app-layering-and-fslogix-profile-containers/
"There were a large number of files deleted from the base OS in this layer. Any updates or major changes to the OS should be done in the base OS layer rather than the application layers" is telling you that during the creation of the application layer, there were many changes done to the base OS layer that will sometimes result in an application layer that does unexpected things when it is deployed to the image. Typically this is caused by inadvertent Windows updates taking place while the application layer was being created. If that is the case, you more than likely do not want to finalize this layer, and instead, you may want to recreate it after disabling Windows updates in your OS layer. The threshold that is used is 20,000 deletes. Should you decide that you want to still finalize the layer, you can change the threshold by navigating to the HKLM\System\CurrentControlSet\Services\Uniservice key and adding a dword value of the name "AllowableDeleteThreshold" and set it to a number up to 1,000,000. This value is checked every ten seconds. The number of deletes that we believe have happened is listed as the "ApproximateDeleteTokenCount" in the same key, and if the number of deletes that have been seen is less than the threshold number that was set, then the shudown will be allowed. If the number of deletes happens to be greater than 1,000,000, we really stress that the layer will almost certainly cause an issue, and should not be finalized. But should you still want to go forward you can disable the checks as is documented below.
Layer Integrity Bypass:
If you have a layer that simply won't ever get to finalize, for whatever reason (like it always thinks it still has a reboot pending, or you don't care about corrupted .NET assemblies and don't want to wait for NGEN to finish), you can tell that single layer to ignore its layer integrity checks and allow you tin shutdown to finalize, using a registry key.
Run regedit.exe and create this key
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Uniservice:]
"BypassLayerCheck"=DWORD 1
The value doesn't matter, all that matters is that the value exists. This will block all layer integrity checks and allow a layer to be finalized regardless as to how bad we think it might be.
You may need to use this key for applications that update their boot level components on every boot. Application Layering can't tell the difference between the application doing an upgrade or simply replacing their files without making a change. You will see this most often in Anti-Virus applications. Therefore once you know that the application has been updated correctly, you can set the value. This will allow the finalize to go through. Note that you don't want to always just set the key because you are blocked with the finalize. You should only do so when you are confident that you know the application is complete.
Shutdown for Finalize is checking to see if anything is still pending that should happen in the layer rather than happen in the image later. If something is, it does not shut down, and instead puts up a statement about the pending issue. Fix the issue (for instance, reboot) and try again. It also writes this information into two log files:
C:\Program Files\Unidesk\Uniservice\Log\LayerIntegrity.txt
C:\Program Files\Unidesk\Uniservice\Log\UniBilcLogs_X.txt
You can't know exactly which UniBilcLogs file it's using, so look for the one with the latest timestamp. That will be for the current boot. Search for "Integrity".
You might think you could bypass the Layer Integrity check by just shutting down the machine normally and finalizing that. But if you try, you will find the ELM will stop the task and return you to the Packaging Machine, because it knows that the Layer Integrity Checks either failed or never ran. You must successfully run that Shutdown for Finalize script to finalize a layer.
The registry key, noted at the end of this article, to bypass or ignore integrity problems still works, and you should be just as reluctant to use it as ever. But it's still a valid way to give up and bypass it .
There are 9 Layer Integrity warnings you can see:
- "a RunOnce script is outstanding - please check and reboot the Packaging Machine"
- "a post-installation reboot is pending - please check and reboot the Packaging Machine"
- "a Microsoft NGen operation is in progress in the background {0}"
- "an MSI install operation is in progress - please check the Packaging Machine"
- "a reboot is pending to update drivers on the boot disk - please check and reboot the Packaging Machine"
- “a Microsoft NGen operation is needed”
- "Software Center Client is configured to run, but the SMSCFG.INI is still present. See https://social.technet.microsoft.com/wiki/contents/articles/23923.implementing-sccm-in-a-xendesktop-vdi-environment.aspx"
- "FSLogix driver is configured to run but the altitude is not set to 138010 - please check and reboot the packaging machine"
- "There were a large number of files deleted from the base OS in this layer. Any updates or major changes to the OS should be done in the base OS layer rather than the application layers"
"A RunOnce script is outstanding" is telling you that there is a key in either of these two locations:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\RunOnce
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\RunOnce
Windows normally deletes those keys on reboot, but we have seen circumstances (especially with our own script, envsetup.cmd) where that doesn't happen. You can manually run the referenced script and delete the key, or just delete the key if the script file no longer exists.
"A post-installation reboot is pending" is looking at six different registry keys. Your first course of action should be to reboot, more than once(in some cases it has taken 3+ reboots), just to make sure that it isn't a real reboot being requested by some software. It may also be helpful, if the problem is NetLogon, to restart the Unidesk Service for Message Management.
First we check for the existence of any of these three:
HKLM\System\CurrentControlSet\Control\Session Manager\PendingFileRenameOperations
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\RebootPending
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired
You can manually modify any of these to suit your needs, including just deleting them.
Then we look for changes in the NetLogon key (if the current value is now different from what it was at bootup), and to see if the computer name doesn't match the active computer name. This is how we determine that a domain-join operation is still waiting for a reboot.
HKLM\SYSTEM\CURRENTCONTROLSET\SERVICES\NETLOGON\Start
HKLM\SYSTEM\CURRENTCONTROLSET\CONTROL\COMPUTERNAME\ACTIVECOMPUTERNAME
HKLM\SYSTEM\CURRENTCONTROLSET\CONTROL\COMPUTERNAME\COMPUTERNAME
Generally you cannot modify these. I've seen some software modify the NETLOGON\Start key on every reboot, so maybe that's happening. If, after cleaning out the top three, you still get the prompt on reboot, you will need to use the flag to ignore layer integrity checks.
"A Microsoft NGen operation is in progress in the background" is telling you that a foreground or background NGEN operation (where .Net assemblies are compiled into native images) is still in progress. Generally this is simply true: the ngen rebuild is still running. To watch it in the foreground, run "ngen update /force". Or you can wait it out, and run "ngen queue status" periodically to see how it's doing, but that will slow it down because the background process pauses every time you check its status in the foreground. Don't reboot or you might cause it to have to start over.
It's important to let NGEN finish. If you kill the process or reboot in the middle, you might wind up with partially written .Net assemblies that crash programs when they show up in an image. You should be patient. However, sometimes we have seen background MSCORSVW.EXE processes, clearly doing nothing, that just don't finish even after hours. A reboot might help those.
We are looking for the following services in the running process: ngen.exe, ngentask.exe, mscorsvw.exe.
"An MSI install operation is in progress" is very specific: it is saying that a system mutex (mutual exclusion object) named precisely Global\_MSIExecute exists. The MSI installer uses that to ensure that only one installer can run at a time. I don't know of anything you can do about this manually, if you are certain that no MSI installations are happening.
(Note, there was in App Layering 4.2 a bug with upgrading an existing Windows 10 layer from 1611 to 1703 where this flag could be set and not cleared.)
"A reboot is pending to update drivers on the boot disk" is telling you that a service or driver that is set to start at system boot time (the START= value in the registry is 0) was modified or installed, and App Layering wants to make sure the modified driver can boot successfully. Normally you just need to reboot once, and the driver will work fine. We have on some occasions seen software (like Microsoft Defender) attempt to modify its driver file on every single boot, triggering this integrity check every time, so no number of reboots is sufficient to clear it.
"a Microsoft NGen operation is needed" is telling you that an application was installed on the packaging machine and that it scheduled items to be updated at a priority level of 3. That means that the ngen will run when idle and that it is simply waiting until there is no more activity. We are blocking because the ngen needs to create the binaries now instead of on every machine that the application will be deployed to in order to ensure that the application will run in the most optimal way. You should run ngen eqi 3 in both the c:\windows\microsoft.net\framework\v4.0.30319 directory and the c:\windows\microsoft.net\framework64\v4.0.30319 dirctory to have the ngen complete the operations that are needed. You can also wait, as the ngen will typically pick up and run on its own after 15 minutes of idle time.
The values that is being examined are HKLM\SOFTWARE\Microsoft\.NETFramework\v2.0.50727\NGenService\Roots\WorkPending and HKLM\SOFTWARE\WOW6432Node\Microsoft\.NETFramework\v2.0.50727\NGenService\Roots\WorkPending. A value of 1 means that there are work items queued up to be processed.
"Software Center Client is configured to run, but the SMSCFG.INI is still present...." is telling you that we have seen that this machine has ccmexec.exe configured as a service and that it is not configured as disabled. Since we know that any layers created on a packaging machine need to be sealed properly in order to deploy correctly in a VDI environment, we are checking to make sure the SMSCFG.ini is not present. See the web page indicated to get an understanding of why the software center client needs to be sealed. We have provided the commands to run in a batch command file that you can use to seal the layer (run c:\windows\setup\scripts\SEALSCCMCLIENT.cmd for an administrator command window).
"FSLogix driver is configured to run but the altitude is not set to 138010..." is telling you that the FSLogix product is installed and is currently configured to start up, but it is not set to the altiude that is required to run when running with application layering (which is 138010). This is to allow FSLogix to run at a higher altitude than application layer such that FSLogix will be able process the profile data for the users correctly. To do, open regedit and navigate to HKLM\System\CurrentControlSet\Services\frxdrvvt\Instances\frxdrvvt and change the Altitude to 138010. The setting will be noticed by application layering in less than 30 seconds and you should be allowed to then shutdown for finalize. For details about why this needs to be set you can see: https://www.citrix.com/blogs/2020/01/07/citrix-app-layering-and-fslogix-profile-containers/
"There were a large number of files deleted from the base OS in this layer. Any updates or major changes to the OS should be done in the base OS layer rather than the application layers" is telling you that during the creation of the application layer, there were many changes done to the base OS layer that will sometimes result in an application layer that does unexpected things when it is deployed to the image. Typically this is caused by inadvertent Windows updates taking place while the application layer was being created. If that is the case, you more than likely do not want to finalize this layer, and instead, you may want to recreate it after disabling Windows updates in your OS layer. The threshold that is used is 20,000 deletes. Should you decide that you want to still finalize the layer, you can change the threshold by navigating to the HKLM\System\CurrentControlSet\Services\Uniservice key and adding a dword value of the name "AllowableDeleteThreshold" and set it to a number up to 1,000,000. This value is checked every ten seconds. The number of deletes that we believe have happened is listed as the "ApproximateDeleteTokenCount" in the same key, and if the number of deletes that have been seen is less than the threshold number that was set, then the shudown will be allowed. If the number of deletes happens to be greater than 1,000,000, we really stress that the layer will almost certainly cause an issue, and should not be finalized. But should you still want to go forward you can disable the checks as is documented below.
Layer Integrity Bypass:
If you have a layer that simply won't ever get to finalize, for whatever reason (like it always thinks it still has a reboot pending, or you don't care about corrupted .NET assemblies and don't want to wait for NGEN to finish), you can tell that single layer to ignore its layer integrity checks and allow you tin shutdown to finalize, using a registry key.
Run regedit.exe and create this key
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Uniservice:]
"BypassLayerCheck"=DWORD 1
The value doesn't matter, all that matters is that the value exists. This will block all layer integrity checks and allow a layer to be finalized regardless as to how bad we think it might be.
You may need to use this key for applications that update their boot level components on every boot. Application Layering can't tell the difference between the application doing an upgrade or simply replacing their files without making a change. You will see this most often in Anti-Virus applications. Therefore once you know that the application has been updated correctly, you can set the value. This will allow the finalize to go through. Note that you don't want to always just set the key because you are blocked with the finalize. You should only do so when you are confident that you know the application is complete.
Was this article helpful?
Yes
No
Powered by
