Tasks won't progress, appear as stalled.
New tasks sit at "Pending"


From maservice.log:

2021-01-11 09:10:25,575 WARN  [20193]        UnitOfWorkImplemen: Transaction is implicitly being rolled back
  at Uni.Appliance.PersistenceLayer.UnitOfWork.UnitOfWorkImplementation.Dispose () [0x00025] in <3c18c9c6b1c94577b82288e36355214e>:0
  at Unidesk.Ma.MaService.Services.WorkItemCheckerService.CheckWorkItems () [0x002cc] in <1cde95bcf00b4efd94ee6ad0e3a8e8dc>:0
  at Uni.Core.Base.CoalescedAction.CallbackWrapper () [0x00020] in <9672185b044649c0beb38c9185a8a6e6>:0
  at Uni.Core.Base.ThreadWrapper.WrapperMethod (System.Action action) [0x00000] in <9672185b044649c0beb38c9185a8a6e6>:0
  at Uni.Core.Base.ThreadWrapper+<>c__DisplayClass5_0.<GetTaskAction>b__0 () [0x00000] in <9672185b044649c0beb38c9185a8a6e6>:0
  at System.Threading.Tasks.Task.InnerInvoke () [0x0000f] in <6de68d9ed0b943838b4ddc538c95c31e>:0
  at System.Threading.Tasks.Task.Execute () [0x00000] in <6de68d9ed0b943838b4ddc538c95c31e>:0
  at System.Threading.Tasks.Task.ExecutionContextCallback (System.Object obj) [0x00006] in <6de68d9ed0b943838b4ddc538c95c31e>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00071] in <6de68d9ed0b943838b4ddc538c95c31e>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <6de68d9ed0b943838b4ddc538c95c31e>:0
  at System.Threading.Tasks.Task.ExecuteWithThreadLocal (System.Threading.Tasks.Task& currentTaskSlot) [0x00034] in <6de68d9ed0b943838b4ddc538c95c31e>:0
  at System.Threading.Tasks.Task.ExecuteEntry (System.Boolean bPreventDoubleExecution) [0x0004a] in <6de68d9ed0b943838b4ddc538c95c31e>:0
  at System.Threading.Tasks.Task.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in <6de68d9ed0b943838b4ddc538c95c31e>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <6de68d9ed0b943838b4ddc538c95c31e>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <6de68d9ed0b943838b4ddc538c95c31e>:0 .

2021-01-11 09:10:25,578 ERROR [20193]        WorkItemCheckerSer: Caught an exception while processing WorkItems:
Uni.Core.Handlers.Exceptions.GlobalizedErrorException`1[Uni.Core.Contract.Results.ItemNotFoundCategory]: MessageId=ItemNotFound, DefaultTitle=, CategoryData={[ItemNotFoundCategory { Item = "ConnectorCachedLayer", AdditionalInfo = 24412162 }]}

Verify the below first:

1. There are no hidden tasks associated with the stuck, pending, stalled tasks
2. The layers are not listed in a template image or associated with another layer
3. Ensure all tasks that can be canceled have been. This may take a few tries and reboots of the ELM

This will clear all tasks from the database.

1. Login to the ELM as root
2. Enter and execute
   1. systemctl stop maservice
   2. On Azure: sudo systemctl stop maservice
3. Connect to MySQL by entering and executing
   1. mysql -uroot -proot MAData
   2. On Azure: sudo mysql -uroot -proot MAData
4. Enter and execute the below commands to clear all tasks
   1. delete from WorkItemContext;
   2. update WorkItem Join WorkTicket on wtiWorkTicketId = wtId SET wtiState = 'Canceled' WHERE wtiState not in('Complete', 'Failed', 'Canceled');
   3. update WorkItem Set wtiUserActionPerformed = 0 Where wtiUserActionPerformed != 0;
   4. delete from StoredRegisteredTask;
   5. UPDATE LayerRevision set lrStatus = 'Broken' where lrStatus in ('Editing', 'Finalizable', 'Finalizing', 'Deleting');
5. Reboot the ELM and check the task queue from the web console
6. Attempt your tasks again, Should the failure persists, export a new set of ELM logs and attach the data to an existing case, via cis.citrix.com, or a new case.

Problem Cause

There can be multiple reasons for this issue. One reason found is when a connector setting is changed while a task using the same connector is already running.