Error in handleInterruptedCoalesceLeaf when trying to scan the Storage on XenServer

Error in handleInterruptedCoalesceLeaf when trying to scan the Storage on XenServer

book

Article ID: CTX202815

calendar_today

Updated On:

Description

  • Error in XenCenter - SR failed to complete the operation
  • Error in /var/log/SMlog
-[25840] 2012-12-14 10:52:35.402274 lock: released /var/lock/sm/lvm-579fdc35-f120-398e-397e-b12179c9531c/83b6fcfd-e9ed-432b-b662-d8d28f9ccca5
[25840] 2012-12-14 10:52:35.402380 lock: closed /var/lock/sm/lvm-579fdc35-f120-398e-397e-b12179c9531c/83b6fcfd-e9ed-432b-b662-d8d28f9ccca5
[25840] 2012-12-14 10:52:35.404959 *** LVHD: EXCEPTION XenAPI.Failure, ['XENAPI_PLUGIN_FAILURE', 'multi', 'CommandException', '5']
File "/opt/xensource/sm/LVMSR", line 1985, in ?
SRCommand.run(LVHDSR, DRIVER_INFO)
File "/opt/xensource/sm/SRCommand.py", line 307, in run
sr = driver(cmd, cmd.sr_uuid)
File "/opt/xensource/sm/SR.py", line 136, in __init__
self.load(sr_uuid)
File "/opt/xensource/sm/LVMSR", line 196, in load
self._undoAllJournals()
File "/opt/xensource/sm/LVMSR", line 1043, in _undoAllJournals
self._undoAllInflateJournals()
File "/opt/xensource/sm/LVMSR", line 1071, in _undoAllInflateJournals
self.vgname, vdi.lvname, uuid)
File "/opt/xensource/sm/lvhdutil.py", line 302, in lvRefreshOnAllSlaves
lvRefreshOnSlaves(session, srUuid, vgName, lvName, vdiUuid, slaves)
File "/opt/xensource/sm/lvhdutil.py", line 297, in lvRefreshOnSlaves
text = session.xenapi.host.call_plugin(slave, "on-slave", "multi", args)
File "/usr/lib/python2.4/site-packages/XenAPI.py", line 234, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python2.4/site-packages/XenAPI.py", line 138, in xenapi_request
result = _parse_result(getattr(self, methodname)(*full_params))
File "/usr/lib/python2.4/site-packages/XenAPI.py", line 208, in _parse_result
raise Failure(result['ErrorDescription'])



OR

# xe sr-scan uuid=579fdc35-f120-398e-397e-b12179c9531c
There was an SR backend failure.
status: non-zero exit
stdout:
stderr: Traceback (most recent call last):
  File "/opt/xensource/sm/LVMoHBASR", line 235, in ?
    SRCommand.run(LVHDoHBASR, DRIVER_INFO)
  File "/opt/xensource/sm/SRCommand.py", line 325, in run
    sr = driver(cmd, cmd.sr_uuid)
  File "/opt/xensource/sm/SR.py", line 135, in __init__
    self.load(sr_uuid)
  File "/opt/xensource/sm/LVMoHBASR", line 98, in load
    self._pathrefresh(LVHDoHBASR)
  File "/opt/xensource/sm/SR.py", line 466, in _pathrefresh
    super(obj, self).load(self.uuid)
  File "/opt/xensource/sm/LVHDSR.py", line 196, in load
    self._undoAllJournals()
  File "/opt/xensource/sm/LVHDSR.py", line 1046, in _undoAllJournals
    self._handleInterruptedCoalesceLeaf()
  File "/opt/xensource/sm/LVHDSR.py", line 807, in _handleInterruptedCoalesceLeaf
    cleanup.gc_force(self.session, self.uuid)
  File "/opt/xensource/sm/cleanup.py", line 2614, in gc_force
    sr.scanLocked(force)
  File "/opt/xensource/sm/cleanup.py", line 1306, in scanLocked
    self.scan(force)
  File "/opt/xensource/sm/cleanup.py", line 2152, in scan
    self._handleInterruptedCoalesceLeaf()
  File "/opt/xensource/sm/cleanup.py", line 2232, in _handleInterruptedCoalesceLeaf
    self._finishInterruptedCoalesceLeaf(uuid, parentUuid)
  File "/opt/xensource/sm/cleanup.py", line 2293, in _finishInterruptedCoalesceLeaf
    vdi.inflateFully()
  File "/opt/xensource/sm/cleanup.py", line 959, in inflateFully
    self.inflate(lvhdutil.calcSizeVHDLV(self.sizeVirt))
  File "/opt/xensource/sm/cleanup.py", line 938, in inflate
    lvhdutil.inflate(self.sr.journaler, self.sr.uuid, self.uuid, size)
  File "/opt/xensource/sm/lvhdutil.py", line 176, in inflate
    lvmCache.setSize(lvName, newSize)
  File "/opt/xensource/sm/lvmcache.py", line 44, in wrapper
    ret = op(self, *args)
  File "/opt/xensource/sm/lvmcache.py", line 126, in setSize
    lvutil.setSize(path, newSize, (newSize < size))
  File "/opt/xensource/sm/lvutil.py", line 445, in setSize
    util.pread(cmd)
  File "/opt/xensource/sm/util.py", line 174, in pread
    raise CommandException(rc, str(cmdlist), stderr.strip())
util.CommandException: 5

  • VMs with disks on that Storage repository will not boot.
  • Cannot add a new Disk to that SR.
  • Cannot delete a disk in that SR.

Resolution

  • Copy the disk UUID from the Error message in SMlog
  • Run #lvscan | grep -i <UUID_OF_DISK>
  • Now you will get an output with the original disk and the inflate disk.
  • Now run #lvremove <path_to_inflate_disk>
  • Now try to Re-scan the SR - It should work.
Example from the above error
  • Copy the highlighted portion from the below
  • Error in /var/log/SMlog
  • -[25840] 2012-12-14 10:52:35.402274 lock: released /var/lock/sm/lvm-579fdc35-f120-398e-397e-b12179c9531c/83b6fcfd-e9ed-432b-b662-d8d28f9ccca5

     
  • lvscan | grep -i 83b6fcfd-e9ed-432b-b662-d8d28f9ccca5
 Output would look like 

  
ACTIVE '/dev/VG_XenStorage-579fdc35-f120-398e-397e-b12179c9531c/VHD-83b6fcfd-e9ed-432b-b662-d8d28f9ccca5' [30.07 GB] inherit
   ACTIVE '/dev/VG_XenStorage-579fdc35-f120-398e-397e-b12179c9531c/inflate_VHD-83b6fcfd-e9ed-432b-b662-d8d28f9ccca5' [4.00 MB] inherit
  • lvremove  /dev/VG_XenStorage-579fdc35-f120-398e-397e-b12179c9531c/inflate_VHD-83b6fcfd-e9ed-432b-b662-d8d28f9ccca5
  Make sure you remove only the inflate disk and not the original disk
 

Problem Cause

Inflate disk was not deleted automatically. The inflate disk would have been created as part of a snapshot or Garbage collection process

Issue/Introduction

Error *** LVHD: EXCEPTION XenAPI.Failure, ['XENAPI_PLUGIN_FAILURE', 'multi', 'CommandException', '5'] when Trying to Scan an SR.