====== Citrix XenServer Troubleshooting ======

**Tools for XenServer Troubleshooting**: http://blogs.citrix.com/2011/05/06/tools-for-xenserver-troubleshooting/

**How to Collect Diagnostic Information for XenServer**: http://support.citrix.com/article/CTX125372

**Helpful XenServer Commands**: http://support.randomsolutions.nl/432213-Helpfull-XenServer-Commands

===== Kernel Panic (Hang on Boot) =====

Try booting with the 'Fallback' kernel and look for clues in the boot messages.

Type ''fallback'' or ''fallback-serial'' at the ''boot:'' prompt.

==== RAID Metadata on Disk ====

See also **[[computing:storage:raid_metadata_removal|RAID Metadata Removal]]**

:!: If you are installing to disks that previously had been used in RAID configurations, you may get a **kernel panic** (hang on boot).

:!: You probably won't see the kernel panic (hang on boot) until **after installing SP1** (XenServer 6.2).

==== Check Disk Space ====

<file>
df -h | grep "/$" | head -n 1
</file>

Looking for files bigger than a given size (specify path and size):

<file>
find / -mount -type f -size +20M -exec ls -lh {} \; | awk '{ print $9 ": " $5 }'
</file>

:!: You can pretty much ignore anything in ''/var/run/sr-mount/'' as it's not on the local filesystem.

:!: Take special care when using wildcards with the ''rm'' command!

The first step might be to delete all files such as downloaded patches from ''/root'':

<file>
ls -al /root
rm -rf /root/*
</file>

===== File System on Control Domain Full =====

https://support.citrix.com/article/CTX202821

http://xenserver.org/component/easyblog/entry/xenserver-root-disk-maintenance.html?Itemid=179

===== Detach a VHD from the Control Domain =====

<note important>It is suggested to rename the REAL virtual disk so you DO NOT delete the wrong one!  Rename the real one because you can't rename the one connected to the control domain.  You can rename the REAL one back to the original name when you are done.</note>

List out the VBD (connector object between VM and VDI) based on the VDI UUID:

<file>
xe vdi-list | grep -B1 -A5 -i "part-of-vm-name"

xe vdi-list name-label="<VDI-name-label>"

xe vbd-list vdi-uuid=<UUID of VDI>

</file>
<note important>Verify the VBD is the one connected to control domain!
</note>

Once done, you want to first try to make VBD inactive (it may already be inactive),

<file>
xe vbd-unplug uuid=<UUID of VBD>
</file>

Once done, you can then break the connection:

<file>
xe vbd-destroy uuid=<UUID of VBD>
</file>

Check in XenCenter to verify and delete the virtual disk.

===== Log Partition Full =====

Delete rotated log files:

<file>
find /var/log/ -type f -name *.gz -exec rm -f {} \;
</file>

Delete ''tapdisk'' log files:

<file>
rm -f /var/log/blktap/*
</file>

Delete applied patches:

:!: Works pool-wide if run on the pool master server.

<file>
for i in `xe patch-list --minimal|tr "," " "`;do xe patch-pool-clean uuid=$i;done
</file>

Delete 'orphaned' patches and backup patches:

:!: Do not delete /var/patch/applied or its contents.

<file>
rm /var/patch/*-*-*-*-*
</file>

For true emergencies or for patch install problems, you can delete patch backups:

<file>
rm -rf /opt/xensource/patch-backup/*-*-*-*-*
</file>

:!: On the Pool Masters, we mount an NFS backup share on /mnt/backup.

Unmount the backup share and delete any 'hidden' files:

<file>
umount /mnt/backup && rm -rf /mnt/backup/*
mount -a
</file>

===== XAPI Database Parsing Tools =====

Windows Version: http://support.citrix.com/article/CTX129070

Linux Version: http://support.citrix.com/article/CTX129217

===== XenTools =====

FIXME Incomplete and needs verification

This is a procedure to fix XenTools networking issues:

  - Remove network interface via XenCenter
  - Uninstall all Citrix XenTools packages from Add/Remove Programs
  - Remove Citrix from filesystem and registry
    * ''c:\program files\citrix''
    * ''c:\program files (x86)\citrix''
    * ''HKLM\Software\Citrix''
  - Run in an admin command shell
    * Stop all running XenTools services
    * ''sc delete xenlite''
    * ''winmgmt /resetrepository''
  - Reboot
  - Check things out and remove any 'ghosted' or hidden network adapters
  - Reboot again
  - Add network adapter in XenCenter
  - Reconfigure the networking until it works properly again
  - Reboot
  - Reinstall XenTools patiently