ZFS Failed Disk Replacement

ZFS Failed Disk Replacement

Identify the problem drive and location exactly before replacing!

Power the server off when manipulating disk drives, if at all possible. This can go a long way towards eliminating self inflicted wounds!

The normal replacement procedure involves fixing the pool first, then removing the failed drive. Replace in-place if you must.

Napp-IT

Replace with spare:

Degraded → Offline → Replace

Replace in-place drive:

Degraded → Offline → Remove → Pull Bad Drive → Install New Drive → Initialize → Add to VDEV

Wait for the resilvering to finish then run a scrub job.

CLI

Replace a Failed Drive

https://www.thegeekdiary.com/solaris-zfs-how-to-offline-online-detach-replace-device-in-a-storage-pool/

Use zpool replace to replace a failed drive. It is safest to replace a failed drive with another existent drive in another slot.

zpool replace <poolname> <faileddisk> <newdisk>

To replace a failed drive by replacing it in-place, it is best to shut down the server, remove and replace the failed drive, then use zpool replace specifying only the drive being replaced.

zpool replace <poolname> <disk>

After initializing a new replacement drive in-place using Napp-IT on OmniOS, this is the command that Napp-IT uses when replacing a drive.

zpool replace -f "poolname" c0t5000C500418A83B3d0 c0t5000C500836A90C7d0

zpool detach will remove a drive from a VDEV, turning a mirrored VDEV into a single drive VDEV. Use zpool attach to recreate the mirror. In general, do not use these commands to replace a failed drive.

Replace Old Drives With New or Bigger

https://www.thegeekdiary.com/solaris-zfs-how-to-offline-online-detach-replace-device-in-a-storage-pool/

It is safest to zpool attach an additional mirror to a mirrored VDEV (which adds redundancy) then remove the drive being upgraded afterwards.

zpool attach <poolname> <existingvdevmember> <newdrive>
# Wait for it to resilver, which adds redundancy, then
zpool detach <poolname> <drivebeingreplaced>

Spares

Usually a spare can replace a faulted disk but remains a spare. The idea behind this is that you replace the faulted disk, then replace the spare with the new and the spare remains a spare.

If you want to remove the spare property, first remove then replace.

Pool Expansion

https://tomasz.korwel.net/2014/01/03/growing-zfs-pool/

https://docs.oracle.com/cd/E36784_01/html/E36835/gazgm.html

CLI

You can expand a ZFS pool by replacing redundant drives on replacing them one at a time. Be sure to wait for the resilvering to finish before replacing the next drive!

Shut down the server to physically replace drives. If you have hotswap capability, you can use zpool offline <poolname> <disk> to tell ZFS to ignore the device being replaced.

# Set the autoexpand property so the new disk space is recognized
zpool set autoexpand=on <poolname>
# Verify everything is online before replacing a disk
zpool status
# Take the old, smaller disk offline before removing it
zpool offline <poolname> <olddisk>
##### Physically replace the disk with the new larger disk #####
zpool online <poolname> <samediskname>
# Optionally kick off the autoexpand of each new larger device
zpool online -e <poolname> <device>

Table of Contents