I’m replacing a disk and and expanding my ZFS 2×RAID-Z2 pool 👇
My main ZFS pool consists of two RAID-Z2 VDEVs with eight disks;
- 8×4 TB
- 6×8 TB + 1×4 TB + 1×2 TB
My 2nd VDEV is a mix of disks, as I didn’t have the 💰 to buy all 8 TB when setting it up. The smallest disk is the limiting factor, so currently the capacity is only 8×2=16 TB (minus two disk parity usable).
I just bought a new 8 TB disk, so by replacing the 2 TB I will double the capacity! Let’s do that now.
First let’s offline the device we are going to replace (I’m using /dev/disk/by-id/ to identify my disks.)
$ sudo zpool offline tank0 ata-WDC_WD2003FZEX-00SRLA0_WD-xxxxxxxxxxxx
Now let us have a look at the pool status.
$ zpool status
NAME STATE READ WRITE CKSUM
tank0 DEGRADED 0 0 0
raidz2-0 ONLINE 0 0 0
ata-ST4000VN008-2DR166_xxxxxxxx ONLINE 0 0 0
ata-ST4000VN008-2DR166_xxxxxxxx ONLINE 0 0 0
ata-ST4000VN008-2DR166_xxxxxxxx ONLINE 0 0 0
ata-WDC_WD40EFRX-68N32N0_WD-xxxxxxxxxxxx ONLINE 0 0 0
ata-WDC_WD40EFRX-68N32N0_WD-xxxxxxxxxxxx ONLINE 0 0 0
ata-WDC_WD40EFRX-68N32N0_WD-xxxxxxxxxxxx ONLINE 0 0 0
ata-WDC_WD40EFRX-68N32N0_WD-xxxxxxxxxxxx ONLINE 0 0 0
ata-WDC_WD40EFRX-68N32N0_WD-xxxxxxxxxxxx ONLINE 0 0 0
raidz2-1 DEGRADED 0 0 0
ata-TOSHIBA_HDWN180_xxxxxxxxxxxx ONLINE 0 0 0
ata-TOSHIBA_HDWN180_xxxxxxxxxxxx ONLINE 0 0 0
ata-TOSHIBA_HDWN180_xxxxxxxxxxxx ONLINE 0 0 0
ata-WDC_WD80EFZX-68UW8N0_xxxxxxxx ONLINE 0 0 0
ata-TOSHIBA_HDWG180_xxxxxxxxxxxx ONLINE 0 0 0
ata-TOSHIBA_HDWG180_xxxxxxxxxxxx ONLINE 0 0 0
ata-WDC_WD2003FZEX-00SRLA0_WD-xxxxxxxxxxxx OFFLINE 0 0 0
ata-ST4000VX007-2DT166_xxxxxxxx ONLINE 0 0 0
Sweet, the device is offline — time to shut the server down and physically replace the disk.
$ sudo shutdown now
Now — the server is back up and it’s time to instruct ZFS to replace the removed device with the disk we just installed.
$ sudo zpool replace tank0 \
ata-WDC_WD2003FZEX-00SRLA0_WD-xxxxxxxxxxxx \
/dev/disk/by-id/ata-TOSHIBA_HDWG180_xxxxxxxxxxxx
$ zpool status
NAME STATE READ WRITE CKSUM
tank0 DEGRADED 0 0 0
raidz2-0 ONLINE 0 0 0
ata-ST4000VN008-2DR166_xxxxxxxx ONLINE 0 0 0
ata-ST4000VN008-2DR166_xxxxxxxx ONLINE 0 0 0
ata-ST4000VN008-2DR166_xxxxxxxx ONLINE 0 0 0
ata-WDC_WD40EFRX-68N32N0_WD-xxxxxxxxxxxx ONLINE 0 0 0
ata-WDC_WD40EFRX-68N32N0_WD-xxxxxxxxxxxx ONLINE 0 0 0
ata-WDC_WD40EFRX-68N32N0_WD-xxxxxxxxxxxx ONLINE 0 0 0
ata-WDC_WD40EFRX-68N32N0_WD-xxxxxxxxxxxx ONLINE 0 0 0
ata-WDC_WD40EFRX-68N32N0_WD-xxxxxxxxxxxx ONLINE 0 0 0
raidz2-1 DEGRADED 0 0 0
ata-TOSHIBA_HDWN180_xxxxxxxxxxxx ONLINE 0 0 0
ata-TOSHIBA_HDWN180_xxxxxxxxxxxx ONLINE 0 0 0
ata-TOSHIBA_HDWN180_xxxxxxxxxxxx ONLINE 0 0 0
ata-WDC_WD80EFZX-68UW8N0_xxxxxxxx ONLINE 0 0 0
ata-TOSHIBA_HDWG180_xxxxxxxxxxxx ONLINE 0 0 0
ata-TOSHIBA_HDWG180_xxxxxxxxxxxx ONLINE 0 0 0
replacing-6 DEGRADED 0 0 0
ata-WDC_WD2003FZEX-00SRLA0_WD-xxxxxxxxxxxx OFFLINE 0 0 0
ata-TOSHIBA_HDWG180_xxxxxxxxxxxx ONLINE 0 0 0 (resilvering)
ata-ST4000VX007-2DT166_xxxxxxxx ONLINE 0 0 0
The disk is replaced and getting resilvered.
…one eternity later…
Once the resilvering is done; this is what the pool looks like.
$ zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
tank0 43.5T 33.0T 10.5T 14.5T 7% 75% 1.00x ONLINE -
The capacity hasn’t increased yet, but we now have 14.5T of expandable space. So let’s run the expand command on the new disk.
$ sudo zpool online -e tank0 ata-TOSHIBA_HDWG180_xxxxxxxxxxxx
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
tank0 58T 33.0T 25.0T - 5% 56% 1.00x ONLINE -
Nice! The pool has grown 👍 Next time I will be replacing the 4 TB to fully utilize all disks 😁
P.S. I did replace a few 2 TB disks with 8 TB before this, but I only needed to run the expand command once — as shown above.
Last commit 2024-04-05, with message: Tag cleanup.
