My main ZFS pool consists of two RAID-Z2 VDEVs with eight disks;

  1. 8×4 TB
  2. 6×8 TB + 1×4 TB + 1×2 TB

My 2nd VDEV is a mix of disks, as I didn’t have the 💰 to buy all 8 TB when setting it up. The smallest disk is the limiting factor, so currently the capacity is only 8×2=16 TB (minus two disk parity usable).

I just bought a new 8 TB disk, so by replacing the 2 TB I will double the capacity! Let’s do that now.

First let’s offline the device we are going to replace (I’m using /dev/disk/by-id/ to identify my disks.)

$ sudo zpool offline tank0 ata-WDC_WD2003FZEX-00SRLA0_WD-xxxxxxxxxxxx

Now let us have a look at the pool status.

$ zpool status

NAME                                            STATE     READ WRITE CKSUM
tank0                                           DEGRADED     0     0     0
  raidz2-0                                      ONLINE       0     0     0
    ata-ST4000VN008-2DR166_xxxxxxxx             ONLINE       0     0     0
    ata-ST4000VN008-2DR166_xxxxxxxx             ONLINE       0     0     0
    ata-ST4000VN008-2DR166_xxxxxxxx             ONLINE       0     0     0
    ata-WDC_WD40EFRX-68N32N0_WD-xxxxxxxxxxxx    ONLINE       0     0     0
    ata-WDC_WD40EFRX-68N32N0_WD-xxxxxxxxxxxx    ONLINE       0     0     0
    ata-WDC_WD40EFRX-68N32N0_WD-xxxxxxxxxxxx    ONLINE       0     0     0
    ata-WDC_WD40EFRX-68N32N0_WD-xxxxxxxxxxxx    ONLINE       0     0     0
    ata-WDC_WD40EFRX-68N32N0_WD-xxxxxxxxxxxx    ONLINE       0     0     0
  raidz2-1                                      DEGRADED     0     0     0
    ata-TOSHIBA_HDWN180_xxxxxxxxxxxx            ONLINE       0     0     0
    ata-TOSHIBA_HDWN180_xxxxxxxxxxxx            ONLINE       0     0     0
    ata-TOSHIBA_HDWN180_xxxxxxxxxxxx            ONLINE       0     0     0
    ata-WDC_WD80EFZX-68UW8N0_xxxxxxxx           ONLINE       0     0     0
    ata-TOSHIBA_HDWG180_xxxxxxxxxxxx            ONLINE       0     0     0
    ata-TOSHIBA_HDWG180_xxxxxxxxxxxx            ONLINE       0     0     0
    ata-WDC_WD2003FZEX-00SRLA0_WD-xxxxxxxxxxxx  OFFLINE      0     0     0
    ata-ST4000VX007-2DT166_xxxxxxxx             ONLINE       0     0     0

Sweet, the device is offline — time to shut the server down and physically replace the disk.

$ sudo shutdown now

Now — the server is back up and it’s time to instruct ZFS to replace the removed device with the disk we just installed.

$ sudo zpool replace tank0 \
    ata-WDC_WD2003FZEX-00SRLA0_WD-xxxxxxxxxxxx \
    /dev/disk/by-id/ata-TOSHIBA_HDWG180_xxxxxxxxxxxx

$ zpool status

NAME                                              STATE     READ WRITE CKSUM
tank0                                             DEGRADED     0     0     0
  raidz2-0                                        ONLINE       0     0     0
    ata-ST4000VN008-2DR166_xxxxxxxx               ONLINE       0     0     0
    ata-ST4000VN008-2DR166_xxxxxxxx               ONLINE       0     0     0
    ata-ST4000VN008-2DR166_xxxxxxxx               ONLINE       0     0     0
    ata-WDC_WD40EFRX-68N32N0_WD-xxxxxxxxxxxx      ONLINE       0     0     0
    ata-WDC_WD40EFRX-68N32N0_WD-xxxxxxxxxxxx      ONLINE       0     0     0
    ata-WDC_WD40EFRX-68N32N0_WD-xxxxxxxxxxxx      ONLINE       0     0     0
    ata-WDC_WD40EFRX-68N32N0_WD-xxxxxxxxxxxx      ONLINE       0     0     0
    ata-WDC_WD40EFRX-68N32N0_WD-xxxxxxxxxxxx      ONLINE       0     0     0
  raidz2-1                                        DEGRADED     0     0     0
    ata-TOSHIBA_HDWN180_xxxxxxxxxxxx              ONLINE       0     0     0
    ata-TOSHIBA_HDWN180_xxxxxxxxxxxx              ONLINE       0     0     0
    ata-TOSHIBA_HDWN180_xxxxxxxxxxxx              ONLINE       0     0     0
    ata-WDC_WD80EFZX-68UW8N0_xxxxxxxx             ONLINE       0     0     0
    ata-TOSHIBA_HDWG180_xxxxxxxxxxxx              ONLINE       0     0     0
    ata-TOSHIBA_HDWG180_xxxxxxxxxxxx              ONLINE       0     0     0
    replacing-6                                   DEGRADED     0     0     0
      ata-WDC_WD2003FZEX-00SRLA0_WD-xxxxxxxxxxxx  OFFLINE      0     0     0
      ata-TOSHIBA_HDWG180_xxxxxxxxxxxx            ONLINE       0     0     0  (resilvering)
    ata-ST4000VX007-2DT166_xxxxxxxx               ONLINE       0     0     0

The disk is replaced and getting resilvered.

…one eternity later…

Once the resilvering is done; this is what the pool looks like.

$ zpool list

NAME      SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank0    43.5T  33.0T  10.5T     14.5T     7%    75%  1.00x  ONLINE  -

The capacity hasn’t increased yet, but we now have 14.5T of expandable space. So let’s run the expand command on the new disk.

$ sudo zpool online -e tank0 ata-TOSHIBA_HDWG180_xxxxxxxxxxxx

NAME      SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank0      58T  33.0T  25.0T         -     5%    56%  1.00x  ONLINE  -

Nice! The pool has grown 👍 Next time I will be replacing the 4 TB to fully utilize all disks 😁

P.S. I did replace a few 2 TB disks with 8 TB before this, but I only needed to run the expand command once — as show above.