A few days ago; sitting outside — drinking coffee and minding my own business, I received an email from my UPS:

Utility power not available.

2021/06/01 18:38:27

My homelab had lost both power sources 😮 This whole area had a power outage — a good opportunity to do a UPS and power down test 😛

Shutdown

26 minutes and 18 seconds after the power loss, I received another email:

The UPS batteries backup time is below the setting limit. [250 sec < 300 sec]

2021/06/01 19:04:46

The UPS has less than 5 minutes of runtime left, and shutdown of all servers was initiated.

I’m using a Raspberry Pi running nut (Network UPS Tools) server to communicate with my UPS, and send shutdown commands to all my servers using upsmon.

More information about the system on my homelab page.

Shutting down the file server and Docker host was quick. But the two hypervisors used a long time, Epsilon shut down just in time before the UPS turned off — Alpha did not.

I’m pretty sure the running KVM virtual machines are to blame. I’ve seen before that they take a long time to stop when shutting the host down. I usually run a script to stop all running VMs before shutting the servers down. I think I need to add that to the upsmon.conf.

Apart from that; the shutdown went well 🙂

Power up

The power was out for about an hour. When it returned; the UPS powered up and so did all the equipment and servers (due to the power on after AC loss BIOS setting). Getting back up after a complete shutdown is always interesting, this time was no different.

  • Raspberry Pi nut server didn’t establish connection with the UPS
  • Servers and containers booted up before the core switch was ready
    • NFS mounting failed
    • Home Assistant was unable to initiate devices
  • Two disk on my main ZFS pool was faulted 😮

Fixing the ZFS pool

  pool: tank0
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
	invalid.  Sufficient replicas exist for the pool to continue
	functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: scrub in progress since Tue Jun  1 19:30:49 2021
	7.78T scanned at 891M/s, 7.32T issued at 838M/s, 39.0T total
	0B repaired, 18.78% done, 0 days 11:00:34 to go
config:

	NAME                      STATE     READ WRITE CKSUM
	tank0                     DEGRADED     0     0     0
	  raidz2-0                ONLINE       0     0     0
	    sdi                   ONLINE       0     0     0
	    sdb                   ONLINE       0     0     0
	    sdh                   ONLINE       0     0     0
	    sdg                   ONLINE       0     0     0
	    sdd                   ONLINE       0     0     0
	    sdf                   ONLINE       0     0     0
	    sdc                   ONLINE       0     0     0
	    sde                   ONLINE       0     0     0
	  raidz2-1                DEGRADED     0     0     0
	    sdm                   ONLINE       0     0     0
	    6835734791109053608   FAULTED      0     0     0  was /dev/sdk1
	    16423333236828266746  FAULTED      0     0     0  was /dev/sdj1
	    sdn                   ONLINE       0     0     0
	    sdl                   ONLINE       0     0     0
	    sdo                   ONLINE       0     0     0
	    sdp                   ONLINE       0     0     0
	    sdq                   ONLINE       0     0     0

I suspect sdk and sdj had been swapped. This is why it’s not a good idea to use /dev/sdX to identify drives in the pool.

I initially created the pool with /dev/disk/by-id, but it got switched after I did an import. Time to switch it back.

I first added the following configuration to /etc/default/zfs:

ZPOOL_IMPORT_PATH="/dev/disk/by-id"

This tells ZFS to use /dev/disk/by-id when importing.

Then I exported and reimported the pool:

$ sudo zpool export tank0
$ sudo zpool import -d /dev/disk/by-id tank0

Voila!

  pool: tank0
 state: ONLINE
  scan: resilvered 3.08M in 0 days 00:00:02 with 0 errors on Tue Jun  1 22:10:56 2021
config:

	NAME                        STATE     READ WRITE CKSUM
	tank0                       ONLINE       0     0     0
	  raidz2-0                  ONLINE       0     0     0
	    wwn-0x5000c500afxxxxxx  ONLINE       0     0     0
	    wwn-0x5000c500b0xxxxxx  ONLINE       0     0     0
	    wwn-0x5000c500b3xxxxxx  ONLINE       0     0     0
	    wwn-0x50014ee265xxxxxx  ONLINE       0     0     0
	    wwn-0x50014ee2b9xxxxxx  ONLINE       0     0     0
	    wwn-0x50014ee211xxxxxx  ONLINE       0     0     0
	    wwn-0x50014ee2b9xxxxxx  ONLINE       0     0     0
	    wwn-0x50014ee265xxxxxx  ONLINE       0     0     0
	  raidz2-1                  ONLINE       0     0     0
	    wwn-0x500003999xxxxxxx  ONLINE       0     0     0
	    wwn-0x500003999xxxxxxx  ONLINE       0     0     0
	    wwn-0x500003999xxxxxxx  ONLINE       0     0     0
	    wwn-0x5000cca26xxxxxxx  ONLINE       0     0     0
	    wwn-0x5000039a5xxxxxxx  ONLINE       0     0     0
	    wwn-0x5000039a5xxxxxxx  ONLINE       0     0     0
	    wwn-0x5000039a5xxxxxxx  ONLINE       0     0     0
	    wwn-0x5000c500bxxxxxxx  ONLINE       0     0     0

errors: No known data errors

That’s some weird looking IDs, but I found this reply on the ServeTheHome forums:

[…] think of wwn like a mac address (it basically is the equivalent in storage land). I’ve never seen issues using wwn. […]

[…] I find WWN more reliable because some manufacturers don’t code serial numbers properly (e.g. those white labeled Seagate Exos X16 drives) and wwns are still unique and fully serviceable. […]

andrewbedia

Seems fine — better than /dev/sdX 🙂 I’ll be adding these wwn IDs to my disk/bay/serial sheet.

Improving power up

It’s easy to forget about optimizing for shutdown recovery, because it very rarely happens. But there are some simple things I can do to improve it.

My PDU has a configurable delay between starting up each of the 8 outputs. This is to prevent everything coming on at once, causing a current spike. The delay is 1 second by default, but can be set up to 240 seconds.

By increasing this delay, and reorganizing the power outputs; I can make sure that the most critical equipment starts first.

  • Network (router, switches)
  • File server
  • Primary hypervisor
  • Secondary hypervisor
  • Docker host

This may also solve the issue with the nut server not being able to communicate with the UPS.

I will also be adding the KVM shutdown script to the upsmon.conf configuration on the hypervisors:

# SHUTDOWNCMD "<command>"
#
# upsmon runs this command when the system needs to be brought down.

SHUTDOWNCMD "/sbin/shutdown -h +0"

I’m hoping to get this done before the next unexpected test 😛

Kids' reaction

Power outages are very rare where I live. So rare in fact that my kids, age 7, can’t remember experiencing one, and doesn’t understand what it means.

During the outage they wanted to watch TV. I explained that the TV doesn’t work because it needs electricity, and we don’t have that right now. So they wanted to play video games — again I explained why that wasn’t possible.

They got really upset and asked: “Does everything need electricity? Is it going to be like this forever?" 😛