Simple homelab status page — bash script, NetBox, and AWS

I set up a super simple status page for the homelab — using a small bash script, NetBox, AWS S3 and CloudFront.

But ended up using a hosted service instead — HetrixTools.

Introduction

I’ve thought about setting up a status page for the homelab any times; I’ve tried Cachet, Statusfy, and Staytus. It’s fun to set up, but not very useful — and in the end I don’t bother updating it…

I wanted to try and set up something super simple — that I don’t have to manually update, and fits into tools I’m already using. This is what I came up with 👇

Bash script

I stumbled onto tinystatus:

tinystatus generate an html status page via shell script.

It does http, ping, or port checks and generates a single html file with the results. Incidents can simply be added to a text file.

I thought the design was a bit plain, and I wanted a few more features; so I forked it and changed a few things:

• Layout — cards, background color, and shadows
• Incidents have type and are HTML formatted
• Small JavaScript to set status to unknown if page was generated more than 5 mins ago

The script runs every 3 minutes, and the html file is uploaded to AWS S3 and served with CloudFront:

#!/bin/bash

export PIPENV_PIPFILE=/home/hebron/dev/netbox-api/Pipfile
pipenv run python3 ../netbox-api/rack-journal.py > incidents.txt

./tinystatus > index.html

if [ -f "index.html" ]; then
    aws s3 cp index.html s3://homelab-status/
else
    echo "index.html not found"
fi

Crontab definition:

*/3 * * * * cd /home/hebron/dev/tinystatus && ./check.sh

All checks are performed locally, which means that some errors won’t be detected — like a port forward issue with the router. If the homelab looses internet connectivity, it won’t be able to upload a new version of the status page. And a small JavaScript will mark all services as unreachable.

You can run the script on a server outside the local network, like a VPS, to do the checks externally, but I’m not doing that.

Incidents from NetBox journal

Instead of manually maintaining the incidents.txt file — I’m pulling journal entries from the NetBox API. I try to maintain a simple journal when I change or test something on servers or the homelab.

And using the NetBox journal entries as “incidents” for the status page means no additional work 🙂

This script is executed from the check.sh bash script above:

import pynetbox
import datetime
import markdown

nb = pynetbox.api(
    'http://omicron.lan.uctrl.net:8001',
    token='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
)

entries = nb.extras.journal_entries.filter(assigned_object_type="dcim.rack", assigned_object_id=1)

for entry in entries:
    date = datetime.datetime.fromisoformat(entry.created.replace("Z", "+00:00")).strftime("%Y-%m-%d %H:%M")
    html = markdown.markdown(entry.comments).replace("\n", "")

    print(f'{entry.kind.value}|<p class="timestamp">{date}</p>{html}')

The journal entries from NetBox is markdown formatted, this script turns them into a single line of HTML. Here is an example of a generated incidents.txt file:

danger|<p class="timestamp">2022-03-11 14:20</p><p>Danger test</p><p><strong>Resolved:</strong> I fixed this thing.</p>
warning|<p class="timestamp">2022-03-11 14:20</p><p>Warning test</p>
success|<p class="timestamp">2022-03-11 14:20</p><p>Success test</p>
info|<p class="timestamp">2022-03-11 14:20</p><h2>Info test</h2><p>I'm doing maintenance.</p>

Concluding

I was pretty happy with the results. Disruptions were detected and the status page updated automatically. And since the page was hosted with AWS S3 and CloudFront — it remained available even when the homelab was offline.

The paragraph above is written in past tense — because in the end, I didn’t end up using this status page solution…

While doing VyOS router testing I needed a more feature rich and reliable way of checking service status from outside my network. I signed up with HetrixTools; they have a really generous free plan with uptime and server monitors.

And status pages; white labeled on your own domain 👍

There is no issues functionality, but you can make announcements on the top of the status page. I’m using the HetrixTools API to put services in maintenance mode (suspending notifications) from some backup scripts.

So this post took a bit of a weird turn; I set out to make a simple self-hosted status page — and ended up finding a nice, free, hosted service 🙂