Migrating Traefik to Nomad for Zero-Downtime Homelab Maintenance

Some Background

My homelab is where I host a variety of services. This Blog, my Gitea instance, and a media server are a few of the workloads my homelab runs. It is made up of a network gateway and three very different devices that I steer workloads towards depending on their characteristics.

When a request comes into my network, it's first proxied through Cloudflare to help prevent DDoS attacks and avoid drawing too much attention to my residential IP address. I'm fortunate to have an amazing ISP where my 'dynamic' IP address seems pretty static, which means I rarely need to update my domain / Cloudflare routing. When those requests come in, they are handled by my UniFi Dream Machine SE. It performs some intrusion prevention and then forwards traffic to what I used to consider my primary server, Pandora.

Pandora received a static IP address on my homelab subnet and had Traefik Proxy listening to web traffic on it. Depending on the service the request was for, it would be routed to either a different port on Pandora or to my other server, Miranda. Pandora is a custom-built server with ample horsepower and redundant storage. I use the ZFS filesystem to create a pool of 8 hard disk drives, so that any two can fail without data loss. It also has a mirrored SSD pool and two small Intel Optane drives for local storage. Those small drives are part of the problem we'll get into later... Miranda is just an old micro PC with a single drive I had lying around, so it's a good fit for stateless workloads or things I don't care about losing data for, such as logs or telemetry.

The naming theme is planets from nerdy fandoms in case you were curious.

The Software Stack

Of course, it wouldn't be a proper homelab without a Raspberry Pi. This is the third device in my cluster, and it acts as the 'brains' of the system. While the purpose of my homelab is to practice digital sovereignty, it's also a good way to build my own cloud skills. I explored doing a traditional VM approach, but it didn't feel 'cloud native' enough. Instead, I chose to use Hashicorp's Nomad and Consul products. If you are not familiar, they are similar to Kubernetes. Still, Hashicorp's approach follows the Unix philosophy of doing one thing and doing it well, whereas K8S is much more expansive in its capabilities and control plane.

My Raspberry Pi is named Radiata (a reference to the supercomputer in Lycoris Recoil) and serves as the server node for my Nomad and Consul cluster. Pandora and Miranda are agents that then execute the workloads they are scheduled to run by Radiata.

This is NOT a high-availability setup. You want at least three server nodes in production to provide redundancy for both Nomad and Consul.

This has worked really well for me because I can define a job, and the cluster will figure out where to run it and ensure it's operational. For my Blog, which is stateless (or rather, has its state embedded), it could run on either Pandora or Radiata without issue. The basic idea is that Nomad schedules the job and registers the service definition with Consul. Traefik Proxy then uses the Consul Catalog to configure itself to direct traffic to the service, including the domain it should be listening on. This means when a valid request I want to listen for is received from Cloudflare through my gateway and forwarded to Traefik, it knows how to direct the request to the right service, regardless of whether that job is running on Pandora or Miranda.

The Trigger

When I was using Alpine Linux as my base OS on bare metal, Pandora was properly scaled and could operate on the small Intel Optane drives, but I ran into a problem. Alpine Linux uses musl instead of glibc (a major factor in its small size), and Pandora has an Nvidia Graphics Card I wanted to leverage for transcoding in the media server. Unfortunately, Nvidia doesn't provide great support for musl, and glibc is required for full functionality. Easy enough fix, I'll just switch to Ubuntu (I've already gone through the learning process with Alpine anyway, so having something batteries included would be a nice change of pace).

I no longer knew what every service running on the OS was, and didn't know everything that was installed, but I was able to leverage the graphics card. This was a fine solution, and it worked well for several months until I first noticed that I ran out of storage. I was able to free up some space by deleting some local logs and offloading several Docker images that were no longer in use, but it was only a temporary fix.

I logged into Pandora the other day, and when I checked on the Hard Drive Pools' health, my heart stopped for a moment longer than it should have. It wasn't running with one drive down; it was running with two faulted drives. One more and the data would be unrecoverable.

  NAME        STATE     READ WRITE CKSUM
  hdd         DEGRADED     0     0     0
    raidz2-0  DEGRADED     0     0     0
      sda     ONLINE       0     0     0
      sdb     ONLINE       0     0     0
      sdc     ONLINE       0     0     0
      sdd     ONLINE       0     0     0
      sde     ONLINE       0     0     0
      sdf     ONLINE       0     0     0
      sdg     FAULTED      0     0     0
      sdh     FAULTED      0     0     0

I tried to check the SMART data on the drives (information that can indicate whether a drive is starting to fail), but I needed to install a package I didn't have room for. It was able to install some of the packages, though, and it started looking like the drives were ok, just not registered in the pool correctly. This was when I decided to upgrade the OS drive to a larger size so I would stop having these problems. It may not be logical, given that metadata is stored within the drive pool itself, but my thought process was that the Hard Drive zpool had a metadata error because the OS drive was out of space. We'll start with replacing the OS drive, and if the HDDs are still faulted on the fresh install, I'd look to replacing them (but they did seem healthy)

The Problem

To swap out the OS drive, I would need to take Pandora offline. This was fine for this Blog because it can run on Miranda. What wasn't fine was that it meant my Traefik Reverse Proxy would also be down, because it runs on the single Pandora host with a fixed IP for the port-forwarding rule. That's a problem because even if my Blog is still running, it couldn't actually receive any traffic to respond to.

At the end of the day, a little downtime wouldn't be a big deal for a blog that gets minimal traffic, but it felt against the spirit of the homelab when a zero-downtime solution was achievable. I am using an orchestrator that can handle massive production workloads after all.

Migrating to Docker

Instead of running Traefik as a local install, we can run it in a Docker container. By running it within Docker, it becomes painless to pass it to Nomad to run it on whatever host is available. My instance of Traefik required configuration files to be saved on the local host. By moving these into Consul's KV store, we can fetch the config files from whichever host ends up running the job.

The Code

We can either use Consul's Web UI to define a blob for a given key, or we can 'upload' a file to a specific key using the consul cli.

consul kv put <key> <value>

Consul KV is a flat key-value namespace that supports 'nesting' keys together with a prefix. So in my case I had the following two commands to run and both keys are nested in the Web UI under the traefik key.

consul kv put traefik/traefik.yml @host_config/traefik.yml
consul kv put traefik/traefik-dynamic.yml @host_config/traefik-dynamic.yml

In Nomad, we then fetch these values as a template block. We read the data in the Consul KV store and save it to a file in the container. An added bonus is that we can instruct Nomad to restart the job when the value of the key is updated.

  template {
    data        = "{{ key \"traefik/traefik.yml\" }}"
    destination = "local/traefik.yml"
    change_mode = "restart"
  }

Setting up the Network

• The initial reason I had Traefik just running on the primary host
• Setting up a 'virtual IP' for port forwarding
• Pointing Unifi to a 'virtual IP (and excluding it from the DHCP range)

By moving the Traefik proxy to an orchestrated job, we ensure it remains online as long as there is an eligible host to run it on. This comes with a new problem, though: because Traefik was only running on Pandora, my port-forwarding configuration could be static. Now that the Traefik proxy can move around, how do I route the port forwarding to a dynamic host?

Introducing Virtual Router Redundancy Protocol (VRRP). By deploying an osizia/keepalived alongside the Traefik Proxy job, the host computer that receives the job effectively assumes a fixed IP address. When a new instance comes online, it checks whether it is the primary or a backup. If it's the primary, it will claim the traffic bound to the IP address, and if the primary instance goes offline, a backup instance will claim ownership of the IP address.

This lets me define a static IP address of 192.168.2.200 on my subnet. From there, I excluded it from my DHCP lease pool and configured my router to forward traffic to this IP instead of Pandora's address. If Pandora goes offline, Radiata schedules Traefik on Miranda, and the 'sidecar task' keepalived automatically takes ownership of the static IP, meaning that, in the worst case, only a couple of packets get dropped during failover.

Success!

Once Traefik was running as a container with a 'static' IP address, I used curl to check whether it could route correctly. I targeted the 'virtual' IP address on the port I had Traefik configured as an entrypoint. By using the -H flag, I was able to specify a domain for the specific service I wanted to access. This was akin to me doing a web request to Cloudflare, but within my home network. This, however, fails because Cloudflare is my certificate provider and because Traefik is configured to use only HTTPS; any curl requests over HTTPS fail to validate the certificate. Including the -k flag tells curl to ignore SSL certificate verification.

curl -k -H "Host: cbraaten.dev" https://192.168.2.200:8443

This gave me my Blog's homepage! Before I was ready to take Pandora offline, though, I needed to update my port forwarding from Pandora's static IP to the floating 'virtual IP'. Because I had already tested with curl, this was just to make sure that my Unifi router would be ok with the IP. Traffic continued to flow as expected and when I stopped the Traefik service on Pandora, it was as if nothing changed! Upon shutting down the native Traefik proxy on Pandora, I had Nomad make two allocations of the Traefik proxy and then drained Miranda of its jobs to force a failover. Sure enough, the swap was seamless. It was now safe to shut down Pandora for maintenance, and my homelab would continue operating.

After installing the new, larger drive, I was successfully able to rebuild Pandora and import the zpool from the prior OS version. The HDD pool reported all 8 drives as healthy and resilvered ~250GB of data 😱. Turns out I had copied a lot of stuff to the server with the pool in a degraded state. 😅

Once Pandora came online and had the proper dependencies (perhaps another blog post on that soon), it joined the Consul and Nomad clusters and immediately started serving traffic once more.