What runs in my homelab, and why it's Kubernetes

I run a four-node Kubernetes cluster at home. It didn't need to be Kubernetes. Proxmox, or one box running Docker, would have been simpler for a homelab, and for a long time one box running Docker is exactly what I had.

How it started

The first version was a single node running Docker containers, deployed and managed with Ansible, with Semaphore as the UI on top. It worked.

The problem was that I already knew Ansible. I knew Docker and I knew virtualisation. The thing I kept reading about and didn't understand was Kubernetes, and I had a feeling it might be useful in my career at some point. I wanted hands-on experience so I could answer questions about it and understand what I was reading.

So in April I stripped Docker out, added two Raspberry Pis, and built a K3s cluster. The whole reason for going with K3s was that I wanted to learn it.

The hardware

All of it was sitting around the house.

The control plane is a Mac Mini from 2014. I ripped the operating system off it and put Ubuntu server on. The two workers are Raspberry Pi 5s running Ubuntu Server, booting from SSDs so the SD cards don't degrade.

From there it's kind of grown. I was using iCloud Photos and moved to Immich, hosted at home, which meant I needed somewhere to put a lot of files. So I added a NAS, and that became the fourth node. Then I did the same with documents: I was using iCloud Drive, and now there's ownCloud in the cluster and my files live on a local-first file server.

The rack is a GeeekPi DeskPi RackMate T1. When I was looking at it, a photo of how someone had theirs set up would have been useful, so here's mine. The two Pis are in the top panel, the Mac Mini is behind the mesh, and the patch panel is at the bottom. The NAS sits next to it.

How it's put together

Everything is declared in one git repo. Argo CD watches the repo and applies it, so if I change something by hand on the cluster it gets reverted.

Public apps come in through a Cloudflare Tunnel with a login in front of them, and I get in for admin over Tailscale. There's one shared Postgres cluster, and no app runs its own.

The diagram leaves a few things out. Secrets are sealed and committed alongside everything else. Every node backs up nightly with restic to Backblaze B2. Prometheus, Grafana and Loki watch the lot and send alerts to Discord.

The apps folder has 57 entries in it. Most of those are plumbing.

What we actually use

Every week, probably three things: Immich, ownCloud, and the sourdough planner I wrote.

Less often there's the finance planner, which comes out once a month when we do the household budget. PriceBuddy runs in the background watching prices on things we want to buy. TREK is for planning trips.

Then there's OpenViking, which is the memory for my AI agents. I don't actively use it. It gets used passively whenever I speak to an agent, which is pretty much constantly.

Experiments

The nice thing about Kubernetes is that I can bring something up from a Helm chart very quickly, test it, and if I don't like it I remove the chart and it's gone.

A lot hasn't survived. The most recent was Hivekeep, a team of named coding agents running in the cluster and opening pull requests. It was a useful test and it's still written up in the repo, but the API costs made it not feasible, so out it came.

It has also fallen over. Not long after I built it, the Mac Mini came up on a different IP address and took the whole cluster down with it.

Would I do it again

As a tech, there have been no downsides. Everything I've wanted to learn, I could learn. Anything I want to test, I can try, and I can do it quickly. If the disposable income is there for the equipment, and you've got the time and the drive to learn, there's no downside to running one.

The thing I'd change is the hardware. Repurposing what I already had was great for getting off the ground. If I were starting again I'd look at old office micropcs, which are better suited to the job from the off.

Right now I'm deciding whether to replace a node or add one. I want GPU power, more CPU, and RAM I can expand later. I haven't decided what infrastructure comes next.