DEV Community

The Homelab Postmortem profile picture

The Homelab Postmortem

I write up homelab failures where the tool reported success — ufw says the port is closed while it answers, tar exits 0 on an empty archive. Everything is reproduced on real hardware before it goes up

Podman's Docker-compatible API resets a container's restart policy to 'no' on any update that omits it

Podman's Docker-compatible API resets a container's restart policy to 'no' on any update that omits it

1
Comments
7 min read
One failed WPA handshake and a headless Raspberry Pi on Trixie stays off WiFi until someone types a command

One failed WPA handshake and a headless Raspberry Pi on Trixie stays off WiFi until someone types a command

1
Comments
8 min read
llama.cpp reads past your pos array on embedding batches for M-RoPE models, and the logits change with the heap

llama.cpp reads past your pos array on embedding batches for M-RoPE models, and the logits change with the heap

1
Comments 1
10 min read
Ollama's gemma4 renderer silently drops tool parameters named type or description, and the model invents a value

Ollama's gemma4 renderer silently drops tool parameters named type or description, and the model invents a value

2
Comments 2
7 min read
llama.cpp reads past its own pos buffer for M-RoPE embeddings, and nothing reports it

llama.cpp reads past its own pos buffer for M-RoPE embeddings, and nothing reports it

Comments
6 min read
Ollama's Responses API accepts previous_response_id, returns 200, and forgets the whole conversation

Ollama's Responses API accepts previous_response_id, returns 200, and forgets the whole conversation

4
Comments 4
6 min read
docker logs stops at a NUL byte, exits 0, and --since says nothing happened

docker logs stops at a NUL byte, exits 0, and --since says nothing happened

Comments 1
7 min read
llama-server ignores the response_format its own README shows, and returns 200

llama-server ignores the response_format its own README shows, and returns 200

Comments 1
6 min read
cloud-init never reads the instance-id Raspberry Pi OS sets, so your second user-data is skipped

cloud-init never reads the instance-id Raspberry Pi OS sets, so your second user-data is skipped

1
Comments
6 min read
vLLM accepts a LoRA it will never apply, and answers with the base model instead

vLLM accepts a LoRA it will never apply, and answers with the base model instead

Comments
6 min read
An Ollama tag that pulls and runs and has no working code in it. The same model at the same quantisation, elsewhere, is fine.

An Ollama tag that pulls and runs and has no working code in it. The same model at the same quantisation, elsewhere, is fine.

Comments
8 min read
llama-cli reports a commit hash that is not in llama.cpp. It belongs to whatever repository you unpacked the source inside.

llama-cli reports a commit hash that is not in llama.cpp. It belongs to whatever repository you unpacked the source inside.

Comments
7 min read
llama-cli prints the error and exits 0. The same binary exits 1 when the model is missing.

llama-cli prints the error and exits 0. The same binary exits 1 when the model is missing.

Comments
4 min read
The same ssh_import_id works in one cloud-init file and does nothing in the next. The difference is one letter.

The same ssh_import_id works in one cloud-init file and does nothing in the next. The difference is one letter.

Comments
8 min read
iptables says your kernel needs upgrading. Upgrading the kernel is what broke it.

iptables says your kernel needs upgrading. Upgrading the kernel is what broke it.

Comments
6 min read
tar backed up your Pi's WiFi config and exited 0. The archive holds one empty directory.

tar backed up your Pi's WiFi config and exited 0. The archive holds one empty directory.

Comments
9 min read
Podman publishes the same port and your firewall still holds. Even as root.

Podman publishes the same port and your firewall still holds. Even as root.

Comments
6 min read
UFW says the port is closed. Docker published it to your whole network anyway.

UFW says the port is closed. Docker published it to your whole network anyway.

Comments
5 min read
A new NetworkManager property broke a decade of headless WiFi scripts

A new NetworkManager property broke a decade of headless WiFi scripts

Comments
5 min read
sysctl -p says it worked. The next reboot disagrees.

sysctl -p says it worked. The next reboot disagrees.

Comments
6 min read
Your Pi accepts every memory limit you set and enforces none of them

Your Pi accepts every memory limit you set and enforces none of them

Comments
6 min read
Your Pi's 2 GB swap file isn't swap, and no swap command will tell you that

Your Pi's 2 GB swap file isn't swap, and no swap command will tell you that

Comments
6 min read
Stripe retries a failed webhook for three days, and I found out by getting the same email three times

Stripe retries a failed webhook for three days, and I found out by getting the same email three times

Comments
4 min read
Raspberry Pi OS Trixie throws away your logs on reboot, and /var/log/journal exists anyway to reassure you it doesn't

Raspberry Pi OS Trixie throws away your logs on reboot, and /var/log/journal exists anyway to reassure you it doesn't

Comments
5 min read
Undervoltage doesn’t look like a power problem. It looks like bad Wi-Fi.

Undervoltage doesn’t look like a power problem. It looks like bad Wi-Fi.

Comments
4 min read
Your Discord bot is ignoring you, and the status output is lying about why

Your Discord bot is ignoring you, and the status output is lying about why

Comments
4 min read
The rpi-clone PARTUUID trap: why your Pi 4B silently refuses to boot from a cloned SSD

The rpi-clone PARTUUID trap: why your Pi 4B silently refuses to boot from a cloned SSD

Comments
5 min read
loading...