~blogwhy-i-write-about-what-breaks

Why I write about what breaks, not what shipped

2024-06-11 · 4 min read · meta

Most engineering writing shows you the finished thing. A clean architecture diagram, a config that works, a benchmark that went the right way. It is useful right up until you deploy it, and then something at 2am disagrees with all of it.

I have spent six years building backend systems, and for most of those years I have also been the person who deploys them, watches them, and gets called when they stop. That combination changes what you find worth writing down. The design is rarely the interesting part. The interesting part is the gap between what the design assumed and what the machine did.

So this is a blog about that gap.

The three questions

Before I commit to a design now, I ask the same three things, in this order:

Question What it catches
How does this fail? The failure mode nobody owns, because it lives between two teams
What does it cost to run? Money, and the attention tax of a thing that needs watching
Can someone else keep it alive? Cleverness that only I can maintain

None of these are about whether the system works on the happy path. Everything works on the happy path. That is what makes the happy path such a bad place to stop thinking.

Most of what I publish here comes out of question one, because that is where I spend my nights.

What you will find here

Roughly four kinds of post, and they mix:

Debugging writeups, where a real failure took me hours and the cause was three layers below where the error appeared. These are the ones I would have wanted to find on Google at the time, and mostly could not.

Design writeups, where I explain a system I built and the trade-off I took knowingly. Serverless, ingestion pipelines, permission models, monitoring.

Homelab and infrastructure, because a homelab is the only production environment where I am allowed to break everything and the only user affected is me. Multiple ISPs, Proxmox, routers, tunnels, local models on hardware I already owned.

Cost and constraint, which is a theme more than a category. Most of what I run costs nothing per month, and that limit has taught me more than a bigger instance ever did.

Three rules I hold this to

Everything I write about runs somewhere I can show you. The homelab is a set of machines in my apartment in Dhaka that I have to fix myself when I break them, usually on a weeknight. The production systems are ones I am on call for. If I have not run a thing, I do not have an opinion about it worth reading.

I say which posts are shipped and which are design. Some of what I want to write about is architecture I am partway through building. That is still worth publishing, but it is a different claim than “this ran in production for a year”, and I will mark which one you are reading.

No invented numbers. If I have a real measurement I will give it, with the hardware it came from. If I do not have one, I will say what I would measure instead of producing a plausible figure. Made up benchmarks are the fastest way to be wrong in public and the easiest thing in the world to write.

Who is writing

I am a lead engineer in Dhaka, Bangladesh. Node and TypeScript most days, NestJS, Postgres, Redis, RabbitMQ, Docker and Kubernetes underneath, Cloudflare Workers at the edge, Proxmox at home. Lately more of my time goes to AI systems that touch real business tools, which turns out to be an access control problem long before it is a model problem.

I have nothing published in the open yet. The thing I keep meaning to build is a backup tool I can point at Postgres, MySQL and Mongo and then forget about, because I once found out during an incident what an untested backup is worth. Until that exists, everything here comes out of systems I run privately or operate at work.

Next post is the map: every machine and service the rest of these posts refer back to, what each one does, and what it costs me.