Building ML systems
for the boring middle layer
between training and the bill.
I’m a machine-learning engineer who spends most of my time on the parts no slide deck loves: inference latency, cluster economics, and the slow craft of making a software system you can reason about. This is where I work in public — in essays, projects, and a quiet reading log.
Recent essays
-
Field notes
Cost-aware autoscaling for inference, or: why p99 lies
Notes from a quarter spent watching a GPU fleet melt money. A small model of demand, a smaller model of cost, and the gap between them.
-
Architecture
Your microservice is a distributed monolith
On call boundaries vs. data boundaries. The graph that matters is the one you can’t see in the org chart.
-
Notes
Quantization, with a spreadsheet
INT8 vs FP8 vs the rest, with a working sheet you can copy. The point is to feel the bits, not memorize a table.
-
Case study
When pgvector is enough
A two-million-row case study, and the thresholds I use before reaching for a dedicated vector store.
From the workshop
-
Subroute
shippingA tiny request router for sharding LLM traffic by token-budget.
~600 LOC of Go. In production behind a 4-node inference cluster.
-
Mistlake
shippingCold-storage layout for inference logs that survives a 90% read-skew.
Cuts the S3 bill by ~38% on our shape. Rust + Parquet, single binary.
-
pgvec-bench
pausedA no-frills benchmark suite for pgvector vs Qdrant.
Mixed read/write, skewed dimensions, realistic recall curves.
From the reading log
-
001.A Philosophy of Software Design — John OusterhoutRereading. The second half hits harder this time. Complexity is the only enemy.rereading
-
002.Designing Data-Intensive Applications — Martin KleppmannStill the reference. Chapters 5 and 11 do a lot of work in my head.reference
-
003.Mixtral of Experts — Mistral AI, 2024Cleanest description of routing-by-expert I’ve read in a paper.great