Things being made, on purpose.

Subroute sits between our gateway and a small cluster of model servers. The clever part isn’t how it routes — it’s how it counts. Every request gets a cheap estimate of its token budget (input length + a sampled completion length), and we route to whichever node has the most remaining headroom in its KV cache.

Result: queue depth and tail latency stopped being correlated, which freed us to tune for throughput without watching p99 blow up. There’s a writeup forthcoming on the cost side; the short version is that we shrank the fleet by 20% with no change in user-visible latency.

Our inference logs are roughly 90% read-skewed: a tiny minority of trace IDs get read constantly (debugging, replay, eval), and the rest sit cold for months. Mistlake exploits that with a two-tier Parquet layout and a small hot index in front. The design isn’t novel; the discipline of refusing to over-engineer it kind of is.

The interesting failure was during the migration: a one-line bug in the tier-promotion thread caused the hot tier to grow at 5× its intended rate for about 36 hours before anyone noticed. The lesson I keep going back to is that cheap monitoring of cheap things is a real cost saver, and the place I keep failing to apply it is during the migration itself.

I got interested in pgvector during the “when is pgvector enough” essay and then realized I didn’t trust any of the benchmarks I could find. So I wrote one. The current state is “runs, but the report writer is a mess”; I’ll come back to it after Subroute settles.

If you’re someone who runs a serious vector workload and you have opinions about what the workload generator is missing, please write.

I wrote tinyrl to convince myself I actually understood the moving parts of RLHF. It does PPO over a small reward model on a toy summarization task. It has every comment I wished a reference implementation had had when I was trying to learn this. It is, on purpose, slow.

People sometimes write to ask if I’ll productionize it. I will not. The point of tinyrl is that it fits in your head.