All posts
Long-form notes on software, distributed systems, and the craft of building. Shipping one a week.
Forecasting Is Now a Library Call: Notes from Running TimesFM 2.5
Google's TimesFM 2.5 shrank to 200M parameters, topped GIFT-Eval zero-shot at release, and now sits behind BigQuery's AI.FORECAST — so I ran it myself on telemetry-shaped series. Zero-shot, it beat seasonal-naive by 15% in 0.6s per forecast on CPU. Then I added a 60% level shift one week before the forecast and the one-line baseline won by 1.5x — while the model's quantile band quietly widened 2.6x. These are my notes on what caught the attention, what to observe before trusting it, and the use cases where I would actually wire it in.
The Handoff Is the Unit of Design: Delegating to Agents Without Losing the System
Once agents write a meaningful share of the code, my output is no longer typed code — it is delegation decisions. These are my notes on the human discipline that makes that work: sizing every handoff to the review I can afford, the brief I hand over instead of big tasks, and the four habits that keep me connected to a system I am no longer typing into — from Bainbridge's 1983 ironies to a METR result that has since reversed its own sign.
Your Queue Won't Drain: The Arithmetic of Backlog Recovery
A healthy-looking consumer fleet can sit on a 3-million-message backlog that never shrinks. These are my notes on the arithmetic behind it: surplus = capacity − arrival, drain = backlog ÷ surplus, and why a fleet sized for steady-state has zero recovery capacity. I built a small TypeScript simulator to watch retry amplification park a correctly sized fleet in a metastable failure, and worked out when shedding beats draining.
Cache Stampedes Are a Capacity Problem, Not a Locking Problem
A distributed lock on every cache miss is the fix most posts reach for; in my own testing it is the one that turns a 400 ms blip into a queue collapse. These are my notes on treating stampedes as an upstream capacity budget — why 10% TTL jitter is a cargo-cult number, what XFetch actually buys, and the rubric I use to decide when a lock earns its keep.
Backpressure Is a Contract Every Caller Must Honor, Not a Setting
A bounded queue in one service does nothing if the caller above it keeps pushing. These are my notes on backpressure as an end-to-end contract: what the callee owes, what every caller owes back, and why the Azure OpenAI retry-amplification incident is what a broken clause looks like at hyperscale.
When a Zone Fails, Don't Let Your Cluster Rebalance Itself to Death
An availability zone drops and the cluster survives the outage — then hurts itself trying to heal. These are my notes from reading Uber's zone-resilient OpenSearch write-up against the allocation-awareness docs: why the re-replication reflex is the real danger, and how declaring every failure domain up front makes a cluster degrade to a stable yellow instead of a rebalance storm.
The Bug That Wouldn't Compile: Porting a Reordering Mistake to Hydro
I planted an arrival-order bug in a hand-rolled Rust fan-in aggregation, reproduced a 51% undercount with a seeded shuffle, then ported the same logic to Hydro 0.17-alpha. The unordered merge refused to compile my non-commutative fold — these are my notes on what that guarantee is, what manual_proof! only attests, and when the seam costs are worth paying.
Architecture as Code with Structurizr DSL: A C4 Workspace Both a Reviewer and an Agent Can Read
I committed a Structurizr DSL workspace next to a Kotlin/Spring billing service and wired ArchUnit into Gradle so CI fails the moment the code drifts from the C4 design. Here is the workspace.dsl, the test that backs it, and what changed when a coding agent could read both files instead of guessing from packages.
Three Phases of Online Data Movement, Read as Three Invariants
Every online migration diagram I have seen looks the same — three arrows between two boxes — and skips the part that matters: what each phase is actually for. These are my notes after re-reading Stripe's DocDB writeups against the gh-ost and Vitess cutover docs, distilled into three invariants I can graph and gate.
Engineering Before Inference: The Question Zero Token Architecture Is Actually Asking
Lately I keep hearing a sentence that would have been absurd three years ago: "I can't do it now — I ran out of tokens." These are my notes from digging into Zero Token Architecture — the idea Kelsey Hightower took from a throwaway post to a PlatformCon keynote — and the principle I want to build on it: every AI call is an architectural decision, and the costs that matter most were never the tokens.