All posts
Long-form notes on software, distributed systems, and the craft of building. Shipping one a week.
Cache Stampedes Are a Capacity Problem, Not a Locking Problem
A distributed lock on every cache miss is the fix most posts reach for; in my own testing it is the one that turns a 400 ms blip into a queue collapse. These are my notes on treating stampedes as an upstream capacity budget — why 10% TTL jitter is a cargo-cult number, what XFetch actually buys, and the rubric I use to decide when a lock earns its keep.
When a Zone Fails, Don't Let Your Cluster Rebalance Itself to Death
An availability zone drops and the cluster survives the outage — then hurts itself trying to heal. These are my notes from reading Uber's zone-resilient OpenSearch write-up against the allocation-awareness docs: why the re-replication reflex is the real danger, and how declaring every failure domain up front makes a cluster degrade to a stable yellow instead of a rebalance storm.
Read-Your-Writes Is a Session Contract, Not a Database Setting
I added a read replica and within a day users reported edits that "didn't save" — they had saved, but the reads raced the replication stream and lost. These are my notes on why read-your-writes is a session contract rather than a replica setting, reproduced with a pinned 50 ms lag in Go, and the three ways to carry it — sticky routing, a GTID-style version token, and a bounded-staleness wait — compared on cost.
Hybrid Logical Clocks: Making Last-Write-Wins Mean the Later Write
Wall-clock last-write-wins keeps the write from the faster clock, not the later event — and silently drops causally newer data under skew. These are my notes on rebuilding a Hybrid Logical Clock in Go: a 64-bit, monotonic, causal timestamp, why its counter stays bounded, and what it costs in CockroachDB-style uncertainty restarts.
Convergence Is a Property of Your Merge Function, Not the Network
I once watched an afternoon of offline edits vanish under a last-writer-wins sync, and the fix was not better networking — it was a better merge function. These are my notes on why CRDT replicas converge: a merge that is commutative, associative, and idempotent. I rebuild a minimal add-wins OR-Set in TypeScript, run it, and weigh what the guarantee costs in tombstones and memory.
Two-Phase Commit on the JVM: The Blocking Problem Nobody Puts in the Diagram
I crashed a Two-Phase Commit coordinator on purpose in a small Kotlin simulation to measure how long participants stay locked when the coordinator vanishes between phases. The result is the part of 2PC the diagrams never show — and the reason I would model most cross-service writes as a saga instead.
Drop the Right Requests First: Priority-Aware Load Shedding Under Overload
Static RPS caps shed the wrong traffic. Concurrency is what saturates a service, not request rate. From my notes after reading the InfoQ piece on overload protection, Uber's January writeup on Cinnamon, and Netflix's QCon SF talk on service-level prioritized load shedding, here is why latency is the right control signal — and how a small priority taxonomy plus an adaptive concurrency limit keep the cheapest traffic shedding first.
Actor-per-Entity vs Postgres Optimistic Locking: A Seat-Reservation Bake-off
I ran the same hot-key seat reservation workload two ways: Postgres with a version column and retries, and a single actor per seat. The actor design did not scale better — it moved the hard problem from concurrency control to routing and rebalance correctness, and that trade was the easier one to reason about under hot keys.
Auditing a Scala Service Against Chad Fowler's Four Regenerative Constraints
I walked a Scala order-processing service from my notes through Chad Fowler's four regenerative constraints. Two passed for free, two would force a real redesign. Here is what I learned about where "loosely coupled module" ends and "regenerative component" begins, and which parts of the redesign I would actually pay for.
AckWait Is a Contract: How a 30-Second Default Took Down My JetStream Consumer
I lost an evening to a NATS JetStream pull consumer that doubled its work in production. The cause was three lines of ConsumerConfig I never wrote. These are my notes on what AckWait actually counts, why MaxDeliver = -1 is the silent footgun, and the 70-line Go contract I now ship on every JetStream consumer.