Notes on Distributed Systems

Distributed systems are hard because they force you to reason about failure. Here are a few principles I keep coming back to.

Assume failure

Every network call can fail, hang, or return a partial result. Design for retries, idempotency, and timeouts from the start.

Prefer simple invariants

The fewer invariants you have, the fewer ways your system can break. A single source of truth is worth a lot of complexity elsewhere.

Observe everything

You can't debug what you can't see. Metrics, logs, and traces are not optional — they're the interface to your running system.

Test the failure paths

Happy-path tests give false confidence. Kill a node, partition the network, and see what actually happens.