Distributed system
Cloud Orchestrator
A distributed orchestrator for service instances — nodes find each other, work moves between them, and a failed machine costs no downtime.
An orchestrator that starts, supervises and relocates service instances across several machines — comparable in scope to established server clouds, but deliberately without Kubernetes underneath. That constraint is the whole point: build everything a cluster needs once, by hand, and find out why the established systems look the way they do.
The idea goes back to 2022. What runs today is the third attempt at it, and the first one where the module boundaries held up long enough to build on.
Structure
The system is cut into roughly 30 modules that build strictly on top of one another. No module may know a module from a higher layer — a rule the build enforces rather than the documentation.
foundation → kernel → transport · persistence · cache → scheduling → …- foundation — result types, error model, configuration, structured logging
- kernel — lifecycle, service registry, event bus
- transport — a custom binary protocol on Netty: framed, versioned, backwards compatible
- persistence & cache — interchangeable stores behind a narrow abstraction
The mesh
Node-to-node communication is the core of it. Nodes discover each other, negotiate a protocol version and hold the connection open with heartbeats. When a node dies, the others notice within seconds and take over its work — verified by tests that start real nodes in separate processes and then kill them on purpose.
Choosing a custom binary protocol over HTTP turned out right. Per-message overhead sits in the low tens of bytes, and negotiating the version during the handshake is what makes rolling updates across a live cluster possible at all.
What I took away
The hardest part was not the networking. It was resisting the shortcut of letting one module reach into another “just this once” while the deadline of my own impatience loomed. Two earlier attempts died exactly there: every shortcut is cheap on the day it is taken and expensive every day after.
Status
The lower layers are done and tested, and the mesh holds. Next comes the scheduling layer: who decides which instance runs on which node — and what happens when two nodes both believe they made that decision.