https://osmangoninahid.github.io/tags/distributed-systems/Recent content in Distributed-Systems on Osman Goni NahidHugoen-USTue, 29 Sep 2026 00:00:00 +0400https://osmangoninahid.github.io/stuck-undeploying-two-ledgers/Tue, 29 Sep 2026 00:00:00 +0400https://osmangoninahid.github.io/stuck-undeploying-two-ledgers/<p>&ldquo;Stuck in Undeploying.&rdquo; &ldquo;Stuck in Pending.&rdquo; &ldquo;Stuck in Terminating.&rdquo; Over a year, that word showed up in more than a hundred tickets on a platform that deploys models onto GPU clusters. The states were different; the shape of the problem was the same every time.</p> <figure><img src="https://osmangoninahid.github.io/images/stuck-two-ledgers.gif" alt="Two ledgers side by side: the platform database says Undeploying for hours while Kubernetes shows the pod gone, a PVC waiting on a finalizer and GPU memory still held; a reconcile loop brings them back in line" loading="lazy"></figure> <p>A platform keeps its own record of what a workload is doing. Kubernetes keeps the truth. &ldquo;Stuck&rdquo; is the gap between them.</p>