https://osmangoninahid.github.io/tags/sre/Recent content in Sre on Osman Goni NahidHugoen-USTue, 29 Sep 2026 00:00:00 +0400https://osmangoninahid.github.io/stuck-undeploying-two-ledgers/Tue, 29 Sep 2026 00:00:00 +0400https://osmangoninahid.github.io/stuck-undeploying-two-ledgers/<p>&ldquo;Stuck in Undeploying.&rdquo; &ldquo;Stuck in Pending.&rdquo; &ldquo;Stuck in Terminating.&rdquo; Over a year, that word showed up in more than a hundred tickets on a platform that deploys models onto GPU clusters. The states were different; the shape of the problem was the same every time.</p> <figure><img src="https://osmangoninahid.github.io/images/stuck-two-ledgers.gif" alt="Two ledgers side by side: the platform database says Undeploying for hours while Kubernetes shows the pod gone, a PVC waiting on a finalizer and GPU memory still held; a reconcile loop brings them back in line" loading="lazy"></figure> <p>A platform keeps its own record of what a workload is doing. Kubernetes keeps the truth. &ldquo;Stuck&rdquo; is the gap between them.</p>https://osmangoninahid.github.io/llm-endpoint-503-root-causes/Tue, 29 Sep 2026 00:00:00 +0400https://osmangoninahid.github.io/llm-endpoint-503-root-causes/<p>For about a year I have been the person the ticket lands on when a model endpoint answers 503, 500, or &ldquo;server unreachable&rdquo;. Different clusters, different models, same ticket title. Here is what it turned out to be, sorted by where the request actually died.</p> <figure><img src="https://osmangoninahid.github.io/images/llm-503-root-causes.gif" alt="One request walking through client, edge, gateway, engine and model; at each layer a card shows the root cause that was found there" loading="lazy"></figure> <p>Almost none of these were the model. Most were something in front of it.</p>