https://osmangoninahid.github.io/tags/scheduling/Recent content in Scheduling on Osman Goni NahidHugoen-USSat, 03 Oct 2026 00:00:00 +0400https://osmangoninahid.github.io/gpu-allocation-model-why-the-number-lies/Sat, 03 Oct 2026 00:00:00 +0400https://osmangoninahid.github.io/gpu-allocation-model-why-the-number-lies/<p>A fixed fleet, many teams, one dashboard number: <em>available GPUs</em>. On a shared cluster that number is not read — it is assembled from four sources. Every capacity ticket I handled this year was two of them disagreeing.</p> <figure><img src="https://osmangoninahid.github.io/images/gpu-allocation-lies.gif" alt="Four sources (Kubernetes nodes, platform records, scheduler queues, batch scheduler) feed one &#39;available GPUs&#39; number; five lies cycle through — a node that left, a GPU idle on one side and busy on the other, a frozen usage sync, MIG slices counted as devices, a Ready node with zero GPUs — then the fix" loading="lazy"></figure> <h2 id="the-model">The model</h2> <pre class="mermaid"> graph TB C[&#34;cluster&lt;br/&gt;nodes · GPU types · MIG slices&#34;] --&gt;|&#34;nodes → tenant&lt;br/&gt;target: k8s | slurm&#34;| T[&#34;tenant&#34;] T --&gt;|&#34;GPUs by type → workspace&#34;| W[&#34;workspace&lt;br/&gt;queue: capability · deserved · borrow&#34;] W --&gt; P[&#34;pods&#34;] </pre> <p>Three levels, three owners. A node serves one <em>target</em> — Kubernetes or the batch scheduler — never both. A workspace is a scheduler queue with a hard ceiling, a deserved share, and a borrowing flag.</p>