https://osmangoninahid.github.io/tags/inference/Recent content in Inference on Osman Goni NahidHugoen-USWed, 23 Sep 2026 00:00:00 +0400https://osmangoninahid.github.io/gpu-numbers-for-llm-inference/Wed, 23 Sep 2026 00:00:00 +0400https://osmangoninahid.github.io/gpu-numbers-for-llm-inference/<p>You have a model. You have a GPU, or a budget for one. Will it serve? TFLOPS will not tell you. These four numbers will, in this order.</p> <pre class="mermaid"> flowchart LR A[&#34;1 Memory&lt;br/&gt;does it fit?&#34;] --&gt; B[&#34;2 Bandwidth&lt;br/&gt;how fast per token?&#34;] B --&gt; C[&#34;3 Precision&lt;br/&gt;what math is native?&#34;] C --&gt; D[&#34;4 Scaling&lt;br/&gt;one card or many?&#34;] A -. &#34;no? shrink it&#34; .-&gt; C </pre> <h2 id="1-memory-does-it-fit">1. Memory: does it fit?</h2> <p>Three things need room on the card.</p>