https://osmangoninahid.github.io/tags/prometheus/Recent content in Prometheus on Osman Goni NahidHugoen-USThu, 24 Sep 2026 00:00:00 +0400https://osmangoninahid.github.io/llm-serving-metrics-that-matter/Thu, 24 Sep 2026 00:00:00 +0400https://osmangoninahid.github.io/llm-serving-metrics-that-matter/<p>A normal web request is one number: how long it took. An LLM request is not one number. It trickles out token by token, so a single &ldquo;duration&rdquo; hides almost everything that matters.</p> <p>Here is one request, and the numbers it leaves behind.</p> <figure><img src="https://osmangoninahid.github.io/images/llm-serving-metrics.gif" alt="One LLM request on a timeline: TTFT, inter-token latency, streaming and total duration, token count, in-flight gauge" loading="lazy"></figure> <h2 id="the-picture-in-words">The picture, in words</h2> <p>Four things happen on that line, and each one is a different metric:</p>