https://osmangoninahid.github.io/tags/helm/Recent content in Helm on Osman Goni NahidHugoen-USWed, 30 Sep 2026 00:00:00 +0400https://osmangoninahid.github.io/release-engineering-ai-platform-many-clusters/Wed, 30 Sep 2026 00:00:00 +0400https://osmangoninahid.github.io/release-engineering-ai-platform-many-clusters/<p>A SaaS deploys one version to one place. A platform sold to enterprises deploys to every customer&rsquo;s cluster, each on the version they last accepted, several with no internet. Release engineering is the part of platform engineering that makes that boring. This is how ours works, and the tickets that shaped it.</p> <figure><img src="https://osmangoninahid.github.io/images/release-migrations.gif" alt="A migration chain of preflight, database, object storage, Kubernetes entities and smoke test; before, a failing step did not stop the chain; after, it aborts and reports; three clusters on different versions walk the same steps" loading="lazy"></figure> <h2 id="the-shape-of-the-problem">The shape of the problem</h2> <ul> <li>Around forty services, each its own repository and pipeline.</li> <li>Two branch kinds everywhere: <code>main</code> and <code>release-X.Y.Z</code>.</li> <li>Thirty-plus Helm charts in one monorepo, from the application down to the message broker and the GPU exporters.</li> <li>Customer clusters on 1.7, 1.10, 1.13 at the same time. Some pull from our registry; some get a mirrored registry inside their network and nothing else.</li> </ul> <p>Every upgrade has to move three things that live in three places: the <strong>database schema</strong>, the <strong>object-storage layout</strong>, and <strong>Kubernetes objects</strong> (labels, services, cluster records). Miss one and the platform comes up &ldquo;healthy&rdquo; and wrong.</p>