Release Process

Versioning

AI Grid Network (AGN) uses Semantic Versioning. The workspace version is defined in workspace.package.version in the root Cargo.toml. Workspace crates inherit that version. Keep the separate crates/gateway/Cargo.toml workspace version and its explicit gateway package version aligned with the root workspace; root version changes do not update them. Package versions alone do not establish image compatibility: qualify the tagged source and intended consumer below.

The five charts published by this release (grid-operator, grid-site, praxis-gateway, grid-mock-providers, and grid-enrollment) have versions matching the AGN release. The appVersion for AGN-owned workloads in the release charts also matches the AGN tag. The praxis-gateway chart is different: its appVersion identifies the default Praxis AI image and may advance independently of AGN.

grid-enrollment joined the release line in 0.2.0. Before that it versioned independently and its image was published elsewhere, so a reader installing from chart defaults could not pull it.

The project-owned grid-gateway operand is versioned with AGN and must be built from the same tagged AGN source. It is distinct from the separately released Praxis AI image used by the praxis-gateway chart.

Release Artifacts

An AGN release publishes:

• ghcr.io/praxis-proxy/grid-gateway:v<version>;
• ghcr.io/praxis-proxy/grid-operator;
• ghcr.io/praxis-proxy/grid-mock-providers;
• ghcr.io/praxis-proxy/grid-overlay-sync;
• ghcr.io/praxis-proxy/grid-enrollment;
• the grid-operator, grid-site, praxis-gateway, grid-mock-providers, and grid-enrollment Helm charts; and
• a GitHub Release containing generated notes and immutable artifact digests.

Build grid-gateway:v<version> from the matching AGN release tag and record its immutable digest in the release manifest. Image publication does not depend on a chart or operator consuming the image.

The praxis-gateway chart uses the separately released ghcr.io/praxis-proxy/ai:<tag> image. Qualify any chart or operator that consumes grid-gateway separately, as described below.

AGN does not publish a Praxis AI image or an AI rollup. The release workflow verifies the pinned official Praxis AI image used by the praxis-gateway chart, including its digest and OCI provenance.

Optional Praxis AI filters are an explicit deployment dependency. Examples or qualifications that require optional filters must document the required Cargo features and require the caller to provide a compatible image. They must not silently substitute a custom-built Praxis AI image.

Pre-release Checklist

Before opening a release preparation pull request:

• Update the workspace version in Cargo.toml and regenerate Cargo.lock.
• Update both workspace.package.version and package.version in crates/gateway/Cargo.toml and regenerate crates/gateway/Cargo.lock.
• Update the five release-chart versions listed above.
• Update their AGN workload appVersion values to the AGN tag.
• Verify the praxis-gateway appVersion and default image match the intended official Praxis AI release.
• Bump every chart value that pins an AGN image tag explicitly, such as praxis-gateway overlay.sidecar.image.tag. An empty tag there resolves to the chart appVersion, which is the Praxis AI version, so these are pinned by hand and the release workflow does not check them.
• Run make generate-deploy and commit the result. The manifests under deploy/operator and deploy/overlays render from charts/grid-operator, so they carry the operator image tag and the kind version label.
• Confirm the release workflow builds ghcr.io/praxis-proxy/grid-gateway:v<version> from the same tagged AGN source, publishes the version tag, and records the resulting immutable digest.
• Update the release workflow’s pinned AI tag, digest, and source revision when the default Praxis AI image changes.
• Run make lint for both workspaces, then make lint-extra.
• Run make test and make doc for the root workspace.
• Run cargo test --locked --manifest-path crates/gateway/Cargo.toml --workspace for the separate Gateway workspace.
• Run make codegen-check, make crds-check, and make deploy-check.
• Run git diff --check and validate the release workflow with actionlint.
• Lint and render the five release charts with their required values.
• Validate affected Forge topologies.
• Run the relevant integration qualifications when routing, overlay, or gateway compatibility changes.
• Confirm generated evidence and local build artifacts are not committed.
• Review pull request labels so generated release notes are useful.

The first-class integration qualifications are documented in the documentation index. They build isolated Kind environments, exercise runtime behavior, record evidence, and clean up their resources.

Integration qualification checklist

Run the individual qualification commands below against fresh source builds and fresh, uniquely tagged images. There is deliberately no aggregate release qualification command. Run the commands sequentially in a dedicated checkout: several use shared Forge state, Kind cluster names, or Docker network names. Until Forge teardown is run-scoped, a concurrent run using the same state directory can lose its clusters even when it uses a different config file.

Use the following one-line command stack after setting IMAGE_TAG and EVIDENCE_DIR for each run. The quota qualification additionally requires a Praxis AI image built with the optional features shown below; the other topologies require the filters documented in their READMEs and may use the official compatible AI image where applicable.

cargo xtask env run-grid-provider-traffic-qualification --forge-config tests/e2e/topologies/grid-provider-traffic/forge.yaml --full --teardown --evidence-dir "$EVIDENCE_DIR"
GRID_XTASK_SIM_IMAGE=ghcr.io/llm-d/llm-d-inference-sim:v0.10.2 cargo xtask env run-grid-static-weighted-qualification --forge-config tests/e2e/topologies/grid-static-weighted/forge.yaml --quick --teardown --evidence-dir "$EVIDENCE_DIR"
cargo xtask env run-grid-token-rate-limit-qualification --forge-config tests/e2e/topologies/grid-token-rate-limit/forge.yaml --image-tag "$IMAGE_TAG" --evidence-dir "$EVIDENCE_DIR"
cargo xtask env run-grid-single-cluster-multi-gateway-qualification --forge-config tests/e2e/topologies/grid-single-cluster-multi-gateway/forge.yaml --evidence-dir "$EVIDENCE_DIR"
cargo xtask env run-grid-combined-site-demo --forge-config tests/e2e/topologies/grid-combined-site/forge.yaml --full --teardown --evidence-dir "$EVIDENCE_DIR"
cargo xtask env run-grid-glb-demo --forge-config tests/e2e/topologies/grid-glb-demo/forge.yaml --full --teardown --evidence-dir "$EVIDENCE_DIR"
cargo xtask env run-grid-glb-demo --forge-config tests/e2e/topologies/grid-glb-demo/forge.yaml --no-ingress --full --teardown --evidence-dir "$EVIDENCE_DIR"
GRID_XTASK_OPERATOR_IMAGE="${GRID_XTASK_OPERATOR_IMAGE:?build a source-matched Grid operator; see the llm-d README}" GRID_XTASK_GRID_GATEWAY_IMAGE="${GRID_XTASK_GRID_GATEWAY_IMAGE:?build a source-matched Grid Gateway; see the llm-d README}" cargo xtask env run-grid-llmd-pool-metrics-demo --forge-config tests/e2e/topologies/grid-llmd-pool-metrics/forge.yaml --full --teardown --evidence-dir "$EVIDENCE_DIR"

For release evidence, use the triggered workflow’s immutable EPP and simulator image references, or reject the run if either reference lacks a sha256 digest or the recorded container image ID is missing. The command above uses mutable defaults when these image variables are unset and does not enforce those evidence requirements by itself.

The checked-in Praxis configurations for these Kind qualifications explicitly enable both insecure_options.allow_private_endpoints and insecure_options.allow_private_upstreams because their test backends use private Kind Service addresses and reserved cluster IPs. These are qualification-only settings; production configurations retain the default private-upstream rejection.

Images and optional features

Each run must use exact image references that are materialized into the resolved Forge configuration and loaded into Kind when imagePullPolicy: Never is used. Record the image IDs or immutable digests in the evidence. Typical local overrides are:

export GRID_XTASK_GATEWAY_IMAGE=praxis-ai:$IMAGE_TAG
export GRID_XTASK_OPERATOR_IMAGE=grid-operator:$IMAGE_TAG
export GRID_XTASK_OVERLAY_SYNC_IMAGE=grid-overlay-sync:$IMAGE_TAG
export GRID_XTASK_MOCK_PROVIDER_IMAGE=grid-mock-providers:$IMAGE_TAG
export GRID_XTASK_SIM_IMAGE=ghcr.io/neuralmagic/vllm-vcr:vllm0.23
export GRID_XTASK_IMAGE_PULL_POLICY=Never

In the current qualification tooling, GRID_XTASK_GATEWAY_IMAGE names the Praxis AI gateway image used by the fixtures. It does not refer to the project-owned grid-gateway operand.

The distributed token-quota qualification requires the optional Praxis AI token-rate-limit-filter. Praxis AI 0.4.0 already includes Basic Auth; a custom quota image only needs to enable the token filter. The qualification’s image label still declares both available filters:

token-rate-limit-filter,praxis-filter/basic-auth-filter

The image label and runtime registrations must match the compiled feature set. The standard official AI image may be suitable for provider selection and other gateway paths, but a quota run must be BLOCKED, never marked PASS, if the required filters are absent or its feature label is unavailable or inaccurate. See the provider traffic README, quota README, and the other topology READMEs for exact image preparation.

Evidence, cleanup, and release gates

Create a new UTC-stamped evidence directory for every run, normally beneath evidence/. Evidence from an older commit, source tree, image tag, or image digest is not valid for the current release. Generated evidence must not be committed.

Use --teardown where the command accepts it; the quota and single-cluster qualifications attempt cleanup by default unless --keep is set. SIGKILL can stop either qualification before cleanup runs, and the quota runner’s default SIGINT path also skips its cleanup guard. Verify the actual cleanup results and ownership before removing anything manually. A missing prerequisite, unavailable image, missing Forge binary, or failed readiness/convergence check is BLOCKED or FAIL, never PASS. Preserve non-2xx responses and first failures in the evidence; do not hide them with retries or reinterpret them as successful routing.

cargo xtask env validate-all is a separate legacy/local validation suite. It does not run the release qualification matrix above and must not be presented as a substitute for these topology-specific commands.

Experimental classification means that a topology is outside the default supported release surface; it does not permit weakening assertions or ignoring failed results. When an experimental path is changed or claimed in release notes, run it and report its evidence independently.

Qualification details

Coverage gaps that remain visible

The llm-d qualification separately rejects a wrong peer certificate pin while SWIM remains connected, then tests a deployed network partition with both operators and the Grid Gateway running. The standalone verify-failover-under-lost-peer operator check stops an operator process and does not cover that deployed boundary.

Geography-first selection after peer auto-discovery remains unqualified. Auto-discovered GridSite resources do not currently carry region and zone from SWIM; a deployed three-tier proof needs that metadata propagation before it can assert geography without manually patching discovered resources.

The optional GLB external-provider proof is a separately configured live check. When its protected credential is absent, the runner reports SKIPPED; that does not count as a passed external-provider qualification.

Tagging A Release

Tags use v<MAJOR>.<MINOR>.<PATCH>. Create the tag only from the reviewed commit after its pull request and CI checks pass. Sign the tag using the project’s normal Git signing configuration:

VERSION="${VERSION:?set VERSION to the workspace release version}"
TAG="v${VERSION}"
git tag -s "$TAG" -m "AGN $TAG"
git push origin "$TAG"

Do not move or recreate a published release tag. Prepare a new patch release for corrections.

Automated Publication

Pushing a valid release tag triggers the Release workflow. The workflow:

1. verifies that the tag matches the workspace version;
2. reruns lint, tests, and documentation validation from the tagged source;
3. verifies the pinned official Praxis AI image and provenance;
4. builds and publishes the version-tagged AGN container images, including grid-gateway, with SBOM and provenance attestations;
5. validates, packages, and publishes the five release charts; and
6. creates the GitHub Release with generated notes and immutable digests.

The workflow can also be dispatched for an existing immutable release tag. A manual dispatch does not replace the requirement for a reviewed, signed tag.

Post-publication Verification

For every release:

• Confirm ghcr.io/praxis-proxy/grid-gateway:v<version> exists and its registry digest matches the digest recorded in the GitHub Release manifest.
• Confirm that deployments using the praxis-gateway chart still resolve the separately released ghcr.io/praxis-proxy/ai:<tag> image; that chart is not evidence that the grid-gateway operand is wired.

grid-gateway consumer qualification

Before deploying the grid-gateway operand:

• Identify the intended chart or operator consumer and verify that its rendered image reference uses the release’s grid-gateway image by its recorded immutable digest, or a version tag that resolves to that digest.
• Preserve the rendered consumer reference with the deployment evidence.

Until that consumer exists and passes this check, record grid-gateway consumption as NOT QUALIFIED. This does not block publication of grid-gateway or the other AGN release artifacts.

Release Notes

AGN uses GitHub Releases for release notes. There is no committed version-specific changelog. Add the user-facing summary, compatibility notes, upgrade considerations, qualification results, and demonstration links to the GitHub Release page.

Release Branches

Release branches are optional. Create one from a release tag only when a supported line needs a backport. Use release/v<MAJOR>.<MINOR>.x, cherry-pick the focused fix, and publish a new patch tag through the normal workflow.