CI Kind E2E Implementation

This document describes the CI coverage for AGN’s Kind-based end-to-end validation suite.

Validation tiers

Gate implementation

CI gating is organized as static + unit + smoke Kind for the fast path. The multi-cluster suite runs on main, nightly, or release validation because it requires multiple Kind clusters and serialized execution.

Sequence:

1. Static checks and unit tests on every PR.
2. validate-operator-routing as the first Kind PR gate.
3. verify-swim-membership and verify-swim-state for distributed membership validation.
4. Full two-provider suite for nightly or release validation.

Single-cluster multi-gateway coverage

The run-grid-single-cluster-multi-gateway-qualification command uses one Kind cluster, one AGN Operator, one GridNetwork, and one GridSite named single, with two consumer gateways and three provider gateways. It is distinct from both a single gateway smoke test and the multi-cluster provider-traffic qualification: the single-cluster test shares one site, Kubernetes control plane, and overlay state, while the multi-cluster test exercises multiple sites connected through SWIM. Provider-selection cursors remain local to each Praxis process. The qualification must prove accepted and serving overlay revisions, per-consumer attributed traffic, provider withdrawal and restoration, consumer failure, and positive and negative security probes before it can report success.

Topology and execution details are maintained in the single-cluster topology README.

Multi-cluster coverage set

Sequencing requirements

Multi-cluster validations run sequentially. The current suite assumes shared cluster names and shared local operator process ports. Sequential execution avoids:

• SWIM UDP port conflicts
• Overlapping writes to the same Kind clusters
• Stale test fixtures from a previous validation affecting the next
• Competing gateway or provider deployments with the same names

Static and unit jobs run in parallel with each other. Kind jobs are serialized until the harness supports per-job cluster name isolation.

Environment requirements

CI runners require:

• Docker access for Kind
• kind
• kubectl
• The repository Rust toolchain
• The nightly toolchain used by formatting checks
• Gateway and mock-provider images available to Kind
• Permission to create and delete local Kind clusters

The Praxis AI gateway image must include the data-plane filters required for AGN routing and by the validation suite. CI consumes a reviewed, published image rather than building an unpinned image during the test job.

Flake controls

Each Kind run starts from a clean environment:

cargo xtask env down -c tests/env/<config>.toml
cargo xtask env up -c tests/env/<config>.toml
cargo xtask env load-gateway-images -c tests/env/<config>.toml

SWIM and failover tests do not use automatic retries. Those tests exercise process lifecycle and membership timing; retrying from a partially mutated cluster state can hide real bugs. If a retry is needed, it is a full clean down / up rerun.

Runtime tiers

Multi-cluster gate requirements

The full multi-cluster suite requires:

• A reviewed Praxis AI gateway image with the required routing filters must be published and referenced by CI.
• Kind cluster names must be either isolated per-job or the suite must run serially on a single runner.
• A reusable cleanup preflight removes all known E2E resources before a suite starts, making sequential runs reliable.