Custom Resource Definitions

API group: grid.praxis.fast/v1alpha1

The AI Grid Network (AGN) Operator defines these resources to describe sites, provider capacity, and routing policy. The established API identities remain unchanged by the project-name update.

All CRDs are cluster-scoped and served at v1alpha1.

SiteInvite

Declares an invitation on the signer cluster. The signer pins the site name and jurisdictions, optionally binds the invite to a public key digest, and limits its lifetime (default 24h, from 5m through 168h). The spec is immutable; delete and recreate an invite to change it.

apiVersion: grid.praxis.fast/v1alpha1
kind: SiteInvite
metadata:
  name: east2
spec:
  siteName: east2
  jurisdictions:
    - eu-west
  lifetime: 24h
  allowDeletedName: false
  # publicKeySHA256: <64 lowercase hex characters>

The status reports the public token ID, the Secret reference holding the one-time token, its expiry, redemption details, and conditions. Treat the referenced Secret as a credential and deliver it and the grid CA bundle to the joining site’s operator namespace out of band. The token itself is never published in status.

In Kubernetes, the controller reads SiteInvite objects, stores each token in a Secret, and revokes unused tokens when an admin deletes an invite. Standalone enrollment does not start the controller. Enrollment enforces an optional key binding. Jurisdiction assignment and the human approval gate remain pending, so a redeemed token currently issues a certificate inline.

The chart stores token Secrets in its release namespace by default. Configure a dedicated namespace in the chart values to separate access to those credentials.

The 0.3.x operator accepts both previous and current CRD spec field names. The previous names are deprecated. Set only one name per field; setting both is rejected by the API schema. The grid-site chart translates previous value names into current CRD fields and also rejects both names together. See the 0.3.x CRD field migration guide for the planned removal and upgrade steps.

GridNetwork

The AGN logical network and top-level tenancy scope. GridNetwork is a cluster-scoped CRD. The current AGN deployment supports only one GridNetwork in the cluster. The operator fails startup when it finds more than one; adding another after startup is not supported.

apiVersion: grid.praxis.fast/v1alpha1
kind: GridNetwork
metadata:
  name: production
spec:
  gridId: ""                    # auto-generated on first join
  seeds:
    - "10.0.0.5:7946"
  consumerGateways:
    - name: inference-gw
      namespace: praxis-system
      siteName: cluster-east   # optional; defaults to network name
      praxisConfig:               # optional; opt-in consumer Praxis config generation
        generate: true
        enableProjectedCredentials: true  # generate filter, then roll out consumer
        supportsProjectedCredentials: true # set only after that rollout
        credentialMountPath: /run/secrets/grid-credentials
        configMapName: praxis-consumer-config
        tlsCertMountPath: /etc/praxis/tls
        mountReconciliation:
          enabled: true
          deploymentName: inference-gw
          containerName: praxis
        clusterEndpoints:           # endpoint topology for load_balancer
          - cluster: site-a
            address: "10.0.0.4:30080"
            transport:
              mode: mutual_tls         # mTLS with CA verification and client cert
              sni: site-a.grid.internal
          - cluster: api-provider
            address: "api.example.internal:443"
            transport:
              mode: tls                # verified server TLS
              sni: api.example.internal
              caSecretRef:
                name: api-provider-ca  # Secret in praxis-system (gateway namespace)
                # key: ca.crt          # optional; defaults to ca.crt
  region: us-east-1
  zone: us-east-1a
  swim:
    probeInterval: 5s           # WAN probe interval
    suspicionTimeout: 10s       # before declaring dead
    gossipNodes: 3              # indirect probe fanout
  tls:
    caSecretRef:
      name: grid-ca
      namespace: praxis-system
    siteSecretRef:
      name: grid-site-cert
      namespace: praxis-system
    swimKeySecretRef:
      name: swim-key
      namespace: praxis-system
  budgetPolicy:                   # optional; absent means no tenants are tracked
    tenants:
      - tenantId: tenant-a
        capUsd: 100.0
      - tenantId: tenant-b
        capUsd: 250.0

Routing policy fields

These optional GridNetwork.spec fields control provider ordering, admission, and request selection. For guidance on composing them to achieve a routing behavior, see the AGN Routing Guide.

geographyFirst and scoreFirst are the only current routing-policy values. Explicit grouping fields such as selectionPolicy.grouping.localityScope are not part of this CRD.

Phases: Pending → Initializing → Active → Degraded

Status fields: gridId, connectedSites, remoteProviderCount, observedGeneration, phase, conditions[], praxisConfigStatus[], mountReconciliationStatus[], budgetStatus[], gatewayAgeReports[], gatewayAgeReportOverflow

Conditions: status.conditions carries a PeerTimeValid condition, written by the operator’s peer poller under its own field manager (grid-operator-peer-signals), never by GridNetwork reconciliation. True means no polled peer has a rejected timestamp; False means at least one does.

A relayed age survives a clock step on every hop, so a step alone never makes this condition false: it points at a peer whose clock or stamping is wrong. Rejected rows are dropped and counted in grid_peer_signals_refused_total{peer,reason="skew"}. Only rejected rows become unavailable for provider measurement; valid rows from the same peer remain available. A peer that does not answer keeps its standing, a peer that leaves membership is dropped, and one clean answer restores its validity. The operator conservatively preserves an existing False condition after restarting until every currently polled peer has answered or left membership. Unavailable secure polling is not a clean round and does not establish recovery. The condition is written only when its status, its message or the generation changes, since each write starts a GridNetwork reconcile.

Peer-age observation and status publication wait for the operator’s existing membership-convergence gate, so an initial empty snapshot cannot erase a live rejection during restart. Signal collection continues while this gate is held. A round begun while held cannot publish even if convergence finishes during collection; the next round uses the converged membership.

remoteProviderCount reflects the number of remote InferenceProvider records received from peer sites via CRDT broadcast. Local providers and records from other GridNetworks are excluded from the count.

Gateway age evidence

conditions[] is a map keyed by condition type. GatewayAgesRejected reports local gateway ingest rejection under its own status manager, independently of peer diagnostics and normal GridNetwork reconciliation. True means a fresh gateway process has unresolved rejected ages. False requires genuine newer clean evidence from every recorded rejecting process. Unknown means the remaining evidence is stale, predates the operator restart, or reporter capacity was exceeded. Conditions never change network phase or routing.

gatewayAgeReports[] retains at most 128 records keyed by random process epoch: replica label, sequence, cumulative skew count, original report flag, effective rejection state, and receipt timestamp. Live freshness uses a five-minute monotonic receipt window. Expiration is evaluated when another report arrives; if reporting stops entirely, the last condition remains until a report arrives. After an operator restart, persisted rejected epochs have unknown freshness until that same epoch submits a strictly newer observation. Wall-clock changes and duplicate reports cannot establish recovery or renew freshness.

A clean report from another replica or replacement process never clears an older rejected epoch. The listener identifies the site, not authoritative pod membership. Once an operator has independently verified that an old process retired, they may explicitly remove its gatewayAgeReports status entry. The next report recomputes the condition. Silence or a timeout is insufficient proof of retirement.

Only clean bookkeeping may be evicted to admit a new process. When all 128 entries hold unresolved evidence, the operator persists gatewayAgeReportOverflow: true and refuses new identities; existing processes can still report recovery. This bounded uncertainty survives recovery and restart. Clear it explicitly only after verifying reporter coverage. Inspect status with kubectl get gridnetwork NAME -o yaml; status edits require RBAC permission for the GridNetwork status subresource.

praxisConfigStatus[] is populated for each gateway with praxisConfig.generate: true, reporting the outcome of the most recent render/apply attempt.

Set praxisConfig.mountReconciliation.enabled: true to delegate generated Secret mounts and coordinated Deployment rollouts to Grid. This requires an explicitly annotated Deployment in GatewayRef.namespace, and a Role in that namespace letting the operator get and patch that one Deployment; the praxis-gateway chart renders it, see Deployment access. The operator manages only its reserved projected volumes and the selected container’s mounts. Secret references must be in the gateway namespace. praxisConfigStatus[].phase of Rendered describes the ConfigMap; only mountReconciliationStatus[].phase: Ready reports a ready gateway revision. With delegation disabled, the operator still publishes a reference-only grid-mount-requirements-<hash> ConfigMap with the document under mount-requirements.json for the gateway owner to consume.

clusterEndpoints[].transport.mode accepts mutual_tls, tls, or plaintext. The optional transport.caSecretRef is valid for tls and names a custom CA Secret in the target GatewayRef.namespace; it has name and optional key fields only, with the key defaulting to ca.crt. Existing manifests must remove its former namespace field and place the Secret in the gateway namespace. CRD pruning removes unknown fields: Warn mode accepts the object and reports a warning, Ignore silently drops the field, and Strict rejects the request. This namespace-local reference is distinct from spec.tls.caSecretRef, which retains its explicit namespace. mutual_tls uses spec.tls.caSecretRef and spec.tls.siteSecretRef and does not accept a custom CA override.

Tenant budget tracking

budgetPolicy.tenants[] opts individual tenants into cumulative spend tracking. AGN merges each site’s locally recorded spend for a tenant into a per-tenant CRDT counter (a GCounter, one slot per originating site) that is gossiped over SWIM alongside provider state, so the reported total reflects spend recorded anywhere in the grid, not just the local site.

For every tenant declared in budgetPolicy, budgetStatus[] reports:

• tenantId: matches budgetPolicy.tenants[].tenantId
• capUsd: copied from the policy, in USD
• spendUsd: the converged cross-site total, in USD
• spendRatio: spendUsd / capUsd, for at-a-glance dashboarding

budgetStatus[] is a status signal only. AGN does not itself degrade or reject traffic when a tenant’s spendRatio reaches or exceeds 1.0; that enforcement decision is expected to live in a gateway-side policy filter (cross-repo, praxis-ai), the same split used for provider_route authorization. Real per-request tenant attribution also depends on upstream work (praxis-ai#130/praxis-ai#104) and does not exist yet.

Because budgetStatus[] is visible to any caller with read access to the GridNetwork resource, and Kubernetes RBAC is not field-level, a reader authorized to view one tenant’s status can see every other tracked tenant’s spend on the same GridNetwork. See grid#48 for the options under consideration if per-tenant confidentiality is required.

For a delegated gateway, mountReconciliationStatus[] reports the rendered requirements revision, the config revision last rolled out, the Deployment generation, and one of RequirementsRendered, WaitingForSecret, MountsReconciling, WaitingForRollout, Ready, or Error. Messages contain resource names and paths only. They never contain Secret data or private keys.

Example status output:

status:
  phase: Active
  gridId: grid-abc123
  connectedSites: 2
  praxisConfigStatus:
    - gatewayName: inference-gw
      namespace: praxis-system
      configMapName: praxis-consumer-config
      phase: Rendered
      reason: ""
      message: "consumer config rendered and applied to praxis-system/praxis-consumer-config"
      observedGeneration: 7
    - gatewayName: fallback-gw
      namespace: default
      configMapName: op-e2e-consumer-config
      phase: Error
      reason: PraxisConfigRenderFailed
      message: "consumer config render: overlay local_site must not be blank"
      observedGeneration: 7

CRD-driven SWIM seeds

spec.seeds is a list of SWIM endpoints (host:port) used to bootstrap SWIM mesh formation. Each entry may be a literal IPv4 address, a bracketed IPv6 address, or a DNS hostname. Hostnames are resolved with a bounded lookup before they are announced to the running SWIM runtime on every GridNetwork reconcile. Re-announcing to an existing peer is idempotent: foca ignores redundant joins.

Runtime update behavior:

Channel-full behavior: If the SWIM announce channel is temporarily full (capacity 16 batches), the announce is silently retried on the next reconcile. Seeds are not guaranteed to be joined within one reconcile cycle under heavy broadcast load, but the retry is automatic.

Scope: spec.seeds targets the SWIM site-membership layer. The SWIM runtime is process-global: all GridNetwork resources in the operator process share the same SWIM node. Seeds from any GridNetwork reach the shared SWIM membership table. Provider CRDT state remains scoped per network.

Self-filtering: The operator removes its own SWIM bind address from spec.seeds before announcing, preventing self-join loops.

spec.tls.swimKeySecretRef: References a Kubernetes Secret containing the 32-byte AES-256-GCM key for SWIM transport authentication. When configured, the GridNetwork controller reads the key from the Secret and configures the SWIM runtime to encrypt all outgoing UDP packets and reject incoming packets that fail authentication before it announces CRD seeds or publishes certificate/provider state for that reconcile.

The Secret must contain a "key" field (or the field named by swimKeySecretRef.key) with exactly 32 bytes. If the Secret is absent, unreadable, or has the wrong length, the reconcile fails before CRD seed announcement and state broadcast. The process-global SWIM runtime keeps any previously loaded key until restart; it does not switch to plaintext for that configured reconcile.

For local development and testing, the GRID_SWIM_ENCRYPT_KEY environment variable (64-character hex) provides an alternative key injection path without requiring a Kubernetes Secret. Because environment variables are visible to same-host process inspectors, this is not the production Secret delivery path.

Routing eligibility remains gated separately by GridSite.status.phase == Active, or Unreachable under poll signal transport, regardless of SWIM encryption configuration. Active status indicates control-plane eligibility only: sufficient trust information exists to include the site in routing overlays. Data-plane security readiness (mTLS handshake completion, client certificate validation, routing config loading) is verified separately at request time.

Stale candidate TTL

spec.staleCandidateTtlSeconds controls when stale (fresh=false) remote candidates are evicted from the rendered overlay.

Local and healthy remote candidates are never evicted. CRDT storage records are not deleted by this mechanism.

Under poll signal transport the operator removes no candidate for liveness. A failed probe, a Dead or Suspect SWIM member, this TTL and the SWIM dead-member TTL leave a remote site’s candidates in place, and the gateway routes around a silent site. A candidate leaves when trust fails, its endpoint changed and awaits a probe, its providers are withdrawn, or its GridSite is deleted.

With site auto discovery on, the same TTL bounds auto-discovered GridSites. When gossip stops vouching for a stub’s site (any SWIM state but Dead), the operator records status.absentSince and clears it if the site returns. Once that is at least N seconds old, the stub is deleted and no longer counts against the 256-site discovery cap. The delete is conditional on the object being unchanged, so a site that rejoins first keeps its stub. Declared GridSites are never deleted. With the TTL absent, stubs use a 24-hour default, so the cap always drains; the overlay still keeps stale candidates.

The operator judges no site absent until it has been up for one gossip verification window (a 10-minute record expiry plus convergence), so a restart never starts clocks or collects. A pass that would delete more than half the stubs, and more than 8, deletes none and logs a warning; a partition looks like mass departure.

GatewayRef.providerHopEndpoints

spec.consumerGateways[].providerHopEndpoints independently configures the provider-hop trust allowlist used by the embedded grid-gateway. Each entry names a provider cluster and declares its verified mTLS mode and SNI. The embedded gateway’s startup upstream configuration must use that TLS identity. This field is independent of praxisConfig; absent or disabled generated consumer Praxis config cannot make embedded serving fail validation or change its provider-hop allowlist.

GatewayRef.praxisConfig

spec.consumerGateways[].praxisConfig opts a gateway into operator-managed consumer Praxis ConfigMap generation.

Grid publishes a valid empty versioned overlay when no candidates remain. Every consumer of that overlay must use a Praxis AI image that accepts empty versioned snapshots; older images, including the chart-default 0.4.0 image, are incompatible. Upgrade and roll out the consumers before deploying this Grid version. Malformed or undeliverable updates still retain the last valid revision; an authoritative empty candidate set does not.

When generate: true, the GridNetwork controller renders a praxis.yaml-keyed ConfigMap in the gateway namespace on each reconcile. The generated config is a complete, runnable Praxis config containing:

• listeners: with one public listener at 0.0.0.0:{listenerPort}
• filter_chains: holding the consumer chain with:
  • intelligent_route reading the versioned routing overlay with expected network, gateway, namespace, and local-site scope. Candidates and selection policy are not copied into startup-only YAML.
  • credential_inject is included when current candidates have credential references or enableProjectedCredentials: true. In projected mode it starts with an empty table and resolves later references under the projected Secret mount; missing files fail closed. Token bytes are never written to the ConfigMap.
  • load_balancer entries for every configured endpoint, including currently inactive providers needed for restoration. Every potentially routable cluster must have a matching clusterEndpoints[] entry with endpoint address and explicit transport configuration. transport.mode is the security switch, not sni presence. Missing transport fails closed
• admin: with a health and metrics listener at 127.0.0.1:9901 under metrics_address
• shutdown_timeout_secs: 5

The generated credential-injection config is for credentials intentionally projected into the consumer gateway’s namespace. For remote provider sites, credentials should normally be mounted only in the remote site or provider-side component that makes the final backend call.

The credential_inject filter is a Praxis AI runtime dependency. The AGN operator can render the config shape, but the deployed Praxis AI image must include that filter for the generated config to start successfully.

The consumer Deployment must mount the gateway’s grid-overlay-<network>-<gateway> ConfigMap key routing-overlay.json at /etc/praxis/routing/routing-overlay.json as a projected volume (never with subPath). Cross-cluster consumers must deliver both the Praxis config ConfigMap and routing overlay to the consumer cluster. The initial migration from static inline candidates requires a consumer rollout; listener, endpoint/TLS, or credential-table changes continue to require a rollout, while route-only overlay revisions hot reload.

For generated consumer configs, first set enableProjectedCredentials: true and roll out the consumer so it loads the filter and mount root. Then set supportsProjectedCredentials: true to attest that rollout and mount the referenced Secret. The Grid controller does not own or restart consumer Deployments. Until readiness is attested, credential-bearing overlay revisions are retained. In projected mode, the generated config uses an empty credential table and resolves the selected Secret reference at request time, failing closed when it cannot read it.

When generate: false or praxisConfig is absent, no generated consumer Praxis ConfigMap is applied. The routing overlay remains active, and the embedded gateway’s serving config is controlled separately by GatewayRef.providerHopEndpoints.

GridSite

Represents another site in the grid. Created manually for seed peers or automatically by SWIM discovery.

apiVersion: grid.praxis.fast/v1alpha1
kind: GridSite
metadata:
  name: cluster-b
  labels:
    grid.praxis.fast/network: production
spec:
  gridNetworkRef: production
  gatewayEndpoint:
    address: egress.cluster-b.example.com:8443
    tls:
      mode: Mutual
      serverName: egress.cluster-b.example.com
  trust:
    canonicalFingerprints:
      - "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
  region: us-east-1
  zone: us-east-1a
  sovereigntyZone: us

Phases: Pending → Discovered → Connecting → Active → Unreachable → Left

Status fields: phase, reason, message, observedGeneration, publicCertPem, capabilities (inference, agentTools, agentToAgent), lastProbeTime, lastTransitionTime, absentSince (auto-discovered sites only)

GridSite lifecycle

SWIM discovery, authentication, and authorization are separate concerns:

• SWIM discovery identifies a peer and records liveness.
• Authentication proves the peer gateway identity, normally through mTLS certificate validation.
• Authorization decides whether that authenticated peer is allowed to participate in the AI Grid Network or carry traffic for a given policy scope.

A discovered SWIM peer is not automatically authorized for routing.

Reason codes (in status.reason):

GridSite phase transitions:

• Pending → Discovered: the GridNetwork controller writes Discovered when a remote SWIM peer is first observed as Alive (requires grid.praxis.fast/auto-discover-sites: "true" label on the GridNetwork).
• Discovered → Connecting: the GridSite controller advances automatically when spec.gatewayEndpoint.address is non-empty. For auto-discovered sites, the egress address comes from the remote operator’s GRID_GATEWAY_ADDRESS env var, propagated via SWIM state broadcast. If the remote operator has not configured GRID_GATEWAY_ADDRESS, the egress address is empty and the site stays Discovered with reason GatewayAddressMissing.
• Connecting: the GridSite controller probes the egress gateway on each reconcile. For Mutual TLS mode, the probe performs a bounded TLS handshake verifying the CA chain, serverName SAN, and required canonicalFingerprints pin. For explicit Plaintext mode, it performs a bounded TCP connect for diagnostics but keeps the site in Connecting; TCP reachability alone cannot make a site routing-eligible. Active also requires mTLS client credentials (siteSecretRef must be configured on the GridNetwork).
• Active → Unreachable: the GridSite controller demotes Active to Unreachable when the probe cannot connect. Identity or trust failures demote Active to Connecting, distinguishing a reachable but unverified endpoint from an unreachable endpoint.

Metrics: grid_site_phase{site,phase} reports each GridSite’s phase as a state set, the shape of kube-state-metrics’ kube_pod_status_phase. The series for the current phase is 1 and the other five are 0, so a site is ready when grid_site_phase{phase="Active"} == 1. It follows the phase the printer column shows, updates on every GridNetwork reconcile, and drops a site’s series once its GridSite is gone. Cardinality is six series per site. grid_site_phase_transition_total still counts the transitions by phase and reason.

spec.gatewayEndpoint.address source: For auto-discovered sites, the egress address is sourced from the remote operator’s GRID_GATEWAY_ADDRESS environment variable, propagated through the SWIM state broadcast. If the remote operator has not configured GRID_GATEWAY_ADDRESS, the field is empty and the site stays Discovered. For manually-applied GridSite resources, set spec.gatewayEndpoint.address explicitly to the data-plane gateway endpoint.

status.publicCertPem: The public site certificate PEM received from the remote site via SWIM state broadcast. Before storage, the operator performs a structural check: private-key markers (PRIVATE KEY) cause the input to be discarded entirely and an error logged. Non-certificate PEM triggers TrustMaterialInvalid status. A valid CERTIFICATE header passes the structural check.

This field contains only the public certificate, never a private key. A non-empty publicCertPem means the remote site has shared its public identity material and the structural check passed. It does not mean:

• The certificate has been chain-verified against a trusted CA.
• The peer is authenticated or authorized for routing.
• The content has been parsed as X.509.

Private keys, bearer tokens, provider credentials, and Kubernetes Secret contents must never be written to status.

spec.gatewayEndpoint.tls fields:

spec.trust fields:

Routing eligibility: GridSite.status.phase == Active is the control-plane eligibility gate for remote CRDT provider records. Active means the control plane has verified the remote site’s identity through TLS, which is sufficient to include the site’s providers in routing overlays for consideration.

Provider records from a peer whose GridSite is in Discovered, Connecting, Pending, or Left are excluded from the routing overlay, and so are those from an Unreachable peer except under poll signal transport, where the gateway judges reachability from signals. Records from a peer with no matching GridSite are also excluded (fail-closed).

GridSite Active is a control-plane eligibility signal. It means AGN has enough site/trust information to consider the site for overlay generation. It does not prove that a Praxis gateway has loaded the latest routing config or authorized provider-side traffic. Data-plane readiness is verified separately at request time by provider gateway filters.

Single-site / combined-site deployments: Site discovery is driven by remote SWIM peers - the Pending -> Discovered transition above requires a remote peer observed Alive. A single or combined cluster has no peers, so its own GridSite stays Pending with reason AwaitingDiscovery. This is expected and does not block local serving: local InferenceProviders are eligible regardless of GridSite.status.phase; only remote CRDT provider records are phase-gated (see “Routing eligibility” above). Do not add SWIM seeds or extra operator replicas to try to force the site Active - there is no second site to discover, and a lone operator legitimately runs a single-node mesh with zero peers. The Active phase and its mTLS gateway probe (spec.gatewayEndpoint + spec.trust) apply to reaching remote sites, or a manually-configured peer gateway endpoint. See Architecture Overview -> Single-Site and Combined Deployments.

See Routing eligibility for the full gating rule.

Example status (Mutual TLS verified):

status:
  phase: Active
  reason: TlsVerified
  message: "TLS handshake succeeded; certificate chain, identity, and pin verified"
  observedGeneration: 5
  lastProbeTime: "2026-07-30T12:00:00Z"
  lastTransitionTime: "2026-07-30T11:55:00Z"

Example status (trust material missing):

status:
  phase: Connecting
  reason: TrustMaterialMissing
  message: "required trust material not available"
  observedGeneration: 3

Example status (gateway address not configured on remote operator):

status:
  phase: Discovered
  reason: GatewayAddressMissing
  message: "gateway address not yet available; cannot advance to Connecting"
  observedGeneration: 2

Status conditions (acceptance criterion open)

The current GridSite status uses a flat phase/reason/message model rather than a Kubernetes-style conditions[] array.

Issue #11’s acceptance criteria include a conditions-based status contract. That criterion is not yet resolved: it requires either (a) implementing a conditions[] array or (b) formally amending the issue to accept the phase/reason/message model as the replacement. Neither has happened at the time of writing.

The flat model is the current contract:

• phase provides the primary lifecycle state; reason encodes the machine-readable probe outcome; message gives a bounded human-readable explanation.
• Richer conditions (e.g. Reachable, IdentityVerified, CertificatePinned) may be added when multi-signal readiness is needed (see docs/architecture/overview.md, Trust and Readiness section).

Tools and automation must not depend on a conditions[] field existing. Match on phase for coarse state and on reason for specific probe outcomes.

InferenceProvider

Represents an inference backend available over the grid.

apiVersion: grid.praxis.fast/v1alpha1
kind: InferenceProvider
metadata:
  name: openai-api
spec:
  gridNetworkRef: production
  providerKind: open_ai          # open_ai | anthropic | bedrock | vertex | self_hosted
  backendKind: api_provider     # local | remote | cloud_managed | api_provider
  endpoint: https://api.openai.com
  models:
    - name: gpt-5-mini
      capabilities: [text_generation]
  auth:
    strategy: bearer_token      # current native path; see Auth doc
    secretRef:
      name: openai-token
      namespace: praxis-system
      key: token
  accessPolicy:
    siteSelector:
      matchLabels: {}           # empty = all sites
  siteSelector:
    matchLabels: {}

This external API example intentionally omits healthCheck because the API does not expose a suitable GET /health endpoint. It also omits metrics scraping because the external API does not provide the provider-pool metrics used by AGN scoring. The referenced openai-token Secret must exist in praxis-system before controller-managed credential projection can become available.

To probe the provider’s default {spec.endpoint}/health URL, add this to the provider spec:

healthCheck: {}

When spec.auth declares a controller-managed bearer token, the operator automatically sends it on same-origin HTTPS health probes. If the endpoint uses a private CA, also configure healthCheck.tls.caSecretRef:

healthCheck:
  tls:
    caSecretRef:
      name: model-gateway-ca
      namespace: praxis-system

The token comes from spec.auth.secretRef on each probe. The probe target must have the same scheme, host, and effective port as spec.endpoint to receive it. healthCheck.path changes the request path; healthCheck.endpoint can change the base URL within that origin. Probes to a different origin or over plain HTTP remain anonymous. Providers without controller-managed bearer auth also probe anonymously. Redirects are not followed; a 3xx response degrades health.

Phases: Pending → Available → Degraded → Unavailable

Readiness: status.conditions carries a Ready condition, written by the operator’s signals loop from each scrape of the provider’s metrics. Readiness is a condition, not a phase: phase follows the provider’s configuration only, and says nothing about whether it serves now.

The operator logs each change of reason once: at WARN when the provider turns not ready, at INFO when it returns to Ready or waits. Each scrape counts in grid_provider_scrape_total{grid_provider,result}, where result is success, no_series, or a failure class, and grid_provider_last_scrape_success_timestamp_seconds{grid_provider} holds the time of the last scrape with the ready-endpoint series.

A provider whose Ready is False is excluded from routing: its site publishes grid_provider_ready 0, and its serving config entry carries admission: none. See Polling Cross-Site Load Signals.

The READY column reads the condition’s status directly, so no status field repeats it. -o wide adds REASON, the condition’s reason, and PHASE.

NAME           PROVIDER      READY   AGE
qwen3-site-a   self_hosted   True    3d
qwen3-site-b   self_hosted   False   3d

spec.capacityWeight is an optional positive relative provider capacity from 1 through 1000, used only with GridNetwork.spec.selectionPolicy.mode: weightedRandom and placementPolicy.strategy: static. If omitted, the effective weight is 1. AGN copies this value to the overlay’s traffic_weight; it does not represent a percentage.

Backend kind

spec.backendKind describes the provider’s placement and policy category:

The value influences scoring and routing policy. It does not require a specific transport implementation; for example, a cloud_managed backend can still be fronted by Praxis.

The current CRD schema represents backendKind as a string rather than an enum. The four values above are the categories AGN recognizes for routing and scoring conversion; an unrecognized value is not converted into a normal scoring candidate. Use a listed value unless the implementation is extended.

Credential projection

spec.auth.secretRef points to a Kubernetes Secret that contains provider credential bytes. For the current native bearer_token path:

1. The operator validates that the Secret exists and contains the referenced key.

2. The routing overlay candidate receives only:

   {
     "credential": {
       "strategy": "bearer_token",
       "secretRef": {
         "name": "anthropic-token",
         "namespace": "praxis-system",
         "key": "token"
       }
     }
   }
   

3. The consumer Praxis config uses credential_inject with a file: source pointing at a mounted Secret file.

Token bytes do not appear in the overlay ConfigMap, intelligent_route candidates, filter metadata, or the consumer Praxis ConfigMap.

Metrics configuration

spec.metricsConfig configures the operator-side Prometheus scrape used during GridNetwork reconciliation. When present, the operator scrapes {spec.endpoint}{metricsConfig.path}, parses the configured signal names, and feeds the resulting BackendMetrics into overlay scoring.

Providers without metricsConfig and signals without configured metric names use neutral metric scores. For scrape failures, plaintext configuration retains the neutral-scoring compatibility behavior; TLS-configured metrics fail closed after any configured stale-sample grace period expires. See Stale metrics grace period in the routing architecture for full semantics.

Signal names

TLS and mTLS

spec.tls configures TLS for provider health checks, metrics scraping, and model discovery. The operator resolves the same CA bundle and optional client identity for each request. This keeps server verification and mTLS consistent across the provider’s control-plane checks.

The tls fields under healthCheck, metricsConfig, and modelDiscovery.openAiModels remain available as per-feature overrides for endpoints that need different trust material or a different client identity. Without an override, each feature inherits spec.tls; if neither is set, the request uses system root certificates. Configured custom CA material replaces system roots for that request, and invalid TLS material fails closed.

Set exactly one of caSecretRef and caConfigMapRef; the configured CA is the only trust used. caSecretRef follows the same SecretRef schema used by spec.auth.secretRef. caConfigMapRef takes name, namespace, and an optional key. clientCertificateSecretRef adds explicit certificateKey and privateKeyKey fields with serde defaults.

Failure behavior: when TLS material cannot be resolved or parsed, the provider phase is downgraded from Available to Degraded with a machine-readable status.reason:

Scrape-time handshake, authorization, timeout, and transport failures are classified in operator logs, not surfaced as InferenceProvider.status.reason. The last successful sample is reused only within staleMetricsSeconds; after that, a TLS-configured provider is marked unhealthy (healthy: false) and excluded from routing. Without TLS, scrape failures retain the neutral-scoring compatibility behavior.

There is no insecureSkipVerify option.

Example (one-way TLS shared across provider checks):

spec:
  tls:
    caSecretRef:
      name: provider-ca
      namespace: grid-system
  metricsConfig:
    path: /metrics
    timeout: 2s
    signalNames:
      queueDepth: vllm:num_requests_waiting

Example (mTLS):

spec:
  tls:
    caSecretRef:
      name: provider-ca
      namespace: grid-system
    clientCertificateSecretRef:
      name: provider-client-cert
      namespace: grid-system
      certificateKey: tls.crt
      privateKeyKey: tls.key
  metricsConfig:
    path: /metrics
    timeout: 2s
    signalNames:
      queueDepth: vllm:num_requests_waiting

Bearer authentication

An llm-d EPP serves /metrics behind TokenReview and SubjectAccessReview by default. metricsConfig.auth sends a credential the EPP can authorize.

Whoever runs the EPP receives the token, and it is valid against the API server, so the operator never sends its own token. The grid-operator chart creates a scraper ServiceAccount allowed only get on the nonResourceURL /metrics (rbac.metricsScraper, default true), and the operator mints a 10-minute token for it with the TokenRequest API, reusing it until two thirds of its lifetime has passed (about 400 seconds). The token is not bound to a Pod: the API server binds a token only to a Pod running as the token’s own ServiceAccount, and the operator runs as another. The token is never logged. A failed mint logs at WARN when the error changes.

A credential goes only to a host proven by the CA in spec.tls or the metricsConfig.tls override: with auth set, an https:// endpoint without either TLS setting is refused rather than trusted through the system roots, and an http:// endpoint is refused unless allowPlaintext is set.

Example (an EPP serving its metrics with the platform service CA):

spec:
  tls:
    caConfigMapRef:
      name: openshift-service-ca.crt
      namespace: grid
      key: service-ca.crt
  metricsConfig:
    endpoint: https://qwen3-epp-service.ai-tenant-site-a.svc:9090
    path: /metrics
    poolName: qwen3-inference-pool
    signalNames:
      queueDepth: llm_d_epp_average_queue_size
      kvCacheUtilization: llm_d_epp_average_kv_cache_utilization
    auth:
      type: serviceAccountToken

Queue depth normalization

AGN does not normalize raw queue counts. Exporters should publish queueDepth as a normalized 0.0 to 1.0 gauge before the operator scrapes it.

Model discovery

spec.modelDiscovery makes the operator poll the backend for the models it serves. Discovery does not yet affect routing or gossip; spec.models remains the routing source.

spec:
  endpoint: https://vllm:8000
  tls:                          # shared by health, metrics, and discovery
    caSecretRef:
      name: provider-ca
      namespace: grid-system
  metricsConfig:
    path: /metrics
  healthCheck:
    path: /health
  modelDiscovery:
    openAiModels:               # GET {endpoint}{path}, reads data[].id
      endpoint: https://vllm:8000 # optional; defaults to spec.endpoint
      path: /v1/models          # default

Use healthCheck.tls, metricsConfig.tls, or modelDiscovery.openAiModels.tls only when that endpoint needs a different CA bundle or client certificate from the shared spec.tls settings.

The effective request URL appears in status.modelDiscoveryUrl. It reflects the configured endpoint and path, regardless of whether a poll succeeds, and is absent when model discovery is not configured.

A failed poll sets status.modelDiscoveryError to the bounded failure category used by grid_model_discovery_total. A successful poll clears it. The discovery loop owns this field separately from the provider’s routing phase and clears it when discovery is disabled. The field reports the latest poll error; the held model set still follows its TTL independently.

status.servedModels is the discovered set itself: the deduplicated model names the loop last observed for this provider (the field is a set; order carries no meaning). A successful poll writes the set and its expiry boundary, including when the names are unchanged. An empty set is present and means the backend serves no models.

status.servedModelsExpiresAt is the RFC 3339 UTC deadline for that exact set, computed from the last successful poll and the discovery TTL. Readers must require this timestamp to be in the future before using servedModels. The discovery loop may not clear the status until its next round after the TTL, so the presence of servedModels alone does not prove freshness. Failed polls do not extend the deadline. When the set is unknown, expired and cleared, or discovery is disabled, both fields are absent; readers must fail closed. Writing either field advances metadata.resourceVersion, so the CRDT revision of the gossiped provider record advances with discovery status changes.

The bearer token comes from spec.auth. A model-discovery URL must use HTTPS when a bearer token is configured. With auth.credentialsManagedExternally, requests carry no credentials, so plain HTTP remains available.

Discovery refuses loopback, localhost, link-local, unspecified, and cloud metadata addresses, including the AWS, GCP, and EKS Pod Identity IPv6 endpoints. Other unique-local IPv6 addresses remain allowed. A literal blocked IP or a localhost URL is reported as config; a hostname that resolves to a blocked address is reported as unreachable.

Discovery runs in its own loop, like the signals scraper, not in reconcile. Discovered sets are held in memory per provider:

• A successful poll replaces the set and renews its deadline; an empty list is a valid result.
• A failed poll keeps the set; its deadline is not renewed.
• A set not renewed within the TTL expires in the monotonic in-memory store. Readers use servedModelsExpiresAt to reject stale status even if the next discovery round has not cleared it yet.
• A provider that is deleted or drops modelDiscovery loses its set on the next round.
• The response is rejected as a whole, never partially applied, when it:
  • is not a 2xx, or exceeds 1 MiB;
  • is not JSON with data[].id;
  • has a blank id, an id over 256 bytes, or an id with control characters;
  • repeats an id (duplicates are treated as malformed, not deduplicated);
  • lists more than 256 models.

Each poll increments grid_model_discovery_total{provider,outcome}, where outcome is one of ok, credential, tls, config, unreachable, timeout, http_status, invalid_response. Failures are also logged.

AgentToolProvider

Represents MCP tool servers available over the grid.

apiVersion: grid.praxis.fast/v1alpha1
kind: AgentToolProvider
metadata:
  name: db-tools
spec:
  gridNetworkRef: production
  protocol: mcp
  endpoint: http://db-tools.tools:8080
  tools:
    - name: database-query
      description: "Query the database"
  auth:
    strategy: bearer_token
    secretRef:
      name: tool-token
      namespace: praxis-system
  accessPolicy:
    siteSelector:
      matchLabels:
        grid.praxis.fast/site: cluster-a

Phases: Pending → Available → Unavailable

Degraded is not currently reachable for this CRD: unlike InferenceProvider’s metrics-scrape path, the MCP tools/list probe has no partial-success state to represent: it either succeeds (Available) or fails outright (Unavailable), mirroring phase_and_reason_from_probe’s and phase_from_matching’s explicit design (both are tested to never emit Degraded).

Status fields: discoveredTools (auto-populated from MCP tools/list; a failed probe preserves the previous list rather than clearing it), matchingSites, reason (machine-readable; see reason codes below), observedGeneration

AgentToolProvider reason codes

An empty reason with phase: Available means SitesMatched; an empty reason with phase: Pending means AwaitingSiteMatch; both are telemetry-only labels (grid_mcp_probe_total, Events), not persisted to status.reason itself.

AgentToAgentProvider

Represents A2A agents available over the grid.

apiVersion: grid.praxis.fast/v1alpha1
kind: AgentToAgentProvider
metadata:
  name: claims-agent
spec:
  gridNetworkRef: production
  protocol: a2a
  endpoint: http://claims-agent.agents:8080
  agentCard:
    skills: [claims-processing, document-review]
    modalities: [text]
  auth:
    strategy: mtls_only
  accessPolicy:
    siteSelector:
      matchLabels:
        grid.praxis.fast/site: cluster-a

Phases: Pending → Available → Degraded → Unavailable