Cross-site site selection

How a grid gateway serving gridServing chooses a site for each request in grid_site_route, what feeds the choice, and what you can change. The routing overlay and the consumer intelligent_route filter are a separate path, described in routing.md.

Site choice

Each site’s operator collects provider metrics, polls its peers, and relays readings under the verified site identity. The gateway polls its local operator. It learns a soft bound from the most held with nothing waiting and a hard bound from the least held at a demonstrated limit. Saturation (rho) combines in-flight load relative to that ceiling with direct KV-cache pressure.

Pressure tiers precede prefix affinity. Selection prefers queue-free sites with room, then queued sites with room, then sites not marked full. If none remain, all healthy sites are eligible. Entirely missing telemetry stays a fallback. Affinity narrows only within the selected tier.

Within either tier with room, selection uses pressure when every remaining site has fresh rho, even if capacity is unknown. With three or more sites, it draws two by capacity and takes the lower rho. With two, weights are capacity over 1 + rho, with each share bounded to 10–90 percent. Unknown capacity receives the mean known capacity, or equal weights when all capacities are unknown. Missing rho uses capacity-weighted selection. Queue-only and saturated selection retain sites tied on the least queue depth. Selection excludes unready sites. A site demoted by gateway health checks is a last resort if it remains admitted.

A site that has published and then sends nothing for the freshness window is silent. A peer is judged at the end of the operator’s last poll round that tried it, and this site at the last read of its operator, so a stalled poller or relay pauses the judgement rather than silencing every peer. A peer whose operator reports no poll rounds is not judged. While any live site serves the model, a silent site takes no request, and shedding does not wait on it. It routes again after two new samples with no full window between them, about two peer polls, so one poll through a lossy link does not bring it back. When every site is silent they take the request, and the pick counts as a fallback.

A peer candidate that publishes load and has sent nothing through a full window of rounds is silent too, so a gateway that starts during a partition does not send overflow to the partitioned site. A candidate publishes load unless it is this site’s provider with no metrics to scrape, or a peer’s provider that this operator has not received load rows for since it started. One that does not publish load is unmeasured, takes no overflow and never holds off shedding. After an operator restart during a partition, the partitioned site’s candidates are unmeasured until it returns. A provider with metricsConfig whose scrape fails still publishes load: it goes silent and is routed around, and its MetricsAvailable condition is False with the scrape error.

A site that returns from a partition takes requests again once both its signals have returned and its backend passes its health check, whichever is later. Silence clears after two new samples, about two peer polls; a mutual_tls backend is probed every 3 s and comes up after two passes. While both are down on every site, the front candidate is tried as a last resort. Requests already sent to a site when it goes down wait out the connect timeout, 2 s by default for a remote site, until grid#419 retries them on another site.

Known issue: from outside its site, the operator cannot tell a peer’s provider that has no metrics from one it cannot scrape. When every candidate that publishes load is silent, a peer candidate with no load rows takes every pick. Those picks count as reason="fallback".

With availability.shedding on, a model sheds once every healthy site serving it has been at its ceiling with work waiting for full_after_ms, and routes again once no sample from a site has shown that for room_after_ms. A shed request gets 429 with Retry-After. A model with no healthy routable site gets 503.

Inputs

These load metrics describe the llm-d EPP integration.

The gateway concludes in-flight as running plus waiting per endpoint, times ready endpoints, plus what flow control holds. The operator adds one grid_provider_config_generation row per qualified batch to scope learning to a provider revision. Fresh pressure remains usable without this row, but learning stays disabled.

The EPP exposes load samples at /metrics. Point metricsConfig.endpoint at the EPP Service. A pod address can reach a standby replica that reports nothing. The gateway keeps no count of its own requests.

Settings

Nothing needs setting. availability.shedding is the one switch, off by default. The other fields of the filter’s availability block, set from the chart’s praxisConfig.render.gridServing.siteRoute.availability, have defaults, and their table and tuning order are in routing.md. On the operator, GRID_SIGNALS_SCRAPE_INTERVAL_SECS (default 1) sets how often it scrapes its EPP, and metricsConfig.staleMetricsSeconds how long a failing scrape keeps the last readiness.

What to watch

grid_route_site_ceiling_soft, grid_route_site_ceiling_hard, grid_route_site_ceiling, grid_route_site_rho_reported, grid_route_site_rho_counted, grid_route_site_rho and grid_route_site_weight per site and cluster; grid_route_selections_total{path} for which arm each request took; grid_route_decisions_total{site,reason} for what was routed and what was refused and why; grid_route_shedding{model} while a model is shed. Every routed response names its site and cluster in x-grid-site and x-grid-backend.

The one poll every site’s load arrives through: grid_signals_poll_total{result} (ok, unreachable, unauthorized, too_large, no_date, bad_identity), grid_signals_last_success_timestamp_seconds, grid_signals_response_bytes against the gateway’s 1 MiB read ceiling (the operator holds the relay under 768 KiB and counts what it left out in grid_signals_relay_truncated_total{store,cut}), and grid_signals_ingest_dropped_total{reason} (skew, site_mismatch, cap, parse) for rows the operator served that the gateway refused. skew is a row stamped more than a second past the operator’s Date or more than a day before it. The gateway logs once when the poll starts failing and once when it recovers.

Symptoms