token_rate_limit

Token-denominated rate limiter: reserves an estimated cost at admission, reconciles against actual usage after the response completes. Evaluates an ordered list of rules, each with its own optional match condition, algorithm choice, and budget.

Configuration Notes

Experimental: requires the token-rate-limit-filter cargo feature, which is off by default and activates the experimental marker. This filter delivers the agreed M1/M2/M6/M7 milestone scope, but its parent proposal is not yet accepted and open questions remain (HA/clustered Redis/Valkey failure modes, and the relationship to Kuadrant’s TokenRateLimitPolicy – see ai#127). The configuration surface may change between releases.

Mirrors the rules:/match: shape from the 00121_token-rate-limiting proposal in praxis-proxy/enhancements, scoped to this milestone’s static header-value matchers, per-rule algorithm choice, configurable estimation strategies (M3, see [EstimationConfig]), M4 token-type weights (default_weights / per-rule weights), graduated soft-limit tiers (S1), and per-rule soft over-quota enforcement (ai#1241). CEL matchers remain deferred (see the module doc comment).

Assumes request identity has already been resolved upstream (this filter doesn’t authenticate callers) – a catch-all rule (no match:) reserves quota for every request that reaches it, including probes and health checks. Scope rules with explicit match: conditions, or place an identity/auth filter earlier in the pipeline. Tracked as follow-on integration work in grid#101.

Observability is group-level by rule, never by user. Metrics carry only bounded rule, algorithm, backend, error, result, capacity, phase, and outcome labels; accounting logs and optional OpenTelemetry spans likewise omit raw subject and bucket-key values. The Prometheus contract is:

• praxis_trl_requests_total{rule,result} (admitted, denied, or soft_over_quota): budget decisions only. Soft over-quota forwards are not reserved or reconciled (meter-only path does not debit the window). Requests rejected before a decision are counted by praxis_trl_unauthenticated_total (401, no trusted subject) and praxis_trl_backend_errors_total (503, fail closed) instead. Hierarchy decisions use the same metrics with rule set to hierarchy:org, hierarchy:team, or hierarchy:user. A configured rule must not use those names.

Accounting log field outcome on hard denials is one of: budget_exhausted (window/bucket capacity), key_capacity (per-rule distinct-key cap), invalid_key, or reservation_capacity. Soft over-quota forwards also use budget_exhausted. Admitted reservations use reserved.

• praxis_trl_unauthenticated_total{rule}

• praxis_trl_tokens_reserved_total{rule}

• praxis_trl_tokens_reconciled_total{rule}

• praxis_trl_tokens_refunded_total{rule}

• praxis_trl_tokens_overage_total{rule}

• praxis_trl_reservations_total{rule,result} (reconciled or orphaned)

• praxis_trl_soft_tier_activations_total{rule,capacity}

• praxis_trl_backend_errors_total{rule,backend,error}: failed reservations (the 503 path), failed reconciliation enqueues, and reconciliations abandoned after their retries. error is one of unavailable, unconfirmed, invalid_response, or configuration_mismatch.

• praxis_trl_sentinel_discovery_total{phase,result}: bounded Sentinel primary-discovery attempts. phase is initial or rediscovery; result is success, failure, or stale when generation fencing discards a result. This metric is emitted only for Sentinel topologies.

• praxis_trl_sentinel_failover_total{outcome}: generation-fenced recovery after a cached connection invalidation. outcome is recovered when a writable connection is returned or failed when the bounded discovery deadline is exhausted. recovered does not prove that Sentinel promoted a replica; it records proxy-side recovery only.

• praxis_trl_fail_open_admissions_total{rule,backend,outcome}: explicit backend.on_failure: open admissions, with outcome equal to bypassed when no mutation was dispatched or unconfirmed when a dispatched operation may have committed. No reservation handle is created for either outcome.

• praxis_trl_backend_reconciliation_total{rule,backend,result}: reconciliations completed by the shared-backend worker.

• praxis_trl_budget_remaining{rule,algorithm}

• praxis_trl_reservations_active{rule}

• praxis_trl_active_keys{rule}

Every previous praxis_ai_token_rate_limit_* name has moved to this prefix; no compatibility aliases are emitted.

budget_remaining is the remaining budget for the key of the most recent admission decision on this replica (admitted or denied), and active_keys is how many keys the backend currently retains. Both are snapshots taken as decisions happen, not continuously refreshed values. Like all Prometheus gauges they are f64 and saturate at the largest exactly representable integer (2^53 - 1).

Gauge scope depends on the backend. With the memory backend every gauge describes this process only, so aggregate replicas with sum. With a shared redis/valkey backend, reservations_active is scoped to the namespace and algorithm, so summing it over rules double-counts; aggregate with max across replicas and rules. active_keys is scoped per rule, so aggregate with max across replicas for each rule. Each replica exports the value it last observed from the shared store. budget_remaining stays per replica and per last decision on either backend: it describes whichever key that replica decided last, so max or sum across replicas says little beyond “some key had this much left”. A replica that stops seeing traffic for a rule keeps exporting its last observation until it does.

The shared redis/valkey backend requires Valkey or Redis 7.0+ (PEXPIRE NX/GT is used). Both selectors use the same backend path and are treated as equal qualification targets. The backend keeps sliding-window usage in 60 fixed sub-windows per window (one per second for windows under a minute); usage leaves the window up to one sub-window late, never early. Changing a window’s length changes its sub-window width and so starts that window’s usage from zero. On the sliding window, concurrent admissions on one key are not serialised, so they can overshoot the budget by their combined estimates for one round trip. A dispatched mutation is never replayed; asynchronous replication may still lose an acknowledged write during failover.

The initial release gate is symmetric against official Redis 8.10.2 (redis:8.10.2-alpine@sha256:3811787313eba226a2ef38658c6ccb91cd5e110edc89c37767de373120a0e5a0) and Red Hat Valkey 8.0.11 (registry.access.redhat.com/rhel10/valkey-8@sha256:5929be16ac020c4851d8dc1f5ead6d48c6da066a9a74349711186925ebd553b9). These are symmetric qualification targets, not a claim that every deployment has passed; support requires both complete product lanes. The shared token bucket, by contrast, serialises admissions per key through an optimistic transaction: one key admits at most about one request per two Redis/Valkey round trips across the whole fleet, and contention shows up first as added latency, up to backend.operation_timeout (default 500 ms), then as 503s. Use a non-global key for high-throughput token-bucket rules so the load spreads over many buckets.

During a rolling upgrade from the earlier scripted shared backend, replicas on the old and new versions keep separate state, so for one window (and until old token buckets have drained) combined admissions can reach about twice the budget. All shared-backend timestamps come from the proxy replicas’ clocks, not the datastore’s: skew between replicas can under-count usage at window edges by up to the skew, and a replica whose clock runs fast trims other replicas’ live reservations and keys from the caps early.

Sentinel HA keeps admission and reconciliation on the discovered writable primary. Healthy operations reuse the cached data connection and do not query Sentinel. Discovery and connection replacement are bounded and coalesced; a dispatched operation whose reply is lost is never replayed. Replication remains asynchronous: an acknowledged write can be lost before promotion, and a partitioned old primary can accept writes later discarded. This backend does not provide RPO=0 or strong consistency. Production deployments should use at least three Sentinels, durable primary/replica persistence, suitable min-replicas-to-write and min-replicas-max-lag, and monitor SENTINEL CKQUORUM.

Admissions, denials, reconciliations, and backend failures also emit structured records on the praxis_ai::token_rate_limit::accounting tracing target: INFO for admissions and settlements, WARN for failures. They are on by default at INFO, so every admitted or denied request produces one line in the operational log stream; keep only failures with runtime.log_overrides: {"praxis_ai::token_rate_limit::accounting": "warn"}, and separate them from other operational logs by filtering on the target field. The records contain bounded policy and token-count fields only. They are best-effort operational audit records, not a durable billing source.

Configuration

Example

filter: token_rate_limit
key:                               # optional: defaults to one shared bucket per rule
  - authenticated_subject          # global | authenticated_subject | ip | model | header: NAME
  - model                          # header first (x-model); body only if already buffered
# key:                             # IP via a forwarding header (right-most hop after trusted_hops)
#   ip:
#     header: x-forwarded-for
#     trusted_hops: 1
#     ipv6_prefix: 64
# key:                             # named header, fail-open when absent
#   header: x-tenant-id
#   missing: fallback
max_keys: 100000                   # optional: per-rule cap on distinct budget keys
backend:                           # optional: defaults to in-process state, shared by every rule
  kind: valkey                      # memory (default) | redis | valkey
  url: "${TOKEN_RATE_LIMIT_VALKEY_URL}"
  namespace: praxis:token_rate_limit
default_weights:                   # optional: omitted types default to 1.0
  input: 1.0
  output: 1.0
  cached_input: 0.1                # prompt-cache hits (token.cache_read)
  cache_write: 1.25                # prompt-cache writes (token.cache_write)
  reasoning: 0.9                   # thinking tokens (token.reasoning)
rules:
  - name: team-alpha                 # human-readable, unique per filter instance
    match:                           # optional: omit for a catch-all rule
      headers:
        x-app-id: alpha
    algorithm: sliding_window        # sliding_window | token_bucket
    window: 1h                       # sliding_window only: window duration
    capacity: 100000                 # max tokens admitted (sliding_window) or held (token_bucket)
    estimation:                      # M3: configurable estimation strategy
      strategy: max_tokens           # fixed | max_tokens | input_plus_max_tokens | model_scaled
      multiplier: 1.2                # optional safety margin (default: 1.0)
      fallback_estimate: 500         # used when max_tokens absent from request
    weights:                         # optional per-rule overlay on default_weights
      cached_input: 0.05
  - name: team-beta
    match:
      headers:
        x-app-id: beta
    algorithm: token_bucket
    capacity: 50000
    refill_rate: 50                  # token_bucket only: tokens refilled per second
    reserved_tokens: 200             # legacy: equivalent to estimation: { strategy: fixed, fallback_estimate: 200 }