token_rate_limit

Token-denominated rate limiter: reserves an estimated cost at admission, reconciles against actual usage after the response completes. Evaluates an ordered list of rules, each with its own optional match condition, algorithm choice, and budget.

Configuration Notes

Experimental: requires the token-rate-limit-filter cargo feature, which is off by default and activates the experimental marker. This filter delivers the agreed M1/M2/M6/M7 milestone scope, but its parent proposal is not yet accepted and open questions remain (HA/clustered-Valkey failure modes, and the relationship to Kuadrant’s TokenRateLimitPolicy – see ai#127). The configuration surface may change between releases.

Mirrors the rules:/match: shape from the 00121_token-rate-limiting proposal in praxis-proxy/enhancements, scoped to this milestone’s static header-value matchers, per-rule algorithm choice, configurable estimation strategies (M3, see [EstimationConfig]), M4 token-type weights (default_weights / per-rule weights), graduated soft-limit tiers (S1), and per-rule soft over-quota enforcement (ai#1241). CEL matchers remain deferred (see the module doc comment).

Assumes request identity has already been resolved upstream (this filter doesn’t authenticate callers) – a catch-all rule (no match:) reserves quota for every request that reaches it, including probes and health checks. Scope rules with explicit match: conditions, or place an identity/auth filter earlier in the pipeline. Tracked as follow-on integration work in grid#101.

Observability is group-level by rule, never by user. Metrics carry only bounded rule, algorithm, backend, error, result, and capacity labels; accounting logs and optional OpenTelemetry spans likewise omit raw subject and bucket-key values. The Prometheus contract is:

• praxis_trl_requests_total{rule,result} (admitted, denied, or soft_over_quota): budget decisions only. Soft over-quota forwards are not reserved or reconciled (meter-only path does not debit the window). Requests rejected before a decision are counted by praxis_trl_unauthenticated_total (401, no trusted subject) and praxis_trl_backend_errors_total (503, fail closed) instead. Hierarchy decisions use the same metrics with rule set to hierarchy:org, hierarchy:team, or hierarchy:user. A configured rule must not use those names.

Accounting log field outcome on hard denials is one of: budget_exhausted (window/bucket capacity), key_capacity (per-rule distinct-key cap), invalid_key, or reservation_capacity. Soft over-quota forwards also use budget_exhausted. Admitted reservations use reserved.

• praxis_trl_unauthenticated_total{rule}

• praxis_trl_tokens_reserved_total{rule}

• praxis_trl_tokens_reconciled_total{rule}

• praxis_trl_tokens_refunded_total{rule}

• praxis_trl_tokens_overage_total{rule}

• praxis_trl_reservations_total{rule,result} (reconciled or orphaned)

• praxis_trl_soft_tier_activations_total{rule,capacity}

• praxis_trl_backend_errors_total{rule,backend,error}: failed reservations (the 503 path), failed reconciliation enqueues, and reconciliations abandoned after their retries. error is one of unavailable, invalid_response, or configuration_mismatch.

• praxis_trl_backend_reconciliation_total{rule,backend,result}: reconciliations completed by a Valkey worker.

• praxis_trl_budget_remaining{rule,algorithm}

• praxis_trl_reservations_active{rule}

• praxis_trl_active_keys{rule}

Every previous praxis_ai_token_rate_limit_* name has moved to this prefix; no compatibility aliases are emitted.

budget_remaining is the remaining budget for the key of the most recent admission decision on this replica (admitted or denied), and active_keys is how many keys the backend currently retains. Both are snapshots taken as decisions happen, not continuously refreshed values. Like all Prometheus gauges they are f64 and saturate at the largest exactly representable integer (2^53 - 1).

Gauge scope depends on the backend. With the memory backend every gauge describes this process only, so aggregate replicas with sum. With the valkey backend, reservations_active is scoped to the namespace and algorithm, so summing it over rules double-counts; aggregate with max across replicas and rules. active_keys is scoped per rule, so aggregate with max across replicas for each rule. Each replica exports the value it last observed from the shared store. budget_remaining stays per replica and per last decision on either backend: it describes whichever key that replica decided last, so max or sum across replicas says little beyond “some key had this much left”. A replica that stops seeing traffic for a rule keeps exporting its last observation until it does.

The valkey backend requires Valkey or Redis 7.0+ (PEXPIRE NX/GT is used). The valkey backend keeps sliding-window usage in 60 fixed sub-windows per window (one per second for windows under a minute); usage leaves the window up to one sub-window late, never early. Changing a window’s length changes its sub-window width and so starts that window’s usage from zero. On the sliding window, concurrent admissions on one key are not serialised, so they can overshoot the budget by their combined estimates for one round trip. Usage written is never lost.

The valkey token bucket, by contrast, serialises admissions per key through an optimistic transaction: one key admits at most about one request per two Valkey round trips across the whole fleet, and contention shows up first as added latency, up to the 500 ms Valkey timeout, then as 503s. Use a non-global key for high-throughput token-bucket rules so the load spreads over many buckets.

During a rolling upgrade from the earlier scripted valkey backend, replicas on the old and new versions keep separate state, so for one window (and until old token buckets have drained) combined admissions can reach about twice the budget. All valkey timestamps come from the proxy replicas’ clocks, not Valkey’s: skew between replicas can under-count usage at window edges by up to the skew, and a replica whose clock runs fast trims other replicas’ live reservations and keys from the caps early.

Admissions, denials, reconciliations, and backend failures also emit structured records on the praxis_ai::token_rate_limit::accounting tracing target: INFO for admissions and settlements, WARN for failures. They are on by default at INFO, so every admitted or denied request produces one line in the operational log stream; keep only failures with runtime.log_overrides: {"praxis_ai::token_rate_limit::accounting": "warn"}, and separate them from other operational logs by filtering on the target field. The records contain bounded policy and token-count fields only. They are best-effort operational audit records, not a durable billing source.

Configuration

Example

filter: token_rate_limit
key:                               # optional: defaults to one shared bucket per rule
  - authenticated_subject          # global | authenticated_subject | ip | model | header: NAME
  - model                          # header first (x-model); body only if already buffered
# key:                             # IP via a forwarding header (right-most hop after trusted_hops)
#   ip:
#     header: x-forwarded-for
#     trusted_hops: 1
#     ipv6_prefix: 64
# key:                             # named header, fail-open when absent
#   header: x-tenant-id
#   missing: fallback
max_keys: 100000                   # optional: per-rule cap on distinct budget keys
backend:                           # optional: defaults to in-process state, shared by every rule
  kind: valkey                      # memory (default) | valkey
  url: "${TOKEN_RATE_LIMIT_VALKEY_URL}"
  namespace: praxis:token_rate_limit
default_weights:                   # optional: omitted types default to 1.0
  input: 1.0
  output: 1.0
  cached_input: 0.1                # prompt-cache hits (token.cache_read)
  cache_write: 1.25                # prompt-cache writes (token.cache_write)
  reasoning: 0.9                   # thinking tokens (token.reasoning)
rules:
  - name: team-alpha                 # human-readable, unique per filter instance
    match:                           # optional: omit for a catch-all rule
      headers:
        x-app-id: alpha
    algorithm: sliding_window        # sliding_window | token_bucket
    window: 1h                       # sliding_window only: window duration
    capacity: 100000                 # max tokens admitted (sliding_window) or held (token_bucket)
    estimation:                      # M3: configurable estimation strategy
      strategy: max_tokens           # fixed | max_tokens | input_plus_max_tokens | model_scaled
      multiplier: 1.2                # optional safety margin (default: 1.0)
      fallback_estimate: 500         # used when max_tokens absent from request
    weights:                         # optional per-rule overlay on default_weights
      cached_input: 0.05
  - name: team-beta
    match:
      headers:
        x-app-id: beta
    algorithm: token_bucket
    capacity: 50000
    refill_rate: 50                  # token_bucket only: tokens refilled per second
    reserved_tokens: 200             # legacy: equivalent to estimation: { strategy: fixed, fallback_estimate: 200 }