OpenAI Conformance

This directory tracks Praxis AI conformance against selected OpenAI API surfaces. The current scope is Conversations only.

This directory also vendors the upstream Anthropic Messages API OpenAPI document at specs/anthropic-spec.json. It is read by the automated PR review prompt (.github/prompts/automated-review.md) when a change touches Anthropic surfaces, and a registry drift check (cargo xtask check-anthropic-messages-registry, wired into make lint) verifies that the Anthropic Messages operation registry’s methods, paths, and operation IDs match the pinned document. There is no oasdiff structural comparison or capability projection for Anthropic. See Anthropic Messages Reference Spec.

Contract Sources

Conformance compares two independent truths:

• OpenAI’s complete pinned contract is checked in at specs/openai-openapi.yaml, with its immutable commit and SHA-256 recorded in specs/openai-openapi-source.json. Every area is projected semantically from this one source during the conformance run.
• Praxis behavior is declared by the Conversations operation registry. Each operation has one route, handling mode, and, when Praxis owns the payload, request and response contract bindings.

The registry generates runtime matching, capability claims, and the local implementation OpenAPI document. The report is derived output; editing it does not add support.

How It Fits Together

Pinned complete OpenAI spec -> semantic area projections ----------------+
                                                                         |
Area registry + Rust operation registry + shared contract types          |
  +-> runtime matcher -> area handlers -> focused runtime tests ----------+-> generated report
  +-> per-area implementation OpenAPI ------------> pinned oasdiff -------+
  +-> capability claims --------------------------------------------------+

The OpenAI pin is the authority for the target contract. The Rust registry and its shared contract types are the authority for what Praxis claims and implements. Conformance is the reproducible comparison between those two sources, backed by tests of the real runtime path.

Verified Conversations request behavior

The Conversations create and update request-body edge cases were checked against api.openai.com on 2026-07-27.

The pinned and current upstream OpenAPI documents omit requestBody.required for update and mark metadata nullable even though the live endpoint requires the body and rejects null metadata. The implementation document follows the verified runtime contract, and both discrepancies are recorded as explicit upstream_spec_exceptions on POST /conversations/{conversation_id} rather than declaring behavior Praxis does not implement.

Code Map

• apis/src/openai/conversations/routes.rs declares each operation once and derives runtime matching, handling modes, capability claims, and contract bindings.
• apis/src/openai/conversations/contracts.rs contains the request and response types shared by runtime behavior and generated schemas.
• apis/src/openai/conversations/openapi.rs builds the implementation OpenAPI document from the registry.
• apis/src/openai/operation.rs provides handling modes and reusable parameter, media-type, request, response, and OpenAPI generation metadata.
• apis/src/openai/conversations/filter.rs and handlers.rs execute the matched operation.
• xtask/src/openai_conformance/area.rs registers each area selector, implementation adapter, support claims, and focused runtime suite.
• xtask/src/openai_conformance/ verifies the full pinned reference, derives semantic area projections, runs oasdiff and runtime checks, and writes the report.
• docs/conformance/specs/ contains the exact complete upstream document and its immutable source manifest.
• docs/conformance/openai-conformance-report.json is the generated snapshot of coverage, owned-contract drift, and runtime verification.
• .github/workflows/openai-conformance.yaml verifies the reference and rejects stale generated artifacts in CI.

Handling Modes

Every operation has one proxy-boundary mode:

• passthrough: forward the operation without payload mutation.
• inspect: read selected fields while preserving the forwarded payload.
• transform: map between input and output contracts owned by Praxis.
• local: terminate the request and produce the response in Praxis.

Only transform and local operations enter owned-contract OpenAPI comparison. All eight current Conversations operations are local.

Rate-limit response ownership

The pinned OpenAI reference declares a 429 response on all eight Conversations operations. Its TooManyRequests contract includes an optional integer Retry-After header (minimum 1) and an ErrorResponse JSON body.

The local Conversations handlers do not produce 429 responses. Their registry therefore does not declare that response. A deployment rate limiter can reject traffic before the handler, but that does not establish the upstream response contract for the locally owned operation. In particular, the token rate limiter returns a bodyless rejection, while OpenAiErrorFormatter formats fatal proxy errors, not arbitrary filter rejections.

The eight missing 429 responses remain actionable response drift. Do not copy the upstream error schema into the implementation document without a runtime path that produces it, or exclude the status from comparison. Closing this gap requires a runtime-owned rate-limit response path and focused tests of its JSON body and optional retry header before changing the registry and regenerating this report.

Authentication ownership

The openai_conversations filter terminates matching requests but does not authenticate clients. Authentication and authorization are deployment-owned: operators must place the required security filter or trusted ingress boundary before Conversations. The generated implementation document therefore does not claim OpenAI bearer authentication.

Area projections exclude only inherited global security when an area declares it deployment-owned. Operation-specific security remains in scope. The report records this choice as inherited_security: "deployment", so the comparison boundary is explicit without hiding operation contracts.

Tooling

The report requires exactly oasdiff 1.23.0. Install the pinned release with:

go install github.com/oasdiff/oasdiff@v1.23.0

This release requires Go 1.26 or newer when installed from source.

The task rejects other versions, disables external references, flattens path-level parameters, and lets oasdiff exclude documentation fields in their OpenAPI context.

Generate the report:

cargo xtask openai-conformance \
  --output-json docs/conformance/openai-conformance-report.json

The default selects all registered areas. Use --area conversations for a focused run. Repeat --area to select multiple registered areas; --area all is equivalent to the default.

This command also runs the focused runtime contract tests recorded in the report. A failing or missing declared test makes the command fail after the JSON result has been written.

Conversation input and output item unions are derived from the same pinned document into the runtime artifact:

cargo xtask openai-conversation-item-contracts
cargo xtask openai-conversation-item-contracts --check

When Conversations is selected, conformance generation performs the check before comparing schemas, so the runtime validator and generated implementation document cannot silently drift from the pinned item union.

Reference Refresh

cargo xtask openai-conformance does not fetch upstream. It always uses the checked-in complete OpenAI reference at docs/conformance/specs/openai-openapi.yaml and verifies its provenance against docs/conformance/specs/openai-openapi-source.json before comparison, then creates area projections in memory. The report retains the reference revision and SHA-256. There is no alternate reference-path option; --openai-spec is rejected. --implementation-spec remains available to override the local implementation document for exactly one selected area, without changing the pinned reference.

Reference refresh is an independent command. To pin an intentional upstream update and replace the complete spec:

cargo xtask openai-conformance-reference --revision <40-character-commit>

To fetch the checked-in immutable revision, verify its source SHA-256, and byte-compare it with the complete vendored document:

cargo xtask openai-conformance-reference --check

Area projection keeps selected path items, path-level parameters, and the recursive closure of local component references. Inherited global security and referenced security schemes are included by default but excluded when an area declares authentication as deployment-owned via without_inherited_security(). Remote and non-component $ref values are rejected.

Reading the Report

Report schema version 3 records the complete reference once, gives every area its own projection digest, implementation source, and inherited_security ownership ("owned" when the area includes global security in its contract, "deployment" when authentication is declared as deployment-owned and excluded from comparison), and keeps three independent dimensions:

1. capability_coverage groups selected operations by handling mode and shows missing or stale support claims.
2. owned_contract_conformance compares local and transform contracts with pinned oasdiff. Operation drift and inherited area drift, such as global authentication, are reported separately.
3. runtime_verification records the focused commands, exact test evidence, and their result from this report run.

Do not collapse these dimensions into one percentage. A route can be covered while its owned schema drifts, and exact operation schemas do not erase a global security mismatch.

Use owned_contract_conformance.fixes_required as the actionable contract backlog.

Upstream specification exceptions

An area may remove an exact drift fingerprint from the actionable backlog only when authoritative or live evidence shows the pinned upstream schema is incomplete. Every exception records its operation, drift channel, exact detail, rationale, and evidence under upstream_spec_exceptions. Generation fails when a declared fingerprint no longer appears, so an upstream correction cannot be masked by a stale exception. Unrelated drift on the same operation remains active.

OpenAI Conversations responses were checked against api.openai.com on 2026-07-27. Omitted and explicitly null create metadata returned {}, while a populated string map round-tripped unchanged. Praxis therefore retains its string-map response schema and records the 12 missing upstream metadata constraints as explicit exceptions.

The same probe covered the update request body. An absent body returned 400 missing_required_parameter and null metadata returned 400 invalid_type, so the pinned upstream is incomplete: it omits requestBody.required and marks metadata nullable. Praxis follows the verified runtime contract — a required body with non-null string-map metadata — and records the seven resulting POST /conversations/{conversation_id} request drift fingerprints as explicit exceptions.

CI Enforcement

CI always runs strict capability and owned-contract conformance after the reference, runtime checks, and generated-artifact verification have passed. Strict conformance records a failed step while known failures remain.

On pull requests, the gate compares normalized failure fingerprints with the base branch’s checked-in report. Existing failures and fixes need no label. New missing operations or drift details fail the workflow unless a reviewer applies conformance-failure-acknowledged. Adding or removing that label reruns the workflow.

The label does not skip conformance. It acknowledges the exact new failure set written to the generated report. Reference integrity, runtime verification, and stale generated artifacts remain hard failures and cannot be acknowledged.

Merge queue and main runs do not have pull-request labels. They regenerate the report and acknowledge strict failure only when it is byte-identical to the checked-in report reviewed on the pull request. Any combined merge-queue change that alters conformance therefore fails until its report is updated and reviewed.

Follow-up fixes remove their fingerprints from the report without needing the label. When no failures remain, the strict gate passes and no acknowledgement step runs.

Change Workflow

When OpenAI-compatible behavior changes:

1. Update the operation registry and the real runtime behavior.
2. Update the shared request or response contract types when Praxis owns them.
3. Add mode-appropriate runtime tests.
4. Run cargo xtask openai-conformance-reference --check.
5. Regenerate openai-conformance-report.json.
6. Commit code, the generated report, and reference artifacts only when the upstream pin was intentionally changed.

Future API areas should add one data-driven path selector, operation registry adapter, generated owned-contract document, and focused runtime suite. They reuse the global OpenAI pin, semantic projector, shared operation metadata, and report pipeline rather than adding another reference artifact, scanning Rust source, or maintaining a handwritten implementation specification.

Anthropic Messages Reference Spec

specs/anthropic-spec.json is the vendored Anthropic API OpenAPI document, used as the authoritative contract when reviewing changes to the Anthropic Messages surfaces under apis/src/anthropic/. It serves two purposes: the registry drift check (cargo xtask check-anthropic-messages-registry, wired into make lint) verifies that every registered operation’s method, path, and operation ID match the pinned spec, and the automated PR review prompt cross-references request and response schemas, field names, types, enums, and optionality without a network fetch. Unlike the OpenAI spec, there is no oasdiff structural comparison, capability projection, or runtime suite wired to it.

Source and provenance

specs/anthropic-spec-source.json pins the exact upstream origin:

• Repository anthropics/anthropic-sdk-python, at an immutable 40-character commit revision.
• Path scripts/mock-spec.json.gz — the mock-server OpenAPI document bundled in that repository. encoding is gzip and source_sha256 is the digest of that upstream gzip.
• anthropic-spec.json is the verbatim gunzip of that pinned gzip, and vendored_sha256 is the digest of the decompressed JSON checked in here.

Verify the vendored file against the manifest:

shasum -a 256 docs/conformance/specs/anthropic-spec.json
# must equal vendored_sha256 in anthropic-spec-source.json

Scope

The spec is the complete upstream Anthropic API document. Today the only Anthropic path Praxis handles is /v1/messages and its subpaths (/v1/messages, /v1/messages/count_tokens, /v1/messages/batches), so that is where conformance review applies. The other stable surfaces in the spec (/v1/complete, /v1/models, /v1/files) are in scope only if a change adds Anthropic handling for them. Paths carrying a ?beta=true query and the platform or console surfaces (agents, deployments, environments, memory stores, organizations, sessions, tunnels, vaults, skills, user profiles) are always out of scope — Praxis does not implement them, so their absence in Praxis is never a finding.

Relationship to the Anthropic Messages operation registry

The Anthropic Messages operation registry in apis/src/anthropic/routes.rs transcribes its methods, paths, operation IDs, and request-body shapes from this pinned document, covering the non-beta /v1/messages surface named above. cargo xtask check-anthropic-messages-registry (wired into make lint) compares every registered operation’s method, path, and operation ID against the pinned spec and fails on divergence, so a spec refresh will flag a registry that has fallen behind.

Refresh

curl -fsSL \
  https://raw.githubusercontent.com/anthropics/anthropic-sdk-python/<revision>/scripts/mock-spec.json.gz \
  | gunzip > docs/conformance/specs/anthropic-spec.json

Then update revision, source_sha256, and vendored_sha256 in specs/anthropic-spec-source.json.