Providers

The providers subsystem connects Veyyon to model APIs and normalizes their auth, request, and response formats.

Responsibility

• Maintain the catalog of supported model providers and their capabilities.
• Resolve a model slug to a provider and its ModelInfo.
• Authenticate requests with API keys, access tokens, or OAuth credentials.
• Translate between the provider-specific wire format and the engine’s protocol types.

Implementation

The provider stack lives in the @veyyon/ai package.

Key concepts

• Provider metadata: a provider’s auth mode and endpoint configuration.
• Model info: per-model capabilities such as context window and vision support.
• Auth material: resolved from API keys, access tokens, or OAuth credentials.

See Models and providers and Provider stack and bring-your-own-key for how to add your own keys and choose models.

Prompt caching

Each provider adapter also sets where the request’s cache markers go, and the shapes differ: Anthropic places up to four cache_control breakpoints, Bedrock interleaves cachePoint blocks, the OpenAI Responses path sends an explicit prompt_cache_breakpoint, and everything else caches implicitly or not at all. Two settings under Settings → Context → Prompt Cache report and optionally block on a cache the provider demonstrably did not use. The full per-provider account, including the breakpoint budget and what invalidates what, is docs/internal/prompt-caching.md.

The first-event budget

A caller declares streamFirstEventTimeoutMs. It is one attempt’s deadline, and it bounds how many stalled attempts a turn pays for: the phase before the first event ends after two. The first stall is retried, because a single connect that never produces an event is common and recoverable. A second consecutive stall is a dead endpoint, and re-spending the deadline there turned a declared 100s into minutes of silence.

The phase ends at the first of these:

• the first event arrives, after which streamIdleTimeoutMs bounds the turn;
• the phase budget is spent and the turn fails with the deadline as its reason.

utils/first-event-budget.ts in @veyyon/ai defines the shape:

Four rules keep this narrow.

A stall is bounded by the budget; a server-directed wait is bounded by the cap. A 429 or 503 carrying retry-after is the server answering and asking for a later attempt, so it is not rejected when the first-event budget is gone. Only a failure where nothing arrived at all is rejected, which makes the guard a veto predicate rather than a fence around a retry loop. The wait itself is bounded separately: maxRetryDelayMs is the longest single server-directed wait a caller sits on, DEFAULT_MAX_DELAY_MS (60s) when it declares none, and a hint above that cap surfaces the refusal instead of sleeping on it. Every retrying path reads the caller’s number: fetchWithRetry for the OpenAI-compatible family, Bedrock, Ollama and Codex, and the Anthropic client and provider ladder for their own retry-after-ms handling.

A deadline that fires ends the phase. A helper that degrades on its own timeout (GitLab Duo’s settings PUT, its project lookup, its model list, and the catalog namespace reader behind them) rethrows when the caller’s deadline is what fired. Reporting it as “nothing found” and continuing spends time nobody granted and reaches the user as a configuration remedy for a network fault.

A refusal states its remedy. 401, 404, 429 and 400 have four different answers: fix the credential, fix the route or the model id, wait, fix the request. Two shapes collapse them into one wording. A status read for a debug log and then discarded: Cursor’s Connect stream did this, so every refusal arrived as “stream ended without a turn_ended update”. A handshake that treats a refusal as one candidate’s silence and reports the remedy for having found nothing: GitLab Duo’s namespace walk did this with a rejected token.

A stream that stopped is not a stream that finished. Every dialect ends a turn with its own marker: finish_reason and [DONE], response.completed, message_stop, finishReason, done: true, messageStop, turn_ended. The end of a body without one is a transport-clean EOF that indicates nothing about the turn. Reporting a normal stop there persists whatever arrived as an answer, and the model reads it back as history on the next turn. Rejecting every such EOF fails turns that were complete, because several compatible servers do not send the marker. stopReasonForTerminallessEof in utils/terminalless-eof defines the judgement for every dialect: visible text is a stop, reasoning with no answer is a length the session can recover, a tool batch counts only when every call parsed, and anything else is an incomplete-stream failure. A provider that seeds stopReason: "stop" before the first byte and never consults this rule writes a blank turn into the session, as Bedrock and Ollama did.

Where each provider’s deadline sits

packages/ai/test/no-api-outlives-the-budget-its-caller-declared.test.ts drives every API in the union against a silent endpoint and pins the observed class per API, so a provider that stops honoring the number turns that suite red.

packages/ai/test/every-provider-refusal-names-what-to-do-about-it.test.ts drives the same fourteen against a refusing transport — 401, 404, 429 with a two-minute retry-after, and 400 — and pins the class each one surfaces, plus the invariant that no refusal echoes the api key back into its message.

packages/ai/test/a-stream-that-stops-mid-turn-is-never-reported-as-a-finished-one.test.ts drives all fourteen against a 200 that closes without a terminal marker and pins each verdict, so a dialect that starts accepting an empty stream as an answer turns red rather than shipping a blank turn.