diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,2287 @@ +# Changelog + +All notable changes to baikai are recorded here. + +The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and +this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [Unreleased] + +## [baikai 0.6.0.0] - 2026-08-28 + +### Added + +- `baikai`: `Baikai.ThinkingLevel.parseThinkingLevel :: Text -> Maybe + ThinkingLevel` and `Baikai.Evidence.parseEvidenceStrength :: Text -> Maybe + EvidenceStrength`, each beside its renderer. Three hand-copied tables — the + evidence schema's level parser, `baikai-agent`'s KDL `effort` decoder, and its + `--require-evidence` parser — now read them instead, so a level or strength + added later cannot be added in one place and missed in three. (REV-2 G.6.) + +- `baikai`: `Baikai.Agent.AgentRunResult` exports its selectors (`provider`, + `exitCode`, `stdout`, `stderr`, `duration`). It exported neither them nor its + constructor, so a consumer without generic-lens could not read a run's exit + code at all. (REV-2 G.6.) + +- `baikai`: `Baikai.Api.normaliseApi :: Api -> Api`, which collapses a `Custom` + tag that spells a built-in API onto that constructor. The registry applies it + to the key it stores and to the tag it is asked for, so a handler registered + under `Custom "anthropic-messages"` answers a model tagged `AnthropicMessages` + and the reverse; the two used to be separate entries and dispatch depended on + which spelling the model happened to carry. Derived `Eq`/`Ord` on `Api` are + deliberately unchanged: altering them would silently rearrange every + `Map Api` a consumer holds. (REV-2 G.4.) + +- `baikai`: `Baikai.Header`, a new module exporting `HeaderName` with + `headerName` and `renderHeaderName`. See the `headers` retype under Changed. + +- `baikai`: `Baikai.Error.ErrorCategory` gains `ContentFiltered` (wire tag + `content_filtered`, never retryable) with the smart constructor + `contentFiltered`. OpenAI's `finish_reason: "content_filter"` and Anthropic's + `refusal` stop now carry it. Both used to be `OtherError`, so the only way to + tell a filtered response from any other non-retryable failure was to match on + the message text. __Breaking__ for a consumer whose `case` over + `ErrorCategory` is exhaustive without a wildcard. (REV-1 1.7 residual.) + +- `baikai` (breaking to construct, not to read): every record that can still + grow a field is now built from an exported base value and refined by record + update, and its constructor is no longer exported — + `Baikai.Provider.Registry.ApiProvider` (`apiProvider` /`apiProviderWith`), + `Baikai.Evidence.ModelCallEvidence` (`baseEvidence`), + `Baikai.Evidence.EvidenceRequest` (`evidenceRequest`), `Baikai.Tool.Tool` + (`mkTool`, with `emptyTool` kept for fixtures), + `Baikai.Embedding.EmbeddingModel` (`emptyEmbeddingModel`), + `Baikai.Cost.Log.CallLogConfig` (`callLogConfig`), + `baikai-trace-otel`'s `OtelSinkOptions` (`defaultOtelSinkOptions`), and + `baikai-agent`'s `AgentCliOptions` (`agentCliOptions`), `AgentCliRun` + (`agentCliRun`), `AgentJob` (`agentJob`) and `AgentConfigPaths` + (`emptyAgentConfigPaths`). Selectors, record update, `OverloadedRecordDot` + reads and generic-lens labels all keep working; only construction from the + constructor stops. Adding `describeThinking` to `ApiProvider` in 0.5.0.0 broke + every third-party registration site, and `strengthCeiling` would have broken + them again; from this release such an addition is a minor bump. (REV-2 G.1.) + +- `baikai`: `Baikai.Provider.apiProvider`, which builds an `ApiProvider` from an + `Api` tag and a streaming producer, deriving `complete` with + `streamingComplete`; and `Baikai.Provider.Registry.apiProviderWith`, which + takes the completer explicitly. Both default `describeThinking` to + "nothing requested, nothing translated" and `strengthCeiling` to + `EvidenceRequestedOnly`, matching `declaredStrength (Custom _)`. + +- `baikai`: `Baikai.Tool.mkTool` — a tool from its name, description and JSON + Schema. A tool built from `emptyTool` and sent unchanged reaches the wire with + `input_schema: null`; `mkTool` has no such shape. + +- `baikai`: `Baikai.Agent.AgentOutputFormat` (`TextFormat`, `JsonFormat`) with + `renderAgentOutputFormat` and `parseAgentOutputFormat`, and + `AgentRunRequest.outputFormat`, defaulting to `TextFormat`. `baikai-claude` + renders `--output-format json` and `baikai-openai` renders `--json`, both + right after the effort flags; `baikai-agent` reads it from + `jobs..output-format`. This is the one setting an evidence record needs + in order to observe a run's session, model and usage, and asking for it used + to require the `provider-args` channel that an operator ceiling closes by + default — an operator should not have to open a privileged channel to get a + record. (REV-2 F.14.) + +- `baikai`: `Baikai.Agent.AgentCeiling` gains three fields and the module gains + the vocabulary they need. `allowedTools :: [Text]` names tool grants the + operator permits beyond the ones `toolGrantsImpliedBy` (also new) says a + capability implies on its own; `maxTimeout :: Maybe NominalDiffTime` and + `maxOutputLimit :: Maybe Int` bound what any job may request, the second + defaulting to the new `defaultMaxOutputLimit` (67108864, sixty-four + mebibytes). `Baikai.Agent.ceilingViolations` is `applyAgentCeiling`'s violation + list on its own, so a caller can concatenate it with violations of its own. + (REV-2 F.3.) + +- `baikai`: `Baikai.Content.toolArgumentsFromText` and + `Baikai.Content.isCutOffToolCall`. The first is the single rule that turns a + tool call's accumulated argument text into its `arguments` value — empty text + is an empty object, non-empty text that does not decode is kept verbatim as a + `String` — and both provider assemblers and core's stream-recovery path now + use it, so the second means the same thing at every layer. + +- `baikai`: new exposed module `Baikai.Provider.Internal.StreamWorker` — the + bounded hand-off both HTTP providers now use between their SSE worker thread + and the consumer draining the stream. `FrameQueue` is a 64-slot `TBQueue` plus + a closed flag; `forkFrameWorker` closes the queue however the body ends, and + `withFrameWorker` runs the consumer under `Stream.bracketIO` so the worker is + killed when the stream stops. The module is exposed like + `Baikai.Provider.Cli.Internal`, outside the PVP promise. See + [docs/adr/0010](docs/adr/0010-a-stream-consumer-that-stops-owns-cancelling-the-producer.md). + +- `baikai`: every Anthropic model in the generated catalog now carries an + explicit `CompatAnthropicMessages` record stating the two request-shaping + facts of its generation: `AnthropicMessagesCompat.thinkingStyle` (which + extended-thinking wire shape it accepts) and the new + `AnthropicMessagesCompat.supportsSamplingParameters` (whether it accepts + `temperature`, `top_p` and `top_k`). Both are sourced from + `baikai/data/models/anthropic.json`, which the fetcher writes from its + curated `anthropicInclude` table, and `baikai-gen-models` now refuses an + `anthropic-messages` entry that reaches it without a `compat` block rather + than falling back to host auto-detection, which cannot know a generation. + This is what fixes `claude-sonnet-5`, whose thinking requests were shaped by + a prefix table that did not know the id. See + [docs/adr/0009](docs/adr/0009-provider-capability-facts-live-in-the-generated-catalog-record.md). + +- `baikai`: two new `Baikai.Evidence.ThinkingAdjustment` constructors, + `SamplingDroppedUnsupportedModel` and `SamplingDroppedUnsupportedApi`, encoding as + `{"kind":"sampling_dropped_unsupported_model","fields":["temperature","top_p"]}` and + `{"kind":"sampling_dropped_unsupported_api","fields":["seed"]}`. They record sampling + parameters removed because the model generation rejects them, or because the API has no + such field on any generation. Both carry a `fields` array and no `requested` level, so + they can appear on a call whose thinking mode is `absent`. + +- `baikai`: `Baikai.Evidence.weakensThinking`, which says whether an adjustment weakens the + thinking the caller asked for. Strict evidence mode filters through it, so a dropped + sampling parameter is recorded without refusing the call — the documented contract is + refusing a call that would weaken the requested *thinking level*. + +- `baikai`: new exposed module `Baikai.Url` — the one place baikai turns a URL + into a host name. `parseUrl` yields a `UrlParts` record with the scheme, host, + port and path, plus flags saying whether userinfo, a query string or a + fragment were present; it never holds their text, so the value cannot carry a + secret into a log line. Alongside it: `urlHost`, `hostMatchesSuffix` (moved + from `Baikai.Compat`, which now re-exports both), `renderEndpoint`, + `stripApiVersion`, and `baseUrlProblem`, which says why a URL is unusable as a + `Model.baseUrl` and what to do instead. See + [docs/adr/0008](docs/adr/0008-one-url-host-parser-and-every-consumer-uses-it.md). + +- `baikai`: new exposed module `Baikai.Provider.Transport.Classify` — the one + rule every HTTP provider uses to classify a transport failure, exporting + `classifyTransportException` plus the per-type functions it composes. The rule + is *where* the failure happened, not what type it is: anything that breaks or + ends the connection after the request went out is `TransientError`, anything + that says the request or the configuration is wrong is not retryable, and a + programming error stays `OtherError`. It understands all three shapes + `http-client` can deliver — an `HttpException` of any constructor, a raw socket + `IOException`, and a raw or wrapped `TLSException` — because the manager wraps + the connect phase but not the body reader. Core gains direct `build-depends` on + `http-types` and `tls`, both already in its install plan. Written for + third-party `Custom` providers built on `http-client` as much as for baikai's + own two. See + [docs/adr/0011](docs/adr/0011-core-owns-transport-failure-classification.md). + +- `baikai`: `Baikai.Error.parseHttpDate` and `Baikai.Error.retryAfterSecondsAt`. + The first parses an HTTP-date in the IMF-fixdate form servers must send plus + the two obsolete forms a recipient must accept; the second converts a + `Retry-After` header in either of its forms to seconds against a reference + instant, clamping a date already in the past to `0`. + `parseRetryAfterSeconds` keeps its integer-only contract, now a deliberate + division of labour rather than a limitation. + +- `baikai`: new exposed module `Baikai.Http` — `canonicalBaseUrl`, + `getClientEnvCached` and `cachedClientEnvCount`, the process-global + `ClientEnv` cache that both HTTP provider packages now share instead of each + keeping its own. Core gains direct `build-depends` on `servant-client`, + `http-client` and `http-client-tls`, which were already in its install plan + through the `openai` SDK. + +- `baikai`: `Baikai.Evidence.ThinkingModeNotTranslated`, encoded as + `"not_translated"`, and `Baikai.Evidence.untranslatedThinking`; and + `Baikai.Evidence.Build.requestedTranslation`. A path where no adapter ran to + translate the caller's level now records the level and says the translation is + unknown, instead of saying nothing was asked. (REV-2 D.2.) + +- `baikai`: `Baikai.Evidence.Build.missingEvidenceError`, + `Baikai.Evidence.Build.strictnessOf` (moved here from `Baikai.Trace`, where it + was private), `Baikai.Stream.requireEvidenceOnTerminal` and + `Baikai.Provider.Registry.requireEvidenceOnResponse`. (REV-2 D.3.) + +- `baikai`: `Baikai.Evidence.usageEnvelope`, and + `Baikai.Evidence.Build.endpointIdentityAt`, `prepareEvidenceAt` and + `minimalEvidenceAt`, which take the base URL the adapter actually resolved. + The three unsuffixed functions remain and pass the model's own field. + (REV-2 D.8, D.11.) + +- `baikai`: `Baikai.Evidence.deriveStrength`, the single rule that turns an + observed model, a provider request id and a response id into an + `EvidenceStrength`. (REV-2 D.10.) + +### Changed + +- `baikai`: catalog refresh. `claude-opus-5` joins the curated Anthropic include + set (adaptive thinking, sampling parameters rejected — the facts + `docs/plans/60-make-anthropic-thinking-style-and-sampling-support-catalog-driven.md` + said whoever curated it in would have to state), and the `gpt-5.6` family + picks up its price cut: `gpt-5.6` and `gpt-5.6-sol` to $4.00/$20.00, + `gpt-5.6-terra` to $2.00/$12.00, `gpt-5.6-luna` to $0.20/$1.20 per Mtok, cache + rates in step. `Baikai.Models.Generated` gains `anthropic_claude_opus_5` and + now carries 36 enabled models. No OpenAI id was added: the `gpt-5.6` family is + still the newest one models.dev reports that speaks + `openai-chat-completions`. + +- `baikai` (breaking): `ResponseFormat`'s `JsonSchema` carries a + `JsonSchemaFormat` record — `name`, `schema`, `strict`, exported + selector-only with the base `jsonSchemaFormat name schema` — instead of + holding the three fields directly. As fields of a sum they were partial + selectors: `name f` on a `JsonObject` crashed at runtime rather than failing to + typecheck, which contradicted the module's own documentation. + `-Wno-partial-fields` is dropped from the module. The JSON encoding is + deliberately unchanged (`{"tag":"JsonSchema","name":…,"schema":…,"strict":…}`) + and is now pinned by a test, because `Options` derives `ToJSON` through it and + at least one consumer keys a cache on the result. (REV-2 G.2.) + +- `baikai`: `Baikai.Context.appendToolResult` returns its input context + unchanged, and runs no dispatcher, when the response is error-shaped. A failed + call has no assistant turn worth replaying and no tool calls to answer; + appending its empty message put a turn into the transcript the model never + took. `runToolLoop` has always stopped on such a response — the documented + direct round trip in `docs/user/tools.md` reaches `appendToolResult` instead, + and now behaves the same way. Its Haddock also stops claiming multi-call + concurrency lives in the dispatcher: the calls are traversed in order. + (REV-2 G.7.) + +- Release metadata (REV-2 G.8): every publishable package now declares + `tested-with: GHC ==9.12.4` and ships its `CHANGELOG.md` (a symlink to the + root one, as `baikai` already did) via `extra-doc-files`, so Hackage shows a + changelog and a tested compiler for all seven. `baikai-claude` and + `baikai-openai` describe what they actually contain — four surfaces each, not + "wraps package X" — and `baikai-trace-otel`'s `streamly-core` bound is + `>=0.3 && <0.5`, matching every other package in the workspace rather than + excluding the 0.4 series the others accept. + +- `baikai` (breaking): `Options.headers` and `Model.headers` are keyed on + `Baikai.Header.HeaderName` — a newtype over a case-insensitive `CI Text` that + keeps the original spelling — instead of `Text`. A header name is + case-insensitive on the wire, so a `Map Text Text` holding both + `Authorization` and `authorization` sent whichever the assembling fold reached + last; the map now holds one entry per header and the last write wins, as a + caller writing two spellings would expect. `HeaderName` has an `IsString` + instance, so `Map.singleton "x-test" "1"` and `#headers` updates keep + compiling; the spelling given is what goes out on the wire and into JSON. + (REV-2 G.5.) + +- `baikai` (breaking): `Options.stopSequences` is `[Text]`, where empty means + "send nothing", instead of `Maybe (Vector Text)` — `Nothing` and `Just []` + were indistinguishable on the wire and only one of them could be right. Plan + 43's rule is lists for caller-side configuration and `Vector` for + provider-bound sequences; this was the one field breaking it. `Options.seed` + is `Maybe Int` rather than `Maybe Integer`: a seed is a machine integer at + every provider that accepts one, and it now sits beside + `timeoutMs :: Maybe Int`. (REV-2 G.5, R14.) + +- `baikai` (breaking): `StopReason.Aborted` is removed. Nothing produced it — + timeouts are `ErrorReason`/`TransientError`, and a consumer abort is recorded + as evidence `CallAborted` — while `responseError`, `eventsFor` and + `runToolLoop` all treated it as a *success*, so a value that reached any of + them would have been silently mishandled. Since 0.6.0.0 a stream consumer that + stops cancels the producer, so no consumer is left to receive such a terminal + either. (REV-2 B.6.) + +- `baikai`: dispatching a model whose `api` is still `emptyModel`'s + `Custom ""` says so — `No provider registered for API: ` — where the message used to end after the + colon. `emptyModel`'s Haddock says the same thing. (REV-2 G.4.) + +- `baikai`: `withTrace` and `withTraceStream` wait at most one second for the + trace sink after writing the shutdown sentinel. On expiry the worker is + abandoned — not killed, which would abort the sink's fold mid-step and lose + its end-of-stream action — the call proceeds, and one stderr line reports + `the trace sink did not confirm delivery within 1000 ms; its worker was + abandoned, and events already queued may still be delivered later`. A sink + that blocked forever used to hold the call forever and swallow the first + attempt to cancel it. A caller under `EvidenceRequired` whose sink did not + confirm delivery gets a failed call, through the same path a throwing sink + takes; `Baikai.Evidence.Build.sinkFailureError` now says "its record was not + confirmed written" rather than "not written", which is the honest claim for + an abandoned worker whose events are still queued. The synthetic terminal a + consumer's abort produces is delivered from a garbage-collection hook and is + not guaranteed before process exit; that was always true and is now stated in + `docs/user/model-call-evidence.md`, `docs/capabilities/call-tracing.md` and + the `Baikai.Trace` module documentation, with the pattern for callers who need + the record. See + [docs/adr/0015](docs/adr/0015-trace-cleanup-is-bounded-and-abort-cleanup-is-gc-eventual.md). + (REV-2 D.5, Theme 7.3.) + +- `baikai`: `Baikai.Trace.Sink.multiSink` runs each member on its own drain + thread behind its own unbounded channel, instead of folding `Fold.tee` across + the list. `Fold.tee` runs one member then the other and lets either's + exception escape, so a single throwing member stopped delivery to every + sibling for the rest of the call and skipped their end-of-stream actions — an + OpenTelemetry span paired with an unwritable file sink was opened and never + ended, and nothing was exported. The step never blocks; the final action sends + every member the sentinel, waits for every member, and reports one aggregate + failure naming each failed member by zero-based index + (`1 of 2 member sinks failed: member 0: …`). (REV-2 D.6.) + +- `baikai`: `AgentSafety.allowedTools` is documented as the __grant__ it is. + On Claude Code it renders `--allowedTools`, whose help reads "list of tool + names to allow": it pre-approves tools the permission mode would otherwise + raise a request for, and in an unattended run a request nobody answers is + denied. The old Haddock called it "optional narrowing of the provider's tool + set", which was the opposite, and `applyAgentCeiling` never looked at it. It + is now bounded: a grant passes when the maximum capability implies it + (`read-only` implies `Read`, `Glob`, `Grep`, `NotebookRead`, `TodoWrite`; + `edit-workspace` adds `Edit`, `MultiEdit`, `Write`, `NotebookEdit`; + `full-access` implies every grant) or when the operator named it in + `policy.allowed-tools`. Matching is exact, so `Bash(git *)` is not `Bash`. + A repository job that grants itself `Bash` under `edit-workspace` — which + passed unexamined before — is now refused with exit 77 before any process is + created. (REV-2 F.3.) + +- `baikai` (breaking): `Baikai.Agent.CeilingViolation` gains five constructors: + `ToolGrantForbidden`, `TimeoutExceeded`, `OutputLimitExceeded`, + `RepositoryScopeForbidden` and `WorkingDirOutsideRepository`. A `case` over + the type that was exhaustive is no longer. + +- `baikai` (behaviour): the default ceiling has a finite `maxOutputLimit`, so + `applyAgentCeiling defaultAgentCeiling` now refuses a request whose + `outputLimit` is `Nothing` — capture without bound is exactly what the + maximum exists to refuse. Jobs resolved through `baikai-agent` are unaffected: + that layer's own default supplies a finite limit, and only an explicit + `output-limit "unlimited"` reaches the ceiling as `Nothing`. + +- `baikai`: a tool call cut off by the output cap is no longer executed. + `runToolLoop` stops with the response and its tool calls intact when any call + is cut off, and `appendToolResult` appends a `ToolResultMessage` with + `isError = True` explaining why instead of calling the dispatcher. Previously + both assemblers replaced truncated arguments with `{}` and a tool loop + happily ran the call with no arguments at all. (REV-2 B.2.) + +- `baikai`: `Baikai.Model.anthropicMessagesCompatFor` no longer overlays a + thinking style guessed from the model id onto a model whose `compat` is + `CompatNone`. `CompatNone` now means host auto-detection alone — the budget + thinking shape, sampling parameters supported. Every catalog model carries an + explicit record, so this changes nothing for them; a **hand-rolled** model + naming an adaptive-era id (`claude-sonnet-5`, `claude-opus-4-7`, + `claude-opus-4-8`, `claude-fable-5`) must now carry + `CompatAnthropicMessages (defaultAnthropicMessagesCompat {thinkingStyle = AnthropicThinkingAdaptive, supportsSamplingParameters = False})` + or start from the catalog value. + +- `baikai`: `Baikai.Evidence.evidenceSchemaVersion` is now + `baikai.model-call-evidence/1.1`. A minor bump: the two sampling adjustment kinds are a + compatible addition, and no previously recorded digest changes. + +- `baikai`: HTTP 413 classifies as `ContextOverflow` rather than `OtherError`, + from the status alone and whatever the body says. 413 *is* the size-limit + status and the caller's remedy — shrink the input — is the same either way; + making the category depend on body wording would recreate for 413 the + inconsistency this release fixes for connection resets. (REV-2 A.7.) + +- `baikai`, `baikai-claude`, `baikai-openai`: an HTTP-date `Retry-After` is + converted to seconds instead of ignored. Both transports use the response's own + `Date` header as the reference instant, falling back to the local clock, so a + CDN-fronted `429` — the common case for a date-valued `Retry-After` — now + carries a hint rather than leaving the caller to guess. (REV-2 A.9.) + +- `baikai`: **breaking.** `Baikai.Embedding.EmbeddingModel.apiKey` is now + `Maybe ApiKeySource` rather than `ApiKeySource`. `Nothing` means the + conventional environment variable for the model's host, from + `defaultApiKeyEnvForBaseUrl` — the same table the chat providers use — and a + host that table does not know refuses with an `AuthError` naming + `EmbeddingModel.apiKey`. Migration: `apiKey = source` becomes + `apiKey = Just source`. `EmbeddingModel` also derives `Eq` and `Generic`, so + the `#field .~ value` idiom works on it as it does on every other record. + (REV-2 E.3.) + +- `baikai`: **breaking.** `AgentRunFailure`'s `RunTimedOut` constructor now + carries a new record `AgentTimedOut` — the configured `limit` plus the + `stdout` and `stderr` a timed-out run drained before its process group was + killed — instead of a bare `NominalDiffTime`. A caller matching + `RunTimedOut limit` becomes `RunTimedOut timedOut` and reads `timedOut ^. + #limit`; `renderAgentRunFailure` is unchanged in what it says. The bytes were + always there, drained from the moment the child was spawned, and were simply + dropped on the timeout path — which is the run an operator most wants an + account of, because the tool started, may have consumed tokens, and may + already have changed the working tree. + +- `baikai`: under `EvidenceRequired`, a successful terminal that carries no + evidence record fails the call with `missingEvidenceError` rather than + returning a silent success with zero `call_evidence` lines. Strict mode + guaranteed that a record which was built and then lost fails the call; it did + not guarantee that one was built. The rule is applied at both dispatch points, + so `completeRequest` with no sink gets the same guarantee as a streaming call; + a failed call keeps the provider's own error, and best effort is unchanged. + See `docs/adr/0014-strict-evidence-means-a-record-exists.md`. (REV-2 D.3.) + +- `baikai`: a caller's thinking level is recorded on every evidence path — the + consumer abort, an unregistered provider, a `complete` handler that threw, and + each provider's `immediateError`. The abort path asks the registered adapter's + own `describeThinking`; the others record `not_translated`. All four used to + record the caller's request as `absent`, which + `docs/adr/0002-requested-translated-observed-are-never-collapsed.md` forbids. + (REV-2 D.2.) + +- **`baikai.model-call-evidence/2.0`.** Two digests cover different bytes, so a + verifier must now select its rules by `schema_version`. `response_commitment` + covers the provider-reported token counts and never baikai's computed cost: + the cost comes from the caller's catalog rates rather than from the response, + so the digest used to change whenever a price was edited and a verifier + holding only the response could not recompute it. `request_configuration` + summarises `output_config` and `response_format` as it already summarised + `tools`, because a structured-output JSON schema carries author-written + `description` strings and is content wherever it appears — the same schema was + stripped from `tools[].input_schema` and survived verbatim through the other + two keys. `thinking.mode` may also now be `"not_translated"`, which is a + compatible addition. (REV-2 D.7, D.11.) + +- **Breaking.** `baikai`: `Baikai.Provider.Registry.ApiProvider` gains a fifth + field, `strengthCeiling :: EvidenceStrength`, and + `Baikai.Evidence.Build.checkEvidenceRequirements` takes that ceiling where it + took an `Api`. The gate compared against `declaredStrength`, a table keyed by + the API tag, which necessarily answered `EvidenceRequestedOnly` for every + `Custom` transport — so a gateway that genuinely observes a model could never + satisfy a strict caller who required that it did. Only a provider knows what + its evidence reaches. `EvidenceRequestedOnly` reproduces the old behaviour for + any custom provider; the four built-in providers fill the field from + `declaredStrength`, which is unchanged in value and still used by the + unattended-agent surface. (REV-2 D.10, G.1.) + +- `baikai`, `baikai-claude`, `baikai-openai`: one strength derivation replaces + three. An observed **response id** now counts as correlation alongside a + captured request-id header, so a host that names its model and its response id + on every chunk but sends no header reaches `model_observed` instead of + `requested_only` — which had put it *below* a host that sent only a header and + named nothing. `anthropicStrength` and `openaiStrength` are removed; + `Baikai.Provider.Cli.Internal.subprocessStrength` keeps its signature and + delegates. (REV-2 D.10.) + +### Removed + +- `baikai` **0.6.0.0** (breaking): the sixteen `_Type` base-value aliases deprecated in + 0.3.0.0 — `_Options`, `_Context`, `_Model`, `_ModelCost`, `_Response`, + `_Usage`, `_Cost`, `_CostBreakdown`, `_Tool`, `_TextContent`, + `_ThinkingContent`, `_ToolCall`, `_ImageContent`, `_EmbeddingModel`, + `_InteractiveLaunchRequest` and `_InteractiveLaunchResult`. Each has an + `empty…` or `zero…` replacement of the same value, named in the pragma that + has been on it since 0.3.0.0. The 0.3.0.0 entry said they remained "for this + release"; 0.4.0.0 and 0.5.0.0 shipped without removing them because no entry + named a version. + `docs/adr/0016-deprecated-names-are-removed-at-the-next-major.md` now fixes + the rule: a name deprecated in `A.B.0.0` is removed in `A.(B+1).0.0`, and + every pragma says so. (REV-2 G.3.) + +- `baikai` **0.6.0.0** (breaking): `Baikai.Trace.newEventId`. It has delegated to + `Baikai.Evidence.newCallId` since 0.5.0.0; call that. (REV-2 G.3.) + +- `baikai` **0.6.0.0** (breaking): `Baikai.Compat.defaultAnthropicThinkingStyle`, deprecated + earlier in this cycle. Nothing in baikai consults it — the thinking style of a + first-party Anthropic model is a field of its generated catalog record + (`Baikai.Models.Generated`); start from that value, or set + `CompatAnthropicMessages` explicitly. + +- `baikai` (breaking): `AgentRunRequest.envPassthrough` is renamed `envRequires`. + The field is a list of variables the job declares it requires, checked as a + precondition; it has never passed anything through, and the KDL key has said + `env-requires` since the setting existed. + +- `baikai` (breaking): `AgentRunFailure.OutputMalformed`, and with it + `baikai-agent`'s exit code 70 and its `internalExitCode` export. Nothing ever + constructed the constructor, and giving it a producer would have been wrong: + the runner treats the tool's output as best-effort observation and its + deliverable is the changed working tree, so a run that edited files correctly + and then printed an unparseable final line would have been reported as a + failure with its exit code and output discarded. A record's `strength` and + `unobserved` fields already say when output could not be read. (REV-2 F.13.) + +### Fixed + +- `baikai`: the terminal event and its evidence record are pushed to the trace + sink exactly once under asynchronous exceptions. The terminal path pushed the + evidence record, pushed the terminal event and only then set the + already-sent flag; an exception delivered between the last two made the + stream finaliser read the flag as unset and push a second `CallEvidence` and + an `aborted` `CallFailed` after the real `CallFinished`, so a sink saw two + records and two contradictory terminals for one call. All three writes now + run inside one `uninterruptibleMask_` with the flag first. (REV-2 D.4.) + +- `baikai`: `Baikai.Cost.Log.closeCallLog` is idempotent. The first caller + claims the handle and waits for the worker; a second returns at once instead + of blocking forever on an `MVar` the worker had already emptied — a shape + `withCallLog` makes easy to reach, since its bracket closes a handle the body + may also have closed. An `appendEntry` after the close enqueues nothing. + +- `baikai`: `reassembleResponse` is total under duplicated, late and + timestamp-less input. The first `EventStart` wins the skeleton and + `responseId` merges with `<|>`, so a later `Nothing` cannot erase an id an + earlier event supplied; events after the first terminal are ignored, so a + producer that keeps talking cannot rewrite the answer; and `latencyMs` falls + back to the reassembler's own wall clock when neither the skeleton nor the + terminal carries a provider timestamp, instead of reporting a zero that reads + as "instant". (REV-2 B.7.) + +- `baikai`: an `EmbeddingModel` pointed at a non-OpenAI host no longer sends + `OPENAI_API_KEY` to it. The default key source was that variable whatever the + base URL said, so pointing the client at DeepSeek handed DeepSeek an OpenAI + credential. It now resolves per host, and refuses an unknown one. New + `resolveEmbeddingKey` and `embeddingClientEnv` expose both decisions without + making a request. (REV-2 E.3.) + +- `baikai`: `Baikai.Embedding.embed` no longer allocates a TLS manager per call. + It used the `openai` SDK's own `getClientEnv`, which builds a fresh manager + every time; it now takes one from `Baikai.Http`'s process-global cache, the + same one the chat providers use, so an embedding call and a chat call to one + host share a connection pool. + +- `baikai`: **a credential in a header is no longer printed.** `Options.headers` + and `Model.headers` went through derived `Show` and `ToJSON` instances that + rendered every value verbatim — while `Baikai.Options`' own documentation + invites callers to put a gateway's `Authorization` header there and the + getting-started guide tells them to `print resp`, which renders the embedded + `Model`. Both types now have hand-written instances that render exactly what + the derived ones did, except that the value of a header whose name looks + credential-carrying (`authorization`, `api-key`, `apikey`, `token`, `secret`, + `cookie`, `password`, or any name ending in `-key`, case-insensitively) prints + as ``. `Baikai.Auth` exports the three pieces — `redactedMarker`, + `isCredentialHeader`, `redactHeaderValues` — so a caller can apply the same + rule to its own logging. Only the rendering changes: the field is untouched, + `Eq` is untouched, and the header is still sent as written. A JSON round trip + of a `Model` is deliberately lossy, since a serialised `Model` is exactly the + thing that should not carry a key. (REV-2 E.2.) + +- `baikai`: an API-key environment variable set to the empty string, or to + nothing but whitespace, now counts as **unset**. `ApiKeyEnv` fails with an + `AuthError` naming the variable and saying it is not set or is empty; + `ApiKeyEnvChain` skips it and continues, and reports every name when none + yields a key. Previously an empty variable resolved to an empty key, which + short-circuited a chain and produced `Authorization: Bearer ` and a provider + 401 that said nothing about the cause. A key with real content is still passed + through untrimmed. (REV-2 E.6.) + +- `baikai`: **the host parse no longer lets a base URL choose which key baikai + sends.** `urlHost` took the text after the *last* `@` anywhere in a URL, so + `https://proxy.example.com/v1?u=@api.openai.com` named the host + `api.openai.com`: `defaultApiKeyEnvForBaseUrl` resolved `OPENAI_API_KEY`, + `autoDetectOpenAICompletions` returned OpenAI's own compatibility record, and + the bearer token went to `proxy.example.com`. Anyone who could set `baseUrl` — + a `Model` decoded from JSON, a proxy override — could pick which provider's + credential to be handed. The same defect broke the benign direction: + `https://api.openai.com/v1/@x` named the host `x` and resolved no key at all. + The authority now ends at the first `/`, `?` or `#`, and userinfo is only ever + the last `@` inside it. (REV-2 A.1 / E.1.) + +- `baikai`: `Baikai.Evidence.Build.sanitizeEndpoint` was a second, separately + written parser that bounded the authority at the first `/` only, so a URL with + a query and no path recorded the wrong host. It is now `renderEndpoint <$> + parseUrl`, which also means a recorded endpoint has a lower-cased scheme and + host; the path keeps its case and trailing slash. + +- `baikai`: `parseCodexJsonlStream` assembles lines in **linear time**. It + previously unpacked every chunk into a stream of bytes and appended them one + at a time with `BS.snoc`, copying the whole accumulator per byte — quadratic + in line length, so one codex event carrying a two-million-character message + cost on the order of a trillion byte moves and in practice never finished. + Lines are now cut out of each chunk with `BS.elemIndex` and `BS.splitAt`, and + the pieces of a line that spans a chunk boundary are joined once. Behaviour is + unchanged: a non-JSON line is still skipped, and a last line without a + trailing newline is still parsed. + +- `baikai`: a Codex custom agent's instructions body renders as a TOML + **literal** multi-line string (`'''`), which interprets nothing, instead of a + basic one (`"""`), which interprets backslash escapes. As a basic string an + instruction as ordinary as "match `\d+`" made Codex refuse to load the file; + `tomllib` rejects the old output with `Unescaped '\' in a string`. A body a + literal string cannot hold — one containing three apostrophes, a bare carriage + return, or a control character other than tab and newline — falls back to a + fully escaped basic string. `tomlString`, which renders `name` and + `description`, now escapes every control character as TOML 1.0 requires + instead of only the five it happened to name. + +- Documentation: `baikai`'s Haddock no longer describes behaviour the code left + behind. The trace event's token counts are `Maybe` because a non-assistant + terminal has no usage, not because the CLI providers report nothing — since + 0.5.0.0 both carry what the tool reported. `EventStart`'s `partial` is a + message skeleton with empty content, zero usage and no stop reason; the api, + provider and model id live on the `Response`. A lifted stream's `EventStart` + carries the final usage and stop reason already filled in, because the + response is complete before the stream begins. `Baikai.CacheRetention` no + longer mentions an OpenAI Responses 24-hour bucket no code emits. System + prompts are documented as living on `Context.systemPrompt` rather than on a + `Baikai.Request` module that no longer exists, `emptyModel`'s `compat` is + described as auto-detection rather than a placeholder, tool dispatch says + calls run one at a time in order, and every reference to a plan number is + gone. (REV-2 H.4.) + +## [baikai-claude 0.6.0.0] - 2026-08-28 + +### Added + +- `baikai-claude`: `Baikai.Provider.Claude.Internal.Request` exports `planRequest`, + `SamplingPlan`, `uncappedMaxTokensFloor` and `normalizeToolCallId` as test seams. + `planThinking` and `describeThinkingFor` are now projections of `planRequest`, so the + strict gate, the request builder and the evidence record read one answer. + +### Changed + +- `baikai-claude`, `baikai-openai` (breaking): each provider's streaming + machinery moved from `Baikai.Provider.

.Api` to + `Baikai.Provider.

.Internal.Stream` — the `SseDriver` seam, `liveSseDriver`, + `

StreamWith`, `Assembler`, `emptyAssembler`, `translate`, and on the OpenAI + side `RawChunk`, `RawToolDelta`, `parseChunk`, `parseFrame`, `TagScanState`, + `scanThinkTags`, `closeOpenStream`, `RawUsage`, `parseUsage` and + `rawUsageToUsage`. `Api` now exports exactly `register`, the provider value + and the live stream function. The `.Internal` module is exposed for the test + suites and sibling packages and, like every `.Internal` module, may change in + any release without a major bump — so changing the assembler stops being a + documented break. `Shape`, `Sse` and `Transport` keep their names and gain the + same no-guarantees header. `_TagScanState` is renamed `emptyTagScanState`. + (REV-2 G.1.) + +- `baikai-claude`, `baikai-openai`: a consumer that stops reading now stops the + provider. Both packages fork their SSE worker under `Stream.bracketIO` and + hand frames through the bounded `FrameQueue` above instead of an unbounded + `Chan`. A consumer that cancels — `Ctrl-C`, `System.Timeout.timeout`, + `cancel` — releases the HTTP connection immediately; a consumer that abandons + the stream (`Stream.take 3`) stops the socket read within 64 further frames + and releases the connection at the next major garbage collection. Previously + the worker read the entire generation into memory for a consumer that would + never look at it, and the provider billed all of it. The three cleanup + strengths are stated in + [docs/adr/0010](docs/adr/0010-a-stream-consumer-that-stops-owns-cancelling-the-producer.md) + and in caller terms in `docs/user/streaming.md`. + +- `baikai-claude`: `anthropic_claude_sonnet_4_6` now sends the adaptive + thinking shape rather than `budget_tokens`. The budget shape is deprecated + for that generation; baikai sends the shape Anthropic documents as current. + +- `baikai-claude`, `baikai-openai`: **behaviour change.** `Options.timeoutMs` of + `Just n` with `n <= 0` is refused as `InvalidRequest` before the action runs, so + no connection is opened. `System.Timeout.timeout` returns immediately at zero + and runs unbounded below it, and the previous `max 0` clamp made both spellings + fail instantly as a *retryable* `TransientError` — a classification a caller's + retry loop re-issues forever for what is a configuration mistake. `Nothing` + remains the only spelling of "no bound". (REV-2 A.10.) + +- `baikai-claude`, `baikai-openai`: an evidence record's `endpoint` names the + host the call actually went to. Both adapters substitute a vendor default for + an empty `Model.baseUrl` inside `prepareCall`, so a call with a perfectly + definite destination recorded `endpoint: null`. Where no adapter ran, `null` + remains the truthful answer. (REV-2 D.8.) + +- `baikai-claude`: the `claude` dependency moves from `^>=1.4` to `^>=1.5`. + 1.5.0 adds a `Pause_Turn` constructor to `Claude.V1.Messages.StopReason`, and + `mapStopReason` matches that type with no wildcard under + `-Werror=incomplete-patterns`, so the bump forced a decision. A paused turn + maps to `Stop`: Anthropic suspends the turn mid-flight for a long-running + server-side tool and expects the caller to send the message back to continue + it, so nothing failed, and `Baikai.StopReason` has no constructor that says + "resume me". Widening that public sum is a breaking change for every consumer + who matches on it exhaustively, and it is not this bump's to make. The general + rule is + [ADR 0018](docs/adr/0018-a-provider-stop-reason-with-no-baikai-equivalent-maps-to-the-nearest-truthful-one.md): + a provider stop reason with no baikai equivalent maps to the constructor that + is truthful about whether the call failed, and the sum widens only when baikai + would behave differently for it. + +- `baikai-claude`: `Messages.StreamUsage` lost its `Generic` instance in `claude` + 1.5.0, so the `message_delta` usage is read through `OverloadedRecordDot` + rather than a generic-lens label. `Messages.max_tokens` and + `Messages.output_config` became ambiguous selectors — `Messages.Fallback` + carries both names — so the provider's tests read them through `^. #max_tokens` + and `^. #output_config` instead. + +### Removed + +- `baikai-claude`, `baikai-openai` **0.6.0.0** (breaking): the eight registration shims — + `registerWith`, `registerWithRegistry` and `registerWithRegistryAndConfig` in + both `Cli` modules, and `registerWithRegistry` in both `Api` modules. Register + the exported provider value instead: + `registerApiProvider (claudeCliProvider cfg)`, + `registerApiProviderWith reg (codexCliProvider cfg)`, + `registerApiProviderWith reg claudeMessagesProvider`. The batch-mode note that + had accumulated on `registerWith` — why `complete` stays on the direct path + rather than going through `streamingComplete` — moves to the provider value it + describes. (REV-2 G.3.) + +- `baikai-claude`, `baikai-openai`: `responseToError` and `classifyErrorText` + (and its private `classifySdkHttpText` half) from both + `.Internal.ErrorClass` modules. Neither package runs a `servant-client` client + on the chat path any more, so the `ClientError` branch was unreachable, and the + text classifiers parsed a string shape the local SSE transports stopped + producing in July. The phrase table `classifyErrorText` held survives as the + message fallback inside `classifyErrorFrame`, pinned through the entry point the + runtime actually uses. Both modules are documented as outside the PVP-stable + surface, so this is not a major bump; version bumps are recorded once, later. + +- **Breaking.** `baikai-claude`: `Baikai.Provider.Claude.Api.anthropicStrength` + and `baikai-openai`: `Baikai.Provider.OpenAI.Api.openaiStrength`, both replaced + by `Baikai.Evidence.deriveStrength`. + +### Fixed + +- `baikai-claude`, `baikai-openai`: a failure that lands while the response body + is streaming is classified as the transient failure it is. A connection reset, + a server closing the socket mid-chunk, a body shorter than its declared length + and a TLS session torn down after the handshake all now terminate the stream + with `TransientError` and `isRetryable = True`, carrying whatever text had + already been drained. Every one of them used to be `OtherError` with + `isRetryable = False`, while the identical failure at connect time was + transient — because `http-client` wraps the connect phase with the manager's + exception wrapper and the body reader with nothing that converts a socket + `IOException` or a `TLSException`, so those reached the worker raw and missed + the `HttpException` branch entirely. (REV-2 A.2.) + +- `baikai-claude`, `baikai-openai`: a transport failure mid-stream now closes + the blocks that were open when it arrived, on both providers, so a consumer + reading raw events and a consumer reassembling them see the same partial + output. Both providers built their terminal from the closed blocks alone and + silently dropped open text, thinking and tool arguments. On the Claude side + this covers `translate (Left …)`, the in-band `error` frame, and the + unexpected end of stream. (REV-2 B.3.) + +- `baikai-claude`: an SSE frame whose event `type` — or whose + `content_block_delta` `delta.type` — the SDK has no constructor for is now + skipped instead of ending the stream with a decode error. The SDK decodes both + with no unknown-tag fallback, so a new frame type from Anthropic used to be a + terminal fault. A frame of a *known* type that still fails to decode remains + one. `Baikai.Provider.Claude.Sse` exports the new `decodeFrame`. (REV-2 B.5.) + +- `baikai-claude`, `baikai-openai`: an empty `data:` heartbeat is ignored, and + on the OpenAI side `[DONE]` is compared after trailing whitespace is trimmed, + so `data: [DONE] ` and `data: [DONE]\r` end the stream rather than failing to + decode. (REV-2 A.8.) + +- `baikai-claude`: every failing stream now begins with `EventStart`. The + producer pre-seeds the start event before the first wire read, exactly as the + OpenAI producer already did, and `message_start` updates the assembler without + emitting a second one. Previously a 401, a rate limit, an in-band `error` + frame or an EOF arriving before `message_start` produced a lone `EventError`, + breaking the protocol `Baikai.Stream.Event` documents. `StartPayload.responseId` + is consequently `Nothing` on both HTTP providers; the provider's message id + rides `TerminalPayload.responseId`, which `reassembleResponse` already prefers. + (REV-2 A.4, REV-1 Theme 1.1.) + +- `baikai-claude`, `baikai-openai`: an asynchronous exception delivered to the + stream worker can no longer strand its consumer. End-of-frames is a flag set + by the worker fork's own `finally` rather than a sentinel value pushed onto + the channel, so a worker that dies without running its normal exit path still + ends the stream in an `EventError`. Previously the consumer blocked until the + runtime's deadlock detector noticed. + +- `baikai-smoke`: two keyed cases against `claude-sonnet-5` — one asking for + thinking (which is a 400 before this release) and one setting `temperature` — plus + `deepseek-chat` and `openrouter/openai/gpt-4o-mini` in `apiCases`, so the tool and + structured-output smokes run against a compatible host that is not OpenAI. + `CompatSmoke` now asserts DeepSeek honoured the output cap rather than only that it + answered, and `CacheSmoke` asserts the cached token classes cost something. + +- `baikai-claude`: a thinking request on `claude-sonnet-5` no longer 400s. It sends + `"thinking":{"type":"adaptive"}` and no `budget_tokens`, because the shape is read off + the model's catalog record rather than guessed from its id. (REV-2 C.1.) + +- `baikai-claude`: `temperature` and `top_p` are no longer sent to a model generation that + rejects them with a 400. They are omitted and the omission is recorded as + `sampling_dropped_unsupported_model` in the call's evidence. `seed`, `frequencyPenalty` + and `presencePenalty`, which the Anthropic Messages API has no field for on any + generation, are recorded as `sampling_dropped_unsupported_api`. (REV-2 C.1, C.5.) + +- `baikai-claude`: a model whose `maxOutputTokens` is `0` no longer sends + `"max_tokens":0`, which Anthropic rejects — and, with thinking set, no longer had its + whole thinking plan discarded for not fitting inside a ceiling of zero. It sends + `uncappedMaxTokensFloor` (1024, the SDK's own default) instead. An explicit + `maxTokens = Just 0` is still forwarded as written. (REV-2 C.2.) + +- `baikai-claude`: replay no longer sends an empty text block or an empty `content` array, + both of which Anthropic rejects. An empty text block is dropped; an assistant turn left + with nothing is dropped whole (it is baikai's own artifact — a block that closed with no + deltas, or only unsigned thinking, which replay already omits); a user turn left with + nothing is refused locally with a message naming the turn. (REV-2 C.3.) + +- `baikai-claude`: tool-call ids that differ only in characters the alphabet forbids, or + only past character 64, no longer normalise onto the same id and misroute a tool result. + A conforming id passes through unchanged — every id Anthropic and OpenAI actually mint + does — and any other is truncated to 51 characters and suffixed with twelve hex + characters of its SHA-256. Two `tool_use` blocks in one turn that still collide are + refused rather than sent. (REV-2 C.7.) + +- Documentation: `baikai-claude`'s and `baikai-openai`'s Haddock point at the + functions that exist. `Baikai.Compat` named + `Baikai.Provider.OpenAI.Api.mkOpenAIResponseFormat`, + `…Api.applyThinkingFormat` and `…Api.translateTextLikeDelta`; the first two + moved to `…Internal.Request` and the third is + `…Internal.Stream.scanThinkTags`. `ThinkingFormat`'s note said the six + non-native shapes all clamp through `compatibleEffort`; three do, Z.ai and + Qwen send a bare toggle, and `ThinkingFormatNone` drops the control. + `immediateError` carried two `-- |` headers where one was intended. + (REV-2 H.4.) + +- `baikai-claude`: an Anthropic call reports its thinking tokens. `Usage.reasoningTokens` + was hard-coded to `Nothing` on this provider because `claude` 1.4.0's + `Messages.Usage` had no breakdown to read; 1.5.0 adds + `output_tokens_details.thinking_tokens`, and both `message_start` and + `message_delta` now fill the field from it. `reasoningTokens` is an + informational subset of `outputTokens`, so no total and no cost moves. + +- `baikai-claude`: the prompt-side token counts survive a server-side tool run. + The final `message_delta` used to contribute only `output_tokens`, and + `inputTokens`, `cacheReadTokens` and `cacheWriteTokens` kept whatever + `message_start` had reported — which is wrong for a call whose prompt grew + mid-stream. `claude` 1.5.0 exposes those three on `Messages.StreamUsage`, and + each is now taken when present. An absent field still keeps the + `message_start` figure rather than zeroing it, so a model that sends only + `output_tokens` is accounted for exactly as before. + +## [baikai-openai 0.6.0.0] - 2026-08-28 + +### Added + +- `baikai-openai`: `Baikai.Provider.OpenAI.Internal.ErrorClass.classifyErrorFrame` + and `Baikai.Provider.OpenAI.Api.parseFrame`, which sort a decoded SSE payload + into a classified in-band error or a completion chunk. + +### Changed + +- `baikai-openai`: **breaking.** `Baikai.Provider.OpenAI.Shape`'s + `injectThinkingShape`, `describeThinkingShape`, `shapeRequestBody` and + `streamRequestBody` take a `Bool` after the compat record — whether the model + advertises reasoning support (`Model.reasoning`). A level on a `reasoning = False` + model now sends no `reasoning_effort`, `reasoning`, `thinking` or `enable_thinking` + key on any host, and records `thinking_dropped_unsupported_model` instead. The model + check runs before the host-format check. This is what stops `gpt-4o-mini` plus a + level from 400ing. (REV-2 C.4.) + +### Fixed + +- `baikai-openai`: an in-band `{"error": …}` frame on a `2xx` stream terminates + the call with the frame's own classification, status and message. Compatible + hosts (OpenRouter, DeepSeek, Together) report an upstream failure they only + learned about after committing to a `200` this way, and `parseChunk` never + looked at `error`. The pre-fix behaviour was worse than a bad category: + OpenRouter's frame carries `choices[0].finish_reason = "error"`, which mapped + to `Stop`, so the call ended as `EventDone` with `errorInfo = Nothing` — a + consumer switching on the terminal saw a *completed* call. A frame with no + `choices` beside the error ended as + `OtherError "openai stream ended without finish_reason"`. (REV-2 A.3.) + +- `baikai-openai`: reasoning that arrives after visible text closes the open + text block before opening the thinking block, so at most one of the two is + open at a time, every `_End` precedes the next `_Start`, and no `contentIndex` + is revisited after a later one. (REV-2 B.4.) + +- `baikai-openai`, `baikai-claude`: **a provider POST no longer follows + redirects.** `http-client`'s default is to follow up to ten with every header + intact, so a 3xx would have re-sent the bearer token (or `x-api-key`) to + whatever host the `Location` header named. `redirectCount` is now zero and the + 3xx is delivered as the one in-band terminal error carrying its status. Each + transport's request builder is exported as `buildRequest`, so the method, the + composed path and the redirect policy are assertable without a connection. + (REV-2 A.5 / E.4.) + +- `baikai-openai`, `baikai-claude`, `baikai`: **the base-URL convention is + stated and enforced.** `Model.baseUrl` and `EmbeddingModel.baseUrl` are the + API *root* — the host, or the prefix a host mounts the API under — because + baikai appends `/v1/chat/completions`, `/v1/messages` or `/v1/embeddings` + itself. A trailing `/v1` is accepted and removed rather than doubled, so + `https://api.deepseek.com/v1` now requests `/v1/chat/completions` instead of + `/v1/v1/chat/completions`. A base URL with no scheme, a scheme other than + `http`/`https`, credentials, a query string, a fragment, or a path that is + already an endpoint is refused as an `InvalidRequest` naming the problem — + and refused *before* a key is read, so an unusable base URL never causes a + credential to be looked up. The message renders the URL without its userinfo + or query, so it is safe to log. `docs/user/models-and-providers.md` gains a + **Base URLs** section stating all of it. (REV-2 A.6.) + +- `baikai-openai`, `baikai-claude`: the `ClientEnv` cache was duplicated in each + package and keyed on the raw base-URL text, so `https://h` and `https://h/` + were two TLS managers and two connection pools to one host. There is now one + cache, in `Baikai.Http`, keyed on the canonical rendering of the parsed base + URL. `Transport.getClientEnvCached` and `Transport.cachedClientEnvCount` are + re-exports of the core functions and keep their signatures. + +- `baikai-openai`: the Codex interactive launcher now **refuses the two approval + policies the installed CLI rejects**. `codex --help` at `codex-cli 0.149.1` + lists exactly `on-request` and `never` for `--ask-for-approval`; + `CodexApprovalUntrusted` and `CodexApprovalOnFailure` are older spellings the + CLI answers with `error: invalid value 'untrusted' for + '--ask-for-approval'`. Rendering them made a launch return `Right` carrying a + non-zero exit code — a session that ran and failed — instead of the `Left + SafetyNotExpressible` this module promises for a policy that cannot be + honoured. They are refused before any process is created, and refused rather + than quietly mapped onto `on-request`, because substituting a different + approval policy would change what the caller asked for. The constructors and + their spellings are unchanged, so code that matches on `CodexApprovalPolicy` + keeps compiling. + +## [baikai-trace-otel 0.4.0.0] - 2026-08-28 + +### Added + +- `baikai-trace-otel`: `OtelSinkOptions` derives `Generic`, so `#spanName` + resolves on it. No `Eq` or `Show`: `OpenTelemetry.Context.Context` has neither, + and an instance that ignored `parentContext` would be a lie. (REV-2 G.6.) + +- `baikai-trace-otel`: `OtelSinkOptions.parentContext :: Maybe Context`, default + `Nothing`. When set, every span the sink opens becomes a child of the span in + that context instead of a root, so a call can be nested under the caller's own + request span. It is a value fixed when the sink is built rather than an action + run per call, because the fold runs on baikai's trace worker thread where the + caller's thread-local context is invisible: capture the context on your own + thread (`ctx <- getContext`, or `Context.insertSpan mySpan Context.empty`) and + build the sink for that request. __Breaking for positional construction__ of + `OtelSinkOptions`; the documented path is a record update on + `defaultOtelSinkOptions`. (REV-2 D.9.) + +### Changed + +- `baikai-trace-otel`: the `baikai.evidence.strength` span attribute is rendered + by `Baikai.Evidence.renderEvidenceStrength`, the function the JSON encoding + uses, instead of a second spelling local to the sink that could drift from it. + +- `baikai-trace-otel`: `gen_ai.response.model` is set only by the evidence + branch, from the model the provider reported. The terminal branch set it from + the *requested* id, and since evidence is pushed before the terminal and + `addAttributes` replaces a key, that both labelled a request as an observation + on every call without evidence and overwrote the genuinely observed value on + every call with one. (REV-2 D.1.) + +## [baikai-effectful 0.4.0.0] - 2026-08-28 + +### Changed + +- `baikai-effectful` (breaking): the version is a **major** bump although this + package's own exports are unchanged. Its `baikai` bound moves to `^>=0.6.0`, + and the `Baikai` effect's three operations are typed in `Model`, `Context`, + `Options` and `Response` — every one of which baikai 0.6.0.0 changes + breakingly. A consumer therefore meets a break through this package even + though nothing in it was renamed, so the number says so rather than making + `0.3.0.4` look like a safe upgrade. + +- `baikai-effectful`: no longer depends on `streamly`. Both stanzas listed it + while every module imports only `Streamly.Data.Fold` and + `Streamly.Data.Stream`, which are `streamly-core`. (REV-2 minor.) + +## [baikai-kit 0.2.0.0] - 2026-08-28 + +### Added + +- `baikai-kit`: `Baikai.Kit.Error` with the closed `KitError` sum, its + `Exception` instance and `renderKitError`; `Baikai.Kit.Path.safeSourcePath`, + which resolves an untrusted relative source below the kit checkout and refuses + a symbolic link in any component or a canonical path outside the checkout; + `Baikai.Kit.Manifest.itemSources`/`ItemSources`, the one pure derivation of an + item's source list, and `supportedManifestVersions`; + `Baikai.Kit.Sidecar.hashEntries`; `Baikai.Kit.Repo.KitRepo`/`RepoRefresh`; + `Baikai.Kit.Install.installFrom`, `renderAvailable` and `UpdateReport`; + `Baikai.Kit.Status.StatusReport`, `UpstreamAvailability` and the now-pure + `renderStatusTable`; `Baikai.Kit.Command.runKitCommand`. `KitState` gains + `KitUpstreamRefused`, rendered `refused`. (REV-2 E.5, F.10, F.11.) + +- `baikai-kit`: `Baikai.Kit.Install.OverwritePolicy` (`KeepLocalEdits`, + `OverwriteLocalEdits`), `reinstallPresent` (the network-free half of + `updateKit`), and `PlannedWrite`/`WriteContent`/`executePlan`/`executePlanWith` + as a test seam. `SidecarMeta` gains `installedFiles` and `installedHash`, + which record what this tool wrote for one provider and the hash of exactly + those bytes; `newSidecarMeta` takes both. `kit update` gains `--force`. + (REV-2 F.12, Theme 8.2.) + +### Changed + +- **Breaking.** `baikai-kit`: every library function returns + `Either KitError a` and prints nothing; only + `Baikai.Kit.Command.runKit` prints `Error: …` and exits 1. `loadManifest`, + `loadManifestMaybe`, `installItem`, `listAvailable`, `uninstallItem`, + `updateKit` and `ensureKitRepo` change shape accordingly, `computeKitHash` + takes the kit root, a base and relative file names, `kitStatus` returns a + `StatusReport` instead of printing, and `KitUpdate`'s report is rendered by + the caller. See `docs/adr/0013-library-code-never-calls-exitfailure.md`. A + consumer that only calls `runKit` and `kitCommandParser` needs no change; one + that calls the library directly binds `Right`. (REV-2 F.11.) + +- `baikai-kit`: a kit is plain files. Install, the content hash and `kit status` + resolve every listed source through `safeSourcePath`, so a kit repository that + commits a symbolic link can no longer have a file read through it and copied + into a provider directory. `kit status` shows such an item as `refused`. + (REV-2 E.5 = F.10.) + +- `baikai-kit`: a manifest whose `version` is not 1 or 2 is refused with + `KitManifestVersionUnsupported` instead of being decoded and installed. + (REV-2 F.12.) + +- `baikai-kit`: an agent that lists several `files` installs all of them. The + first becomes the provider's agent file as before, and each remaining file + goes into a resource directory named after the agent beside it + (`//`), which uninstall removes with the agent. Only + the first file used to be installed. (REV-2 F.12.) + +- `baikai-kit`: `kit update` skips an item whose installed files no longer hash + to what its sidecar recorded, printing the `--force` invocation that would + overwrite them; `kit update --force` reinstalls anyway. Sidecars written + before this release carry no such hash and are updated without the check. + (REV-2 Theme 8.2.) + +### Removed + +- **Breaking.** `baikai-kit`: `Baikai.Kit.Path.safeUnder` (exported and unused), + `Baikai.Kit.Manifest.agentSources` (replaced by `itemSources`) and + `Baikai.Kit.Install.uninstallOutcomes` (absorbed by `uninstallItem`, which now + returns the outcomes for the caller to render). The internal `requireSafe` and + `Baikai.Kit.Status.resolveCacheOrEmpty` are gone with the exits they wrapped. + +### Fixed + +- `baikai-kit`: `kit status` with no cache and no network prints + `No kit items installed.` and exits 0. It used to exit 1: the guard around + `ensureKitRepo` caught `IOException`, which is not what `exitFailure` throws. + (REV-2 F.11.) + +- `baikai-kit`: `Baikai.Kit.Status.upstreamHash` joined the manifest `path` + without validating it, a second unsanitised join that grew after the July + hardening pass validated the first. Both now go through `itemSources` and + `safeSourcePath`. (REV-2 Theme 8.1.) + +- `baikai-kit`: an install that fails while renaming files into place now + restores what was there before, or names the paths it could not restore. + Phase two was a bare loop of renames, so a failure part-way left earlier + renames in place while the message said "no changes were made". Temporary + files are also created with `openTempFile`, so two concurrent installs of one + item no longer clobber each other's staging file, and a destination that is a + directory is refused before anything is written. (REV-2 F.12.) + +- `baikai-kit`: `Baikai.Kit.Install.stripYamlFrontmatter` normalises line + endings to LF on every branch. Input without frontmatter, and input whose + frontmatter is never closed, used to keep their `\r` characters and leak them + into the Codex agent TOML. (REV-2 Theme 8.7.) + +- `baikai-kit`: an `IOException` raised while reinstalling during `kit update` + is returned as `KitWriteFailed` instead of escaping as an uncaught exception. + (REV-2 Theme 8.4.) + +## [baikai-agent 0.2.0.0] - 2026-08-28 + +### Added + +- `baikai-agent`: three operator-only `policy` keys — `policy.allowed-tools`, + `policy.max-timeout` (a duration or `"unlimited"`) and + `policy.max-output-limit` (a byte count or `"unlimited"`) — each defaulting + from `defaultAgentCeiling`, and all six ceiling fields now printed by + `agent show` and carried in its `--json` object. + +- `baikai-agent`: `Baikai.Agent.Config.repositoryScopeViolations`, which reads + the resolution report to say which values the untrusted repository file was + not allowed to supply at all. `Baikai.Agent.Cli` concatenates its answer with + the pure ceiling's, so an operator sees one refusal naming every problem. + +### Changed + +- `baikai-agent` (breaking): `AgentConfigScope`'s constructors are + `AgentUserScope` and `AgentRepositoryScope`. `UserScope` collided with + `baikai-kit`'s `KitScope` constructor of the same name, the one clash between + two baikai-family packages. (REV-2 G.5.) + +- `baikai-agent` (breaking): a relative `working-dir` resolves against the + repository root rather than the process's own directory, so `working-dir "."` + means the checkout whichever file declared it. Resolving against the process + directory made `"."` mean two places when two documents defined one job, since + which one it was depended on which layer won. An absolute path is unchanged. + (REV-2 F.14.) + +- `baikai-agent` (breaking): every `--json` output is now built with `aeson` + rather than a hand-rolled writer, and `agent show --json` always emits one + object with the same seven keys — `job`, `outcome` (`shown`, `refused` or + `failed`), `exitCode`, `message`, `configuration`, `ceiling`, `command` — + with `null` for the parts that do not apply. Previously a refusal emitted a + different shape from a success and a document that would not parse emitted a + bare resolution report or nothing at all, so a reader had to know which + failure mode it was looking at before it could find the exit code. `run --json` + keeps its `outcome` values and `list --json` is unchanged. (REV-2 F.14.) + +- `baikai-agent` (breaking): `--run-id` or `--require-evidence` without either + `--evidence-file` or `--json` is now a usage error (64) naming both fixes. + Before, the record was built — a `--version` probe of the tool and two digests + — and then dropped. Under `--json` the record now travels in the envelope as + `evidence`, encoded by the same `ToJSON` `--evidence-file` writes. + +- `baikai-agent`: `agent show` and `agent run` no longer print another job's + unknown-key warnings, or the operator file's `policy` keys. The declaration + describes one job and the ceiling is a separate declaration, so `settei` warns + about both; neither is a mistake and a document with four jobs printed three + jobs' worth of noise on every run. A misspelled key inside the selected job + still warns, and a `policy` node in the *repository* document earns exactly one + notice saying it has no effect. `Baikai.Agent.Config` exports the two filters, + `relevantWarnings` and `repositoryPolicyNotice`. (REV-2 F.13.) + +- `baikai-agent`: an evidence record's `endpoint` resolves a relative executable + against the job's working directory before probing it, because that is what + the child execs. A job whose `executable` is `./bin/agent` previously reported + a path resolved against the parent's own directory, which does not exist. + `Baikai.Agent.Run` exports `executableForEvidence`. (REV-2 F.13.) + +- `baikai-agent`: a failed run's `error_info.message` keeps the last + `errorInfoStderrTailBytes` (4096) bytes of standard error, prefixed with how + many earlier bytes were dropped, instead of the whole captured stream — which + the output limit allows to reach four mebibytes by default. `Baikai.Agent.Run` + exports the constant. (REV-2 F.13.) + +- `baikai-agent`: `--evidence-file` stages through a uniquely named temporary + file created with `O_EXCL` beside the destination, instead of the destination + plus `.partial`. A symbolic link planted at the old, guessable name was + followed, which let an unattended run overwrite a file of the planter's + choosing. (REV-2 F.13.) + +- `baikai-agent` (breaking): an operator configuration file that lies inside the + repository root is refused with exit 78, naming the file and the root, and no + ceiling is established. The source list already refused the repository + *document*; this closes the shape where the repository supplies the *operator* + document, which both `--user-config .baikai/policy.kdl` and + `XDG_CONFIG_HOME=$PWD/.baikai` produce. `--user-config`, `XDG_CONFIG_HOME` and + `HOME` remain the operator's own inputs: the ceiling is exactly as trustworthy + as the process environment that selects it, and the guide now says so. + (REV-2 F.4.) + +- `baikai-agent` (breaking): an unrecognised key under the operator file's + `policy` node is an error rather than a warning, naming the file and every + such key. Everywhere else a forward-compatible file should not stop an older + binary; under `policy` a misspelling would silently leave the default ceiling + in force, which for the one node whose purpose is limiting authority is + indefensible. Two `AgentConfigError` constructors are added, + `CeilingFileInsideRepository` and `UnknownPolicySetting`. + +- `baikai-agent` (breaking): `AgentConfigPaths` gains `repositoryRoot`, the + directory the process runs in. `--config PATH` chooses which file supplies + repository-scope settings and does not move the root, because the root is what + confines a repository-supplied `working-dir`. + +- `baikai-agent` (breaking): a repository configuration file may no longer set + `executable` or a non-empty `extra-dirs`, and its `working-dir` must resolve — + after following symbolic links — inside the repository root. Each is refused + with exit 77 naming the setting, or naming both directories. The operator's + own file and `--set` may still set all three. `executable` turns configuration + into code execution with the operator's environment and the prompt on standard + input; `extra-dirs` inside the root adds nothing the working directory does not + already give, so the only ones a checkout would ask for are outside it. + (REV-2 F.3.) + +### Removed + +- `baikai-agent` (breaking): the `BAIKAI_AGENT_EXECUTABLE` environment binding. + An environment variable is inherited by every child process and is easy to set + by accident, and naming the program to run is the widest widening there is. + An operator whose installation is not on `PATH` writes `executable` in their + own configuration file or passes `--set`. + +### Fixed + +- `baikai-agent`: a timed-out run now **escalates to `SIGKILL`**. The runner + interrupts the child's whole process group, then terminates it, then kills it, + each of the first two stages bounded by the grace period and ended early once + the leader has been reaped and no member of the group is left. Previously the + last resort was `terminateProcess` followed by an *unbounded* wait, so a + coding agent that ignored `SIGTERM` — or a grandchild holding the output pipe + — hung the run for as long as it chose to live, with the deadline already + past. Polling the group rather than waiting on the leader alone is also what + gives a grandchild the same grace the agent gets. + +- `baikai-agent`: a timed-out run **reports the output it drained**. `baikai + agent run` prints it under the same stream discipline a finished run gets, so + `response=$(baikai agent run job)` under `capture` receives the partial answer + with `$?` set to 75, and `--json`'s failure envelope carries the same + `stdout`, `stdoutTruncated`, `stderr` and `stderrTruncated` fields. A drain + interrupted because something outside the process group still held the pipe + open keeps its bytes too, reported as truncated. + +- `baikai-agent`: the `baikai` command writes its output as **UTF-8 bytes** + rather than through the locale encoding. Where an unattended run actually + happens — cron, a systemd unit, a container — the environment says `LANG=C`, + and on a platform whose locale encoding follows it a single accented character + in the agent's answer made the write throw after the run had already finished: + exit 1, answer lost. This mirrors what the prompt read and the prompt write + have always done. + +- `baikai-agent`: the `baikai` executable now links the **threaded runtime** + (`ghc-options: -threaded` on the `executable baikai` stanza). Without it a + blocking operating-system call — the `waitpid` inside + `System.Process.waitForProcess` — stopped every Haskell thread in the + installed binary, so a job's configured `timeout` could never fire and a + coding agent that wrote more than one pipe buffer deadlocked against the + runner's drain threads. Both defects existed only in the shipped executable: + the test suite was already compiled `-threaded`, so every runner test passed + under a runtime the binary did not have. + + The suite now proves the runtime the binary ships with rather than its own. + `baikai-agent/test/BinaryTests.hs` spawns the built executable — cabal builds + it first and puts it on the suite's `PATH` through + `build-tool-depends: baikai-agent:baikai` — asserts that `baikai +RTS --info` + reports `rts_thr`, and runs `baikai agent run` against a stub agent that + outlives its deadline, requiring exit 75 within seconds and the whole process + group gone. See + [docs/adr/0006](docs/adr/0006-a-process-spawning-executable-ships-on-the-threaded-runtime.md). + +## [baikai 0.5.0.0] - 2026-08-05 + +### Added + +- `baikai`: new exposed module `Baikai.Agent`, the provider-neutral vocabulary + for an **unattended coding-agent run** — a run with no terminal and no human, + which owns its own tool loop, may change files inside directories the caller + authorized, and returns a process result rather than a `Response`. It defines + `AgentRunRequest` (with a required `workingDir`), `AgentRunResult`, the + `AgentCapability` profile (`read-only`, `edit-workspace`, `full-access`), + `AgentSafety`, the `AgentOutputMode` and `AgentCapturedOutput` output + discipline, the `AgentCommand` renderer/runner boundary with an explicit + prompt transport, and the `AgentRenderError` / `AgentRunFailure` taxonomies. + +- `baikai`: the operator policy ceiling — `AgentCeiling`, + `defaultAgentCeiling`, `CeilingViolation`, and the pure `applyAgentCeiling`. + It returns a request unchanged when it is within the ceiling and reports + every violation when it is not; it never clamps an over-broad request to the + permitted value. The default ceiling permits read-only and edit-workspace + authority and refuses full access and raw provider arguments. + + `Baikai.Agent` itself is vocabulary and pure policy algebra only: it spawns no + process and renders no command-line flags. Those live in the vendor packages + and in `baikai-agent`, below. The module is deliberately not re-exported from + the umbrella `Baikai` module, because its field accessors share names with + `Baikai.Interactive`, so `import Baikai` continues to compile unchanged. + +- `baikai`: new exposed module `Baikai.Evidence`, the vocabulary for + **verifiable model-call evidence** — a record of what actually crossed the + boundary to a provider, as opposed to what the process was configured to ask + for. It defines `ModelCallEvidence` and the `evidenceSchemaVersion` string + consumers pin against, `Observed` (a deliberate non-`Maybe` for a value the + provider either did or did not report, with no function that supplies a + default), `ThinkingTranslation` with its `ThinkingMode` and + `ThinkingAdjustment` enumerations describing what a requested + reasoning-effort level actually became on the wire and every clamp, collapse, + or drop applied on the way, `EndpointIdentity` and `TransportKind`, + `CallStatus`, and the ascending `EvidenceStrength` scale. + + It also provides the canonical hashing core: `canonicalEncode` gives a JSON + value exactly one byte representation (object keys sorted, no insignificant + whitespace, numbers normalised so `1`, `1.0`, `1.00`, and `1e0` all encode as + `1`, and a hand-written string escaper so an aeson upgrade cannot silently + invalidate a recorded digest); `commitmentDigest` hashes a full request + envelope, and `configurationDigest` hashes an allow-list projection + (`configurationProjection`) that keeps configuration and replaces content with + structural summaries, so two calls that ask the same model the same way about + different subjects agree. The two digests are separate on purpose: the first + binds a record to a particular request, the second is safe to compare across + runs that legitimately differ in content. + + Nothing constructs a `ModelCallEvidence` from a real call yet, and no existing + behaviour changed. New dependencies: `cryptohash-sha256` and + `base16-bytestring`, both single-purpose packages chosen over a full + cryptographic framework. + +- `baikai`: `Options` gains an `evidence` field carrying an optional + `EvidenceRequest` — the caller's run identifier, retry provenance, and how + strictly they need evidence. A call whose `evidence` is `Nothing`, which is + every call that does not opt in, behaves exactly as it did before: no digest + is computed and no evidence is emitted. + +- (Entry added 2026-08-27; the behaviour shipped in 0.5.0.0.) `baikai`: **strict + evidence mode**. `EvidenceStrictness` is `EvidenceBestEffort` or + `EvidenceRequired !EvidenceStrength`, and a caller who asks for the second + gets a call that **refuses to start** — before any request is built or any + connection opened — when the configuration cannot reach the strength asked + for: `Baikai.Evidence.Build.checkEvidenceRequirements` compares the + requirement against what the provider can deliver and against the thinking + translation, and `completeRequest` / `streamRequest` return an error-shaped + response or a terminal `EventError` instead of dispatching. The gate is + pre-dispatch by design; that is the only point at which refusing is still + free. + +- (Entry added 2026-08-27; the behaviour shipped in 0.5.0.0.) `baikai`: + **sink-failure semantics under strict mode**. `Baikai.Evidence.Build` + exports `onSinkFailure`, `sinkFailureIsFatal` and `sinkFailureError`: a trace + sink that throws fails an `EvidenceRequired` caller's call, because a record + the sink did not confirm written is not a record, while a best-effort caller's + call succeeds with the failure reported on stderr. + +- (Entry added 2026-08-27; the behaviour shipped in 0.5.0.0.) **Breaking.** + `baikai`: `Baikai.Provider.Registry.ApiProvider` gained a fourth field, + `describeThinking :: Model -> Options -> ThinkingTranslation`, which the + pre-dispatch strictness gate calls to learn what a provider would do with the + caller's reasoning-effort request without sending anything. Every third-party + provider constructed with the `ApiProvider` constructor stopped compiling. + This was not recorded at the time; it is the defect that made 0.6.0.0 hide the + constructor behind `apiProvider` so that the next field addition is a minor + release. + +- `baikai`: model-call evidence is now **produced and emitted**. A caller who + sets `Options.evidence` gets exactly one `call_evidence` line per call from + their trace sink, under every way a call can end: success, provider failure, a + consumer that abandons the stream (status `aborted`, not `failed` — an abort + is the consumer's doing and reporting it as a provider failure would + misattribute it), and dispatch that found no registered handler. + + New exposed module `Baikai.Evidence.Build` bridges the vocabulary to the + `Model` and `Options` records: `minimalEvidence` and `prepareEvidence` build a + record, `dispatchEnvelope` supplies the request envelope for the paths where + no adapter ran, `sanitizeEndpoint` reduces a base URL to scheme/host/port/path + with the query string and any userinfo dropped wholesale, and `onSinkFailure` + is the hook a future release replaces to make a strict caller's call fail when + the trace sink does. + + Every record this release produces has `strength` `requested_only` and every + provider-observed field set to `"unobserved"`. That is not a placeholder: it + is a truthful record for a transport that has not yet been taught to observe + anything. Later releases teach each transport to observe more. + + (Correction added 2026-08-27: the two paragraphs above describe the release + inaccurately and are kept as shipped rather than rewritten. `onSinkFailure` + did not await a future release — it shipped in 0.5.0.0 together with + `sinkFailureIsFatal` and `sinkFailureError`, which already fail a strict + caller's call when the sink throws. And not every 0.5.0.0 record has `strength` + `requested_only`: the provider entries below describe what each transport + reports, and the HTTP adapters reach `correlated` and `model_observed`.) + + **A caller who does not opt in pays nothing.** With `Options.evidence` absent + no digest is computed, no call identifier is generated, no evidence event is + emitted, and the request envelope is never even forced — the gate lives inside + the shared builder rather than at each adapter's call site, and the envelope + parameter is deliberately lazy. Both facts are guarded by tests. + +- `baikai`: `TraceEvent` gains a `CallEvidence` constructor, encoded as + `{"kind":"call_evidence", …}`. A consumer whose pattern match over `TraceEvent` + is exhaustive must add a branch; one with a wildcard is unaffected. Filter for + it with `jq 'select(.kind == "call_evidence") | .evidence'`. Note that a trace + line carries its fields alongside the `kind` discriminator rather than nested + under a `data` key, and that the evidence record inside spells its own fields + in snake_case — the two encodings differ deliberately, because an evidence + record must render an absent field as explicit `null` while a trace line drops + it to stay small. + +- `baikai`: `Baikai.Provider.Cli.Internal` — the module the two subprocess + providers share — gains the vocabulary for reading what a coding-agent CLI + reported about its own run. `CodexRunReport` and the new + `parseCodexJsonlStream :: Stream IO ByteString -> IO CodexRunReport` fold the + `codex exec --json` event stream into its assistant text, its thread + identifier, and its token counts, instead of concatenating agent-message text + and discarding everything else. `ClaudeCliReport` and + `decodeClaudeCliResult` do the same for `claude -p --output-format json`. + Every field but the message text is optional, because both tools' event + schemas have changed across versions and an absent field is a genuine absence + rather than a parse failure. **Breaking** for anyone calling + `parseCodexJsonlStream` directly: its result type is no longer `Text`. This is + an internal module and is documented as outside the PVP guarantee. + +- `baikai`: `Baikai.Provider.Cli.Internal` also gains `ExecutableIdentity` and + `executableIdentity`, which resolve a configured executable name to an + absolute path and read the tool's own `--version` line. The probe is cached + per resolved name for the lifetime of the process, because spawning it per + model call would roughly double the process cost of the cheapest possible + call, and it is bounded by a five-second timeout so a tool that hangs on + `--version` cannot wedge a model call. (Corrected 2026-08-27: the entry said + two seconds; `versionProbeMicros` has always been five.) A probe that fails records the version + as absent rather than failing the call. It is only ever called from inside + the evidence branch: a caller who asked for no evidence must not pay for a + process whose only purpose is to describe a tool they were about to run + anyway. + +- `baikai`: `subprocessStrength` and `cliResponseEnvelope`, also in + `Baikai.Provider.Cli.Internal`. The former derives a subprocess call's + evidence strength from what the tool reported and **nothing else** — the exit + status is deliberately not one of its arguments. The latter spells the + response-commitment envelope with the same three keys, in the same shapes, as + the two API transports build by hand, so a verifier holding a response can + recompute the digest without first knowing which transport served it. + +- `baikai`: `Baikai.Agent` gains `AgentRunOutcome` and `agentRunOutcome`. It + pairs what an unattended run did — the existing + `Either AgentRunFailure AgentRunResult` — with the evidence the runner built + for it. The evidence is a sibling of the outcome rather than a field on + `AgentRunResult` because the run that most needs a record is one that did not + produce a result: a run killed by its own timeout reports + `Left (RunTimedOut …)`, so a record hanging off the `Right` would be + unreachable exactly there. + +### Fixed + +- `baikai`: a `call_evidence` event is now emitted **before** its call's + terminal `call_finished` or `call_failed`, rather than after. The + OpenTelemetry sink ends and removes a call's span on the terminal, so under + the old order its evidence-attribute branch was unreachable from any real + call and every backend saw a span with no evidence on it — nothing failed, + the attributes were simply never there. No consumer can have depended on the + old order, because no consumer has ever seen a `call_evidence` line. + +- `baikai`: the `ThinkingFormatOpenAI` Haddock in `Baikai.Compat` listed the + native `reasoning_effort` vocabulary as `minimal | low | medium | high`, which + predates `xhigh` and `max`. It now lists all six and states that this shape + alone sends the canonical baikai level verbatim while the other six clamp + through `compatibleEffort`. No behaviour changed: the native path's exclusion + from that clamp is deliberate and is guarded by two named tests in + `baikai-openai/test/ShapeSpec.hs`. A reader who consulted the comment to + decide whether `xhigh` was safe to use against OpenAI has until now been told + something untrue. + +### Changed + +- **Breaking:** `baikai`: `TerminalPayload` gains an `evidence` field and the two + terminal smart constructors take it as their new first argument: + `doneTerminal :: Maybe ModelCallEvidence -> Maybe Text -> StopReason -> Message -> TerminalPayload` + and `errorTerminal` likewise. `Response` gains the same field. A custom + provider implementation must pass `Nothing` (or a record it builds through + `Baikai.Evidence.Build`); a custom `Response` built with the record + constructor must add `evidence = Nothing`. Code that only pattern-matches on + these types is unaffected. + +- **Breaking:** `baikai`: `CallFinished` gains `cachedInputTokens`, + `cacheWriteTokens`, `reasoningTokens`, and `totalTokens`. The trace path used + to drop counts that `Baikai.Cost.Log.CallLogEntry` kept from the same `Usage` + value, which made the cost log strictly more faithful than the trace. + +- **Breaking:** `baikai`: a computed cost of **zero is now reported as zero** + rather than suppressed, in `CallFinished` and at all three `CallLogEntry` + construction sites. Previously `usd` was omitted whenever the cost came out at + zero, so "this call was free" and "baikai could not price this call" were + indistinguishable — and the subscription-based CLI providers always price at + zero, so that was the common case rather than a corner. **A cost dashboard + that treated an absent `usd` as "unpriced" will now count those calls as + costing zero.** That is the correct reading, but it changes what such a + dashboard shows. + +- **Breaking:** `baikai`: `FromJSON TraceEvent` is written out by hand instead of + derived. The three pre-existing kinds decode exactly as before; a + `call_evidence` line fails to parse with a message saying to read it as a + plain `Data.Aeson.Value`. `ModelCallEvidence` has no `FromJSON` on purpose — + it embeds a `Cost` whose exact `Rational` amounts encode through an + approximating `Scientific`, so a decoder would return a different value than + was encoded — and manufacturing that fidelity would be the precise failure + this vocabulary exists to eliminate. + +- `baikai`: `Baikai.Trace.Sink.renderHuman` renders a `CallEvidence` event as a + single `EVIDENCE run=… call=… strength=…` line rather than the whole record. A + human-readable sink is for watching calls go by; the full record is meant to + be read out of `fileSink` output by a machine. + +- `baikai`: call identifiers on the trace path are now globally unique. + `Baikai.Evidence.newCallId` produces 32 lowercase hexadecimal characters + carrying 128 bits — 48 bits of Unix time in milliseconds, 48 bits of a + per-process random seed drawn once from `/dev/urandom`, and a 32-bit counter. + The previous generator combined the process-start *second* with a + process-local counter into 16 characters, so two processes started within the + same second emitted identical identifier sequences; its own documentation + claimed only per-process uniqueness. Identifiers still sort chronologically + and are still not secrets. + + `Baikai.Trace.newEventId` keeps its name and signature, delegates to + `newCallId`, and is now deprecated. Anything that pinned the 16-character + width — a log parser, a fixture, a column type — must widen to 32. + +- `baikai`: `renderCeilingViolation` no longer prints the raw provider arguments + a `ProviderArgsForbidden` violation carries. It reports how many were + requested and states that their values are not shown. Raw provider arguments + are the one part of a job description that can hold a credential — the + configuration layer classifies the setting secret for that reason — and a + refusal message that quoted them defeated the classification. The constructor + keeps its `[Text]` payload so a programmatic caller can still inspect it. + +## [baikai-claude 0.5.0.0] - 2026-08-05 + +### Added + +- `baikai-claude`: new exposed module `Baikai.Provider.Claude.Agent` with + `ClaudeAgentConfig`, `defaultClaudeAgentConfig`, and `claudeAgentCommand`, a + pure renderer from an unattended `AgentRunRequest` to the `claude` argument + vector. It maps the capability profile onto `--permission-mode` + (`plan` / `acceptEdits` / `bypassPermissions`), joins a tool allow-list into + one `--allowedTools` argument, repeats `--add-dir` per extra directory, always + emits `-p`, and emits `--no-session-persistence` unless `persistSession` is + set. The prompt travels on standard input and appears nowhere in the argument + vector. A request naming a different provider is refused with + `ProviderMismatch`. Nothing is spawned. + +- `baikai-claude`: the Anthropic Messages provider now fills in the evidence + record it previously left blank. It records the model **Anthropic reported + running** (read from the `message_start` event, which the adapter already + decoded for the response id and then discarded), Anthropic's `request-id` + correlation header, the response id, the token counts Anthropic actually + reported, and a commitment digest over the assembled response. A field the + provider did not report stays `"unobserved"` and is never backfilled from the + request — in particular, a stream that fails before `message_start` reports no + observed model at all. `strength` is `model_observed` when both the model and a + correlation identifier arrived, `correlated` when only the identifier did, and + `requested_only` otherwise; a 2xx status never raises it, because a 200 means + the request was accepted, not that any particular model ran. + `fully_observed` is unreachable on this transport, since Anthropic does not + echo the thinking configuration it applied. + +- `baikai-claude`: an evidence record's `thinking` field now describes what the + caller's reasoning-effort preference actually became on the wire, including + three downgrades that were previously invisible everywhere in baikai's output: + asking for thinking on a model that does not advertise `reasoning` + (`thinking_dropped_unsupported_model`); asking for a level whose token budget + does not fit under the resolved output-token ceiling + (`thinking_dropped_budget_exceeded`, carrying both colliding numbers), which is + reachable by lowering `maxTokens` alone; and asking for `high` on an + adaptive-thinking model, which sends no effort field and so is + wire-indistinguishable from taking Anthropic's default depth + (`effort_omitted`). `minimal` on an adaptive model reports `effort_clamped`, + because Anthropic's adaptive vocabulary has no `minimal`. + +- `baikai-claude`: new exports from `Baikai.Provider.Claude.Sse` — + `ResponseMetadata` and `capturedHeaderNames` — and from + `Baikai.Provider.Claude.Api` — `claudeMessagesStreamWith`, `SseDriver`, and + `anthropicStrength`. Response-header capture is an **allow-list** + (`request-id`, `x-request-id`, `cf-ray`, in that preference order), not a + denylist, so a header a future gateway adds is not recorded by default. + +- `baikai-claude` and `baikai-openai`: both subprocess providers now fill in the + evidence record they previously left blank, and both export the translation + function that describes it — `claudeCliThinking` and `codexCliThinking`. They + record the session or thread identifier the tool reported, the token counts it + reported, the model it named when it names one, the resolved executable path + in place of an endpoint URL, the tool's own `--version` string as the + implementation version (for this transport the tool *is* the implementation), + a request commitment over the rendered argument vector, and a response + commitment over the assembled answer. + + **A zero exit status never raises the strength.** A coding-agent CLI that + exits zero has demonstrated that it ran and did not crash; it has not stated + which model served the request. Subprocess calls almost always exit zero, so + encoding that as corroboration would make the weakest evidence in the system + look like the strongest. `strength` is `model_observed` only when the tool + named both an identifier and a model, `correlated` when it named only an + identifier, and `requested_only` otherwise. + + The two transports differ in how far they can get. `claude` names the model + that consumed tokens in its result event's `modelUsage` map, complete with a + context-window variant marker such as `[1m]`, so a Claude CLI run can reach + `model_observed`. `codex-cli 0.146.0` names no model anywhere in its event + stream, so **no** Codex CLI run can exceed `correlated` — backfilling the + `--model` flag baikai passed would report the request as an observation. + +- `baikai-claude`: an evidence record's `thinking` field now describes what a + reasoning-effort request became on the `claude` command line: mode `flag`, + wire field `--effort`, and an `effort_clamped` adjustment recording the + `minimal` → `low` collapse, because the tool's `--effort` flag has no + `minimal`. A caller asking for `minimal` and a caller asking for `low` produce + byte-identical argument vectors — and therefore identical request commitment + digests — so the translation is the only place that difference survives. + +- **Breaking:** `baikai-claude` and `baikai-openai`: `claudeAgentCommand` and + `codexAgentCommand` return `(AgentCommand, ThinkingTranslation)` rather than + `AgentCommand`. The runner deliberately imports no vendor renderer, so it + cannot derive the translation and has to be handed it. A caller that only + wants the command writes `fmap fst`. Both modules also export the translation + function alone — `claudeAgentThinking` and `codexAgentThinking` — for asking + what a level would become without rendering anything. + +### Fixed + +- **Loud:** `baikai-claude` and `baikai-openai`: both subprocess providers + hardcoded `usage = zeroUsage` on every call, so a cost dashboard saw every + `claude -p` and `codex exec` call as consuming no tokens and costing nothing. + Both tools report their own token counts and baikai now carries them through, + normalized into the disjoint `Usage` convention: `claude`'s counts are + Anthropic-shaped and already disjoint, while `codex` reports OpenAI-style + inclusive prompt counts, so its cached tokens are subtracted out of + `inputTokens`. `claude` additionally reports a `total_cost_usd`, which now + populates `Usage.cost` exactly rather than being reported as zero. + + **A dashboard that read these calls as free will now see real tokens and, for + `claude`, a real cost.** That is the correction, not a regression — but it + changes what existing reports show, and totals over historical data will not + match totals over new data. + +- `baikai-claude`: `Response.responseId` was always `Nothing` on the `claude -p` + transport even though `ClaudeCliResult` decoded the tool's `session_id` one + screen earlier and then dropped it. It now carries that identifier, on both + the successful and the failed terminal. `baikai-openai`: the same for + `codex exec`, whose thread identifier was filtered out of the event stream + along with everything that was not an `agent_message`. These are the handles + each vendor's support tooling looks a run up by. + +### Changed + +- **Breaking:** `baikai-claude`: `Baikai.Provider.Claude.Sse`'s four streaming + entry points — `claudeSseStream`, `claudeSseStreamValue`, + `claudeSseStreamValueWithHeaders`, and `sseFromResponse` — take a new + `ResponseMetadata -> IO ()` callback immediately before the existing per-event + callback. It fires exactly once, before the first event, on both the success + and the non-2xx path. Pass `(\_ -> pure ())` to keep the previous behaviour. + The callback is separate rather than a widening of the per-event one because + the per-event callback runs once per SSE frame and response-level data does not + belong on that path. + +- **Breaking:** `baikai-claude`: `Baikai.Provider.Claude.Internal.Request`'s + `mapRequest` now returns + `Either Text (Messages.CreateMessage, ThinkingTranslation)` and + `computeThinking` returns `(ThinkingPlan, ThinkingTranslation)`. Take `fst` to + keep the previous value. This module is exposed for provider tests and + debugging and its header states it is not covered by PVP compatibility + guarantees, but the change is recorded here because that is not a licence to + break a consumer silently. + +- **Breaking:** `baikai-claude`: `claudeInteractiveCommand` now returns + `Either AgentRenderError (FilePath, [String])` and `launchClaudeInteractive` + returns `IO (Either AgentRenderError InteractiveLaunchResult)`. A request + whose `safety` is a `CodexSandbox` policy — which Claude Code cannot express + — is refused with `SafetyNotExpressible AgentClaude`, naming the rejected + sandbox mode and approval policy and suggesting `ClaudeAllowedTools` or + `DefaultSafety`. Previously the policy was silently discarded and an + **unrestricted** Claude session was started and reported as a success. A + `Left` means no process was started; a `Right` with a non-zero exit code + means the session ran and exited non-zero. `DefaultSafety` and an empty + `ClaudeAllowedTools` list still render no safety flag and are never refused, + and no previously rendered argument vector changed. Callers must handle the + refusal branch. + +## [baikai-openai 0.5.0.0] - 2026-08-05 + +### Added + +- `baikai-openai`: new exposed module `Baikai.Provider.OpenAI.Agent` with + `CodexAgentConfig`, `defaultCodexAgentConfig`, and `codexAgentCommand`, the + same renderer for `codex exec`. It maps the capability profile onto + `--sandbox` (`read-only` / `workspace-write` / `danger-full-access`), emits + `--cd` for the working root, and defaults `--skip-git-repo-check` and + `--ephemeral` on. A request carrying a tool allow-list is **refused** with + `UnsupportedToolRestriction`, because `codex exec` has no such flag and running + it with unrestricted tools would grant more authority than the caller asked + for. Nothing is spawned. + +- `baikai-openai`: an evidence record's `thinking` field now describes what the + caller's reasoning-effort preference became on the wire for the specific host + the call went to, across **all seven** OpenAI-compatible wire shapes. The + OpenAI-native shape sends the canonical level verbatim and records no + adjustment, because it expresses every level exactly. The four shapes that + carry an effort word for a non-native host record `effort_clamped` whenever + the word differs from the canonical name — `minimal` becomes `low`, and both + `xhigh` and `max` become `high`. Z.ai and Qwen accept a bare + `enable_thinking: true` with no depth, so **every** level records + `effort_collapsed_to_toggle`: a caller asking for `max` and a caller asking + for `low` produce byte-identical requests there, and only the evidence record + can tell them apart. A host with no reasoning controls records + `thinking_dropped_unsupported_host` where the option previously vanished with + no trace. A forty-two-row table test pins the translation and the shaped + request body for every shape at every level. + +- `baikai-openai`: the Chat Completions provider now fills in the evidence record + it previously left blank. It records the model **the host reported running** + (read from the first streamed chunk carrying a top-level `model` field and + never overwritten by a later one), the host's `x-request-id` correlation + header, the response id, the token counts the host actually reported, and a + commitment digest over the assembled response. A field the host did not report + stays `"unobserved"` and is never backfilled from the request — in particular, + a call that fails before any chunk arrives reports no observed model at all. + `strength` is `model_observed` when both the model and a correlation + identifier arrived, `correlated` when only the identifier did, and + `requested_only` otherwise; a 2xx status never raises it, because a 200 means + the request was accepted, not that any particular model ran. + `fully_observed` is unreachable on this transport, since no host in this + ecosystem echoes the reasoning configuration it applied. + +- `baikai-openai`: new exports from `Baikai.Provider.OpenAI.Sse` — + `ResponseMetadata` and `capturedHeaderNames` — and from + `Baikai.Provider.OpenAI.Api` — `openaiChatStreamWith` and `SseDriver`. + Response-header capture is an **allow-list** (`x-request-id`, `request-id`, + `x-amzn-requestid`, `x-ms-request-id`, `cf-ray`, in that preference order), + not a denylist, so a header a future gateway adds is not recorded by default. + The list is longer than the Anthropic one because this transport speaks to an + open-ended set of hosts and the gateways commonly in front of them. + +- `baikai-openai`: the same field for `codex exec`: mode `flag`, wire field + `model_reasoning_effort`, and **no** adjustments at any level. Codex is the + only transport in baikai that expresses all six canonical levels exactly, and + a test asserts each one reaches the command line verbatim. + +### Fixed + +- `baikai-openai`: `Response.responseId` was always `Nothing` on the Chat + Completions transport, although every compatible host sends a top-level `id` + on every streamed chunk. It now carries the identifier the host reported, on + both the successful and the failed terminal. + +### Changed + +- **Breaking:** `baikai-openai`: `Baikai.Provider.OpenAI.Sse`'s four streaming + entry points — `openaiSseStream`, `openaiSseStreamValue`, + `openaiSseStreamValueWithHeaders`, and `sseFromResponse` — take a new + `ResponseMetadata -> IO ()` callback immediately before the existing per-chunk + callback. It fires exactly once, before the first chunk, on both the success + and the non-2xx path — a failed call's correlation identifier is if anything + more valuable than a successful one's. Pass `(\_ -> pure ())` to keep the + previous behaviour. The callback is separate rather than a widening of the + per-chunk one because that one runs once per SSE frame and response-level data + does not belong on that path. + +- **Breaking:** `baikai-openai`: `Baikai.Provider.OpenAI.Api`'s `RawChunk` gains + `model` and `responseId` fields, both `Maybe Text`. Code that pattern-matches + on `RawChunk` is unaffected; code that constructs one with record syntax must + add them. + +- **Breaking:** `baikai-openai`: `Baikai.Provider.OpenAI.Shape`'s + `shapeRequestBody`, `streamRequestBody`, and `injectThinkingShape` now return + `(Aeson.Value, ThinkingTranslation)` instead of a bare body. Take `fst` to + keep the previous value. The description has to travel out of the shaping step + because nothing downstream can recompute it: it depends on the host's + `ThinkingFormat`, which only the compat lookup knows. **No request body + changed** — every one of the seven shapes puts exactly the same bytes on the + wire as before. + +- **Breaking:** `baikai-openai`: `codexInteractiveCommand` now returns + `Either AgentRenderError (FilePath, [String])` and `launchCodexInteractive` + returns `IO (Either AgentRenderError InteractiveLaunchResult)`. A request + whose `safety` is a non-empty `ClaudeAllowedTools` list — which `codex` has + no flag for — is refused with `SafetyNotExpressible AgentCodex`, quoting the + rejected tools and suggesting `CodexSandbox` or `DefaultSafety`. Previously + the allow-list was silently discarded and Codex was started with its default + sandbox. The same `Left`/`Right` reading applies, `DefaultSafety` and an + empty allow-list are never refused, and no previously rendered argument + vector changed. Callers must handle the refusal branch. + + Both changes make the interactive surface honor the same contract as the new + unattended surface: a safety policy the chosen provider cannot express fails + visibly instead of silently becoming a weaker policy. Downstream consumers + must adapt before upgrading; the known one is `shinzui/seihou`, whose + `Seihou.CLI.AgentLaunchExec` module builds interactive launch requests. + +## [baikai-trace-otel 0.3.0.3] - 2026-08-05 + +### Added + +- `baikai-trace-otel`: the sink attaches an evidence record's salient fields to + the open span as flat attributes (`baikai.evidence.run_id`, + `baikai.evidence.call_id`, `baikai.evidence.strength`, the two digests, and + `gen_ai.response.model` only when the provider actually reported one) rather + than serialising the record into one blob. A `CallEvidence` event neither + opens nor closes a span. + +### Changed + +- `baikai-trace-otel`: widened its `baikai` bound to admit `0.5`. No API change. + +## [baikai-effectful 0.3.0.3] - 2026-08-05 + +### Changed + +- Widened its `baikai` bound to admit `0.5`. No API change; the package's + own surface is untouched. + +## [baikai-kit 0.1.0.4] - 2026-08-05 + +### Changed + +- Widened its `baikai` bound to admit `0.5`. No API change; the package's + own surface is untouched. + +## [baikai-agent 0.1.0.0] - 2026-08-05 + +### Added + +- `baikai-agent`: **new package** (`0.1.0.0`) holding the unattended + coding-agent runner. `Baikai.Agent.Run.runAgentCommand` takes an + `AgentRunRequest` and an already-rendered `AgentCommand` and spawns the tool + with no terminal and no human present. It delivers the prompt on standard + input and closes the handle, drains standard output and standard error + concurrently so a chatty agent cannot deadlock on a full pipe, retains at most + `outputLimit` bytes per stream while reading and discarding the excess, and + honors the three output disciplines. Preconditions run before any spawn: a + missing working directory is `WorkingDirMissing` and unset or empty declared + variables are `MissingEnvironment`, listing all of them at once. On timeout + the child's whole process group is interrupted, given a grace period, and then + terminated, so the agent's own child processes go with it; the failure reports + the configured limit. A non-zero exit code is a successful run carrying that + code, not a failure. The runner consumes an already-rendered `AgentCommand` + and never imports a vendor renderer, so it is exercised entirely with + hand-written argument vectors. Its POSIX-signal escalation is conditional on a + non-Windows build. + +- `baikai-agent`: new exposed module `Baikai.Agent.Config`, the layered + configuration layer. `resolveAgentJob` resolves one named job across five + layers — built-in defaults, the operator file, the repository file, the + environment, then command-line overrides, later layers winning — and returns + the resolved `AgentJob` together with a report attributing every value to the + file, line, and column it came from. `agentJobRequest` converts a job into an + `AgentRunRequest`, taking the prompt at call time. `listAgentJobs` enumerates + configured job names, sorted, each attributed to the highest-precedence scope + defining it. `defaultAgentConfigPaths` locates + `$XDG_CONFIG_HOME/baikai/agents.kdl` (or `$HOME/.config/baikai/agents.kdl`) + and `./.baikai/agents.kdl`, with no upward search through parent directories. + + The **policy ceiling** is loaded by a separate function, `loadAgentCeiling`, + against a separate source list containing the operator file and nothing else: + no repository file, environment variable, or command-line override can raise + it. `applyCeilingToJob` refuses an over-broad request with `CeilingRejected` + rather than clamping it. With no operator file the ceiling is + `defaultAgentCeiling`. `safety.provider-args` is classified secret and renders + as `` in any report or structured error. + + New dependencies: `settei`, `settei-env`, `settei-kdl`, and + `settei-optparse-applicative` (all `^>=0.2`, published on Hackage at + `0.2.0.0`), plus `containers` and `filepath`. `settei-formats` is deliberately + excluded, because it bundles Dhall loading and repository configuration is + untrusted input here. + +- `baikai-agent`: the **`baikai` executable**, with the `agent run`, + `agent show`, and `agent list` commands, and the `Baikai.Agent.Cli` module + that implements them. A shell script now invokes one stable command, supplies + a prompt on standard input, and selects Claude Code or Codex entirely through + configuration. + + `agent run` resolves the named job, caps it against the operator ceiling, + renders it through the vendor renderer for its provider, and spawns it. The + agent's own exit code passes through unchanged; Baikai's own failures use 64 + and above following the `sysexits` convention — 64 for a usage error or an + empty prompt, 69 when the executable could not be started, 70 for malformed + output, 75 for a timeout, 77 for a policy refusal, and 78 for a configuration + problem. The prompt comes from `--prompt-stdin`, `--prompt-file`, or + `--prompt`, which are mutually exclusive, and is decoded as UTF-8 explicitly + rather than through the handle's locale encoding. + + `agent show` performs the whole pipeline except spawning and prints each + resolved value with the file, line, and column it came from, the policy + ceiling in force and where it was read, and the exact argument vector that + would be spawned — with `` in place of any raw provider argument. A + job whose policy is refused prints its configuration first and then the + refusal. `agent list` enumerates configured jobs and the scope each came from. + + Every Baikai diagnostic goes to standard error. The agent's own output follows + the job's output mode, so `response=$(baikai agent run job)` yields the + agent's answer alone for a capturing job. `--set KEY=VALUE` overrides one + setting of the selected job through `settei`'s own command-line source, so an + override is attributed with the same fidelity as a file. `--json` emits + exactly one JSON object per command. + + New dependencies for `baikai-agent`: `baikai-claude`, `baikai-openai`, and + `optparse-applicative`. The provider packages are needed only so that + `renderJobCommand`, the single provider dispatch point in the codebase, can + reach both renderers. This is the first dependency in the workspace from + `baikai-agent` onto the provider packages, so `baikai-agent` now publishes + after all three of `baikai`, `baikai-claude`, and `baikai-openai`. + + The user guide `docs/user/unattended-agent-runs.md` documents the whole + surface: the three commands with their flags, exit codes, and stream + discipline; the KDL job format and layer precedence; the operator ceiling and + redaction; the capability mapping tables for both tools; and a before-and-after + migration of a script that embeds provider flags today. + `docs/user/cli-providers.md` and `docs/user/interactive-launches.md` link to + it, and the capability mapping tables moved there from the latter. + +- `baikai-agent`: **an unattended coding-agent run now produces model-call + evidence.** This surface previously had no observability of any kind: no trace + sink, no `Response`, no usage, no identifiers. An operator could show that a + process started, exited, and took some time; they could not show which model + ran, which reasoning effort was applied, or which agent session the run + corresponds to in the vendor's records. + + A record carries the run and call identifiers, the resolved executable and its + own reported version, digests over the request, the requested model and what + the reasoning-effort request became on the command line, whatever the tool + reported about itself, the outcome, and an honest strength. + + **A zero exit status never raises the strength.** On this surface that rule + matters more than anywhere else, because almost every unattended run exits + zero. A coding agent that exits zero has demonstrated that it ran, not which + model served it. + + Two things gate what a record can prove, and neither is the default. The job + must **capture** output — under `inherit` the agent's bytes went to the + operator's terminal and baikai never held them — and the tool must be + configured to print a structured format, which means `--output-format json` + for `claude` or `--json` for `codex exec` through the job's `provider-args`. + Without both, the tool's session identifier, model, and token counts are + genuinely unavailable and the record says `"unobserved"` rather than inferring + anything. A timed-out run records `aborted`; a run that never started records + nothing at all. + +- **Breaking:** `baikai-agent`: `Baikai.Agent.Run.runAgentCommand` takes two new + leading arguments and returns the new outcome type: + `Maybe EvidenceRequest -> ThinkingTranslation -> AgentRunRequest -> AgentCommand -> IO AgentRunOutcome`. + A caller who wants the previous behaviour passes `Nothing` and + `Baikai.Evidence.noThinkingRequested` and reads the `outcome` field; that path + is byte-for-byte what it was, and costs what it cost — no digest is computed, + no call identifier is generated, and the tool is not invoked a second time to + read its version. + +- `baikai-agent`: `baikai agent run` gains `--evidence-file PATH` and + `--run-id TEXT`. Supplying neither leaves the run on the pre-existing path at + the pre-existing cost; supplying either turns recording on, with the job's own + name standing in as the run identifier when only a destination is given. The + file is written atomically — a staging file beside the destination, then a + rename — so a reader polling the path never sees a half-written object, and it + is never appended to. A failed write is reported on standard error and never + changes the exit code, because the agent's own status is what a calling script + branches on. `docs/user/unattended-agent-runs.md` documents both options and, + more importantly, what the record does and does not prove. + +- `baikai-agent`: `baikai agent run` gains `--require-evidence STRENGTH`, taking + `requested_only`, `correlated`, `model_observed`, or `fully_observed` — the + same words a record's `strength` field spells, so what one record showed can + be passed back as the next run's requirement. A job whose configuration cannot + produce evidence of at least that strength is refused before anything is + spawned, exiting 77 — the code a ceiling violation and an inexpressible safety + policy already use, so a script branching on 77 needs no new case. + +## [baikai-claude 0.4.0.1] - 2026-07-30 + +### Fixed + +- Widened the `crypton` bound from `^>=1.0` to `>=1.0 && <1.2` so consumers can + build `baikai-claude` alongside packages that require `crypton` 1.1.x (for + example `pg-migrate-1.1.0.0`), which previously had no solvable build plan. + The only `crypton` use is `Crypto.Hash` (`Digest`, `SHA256`) in + `Baikai.Provider.Claude.Transport`, whose API is identical across the 1.0/1.1 + boundary. No API change. + +## [baikai 0.4.1.0] - 2026-07-20 + +### Changed + +- Version bump only; no library API or code changes. Released so the umbrella + release tag `baikai-0.4.1.0` names a fresh core version alongside the breaking + `baikai-claude` / `baikai-openai` 0.4.0.0 releases, matching the tag + convention downstream consumers pin against. + +## [baikai-claude 0.4.0.0] - 2026-07-20 + +### Changed + +- **Breaking:** `claudeCliCommand` now takes the `Options` record and forwards + `Options.thinking` to batch `claude -p` as `--effort ` (`minimal` + collapses to `low`, matching the interactive launcher and the claude CLI's + lack of a `minimal` value). `thinking = Nothing` emits no effort flag, keeping + existing argv byte-for-byte. The added parameter is a PVP-major signature + change. + +## [baikai-openai 0.4.0.0] - 2026-07-20 + +### Changed + +- **Breaking:** `codexCliCommand` now takes the `Options` record and forwards + `Options.thinking` to `codex exec` as `-c model_reasoning_effort=` for + all six effort levels. `thinking = Nothing` emits no override, keeping + existing argv byte-for-byte. The added parameter is a PVP-major signature + change. + +## [baikai 0.4.0.0] - 2026-07-20 + +### Added + +- Added `ThinkingXHigh` and `ThinkingMax` to the exported `ThinkingLevel` + vocabulary and added a defaulted `InteractiveLaunchRequest.effort` field. + Extending the closed sum type is a PVP-major API change for downstream + exhaustive matches. + +## [baikai-claude 0.3.0.2] - 2026-07-20 + +### Added + +- Added `--effort` rendering to interactive Claude Code launches and preserved + `xhigh` / `max` on native adaptive Anthropic API requests, with larger fixed + budgets for manual-thinking models. + +### Changed + +- Bumped the internal `baikai` dependency bound to `^>=0.4.0` for the + baikai 0.4.0.0 release. + +## [baikai-openai 0.3.0.2] - 2026-07-20 + +### Added + +- Added `model_reasoning_effort` overrides to interactive Codex launches and + preserved `xhigh` / `max` in native OpenAI request JSON; non-native + OpenAI-compatible request shapes continue to clamp them to `high`. + +### Changed + +- Bumped the internal `baikai` dependency bound to `^>=0.4.0` for the + baikai 0.4.0.0 release. + +## [baikai-trace-otel 0.3.0.2] - 2026-07-20 + +### Changed + +- Bumped the internal `baikai` dependency bound to `^>=0.4.0` for the + baikai 0.4.0.0 release. No API changes. + +## [baikai-effectful 0.3.0.2] - 2026-07-20 + +### Changed + +- Bumped the internal `baikai` dependency bound to `^>=0.4.0` for the + baikai 0.4.0.0 release. No API changes. + +## [baikai-kit 0.1.0.3] - 2026-07-20 + +### Changed + +- Bumped the internal `baikai` dependency bound to `^>=0.4.0` for the + baikai 0.4.0.0 release. No API changes. + +## [baikai 0.3.1.0] - 2026-07-15 + +### Added + +- Added `claude-sonnet-5` to the Anthropic model catalog (1M context window, + 128k max output, `tool_call` + reasoning). +- Added the `gpt-5.6` family — `gpt-5.6`, `gpt-5.6-luna`, `gpt-5.6-sol`, and + `gpt-5.6-terra` — to the OpenAI model catalog (chat-completions with + `tool_call` support). + +### Changed + +- Corrected `claude-sonnet-4-5` context window to 1M tokens and + `claude-sonnet-4-6` max output to 128k tokens in the catalog. +- Added PVP-compliant upper bounds to all previously-unbounded library and + executable dependencies. + +## [baikai-claude 0.3.0.1] - 2026-07-15 + +### Changed + +- Added PVP-compliant upper bounds to all previously-unbounded library and + executable dependencies. + +## [baikai-openai 0.3.0.1] - 2026-07-15 + +### Changed + +- Added PVP-compliant upper bounds to all previously-unbounded library and + executable dependencies. + +## [baikai-trace-otel 0.3.0.1] - 2026-07-15 + +### Changed + +- Added PVP-compliant upper bounds to all previously-unbounded library and + executable dependencies. + +## [baikai-effectful 0.3.0.1] - 2026-07-15 + +### Changed + +- Added PVP-compliant upper bounds to all previously-unbounded library and + executable dependencies. + +## [baikai-kit 0.1.0.2] - 2026-07-15 + +### Changed + +- Added PVP-compliant upper bounds to all previously-unbounded library and + executable dependencies. + +## [baikai 0.3.0.0] - 2026-07-03 + +### Added + +- Added the documented record-update bases `emptyOptions`, `emptyContext`, + `emptyModel`, `emptyResponse`, `emptyTool`, `emptyTextContent`, + `emptyThinkingContent`, `emptyToolCall`, `emptyImageContent`, + `emptyEmbeddingModel`, plus zero-valued bases `zeroUsage`, `zeroCost`, + `zeroCostBreakdown`, and `zeroModelCost`. +- Added `firstEmbedding`, a total accessor for OpenAI-compatible embedding + responses. +- Added `responseError`, `errorResponse`, `httpError`, and + `parseRetryAfterSeconds` for the in-band error contract. + +### Changed + +- **Breaking:** Constructors for evolvable records are no longer exported: + `Options`, `Context`, `Model`, `OpenAICompletionsCompat`, + `AnthropicMessagesCompat`, and `InteractiveLaunchRequest` are built from + exported base values plus record updates. +- **Breaking:** The `_X` base values are deprecated in favor of the new + `empty*` and `zero*` names; the aliases remain for this release. +- **Breaking:** Removed `unModel`; use `mkModel` or `emptyModel` record + updates. +- **Breaking:** Renamed `InteractiveLaunchRequest.model` to `modelId`. +- **Breaking:** `Response.latencyMs` and trace event `latencyMs` fields are + now `Int`. +- **Breaking:** `completeRequest` / `completeRequestWith` no longer throw + `BaikaiError` for unregistered API tags; they return an error-shaped + `Response`. +- **Breaking:** CLI providers now report subprocess/decode/provider failures + in-band as error-shaped `Response`s. +- **Breaking:** `errorTerminal` now requires a `BaikaiError`, enforcing + structured error details for `EventError` construction sites. +- Documented that `Baikai.Prelude` is a convenience module outside the PVP + stability contract and that `.Internal` modules have no compatibility + guarantees. + +### Fixed + +- Empty embedding `data` arrays now produce a typed `decodeError` instead of + crashing on an empty vector. +- The model-fetch JSON renderer now delegates string escaping to aeson. +- The model generator now fails on sanitized Haskell identifier collisions + instead of rendering duplicate bindings. +- Live HTTP status, `Retry-After`, and network-failure classification now + works on both API providers. +- `content_filter` / Anthropic refusals terminate as classified `EventError` + terminals, and `liftCompleteToStream` preserves error-shaped responses. + +## [baikai-claude 0.3.0.0] - 2026-07-03 + +### Changed + +- **Breaking:** `Baikai.Provider.Claude.ErrorClass` moved to + `Baikai.Provider.Claude.Internal.ErrorClass`. +- **Breaking:** `mapRequest` and pure request-shaping helpers moved from + `Baikai.Provider.Claude.Api` to + `Baikai.Provider.Claude.Internal.Request`. +- **Breaking:** `ClaudeCliConfig` and `ClaudeInteractiveConfig` constructors + are no longer exported; start from their default config values and update + fields. +- **Breaking:** CLI and interactive `extraArgs` fields are now `[Text]`. + +## [baikai-openai 0.3.0.0] - 2026-07-03 + +### Changed + +- **Breaking:** `Baikai.Provider.OpenAI.ErrorClass` moved to + `Baikai.Provider.OpenAI.Internal.ErrorClass`. +- **Breaking:** `mapRequest` and pure request-shaping helpers moved from + `Baikai.Provider.OpenAI.Api` to + `Baikai.Provider.OpenAI.Internal.Request`. +- **Breaking:** `CodexCliConfig` and `CodexInteractiveConfig` constructors are + no longer exported; start from their default config values and update fields. +- **Breaking:** CLI and interactive `extraArgs` fields are now `[Text]`. + +## [baikai-trace-otel 0.3.0.0] - 2026-07-03 + +### Changed + +- Updated the `baikai` dependency bound to `^>=0.3.0`. +- Adjusted to the core trace event `latencyMs :: Int` type. + +## [baikai-effectful 0.3.0.0] - 2026-07-03 + +### Changed + +- Updated the `baikai` dependency bound to `^>=0.3.0`. + +## [baikai-kit 0.1.0.1] - 2026-07-03 + +### Changed + +- Updated the `baikai` dependency bound to `^>=0.3.0`. + +## [baikai 0.2.0.0] - 2026-06-21 + +### Added + +- `Usage`, `Cost`, and `CostBreakdown` now have `Semigroup`/`Monoid` + instances that add field-by-field, plus `sumUsage :: Foldable f => f + Usage -> Usage`, so callers can total per-call usage and cost. + `reasoningTokens` combines as presence-wins (`Nothing` only when both + operands are `Nothing`). +- A categorised error model: `BaikaiError` is now a record carrying an + `ErrorCategory` (`AuthError`, `RateLimited`, `ContextOverflow`, + `InvalidRequest`, `TransientError`, `DecodeFailure`, `ProcessFailure`, + `ProviderUnavailable`, `OtherError`), an optional HTTP `httpStatus`, a + `retryAfterSeconds` hint, and a subprocess `exitCode`. New smart + constructors (`providerError`, `invalidRequest`, `decodeError`, + `processError`, `rateLimited`, `authError`, `providerUnavailable`), + the `isRetryable` predicate, and the pure `classifyHttpStatus` / + `classifyHttpStatusWithBody` helpers let callers implement retry + policy without parsing error text. `ErrorCategory` and `BaikaiError` + serialize to JSON. +- `Response` and the streaming `EventError`'s `TerminalPayload` now + carry `errorInfo :: Maybe BaikaiError`, so a failed `completeRequest` + (or a drained stream) exposes the structured category/retry hint + in-band. `Baikai.Stream.Event` gains `doneTerminal` / `errorTerminal` + constructors. + +### Changed + +- **Breaking:** `BaikaiError`'s four flat constructors + (`ProviderError`, `RequestInvalid`, `DecodeError`, `ProcessError`) + were replaced by the record above. Migrate by lowercasing to the + smart constructors — `ProviderError "x"` becomes `providerError "x"`, + `ProcessError n "x"` becomes `processError n "x"`, etc. +- **Breaking:** `Baikai.Stream.Event.TerminalPayload` and + `Baikai.Response.Response` gained an `errorInfo` field; build + `TerminalPayload` via `doneTerminal` / `errorTerminal`. + +### Fixed + +- Restored JSON decoding for `BaikaiError` values with omitted optional + metadata fields. + +## [baikai-claude 0.2.0.0] - 2026-06-21 + +### Added + +- The Anthropic API and `claude -p` CLI providers now classify failures + into the typed `BaikaiError` categories: HTTP errors (via the caught + `servant-client` `ClientError`) map status/`Retry-After`/body onto + `AuthError` / `RateLimited` / `ContextOverflow` / `InvalidRequest` / + `TransientError`, and mid-stream Anthropic `error` events are + classified by their error type. The result is surfaced on + `Response.errorInfo`. + +## [baikai-openai 0.2.0.0] - 2026-06-21 + +### Added + +- The OpenAI/OpenAI-compatible API and `codex exec` CLI providers now + classify failures into the typed `BaikaiError` categories the same way + as `baikai-claude` (HTTP `ClientError` for status-based errors, + streamed error text for mid-stream errors), surfaced on + `Response.errorInfo`. + +## [baikai-trace-otel 0.2.0.0] - 2026-06-21 + +### Changed + +- Updated the `baikai` dependency bound to `^>=0.2.0` for compatibility with + the `baikai 0.2.0.0` breaking API release. + +## [baikai-effectful 0.2.0.0] - 2026-06-21 + +### Changed + +- Updated the `baikai` dependency bound to `^>=0.2.0` for compatibility with + the `baikai 0.2.0.0` breaking API release. + +## [baikai 0.1.1.0] - 2026-06-12 + +### Added + +- Added provider-agnostic `ResponseFormat` support on `Options`, including + plain JSON-object mode and named JSON-schema mode. +- Added `Baikai.Embedding`, an OpenAI `/v1/embeddings` client for text + embeddings. + +## [baikai-claude 0.1.1.0] - 2026-06-12 + +### Added + +- Mapped baikai `ResponseFormat` options onto Anthropic `output_config` for + Claude API requests. +- Exported `mapRequest` for request-mapping tests and downstream inspection. + +## [baikai-openai 0.1.1.0] - 2026-06-12 + +### Added + +- Mapped baikai `ResponseFormat` options onto OpenAI Chat Completions + `response_format`. +- Exported `mapRequest` for request-mapping tests and downstream inspection. + +## [baikai-effectful 0.1.0.0] - 2026-06-12 + +### Added + +- Initial release: effectful binding for baikai with the `Baikai` dynamic + effect, `complete`, `streamCollect`, `streamEach`, and registry-backed + interpreters. + +## [baikai 0.1.0.0] - 2026-06-04 + +### Added + +- Initial release: unified Haskell interface for working with multiple AI + providers. Core modules including `Baikai`, `Baikai.Prelude`, `Baikai.Api`, + `Baikai.Provider`, `Baikai.Provider.Registry`, `Baikai.Response`, + `Baikai.Stream`, `Baikai.Tool`, `Baikai.Trace`, and the cost/usage modules. +- Depends on released `streamly` (`>=0.11 && <0.13`) and `streamly-core` + (`>=0.3 && <0.5`) from Hackage, so all dependencies resolve from Hackage. + +## [baikai-claude 0.1.0.0] - 2026-06-04 + +### Added + +- Initial release: Anthropic Claude providers for the baikai abstraction, + wrapping the `claude` package for both the Anthropic API and the `claude -p` + CLI (`Baikai.Provider.Claude.Api`, `.Cli`, `.Interactive`). + +## [baikai-openai 0.1.0.0] - 2026-06-04 + +### Added + +- Initial release: OpenAI providers for the baikai abstraction, wrapping the + `openai` package for OpenAI's Chat Completions API + (`Baikai.Provider.OpenAI.Api`, `.Cli`, `.Interactive`). + +## [baikai-trace-otel 0.1.0.0] - 2026-06-04 + +### Added + +- Initial release: OpenTelemetry `TraceSink` adapter for baikai + (`Baikai.Trace.Sink.OpenTelemetry`), emitting one OTel span per provider call + with GenAI semantic-convention attributes plus baikai cost and latency. diff --git a/app/Main.hs b/app/Main.hs --- a/app/Main.hs +++ b/app/Main.hs @@ -13,14 +13,15 @@ runAgentCli, ) import Control.Lens ((^.)) +import Data.ByteString qualified as BS import Data.Generics.Labels () import Data.Text qualified as Text -import Data.Text.IO qualified as TextIO +import Data.Text.Encoding qualified as Text import Options.Applicative qualified as Options import Settei.Env (EnvSnapshot, envSnapshot) import System.Environment (getEnvironment) import System.Exit (ExitCode (..), exitWith) -import System.IO (stderr, stdout) +import System.IO (hFlush, stderr, stdout) main :: IO () main = do @@ -38,10 +39,24 @@ values <- getEnvironment pure (envSnapshot [(Text.pack name, Text.pack value) | (name, value) <- values]) +-- | Write both streams as UTF-8 bytes. +-- +-- Explicitly encoded rather than written with 'Data.Text.IO.hPutStr', +-- which encodes through the handle's locale encoding: under @LANG=C@ — +-- cron, systemd, a minimal container — a single non-ASCII character in +-- the agent's answer made that throw @invalid argument@ after the run +-- had already finished, losing the answer and exiting 1. +-- +-- Nothing here can throw on encoding. Invalid UTF-8 in the child's +-- output became U+FFFD when 'Baikai.Agent.Cli.decoded' decoded it +-- leniently, so what arrives is always encodable, and this is the same +-- discipline the prompt read and the prompt write already use. emit :: AgentCliRun -> IO () emit finished = do - TextIO.hPutStr stdout (finished ^. #standardOutput) - TextIO.hPutStr stderr (finished ^. #standardError) + BS.hPut stdout (Text.encodeUtf8 (finished ^. #standardOutput)) + BS.hPut stderr (Text.encodeUtf8 (finished ^. #standardError)) + hFlush stdout + hFlush stderr exitCodeFrom :: Int -> ExitCode exitCodeFrom 0 = ExitSuccess diff --git a/baikai-agent.cabal b/baikai-agent.cabal --- a/baikai-agent.cabal +++ b/baikai-agent.cabal @@ -1,7 +1,7 @@ -cabal-version: 3.4 -name: baikai-agent -version: 0.1.0.0 -synopsis: Unattended coding-agent runs for the Baikai abstraction +cabal-version: 3.4 +name: baikai-agent +version: 0.2.0.0 +synopsis: Unattended coding-agent runs for the Baikai abstraction description: Runs a local coding-agent command-line tool with no terminal and no human present: delivers the prompt on standard input, drains both @@ -10,13 +10,15 @@ vocabulary lives in @Baikai.Agent@ in the core package, and vendor packages own the translation into each tool's argument vector. -category: AI -license: BSD-3-Clause -license-file: LICENSE -author: Nadeem Bitar -maintainer: nadeem@gmail.com -copyright: (c) 2026 Nadeem Bitar -build-type: Simple +category: AI +license: BSD-3-Clause +license-file: LICENSE +author: Nadeem Bitar +maintainer: nadeem@gmail.com +copyright: (c) 2026 Nadeem Bitar +build-type: Simple +tested-with: GHC ==9.12.4 +extra-doc-files: CHANGELOG.md common common-options ghc-options: @@ -57,9 +59,9 @@ build-depends: , aeson ^>=2.2 - , baikai ^>=0.5.0 - , baikai-claude ^>=0.5 - , baikai-openai ^>=0.5 + , baikai ^>=0.6.0 + , baikai-claude ^>=0.6 + , baikai-openai ^>=0.6 , base >=4.20 && <5 , bytestring ^>=0.12 , containers ^>=0.7 @@ -89,12 +91,21 @@ hs-source-dirs: app main-is: Main.hs + -- The runner waits on the child with waitForProcess under + -- System.Timeout and drains the child's pipes on forked threads. + -- Without the threaded runtime that wait blocks every Haskell thread, + -- so the configured timeout can never fire and a chatty child + -- deadlocks the run on a full pipe. BinaryTests proves it against this + -- executable rather than against the suite's own runtime. + ghc-options: -threaded + -- Deliberately thin: everything real lives in Baikai.Agent.Cli so the -- whole command-line surface is reachable from the test suite without -- spawning the built binary. build-depends: , baikai-agent , base >=4.20 && <5 + , bytestring ^>=0.12 , generic-lens ^>=2.3 , lens ^>=5.3 , optparse-applicative ^>=0.19 @@ -102,18 +113,26 @@ , text ^>=2.1 test-suite baikai-agent-test - import: common-options - type: exitcode-stdio-1.0 - hs-source-dirs: test - main-is: Main.hs + import: common-options + type: exitcode-stdio-1.0 + hs-source-dirs: test + main-is: Main.hs other-modules: + BinaryTests CliTests ConfigTests EvidenceTests + PublicSurfaceSpec -- -threaded is not optional here: the runner forks threads to drain -- pipes and relies on System.Timeout interrupting a blocking wait. - ghc-options: -threaded -with-rtsopts=-N + ghc-options: -threaded -with-rtsopts=-N + + -- BinaryTests spawns the shipped executable rather than exercising the + -- library in this process. Naming it here makes cabal build it before + -- this suite and put it on the PATH the suite inherits, so the case + -- runs the binary a user installs instead of this suite's own runtime. + build-tool-depends: baikai-agent:baikai build-depends: , aeson , baikai diff --git a/src/Baikai/Agent/Cli.hs b/src/Baikai/Agent/Cli.hs --- a/src/Baikai/Agent/Cli.hs +++ b/src/Baikai/Agent/Cli.hs @@ -25,12 +25,23 @@ ( -- * The parsed command line AgentCliCommand (..), PromptSource (..), - AgentCliOptions (..), + AgentCliOptions + ( command, + overrides, + userConfig, + repoConfig, + jsonOutput, + evidenceFile, + runId, + requiredEvidence + ), + agentCliOptions, agentCliParser, agentCliParserInfo, -- * Running it - AgentCliRun (..), + AgentCliRun (exitCode, standardOutput, standardError), + agentCliRun, runAgentCli, runAgentCliWithPaths, @@ -40,7 +51,6 @@ -- * Exit codes usageExitCode, unavailableExitCode, - internalExitCode, timeoutExitCode, refusedExitCode, configExitCode, @@ -58,10 +68,11 @@ AgentOutputMode (..), AgentPromptTransport (..), AgentProvider (..), - AgentRenderError, + AgentRenderError (..), AgentRunFailure (..), AgentRunRequest, AgentRunResult, + CeilingViolation, renderAgentCapability, renderAgentProvider, renderAgentRenderError, @@ -71,12 +82,16 @@ ( AgentConfigPaths (..), AgentJob, agentJobRequest, - applyCeilingToJob, + ceilingViolations, defaultAgentConfigPaths, + emptyAgentConfigPaths, listAgentJobs, loadAgentCeiling, + relevantWarnings, renderAgentConfigError, renderAgentConfigScope, + repositoryPolicyNotice, + repositoryScopeViolations, resolveAgentJob, ) import Baikai.Agent.Run (runAgentCommand) @@ -87,6 +102,7 @@ ModelCallEvidence, ThinkingTranslation, evidenceRequest, + parseEvidenceStrength, ) import Baikai.Provider.Claude.Agent ( ClaudeAgentConfig, @@ -99,20 +115,20 @@ defaultCodexAgentConfig, ) import Control.Applicative ((<|>)) -import Control.Exception (IOException, displayException, try) +import Control.Exception (IOException, bracketOnError, displayException, try) import Control.Lens ((&), (.~), (^.)) import Data.Aeson qualified as Aeson +import Data.Aeson.Key qualified as Key +import Data.Aeson.Types (Pair) import Data.ByteString qualified as BS import Data.ByteString.Lazy qualified as BSL -import Data.Char (isControl, ord) import Data.Generics.Labels () import Data.List.NonEmpty qualified as NonEmpty -import Data.Maybe (fromMaybe) +import Data.Maybe (fromMaybe, isJust, isNothing) import Data.Text (Text) import Data.Text qualified as Text import Data.Text.Encoding qualified as Text import GHC.Generics (Generic) -import Numeric (showHex) import Options.Applicative (Parser, ParserInfo) import Options.Applicative qualified as Options import Settei.Env (EnvSnapshot) @@ -127,9 +143,21 @@ reportNodes, ) import Settei.Value (RawValue (..)) -import System.Directory (doesFileExist, renameFile) +import System.Directory + ( doesFileExist, + getCurrentDirectory, + removeFile, + renameFile, + ) import System.Exit (ExitCode (..)) -import System.IO (stdin) +import System.FilePath + ( dropTrailingPathSeparator, + normalise, + takeDirectory, + takeFileName, + (), + ) +import System.IO (hClose, openBinaryTempFileWithDefaultPermissions, stdin) -- | Which of the three commands was asked for. data AgentCliCommand @@ -151,6 +179,9 @@ deriving stock (Eq, Show, Generic) -- | The parsed command line, before any file is opened. +-- +-- Construction: the constructor is deliberately not exported. Start +-- from 'agentCliOptions' and override fields by record update. data AgentCliOptions = AgentCliOptions { command :: !AgentCliCommand, -- | Parsed @--set@ overrides, __with keys as the operator wrote @@ -185,6 +216,9 @@ -- the @inherit@ and @tee@ output modes the agent's own output goes -- straight to the real process streams and bypasses this record -- entirely, which is correct and is what the motivating consumer wants. +-- +-- Construction: the constructor is deliberately not exported. Start +-- from 'agentCliRun' and override fields by record update. data AgentCliRun = AgentCliRun { exitCode :: !Int, standardOutput :: !Text, @@ -194,7 +228,6 @@ usageExitCode, unavailableExitCode, - internalExitCode, timeoutExitCode, refusedExitCode, configExitCode :: @@ -213,9 +246,6 @@ -- | The coding-agent executable could not be started. unavailableExitCode = 69 --- | The agent produced output the caller could not interpret. -internalExitCode = 70 - -- | The run exceeded its timeout and its process group was terminated. timeoutExitCode = 75 @@ -244,6 +274,30 @@ <> Options.failureCode usageExitCode ) +-- | The command line for one command, with every other option at the +-- value the parser produces when the flag is absent. +agentCliOptions :: AgentCliCommand -> AgentCliOptions +agentCliOptions cliCommand = + AgentCliOptions + { command = cliCommand, + overrides = [], + userConfig = Nothing, + repoConfig = Nothing, + jsonOutput = False, + evidenceFile = Nothing, + runId = Nothing, + requiredEvidence = Nothing + } + +-- | A run result carrying an exit code and no output on either stream. +agentCliRun :: Int -> AgentCliRun +agentCliRun code = + AgentCliRun + { exitCode = code, + standardOutput = Text.empty, + standardError = Text.empty + } + -- | The top level takes one subcommand group, @agent@, leaving room for -- future groups without minting a new executable. agentCliParser :: Parser AgentCliOptions @@ -467,17 +521,19 @@ \strength: requested_only, correlated, model_observed, or fully_observed" ) where - parse = \case - "requested_only" -> Right EvidenceRequestedOnly - "correlated" -> Right EvidenceCorrelated - "model_observed" -> Right EvidenceModelObserved - "fully_observed" -> Right EvidenceFullyObserved - other -> - Left - ( "unknown evidence strength: " - <> other - <> " (expected requested_only, correlated, model_observed, or fully_observed)" - ) + -- The strength table lives beside its renderer, in + -- 'Baikai.Evidence.parseEvidenceStrength', so this parser cannot + -- fall behind the names an evidence record spells. + parse other = + maybe + ( Left + ( "unknown evidence strength: " + <> other + <> " (expected requested_only, correlated, model_observed, or fully_observed)" + ) + ) + Right + (parseEvidenceStrength (Text.pack other)) -- -------------------------------------------------------------------- -- Provider dispatch @@ -551,17 +607,28 @@ -- | Discovery, with explicit paths winning per scope. When both scopes -- are explicit nothing is discovered at all, so a fully specified -- invocation never reads @HOME@ or @XDG_CONFIG_HOME@. +-- The repository root is always the process's own directory. @--config@ +-- chooses which file supplies repository-scope settings; it does not +-- move the repository, because the root is what confines a +-- repository-supplied working directory. effectiveConfigPaths :: AgentCliOptions -> IO AgentConfigPaths -effectiveConfigPaths options = +effectiveConfigPaths options = do + root <- getCurrentDirectory case (options ^. #userConfig, options ^. #repoConfig) of (Just user, Just repo) -> - pure AgentConfigPaths {userConfig = Just user, repoConfig = Just repo} + pure + emptyAgentConfigPaths + { userConfig = Just user, + repoConfig = Just repo, + repositoryRoot = root + } (user, repo) -> do discovered <- defaultAgentConfigPaths pure - AgentConfigPaths + emptyAgentConfigPaths { userConfig = user <|> discovered ^. #userConfig, - repoConfig = repo <|> discovered ^. #repoConfig + repoConfig = repo <|> discovered ^. #repoConfig, + repositoryRoot = root } successfulRun :: Text -> Text -> AgentCliRun @@ -582,7 +649,8 @@ pure $ case listed of Left problem -> failedRun configExitCode (renderAgentConfigError problem <> "\n") Right entries - | options ^. #jsonOutput -> successfulRun (jsonArray (map entryJson entries) <> "\n") "" + | options ^. #jsonOutput -> + successfulRun (encodeEnvelope (Aeson.toJSON (map entryJson entries))) "" -- An empty list is a normal state, not an error, and the note -- saying so goes to standard error so that a script piping the -- list never has to filter prose out of its data. @@ -601,10 +669,10 @@ else "" ) entryJson entry = - jsonObject - [ ("name", jsonString (entry ^. #name)), - ("scope", jsonString (renderAgentConfigScope (entry ^. #scope))), - ("definingScopes", Text.pack (show (entry ^. #definingScopes))) + Aeson.object + [ "name" Aeson..= (entry ^. #name), + "scope" Aeson..= renderAgentConfigScope (entry ^. #scope), + "definingScopes" Aeson..= (entry ^. #definingScopes) ] -- -------------------------------------------------------------------- @@ -617,7 +685,11 @@ report :: !ResolutionReport, warnings :: !Text, ceiling :: !AgentCeiling, - ceilingSource :: !Text + ceilingSource :: !Text, + -- | Violations that depend on which file supplied a value, which + -- the pure ceiling check cannot see. They are computed once here and + -- concatenated with it, so an operator sees one refusal. + scopeViolations :: ![CeilingViolation] } deriving stock (Generic) @@ -643,7 +715,15 @@ case loaded of Left problem -> pure (Left (configFailure (renderAgentConfigError problem))) Right resolved -> do - let warningsText = renderWarningsText (resolved ^. #warnings) + -- Not every warning settei raises is about this run. The + -- declaration describes one job, so every key of every *other* + -- job warns, and so does the operator file's policy node; neither + -- is a mistake and neither is the operator's to fix. A repository + -- policy node is different: it does nothing, and whoever wrote it + -- believed it would, so it earns one notice. + let warningsText = + renderWarningsText (relevantWarnings jobName (resolved ^. #warnings)) + <> fromMaybe "" (repositoryPolicyNotice (resolved ^. #warnings)) case resolved ^. #answer of Left problems -> pure @@ -661,24 +741,35 @@ -- path to it. It calls loadAgentCeiling and adds no override -- of its own. loadedCeiling <- loadAgentCeiling paths - pure $ case loadedCeiling of - Left problem -> - Left - StageFailure - { exitCode = configExitCode, - message = renderAgentConfigError problem, - report = Nothing, - warnings = warningsText - } - Right ceiling' -> - Right - StagedJob - { job, - report = resolved ^. #report, - warnings = warningsText, - ceiling = ceiling', - ceilingSource = ceilingSourceLabel paths - } + let loadedOrFailed = case loadedCeiling of + Left problem -> + Left + StageFailure + { exitCode = configExitCode, + message = renderAgentConfigError problem, + report = Nothing, + warnings = warningsText + } + Right ceiling' -> Right ceiling' + case loadedOrFailed of + Left failure -> pure (Left failure) + Right ceiling' -> do + -- Provenance first, against the job as configured: the + -- scope check composes the root itself and must see the + -- spelling the document used. + scopeViolations <- + repositoryScopeViolations paths (resolved ^. #report) jobName job + pure + ( Right + StagedJob + { job = rootedJob paths job, + report = resolved ^. #report, + warnings = warningsText, + ceiling = ceiling', + ceilingSource = ceilingSourceLabel paths, + scopeViolations + } + ) where configFailure text = StageFailure @@ -688,6 +779,28 @@ warnings = "" } +-- | Resolve the job's working directory against the repository root. +-- +-- A relative @working-dir@ has to resolve against /something/, and the +-- process's own directory is the wrong answer for a value that may have +-- come from either file: two documents defining one job would make +-- @\".\"@ mean two places depending on which layer won. The repository +-- root is the same directory whichever file declared it, so +-- @working-dir \".\"@ means \"this checkout\" everywhere. +-- +-- An absolute path passes through @\<\/\>@ unchanged, so a job naming +-- one is untouched. The result is normalised and stripped of a trailing +-- separator, so @\".\"@ yields the root itself rather than +-- @\\/.@ — this path is rendered into @codex exec --cd@, printed +-- by @agent show@, and recorded as an evidence endpoint, and all three +-- should read as the directory a person would name. +rootedJob :: AgentConfigPaths -> AgentJob -> AgentJob +rootedJob paths job = + job + & #workingDir + .~ dropTrailingPathSeparator + (normalise ((paths ^. #repositoryRoot) (job ^. #workingDir))) + ceilingSourceLabel :: AgentConfigPaths -> Text ceilingSourceLabel paths = case paths ^. #userConfig of Nothing -> "built-in default (no operator configuration file)" @@ -726,14 +839,32 @@ showCommand paths snapshot options jobName = do staged <- stageJob paths snapshot options jobName pure $ case staged of - Left failure -> - AgentCliRun - { exitCode = failure ^. #exitCode, - -- A failed resolution's provenance is exactly what an explain - -- command is for, so the report is printed when there is one. - standardOutput = maybe "" (renderReport options) (failure ^. #report), - standardError = failure ^. #warnings <> failure ^. #message <> "\n" - } + Left failure + | options ^. #jsonOutput -> + AgentCliRun + { exitCode = failure ^. #exitCode, + standardOutput = + encodeEnvelope + ( showJson + jobName + "failed" + (failure ^. #exitCode) + (Just (failure ^. #message)) + (failure ^. #report) + Nothing + Nothing + ), + standardError = failure ^. #warnings + } + | otherwise -> + AgentCliRun + { exitCode = failure ^. #exitCode, + -- A failed resolution's provenance is exactly what an + -- explain command is for, so the report is printed when + -- there is one. + standardOutput = maybe "" renderEffectiveConfig (failure ^. #report), + standardError = failure ^. #warnings <> failure ^. #message <> "\n" + } Right stagedJob -> explain options jobName stagedJob -- | Print the effective configuration, the ceiling, and the command @@ -746,7 +877,17 @@ | options ^. #jsonOutput -> AgentCliRun { exitCode = refusedExitCode, - standardOutput = jsonShow (Just (renderAgentRenderError refusal)) Nothing <> "\n", + standardOutput = + encodeEnvelope + ( showJson + jobName + "refused" + refusedExitCode + (Just (renderAgentRenderError refusal)) + (Just (staged ^. #report)) + (Just (staged ^. #ceilingSource, staged ^. #ceiling)) + Nothing + ), standardError = staged ^. #warnings } | otherwise -> @@ -758,7 +899,19 @@ } Right command | options ^. #jsonOutput -> - successfulRun (jsonShow Nothing (Just command) <> "\n") (staged ^. #warnings) + successfulRun + ( encodeEnvelope + ( showJson + jobName + "shown" + 0 + Nothing + (Just (staged ^. #report)) + (Just (staged ^. #ceilingSource, staged ^. #ceiling)) + (Just command) + ) + ) + (staged ^. #warnings) | otherwise -> successfulRun (textSections <> "\n" <> renderCommandSection command) @@ -780,7 +933,7 @@ . #providerArgs .~ ["" | _ <- staged ^. #job . #providerArgs] rendered = do - _ <- applyCeilingToJob (staged ^. #ceiling) request + ceilingGuard staged request fst <$> renderJobCommand (staged ^. #job) displayRequest textSections = "job \"" @@ -791,21 +944,7 @@ <> staged ^. #ceilingSource <> "\n" <> renderCeiling (staged ^. #ceiling) - jsonShow refusal command = - jsonObject - ( [ ("job", jsonString jobName), - ("configuration", renderResolutionJson (staged ^. #report)), - ("ceiling", ceilingJson (staged ^. #ceilingSource) (staged ^. #ceiling)) - ] - <> maybe [] (\message -> [("refused", jsonString message)]) refusal - <> maybe [] (\value -> [("command", commandJson value)]) command - ) -renderReport :: AgentCliOptions -> ResolutionReport -> Text -renderReport options report - | options ^. #jsonOutput = renderResolutionJson report <> "\n" - | otherwise = renderEffectiveConfig report - -- | Render every resolved value with the file, line, and column it came -- from. -- @@ -860,12 +999,33 @@ <> maybe "" (\line -> ":" <> Text.pack (show line)) (location ^. #line) <> maybe "" (\column -> ":" <> Text.pack (show column)) (location ^. #column) +-- | Every way this job exceeds the operator's policy, as one refusal. +-- +-- Two lists are concatenated because they answer different questions. +-- 'ceilingViolations' compares the request against the ceiling and knows +-- nothing about files; 'repositoryScopeViolations', already computed +-- while staging, says which values the untrusted repository file was not +-- allowed to supply at all. Reporting them together means an operator +-- fixing a job description sees every problem in one run, which is the +-- same property 'Baikai.Agent.applyAgentCeiling' has on its own. +-- +-- The request is never clamped to fit: a job that asked for more than it +-- may have is an error to report, not a request to quietly weaken. +ceilingGuard :: StagedJob -> AgentRunRequest -> Either AgentRenderError () +ceilingGuard staged request = + case (staged ^. #scopeViolations) <> ceilingViolations (staged ^. #ceiling) request of + [] -> Right () + violations -> Left (CeilingRejected violations) + renderCeiling :: AgentCeiling -> Text renderCeiling ceiling' = Text.unlines [ " max-capability " <> renderAgentCapability (ceiling' ^. #maxCapability), " allow-provider-args " <> renderBool (ceiling' ^. #allowProviderArgs), - " allowed-providers " <> renderProviders (ceiling' ^. #allowedProviders) + " allowed-providers " <> renderProviders (ceiling' ^. #allowedProviders), + " allowed-tools " <> renderGrants (ceiling' ^. #allowedTools), + " max-timeout " <> renderUnlimited showText (ceiling' ^. #maxTimeout), + " max-output-limit " <> renderUnlimited showText (ceiling' ^. #maxOutputLimit) ] where renderBool True = "true" @@ -873,6 +1033,14 @@ renderProviders [] = "none" renderProviders providers = Text.intercalate ", " (map renderAgentProvider providers) + -- An empty grant list is not "none at all": the capability still + -- implies its own grants, and saying so stops an operator reading + -- this line as "no tools". + renderGrants [] = "(none beyond the capability)" + renderGrants grants = Text.intercalate ", " grants + renderUnlimited render = maybe "unlimited" render + showText :: (Show a) => a -> Text + showText = Text.pack . show -- | The rendered command, one flag per line. -- @@ -916,6 +1084,14 @@ Text -> PromptSource -> IO AgentCliRun +runCommand _paths _snapshot options _jobName _promptSource + | evidenceWithoutADestination options = + pure + ( failedRun + usageExitCode + "--run-id and --require-evidence produce an evidence record, which \ + \needs a destination: add --evidence-file PATH or --json\n" + ) runCommand paths snapshot options jobName promptSource = do staged <- stageJob paths snapshot options jobName case staged of @@ -949,24 +1125,25 @@ Right (request, command, translation) -> do ran <- runAgentCommand (evidenceRequestFor options) translation request command written <- writeEvidenceFile (options ^. #evidenceFile) (ran ^. #evidence) - pure (interpret options staged request (ran ^. #outcome) written) + pure (interpret options staged request (ran ^. #outcome) (ran ^. #evidence) written) where request0 = agentJobRequest (staged ^. #job) promptBody prepared = do - permitted <- applyCeilingToJob (staged ^. #ceiling) request0 - (command, translation) <- renderJobCommand (staged ^. #job) permitted - pure (permitted, command, translation) + ceilingGuard staged request0 + (command, translation) <- renderJobCommand (staged ^. #job) request0 + pure (request0, command, translation) refusedRun message | options ^. #jsonOutput = AgentCliRun { exitCode = refusedExitCode, standardOutput = - jsonObject - [ ("outcome", jsonString "refused"), - ("exitCode", Text.pack (show refusedExitCode)), - ("message", jsonString message) - ] - <> "\n", + encodeEnvelope + ( Aeson.object + [ "outcome" Aeson..= ("refused" :: Text), + "exitCode" Aeson..= refusedExitCode, + "message" Aeson..= message + ] + ), standardError = staged ^. #warnings } | otherwise = @@ -977,6 +1154,10 @@ StagedJob -> AgentRunRequest -> Either AgentRunFailure AgentRunResult -> + -- | The run's evidence record, when one was built. Under @--json@ it + -- travels in the envelope, which is the destination a caller who + -- named no file asked for. + Maybe ModelCallEvidence -> -- | Whatever went wrong writing the evidence file, appended to -- standard error. A failed write never changes the exit code: the -- agent's own status is what a calling script branches on, and @@ -984,18 +1165,57 @@ -- not be written would be the worse surprise. Text -> AgentCliRun -interpret options staged request result evidenceNote = case result of +interpret options staged request result record evidenceNote = case result of + -- A timed-out run is still a run: the tool started, printed, and may + -- have changed the working tree. Its drained output is reported with + -- exactly the stream discipline a finished run gets, so + -- `response=$(baikai agent run job)` under `capture` receives the + -- partial answer with $? set to the timeout code. Under `tee` the + -- bytes were echoed while draining and are not repeated; under + -- `inherit` there is nothing to report. + Left failure@(RunTimedOut timedOut) + | options ^. #jsonOutput -> + AgentCliRun + { exitCode = failureExitCode failure, + standardOutput = + encodeEnvelope + ( Aeson.object + ( [ "outcome" Aeson..= ("failed" :: Text), + "exitCode" Aeson..= failureExitCode failure, + "message" Aeson..= renderAgentRunFailure failure + ] + <> streamFields "stdout" (timedOut ^. #stdout) + <> streamFields "stderr" (timedOut ^. #stderr) + <> evidenceField record + ) + ), + standardError = staged ^. #warnings <> evidenceNote + } + | otherwise -> + AgentCliRun + { exitCode = failureExitCode failure, + standardOutput = if capturing then decoded (timedOut ^. #stdout) else "", + standardError = + staged ^. #warnings + <> evidenceNote + <> (if capturing then decoded (timedOut ^. #stderr) else "") + <> renderAgentRunFailure failure + <> "\n" + } Left failure | options ^. #jsonOutput -> AgentCliRun { exitCode = failureExitCode failure, standardOutput = - jsonObject - [ ("outcome", jsonString "failed"), - ("exitCode", Text.pack (show (failureExitCode failure))), - ("message", jsonString (renderAgentRunFailure failure)) - ] - <> "\n", + encodeEnvelope + ( Aeson.object + ( [ "outcome" Aeson..= ("failed" :: Text), + "exitCode" Aeson..= failureExitCode failure, + "message" Aeson..= renderAgentRunFailure failure + ] + <> evidenceField record + ) + ), standardError = staged ^. #warnings <> evidenceNote } | otherwise -> @@ -1006,7 +1226,7 @@ | options ^. #jsonOutput -> AgentCliRun { exitCode = resultExitCode ran, - standardOutput = resultJson ran <> "\n", + standardOutput = encodeEnvelope (resultJson record ran), standardError = staged ^. #warnings <> evidenceNote <> truncationNotes ran } | otherwise -> @@ -1029,6 +1249,19 @@ -- captured nothing at all. capturing = request ^. #output == CaptureOutput +-- | Whether the caller asked for a record and named nowhere to put it. +-- +-- Building one costs a @--version@ probe of the tool and two digests, so +-- a record that is built and dropped is measurable work proving nothing. +-- There is no sensible default destination either: writing to a guessed +-- path would surprise, and printing it on standard output would corrupt +-- the agent's own answer, which a capturing script is reading. +evidenceWithoutADestination :: AgentCliOptions -> Bool +evidenceWithoutADestination options = + (isJust (options ^. #runId) || isJust (options ^. #requiredEvidence)) + && isNothing (options ^. #evidenceFile) + && not (options ^. #jsonOutput) + -- | The caller's evidence request, or 'Nothing' when they asked for -- none. -- @@ -1060,11 +1293,22 @@ -- | Write one evidence record to the operator's chosen path, returning -- whatever went wrong. -- --- The write is atomic — a temporary file beside the destination, then a --- rename — so a reader polling the path never sees a half-written --- record. It never appends: each run writes one complete object, and an --- operator wanting a log of many runs points each at its own path. +-- The write is atomic — a uniquely named staging file beside the +-- destination, then a rename — so a reader polling the path never sees a +-- half-written record. It never appends: each run writes one complete +-- object, and an operator wanting a log of many runs points each at its +-- own path. -- +-- The staging name is __not__ the destination plus a suffix. +-- @openBinaryTempFileWithDefaultPermissions@ creates the file itself +-- under a fresh name, with @O_EXCL@, so a symbolic link planted at a +-- guessable path is never followed — an unattended run writing its +-- record into whatever a predictable name pointed at is a file overwrite +-- an attacker chooses. The default-permissions variant keeps the mode a +-- plain @writeFile@ would have given rather than the @0600@ the ordinary +-- temp-file function uses, so an operator's umask still decides who can +-- read the record. +-- -- Nothing is written when the operator named no path, and nothing is -- written when the run produced no evidence, which is the case where the -- tool never started. An empty file would claim a run happened. @@ -1072,12 +1316,22 @@ writeEvidenceFile Nothing _ = pure "" writeEvidenceFile (Just _) Nothing = pure "" writeEvidenceFile (Just path) (Just record) = do - let staging = path <> ".partial" written <- try - ( do - BSL.writeFile staging (Aeson.encode record) - renameFile staging path + ( bracketOnError + ( openBinaryTempFileWithDefaultPermissions + (takeDirectory path) + (takeFileName path <> ".partial") + ) + -- Only on failure: a rename that succeeded has taken the name + -- away, and removing the destination would be the opposite of + -- what this function is for. + (\(staging, handle) -> hClose handle >> removeFile staging) + ( \(staging, handle) -> do + BSL.hPut handle (Aeson.encode record) + hClose handle + renameFile staging path + ) ) :: IO (Either IOException ()) pure $ case written of @@ -1100,7 +1354,6 @@ SpawnFailed _ _ -> unavailableExitCode WorkingDirMissing _ -> configExitCode MissingEnvironment _ -> configExitCode - OutputMalformed _ -> internalExitCode -- Policy said no and nothing was started, which is exactly what -- 'refusedExitCode' means for a ceiling violation or a provider that -- cannot express a safety policy. A script that already branches on @@ -1123,28 +1376,50 @@ OutputCaptured bytes -> Text.decodeUtf8Lenient bytes OutputTruncated bytes -> Text.decodeUtf8Lenient bytes -resultJson :: AgentRunResult -> Text -resultJson result = - jsonObject - ( [ ("outcome", jsonString "ran"), - ("exitCode", Text.pack (show (resultExitCode result))), - ("provider", jsonString (renderAgentProvider (result ^. #provider))), - ("durationSeconds", Text.pack (show (realToFrac (result ^. #duration) :: Double))) +resultJson :: Maybe ModelCallEvidence -> AgentRunResult -> Aeson.Value +resultJson record result = + Aeson.object + ( [ "outcome" Aeson..= ("ran" :: Text), + "exitCode" Aeson..= resultExitCode result, + "provider" Aeson..= renderAgentProvider (result ^. #provider), + "durationSeconds" Aeson..= (realToFrac (result ^. #duration) :: Double) ] <> streamFields "stdout" (result ^. #stdout) <> streamFields "stderr" (result ^. #stderr) + <> evidenceField record ) - where - streamFields _ OutputNotCaptured = [] - streamFields label captured = - [ (label, jsonString (decoded captured)), - ( label <> "Truncated", - case captured of - OutputTruncated _ -> "true" - _ -> "false" - ) - ] +-- | The run's evidence record as one JSON field, or no field at all. +-- +-- Absent rather than @null@ when none was built, for the same reason a +-- stream that was never captured contributes nothing: a reader can tell +-- \"no record was asked for\" from \"a record exists and says nothing\". +-- 'ModelCallEvidence' has its own 'Aeson.ToJSON', which is the same +-- encoder @--evidence-file@ writes, so the two destinations cannot +-- drift. +evidenceField :: Maybe ModelCallEvidence -> [Pair] +evidenceField Nothing = [] +evidenceField (Just record) = ["evidence" Aeson..= record] + +-- | One captured stream as JSON fields: its text and whether it was cut +-- off at the configured output limit. +-- +-- A stream that was never captured contributes nothing rather than an +-- empty string, so a reader can tell \"the agent printed nothing\" from +-- \"the bytes went to the terminal and were never Baikai's to report\". +-- +-- Shared by a finished run and by a timed-out one, which carries the +-- same two streams. +streamFields :: Text -> AgentCapturedOutput -> [Pair] +streamFields _ OutputNotCaptured = [] +streamFields label captured = + [ Key.fromText label Aeson..= decoded captured, + Key.fromText (label <> "Truncated") + Aeson..= case captured of + OutputTruncated _ -> True + _ -> False + ] + -- -------------------------------------------------------------------- -- Prompts -- -------------------------------------------------------------------- @@ -1182,55 +1457,92 @@ promptSourceLabel (PromptInline _) = "--prompt" -- -------------------------------------------------------------------- --- A very small JSON writer +-- JSON -- -------------------------------------------------------------------- --- | Hand-rolled rather than pulled from @aeson@: the package needs --- exactly three shapes, and the alternative is a dependency the library --- otherwise has no use for. -jsonObject :: [(Text, Text)] -> Text -jsonObject fields = - "{" <> Text.intercalate "," [jsonString name <> ":" <> value | (name, value) <- fields] <> "}" +-- | One JSON value as a line of output. +-- +-- Every @--json@ path goes through this, so the encoder — and therefore +-- the escaping — is @aeson@'s in every case. The package already +-- depends on it to write evidence records. +encodeEnvelope :: Aeson.Value -> Text +encodeEnvelope value = Text.decodeUtf8Lenient (BSL.toStrict (Aeson.encode value)) <> "\n" -jsonArray :: [Text] -> Text -jsonArray values = "[" <> Text.intercalate "," values <> "]" +-- | The one shape @agent show --json@ emits, whatever happened. +-- +-- A reader should not have to know which of three failure modes it is +-- looking at before it can find the exit code: every field is always +-- present, and the ones that do not apply are @null@. That is why a +-- document that would not even parse still produces an object here, +-- where it used to produce a bare resolution report or nothing at all. +showJson :: + -- | The job that was asked about. + Text -> + -- | @shown@, @refused@, or @failed@. + Text -> + -- | The exit code the command is about to return. + Int -> + -- | The refusal or the configuration error, when there was one. + Maybe Text -> + -- | The resolution report, when resolution got far enough to produce + -- one. A failed resolution still carries provenance, and for an + -- explain command that is exactly what the operator needs. + Maybe ResolutionReport -> + -- | The ceiling and the file it came from, when one was loaded. + Maybe (Text, AgentCeiling) -> + -- | The rendered command, when one was rendered. + Maybe AgentCommand -> + Aeson.Value +showJson jobName outcomeWord code message report ceiling' command = + Aeson.object + [ "job" Aeson..= jobName, + "outcome" Aeson..= outcomeWord, + "exitCode" Aeson..= code, + "message" Aeson..= message, + "configuration" Aeson..= fmap reportJson report, + "ceiling" Aeson..= fmap (uncurry ceilingJson) ceiling', + "command" Aeson..= fmap commandJson command + ] -jsonString :: Text -> Text -jsonString value = "\"" <> Text.concatMap escape value <> "\"" +-- | @settei@'s own JSON rendering of a resolution report, as a value +-- rather than as text, so it nests inside the envelope instead of +-- appearing as an escaped string. +-- +-- The fallback keeps the text: a report that will not decode is still +-- worth showing, and refusing to emit an envelope over it would lose +-- the exit code too. +reportJson :: ResolutionReport -> Aeson.Value +reportJson report = + fromMaybe + (Aeson.String rendered) + (Aeson.decodeStrict (Text.encodeUtf8 rendered)) where - escape '"' = "\\\"" - escape '\\' = "\\\\" - escape '\n' = "\\n" - escape '\r' = "\\r" - escape '\t' = "\\t" - escape character - | isControl character = - "\\u" <> Text.justifyRight 4 '0' (Text.pack (showHex (ord character) "")) - | otherwise = Text.singleton character + rendered = renderResolutionJson report -ceilingJson :: Text -> AgentCeiling -> Text +ceilingJson :: Text -> AgentCeiling -> Aeson.Value ceilingJson sourceLabel ceiling' = - jsonObject - [ ("source", jsonString sourceLabel), - ("maxCapability", jsonString (renderAgentCapability (ceiling' ^. #maxCapability))), - ( "allowProviderArgs", - if ceiling' ^. #allowProviderArgs then "true" else "false" - ), - ( "allowedProviders", - jsonArray (map (jsonString . renderAgentProvider) (ceiling' ^. #allowedProviders)) - ) + Aeson.object + [ "source" Aeson..= sourceLabel, + "maxCapability" Aeson..= renderAgentCapability (ceiling' ^. #maxCapability), + "allowProviderArgs" Aeson..= (ceiling' ^. #allowProviderArgs), + "allowedProviders" + Aeson..= map renderAgentProvider (ceiling' ^. #allowedProviders), + "allowedTools" Aeson..= (ceiling' ^. #allowedTools), + -- Null rather than a word, because "unlimited" is the absence of a + -- bound and a reader branching on null needs no vocabulary. + "maxTimeout" + Aeson..= fmap (Text.pack . show) (ceiling' ^. #maxTimeout), + "maxOutputLimit" Aeson..= (ceiling' ^. #maxOutputLimit) ] -commandJson :: AgentCommand -> Text +commandJson :: AgentCommand -> Aeson.Value commandJson command = - jsonObject - [ ("executable", jsonString (Text.pack (command ^. #executable))), - ("arguments", jsonArray (map (jsonString . Text.pack) (command ^. #arguments))), - ( "promptTransport", - jsonString - ( case command ^. #promptTransport of - PromptOnStdin -> "stdin" - PromptAsArgument -> "argument" - ) - ) + Aeson.object + [ "executable" Aeson..= Text.pack (command ^. #executable), + "arguments" Aeson..= map Text.pack (command ^. #arguments), + "promptTransport" + Aeson..= ( case command ^. #promptTransport of + PromptOnStdin -> "stdin" :: Text + PromptAsArgument -> "argument" + ) ] diff --git a/src/Baikai/Agent/Config.hs b/src/Baikai/Agent/Config.hs --- a/src/Baikai/Agent/Config.hs +++ b/src/Baikai/Agent/Config.hs @@ -21,12 +21,29 @@ -- 'applyCeilingToJob' is where that ask is refused. module Baikai.Agent.Config ( -- * The configured shape of one job - AgentJob (..), + AgentJob + ( provider, + executable, + modelId, + effort, + workingDir, + extraDirs, + capability, + allowedTools, + providerArgs, + timeout, + output, + outputFormat, + outputLimit, + envRequires + ), + agentJob, agentJobConfig, agentJobRequest, -- * Where configuration lives - AgentConfigPaths (..), + AgentConfigPaths (userConfig, repoConfig, repositoryRoot), + emptyAgentConfigPaths, AgentConfigScope (..), renderAgentConfigScope, defaultAgentConfigPaths, @@ -38,6 +55,10 @@ agentCeilingConfig, loadAgentCeiling, applyCeilingToJob, + ceilingViolations, + repositoryScopeViolations, + relevantWarnings, + repositoryPolicyNotice, -- * Enumeration AgentJobEntry (..), @@ -59,25 +80,31 @@ import Baikai.Agent ( AgentCapability, AgentCeiling, + AgentOutputFormat (..), AgentOutputMode (..), AgentProvider, AgentRenderError (..), AgentRunRequest, AgentSafety, + CeilingViolation (..), agentRunRequest, agentSafety, applyAgentCeiling, + ceilingViolations, defaultAgentCeiling, parseAgentCapability, + parseAgentOutputFormat, parseAgentOutputMode, parseAgentProvider, renderAgentCapability, + renderAgentOutputFormat, renderAgentOutputMode, renderAgentProvider, ) -import Baikai.ThinkingLevel (ThinkingLevel (..), renderThinkingLevel) +import Baikai.ThinkingLevel (ThinkingLevel, parseThinkingLevel, renderThinkingLevel) import Control.Lens ((&), (.~), (^.)) import Data.Generics.Labels () +import Data.List (isPrefixOf) import Data.List.NonEmpty qualified as NonEmpty import Data.Map.Strict qualified as Map import Data.Set qualified as Set @@ -97,14 +124,16 @@ environmentSource, renderEnvErrorsText, ) +import Settei.Error (ConfigWarning (..), UnknownKeyProblem (..)) import Settei.Kdl ( kdlSourceOptions, readKdlSource, renderKdlErrorsText, withKdlSourcePath, ) -import Settei.Key (Key, keySegments, parseKey) +import Settei.Key (Key, keySegments, parseKey, renderKey) import Settei.Optparse (CliOverride, cliSources) +import Settei.Report (ResolutionReport, reportNodes) import Settei.Resolve (ResolveResult, defaultResolveOptions, resolve) import Settei.Setting ( Setting, @@ -121,14 +150,13 @@ boundedIntegralDecoder, decodeFailure, decoder, - enumDecoder, parsedDecoder, runDecoder, textDecoder, ) -import System.Directory (doesFileExist) +import System.Directory (canonicalizePath, doesFileExist, getCurrentDirectory) import System.Environment (lookupEnv) -import System.FilePath (()) +import System.FilePath (pathSeparator, ()) -- | The configured shape of one named job. -- @@ -141,6 +169,9 @@ -- There is deliberately no @name@ field. The name is how the job was -- looked up, not a property of it, and storing it would invite a -- mismatch between the two. +-- Construction: the constructor is deliberately not exported. Start +-- from 'agentJob', which takes the three required fields, and override +-- the rest by record update. data AgentJob = AgentJob { -- | Which coding-agent tool to run. Required. provider :: !AgentProvider, @@ -161,7 +192,8 @@ -- that forgot to state its authority must not silently receive -- some. capability :: !AgentCapability, - -- | Optional narrowing of the provider's tool set. + -- | Tools granted beyond what the capability implies. Bounded by + -- the operator's ceiling; see 'Baikai.Agent.toolGrantsImpliedBy'. allowedTools :: ![Text], -- | Raw provider arguments passed through verbatim. Classified -- __secret__, because it is the one field an operator could write a @@ -171,6 +203,8 @@ timeout :: !(Maybe NominalDiffTime), -- | What to do with the child's output streams. output :: !AgentOutputMode, + -- | What shape the tool should print its final answer in. + outputFormat :: !AgentOutputFormat, -- | Maximum captured bytes per stream. 'Nothing' means unbounded. outputLimit :: !(Maybe Int), -- | Names of environment variables the job declares it requires. @@ -186,13 +220,13 @@ -- @FileSource \"KDL v2\"@ — naming the /format/, not the file — so the -- user and repository documents are indistinguishable by kind. data AgentConfigScope - = UserScope - | RepositoryScope + = AgentUserScope + | AgentRepositoryScope deriving stock (Eq, Ord, Show, Generic) renderAgentConfigScope :: AgentConfigScope -> Text -renderAgentConfigScope UserScope = "user configuration" -renderAgentConfigScope RepositoryScope = "repository configuration" +renderAgentConfigScope AgentUserScope = "user configuration" +renderAgentConfigScope AgentRepositoryScope = "repository configuration" -- | One configured job name and the scope that supplied the winning -- definition of it. @@ -213,9 +247,24 @@ -- An explicit record with a pure resolution path underneath is what lets -- a test point at a temporary directory. Nothing below this record reads -- the real @HOME@ or @XDG_CONFIG_HOME@. +-- Construction: the constructor is deliberately not exported. Start +-- from 'emptyAgentConfigPaths' (or 'defaultAgentConfigPaths', which +-- discovers real paths) and override fields by record update. data AgentConfigPaths = AgentConfigPaths { userConfig :: !(Maybe FilePath), - repoConfig :: !(Maybe FilePath) + repoConfig :: !(Maybe FilePath), + -- | The repository the run is working in: the directory the process + -- was started in, which is also the directory + -- 'defaultAgentConfigPaths' looks for @.baikai\/agents.kdl@ under. + -- + -- @--config PATH@ chooses which /file/ supplies repository-scope + -- settings; it does not move the root. The root is what confines a + -- repository-supplied @working-dir@ and what refuses an operator + -- file that lies inside the checkout, and both properties would be + -- defeated by a flag that could move it. It is an explicit field + -- rather than a call to 'getCurrentDirectory' at each use so a test + -- can point it at a temporary directory. + repositoryRoot :: !FilePath } deriving stock (Eq, Show, Generic) @@ -230,6 +279,25 @@ ConfigFileUnreadable !FilePath !Text | -- | The job name, and why it cannot address a configuration key. InvalidJobName !Text !Text + | -- | The operator configuration file, then the repository root it + -- lies inside. + -- + -- The ceiling is the operator's limit on what an untrusted checkout + -- may ask for, so a ceiling file the checkout could have written is + -- not a ceiling. This is the one shape of that hole that can be + -- closed from inside the process: @--user-config .baikai\/policy.kdl@ + -- and @XDG_CONFIG_HOME=$PWD\/.baikai@ both name a path under the + -- checkout, and both are now refused. + CeilingFileInsideRepository !FilePath !FilePath + | -- | The operator configuration file, then every key under its + -- @policy@ node that the ceiling schema does not declare. + -- + -- Everywhere else an unrecognised key is a warning, because a + -- forward-compatible file should not stop an older binary. Under + -- @policy@ it is an error: this is the one node whose purpose is to + -- limit authority, and a misspelling that silently left the default + -- in force would give an operator a ceiling they did not write. + UnknownPolicySetting !FilePath ![Text] deriving stock (Eq, Show, Generic) renderAgentConfigError :: AgentConfigError -> Text @@ -237,6 +305,19 @@ "could not read the configuration file " <> Text.pack path <> ": " <> message renderAgentConfigError (InvalidJobName jobName why) = "invalid job name " <> jobName <> ": " <> why +renderAgentConfigError (CeilingFileInsideRepository path root) = + "the operator configuration file " + <> Text.pack path + <> " lies inside the repository " + <> Text.pack root + <> ", so the repository could have written the policy ceiling; move it \ + \outside the checkout or pass --user-config with a path outside it" +renderAgentConfigError (UnknownPolicySetting path keys) = + "the operator configuration file " + <> Text.pack path + <> " sets a policy key that does not exist: " + <> Text.intercalate ", " keys + <> "; a misspelled ceiling key would silently leave the default in force" -- | Bytes per stream captured when no layer states a limit. -- @@ -338,19 +419,20 @@ "one of: inherit, capture, tee" (maybe (Left "unknown output mode") Right . parseAgentOutputMode) --- | The six canonical reasoning-effort names. The list lives in --- @baikai\/src\/Baikai\/ThinkingLevel.hs@, which has a renderer but no --- parser, so a future level must be added in both places. +outputFormatDecoder :: Decoder AgentOutputFormat +outputFormatDecoder = + parsedDecoder + "one of: text, json" + (maybe (Left "unknown output format") Right . parseAgentOutputFormat) + +-- | The canonical reasoning-effort names, read through +-- 'Baikai.ThinkingLevel.parseThinkingLevel' so this file cannot fall +-- behind the level set it names. effortDecoder :: Decoder ThinkingLevel effortDecoder = - enumDecoder - [ ("minimal", ThinkingMinimal), - ("low", ThinkingLow), - ("medium", ThinkingMedium), - ("high", ThinkingHigh), - ("xhigh", ThinkingXHigh), - ("max", ThinkingMax) - ] + parsedDecoder + "one of: minimal, low, medium, high, xhigh, max" + (maybe (Left "unknown effort") Right . parseThinkingLevel) pathDecoder :: Decoder FilePath pathDecoder = fmap Text.unpack textDecoder @@ -380,6 +462,16 @@ Right (bytes :: Int) | bytes > 0 -> Right (Just bytes) _ -> Left (decodeFailure key "a positive number of bytes, or the word unlimited") +-- | A duration, or the word @unlimited@ for no bound at all. +-- +-- The same spelling @output-limit@ uses for its own no-bound case, so an +-- operator learns one word rather than two. +maxTimeoutDecoder :: Decoder (Maybe NominalDiffTime) +maxTimeoutDecoder = + decoder $ \key raw -> case raw of + RawText "unlimited" -> Right Nothing + _ -> fmap Just (runDecoder durationDecoder key raw) + providerSetting :: Text -> Setting AgentProvider providerSetting jobName = publicSettingWithRenderer @@ -429,11 +521,17 @@ capabilityDecoder renderAgentCapability +-- | Tools the run is granted beyond what its capability implies. +-- +-- A grant, not a narrowing: on Claude Code this becomes +-- @--allowedTools@, which pre-approves the named tools. The ceiling +-- bounds it, so a repository asking for @Bash@ under @edit-workspace@ is +-- refused until the operator writes @policy.allowed-tools@. allowedToolsSetting :: Text -> Setting [Text] allowedToolsSetting jobName = publicShowSetting (jobKey jobName "safety.allowed-tools") - "Narrowing of the provider's tool set" + "Tools granted beyond what the capability implies" (scalarOrListDecoder textDecoder) -- | Raw provider arguments, classified __secret__. @@ -469,6 +567,14 @@ outputModeDecoder renderAgentOutputMode +outputFormatSetting :: Text -> Setting AgentOutputFormat +outputFormatSetting jobName = + publicSettingWithRenderer + (jobKey jobName "output-format") + "Shape the tool should print its final answer in" + outputFormatDecoder + renderAgentOutputFormat + outputLimitSetting :: Text -> Setting (Maybe Int) outputLimitSetting jobName = publicShowSetting @@ -487,6 +593,35 @@ emptyListDefault ruleName = constantDefault (RuleName ruleName) "nothing configured" [] +-- | A job from its three required fields: which tool to run, the +-- directory to run it in, and how much filesystem authority it asks +-- for. +-- +-- Every optional field starts empty and every defaulted one at the +-- value the KDL schema would supply for an absent setting — no +-- executable override, no model or effort override, no extra +-- directories, no tool grants, no provider arguments, no timeout, +-- 'InheritOutput', 'TextFormat', 'defaultOutputLimit' and no required +-- environment. Override by record update. +agentJob :: AgentProvider -> FilePath -> AgentCapability -> AgentJob +agentJob jobProvider jobWorkingDir jobCapability = + AgentJob + { provider = jobProvider, + executable = Nothing, + modelId = Nothing, + effort = Nothing, + workingDir = jobWorkingDir, + extraDirs = [], + capability = jobCapability, + allowedTools = [], + providerArgs = [], + timeout = Nothing, + output = InheritOutput, + outputFormat = TextFormat, + outputLimit = Just defaultOutputLimit, + envRequires = [] + } + -- | The declaration for one named job. -- -- The keys are built from the job name because @settei@'s 'Config' @@ -511,13 +646,20 @@ <*> required (workingDirSetting jobName) <*> withDefault (extraDirsSetting jobName) (emptyListDefault "no-extra-dirs") <*> required (capabilitySetting jobName) - <*> withDefault (allowedToolsSetting jobName) (emptyListDefault "no-tool-restriction") + <*> withDefault (allowedToolsSetting jobName) (emptyListDefault "no-tool-grants") <*> withDefault (providerArgsSetting jobName) (emptyListDefault "no-provider-args") <*> optional (timeoutSetting jobName) <*> withDefault (outputSetting jobName) (constantDefault (RuleName "inherit-output") "no output discipline configured" InheritOutput) <*> withDefault + (outputFormatSetting jobName) + ( constantDefault + (RuleName "text-output-format") + "no output format configured" + TextFormat + ) + <*> withDefault (outputLimitSetting jobName) ( constantDefault (RuleName "default-output-limit") @@ -542,8 +684,9 @@ & #safety .~ requestedSafety & #timeout .~ (job ^. #timeout) & #output .~ (job ^. #output) + & #outputFormat .~ (job ^. #outputFormat) & #outputLimit .~ (job ^. #outputLimit) - & #envPassthrough .~ (job ^. #envRequires) + & #envRequires .~ (job ^. #envRequires) where requestedSafety :: AgentSafety requestedSafety = @@ -553,12 +696,16 @@ -- | Environment variables that may influence a job, bound explicitly. -- --- The set is small and deliberately chosen: the provider, the model, the --- executable, and the timeout. The capability, the tool list, and the --- raw provider arguments are __not__ bound. An environment variable is --- easy to set accidentally and is inherited by every child process, so --- letting one widen a job's authority would create exactly the ambient --- influence the ceiling exists to prevent. +-- The set is small and deliberately chosen: the provider, the model, and +-- the timeout. The capability, the tool list, the raw provider +-- arguments, and __the executable__ are not bound. An environment +-- variable is easy to set accidentally and is inherited by every child +-- process, so letting one widen a job's authority would create exactly +-- the ambient influence the ceiling exists to prevent — and naming the +-- program to run is the widest widening there is, since the named +-- program inherits the operator's environment and receives the prompt on +-- its standard input. An operator whose installation is not on @PATH@ +-- writes @executable@ in their own file or passes @--set@. -- -- The binding list is a function of the job name rather than a module -- constant, because every key contains the name. Forcing it for any name @@ -572,11 +719,24 @@ ( bindings [ binding (EnvName "BAIKAI_AGENT_PROVIDER") (jobKey jobName "provider"), binding (EnvName "BAIKAI_AGENT_MODEL") (jobKey jobName "model"), - binding (EnvName "BAIKAI_AGENT_EXECUTABLE") (jobKey jobName "executable"), binding (EnvName "BAIKAI_AGENT_TIMEOUT") (jobKey jobName "timeout") ] ) +-- | No configuration files, rooted at the current directory as a +-- relative path. +-- +-- The starting point for a caller who supplies explicit paths; +-- 'defaultAgentConfigPaths' is the one that discovers them and resolves +-- the repository root. +emptyAgentConfigPaths :: AgentConfigPaths +emptyAgentConfigPaths = + AgentConfigPaths + { userConfig = Nothing, + repoConfig = Nothing, + repositoryRoot = "." + } + -- | Locate the two configuration files, treating a missing file as a -- normal state rather than an error. -- @@ -600,7 +760,8 @@ userPath = fmap (\base -> base "baikai" "agents.kdl") configBase userConfig <- maybe (pure Nothing) whenPresent userPath repoConfig <- whenPresent (".baikai" "agents.kdl") - pure AgentConfigPaths {userConfig, repoConfig} + repositoryRoot <- getCurrentDirectory + pure AgentConfigPaths {userConfig, repoConfig, repositoryRoot} where nonEmptyPath dir = if null dir then Nothing else Just dir whenPresent path = do @@ -623,12 +784,12 @@ -- scope that produced it. loadScopeSources :: AgentConfigPaths -> IO (Either AgentConfigError [(AgentConfigScope, Source)]) loadScopeSources paths = do - userLoaded <- loadScope UserScope (paths ^. #userConfig) - repoLoaded <- loadScope RepositoryScope (paths ^. #repoConfig) + userLoaded <- loadScope AgentUserScope (paths ^. #userConfig) + repoLoaded <- loadScope AgentRepositoryScope (paths ^. #repoConfig) pure $ do userSources <- userLoaded repoSources <- repoLoaded - Right (map ((,) UserScope) userSources <> map ((,) RepositoryScope) repoSources) + Right (map ((,) AgentUserScope) userSources <> map ((,) AgentRepositoryScope) repoSources) -- | Resolve one named job across all five layers. -- @@ -693,12 +854,36 @@ "no operator policy configured" (defaultAgentCeiling ^. #allowedProviders) ) + <*> withDefault + allowedToolsCeilingSetting + ( constantDefault + (RuleName "default-allowed-tool-grants") + "no operator policy configured" + (defaultAgentCeiling ^. #allowedTools) + ) + <*> withDefault + maxTimeoutSetting + ( constantDefault + (RuleName "default-max-timeout") + "no operator policy configured" + (defaultAgentCeiling ^. #maxTimeout) + ) + <*> withDefault + maxOutputLimitSetting + ( constantDefault + (RuleName "default-max-output-limit") + "no operator policy configured" + (defaultAgentCeiling ^. #maxOutputLimit) + ) where - buildCeiling cap rawArgs providers = + buildCeiling cap rawArgs providers grants timeoutMax outputMax = defaultAgentCeiling & #maxCapability .~ cap & #allowProviderArgs .~ rawArgs & #allowedProviders .~ providers + & #allowedTools .~ grants + & #maxTimeout .~ timeoutMax + & #maxOutputLimit .~ outputMax maxCapabilitySetting = publicSettingWithRenderer (validKey "policy.max-capability") @@ -715,6 +900,21 @@ (validKey "policy.allowed-providers") "Providers jobs may select" (scalarOrListDecoder providerDecoder) + allowedToolsCeilingSetting = + publicShowSetting + (validKey "policy.allowed-tools") + "Tool grants permitted beyond the ones the maximum capability implies" + (scalarOrListDecoder textDecoder) + maxTimeoutSetting = + publicShowSetting + (validKey "policy.max-timeout") + "Longest wall-clock limit any job may request" + maxTimeoutDecoder + maxOutputLimitSetting = + publicShowSetting + (validKey "policy.max-output-limit") + "Largest per-stream capture any job may request" + outputLimitDecoder -- | Load the operator's policy ceiling. -- @@ -729,28 +929,72 @@ -- would add. Someone \"fixing an inconsistency\" by adding the -- repository source here would silently remove the security property -- while every test that does not specifically check it kept passing. +-- Refusing a file inside the repository root, below, is the other half of +-- the same property: a file the repository could have written is not the +-- operator's, whichever flag or variable named it. -- -- With no user file the ceiling is 'defaultAgentCeiling': read-only and --- edit-workspace are permitted, full access is refused, and raw provider --- arguments are refused. +-- edit-workspace are permitted, full access is refused, raw provider +-- arguments are refused, no tool grant beyond the capability is +-- permitted, a run may be untimed, and a capture is bounded at +-- 'Baikai.Agent.defaultMaxOutputLimit'. loadAgentCeiling :: AgentConfigPaths -> IO (Either AgentConfigError AgentCeiling) loadAgentCeiling paths = do - userLoaded <- loadScope UserScope (paths ^. #userConfig) - pure $ do - userSources <- userLoaded - let resolved = resolve defaultResolveOptions userSources agentCeilingConfig - case resolved ^. #answer of - Left problems -> - Left - ( ConfigFileUnreadable - (maybe "" id (paths ^. #userConfig)) - (renderCeilingProblems problems) - ) - Right ceiling' -> Right ceiling' + located <- ceilingFileLocation paths + case located of + Just problem -> pure (Left problem) + Nothing -> do + userLoaded <- loadScope AgentUserScope (paths ^. #userConfig) + pure $ do + userSources <- userLoaded + let resolved = resolve defaultResolveOptions userSources agentCeilingConfig + case unknownPolicyKeys (resolved ^. #warnings) of + keys@(_ : _) -> Left (UnknownPolicySetting ceilingFileLabel keys) + [] -> case resolved ^. #answer of + Left problems -> + Left (ConfigFileUnreadable ceilingFileLabel (renderCeilingProblems problems)) + Right ceiling' -> Right ceiling' where + ceilingFileLabel = maybe "" id (paths ^. #userConfig) renderCeilingProblems problems = Text.intercalate "; " (map (Text.pack . show) (NonEmpty.toList problems)) +-- | Refuse an operator configuration file that lies inside the +-- repository, naming both paths. +-- +-- Only a file that exists is checked, so the normal case — no operator +-- file at all — costs nothing, and a path that names nothing is reported +-- by the read instead. Both sides are canonicalised, so a symbolic link +-- into the checkout is caught too. +ceilingFileLocation :: AgentConfigPaths -> IO (Maybe AgentConfigError) +ceilingFileLocation paths = case paths ^. #userConfig of + Nothing -> pure Nothing + Just path -> do + present <- doesFileExist path + if not present + then pure Nothing + else do + canonicalFile <- canonicalizePath path + canonicalRoot <- canonicalizePath (paths ^. #repositoryRoot) + pure + ( if isInside canonicalRoot canonicalFile + then Just (CeilingFileInsideRepository canonicalFile canonicalRoot) + else Nothing + ) + +-- | Every unrecognised key under the @policy@ node, rendered. +-- +-- Warnings about anything else are dropped rather than reported: the +-- operator file also holds job defaults, and the ceiling schema declares +-- none of them, so every @jobs@ key in it warns here and none of them is +-- a mistake. +unknownPolicyKeys :: [ConfigWarning] -> [Text] +unknownPolicyKeys warnings = + [ renderKey key + | UnknownKeyWarning UnknownKeyProblem {key} <- warnings, + NonEmpty.head (keySegments key) == "policy" + ] + -- | Refuse a request that exceeds the operator's ceiling. -- -- The comparison itself is 'applyAgentCeiling', which is pure and @@ -764,6 +1008,139 @@ case applyAgentCeiling ceiling' request of Left violations -> Left (CeilingRejected violations) Right permitted -> Right permitted + +-- | The violations that depend on __which file__ supplied a value. +-- +-- 'Baikai.Agent.applyAgentCeiling' cannot produce these: it sees a +-- request, and a request carries no record of where each field came +-- from. This function reads that record from the resolution report and +-- returns violations in the same vocabulary, so a caller concatenates +-- the two lists and reports one refusal naming everything at once. +-- +-- Three settings are treated specially, and the reasoning is the same in +-- each case — the repository file is untrusted input, so it may not +-- choose things that reach outside itself. +-- +-- @executable@ names the program to run. A repository that could set it +-- turns a configuration file into code execution with the operator's +-- environment and the prompt on standard input; an operator whose +-- installation is not on @PATH@ sets it in their own file or with +-- @--set@. +-- +-- @extra-dirs@ names directories the run may reach __beyond__ its +-- working directory. Inside the repository it adds nothing the working +-- directory does not already give, so the only extra directory a +-- repository would ask for is one outside the checkout — the operator's +-- grant to make. An empty resolved list is not a violation, so a +-- repository may write @extra-dirs@ with no arguments. +-- +-- @working-dir@ must resolve inside 'repositoryRoot'. Canonicalising +-- first is what defeats a committed symbolic link: @work -> \/@ would +-- otherwise pass a textual prefix check. +repositoryScopeViolations :: + AgentConfigPaths -> + ResolutionReport -> + -- | The job the report resolved. + Text -> + AgentJob -> + IO [CeilingViolation] +repositoryScopeViolations paths report jobName job = do + workingDirViolations <- + if setByRepository "working-dir" + then confineWorkingDir (paths ^. #repositoryRoot) (job ^. #workingDir) + else pure [] + pure + ( concat + [ [RepositoryScopeForbidden "executable" | setByRepository "executable"], + [ RepositoryScopeForbidden "extra-dirs" + | setByRepository "extra-dirs", + not (null (job ^. #extraDirs)) + ], + workingDirViolations + ] + ) + where + setByRepository leaf = any (matches leaf) (reportNodes report) + matches leaf node = + node ^. #key == jobKey jobName leaf + && maybe + False + ((== renderAgentConfigScope AgentRepositoryScope) . (^. #name)) + (node ^. #origin) + +-- | Refuse a working directory that resolves outside the repository +-- root, naming both the resolved directory and the root. +-- +-- An absolute @working-dir@ passes through @\<\/\>@ unchanged, so one +-- expression covers both spellings, and a relative one resolves against +-- the root rather than the process directory — which makes +-- @working-dir \"\.\"@ mean the repository whichever file declared it. +-- +-- @canonicalizePath@ resolves the longest prefix that exists, so a +-- working directory that does not exist yet still gets an answer here +-- and is reported later, by the runner, as 'WorkingDirMissing'. +confineWorkingDir :: FilePath -> FilePath -> IO [CeilingViolation] +confineWorkingDir root workingDir = do + canonicalRoot <- canonicalizePath root + resolved <- canonicalizePath (root workingDir) + pure + [ WorkingDirOutsideRepository resolved canonicalRoot + | not (isInside canonicalRoot resolved) + ] + +-- | Whether a canonical path is the given canonical directory or lies +-- under it. Both arguments must already be canonical; comparing +-- uncanonicalised paths would let a symbolic link through. +isInside :: FilePath -> FilePath -> Bool +isInside base path = path == base || (base <> [pathSeparator]) `isPrefixOf` path + +-- | The unknown-key warnings a run of one named job should show. +-- +-- @settei@ warns about every addressable leaf its declaration does not +-- recognise, and the declaration here describes __one__ job, so +-- resolving @demo@ in a document that also defines @release@ warns about +-- every key of @release@ — none of which is a mistake, and none of which +-- the operator asked about. The operator file's @policy@ node warns for +-- the same reason: the job declaration does not describe the ceiling. +-- +-- Filtering here rather than asking @settei@ for a per-key policy is not +-- a workaround. Its only resolver option is @unknownKeyPolicy@, which is +-- @WarnUnknownKeys@ or @RejectUnknownKeys@ for the whole resolution; +-- there is no per-key setting to ask for. Rebuilding a filtered source +-- before resolving would duplicate the adapter and lose the annotations +-- provenance depends on. +-- +-- A misspelled key /inside the selected job/ still warns, which is the +-- case that matters: it is the one that silently leaves a default in +-- force. +relevantWarnings :: Text -> [ConfigWarning] -> [ConfigWarning] +relevantWarnings jobName = filter keep + where + keep (UnknownKeyWarning UnknownKeyProblem {key}) = + case NonEmpty.toList (keySegments key) of + "jobs" : name : _ -> name == jobName + "policy" : _ -> False + -- A stray top-level node belongs to neither and is probably a + -- mistake, so it is kept. + _ -> True + +-- | One notice when the __repository__ document carries a @policy@ node. +-- +-- A checkout writing a ceiling is not an error — the ceiling is read +-- from the operator file only, so the node simply does nothing — but +-- someone who wrote it believed it would, and saying nothing leaves them +-- with a policy they think is in force. One line, not one per key. +repositoryPolicyNotice :: [ConfigWarning] -> Maybe Text +repositoryPolicyNotice warnings + | any fromRepositoryPolicy warnings = + Just + "the repository configuration contains a policy node; it has no effect, \ + \because the ceiling is read from the operator file only\n" + | otherwise = Nothing + where + fromRepositoryPolicy (UnknownKeyWarning UnknownKeyProblem {key, origin}) = + NonEmpty.head (keySegments key) == "policy" + && origin ^. #name == renderAgentConfigScope AgentRepositoryScope -- | Every configured job name, sorted, each attributed to the -- highest-precedence scope defining it. diff --git a/src/Baikai/Agent/Run.hs b/src/Baikai/Agent/Run.hs --- a/src/Baikai/Agent/Run.hs +++ b/src/Baikai/Agent/Run.hs @@ -23,6 +23,10 @@ agentRequestEnvelope, agentConfigurationEnvelope, + -- * Evidence detail + errorInfoStderrTailBytes, + executableForEvidence, + -- * Exposed for testing timeoutMicros, ) @@ -38,11 +42,13 @@ AgentRunOutcome (..), AgentRunRequest, AgentRunResult, + AgentTimedOut (..), agentRunOutcome, agentRunResult, renderAgentProvider, renderAgentRunFailure, ) +import Baikai.Agent qualified as Agent import Baikai.Api (Api (..)) import Baikai.Error (BaikaiError, processError, providerError) import Baikai.Evidence @@ -72,7 +78,7 @@ subprocessStrength, ) import Baikai.Usage (Usage) -import Control.Concurrent (forkIO) +import Control.Concurrent (ThreadId, forkIO, killThread, myThreadId, threadDelay) import Control.Concurrent.MVar (MVar, newEmptyMVar, putMVar, takeMVar) import Control.Exception ( SomeAsyncException (..), @@ -83,12 +89,12 @@ try, ) import Control.Lens ((&), (.~), (^.)) -import Control.Monad (void) +import Control.Monad (unless, void) import Data.Aeson (Value) import Data.Aeson qualified as Aeson import Data.ByteString qualified as BS import Data.Generics.Labels () -import Data.Maybe (fromMaybe) +import Data.Maybe (fromMaybe, isJust) import Data.Text (Text) import Data.Text qualified as Text import Data.Text.Encoding qualified as Text @@ -97,7 +103,9 @@ import System.Directory (doesDirectoryExist) import System.Environment (lookupEnv) import System.Exit (ExitCode (..)) -import System.IO (Handle, hClose, hFlush, stderr, stdout) +import System.FilePath (isPathSeparator, isRelative, ()) +import System.IO (Handle, hClose, hFlush) +import System.IO qualified as SystemIO import System.Process qualified as P import System.Timeout qualified as Timeout #if defined(BAIKAI_POSIX_SIGNALS) @@ -120,8 +128,9 @@ -- * working directory absent — @Left WorkingDirMissing@, nothing spawned -- * a declared variable unset or empty — @Left MissingEnvironment@ -- * executable not startable — @Left SpawnFailed@ --- * still running at the deadline — @Left RunTimedOut@, whole process --- group terminated +-- * still running at the deadline — @Left RunTimedOut@, whose whole +-- process group was interrupted, then terminated, then killed, and +-- which carries the output drained before the kill -- -- __Evidence.__ The first argument is the caller's request for a -- verifiable record of the run. 'Nothing' means they want none, which is @@ -141,9 +150,11 @@ -- __What evidence from this surface can and cannot prove.__ The tool's -- own session identifier, model, and token counts are read out of -- captured standard output, so they are available only when the run --- captured output /and/ the tool was configured to print a structured --- format — @--output-format json@ for @claude@, @--json@ for --- @codex exec@, neither of which the vendor renderers emit by default. +-- captured output /and/ asked the tool for a structured format — +-- 'Baikai.Agent.JsonFormat', which the vendor renderers turn into +-- @--output-format json@ for @claude@ and @--json@ for @codex exec@. +-- Neither is the default, because a structured stream is not what an +-- operator watching a run wants to read. -- Under 'InheritOutput' there is nothing to read at all. Absent those, -- every tool-reported field stays 'Unobserved' and the strength stays at -- 'Baikai.Evidence.EvidenceRequestedOnly'. An operator who needs @@ -176,7 +187,7 @@ if not dirExists then pure (agentRunOutcome (Left (WorkingDirMissing (req ^. #workingDir)))) else do - missing <- missingEnvironment (req ^. #envPassthrough) + missing <- missingEnvironment (req ^. #envRequires) if not (null missing) then pure (agentRunOutcome (Left (MissingEnvironment missing))) else do @@ -324,7 +335,7 @@ Nothing -> pure Nothing Just (st, failure) -> do cid <- newCallId - identity <- executableIdentity (cmd ^. #executable) + identity <- executableIdentity (executableForEvidence req cmd) (session, model, tokens) <- observeToolOutput (req ^. #provider) result pure . Just $ baseEvidence @@ -363,9 +374,63 @@ ExitFailure code -> Just ( CallFailed, - Just (processError code (capturedText (ran ^. #stderr))) + Just (processError code (stderrTail (ran ^. #stderr))) ) +-- | How much of a failed run's standard error the evidence record +-- keeps. +-- +-- Four kibibytes is a few dozen lines, which is where a failing tool's +-- actual reason lives; the bytes before it are the transcript of the +-- work, which the record is not the place for. Without a bound the +-- record embeds whatever the output limit allowed — up to four mebibytes +-- by default, and more if the operator raised it — in a field a reader +-- expects to be one message. +errorInfoStderrTailBytes :: Int +errorInfoStderrTailBytes = 4096 + +-- | The tail of a captured stream, as text, prefixed with what was +-- dropped. +-- +-- The prefix matters: a message that silently begins mid-sentence reads +-- as a corrupted record rather than a bounded one. Truncation is by +-- bytes and the decode is lenient, so a multi-byte character split at +-- the boundary becomes a replacement character rather than a decode +-- failure. +stderrTail :: AgentCapturedOutput -> Text +stderrTail captured = case Agent.capturedBytes captured of + Nothing -> "" + Just bytes + | dropped <= 0 -> Text.decodeUtf8Lenient bytes + | otherwise -> + "[stderr truncated to the last " + <> Text.pack (show errorInfoStderrTailBytes) + <> " of " + <> Text.pack (show (BS.length bytes)) + <> " bytes] " + <> Text.decodeUtf8Lenient (BS.drop dropped bytes) + where + dropped = BS.length bytes - errorInfoStderrTailBytes + +-- | The path the child actually execs. +-- +-- A relative path containing a separator is resolved by the operating +-- system against the process's working directory, and the runner sets +-- that to the request's @workingDir@ — so a job whose @executable@ is +-- @.\/bin\/agent@ runs @\\/bin\/agent@. The evidence probe +-- runs in the /parent/, whose working directory is somewhere else +-- entirely, so it has to resolve the same way or it reports a path that +-- does not exist. +-- +-- A bare name with no separator is resolved on @PATH@ by both, and an +-- absolute path is absolute for both, so neither needs adjusting. +executableForEvidence :: AgentRunRequest -> AgentCommand -> FilePath +executableForEvidence req cmd + | any isPathSeparator exe && isRelative exe = (req ^. #workingDir) exe + | otherwise = exe + where + exe = cmd ^. #executable + -- | Where the run went, recorded without a URL, because there is no -- request and no server. -- @@ -403,12 +468,12 @@ -- | What the tool said about its own run, read out of captured standard -- output. -- --- Best-effort by necessity. Neither vendor renderer asks its tool for a --- structured format — @claude@ gets no @--output-format json@ and --- @codex exec@ gets no @--json@ — because that would change what an --- operator watching the run sees. So this parses if the operator --- configured one through the job's extra arguments, and reports honest --- silence if not. Under 'InheritOutput' there are no bytes at all. +-- Best-effort by necessity. A job asks for a structured format by +-- setting 'Baikai.Agent.JsonFormat' — @output-format \"json\"@ in a +-- configuration file — and the default is the tool's own text, because +-- a structured stream would change what an operator watching the run +-- sees. So this parses what it recognises and reports honest silence +-- otherwise. Under 'InheritOutput' there are no bytes at all. -- -- Truncated output is still parsed: @codex@\'s newline-delimited stream -- yields every complete line, which is more than nothing. Only the @@ -468,22 +533,21 @@ responseCommitmentOf _ _ _ = Unobserved -- | The standard output bytes a run captured, complete or truncated. +-- +-- A timed-out run has bytes too: the group was killed, but whatever it +-- printed before that was drained and kept. Reading them here is what +-- lets a codex run cut off mid-stream still yield the thread identifier +-- from the complete lines it managed to print, which is what +-- 'observeToolOutput' promises. capturedBytes :: Either AgentRunFailure AgentRunResult -> Maybe BS.ByteString capturedBytes = \case Right ran -> case ran ^. #stdout of OutputCaptured bytes -> Just bytes OutputTruncated bytes -> Just bytes OutputNotCaptured -> Nothing + Left (RunTimedOut timedOut) -> Agent.capturedBytes (timedOut ^. #stdout) Left _ -> Nothing --- | Captured bytes as text, for an error message. Absent capture yields --- the empty string rather than a claim about what the tool said. -capturedText :: AgentCapturedOutput -> Text -capturedText = \case - OutputCaptured bytes -> Text.decodeUtf8Lenient bytes - OutputTruncated bytes -> Text.decodeUtf8Lenient bytes - OutputNotCaptured -> "" - spawn :: AgentRunRequest -> AgentCommand -> IO (Either AgentRunFailure AgentRunResult) spawn req cmd = do @@ -498,7 +562,7 @@ P.create_group = True, -- The child inherits the parent's environment in full. Both -- tools need HOME, PATH, and their own credential files; - -- envPassthrough is a precondition check, not a filter. + -- envRequires is a precondition check, not a filter. P.env = Nothing } stdinSpec = case cmd ^. #promptTransport of @@ -549,20 +613,33 @@ -- that the timeout below can fire while a stream is still open. A -- drain on this thread would block until the child exits, which would -- make the timeout unreachable in the two capturing modes. - outVar <- forkDrain limit teeOut mOut - errVar <- forkDrain limit teeErr mErr + outDrain <- forkDrain limit teeOut mOut + errDrain <- forkDrain limit teeErr mErr waited <- waitWithTimeout (timeoutMicros (req ^. #timeout)) ph case waited of Nothing -> do terminateGroup ph + capturedOut <- collect outDrain + capturedErr <- collect errDrain -- Report the configured limit rather than the measured elapsed -- time: the caller asked for a limit and wants to be told which -- one was hit, and the elapsed time is slightly larger because of - -- the grace period. - pure (Left (RunTimedOut (maybe 0 id (req ^. #timeout)))) + -- the grace period. The drained bytes travel with it: the runner + -- already holds them, and a timed-out run is the one an operator + -- most wants an account of. + pure + ( Left + ( RunTimedOut + AgentTimedOut + { limit = fromMaybe 0 (req ^. #timeout), + stdout = capturedOut, + stderr = capturedErr + } + ) + ) Just code -> do - capturedOut <- takeMVar outVar - capturedErr <- takeMVar errVar + capturedOut <- takeMVar (snd outDrain) + capturedErr <- takeMVar (snd errDrain) end <- getCurrentTime pure ( Right @@ -574,9 +651,26 @@ where limit = req ^. #outputLimit (teeOut, teeErr) = case req ^. #output of - TeeOutput -> (Just stdout, Just stderr) + -- Qualified: 'AgentTimedOut' has fields of the same two names, + -- and this module constructs one. + TeeOutput -> (Just SystemIO.stdout, Just SystemIO.stderr) InheritOutput -> (Nothing, Nothing) CaptureOutput -> (Nothing, Nothing) + -- Take what a drain has after the group has been killed. + -- + -- Killing the group closes the write end of each pipe, so the drain + -- reaches end of file and answers on its own — unless a process + -- outside the group inherited the pipe and still holds it open, in + -- which case the drain is blocked on a read that will never return. + -- One grace period is allowed for the ordinary case, then the drain + -- is interrupted and answers with what it has. + -- + -- Interrupted rather than closed: a thread blocked in 'BS.hGetSome' + -- holds the handle's own lock for the whole read, so an 'hClose' + -- from here would block on that lock instead of ending the read. + collect (tid, var) = + Timeout.timeout gracePeriodMicros (takeMVar var) + >>= maybe (killThread tid >> takeMVar var) pure -- | Write the prompt and close the handle, on its own thread. -- @@ -602,23 +696,38 @@ -- | Drain one stream on its own thread, delivering the result through -- an 'MVar'. A stream that was inherited rather than piped has no -- handle and yields 'OutputNotCaptured' immediately. +-- +-- The thread identifier comes back with the 'MVar' because the timeout +-- path may have to interrupt a drain that is blocked on a pipe a +-- survivor outside the process group still holds open; 'consume' does +-- that in its @collect@. For an inherited stream there is no drain +-- thread, so the caller's own identifier is returned beside an +-- already-full 'MVar' — @collect@ never reaches the interrupt for a +-- variable it can take immediately, so that identifier is never used. forkDrain :: - Maybe Int -> Maybe Handle -> Maybe Handle -> IO (MVar AgentCapturedOutput) + Maybe Int -> + Maybe Handle -> + Maybe Handle -> + IO (ThreadId, MVar AgentCapturedOutput) forkDrain limit tee source = do var <- newEmptyMVar case source of - Nothing -> putMVar var OutputNotCaptured - Just h -> - void . forkIO $ do - -- A plain 'try' rather than 'trySync' here on purpose: nothing - -- delivers an asynchronous exception to this thread, and - -- re-throwing one would leave the MVar empty and hang whoever - -- takes it. A drain that fails yields no capture rather than - -- propagating; the handles are closed under us on timeout, and - -- that is a normal end rather than an error. + Nothing -> do + putMVar var OutputNotCaptured + here <- myThreadId + pure (here, var) + Just h -> do + -- A plain 'try' rather than 'trySync' here on purpose: an + -- asynchronous exception re-thrown from this thread would leave + -- the MVar empty and hang whoever takes it. A drain that fails + -- yields no capture rather than propagating; the handles are + -- closed under us on timeout, and that is a normal end rather + -- than an error. An interrupt aimed at the read itself is caught + -- inside 'drain', which answers with the bytes it already has. + tid <- forkIO $ do result <- try (drain limit tee h) :: IO (Either SomeException AgentCapturedOutput) putMVar var (either (const OutputNotCaptured) id result) - pure var + pure (tid, var) -- | Read a stream to the end, retaining at most the byte limit. -- @@ -635,23 +744,32 @@ drain limit tee h = go [] 0 False where go chunks retained dropped = do - chunk <- BS.hGetSome h chunkSize - if BS.null chunk - then - let bytes = BS.concat (reverse chunks) - in pure (if dropped then OutputTruncated bytes else OutputCaptured bytes) - else do - echo chunk - case limit of - Nothing -> go (chunk : chunks) (retained + BS.length chunk) dropped - Just cap - | retained >= cap -> go chunks retained True - | otherwise -> do - let kept = BS.take (cap - retained) chunk - go - (kept : chunks) - (retained + BS.length kept) - (dropped || BS.length kept < BS.length chunk) + -- A read that fails ends the drain with what it already has + -- rather than with nothing. Two things end a read this way and + -- neither is an error: the handle goes away when the child's side + -- of the pipe is closed under us, and the timeout path interrupts + -- a drain still blocked on a pipe a survivor holds open. Either + -- way more output may have existed, so the answer is truncated. + attempt <- try (BS.hGetSome h chunkSize) :: IO (Either SomeException BS.ByteString) + case attempt of + Left _ -> pure (OutputTruncated (BS.concat (reverse chunks))) + Right chunk -> + if BS.null chunk + then + let bytes = BS.concat (reverse chunks) + in pure (if dropped then OutputTruncated bytes else OutputCaptured bytes) + else do + echo chunk + case limit of + Nothing -> go (chunk : chunks) (retained + BS.length chunk) dropped + Just cap + | retained >= cap -> go chunks retained True + | otherwise -> do + let kept = BS.take (cap - retained) chunk + go + (kept : chunks) + (retained + BS.length kept) + (dropped || BS.length kept < BS.length chunk) -- Flush per chunk: an unattended run can take many minutes, and an -- operator watching a log wants progress rather than a silent block -- that appears all at once at the end. @@ -715,11 +833,29 @@ -- agent outright can leave the very children this is meant to reach -- still running. Only a group-wide terminate collects them. -- +-- Escalation is @SIGINT@, then @SIGTERM@, then @SIGKILL@, each to the +-- whole group. The first two can be caught, ignored, or handled slowly, +-- and a coding agent that ignores both is exactly the run an operator +-- needs a deadline for; @SIGKILL@ cannot be caught, ignored, or delayed, +-- so it is the last word and the only stage whose wait is unbounded. +-- +-- Each earlier stage is bounded by 'gracePeriodMicros' and ends as soon +-- as the leader has been reaped /and/ no member of the group is left. +-- Polling the group rather than only waiting on the leader is what gives +-- a grandchild the same grace the agent gets: waiting on the leader +-- alone would send all three signals in a burst whenever the agent +-- itself stopped promptly and its children did not. +-- -- Every signal is wrapped because a process that has already exited -- makes these throw, and a race between the timeout firing and the -- process exiting on its own is normal rather than exceptional. The -- final wait always runs so the child is reaped instead of lingering as -- a zombie. +-- +-- Without POSIX signals the two group calls do nothing and only the +-- leader can be reached, which 'killGroupOrLeader' does with +-- 'P.terminateProcess'; survivors that ignored the interrupt are missed, +-- as they always were on such a platform. terminateGroup :: P.ProcessHandle -> IO () terminateGroup ph = do -- Read the identifier before any wait: 'P.getPid' yields 'Nothing' @@ -727,14 +863,68 @@ -- leader. leader <- P.getPid ph _ <- trySync (P.interruptProcessGroupOf ph) - stopped <- Timeout.timeout gracePeriodMicros (P.waitForProcess ph) - _ <- trySync (terminateProcessGroup leader) - case stopped of - Just _ -> pure () - Nothing -> do - _ <- trySync (P.terminateProcess ph) - _ <- trySync (P.waitForProcess ph) - pure () + settled <- awaitGroup leader + unless settled $ do + _ <- trySync (terminateProcessGroup leader) + settledAfterTerm <- awaitGroup leader + unless settledAfterTerm $ do + _ <- trySync (killGroupOrLeader ph leader) + -- Unbounded on purpose: SIGKILL cannot be ignored, so this + -- returns as soon as the operating system has torn the leader + -- down, and waiting is how it is reaped rather than left a zombie. + void (trySync (P.waitForProcess ph)) + where + -- True once the leader has been reaped and no member of its group + -- remains. Polled rather than waited on, because there is no call + -- that blocks until a whole group is empty. + -- + -- The order within a poll matters: 'P.getProcessExitCode' reaps an + -- exited leader, and an unreaped leader is itself still a member of + -- the group, so asking about the group first would always say it is + -- alive. + awaitGroup leader = go (max 1 (gracePeriodMicros `div` pollIntervalMicros)) + where + go :: Int -> IO Bool + go 0 = pure False + go n = do + exited <- P.getProcessExitCode ph + alive <- groupAlive leader + if isJust exited && not alive + then pure True + else threadDelay pollIntervalMicros >> go (n - 1) + +-- | How often 'terminateGroup' asks whether a signalled group has gone. +pollIntervalMicros :: Int +pollIntervalMicros = 50000 + +-- | Whether any process remains in the group named by this leader. +-- +-- Asked with the null signal, which delivers nothing and exists for +-- exactly this question: it fails with @ESRCH@ once the group has no +-- members left. On a platform without POSIX signals there is no way to +-- ask, and 'False' keeps 'terminateGroup' resting on the leader's own +-- exit code, which is all such a platform ever knew. +groupAlive :: Maybe P.Pid -> IO Bool +#if defined(BAIKAI_POSIX_SIGNALS) +groupAlive Nothing = pure False +groupAlive (Just leader) = + either (const False) (const True) + <$> trySync (Signals.signalProcessGroup Signals.nullSignal leader) +#else +groupAlive _ = pure False +#endif + +-- | The last resort: @SIGKILL@ to the whole group where POSIX signals +-- exist, and otherwise a terminate of the leader alone, which is all the +-- platform can reach. +killGroupOrLeader :: P.ProcessHandle -> Maybe P.Pid -> IO () +#if defined(BAIKAI_POSIX_SIGNALS) +killGroupOrLeader _ Nothing = pure () +killGroupOrLeader _ (Just leader) = + Signals.signalProcessGroup Signals.sigKILL leader +#else +killGroupOrLeader ph _ = P.terminateProcess ph +#endif -- | Send @SIGTERM@ to a whole process group, named by its leader. -- diff --git a/test/BinaryTests.hs b/test/BinaryTests.hs new file mode 100644 --- /dev/null +++ b/test/BinaryTests.hs @@ -0,0 +1,391 @@ +-- | Tests that spawn the __built__ @baikai@ executable. +-- +-- Every other case in this suite exercises the library in this test +-- binary's own process, under this test binary's own runtime system. A +-- test suite's @ghc-options@ are evidence about the suite and about +-- nothing else, so a suite compiled @-threaded@ can pass every runner +-- case while the installed executable, compiled without it, cannot keep +-- the same contract: in the non-threaded runtime a blocking operating +-- system call such as the @waitpid@ inside 'System.Process.waitForProcess' +-- stops every Haskell thread until it returns, so the timeout can never +-- fire and a child that fills a pipe deadlocks against its parent. +-- +-- The cases here therefore run the real binary, as a user who typed +-- @cabal install baikai-agent@ would. +module BinaryTests (binaryTests) where + +import Control.Concurrent (forkIO, threadDelay) +import Control.Concurrent.MVar (MVar, newEmptyMVar, newMVar, putMVar, takeMVar) +import Data.ByteString (ByteString) +import Data.ByteString qualified as BS +import Data.ByteString.Char8 qualified as BS8 +import Data.Maybe (fromMaybe) +import Data.Text (Text) +import Data.Text qualified as Text +import Data.Text.Encoding qualified as Text +import Data.Time.Clock (diffUTCTime, getCurrentTime) +import System.Directory + ( createDirectoryIfMissing, + doesFileExist, + findExecutable, + getPermissions, + setOwnerExecutable, + setPermissions, + ) +import System.Environment (lookupEnv) +import System.Exit (ExitCode (..)) +import System.FilePath (()) +import System.IO (Handle) +import System.IO.Temp (withSystemTempDirectory) +import System.Process qualified as P +import System.Timeout qualified as Timeout +import Test.Tasty (TestTree, testGroup) +import Test.Tasty.HUnit (assertBool, assertFailure, testCase, (@?=)) + +binaryTests :: TestTree +binaryTests = + testGroup + "the built baikai binary" + [ reportsTheThreadedRuntimeTest, + timesOutAHungAgentTest, + writesUtf8UnderCLocaleTest + ] + +-- -------------------------------------------------------------------- +-- Locating and running the executable +-- -------------------------------------------------------------------- + +-- | The built executable. +-- +-- Under @cabal test@ the @build-tool-depends: baikai-agent:baikai@ entry +-- on this suite makes cabal build the executable first and put its +-- directory on the @PATH@ of the process that runs this binary, so a +-- plain 'findExecutable' finds the freshly built one. +-- @BAIKAI_AGENT_TEST_EXECUTABLE@ overrides that for a test binary run by +-- hand, for example +-- @BAIKAI_AGENT_TEST_EXECUTABLE=$(cabal list-bin baikai-agent:exe:baikai)@. +-- +-- Absence is a hard failure rather than a skip: a silent skip here would +-- turn the one case that proves what the binary ships with into no case +-- at all. +builtBaikai :: IO FilePath +builtBaikai = do + override <- lookupEnv "BAIKAI_AGENT_TEST_EXECUTABLE" + found <- maybe (findExecutable "baikai") (pure . Just) override + maybe + ( assertFailure + "no baikai executable found: run this suite through `cabal test`, \ + \or set BAIKAI_AGENT_TEST_EXECUTABLE to the built binary" + ) + pure + found + +-- | Run the executable and collect everything it wrote. +-- +-- The environment is explicit: @PATH@ from this process (the child needs +-- @sh@, and under @cabal test@ that @PATH@ is also how the stub scripts +-- find their own tools), and @HOME@ and @XDG_CONFIG_HOME@ pointing at +-- the temporary directory so that a developer's real +-- @~\/.config\/baikai\/agents.kdl@ is never consulted. Anything a case +-- adds wins over those defaults. +-- +-- Both output pipes are drained on forked threads before the wait, for +-- the same reason the runner itself does it: a pipe holds a bounded +-- amount of data and a parent that waits first deadlocks a chatty child. +runBaikai :: + FilePath -> + [String] -> + [(String, String)] -> + FilePath -> + IO (ExitCode, ByteString, ByteString) +runBaikai exe args extraEnv workingDir = do + parentPath <- fromMaybe "/usr/bin:/bin" <$> lookupEnv "PATH" + let environment = + [ ("PATH", parentPath), + ("HOME", workingDir), + ("XDG_CONFIG_HOME", workingDir) + ] + <> extraEnv + spec = + (P.proc exe args) + { P.cwd = Just workingDir, + P.env = Just environment, + P.std_in = P.NoStream, + P.std_out = P.CreatePipe, + P.std_err = P.CreatePipe + } + P.withCreateProcess spec $ \_ mOut mErr ph -> do + outVar <- drainAsync mOut + errVar <- drainAsync mErr + code <- P.waitForProcess ph + out <- takeMVar outVar + err <- takeMVar errVar + pure (code, out, err) + +-- | Read one handle to end of file on its own thread. +drainAsync :: Maybe Handle -> IO (MVar ByteString) +drainAsync Nothing = newMVar BS.empty +drainAsync (Just h) = do + var <- newEmptyMVar + _ <- forkIO (BS.hGetContents h >>= putMVar var) + pure var + +-- -------------------------------------------------------------------- +-- The runtime probe +-- -------------------------------------------------------------------- + +-- | @+RTS --info@ prints the runtime the binary was linked against. +-- +-- The @RTS way@ entry reads @rts_thr@ for the threaded runtime; a binary +-- built without @-threaded@ reads @rts_v@. Only the way name is asserted, +-- because the surrounding spacing of the printed pair is GHC's to change +-- (9.12.4 prints @(\"RTS way\", \"rts_thr\")@, with a space). The probe +-- itself is permitted under GHC's default @-rtsopts=some@, so the +-- executable needs no @-rtsopts@ of its own for this to work. +reportsTheThreadedRuntimeTest :: TestTree +reportsTheThreadedRuntimeTest = + testCase "baikai +RTS --info reports the threaded runtime" $ + withSystemTempDirectory "baikai-agent-binary" $ \dir -> do + exe <- builtBaikai + (_, out, _) <- runBaikai exe ["+RTS", "--info"] [] dir + assertBool + ( "expected the threaded runtime in `baikai +RTS --info`; a stale or \ + \unthreaded binary reports rts_v. Output was:\n" + <> BS8.unpack out + ) + (BS8.pack "\"rts_thr\"" `BS.isInfixOf` out) + +-- -------------------------------------------------------------------- +-- The timeout, on the shipped runtime +-- -------------------------------------------------------------------- + +-- | A stub coding agent that outlives its timeout. +-- +-- It records its own process id and the process id of a background +-- @sleep@ so the case can prove the whole process group was reached, +-- prints one line so the case can prove drained output survives, and +-- then waits for a child that will not return for two minutes. +hangingStub :: Text +hangingStub = + Text.pack + ( unlines + [ "#!/bin/sh", + -- Ignoring both polite signals is what makes this case prove + -- the escalation rather than the first signal: SIGKILL is the + -- only thing left that can end this shell, and the ignored + -- disposition is inherited by the background sleep too. + "trap '' INT TERM", + "echo \"$$\" > \"$BAIKAI_TEST_PIDFILE\"", + "printf 'partial output before the hang\\n'", + "sleep 120 &", + "echo \"$!\" >> \"$BAIKAI_TEST_PIDFILE\"", + "wait" + ] + ) + +-- | An operator-scope job that runs the stub under a short deadline. +-- +-- Operator scope rather than repository scope on purpose: the policy +-- ceiling is loaded from the operator file, so a job that names its own +-- @executable@ is safe here whatever the repository scope is later +-- allowed to set. +-- +-- Three seconds rather than one. The deadline has to clear the time the +-- operating system takes to start @\/bin\/sh@ while the rest of this +-- suite runs in parallel, and one second does not: with @timeout \"1s\"@ +-- this case fails on a loaded machine having never created the process +-- id file at all, because the stub's first line had not run when the +-- group was killed. Three seconds is still forty times shorter than the +-- stub's own sleep, so what the case proves — that the deadline stopped +-- the child rather than the child finishing — is unchanged. +captureJob :: String -> FilePath -> FilePath -> String -> String +captureJob jobName executable workspace deadline = + unlines + [ "jobs {", + " " <> jobName <> " {", + " provider \"claude\"", + " executable \"" <> executable <> "\"", + " working-dir \"" <> workspace <> "\"", + " output \"capture\"", + " timeout \"" <> deadline <> "\"", + " safety { capability \"read-only\" }", + " }", + "}" + ] + +timesOutAHungAgentTest :: TestTree +timesOutAHungAgentTest = + testCase "agent run stops a child that outlives its timeout, exit 75" $ + withSystemTempDirectory "baikai-agent-binary" $ \dir -> do + exe <- builtBaikai + let workspace = dir "workspace" + stub = dir "hang.sh" + pidFile = dir "pids" + -- The operator file must lie outside the directory the command + -- runs in, which is the repository as far as baikai is + -- concerned: a ceiling file a checkout could have written is + -- refused, and `--user-config` naming a path under the checkout + -- is exactly that shape. + repositoryRoot = dir "repo" + configPath = dir "agents.kdl" + createDirectoryIfMissing True workspace + createDirectoryIfMissing True repositoryRoot + writeExecutable stub hangingStub + writeFile configPath (captureJob "hang" stub workspace "3s") + started <- getCurrentTime + -- Bounded so that the pre-fix behaviour — a timeout that can never + -- fire — fails in thirty seconds instead of waiting out the + -- stub's two-minute sleep. + outcome <- + Timeout.timeout 30000000 $ + runBaikai + exe + [ "agent", + "run", + "hang", + "--prompt", + "go", + "--user-config", + configPath + ] + [("BAIKAI_TEST_PIDFILE", pidFile)] + repositoryRoot + finished <- getCurrentTime + let elapsed = diffUTCTime finished started + case outcome of + Nothing -> + assertFailure + "the run never returned: the configured timeout did not stop the \ + \child. This is what an executable built without -threaded does, \ + \because waitForProcess blocks every Haskell thread." + Just (code, out, err) -> do + assertBool + ( "expected exit 75 (the timeout code). stdout:\n" + <> BS8.unpack out + <> "\nstderr:\n" + <> BS8.unpack err + ) + (code == ExitFailure 75) + assertBool + ( "expected the deadline plus the grace periods, not the stub's \ + \own two minutes; the run took " + <> show elapsed + ) + (elapsed < 20) + -- The bytes the stub printed before the kill are reported + -- rather than discarded, which for a real coding agent is the + -- partial answer an operator most wants from a timed-out run. + assertBool + ("expected the drained line on standard output, got: " <> show out) + (out == BS8.pack "partial output before the hang\n") + pids <- recordedPids pidFile + assertBool + ( "the stub never recorded a process id, so the case proved \ + \nothing. stdout:\n" + <> BS8.unpack out + <> "\nstderr:\n" + <> BS8.unpack err + ) + (not (null pids)) + awaitAllGone pids + +-- | The process ids the stub wrote, one per line. +recordedPids :: FilePath -> IO [String] +recordedPids path = do + present <- doesFileExist path + if not present + then pure [] + else do + contents <- readFile path + pure [line | line <- lines contents, not (null line)] + +-- | Fail unless every recorded process is gone. +-- +-- Polled rather than checked once: the runner returns as soon as it has +-- reaped the group's leader, and the kernel may take a moment longer to +-- finish reaping a grandchild. +awaitAllGone :: [String] -> IO () +awaitAllGone pids = go (40 :: Int) + where + go n = do + alive <- filterM' processAlive pids + case (alive, n) of + ([], _) -> pure () + (remaining, 0) -> + assertFailure + ( "these processes outlived the run: " + <> unwords remaining + <> " — the timeout did not reach the whole process group" + ) + (_, _) -> threadDelay 50000 >> go (n - 1) + filterM' p xs = concat <$> mapM (\x -> (\keep -> [x | keep]) <$> p x) xs + +-- | Whether a process exists, asked with the null signal: @kill -0@ +-- delivers nothing and fails when no such process is there. +processAlive :: String -> IO Bool +processAlive pid = do + (code, _, _) <- P.readProcessWithExitCode "kill" ["-0", pid] "" + pure (code == ExitSuccess) + +-- -------------------------------------------------------------------- +-- Small helpers +-- -------------------------------------------------------------------- + +-- | Write a shell script as UTF-8 bytes and make it executable. +-- +-- Encoded explicitly rather than written with 'writeFile', which encodes +-- through this process's locale: a stub whose whole purpose is to print +-- non-ASCII text could not otherwise be written at all under @LANG=C@. +writeExecutable :: FilePath -> Text -> IO () +writeExecutable path body = do + BS.writeFile path (Text.encodeUtf8 body) + perms <- getPermissions path + setPermissions path (setOwnerExecutable True perms) + +-- -------------------------------------------------------------------- +-- Output encoding, on the shipped binary +-- -------------------------------------------------------------------- + +-- | What the stub prints, and therefore what the command must print +-- back: a Latin-1-representable letter, a character outside Latin-1, and +-- one outside the Basic Multilingual Plane's Latin range. +utf8Answer :: Text +utf8Answer = Text.pack "réconcilier — 文法" + +-- | The command writes UTF-8 whatever the locale says. +-- +-- @LANG=C@ is what cron, systemd units, and minimal containers give a +-- process. Before this was fixed the command encoded its output through +-- that locale, so a single non-ASCII character in the agent's answer +-- made the write throw @invalid argument@ /after/ the run had finished: +-- exit 1, and the answer lost. +-- +-- The environment is set explicitly and contains no other @LC_@ +-- variable, so nothing else can quietly restore a UTF-8 locale. +writesUtf8UnderCLocaleTest :: TestTree +writesUtf8UnderCLocaleTest = + testCase "the command writes UTF-8 under LANG=C" $ + withSystemTempDirectory "baikai-agent-binary" $ \dir -> do + exe <- builtBaikai + let workspace = dir "workspace" + stub = dir "say.sh" + -- Outside the directory the command runs in; see the timeout + -- case above for why. + repositoryRoot = dir "repo" + configPath = dir "agents.kdl" + createDirectoryIfMissing True workspace + createDirectoryIfMissing True repositoryRoot + writeExecutable + stub + (Text.pack "#!/bin/sh\nprintf '" <> utf8Answer <> Text.pack "\\n'\n") + writeFile configPath (captureJob "say" stub workspace "30s") + (code, out, err) <- + runBaikai + exe + ["agent", "run", "say", "--prompt", "go", "--user-config", configPath] + [("LANG", "C"), ("LC_ALL", "C")] + repositoryRoot + assertBool + ("expected a successful run; stderr was:\n" <> BS8.unpack err) + (code == ExitSuccess) + out @?= Text.encodeUtf8 (utf8Answer <> Text.pack "\n") diff --git a/test/CliTests.hs b/test/CliTests.hs --- a/test/CliTests.hs +++ b/test/CliTests.hs @@ -18,6 +18,7 @@ AgentCliOptions (..), AgentCliRun, PromptSource (..), + agentCliOptions, agentCliParserInfo, configExitCode, readPromptSource, @@ -26,14 +27,16 @@ runAgentCliWithPaths, usageExitCode, ) -import Baikai.Agent.Config (AgentConfigPaths (..), AgentJob, resolveAgentJob) +import Baikai.Agent.Config (AgentConfigPaths (..), AgentJob, emptyAgentConfigPaths, resolveAgentJob) import Baikai.Evidence (EvidenceStrength (..), evidenceSchemaVersion) -import Control.Lens ((^.)) +import Control.Lens ((&), (.~), (^.)) import Data.Aeson qualified as Aeson import Data.Aeson.KeyMap qualified as KeyMap import Data.Generics.Labels () +import Data.List (isInfixOf) import Data.Text (Text) import Data.Text qualified as Text +import Data.Text.Encoding qualified as Text import Data.Text.IO qualified as TextIO import GHC.IO.Handle (hDuplicate, hDuplicateTo) import Options.Applicative qualified as Options @@ -41,13 +44,16 @@ import Settei.Key (parseKey) import Settei.Optparse (cliOverride) import System.Directory - ( doesFileExist, + ( createDirectoryIfMissing, + createFileLink, + doesFileExist, getPermissions, + listDirectory, setOwnerExecutable, setPermissions, ) import System.Environment (setEnv) -import System.FilePath (()) +import System.FilePath (takeDirectory, ()) import System.IO (IOMode (..), hClose, openFile, stdin) import System.IO.Temp (withSystemTempDirectory) import Test.Tasty (TestTree, testGroup) @@ -73,16 +79,27 @@ [ showExplainsWithProvenanceTest, showRedactsProviderArgumentsTest, showPrintsConfigurationBeforeRefusalTest, - showReportsAnUnreadableFileTest + showReportsAnUnreadableFileTest, + showJsonFailureIsAnEnvelopeTest, + showJsonRefusalIsAnEnvelopeTest, + aRelativeWorkingDirIsTheRepositoryTest ], testGroup "the sync-keiro-dsl fixture" [ syncKeiroDslRunsTest, + bashGrantIsRefusedUnderTheDefaultCeilingTest, swappingTheProviderIsAConfigurationChangeTest, swappingTheProviderRefusesATheToolListTest, theCeilingRefusesBeforeAnythingIsStartedTest ], testGroup + "repository scope through the command" + [ repositoryExecutableIsRefusedThroughTheCommandTest, + showListsTheCeilingFieldsTest, + ceilingInsideTheRepoExitsSeventyEightTest, + anOperatorPolicyNodeIsNotWarnedAboutTest + ], + testGroup "agent run" [ propagatesTheAgentExitCodeTest, inheritModeCapturesNothingTest, @@ -90,7 +107,10 @@ refusesAnEmptyPromptTest, writesTheEvidenceFileTest, writesNoEvidenceFileByDefaultTest, - refusesAnImpossibleEvidenceRequirementTest + refusesAnImpossibleEvidenceRequirementTest, + runIdWithoutADestinationIsAUsageErrorTest, + jsonCarriesTheEvidenceRecordTest, + stagingFileCannotBePrePlantedTest ], testGroup "prompts" @@ -116,17 +136,7 @@ -- configuration scope. Tests supply the paths separately, so the two -- path fields stay 'Nothing' here and are never consulted. options :: AgentCliCommand -> AgentCliOptions -options command = - AgentCliOptions - { command, - overrides = [], - userConfig = Nothing, - repoConfig = Nothing, - jsonOutput = False, - evidenceFile = Nothing, - runId = Nothing, - requiredEvidence = Nothing - } +options = agentCliOptions withOverride :: Text -> Text -> AgentCliOptions -> AgentCliOptions withOverride key value opts = @@ -144,10 +154,41 @@ withEvidence path outerRun opts = opts {evidenceFile = Just path, runId = Just outerRun} +-- | The checkout a test's repository document belongs to. +-- +-- Real layout, not a convenience: a repository file may not set +-- @executable@ or @extra-dirs@ and its @working-dir@ must resolve inside +-- this directory, so a test that wrote both documents into one directory +-- would be asserting against a shape the code refuses. +repositoryRootIn :: FilePath -> FilePath +repositoryRootIn dir = dir "repo" + +-- | Where the repository document lands, for a test that names it. +repositoryDocumentIn :: FilePath -> FilePath +repositoryDocumentIn dir = repositoryRootIn dir ".baikai" "agents.kdl" + +-- | Write the operator document, if there is one, outside the checkout, +-- and the repository document inside it. +pathsIn :: FilePath -> Maybe Text -> Text -> IO AgentConfigPaths +pathsIn dir operatorDoc repoDoc = do + userConfig <- traverse (writeIn (dir "operator" "agents.kdl")) operatorDoc + repoConfig <- Just <$> writeIn (repositoryDocumentIn dir) repoDoc + pure + emptyAgentConfigPaths + { userConfig, + repoConfig, + repositoryRoot = repositoryRootIn dir + } + where + writeIn path body = do + createDirectoryIfMissing True (takeDirectory path) + TextIO.writeFile path body + pure path + -- | Paths naming only a repository document, which is the normal state -- for an operator who has written no policy file. -repositoryOnly :: FilePath -> AgentConfigPaths -repositoryOnly path = AgentConfigPaths {userConfig = Nothing, repoConfig = Just path} +repositoryOnly :: FilePath -> Text -> IO AgentConfigPaths +repositoryOnly dir repoDoc = pathsIn dir Nothing repoDoc -- | Write a KDL document into a temporary directory and hand back the -- workspace directory and the document's path. @@ -210,21 +251,23 @@ resolveOne :: Text -> Text -> IO AgentJob resolveOne document jobName = withWorkspace $ \dir -> do - path <- writeDocument dir "repo.kdl" document - loaded <- resolveAgentJob (repositoryOnly path) noEnvironment [] jobName + paths <- repositoryOnly dir document + loaded <- resolveAgentJob paths noEnvironment [] jobName case loaded of Left problem -> assertFailure ("loading failed: " <> show problem) Right resolved -> case resolved ^. #answer of Left problems -> assertFailure ("resolution failed: " <> show problems) Right job -> pure job +-- | A job with nothing but the required settings. It is never spawned, +-- so the working directory is the repository itself. minimalJob :: Text -> Text -> Text minimalJob provider extra = Text.unlines [ "jobs {", " demo {", " provider \"" <> provider <> "\"", - " working-dir \"/tmp\"", + " working-dir \".\"", extra, " safety { capability \"edit-workspace\" }", " }", @@ -268,7 +311,10 @@ testCase "an empty list exits 0 and keeps standard output empty" $ do -- An empty list is a normal state, not an error, and a script -- piping the output should never have to filter prose out of data. - finished <- run AgentConfigPaths {userConfig = Nothing, repoConfig = Nothing} (options AgentList) + finished <- + run + emptyAgentConfigPaths + (options AgentList) finished ^. #exitCode @?= 0 finished ^. #standardOutput @?= "" assertBool @@ -279,10 +325,9 @@ listsConfiguredJobsTest = testCase "configured jobs are listed, sorted, with their scope" $ withWorkspace $ \dir -> do - path <- - writeDocument + paths <- + repositoryOnly dir - "repo.kdl" ( Text.unlines [ "jobs {", " zebra { provider \"claude\" }", @@ -290,7 +335,7 @@ "}" ] ) - finished <- run (repositoryOnly path) (options AgentList) + finished <- run paths (options AgentList) finished ^. #exitCode @?= 0 let listed = Text.lines (finished ^. #standardOutput) map (take 1 . Text.words) listed @?= [["alpha"], ["zebra"]] @@ -309,8 +354,9 @@ -- renderResolutionText drops the location, so this is also the test -- that the command-line layer walks the report itself. withWorkspace $ \dir -> do - path <- writeDocument dir "repo.kdl" (minimalJob "claude" "") - finished <- run (repositoryOnly path) (options (AgentShow "demo")) + paths <- repositoryOnly dir (minimalJob "claude" "") + let path = repositoryDocumentIn dir + finished <- run paths (options (AgentShow "demo")) let output = finished ^. #standardOutput finished ^. #exitCode @?= 0 assertBool @@ -323,6 +369,9 @@ ("a line and column follow the path: " <> Text.unpack output) ((Text.pack path <> ":3:") `Text.isInfixOf` output) assertBool + ("the output format is named: " <> Text.unpack output) + ("jobs.demo.output-format" `Text.isInfixOf` output) + assertBool ("the ceiling is shown: " <> Text.unpack output) ("max-capability" `Text.isInfixOf` output) assertBool @@ -341,20 +390,15 @@ -- vector either, which is the easier of the two to overlook. withWorkspace $ \dir -> do - userPath <- - writeDocument - dir - "user.kdl" - (Text.unlines ["policy {", " allow-provider-args #true", "}"]) - repoPath <- - writeDocument + paths <- + pathsIn dir - "repo.kdl" + (Just (Text.unlines ["policy {", " allow-provider-args #true", "}"])) ( Text.unlines [ "jobs {", " demo {", " provider \"claude\"", - " working-dir \"/tmp\"", + " working-dir \".\"", " safety {", " capability \"edit-workspace\"", " provider-args \"--api-key\" \"sk-not-a-real-key\"", @@ -363,10 +407,7 @@ "}" ] ) - finished <- - run - AgentConfigPaths {userConfig = Just userPath, repoConfig = Just repoPath} - (options (AgentShow "demo")) + finished <- run paths (options (AgentShow "demo")) let output = finished ^. #standardOutput <> finished ^. #standardError finished ^. #exitCode @?= 0 assertBool @@ -385,21 +426,20 @@ -- A job the ceiling refuses is precisely the case an operator most -- needs `show` for; printing nothing would hide it. withWorkspace $ \dir -> do - path <- - writeDocument + paths <- + repositoryOnly dir - "repo.kdl" ( Text.unlines [ "jobs {", " demo {", " provider \"claude\"", - " working-dir \"/tmp\"", + " working-dir \".\"", " safety { capability \"full-access\" }", " }", "}" ] ) - finished <- run (repositoryOnly path) (options (AgentShow "demo")) + finished <- run paths (options (AgentShow "demo")) finished ^. #exitCode @?= refusedExitCode assertBool ("the configuration was printed: " <> Text.unpack (finished ^. #standardOutput)) @@ -410,12 +450,123 @@ && "edit-workspace" `Text.isInfixOf` (finished ^. #standardError) ) +-- | Every `show --json` path emits one object with the same seven keys, +-- so a reader never has to know which of three failure modes it is +-- looking at before it can find the exit code. +showEnvelopeOf :: AgentCliRun -> IO (KeyMap.KeyMap Aeson.Value) +showEnvelopeOf finished = + case Aeson.eitherDecodeStrict (Text.encodeUtf8 (finished ^. #standardOutput)) of + Left problem -> + assertFailure + ( "the envelope did not parse: " + <> problem + <> "\n" + <> Text.unpack (finished ^. #standardOutput) + ) + Right (Aeson.Object envelope) -> do + mapM_ + ( \key -> + assertBool + ("expected the key " <> show key <> " in: " <> show envelope) + (KeyMap.member key envelope) + ) + ["job", "outcome", "exitCode", "message", "configuration", "ceiling", "command"] + pure envelope + Right other -> assertFailure ("expected one JSON object, got: " <> show other) + +showJsonFailureIsAnEnvelopeTest :: TestTree +showJsonFailureIsAnEnvelopeTest = + testCase "a malformed document under --json still produces one envelope" $ + -- The path that used to print a bare resolution report, or nothing + -- at all when there was no report to print. + withWorkspace $ \dir -> do + paths <- repositoryOnly dir "jobs {\n demo {\n provider \"claude\"\n" + finished <- + run paths ((options (AgentShow "demo")) {jsonOutput = True}) + finished ^. #exitCode @?= configExitCode + envelope <- showEnvelopeOf finished + KeyMap.lookup "outcome" envelope @?= Just (Aeson.String "failed") + KeyMap.lookup "exitCode" envelope @?= Just (Aeson.Number 78) + KeyMap.lookup "job" envelope @?= Just (Aeson.String "demo") + KeyMap.lookup "command" envelope @?= Just Aeson.Null + +showJsonRefusalIsAnEnvelopeTest :: TestTree +showJsonRefusalIsAnEnvelopeTest = + testCase "a refused job under --json produces the same envelope" $ + withWorkspace $ \dir -> do + paths <- + repositoryOnly + dir + ( Text.unlines + [ "jobs {", + " demo {", + " provider \"claude\"", + " working-dir \".\"", + " safety { capability \"full-access\" }", + " }", + "}" + ] + ) + finished <- run paths ((options (AgentShow "demo")) {jsonOutput = True}) + finished ^. #exitCode @?= refusedExitCode + envelope <- showEnvelopeOf finished + KeyMap.lookup "outcome" envelope @?= Just (Aeson.String "refused") + KeyMap.lookup "exitCode" envelope @?= Just (Aeson.Number 77) + -- The configuration and the ceiling are both present: a refused + -- job is exactly the case an operator most needs them for. + assertBool + ("expected a configuration object: " <> show envelope) + (KeyMap.lookup "configuration" envelope /= Just Aeson.Null) + assertBool + ("expected a ceiling object: " <> show envelope) + (KeyMap.lookup "ceiling" envelope /= Just Aeson.Null) + case KeyMap.lookup "message" envelope of + Just (Aeson.String message) -> + assertBool + ("the refusal names both values: " <> Text.unpack message) + ("full-access" `Text.isInfixOf` message && "edit-workspace" `Text.isInfixOf` message) + other -> assertFailure ("expected a message string, got: " <> show other) + +aRelativeWorkingDirIsTheRepositoryTest :: TestTree +aRelativeWorkingDirIsTheRepositoryTest = + testCase "working-dir \".\" means the repository, whichever file declared it" $ + -- Resolving against the process's own directory would be wrong for a + -- value that may come from either file: two documents defining one + -- job would make "." mean two places depending on which layer won. + withWorkspace $ \dir -> do + paths <- + repositoryOnly + dir + ( Text.unlines + [ "jobs {", + " demo {", + " provider \"codex\"", + " working-dir \".\"", + " safety { capability \"read-only\" }", + " }", + "}" + ] + ) + finished <- run paths ((options (AgentShow "demo")) {jsonOutput = True}) + finished ^. #exitCode @?= 0 + envelope <- showEnvelopeOf finished + case KeyMap.lookup "command" envelope of + Just (Aeson.Object command) -> + case KeyMap.lookup "arguments" command of + Just (Aeson.Array args) -> + assertBool + ("expected --cd to name the repository root: " <> show args) + (Aeson.String (Text.pack (repositoryRootIn dir)) `elem` args) + other -> assertFailure ("expected an arguments array, got: " <> show other) + other -> assertFailure ("expected a command object, got: " <> show other) + showReportsAnUnreadableFileTest :: TestTree showReportsAnUnreadableFileTest = testCase "a malformed document exits with the configuration code" $ withWorkspace $ \dir -> do - path <- writeDocument dir "repo.kdl" "jobs {\n demo {\n provider \"claude\"\n" - finished <- run (repositoryOnly path) (options (AgentShow "demo")) + paths <- repositoryOnly dir "jobs {\n demo {\n provider \"claude\"\n" + let path = repositoryDocumentIn dir + finished <- run paths (options (AgentShow "demo")) finished ^. #exitCode @?= configExitCode assertBool ("the file is named: " <> Text.unpack (finished ^. #standardError)) @@ -425,6 +576,27 @@ -- The sync-keiro-dsl fixture -- -------------------------------------------------------------------- +-- | The operator half of the motivating fixture. +-- +-- Two things live here that a repository file may not supply. The +-- @executable@ names which program runs, which is operator authority by +-- definition. And @Bash@ and @Skill@ are grants outside what +-- @edit-workspace@ implies, so the operator has to say they are +-- permitted; the eight-tool list in the repository document below is +-- unchanged, and it is that operator line which now lets it through. +syncKeiroDslOperatorDocument :: FilePath -> Text +syncKeiroDslOperatorDocument executable = + Text.unlines + [ "policy {", + " allowed-tools \"Bash\" \"Skill\"", + "}", + "jobs {", + " sync-keiro-dsl {", + " executable \"" <> Text.pack executable <> "\"", + " }", + "}" + ] + -- | The translation of the motivating script's launch into -- configuration. -- @@ -433,14 +605,13 @@ -- output. The extra directory is deliberately __not__ here: it arrives -- on the command line as a single @--set@, which is what makes the -- "no provider flags in the script" claim testable. -syncKeiroDslDocument :: FilePath -> FilePath -> Text -syncKeiroDslDocument workingDir executable = +syncKeiroDslDocument :: FilePath -> Text +syncKeiroDslDocument workingDir = Text.unlines [ "jobs {", " sync-keiro-dsl {", " provider \"claude\"", " working-dir \"" <> Text.pack workingDir <> "\"", - " executable \"" <> Text.pack executable <> "\"", " output \"capture\"", " env-requires \"BAIKAI_TEST_CLAUDE_ARGV\" \"BAIKAI_TEST_CLAUDE_STDIN\"", " safety {", @@ -469,10 +640,14 @@ "claude" (recordingAgent "BAIKAI_TEST_CLAUDE_ARGV" "BAIKAI_TEST_CLAUDE_STDIN") promptPath <- writeDocument dir "prompt.txt" fixturePrompt - configPath <- writeDocument dir "repo.kdl" (syncKeiroDslDocument dir executable) + paths <- + pathsIn + dir + (Just (syncKeiroDslOperatorDocument executable)) + (syncKeiroDslDocument (repositoryRootIn dir)) finished <- run - (repositoryOnly configPath) + paths ( withOverride "extra-dirs" (Text.pack keiroPath) @@ -498,6 +673,128 @@ ("the agent's answer is on standard output: " <> Text.unpack (finished ^. #standardOutput)) ("reconciled the lexical surface" `Text.isInfixOf` (finished ^. #standardOutput)) +bashGrantIsRefusedUnderTheDefaultCeilingTest :: TestTree +bashGrantIsRefusedUnderTheDefaultCeilingTest = + testCase "A REPOSITORY TOOL GRANT NEEDS AN OPERATOR GRANT" $ + -- The same repository document as the fixture above, with no + -- operator policy. `--allowedTools Bash` is a grant: it pre-approves + -- shell access that the permission mode would otherwise have raised + -- a request for, and with nobody present that request is denied. So + -- an unattended run must not get it because a checkout asked. + withWorkspace $ \dir -> do + let argvRecord = dir "argv" + -- Deliberately no setEnv: the two variables the fixture document + -- declares under `env-requires` are checked by the runner, and the + -- ceiling refuses before the runner is reached. Setting them here + -- would race the fixture case above, which runs in parallel and + -- reads the same two process-global variables from its own fake. + executable <- + writeFakeAgent + dir + "claude" + (recordingAgent "BAIKAI_TEST_CLAUDE_ARGV" "BAIKAI_TEST_CLAUDE_STDIN") + promptPath <- writeDocument dir "prompt.txt" fixturePrompt + paths <- + pathsIn + dir + -- The executable only: no `policy` node, so the default + -- ceiling is in force. + (Just (operatorJob "sync-keiro-dsl" executable)) + (syncKeiroDslDocument (repositoryRootIn dir)) + finished <- + run paths (options (AgentRun "sync-keiro-dsl" (PromptFile promptPath))) + finished ^. #exitCode @?= refusedExitCode + let message = finished ^. #standardError + mapM_ + ( \fragment -> + assertBool + ("expected " <> Text.unpack fragment <> " in: " <> Text.unpack message) + (fragment `Text.isInfixOf` message) + ) + ["Bash", "edit-workspace", "policy.allowed-tools"] + -- Grants the capability already implies are not named, because + -- they are not the problem. + assertBool + ("Read is implied and must not be named: " <> Text.unpack message) + (not ("Read" `Text.isInfixOf` message)) + started <- doesFileExist argvRecord + assertBool "nothing was started" (not started) + +repositoryExecutableIsRefusedThroughTheCommandTest :: TestTree +repositoryExecutableIsRefusedThroughTheCommandTest = + testCase "a repository file naming the executable is refused before anything starts" $ + withWorkspace $ \dir -> do + let argvRecord = dir "argv" + executable <- + writeFakeAgent dir "claude" ("#!/bin/sh\ntouch '" <> Text.pack argvRecord <> "'\n") + promptPath <- writeDocument dir "prompt.txt" fixturePrompt + paths <- + repositoryOnly + dir + ( Text.unlines + [ "jobs {", + " demo {", + " provider \"claude\"", + " working-dir \"" <> Text.pack (repositoryRootIn dir) <> "\"", + " executable \"" <> Text.pack executable <> "\"", + " safety { capability \"edit-workspace\" }", + " }", + "}" + ] + ) + finished <- run paths (options (AgentRun "demo" (PromptFile promptPath))) + finished ^. #exitCode @?= refusedExitCode + assertBool + ("the setting is named: " <> Text.unpack (finished ^. #standardError)) + ("executable" `Text.isInfixOf` (finished ^. #standardError)) + started <- doesFileExist argvRecord + assertBool "the executable was never invoked" (not started) + +ceilingInsideTheRepoExitsSeventyEightTest :: TestTree +ceilingInsideTheRepoExitsSeventyEightTest = + testCase "an operator file inside the repository is a configuration error" $ + -- `--user-config .baikai/policy.kdl` is a one-line way to make the + -- checkout supply its own ceiling. No ceiling could be established, + -- so this is a configuration error rather than a policy refusal. + withWorkspace $ \dir -> do + let insidePath = repositoryRootIn dir ".baikai" "policy.kdl" + _ <- repositoryOnly dir (minimalJob "claude" "") + createDirectoryIfMissing True (takeDirectory insidePath) + TextIO.writeFile + insidePath + (Text.unlines ["policy {", " max-capability \"full-access\"", "}"]) + paths <- repositoryOnly dir (minimalJob "claude" "") + finished <- + run (paths & #userConfig .~ Just insidePath) (options (AgentShow "demo")) + finished ^. #exitCode @?= configExitCode + assertBool + ("the file is named: " <> Text.unpack (finished ^. #standardError)) + ("policy.kdl" `Text.isInfixOf` (finished ^. #standardError)) + +showListsTheCeilingFieldsTest :: TestTree +showListsTheCeilingFieldsTest = + testCase "show prints every field of the ceiling" $ + -- An operator reading `show` to find out why a job was refused needs + -- to see the limit that refused it, so every field is printed. + withWorkspace $ \dir -> do + paths <- repositoryOnly dir (minimalJob "claude" "") + finished <- run paths (options (AgentShow "demo")) + finished ^. #exitCode @?= 0 + let output = finished ^. #standardOutput + mapM_ + ( \field -> + assertBool + ("expected " <> Text.unpack field <> " in: " <> Text.unpack output) + (field `Text.isInfixOf` output) + ) + [ "max-capability", + "allow-provider-args", + "allowed-providers", + "allowed-tools", + "max-timeout", + "max-output-limit" + ] + swappingTheProviderIsAConfigurationChangeTest :: TestTree swappingTheProviderIsAConfigurationChangeTest = testCase "changing only the provider line moves the run to codex" $ @@ -515,16 +812,15 @@ "codex" (recordingAgent "BAIKAI_TEST_CODEX_ARGV" "BAIKAI_TEST_CODEX_STDIN") promptPath <- writeDocument dir "prompt.txt" fixturePrompt - configPath <- - writeDocument + paths <- + pathsIn dir - "repo.kdl" + (Just (operatorJob "sync-keiro-dsl" executable)) ( Text.unlines [ "jobs {", " sync-keiro-dsl {", " provider \"codex\"", - " working-dir \"" <> Text.pack dir <> "\"", - " executable \"" <> Text.pack executable <> "\"", + " working-dir \"" <> Text.pack (repositoryRootIn dir) <> "\"", " output \"capture\"", " safety { capability \"edit-workspace\" }", " }", @@ -532,9 +828,7 @@ ] ) finished <- - run - (repositoryOnly configPath) - (options (AgentRun "sync-keiro-dsl" (PromptFile promptPath))) + run paths (options (AgentRun "sync-keiro-dsl" (PromptFile promptPath))) finished ^. #exitCode @?= 0 argv <- recordedArgv argvRecord argv @@ -542,7 +836,7 @@ "--sandbox", "workspace-write", "--cd", - Text.pack dir, + Text.pack (repositoryRootIn dir), "--skip-git-repo-check", "--ephemeral" ] @@ -559,17 +853,19 @@ let argvRecord = dir "argv" executable <- writeFakeAgent dir "codex" "#!/bin/sh\ntouch \"$1\"\n" promptPath <- writeDocument dir "prompt.txt" fixturePrompt - configPath <- - writeDocument + paths <- + pathsIn dir - "repo.kdl" + (Just (operatorJob "sync-keiro-dsl" executable)) ( Text.unlines [ "jobs {", " sync-keiro-dsl {", " provider \"codex\"", - " working-dir \"" <> Text.pack dir <> "\"", - " executable \"" <> Text.pack executable <> "\"", + " working-dir \"" <> Text.pack (repositoryRootIn dir) <> "\"", " safety {", + -- Read and Write are implied by edit-workspace, so the + -- ceiling permits them; the refusal below is the Codex + -- renderer's, which is what this case is about. " capability \"edit-workspace\"", " allowed-tools \"Read\" \"Write\"", " }", @@ -578,9 +874,7 @@ ] ) finished <- - run - (repositoryOnly configPath) - (options (AgentRun "sync-keiro-dsl" (PromptFile promptPath))) + run paths (options (AgentRun "sync-keiro-dsl" (PromptFile promptPath))) finished ^. #exitCode @?= refusedExitCode assertBool ("the message names the sandbox alternative: " <> Text.unpack (finished ^. #standardError)) @@ -599,25 +893,22 @@ executable <- writeFakeAgent dir "claude" ("#!/bin/sh\ntouch '" <> Text.pack argvRecord <> "'\n") promptPath <- writeDocument dir "prompt.txt" fixturePrompt - configPath <- - writeDocument + paths <- + pathsIn dir - "repo.kdl" + (Just (operatorJob "sync-keiro-dsl" executable)) ( Text.unlines [ "jobs {", " sync-keiro-dsl {", " provider \"claude\"", - " working-dir \"" <> Text.pack dir <> "\"", - " executable \"" <> Text.pack executable <> "\"", + " working-dir \"" <> Text.pack (repositoryRootIn dir) <> "\"", " safety { capability \"full-access\" }", " }", "}" ] ) finished <- - run - (repositoryOnly configPath) - (options (AgentRun "sync-keiro-dsl" (PromptFile promptPath))) + run paths (options (AgentRun "sync-keiro-dsl" (PromptFile promptPath))) finished ^. #exitCode @?= refusedExitCode assertBool ("the refusal names both values: " <> Text.unpack (finished ^. #standardError)) @@ -631,22 +922,42 @@ -- agent run -- -------------------------------------------------------------------- --- | A job rooted in the workspace, running the given script, capturing --- output unless told otherwise. -scriptedJob :: FilePath -> FilePath -> Text -> Text -scriptedJob dir executable outputMode = +-- | The operator half of a job: which program runs, and nothing else. +-- +-- Every fixture that spawns a fake agent needs one of these, because a +-- repository file that could name the program to run would turn a +-- checkout into code execution with the operator's environment. +operatorJob :: Text -> FilePath -> Text +operatorJob jobName executable = Text.unlines [ "jobs {", + " " <> jobName <> " {", + " executable \"" <> Text.pack executable <> "\"", + " }", + "}" + ] + +-- | The repository half of a scripted job: everything a checkout is +-- allowed to say. It is rooted in the checkout, which is where a +-- repository working directory has to stay. +repositoryJob :: FilePath -> Text -> Text +repositoryJob dir outputMode = + Text.unlines + [ "jobs {", " demo {", " provider \"claude\"", - " working-dir \"" <> Text.pack dir <> "\"", - " executable \"" <> Text.pack executable <> "\"", + " working-dir \"" <> Text.pack (repositoryRootIn dir) <> "\"", " output \"" <> outputMode <> "\"", " safety { capability \"edit-workspace\" }", " }", "}" ] +-- | Both halves of a scripted job, written where they belong. +scriptedPaths :: FilePath -> FilePath -> Text -> IO AgentConfigPaths +scriptedPaths dir executable outputMode = + pathsIn dir (Just (operatorJob "demo" executable)) (repositoryJob dir outputMode) + propagatesTheAgentExitCodeTest :: TestTree propagatesTheAgentExitCodeTest = testCase "the agent's own exit code passes through unchanged" $ @@ -655,9 +966,8 @@ -- could not start" needs the codes to stay separate. withWorkspace $ \dir -> do executable <- writeFakeAgent dir "claude" "#!/bin/sh\ncat > /dev/null\nexit 3\n" - configPath <- writeDocument dir "repo.kdl" (scriptedJob dir executable "capture") - finished <- - run (repositoryOnly configPath) (options (AgentRun "demo" (PromptInline "do the thing"))) + paths <- scriptedPaths dir executable "capture" + finished <- run paths (options (AgentRun "demo" (PromptInline "do the thing"))) finished ^. #exitCode @?= 3 -- Nothing extra is narrated: the agent has already explained -- itself on its own standard error. @@ -672,9 +982,8 @@ withWorkspace $ \dir -> do executable <- writeFakeAgent dir "claude" "#!/bin/sh\ncat > /dev/null\necho 'inherited line'\n" - configPath <- writeDocument dir "repo.kdl" (scriptedJob dir executable "inherit") - finished <- - run (repositoryOnly configPath) (options (AgentRun "demo" (PromptInline "do the thing"))) + paths <- scriptedPaths dir executable "inherit" + finished <- run paths (options (AgentRun "demo" (PromptInline "do the thing"))) finished ^. #exitCode @?= 0 finished ^. #standardOutput @?= "" finished ^. #standardError @?= "" @@ -683,10 +992,8 @@ reportsAMissingExecutableTest = testCase "a missing coding-agent binary exits 69" $ withWorkspace $ \dir -> do - configPath <- - writeDocument dir "repo.kdl" (scriptedJob dir (dir "not-installed") "capture") - finished <- - run (repositoryOnly configPath) (options (AgentRun "demo" (PromptInline "do the thing"))) + paths <- scriptedPaths dir (dir "not-installed") "capture" + finished <- run paths (options (AgentRun "demo" (PromptInline "do the thing"))) finished ^. #exitCode @?= 69 assertBool ("the missing program is named: " <> Text.unpack (finished ^. #standardError)) @@ -705,10 +1012,10 @@ dir "claude" "#!/bin/sh\ncat > /dev/null\necho 'the task is done'\n" - configPath <- writeDocument dir "repo.kdl" (scriptedJob dir executable "capture") + paths <- scriptedPaths dir executable "capture" finished <- run - (repositoryOnly configPath) + paths (withEvidence evidencePath "outer-run-7" (options (AgentRun "demo" (PromptInline "do the thing")))) finished ^. #exitCode @?= 0 -- Nothing about the evidence file leaks onto the agent's own @@ -729,10 +1036,118 @@ (KeyMap.lookup "call_id" o /= Nothing) Right other -> assertFailure ("expected one JSON object, got: " <> show other) -- The write is atomic through a staging file, which must not be - -- left behind. - leftover <- doesFileExist (evidencePath <> ".partial") - assertBool "the staging file was renamed away" (not leftover) + -- left behind. The name is unique rather than the destination plus + -- a suffix, so the check is a directory listing. + leftovers <- listDirectory dir + assertBool + ("no staging file remains: " <> show leftovers) + (not (any (".partial" `isInfixOf`) leftovers)) +runIdWithoutADestinationIsAUsageErrorTest :: TestTree +runIdWithoutADestinationIsAUsageErrorTest = + testCase "asking for a record with nowhere to put it is a usage error" $ + -- Building one costs a --version probe of the tool and two digests, + -- so a record that is built and dropped is measurable work proving + -- nothing. There is no safe default destination either: standard + -- output belongs to the agent's own answer. + withWorkspace $ \dir -> do + executable <- + writeFakeAgent dir "claude" "#!/bin/sh\ncat > /dev/null\necho done\n" + paths <- scriptedPaths dir executable "capture" + finished <- + run + paths + ((options (AgentRun "demo" (PromptInline "do the thing"))) {runId = Just "r"}) + finished ^. #exitCode @?= usageExitCode + assertBool + ("the fix is named: " <> Text.unpack (finished ^. #standardError)) + ( "--evidence-file" `Text.isInfixOf` (finished ^. #standardError) + && "--json" `Text.isInfixOf` (finished ^. #standardError) + ) + +jsonCarriesTheEvidenceRecordTest :: TestTree +jsonCarriesTheEvidenceRecordTest = + testCase "--json carries the evidence record in its envelope" $ + -- The destination a caller who names no file asked for. The record + -- is encoded by the same ToJSON --evidence-file writes, so the two + -- destinations cannot drift. + withWorkspace $ \dir -> do + executable <- + writeFakeAgent dir "claude" "#!/bin/sh\ncat > /dev/null\necho done\n" + paths <- scriptedPaths dir executable "capture" + finished <- + run + paths + ( (options (AgentRun "demo" (PromptInline "do the thing"))) + { runId = Just "outer-run-11", + jsonOutput = True + } + ) + finished ^. #exitCode @?= 0 + case Aeson.eitherDecodeStrict (Text.encodeUtf8 (finished ^. #standardOutput)) of + Left problem -> assertFailure ("the envelope did not parse: " <> problem) + Right (Aeson.Object envelope) -> case KeyMap.lookup "evidence" envelope of + Just (Aeson.Object record) -> + KeyMap.lookup "run_id" record @?= Just (Aeson.String "outer-run-11") + other -> assertFailure ("expected an evidence object, got: " <> show other) + Right other -> assertFailure ("expected one JSON object, got: " <> show other) + +stagingFileCannotBePrePlantedTest :: TestTree +stagingFileCannotBePrePlantedTest = + testCase "A PLANTED STAGING FILE IS NEVER WRITTEN THROUGH" $ + -- An unattended run writing its record into whatever a predictable + -- name pointed at is a file overwrite an attacker chooses. The + -- staging file is created under a fresh name with O_EXCL, so a + -- symbolic link already sitting at the guessable name is not used. + withWorkspace $ \dir -> do + let evidencePath = dir "evidence.json" + canary = dir "canary" + writeFile canary "do not overwrite me" + createFileLink canary (evidencePath <> ".partial") + executable <- + writeFakeAgent dir "claude" "#!/bin/sh\ncat > /dev/null\necho done\n" + paths <- scriptedPaths dir executable "capture" + finished <- + run + paths + (withEvidence evidencePath "outer-run-8" (options (AgentRun "demo" (PromptInline "go")))) + finished ^. #exitCode @?= 0 + readFile canary >>= (@?= "do not overwrite me") + recorded <- Aeson.eitherDecodeFileStrict evidencePath + case recorded of + Left problem -> assertFailure ("the record did not parse as JSON: " <> problem) + Right (Aeson.Object o) -> + KeyMap.lookup "run_id" o @?= Just (Aeson.String "outer-run-8") + Right other -> assertFailure ("expected one JSON object, got: " <> show other) + -- The planted link is still there, unwritten; nothing else called + -- ".partial" was left behind. + leftovers <- listDirectory dir + filter (".partial" `isInfixOf`) leftovers @?= ["evidence.json.partial"] + +anOperatorPolicyNodeIsNotWarnedAboutTest :: TestTree +anOperatorPolicyNodeIsNotWarnedAboutTest = + testCase "show against an operator policy node prints nothing on standard error" $ + -- The ceiling schema is a separate declaration, so the job + -- resolution does not recognise `policy` and settei warns about + -- every key of it. None of them is a mistake. + withWorkspace $ \dir -> do + paths <- + pathsIn + dir + ( Just + ( Text.unlines + [ "policy {", + " max-capability \"full-access\"", + " allow-provider-args #true", + "}" + ] + ) + ) + (minimalJob "claude" "") + finished <- run paths (options (AgentShow "demo")) + finished ^. #exitCode @?= 0 + finished ^. #standardError @?= "" + writesNoEvidenceFileByDefaultTest :: TestTree writesNoEvidenceFileByDefaultTest = testCase "a run that named no evidence destination writes nothing" $ @@ -740,9 +1155,8 @@ let evidencePath = dir "evidence.json" executable <- writeFakeAgent dir "claude" "#!/bin/sh\ncat > /dev/null\necho 'the task is done'\n" - configPath <- writeDocument dir "repo.kdl" (scriptedJob dir executable "capture") - finished <- - run (repositoryOnly configPath) (options (AgentRun "demo" (PromptInline "do the thing"))) + paths <- scriptedPaths dir executable "capture" + finished <- run paths (options (AgentRun "demo" (PromptInline "do the thing"))) finished ^. #exitCode @?= 0 written <- doesFileExist evidencePath assertBool "no evidence file appeared" (not written) @@ -763,13 +1177,19 @@ writeFakeAgent dir "claude" ("#!/bin/sh\ntouch '" <> Text.pack argvRecord <> "'\n") -- `inherit` sends the agent's bytes to the terminal, so baikai -- holds nothing and can observe nothing however the run goes. - configPath <- writeDocument dir "repo.kdl" (scriptedJob dir executable "inherit") + paths <- scriptedPaths dir executable "inherit" finished <- run - (repositoryOnly configPath) + paths ( requiringEvidence EvidenceCorrelated - (options (AgentRun "demo" (PromptInline "do the thing"))) + -- A destination, because a record with nowhere to go is + -- now a usage error and this case is about the refusal. + ( withEvidence + (dir "evidence.json") + "outer-run-9" + (options (AgentRun "demo" (PromptInline "do the thing"))) + ) ) finished ^. #exitCode @?= refusedExitCode assertBool @@ -789,10 +1209,9 @@ testCase "an empty prompt is a usage error, not an expensive run" $ withWorkspace $ \dir -> do executable <- writeFakeAgent dir "claude" "#!/bin/sh\nexit 0\n" - configPath <- writeDocument dir "repo.kdl" (scriptedJob dir executable "capture") + paths <- scriptedPaths dir executable "capture" emptyPrompt <- writeDocument dir "empty.txt" "" - finished <- - run (repositoryOnly configPath) (options (AgentRun "demo" (PromptFile emptyPrompt))) + finished <- run paths (options (AgentRun "demo" (PromptFile emptyPrompt))) finished ^. #exitCode @?= usageExitCode assertBool ("the empty source is named: " <> Text.unpack (finished ^. #standardError)) diff --git a/test/ConfigTests.hs b/test/ConfigTests.hs --- a/test/ConfigTests.hs +++ b/test/ConfigTests.hs @@ -9,34 +9,53 @@ module ConfigTests (configTests) where import Baikai.Agent - ( AgentOutputMode (..), + ( AgentCapability (..), + AgentOutputFormat (..), + AgentOutputMode (..), AgentProvider (..), + CeilingViolation (..), renderAgentRenderError, ) import Baikai.Agent.Config - ( AgentConfigPaths (..), + ( AgentConfigError (..), + AgentConfigPaths (..), AgentConfigScope (..), AgentJob, agentEnvBindings, agentJobRequest, applyCeilingToJob, defaultOutputLimit, + emptyAgentConfigPaths, listAgentJobs, loadAgentCeiling, parseDuration, + relevantWarnings, renderAgentConfigError, + repositoryPolicyNotice, + repositoryScopeViolations, resolveAgentJob, ) import Control.Lens ((^.)) import Data.Generics.Labels () +import Data.List (isPrefixOf, isSuffixOf) import Data.Text (Text) import Data.Text qualified as Text import Data.Text.IO qualified as TextIO import Settei.Env (EnvSnapshot, envSnapshot) +import Settei.Error (ConfigWarning) import Settei.Key (parseKey) import Settei.Optparse (CliOverride, cliOverride) -import Settei.Render (renderErrorsText, renderResolutionJson, renderResolutionText) -import System.FilePath (()) +import Settei.Render + ( renderErrorsText, + renderResolutionJson, + renderResolutionText, + renderWarningsText, + ) +import System.Directory + ( createDirectoryIfMissing, + createDirectoryLink, + ) +import System.FilePath (takeDirectory, ()) import System.IO.Temp (withSystemTempDirectory) import Test.Tasty (TestTree, testGroup) import Test.Tasty.HUnit (assertBool, assertFailure, testCase, (@?=)) @@ -52,6 +71,7 @@ environmentLayerTest, defaultsTest, singleArgumentListTest, + outputFormatTest, missingFileTest ], testGroup @@ -72,9 +92,27 @@ defaultCeilingPermitsEditWorkspaceTest, userFileRaisesCeilingTest, repositoryFileCannotRaiseTheCeilingTest, - commandLineCannotRaiseTheCeilingTest + commandLineCannotRaiseTheCeilingTest, + ceilingFileInsideRepositoryIsRefusedTest, + ceilingFileOutsideTheRepositoryLoadsTest, + unknownPolicyKeyIsAnErrorTest ], testGroup + "unknown keys" + [ otherJobsDoNotWarnTest, + typoInTheSelectedJobStillWarnsTest, + repositoryPolicyNodeIsNoticedOnceTest + ], + testGroup + "repository scope" + [ repositoryExecutableIsRefusedTest, + repositoryExtraDirsAreRefusedTest, + operatorScopeMaySetBothTest, + workingDirMustStayInsideTheRootTest, + executableIsNotEnvBoundTest, + ceilingPolicyKeysTest + ], + testGroup "enumeration" [ listJobsTest ], @@ -84,15 +122,26 @@ -- | Write the supplied documents into a temporary directory and hand -- back paths pointing at them. Either scope may be absent, which is the -- normal state for an operator who has written no policy file. +-- +-- The two documents go into __different__ directories, laid out the way +-- a real machine lays them out: the repository document under a checkout +-- root, and the operator document in a sibling directory outside it. +-- That is not cosmetic. A repository-supplied @working-dir@ is confined +-- to the root, and an operator file that lies inside the root is refused +-- outright, so a layout that put both files in one directory would make +-- every scope test assert against a shape the code refuses. withConfigs :: Maybe Text -> Maybe Text -> (AgentConfigPaths -> IO a) -> IO a withConfigs userDoc repoDoc action = withSystemTempDirectory "baikai-agent-config" $ \dir -> do - userConfig <- traverse (writeDoc dir "user.kdl") userDoc - repoConfig <- traverse (writeDoc dir "repo.kdl") repoDoc - action AgentConfigPaths {userConfig, repoConfig} + let repositoryRoot = dir "repo" + createDirectoryIfMissing True repositoryRoot + userConfig <- traverse (writeDoc (dir "operator" "agents.kdl")) userDoc + repoConfig <- + traverse (writeDoc (repositoryRoot ".baikai" "agents.kdl")) repoDoc + action emptyAgentConfigPaths {userConfig, repoConfig, repositoryRoot} where - writeDoc dir fileName body = do - let path = dir fileName + writeDoc path body = do + createDirectoryIfMissing True (takeDirectory path) TextIO.writeFile path body pure path @@ -129,18 +178,33 @@ -- | A complete job that asks for the given capability. jobDoc :: Text -> Text -jobDoc capability = +jobDoc capability = jobDocWith capability [] + +-- | The same job with extra lines inside its @demo@ node. +-- +-- The extras go inside rather than into a second document, because two +-- top-level @jobs@ nodes in one KDL document are a /repeated/ node, +-- which settei reads as an array and then cannot traverse — the failure +-- reads @cannot traverse jobs through array@ and looks like a resolution +-- bug rather than a malformed fixture. +jobDocWith :: Text -> [Text] -> Text +jobDocWith capability extras = Text.unlines - [ "jobs {", - " demo {", - " provider \"claude\"", - " working-dir \"/tmp\"", - " safety {", - " capability \"" <> capability <> "\"", - " }", - " }", - "}" - ] + ( [ "jobs {", + " demo {", + " provider \"claude\"", + -- The repository root, which is where a repository-supplied + -- working directory has to stay. + " working-dir \".\"" + ] + <> map (" " <>) extras + <> [ " safety {", + " capability \"" <> capability <> "\"", + " }", + " }", + "}" + ] + ) repositoryBeatsUserTest :: TestTree repositoryBeatsUserTest = @@ -193,6 +257,7 @@ job ^. #envRequires @?= [] job ^. #timeout @?= Nothing job ^. #executable @?= Nothing + job ^. #outputFormat @?= TextFormat singleArgumentListTest :: TestTree singleArgumentListTest = @@ -236,6 +301,22 @@ resolveJob paths noEnvironment [override "jobs.none.extra-dirs" "/from-flag"] "none" overridden ^. #extraDirs @?= ["/from-flag"] +outputFormatTest :: TestTree +outputFormatTest = + testCase "output-format selects the structured result shape" $ + -- The setting that lets an evidence record observe a run without + -- opening the raw-argument channel a ceiling closes by default. + withConfigs Nothing (Just (jobDocWith "read-only" ["output-format \"json\""])) $ \paths -> do + job <- resolveJob paths noEnvironment [] "demo" + job ^. #outputFormat @?= JsonFormat + -- A misspelling fails rather than falling back to text. + shouldFail <- + withConfigs Nothing (Just (jobDocWith "read-only" ["output-format \"jsonl\""])) $ + \badPaths -> resolutionFailure badPaths "demo" + assertBool + ("the offending key is named: " <> Text.unpack shouldFail) + (Text.isInfixOf "jobs.demo.output-format" shouldFail) + missingFileTest :: TestTree missingFileTest = testCase "with no configuration at all the required settings are named" $ @@ -510,6 +591,352 @@ ("the refusal names the permitted maximum: " <> Text.unpack message) (Text.isInfixOf "edit-workspace" message) +-- -------------------------------------------------------------------- +-- Unknown keys +-- -------------------------------------------------------------------- + +-- | The warnings a run of the given job should actually show. +warningsFor :: AgentConfigPaths -> Text -> IO [ConfigWarning] +warningsFor paths jobName = do + loaded <- resolveAgentJob paths noEnvironment [] jobName + case loaded of + Left problem -> + assertFailure ("loading failed: " <> Text.unpack (renderAgentConfigError problem)) + Right resolved -> pure (relevantWarnings jobName (resolved ^. #warnings)) + +-- | Every warning settei raised, before filtering. +allWarningsFor :: AgentConfigPaths -> Text -> IO [ConfigWarning] +allWarningsFor paths jobName = do + loaded <- resolveAgentJob paths noEnvironment [] jobName + case loaded of + Left problem -> + assertFailure ("loading failed: " <> Text.unpack (renderAgentConfigError problem)) + Right resolved -> pure (resolved ^. #warnings) + +otherJobsDoNotWarnTest :: TestTree +otherJobsDoNotWarnTest = + testCase "another job's keys are not warned about" + $ + -- The declaration describes one job, so settei warns about every + -- leaf of every other job in the document. None of them is a + -- mistake, and a file with four jobs printed three jobs' worth of + -- noise on every run. + withConfigs + (Just (Text.unlines ["policy {", " max-capability \"full-access\"", "}"])) + ( Just + ( Text.unlines + [ "jobs {", + " demo { provider \"claude\"; working-dir \".\"", + " safety { capability \"read-only\" }", + " }", + -- A second job in the same node. Two top-level `jobs` + -- nodes would be a repeated node, which settei reads as + -- an array and cannot traverse at all. + " release { provider \"codex\"; working-dir \".\"", + " timout \"5m\"", + " safety { capability \"read-only\" }", + " }", + "}" + ] + ) + ) + $ \paths -> do + -- The premise: settei really does warn about all of it. + raised <- allWarningsFor paths "demo" + assertBool + ("expected settei to warn about the other job: " <> show raised) + (not (null raised)) + kept <- warningsFor paths "demo" + kept @?= [] + +typoInTheSelectedJobStillWarnsTest :: TestTree +typoInTheSelectedJobStillWarnsTest = + testCase "a misspelled key inside the selected job still warns" $ + -- The case that matters: this one silently leaves a default in + -- force, so it must survive the filter. + withConfigs Nothing (Just (jobDocWith "read-only" ["timout \"5m\""])) $ \paths -> do + kept <- warningsFor paths "demo" + assertBool + ("expected the misspelled key to be named: " <> show kept) + (Text.isInfixOf "jobs.demo.timout" (renderWarningsText kept)) + +repositoryPolicyNodeIsNoticedOnceTest :: TestTree +repositoryPolicyNodeIsNoticedOnceTest = + testCase "a repository policy node earns exactly one notice" + $ + -- It does nothing, because the ceiling is read from the operator + -- file only — but whoever wrote it believed it would, and saying + -- nothing leaves them with a policy they think is in force. + withConfigs + Nothing + ( Just + ( jobDoc "read-only" + <> Text.unlines + [ "policy {", + " max-capability \"full-access\"", + " allow-provider-args #true", + "}" + ] + ) + ) + $ \paths -> do + raised <- allWarningsFor paths "demo" + case repositoryPolicyNotice raised of + Nothing -> assertFailure "expected a notice about the repository policy node" + Just notice -> + assertBool + ("expected one line explaining it: " <> Text.unpack notice) + ( Text.isInfixOf "no effect" notice + && length (Text.lines notice) == 1 + ) + -- The operator's own policy node earns none: it is the file the + -- ceiling is read from. + withConfigs + (Just (Text.unlines ["policy {", " max-capability \"full-access\"", "}"])) + (Just (jobDoc "read-only")) + $ \operatorPaths -> do + operatorRaised <- allWarningsFor operatorPaths "demo" + repositoryPolicyNotice operatorRaised @?= Nothing + +-- -------------------------------------------------------------------- +-- Repository scope +-- -------------------------------------------------------------------- + +-- | The violations that depend on which file supplied a value. +-- +-- These cannot come from the pure ceiling check, which sees a request +-- and not its provenance, so they are computed from the resolution +-- report instead. +scopeViolationsFor :: AgentConfigPaths -> Text -> IO [CeilingViolation] +scopeViolationsFor paths jobName = do + loaded <- resolveAgentJob paths noEnvironment [] jobName + case loaded of + Left problem -> + assertFailure ("loading failed: " <> Text.unpack (renderAgentConfigError problem)) + Right resolved -> case resolved ^. #answer of + Left problems -> + assertFailure ("resolution failed: " <> Text.unpack (renderErrorsText problems)) + Right job -> repositoryScopeViolations paths (resolved ^. #report) jobName job + +repositoryExecutableIsRefusedTest :: TestTree +repositoryExecutableIsRefusedTest = + testCase "A REPOSITORY FILE CANNOT SET THE EXECUTABLE" + $ + -- `executable` turns a configuration file into code execution: the + -- named program inherits the operator's environment and receives the + -- prompt on its standard input. A checkout must not choose it. + withConfigs + Nothing + (Just (jobDocWith "edit-workspace" ["executable \"/opt/bin/claude\""])) + $ \paths -> do + violations <- scopeViolationsFor paths "demo" + violations @?= [RepositoryScopeForbidden "executable"] + +repositoryExtraDirsAreRefusedTest :: TestTree +repositoryExtraDirsAreRefusedTest = + testCase "a repository file cannot grant itself extra directories" + $ + -- Inside the root `extra-dirs` adds nothing the working directory + -- does not already give, so the only ones a repository would ask for + -- are outside it. + withConfigs + Nothing + (Just (jobDocWith "edit-workspace" ["extra-dirs \"/Users/op/.ssh\""])) + $ \paths -> do + violations <- scopeViolationsFor paths "demo" + violations @?= [RepositoryScopeForbidden "extra-dirs"] + +operatorScopeMaySetBothTest :: TestTree +operatorScopeMaySetBothTest = + testCase "the operator's own file may set the executable and extra directories" + $ withConfigs + ( Just + ( operatorSettingsDoc + ["executable \"/opt/bin/claude\"", "extra-dirs \"/Users/op/.ssh\""] + ) + ) + (Just (jobDoc "edit-workspace")) + $ \paths -> do + violations <- scopeViolationsFor paths "demo" + violations @?= [] + +workingDirMustStayInsideTheRootTest :: TestTree +workingDirMustStayInsideTheRootTest = + testCase "a repository working directory must resolve inside the repository" $ + withSystemTempDirectory "baikai-agent-workdir" $ \dir -> do + let repositoryRoot = dir "repo" + createDirectoryIfMissing True (repositoryRoot ".baikai") + -- A checkout could commit a symbolic link out of itself, so the + -- check canonicalises before comparing; a textual prefix test + -- would let this through. + createDirectoryLink "/" (repositoryRoot "escape") + let check workingDir = do + TextIO.writeFile + (repositoryRoot ".baikai" "agents.kdl") + (workingDirJobDoc workingDir) + scopeViolationsFor + emptyAgentConfigPaths + { userConfig = Nothing, + repoConfig = Just (repositoryRoot ".baikai" "agents.kdl"), + repositoryRoot + } + "demo" + check "." >>= (@?= []) + check "sub" >>= (@?= []) + inParent <- check ".." + case inParent of + [WorkingDirOutsideRepository _ _] -> pure () + other -> assertFailure ("expected one out-of-root violation, got: " <> show other) + throughLink <- check "escape/etc" + case throughLink of + [WorkingDirOutsideRepository resolved reportedRoot] -> do + -- The violation names where the link actually led, not the + -- spelling the document used, so an operator reading it sees + -- the escape. The exact string is the fully canonical one, + -- which on macOS makes @\/etc@ read @\/private\/etc@ — hence + -- the suffix rather than an equality. + assertBool + ("expected the link's target in the violation: " <> resolved) + ("etc" `isSuffixOf` resolved) + assertBool + ("expected a directory outside the root: " <> resolved) + (not ((reportedRoot <> "/") `isPrefixOf` resolved)) + other -> assertFailure ("expected one out-of-root violation, got: " <> show other) + +executableIsNotEnvBoundTest :: TestTree +executableIsNotEnvBoundTest = + testCase "no environment variable names the executable" $ + -- An environment variable is inherited by every child process and is + -- easy to set by accident, so the one setting that chooses which + -- program runs is not bound to one. + withConfigs Nothing (Just (jobDoc "read-only")) $ \paths -> do + job <- + resolveJob paths (envSnapshot [("BAIKAI_AGENT_EXECUTABLE", "/evil")]) [] "demo" + job ^. #executable @?= Nothing + +ceilingPolicyKeysTest :: TestTree +ceilingPolicyKeysTest = + testCase "the operator's file sets the grant list and both maxima" + $ withConfigs + ( Just + ( Text.unlines + [ "policy {", + " allowed-tools \"Bash\"", + " max-timeout \"2h\"", + " max-output-limit \"unlimited\"", + "}" + ] + ) + ) + (Just (jobDoc "read-only")) + $ \paths -> do + loaded <- loadAgentCeiling paths + case loaded of + Left problem -> + assertFailure + ("loading the ceiling failed: " <> Text.unpack (renderAgentConfigError problem)) + Right ceiling' -> do + ceiling' ^. #allowedTools @?= ["Bash"] + ceiling' ^. #maxTimeout @?= Just 7200 + ceiling' ^. #maxOutputLimit @?= Nothing + -- The unset keys keep their defaults, so an operator who + -- writes one line still gets a complete ceiling. + ceiling' ^. #maxCapability @?= AgentEditWorkspace + +-- | An operator document supplying the given lines to the @demo@ job. +operatorSettingsDoc :: [Text] -> Text +operatorSettingsDoc settings = + Text.unlines + (["jobs {", " demo {"] <> map (" " <>) settings <> [" }", "}"]) + +-- | A complete job rooted at the given working directory. +workingDirJobDoc :: Text -> Text +workingDirJobDoc workingDir = + Text.unlines + [ "jobs {", + " demo {", + " provider \"claude\"", + " working-dir \"" <> workingDir <> "\"", + " safety { capability \"read-only\" }", + " }", + "}" + ] + +ceilingFileInsideRepositoryIsRefusedTest :: TestTree +ceilingFileInsideRepositoryIsRefusedTest = + testCase "A CEILING FILE INSIDE THE REPOSITORY IS REFUSED" $ + -- The other half of "no repository file can raise the ceiling". The + -- source list already refuses the repository *document*; this closes + -- the shape where the repository supplies the *operator* document, + -- which `--user-config .baikai/policy.kdl` and + -- `XDG_CONFIG_HOME=$PWD/.baikai` both produce. + withSystemTempDirectory "baikai-agent-ceiling-inside" $ \dir -> do + let repositoryRoot = dir "repo" + insidePath = repositoryRoot ".baikai" "policy.kdl" + createDirectoryIfMissing True (takeDirectory insidePath) + TextIO.writeFile insidePath raisingPolicyDoc + loaded <- + loadAgentCeiling + emptyAgentConfigPaths + { userConfig = Just insidePath, + repoConfig = Nothing, + repositoryRoot + } + case loaded of + Right ceiling' -> + assertFailure ("the checkout supplied its own ceiling: " <> show ceiling') + Left problem -> do + case problem of + CeilingFileInsideRepository _ _ -> pure () + other -> assertFailure ("expected a location refusal, got: " <> show other) + let message = renderAgentConfigError problem + assertBool + ("both paths are named: " <> Text.unpack message) + ( Text.isInfixOf "policy.kdl" message + && Text.isInfixOf "repo" message + ) + +ceilingFileOutsideTheRepositoryLoadsTest :: TestTree +ceilingFileOutsideTheRepositoryLoadsTest = + testCase "the same file one directory above the repository loads" $ + -- The companion to the case above: the refusal is about where the + -- file is, not about what it says. + withSystemTempDirectory "baikai-agent-ceiling-outside" $ \dir -> do + let repositoryRoot = dir "repo" + outsidePath = dir "policy.kdl" + createDirectoryIfMissing True repositoryRoot + TextIO.writeFile outsidePath raisingPolicyDoc + loaded <- + loadAgentCeiling + emptyAgentConfigPaths + { userConfig = Just outsidePath, + repoConfig = Nothing, + repositoryRoot + } + case loaded of + Left problem -> + assertFailure + ("expected the ceiling to load: " <> Text.unpack (renderAgentConfigError problem)) + Right ceiling' -> ceiling' ^. #maxCapability @?= AgentFullAccess + +unknownPolicyKeyIsAnErrorTest :: TestTree +unknownPolicyKeyIsAnErrorTest = + testCase "a misspelled policy key is an error, not a warning" + $ + -- Everywhere else an unrecognised key is a warning. Under `policy` a + -- typo would silently leave the default in force, which for the one + -- node whose purpose is limiting authority is indefensible. + withConfigs + (Just (Text.unlines ["policy {", " max-capabilty \"read-only\"", "}"])) + (Just (jobDoc "read-only")) + $ \paths -> do + loaded <- loadAgentCeiling paths + case loaded of + Right ceiling' -> + assertFailure ("the typo was ignored: " <> show ceiling') + Left (UnknownPolicySetting _ keys) -> keys @?= ["policy.max-capabilty"] + Left other -> assertFailure ("expected an unknown-key refusal, got: " <> show other) + listJobsTest :: TestTree listJobsTest = testCase "job names are sorted and attributed to the winning scope" @@ -531,7 +958,7 @@ assertFailure ("listing failed: " <> Text.unpack (renderAgentConfigError problem)) Right entries -> do map (^. #name) entries @?= ["demo", "user-only"] - map (^. #scope) entries @?= [RepositoryScope, UserScope] + map (^. #scope) entries @?= [AgentRepositoryScope, AgentUserScope] -- A name defined in two files is reported once, with the -- count, because a bare name hides a real source of -- confusion. diff --git a/test/EvidenceTests.hs b/test/EvidenceTests.hs --- a/test/EvidenceTests.hs +++ b/test/EvidenceTests.hs @@ -26,6 +26,7 @@ import Baikai.Agent.Run ( agentConfigurationEnvelope, agentRequestEnvelope, + errorInfoStderrTailBytes, runAgentCommand, ) import Baikai.Evidence @@ -49,7 +50,8 @@ import Data.Text (Text) import Data.Text qualified as Text import System.Directory - ( doesFileExist, + ( createDirectoryIfMissing, + doesFileExist, getPermissions, setOwnerExecutable, setPermissions, @@ -69,6 +71,8 @@ timedOutRunTest, inheritedOutputTest, nothingStartedTest, + relativeExecutableEndpointTest, + errorInfoIsBoundedTest, optOutTest, strictRefusalTests, digestTests @@ -196,6 +200,57 @@ Right ran -> assertFailure ("expected a spawn failure, got: " <> show ran) Left _ -> pure () outcome ^. #evidence @?= Nothing + +relativeExecutableEndpointTest :: TestTree +relativeExecutableEndpointTest = + testCase "a relative executable is resolved against the working directory" $ + -- The child execs relative to the working directory the runner sets, + -- so the evidence probe — which runs in the parent, whose working + -- directory is somewhere else entirely — has to resolve the same + -- way. Before this it probed the parent's own directory and reported + -- a path that does not exist. + withSystemTempDirectory "baikai-agent-relative" $ \dir -> do + createDirectoryIfMissing True (dir "bin") + _ <- writeFakeExecutable (dir "bin") "fake" "#!/bin/sh\ncat > /dev/null\nexit 0\n" + outcome <- run (wanted "run-relative") dir ("." "bin" "fake") [] + ev <- oneEvidence outcome + case field "endpoint" ev of + Just (Object o) -> + KeyMap.lookup "endpoint" o + @?= Just (String (Text.pack (dir "bin" "fake"))) + other -> assertFailure ("expected an endpoint object, got: " <> show other) + +errorInfoIsBoundedTest :: TestTree +errorInfoIsBoundedTest = + testCase "a failing run's error message keeps the tail of its standard error" + $ + -- The output limit lets a captured stream reach mebibytes, and + -- before this the whole of it went into one error message. The last + -- few kibibytes are where a failing tool's actual reason lives, and + -- the prefix says what was dropped so the message does not read as a + -- corrupted record. + withFake + "#!/bin/sh\ncat > /dev/null\n" + "i=0; while [ $i -lt 2000 ]; do \ + \printf 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\\n' >&2; \ + \i=$((i+1)); done; printf 'final: reason\\n' >&2" + "\nexit 1\n" + $ \dir exe -> do + ev <- oneEvidence =<< run (wanted "run-noisy") dir exe [] + case field "error_info" ev of + Just (Object o) -> case KeyMap.lookup "message" o of + Just (String message) -> do + assertBool + ("expected a bounded message, got " <> show (Text.length message) <> " characters") + (Text.length message < errorInfoStderrTailBytes + 80) + assertBool + ("expected the tool's last line in: " <> Text.unpack message) + ("final: reason" `Text.isInfixOf` message) + assertBool + ("expected the truncation prefix in: " <> Text.unpack message) + ("[stderr truncated" `Text.isPrefixOf` message) + other -> assertFailure ("expected a message string, got: " <> show other) + other -> assertFailure ("expected an error_info object, got: " <> show other) optOutTest :: TestTree optOutTest = diff --git a/test/Main.hs b/test/Main.hs --- a/test/Main.hs +++ b/test/Main.hs @@ -14,6 +14,7 @@ ) import Baikai.Agent.Run (runAgentCommand, timeoutMicros) import Baikai.Evidence (noThinkingRequested) +import BinaryTests (binaryTests) import CliTests (cliTests) import ConfigTests (configTests) import Control.Concurrent (threadDelay) @@ -25,8 +26,10 @@ import Data.Text.Encoding qualified as Text import Data.Time.Clock (diffUTCTime, getCurrentTime) import EvidenceTests (evidenceTests) +import PublicSurfaceSpec qualified import System.Directory ( doesFileExist, + findExecutable, getPermissions, setOwnerExecutable, setPermissions, @@ -44,7 +47,9 @@ "baikai-agent" [ runTests, configTests, - cliTests + cliTests, + binaryTests, + PublicSurfaceSpec.tests ] runTests :: TestTree @@ -60,6 +65,9 @@ missingEnvironmentTest, timeoutTest, processGroupTest, + keepsDrainedOutputOnTimeoutTest, + escalatesToKillTest, + escapedPipeHolderTest, outputLimitTest, inheritOutputTest, promptAsArgumentTest, @@ -205,7 +213,7 @@ -- Names chosen to be absent rather than unset here, so the test -- never mutates the suite's own environment. let names = ["BAIKAI_AGENT_TEST_ABSENT_ONE", "BAIKAI_AGENT_TEST_ABSENT_TWO"] - req = capturingRequest dir "ignored" & #envPassthrough .~ names + req = capturingRequest dir "ignored" & #envRequires .~ names outcome <- runPlain req (stdinCommand (dir "unused") [] "ignored") case outcome of Left (MissingEnvironment missing) -> missing @?= names @@ -220,7 +228,7 @@ outcome <- runPlain req (stdinCommand exe [] "ignored") end <- getCurrentTime case outcome of - Left (RunTimedOut limit) -> limit @?= 1 + Left (RunTimedOut timedOut) -> timedOut ^. #limit @?= 1 other -> assertFailure ("expected RunTimedOut, got: " <> show other) -- Without this assertion the test would pass just as well by -- waiting for the script to finish, which proves nothing about @@ -250,6 +258,100 @@ waitSeconds 4 survived <- doesFileExist marker assertBool "the grandchild was terminated with its group" (not survived) + +-- | A timed-out run reports what it drained before the kill. +-- +-- The bytes were always there — the runner drains both pipes from the +-- moment it spawns — and were simply dropped on the timeout path. For a +-- coding agent the partial answer is often the whole point of the run. +-- +-- Three seconds rather than one, here and in the two cases below. Each +-- of them asserts something about work the child actually did, so the +-- child has to have run; starting @\/bin\/sh@ while the rest of this +-- suite runs in parallel takes longer than a second on a loaded machine, +-- and a one-second deadline made these cases kill a shell that had not +-- yet reached its first line. Three seconds is still far below the +-- thirty each stub sleeps for, so the deadline is still what ends them. +keepsDrainedOutputOnTimeoutTest :: TestTree +keepsDrainedOutputOnTimeoutTest = + testCase "a timed-out run reports the output it drained before the kill" + $ withFakeExecutable + "chatty-sleeper" + "#!/bin/sh\nprintf 'partial\\n'\nsleep 30\n" + $ \dir exe -> do + let req = capturingRequest dir "ignored" & #timeout .~ Just 3 + outcome <- runPlain req (stdinCommand exe [] "ignored") + case outcome of + Left (RunTimedOut timedOut) -> do + capturedBytes (timedOut ^. #stdout) @?= Just "partial\n" + capturedBytes (timedOut ^. #stderr) @?= Just "" + other -> assertFailure ("expected RunTimedOut, got: " <> show other) + +-- | A child that ignores both polite signals still dies. +-- +-- @trap '' INT TERM@ is a shell asking to be left alone, which is +-- exactly the coding agent a deadline exists for. Only SIGKILL ends it, +-- and before this escalation existed the runner waited on such a child +-- for as long as it chose to live. +escalatesToKillTest :: TestTree +escalatesToKillTest = + testCase "a child that ignores INT and TERM is killed within the grace periods" + $ withFakeExecutable + "stubborn" + "#!/bin/sh\ntrap '' INT TERM\nsleep 30\n" + $ \dir exe -> do + let req = capturingRequest dir "ignored" & #timeout .~ Just 3 + start <- getCurrentTime + outcome <- runPlain req (stdinCommand exe [] "ignored") + end <- getCurrentTime + case outcome of + Left (RunTimedOut _) -> pure () + other -> assertFailure ("expected RunTimedOut, got: " <> show other) + -- Three seconds of deadline plus two grace periods of two + -- seconds each, with room to spare; the pre-escalation runner + -- would have waited the script's full thirty. + assertBool + ("killed rather than waited out; the run took " <> show (diffUTCTime end start)) + (diffUTCTime end start < 12) + +-- | A pipe held open from outside the process group does not hang the +-- run. +-- +-- The group is dead, so nothing more will be written, but the drain is +-- still blocked on a read that will never reach end of file because +-- another process holds the write end. The runner interrupts the drain +-- and reports what it has. @perl@ is used only because macOS ships no +-- @setsid@ binary; where it is absent the case explains itself and +-- passes, because a missing tool is not a defect in this package. +escapedPipeHolderTest :: TestTree +escapedPipeHolderTest = + testCase "a pipe held open outside the group does not hang the run" $ do + perl <- findExecutable "perl" + case perl of + Nothing -> putStrLn "skipped: perl not found, so no pipe holder can leave the group" + Just _ -> + withFakeExecutable + "escapes-the-group" + "#!/bin/sh\nperl -MPOSIX -e 'setsid(); sleep 30' &\nprintf 'held\\n'\nsleep 30\n" + $ \dir exe -> do + -- Five seconds rather than three: this stub has to start a + -- second interpreter and let it leave the process group + -- before the deadline, and under a loaded parallel suite + -- three seconds is not always enough for that. When it is + -- not, nothing holds the pipe and the case proves nothing. + let req = capturingRequest dir "ignored" & #timeout .~ Just 5 + start <- getCurrentTime + outcome <- runPlain req (stdinCommand exe [] "ignored") + end <- getCurrentTime + case outcome of + Left (RunTimedOut timedOut) -> + capturedBytes (timedOut ^. #stdout) @?= Just "held\n" + other -> assertFailure ("expected RunTimedOut, got: " <> show other) + assertBool + ( "the blocked drain was interrupted rather than waited on; the run took " + <> show (diffUTCTime end start) + ) + (diffUTCTime end start < 14) outputLimitTest :: TestTree outputLimitTest = diff --git a/test/PublicSurfaceSpec.hs b/test/PublicSurfaceSpec.hs new file mode 100644 --- /dev/null +++ b/test/PublicSurfaceSpec.hs @@ -0,0 +1,57 @@ +{-# LANGUAGE OverloadedRecordDot #-} + +-- | A downstream consumer's view of @baikai-agent@'s library, compiled. +-- +-- Imports only the two public modules and builds each record the way a +-- consumer now must: from its exported base value by record update, +-- since none of the four constructors is exported any more. The +-- compilation is the test. +module PublicSurfaceSpec (tests) where + +import Baikai.Agent + ( AgentCapability (AgentReadOnly), + AgentOutputFormat (JsonFormat), + AgentOutputMode (CaptureOutput), + AgentProvider (AgentClaude), + ) +import Baikai.Agent.Cli + ( AgentCliCommand (AgentList), + AgentCliOptions (jsonOutput, runId), + AgentCliRun (exitCode, standardOutput), + agentCliOptions, + agentCliRun, + ) +import Baikai.Agent.Config + ( AgentConfigPaths (repoConfig, repositoryRoot, userConfig), + AgentJob (capability, modelId, output, outputFormat, provider, workingDir), + agentJob, + emptyAgentConfigPaths, + ) +import Test.Tasty (TestTree, testGroup) +import Test.Tasty.HUnit (testCase, (@?=)) + +tests :: TestTree +tests = + testGroup + "public surface (baikai-agent)" + [ testCase "agentJob supplies the three required fields and sane defaults" $ do + let job = (agentJob AgentClaude "." AgentReadOnly) {modelId = Just "probe-model"} + job.provider @?= AgentClaude + job.workingDir @?= "." + job.capability @?= AgentReadOnly + job.modelId @?= Just "probe-model" + job.output @?= (agentJob AgentClaude "." AgentReadOnly).output + (job {outputFormat = JsonFormat}).outputFormat @?= JsonFormat + (job {output = CaptureOutput}).output @?= CaptureOutput, + testCase "the CLI records are built from their bases" $ do + let opts = (agentCliOptions AgentList) {jsonOutput = True, runId = Just "run-1"} + opts.jsonOutput @?= True + opts.runId @?= Just "run-1" + let run = (agentCliRun 0) {standardOutput = "ok"} + run.exitCode @?= 0 + run.standardOutput @?= "ok", + testCase "emptyAgentConfigPaths names no file and roots at the current directory" $ do + emptyAgentConfigPaths.userConfig @?= Nothing + emptyAgentConfigPaths.repoConfig @?= Nothing + emptyAgentConfigPaths.repositoryRoot @?= "." + ]