diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,165 @@ ## [Unreleased] +## [baikai 0.7.2.0] - 2026-09-30 + +### Added + +- Curated GPT-6.1 Sol on OpenAI Responses and Claude Sonnet 5.5 on Anthropic + Messages (`openai_gpt_6_1_sol`, `anthropic_claude_sonnet_5_5`), with endpoint + compatibility facts, standard prices, GPT-6.1 Sol's 272K-token context tier, + and Sonnet 5.5's one-hour cache-write rate. Sonnet 5.5 rejects forced tool + choice locally and has no fast mode. Both passed live text and function-tool + acceptance on 2026-09-30 through `/v1/responses` and `/v1/messages` + ([record](docs/validation/plan-85/2026-09-30-complete.json), + [plan 85](docs/plans/85-prove-gpt-6-1-sol-and-claude-sonnet-5-5-live-compatibility.md)). + The focused smoke runner gains the `sol61-*` and `sonnet55-*` cases. +- `Baikai.ResponseFormat.StructuredOutputSupport` + (`NativeJsonSchema | NoStructuredOutput`), `declaredStructuredOutput :: Api -> + StructuredOutputSupport`, and a `structuredOutput` field on `ApiProvider` + (default `NoStructuredOutput` in `apiProviderWith`), so a caller can ask + whether a transport enforces a schema without calling it. Every built-in + provider declares `NativeJsonSchema`. + +## [baikai-claude 0.7.1.0] - 2026-09-30 + +Requires `baikai >=0.7.2`. + +### Added + +- `claude -p` honours a `JsonSchema` response format (IR-11): it receives + `--json-schema ''` and the response text is the tool's validated + `structured_output`. A missing `structured_output` is a `DecodeFailure`; a + `claude` too old for the flag yields an `InvalidRequest` error with the exit + code rather than unconstrained text. `JsonObject`, `name` and `strict` are not + forwarded; requests without a schema render the same argument vector as + before. Both Claude providers declare `structuredOutput = NativeJsonSchema`. + +### Fixed + +- Anthropic adaptive `ThinkingHigh` now sends + `output_config.effort: "high"` instead of omitting the field. Claude Opus 5.5 + defaults to `medium`, so it previously ran `ThinkingHigh` at medium effort; + other adaptive models default to `high` and behave as before. Evidence for + these calls records `effortText = "high"` and no longer carries + `effort_omitted`, so strict evidence mode no longer refuses them. + `Baikai.Evidence.EffortOmitted` stays exported, and older records still decode. + +## [baikai-openai 0.7.1.0] - 2026-09-30 + +Requires `baikai >=0.7.2`. + +### Added + +- `codex exec` honours a `JsonSchema` response format (IR-11): the schema is + written to a temporary file passed as `--output-schema ` and deleted + however the call ends. A `codex` too old for the flag yields an + `InvalidRequest` error with the exit code rather than unconstrained text. + `JsonObject`, `name` and `strict` are not forwarded; requests without a schema + render the same argument vector as before. All three OpenAI providers declare + `structuredOutput = NativeJsonSchema`. +- `codexCliCommandWith`, which renders the `codex exec` + vector with a given `--output-schema` file. `codexCliCommand` is unchanged + and never renders the flag. + +## [baikai 0.7.1.0] - 2026-09-23 + +### Added + +- Curated GPT-6 Sol and Luna on OpenAI Responses and Claude Opus 5.5 on + Anthropic Messages (`openai_gpt_6_sol`, `openai_gpt_6_luna`, + `anthropic_claude_opus_5_5`), with endpoint compatibility and standard, + long-context, cache-duration, and fast-mode prices where applicable. All + three passed live acceptance on 2026-09-23. Sol and Luna dispatch to + `OpenAIResponses`, so calling them requires the + `Baikai.Provider.OpenAI.Responses.register` call that `baikai-openai 0.7.0.0` + introduced. + +## [baikai-kit 0.3.0.0] - 2026-09-23 + +Closes four improvement requests from the tools that ship `baikai-kit` as their +`kit` command: project scope resolves from a configurable root (IR-8), +`kit status` reports local edits separately from upstream drift (IR-7), +`kit install` without a name asks a tool-supplied chooser (IR-6), and `list`, +`status` and `update` print versioned JSON (IR-9). A consumer raising its bound +builds `KitConfig` with `kitConfig`, passes it to `kitCommandParser`, and +matches on `StatusRow.conditions`; each break is at a call site the compiler +names. + +### Added + +- `baikai-kit`: `KitConfig.projectRoot :: IO FilePath` says where project scope + lives. Install, status, update, uninstall and `agentDirsForSession` all derive + project-scope paths from it, so they agree whichever subdirectory a command + runs from. `kitConfig` builds a configuration with every optional field at its + default (project scope is the current directory, as before); + `projectRootByMarkers [".git", ".mytool"]` is a ready-made resolver that walks + up to the nearest marker and falls back to the current directory, and + `findProjectRoot` is the underlying walk. Resolves IR-8. + +- `baikai-kit`: `kit status` reports local edits. It runs the same + installed-file check `kit update` uses to skip an item, and shows + `modified` for an edited copy and `edits-unknown` for one whose sidecar + predates the installed-file hash. The check is exported as + `checkLocalEdits`, returning `LocalEdits` (`Unedited`, `Edited`, + `EditsUnknown`). Resolves IR-7. + +- `baikai-kit`: `kit install` with no name asks a chooser the tool supplies in + the new `KitConfig.chooseItem :: Maybe (KitManifest -> IO (Maybe Text))` + field. The engine refreshes the kit, passes the whole manifest, and installs + what the chooser returns; a cancelled choice prints + `No item chosen; nothing installed.` and exits 0. With no chooser (the + `kitConfig` default) the command fails with the new `KitItemNameRequired` + error, which tells the user to pass `NAME`. The engine ships no picker. + `KitCommand` derives `Eq`, and `kit install --help` names the tool's + `./agents` directory. Resolves IR-6. + +- `baikai-kit`: `kit list`, `kit status` and `kit update` accept `--json` and + print exactly one versioned JSON document on stdout + (`{"formatVersion": 1, "document": "kit-list" | "kit-status" | "kit-update", …}`); + warnings and the first-clone notice go to stderr, and a failed command writes + nothing to stdout. The shapes are written by explicit encoders — + `Baikai.Kit.Json.listDocument`, `statusDocument`, `updateDocument`, and + `kitJsonFormatVersion` — so library callers get the same values, and they are + pinned by golden tests. `Baikai.Kit.Command.OutputFormat` selects the mode, + and `Baikai.Kit.Status.InstalledCopy` / `installedCopies` report where each + item is installed. Resolves IR-9. + +### Changed + +- `baikai-kit`: `KitConfig` gains the strict field `projectRoot`, so a record + literal must set it; build the configuration with + `kitConfig toolName repoUrl providers` instead and override fields with record + update syntax. `KitConfig`'s `Show` instance is now hand-written and prints + `` for the resolver. __Breaking__. + +- `baikai-kit`: `kit status` conditions compose, and `dirty` is renamed + `changed-upstream` (it meant the upstream sources changed without a version + bump, not local edits); `dirty+outdated` now reads + `outdated+changed-upstream`. `StatusRow.state :: KitState` is replaced by + `StatusRow.conditions :: [KitCondition]` (sorted; empty means up to date), + `renderState` by `conditionLabel` and `renderConditions`, and `classify` + returns `[KitCondition]`. `KitUpToDate`, `KitDirty` and `KitDirtyOutdated` + are gone; match on the list instead. __Breaking__. + +- `baikai-kit`: `KitInstall` takes `Maybe Text` (`Nothing` asks the chooser), + `kitCommandParser` takes the `KitConfig` (migration: `kitCommandParser` + becomes `kitCommandParser myKitConfig`), `KitConfig` gains the `chooseItem` + field (set by `kitConfig`), and `KitError` gains `KitItemNameRequired`. + __Breaking__. + +- `baikai-kit`: `KitList`, `KitStatus` and `KitUpdate` gain a trailing + `OutputFormat` field (`HumanOutput` for the previous behaviour). + __Breaking__. + +## [baikai-effectful 0.4.0.2] - 2026-09-15 + +### Changed (dependencies) + +- Requires `effectful-core ^>=2.7` (was `^>=2.6`). No API change: none of the + 2.7 breaking APIs (`LocalEnv`'s second type parameter, `SharedSuffix`, + `KnownEffects`, the ticked strict modules) are used. + ## [baikai 0.7.0.0] - 2026-09-08 ### Added diff --git a/baikai-openai.cabal b/baikai-openai.cabal --- a/baikai-openai.cabal +++ b/baikai-openai.cabal @@ -1,27 +1,33 @@ -cabal-version: 3.4 -name: baikai-openai -version: 0.7.0.0 -synopsis: OpenAI providers for the baikai abstraction +cabal-version: 3.4 +name: baikai-openai +version: 0.7.1.0 +synopsis: OpenAI providers for the baikai abstraction description: OpenAI-compatible backends for baikai: Chat Completions over SSE against OpenAI and any host that emulates it, the codex exec batch provider, a launcher for interactive Codex sessions, and the renderer for unattended codex runs driven by baikai-agent. -category: AI -license: BSD-3-Clause -license-file: LICENSE -author: Nadeem Bitar -maintainer: nadeem@gmail.com -copyright: (c) 2026 Nadeem Bitar -build-type: Simple -tested-with: GHC ==9.12.4 +category: AI +license: BSD-3-Clause +license-file: LICENSE +author: Nadeem Bitar +maintainer: nadeem@gmail.com +copyright: (c) 2026 Nadeem Bitar +build-type: Simple +tested-with: ghc ==9.12.4 extra-doc-files: CHANGELOG.md common common-options ghc-options: - -Wall -Wcompat -Widentities -Wincomplete-uni-patterns - -Wincomplete-record-updates -Wredundant-constraints - -fhide-source-paths -Wmissing-export-lists -Wpartial-fields + -Wall + -Wcompat + -Widentities + -Wincomplete-uni-patterns + -Wincomplete-record-updates + -Wredundant-constraints + -fhide-source-paths + -Wmissing-export-lists + -Wpartial-fields -Wmissing-deriving-strategies -- Exhaustiveness is an error, not a warning. A non-exhaustive match @@ -36,10 +42,11 @@ -- fail the build on warnings that are stylistic or that a future GHC -- invents, and would push people toward blanket suppression. ghc-options: - -Werror=incomplete-patterns -Werror=incomplete-uni-patterns + -Werror=incomplete-patterns + -Werror=incomplete-uni-patterns -Werror=incomplete-record-updates - default-language: GHC2024 + default-language: GHC2024 default-extensions: DeriveAnyClass DuplicateRecordFields @@ -47,8 +54,8 @@ OverloadedStrings library - import: common-options - hs-source-dirs: src + import: common-options + hs-source-dirs: src exposed-modules: Baikai.Provider.OpenAI.Agent Baikai.Provider.OpenAI.Api @@ -66,36 +73,37 @@ Baikai.Provider.OpenAI.Sse Baikai.Provider.OpenAI.Transport - other-modules: Paths_baikai_openai + other-modules: Paths_baikai_openai autogen-modules: Paths_baikai_openai build-depends: - , aeson ^>=2.2 - , baikai ^>=0.7.0 - , base >=4.20 && <5 - , base64-bytestring ^>=1.2 - , bytestring ^>=0.12 - , case-insensitive ^>=1.2 - , containers ^>=0.7 - , generic-lens ^>=2.3 - , http-client ^>=0.7 - , http-client-tls >=0.3 && <0.5 - , http-types ^>=0.12 - , lens ^>=5.3 - , openai ^>=2.5 - , process ^>=1.6 - , scientific ^>=0.3 - , servant-client ^>=0.20 - , streamly >=0.11 && <0.13 - , streamly-core >=0.3 && <0.5 - , text ^>=2.1 - , time ^>=1.14 - , vector ^>=0.13 + aeson ^>=2.2, + baikai ^>=0.7.2, + base >=4.20 && <5, + base64-bytestring ^>=1.2, + bytestring ^>=0.12, + case-insensitive ^>=1.2, + containers ^>=0.7, + directory ^>=1.3, + generic-lens ^>=2.3, + http-client ^>=0.7, + http-client-tls >=0.3 && <0.5, + http-types ^>=0.12, + lens ^>=5.3, + openai ^>=2.5, + process ^>=1.6, + scientific ^>=0.3, + servant-client ^>=0.20, + streamly >=0.11 && <0.13, + streamly-core >=0.3 && <0.5, + text ^>=2.1, + time ^>=1.14, + vector ^>=0.13, test-suite baikai-openai-test - import: common-options - type: exitcode-stdio-1.0 + import: common-options + type: exitcode-stdio-1.0 hs-source-dirs: test - main-is: Main.hs + main-is: Main.hs other-modules: BillingSpec CliEvidenceSpec @@ -114,31 +122,32 @@ ResponsesTransportSpec ShapeSpec SseSpec + StructuredCliSpec TransportSpec build-depends: - , aeson - , baikai ^>=0.7.0 - , baikai-openai - , base >=4.20 && <5 - , bytestring - , case-insensitive - , containers - , directory - , filepath - , generic-lens - , http-client - , http-types - , lens ^>=5.3 - , network - , openai - , servant-client - , stm - , streamly-core >=0.3 && <0.5 - , tasty - , tasty-hunit - , temporary - , text ^>=2.1 - , time - , tls - , vector + aeson, + baikai ^>=0.7.2, + baikai-openai, + base >=4.20 && <5, + bytestring, + case-insensitive, + containers, + directory, + filepath, + generic-lens, + http-client, + http-types, + lens ^>=5.3, + network, + openai, + servant-client, + stm, + streamly-core >=0.3 && <0.5, + tasty, + tasty-hunit, + temporary, + text ^>=2.1, + time, + tls, + vector, diff --git a/src/Baikai/Provider/OpenAI/Api.hs b/src/Baikai/Provider/OpenAI/Api.hs --- a/src/Baikai/Provider/OpenAI/Api.hs +++ b/src/Baikai/Provider/OpenAI/Api.hs @@ -37,6 +37,7 @@ import Baikai.Provider.OpenAI.Internal.Stream (liveSseDriver, openaiChatStreamWith) import Baikai.Provider.OpenAI.Shape (describeThinkingShape) import Baikai.Provider.Registry (registerApiProvider) +import Baikai.ResponseFormat (declaredStructuredOutput) import Baikai.Stream.Event (AssistantMessageEvent) import Control.Lens ((&), (.~), (^.)) import Data.Generics.Labels () @@ -59,6 +60,7 @@ describeThinkingShape (openaiCompletionsCompatFor m) (m ^. #reasoning) opts ) & #strengthCeiling .~ Ev.declaredStrength OpenAIChatCompletions + & #structuredOutput .~ declaredStructuredOutput OpenAIChatCompletions -- | Streaming producer for the OpenAI Chat Completions API. -- diff --git a/src/Baikai/Provider/OpenAI/Cli.hs b/src/Baikai/Provider/OpenAI/Cli.hs --- a/src/Baikai/Provider/OpenAI/Cli.hs +++ b/src/Baikai/Provider/OpenAI/Cli.hs @@ -18,6 +18,18 @@ -- it ran, not which model served the request, so a successful exit -- never raises the recorded 'Baikai.Evidence.EvidenceStrength' — see -- 'Baikai.Provider.Cli.Internal.subprocessStrength'. +-- +-- Structured output: a 'Baikai.ResponseFormat.JsonSchema' in +-- 'Baikai.Options.responseFormat' is written, as UTF-8 JSON bytes, to a +-- temporary file passed as @--output-schema @; the file is removed +-- when the call ends, however it ends. The response text is the final +-- agent message, which the tool has constrained to the schema. Codex +-- submits the schema in strict mode, so it must satisfy OpenAI's strict +-- rules (for example @additionalProperties: false@). An installed +-- @codex@ too old to know the flag yields an 'Baikai.Error.InvalidRequest' +-- error carrying the exit code, never unconstrained text. +-- 'Baikai.ResponseFormat.JsonObject' and the schema's @name@ and +-- @strict@ have no CLI analogue and are not forwarded. module Baikai.Provider.OpenAI.Cli ( CodexCliConfig, executable, @@ -26,6 +38,7 @@ skipGitRepoCheck, ephemeral, codexCliCommand, + codexCliCommandWith, codexCliPrompt, codexCliThinking, defaultCodexCliConfig, @@ -50,17 +63,21 @@ registerApiProvider, ) import Baikai.Response qualified as Resp +import Baikai.ResponseFormat (ResponseFormat (..), declaredStructuredOutput) import Baikai.StopReason (StopReason (..)) import Baikai.Stream (liftCompleteToStream) import Baikai.ThinkingLevel (ThinkingLevel, renderThinkingLevel) import Baikai.Usage (Usage, zeroUsage) import Control.Concurrent (forkIO) import Control.Concurrent.MVar (newEmptyMVar, putMVar, takeMVar) -import Control.Exception (SomeException, displayException, fromException, try) +import Control.Exception (IOException, SomeException, bracket, displayException, fromException, try) import Control.Lens ((&), (.~), (^.)) +import Control.Monad (void) +import Data.Aeson qualified as Aeson import Data.ByteString qualified as BS +import Data.ByteString.Lazy qualified as LBS import Data.Generics.Labels () -import Data.Maybe (fromMaybe) +import Data.Maybe (fromMaybe, isJust) import Data.Text (Text) import Data.Text qualified as Text import Data.Time.Clock (UTCTime, diffUTCTime, getCurrentTime) @@ -68,8 +85,9 @@ import GHC.Generics (Generic) import Streamly.Data.Stream (Stream) import Streamly.Data.Stream qualified as Stream +import System.Directory (getTemporaryDirectory, removeFile) import System.Exit (ExitCode (..)) -import System.IO (Handle) +import System.IO (Handle, hClose, openBinaryTempFile) import System.Process qualified as P -- | Configuration for the @codex exec --json@ subprocess. @@ -114,6 +132,7 @@ -- control is a command-line flag derived from Options alone. & #describeThinking .~ (\_ opts -> codexCliThinking opts) & #strengthCeiling .~ Ev.declaredStrength OpenAICompletionsCli + & #structuredOutput .~ declaredStructuredOutput OpenAICompletionsCli modelArgs :: Model -> [String] modelArgs m = case Text.strip (m ^. #modelId) of @@ -140,8 +159,21 @@ -- | Render the executable and arguments for a @codex exec --json@ -- batch call. The prompt is preceded by @--@ so dash-leading prompts -- cannot be parsed as options. +-- +-- This never renders @--output-schema@, even when +-- 'Baikai.Options.responseFormat' carries a schema: the schema file +-- exists only while a call runs, so a pure renderer has no path to +-- name. Use 'codexCliCommandWith' to see the vector a schema call +-- spawns. codexCliCommand :: CodexCliConfig -> Model -> Context -> Options -> (FilePath, [String]) -codexCliCommand cfg m ctx opts = +codexCliCommand cfg = codexCliCommandWith cfg Nothing + +-- | 'codexCliCommand' with the schema file to pass as +-- @--output-schema@, rendered after the reasoning-effort override and +-- before 'extraArgs'. 'Nothing' renders exactly 'codexCliCommand'. +codexCliCommandWith :: + CodexCliConfig -> Maybe FilePath -> Model -> Context -> Options -> (FilePath, [String]) +codexCliCommandWith cfg schemaFile m ctx opts = ( cfg ^. #executable, ["exec"] <> modelArgs m @@ -149,6 +181,7 @@ <> ["--skip-git-repo-check" | cfg ^. #skipGitRepoCheck] <> ["--ephemeral" | cfg ^. #ephemeral] <> effortArgs opts + <> maybe [] (\path -> ["--output-schema", path]) schemaFile <> fmap Text.unpack (cfg ^. #extraArgs) <> ["--", Text.unpack (codexCliPrompt ctx)] ) @@ -192,11 +225,44 @@ adjustments = [Ev.EffortClamped lvl wire | wire /= renderThinkingLevel lvl] } +-- | The schema a caller asked the tool to enforce, if any. +requestedSchema :: Options -> Maybe Aeson.Value +requestedSchema opts = case opts ^. #responseFormat of + Just (JsonSchema f) -> Just (f ^. #schema) + _ -> Nothing + +-- | Write the schema to a fresh temporary file for the duration of one +-- action, and remove it afterwards however the action ends. +-- +-- The bytes come from 'Aeson.encode', which is UTF-8 by construction; +-- nothing here goes through a locale-dependent text handle. A failure +-- to remove the file is ignored: the call's outcome is already decided, +-- and a stray file in the temporary directory is harmless. +withSchemaFile :: Aeson.Value -> (FilePath -> IO a) -> IO a +withSchemaFile schemaDoc k = do + dir <- getTemporaryDirectory + bracket + (openBinaryTempFile dir "baikai-codex-schema.json") + (\(path, h) -> hClose h >> void (try (removeFile path) :: IO (Either IOException ()))) + ( \(path, h) -> do + LBS.hPut h (Aeson.encode schemaDoc) + hClose h + k path + ) + +compactJson :: Aeson.Value -> String +compactJson = Text.unpack . Internal.decodeUtf8Lenient . LBS.toStrict . Aeson.encode + runCodexCli :: CodexCliConfig -> Model -> Context -> Options -> IO Resp.Response runCodexCli cfg m ctx opts = do - let (exe, args) = codexCliCommand cfg m ctx opts - procSpec = - (P.proc exe args) + let schema = requestedSchema opts + -- The envelope names the schema itself where the spawned vector + -- names its temporary file: the random path would make the + -- request commitment unreproducible, and the schema is what + -- actually crossed the boundary. + (exe, args) = codexCliCommandWith cfg (compactJson <$> schema) m ctx opts + procSpec schemaFile = + (P.proc exe (snd (codexCliCommandWith cfg schemaFile m ctx opts))) { P.std_in = P.NoStream, P.std_out = P.CreatePipe, P.std_err = P.CreatePipe, @@ -220,7 +286,15 @@ st mErr traverse (observeCodexCli exe mReport st) prepared - result <- Internal.trySync (P.withCreateProcess procSpec (consume start mkEv m)) + launch schemaFile = + P.withCreateProcess (procSpec schemaFile) (consume start mkEv (isJust schema) m) + -- Writing the schema file sits inside 'Internal.trySync' too, so a + -- temporary directory that cannot be written becomes an error-shaped + -- response rather than an exception escaping the provider. + result <- + Internal.trySync $ case schema of + Nothing -> launch Nothing + Just schemaDoc -> withSchemaFile schemaDoc (launch . Just) case result of Right resp -> pure resp Left ex -> do @@ -238,13 +312,15 @@ Maybe BaikaiError -> IO (Maybe Ev.ModelCallEvidence) ) -> + -- | Whether @--output-schema@ was sent. + Bool -> Model -> Maybe Handle -> Maybe Handle -> Maybe Handle -> P.ProcessHandle -> IO Resp.Response -consume start mkEv m _ mOut mErr ph = do +consume start mkEv schemaSent m _ mOut mErr ph = do case (mOut, mErr) of (Nothing, _) -> errorNow (providerError "codex: stdout handle missing") (_, Nothing) -> errorNow (providerError "codex: stderr handle missing") @@ -260,7 +336,11 @@ end <- getCurrentTime case exitCode of ExitFailure n -> do - let err = processError n (Internal.decodeUtf8Lenient errBytes) + let stderr = Internal.decodeUtf8Lenient errBytes + flagRejected + | schemaSent = Internal.unsupportedFlagError "codex" "--output-schema" n stderr + | otherwise = Nothing + err = fromMaybe (processError n stderr) flagRejected -- The event stream was drained before the exit status was -- known, so a failed run may still have named its thread and -- its token counts. Those are genuine observations and are diff --git a/src/Baikai/Provider/OpenAI/Responses.hs b/src/Baikai/Provider/OpenAI/Responses.hs --- a/src/Baikai/Provider/OpenAI/Responses.hs +++ b/src/Baikai/Provider/OpenAI/Responses.hs @@ -16,6 +16,7 @@ import Baikai.Provider.OpenAI.Responses.Request (describeThinking) import Baikai.Provider.OpenAI.Responses.Stream (liveResponsesDriver, openaiResponsesStreamWith) import Baikai.Provider.Registry (registerApiProvider) +import Baikai.ResponseFormat (declaredStructuredOutput) import Baikai.Stream.Event (AssistantMessageEvent) import Control.Lens ((&), (.~)) import Data.Generics.Labels () @@ -30,6 +31,7 @@ apiProvider OpenAIResponses openaiResponsesStream & #describeThinking .~ describeThinking & #strengthCeiling .~ Ev.declaredStrength OpenAIResponses + & #structuredOutput .~ declaredStructuredOutput OpenAIResponses openaiResponsesStream :: Model -> Context -> Options -> Stream IO AssistantMessageEvent openaiResponsesStream = openaiResponsesStreamWith liveResponsesDriver diff --git a/test/Main.hs b/test/Main.hs --- a/test/Main.hs +++ b/test/Main.hs @@ -56,6 +56,7 @@ import ShapeSpec qualified import SseSpec qualified import Streamly.Data.Stream qualified as Stream +import StructuredCliSpec qualified import System.Directory (getPermissions, getTemporaryDirectory, setOwnerExecutable, setPermissions) import System.Environment (lookupEnv, setEnv, unsetEnv) import System.FilePath (()) @@ -113,6 +114,7 @@ ResponsesTransportSpec.tests, ShapeSpec.tests, SseSpec.tests, + StructuredCliSpec.tests, TransportSpec.tests ] diff --git a/test/ResponsesSpec.hs b/test/ResponsesSpec.hs --- a/test/ResponsesSpec.hs +++ b/test/ResponsesSpec.hs @@ -3,7 +3,7 @@ module ResponsesSpec (tests) where import Baikai hiding (model, schema) -import Baikai.Models.Generated (openai_gpt_6_astra) +import Baikai.Models.Generated (openai_gpt_6_1_sol, openai_gpt_6_astra, openai_gpt_6_luna, openai_gpt_6_sol) import Baikai.Provider.OpenAI.Responses.Request qualified as R import Control.Lens ((&), (.~)) import Control.Monad (forM_) @@ -21,7 +21,25 @@ tests = testGroup "Responses request mapping" - [ testCase "stateless request carries text, image, system, cap and metadata" $ do + [ testCase "Sol, GPT-6.1 Sol, and Luna catalog bindings retain Responses shaping and replay identity" $ + forM_ [openai_gpt_6_sol, openai_gpt_6_1_sol, openai_gpt_6_luna] $ \m -> do + m.api @?= OpenAIResponses + let opts = emptyOptions & #thinking .~ Just ThinkingLow & #temperature .~ Just 0.5 & #topP .~ Just 0.8 + thoughtForModel = emptyThinkingContent & #replayState .~ Just (ThinkingReplay OpenAIResponses m.modelId (V.singleton reasoningItem)) + response = emptyResponse & #message . #content .~ V.fromList [AssistantThinking thoughtForModel, AssistantToolCall (ToolCall "call_7" "lookup" (object ["x" .= (1 :: Int)]))] & #message . #stopReason .~ ToolUse + ctx = contextOf [user "go"] & #tools .~ V.singleton tool + first <- mapped m ctx opts + field "model" first.requestBody @?= Just (String m.modelId) + field "reasoning" first.requestBody @?= Just (object ["effort" .= ("low" :: Text)]) + field "temperature" first.requestBody @?= Nothing + field "top_p" first.requestBody @?= Nothing + next <- appendToolResult ctx response (\_ -> pure (toolResultText "found")) + second <- mapped m next opts + let items = inputItems second + items !! 1 @?= reasoningItem + field "call_id" (items !! 2) @?= Just (String "call_7") + field "call_id" (items !! 3) @?= Just (String "call_7"), + testCase "stateless request carries text, image, system, cap and metadata" $ do let ctx = (systemUser "system instruction" "hello") & #messages .~ V.singleton (UserMessage UserPayload {content = V.fromList [UserText (TextContent "hello"), UserImage (ImageContent "abc" "image/png")], timestamp = Nothing}) diff --git a/test/StructuredCliSpec.hs b/test/StructuredCliSpec.hs new file mode 100644 --- /dev/null +++ b/test/StructuredCliSpec.hs @@ -0,0 +1,271 @@ +-- | Structured-output passthrough for the @codex exec --json@ +-- subprocess provider. +-- +-- The process cases run a real child process: a few lines of @sh@ +-- written into a temporary directory that record the argument vector, +-- copy the @--output-schema@ file before the provider can delete it, +-- print a canned event stream and stderr, and exit with a chosen code. +module StructuredCliSpec (tests) where + +import Baikai +import Baikai.Provider.OpenAI.Cli qualified as CodexCli +import Control.Lens ((&), (.~), (^.)) +import Data.Aeson (Value (..), (.=)) +import Data.Aeson qualified as Aeson +import Data.Aeson.KeyMap qualified as KeyMap +import Data.ByteString qualified as BS +import Data.ByteString.Lazy qualified as LBS +import Data.Generics.Labels () +import Data.List (isPrefixOf) +import Data.Text (Text) +import Data.Text qualified as Text +import Data.Text.Encoding qualified as Text +import Data.Text.IO qualified as TextIO +import Data.Vector qualified as Vector +import System.Directory (doesFileExist, getPermissions, setOwnerExecutable, setPermissions) +import System.FilePath (()) +import System.IO.Temp (withSystemTempDirectory) +import Test.Tasty (TestTree, testGroup) +import Test.Tasty.HUnit (assertBool, assertFailure, testCase, (@?=)) + +tests :: TestTree +tests = + testGroup + "StructuredCliSpec: codex exec --output-schema passthrough" + [ renderingTests, + processTests, + testCase "the provider declares NativeJsonSchema without spawning anything" $ + CodexCli.codexCliProvider CodexCli.defaultCodexCliConfig ^. #structuredOutput + @?= NativeJsonSchema + ] + +-- ============================================================ +-- Pure argument rendering +-- ============================================================ + +renderingTests :: TestTree +renderingTests = + testGroup + "argument rendering" + [ testCase "JsonObject renders the same vector as no response format" $ + CodexCli.codexCliCommand cfg testModel testContext (emptyOptions & #responseFormat .~ Just JsonObject) + @?= CodexCli.codexCliCommand cfg testModel testContext emptyOptions, + testCase "codexCliCommand renders no --output-schema even for a JsonSchema request" $ + assertBool + "no schema flag" + ("--output-schema" `notElem` snd (CodexCli.codexCliCommand cfg testModel testContext schemaOptions)), + testCase "codexCliCommandWith Nothing is codexCliCommand" $ + CodexCli.codexCliCommandWith cfg Nothing testModel testContext schemaOptions + @?= CodexCli.codexCliCommand cfg testModel testContext schemaOptions, + testCase "codexCliCommandWith places --output-schema before the extra args" $ do + let (_, args) = CodexCli.codexCliCommandWith cfg (Just "/tmp/s.json") testModel testContext schemaOptions + case break (== "--output-schema") args of + (_, "--output-schema" : "/tmp/s.json" : rest) -> + assertBool ("extra args follow the schema: " <> show rest) (["--color", "never"] `isPrefixOf` rest) + _ -> assertFailure ("no --output-schema in " <> show args) + ] + where + cfg = CodexCli.defaultCodexCliConfig {CodexCli.extraArgs = ["--color", "never"]} + +-- ============================================================ +-- A real child process +-- ============================================================ + +processTests :: TestTree +processTests = + testGroup + "fake codex process" + [ testCase "the schema file carries the schema, the reply is returned unchanged, the file is removed" $ + withFakeCodex $ \run -> do + (resp, captured) <- run schemaOptions (Fake conformingEvents "" 0) + responseError resp @?= Nothing + flattenAssistantText (resp ^. #message . #content) @?= conformingText + case captured of + Nothing -> assertFailure "the tool received no --output-schema" + Just (path, bytes) -> do + Aeson.decodeStrict bytes @?= Just itemsSchema + doesFileExist (Text.unpack path) >>= (@?= False), + testCase "the schema file is removed after a failed run too" $ + withFakeCodex $ \run -> do + (resp, captured) <- run schemaOptions (Fake [] "the tool refused\n" 3) + fmap (^. #category) (responseError resp) @?= Just ProcessFailure + case captured of + Nothing -> assertFailure "the tool received no --output-schema" + Just (path, _) -> doesFileExist (Text.unpack path) >>= (@?= False), + testCase "no response format sends no --output-schema" $ + withFakeCodex $ \run -> do + (resp, captured) <- run emptyOptions (Fake conformingEvents "" 0) + responseError resp @?= Nothing + fmap fst captured @?= Nothing, + testCase "codex rejecting --output-schema is InvalidRequest with the exit code" $ + withFakeCodex $ \run -> do + (resp, _) <- run schemaOptions (Fake [] "error: unexpected argument '--output-schema' found\n" 2) + case responseError resp of + Nothing -> assertFailure "expected an error-shaped response" + Just e -> do + e ^. #category @?= InvalidRequest + e ^. #exitCode @?= Just 2, + testCase "the same stderr without a schema request stays a ProcessFailure" $ + withFakeCodex $ \run -> do + (resp, _) <- run emptyOptions (Fake [] "error: unexpected argument '--output-schema' found\n" 2) + fmap (^. #category) (responseError resp) @?= Just ProcessFailure, + -- The spawned vector names a fresh random path each call; the + -- evidence envelope names the schema instead, so the commitment + -- is reproducible and still changes with the schema. + testCase "the request commitment is reproducible and commits to the schema" $ + withFakeCodex $ \run -> do + (first, _) <- run (schemaOptions & #evidence .~ Just (evidenceRequest "run-87")) (Fake conformingEvents "" 0) + (second, _) <- run (schemaOptions & #evidence .~ Just (evidenceRequest "run-87")) (Fake conformingEvents "" 0) + (plain, _) <- run (emptyOptions & #evidence .~ Just (evidenceRequest "run-87")) (Fake conformingEvents "" 0) + let c1 = encodedCommitment first + assertBool ("a commitment was recorded: " <> show c1) (c1 /= Nothing) + c1 @?= encodedCommitment second + assertBool "the schema is part of the commitment" (c1 /= encodedCommitment plain) + ] + +-- | What the fake prints and how it exits. +data Fake = Fake + { eventLines :: [Text], + stderrBody :: Text, + code :: Int + } + +-- | Write one fake @codex@ and hand back a function that runs a call +-- against it and returns the response with the schema path and bytes +-- the tool received. +-- +-- Calls whose commitments are compared must share one fake, because +-- the argument vector the commitment digests begins with the +-- executable's path. +withFakeCodex :: ((Options -> Fake -> IO (Response, Maybe (Text, BS.ByteString))) -> IO a) -> IO a +withFakeCodex k = + withSystemTempDirectory "baikai-codex-structured" $ \dir -> do + let outPath = dir "stdout" + errPath = dir "stderr" + codePath = dir "code" + pathPath = dir "schema-path" + copyPath = dir "schema-copy" + exe <- + writeFakeExecutable dir "codex" $ + unlines + [ "#!/bin/sh", + "if [ \"$1\" = \"--version\" ]; then echo 'codex-cli 9.9.9'; exit 0; fi", + "rm -f '" <> pathPath <> "' '" <> copyPath <> "'", + "prev=", + "for a in \"$@\"; do", + " if [ \"$prev\" = \"--output-schema\" ]; then", + " printf '%s' \"$a\" > '" <> pathPath <> "'", + " cp \"$a\" '" <> copyPath <> "'", + " fi", + " prev=\"$a\"", + "done", + "cat '" <> outPath <> "'", + "cat '" <> errPath <> "' >&2", + "exit $(cat '" <> codePath <> "')" + ] + reg <- newProviderRegistry + registerApiProviderWith + reg + (CodexCli.codexCliProvider CodexCli.defaultCodexCliConfig {CodexCli.executable = exe}) + k $ \opts fake -> do + TextIO.writeFile outPath (Text.unlines (eventLines fake)) + TextIO.writeFile errPath (stderrBody fake) + writeFile codePath (show (code fake)) + resp <- completeRequestWith reg testModel testContext opts + sent <- doesFileExist pathPath + captured <- + if sent + then do + path <- TextIO.readFile pathPath + bytes <- BS.readFile copyPath + pure (Just (path, bytes)) + else pure Nothing + pure (resp, captured) + +writeFakeExecutable :: FilePath -> String -> String -> IO FilePath +writeFakeExecutable dir name body = do + let path = dir name + writeFile path body + perms <- getPermissions path + setPermissions path (setOwnerExecutable True perms) + pure path + +-- | The encoded @request_commitment@ of a response's evidence record. +encodedCommitment :: Response -> Maybe Value +encodedCommitment resp = case Aeson.toJSON <$> (resp ^. #evidence) of + Just (Object o) -> KeyMap.lookup "request_commitment" o + _ -> Nothing + +-- ============================================================ +-- Fixtures +-- ============================================================ + +testModel :: Model +testModel = + emptyModel + & #modelId .~ "gpt-5.6" + & #api .~ OpenAICompletionsCli + & #provider .~ "openai" + +testContext :: Context +testContext = emptyContext & #messages .~ Vector.singleton (user "List two fruits.") + +schemaOptions :: Options +schemaOptions = + emptyOptions & #responseFormat .~ Just (JsonSchema (jsonSchemaFormat "fruits" itemsSchema)) + +-- | An object holding an array of objects, one field an enum, in the +-- strict shape codex requires. +itemsSchema :: Value +itemsSchema = + Aeson.object + [ "type" .= ("object" :: Text), + "properties" + .= Aeson.object + [ "items" + .= Aeson.object + [ "type" .= ("array" :: Text), + "items" + .= Aeson.object + [ "type" .= ("object" :: Text), + "properties" + .= Aeson.object + [ "label" .= Aeson.object ["type" .= ("string" :: Text)], + "level" + .= Aeson.object + [ "type" .= ("string" :: Text), + "enum" .= (["low", "high"] :: [Text]) + ] + ], + "required" .= (["label", "level"] :: [Text]), + "additionalProperties" .= False + ] + ] + ], + "required" .= (["items"] :: [Text]), + "additionalProperties" .= False + ] + +conformingText :: Text +conformingText = + "{\"items\":[{\"label\":\"Strawberry\",\"level\":\"low\"},{\"label\":\"Mango\",\"level\":\"high\"}]}" + +-- | The event stream @codex-cli 0.159.0@ emits for an @--output-schema@ +-- run: the structured reply is the text of the ordinary final +-- @agent_message@. +conformingEvents :: [Text] +conformingEvents = + [ "{\"type\":\"thread.started\",\"thread_id\":\"019fd471-4a48-7c83-be67-6b7c49646e87\"}", + "{\"type\":\"turn.started\"}", + Text.decodeUtf8 . LBS.toStrict . Aeson.encode $ + Aeson.object + [ "type" .= ("item.completed" :: Text), + "item" + .= Aeson.object + [ "id" .= ("item_0" :: Text), + "type" .= ("agent_message" :: Text), + "text" .= conformingText + ] + ], + "{\"type\":\"turn.completed\",\"usage\":{\"input_tokens\":10,\"cached_input_tokens\":0,\"output_tokens\":20}}" + ]