diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,96 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.4.3.0] - 2026-11-04 + +### Added + +- `LLM.Core.Types.validateConversation` (also in `LLM.Core`): problems in + the tool-call structure of a conversation, by turn index. +- `LLM.Tools.FsLimits.maxSearchBytes` (64 MiB): the file bytes one `grep` + call may read. +- `LLM.Tools.MultiReplaceInFile.Replacement`, the type of one edit in + `MultiReplaceInFileToolArgs`, exported without its constructor. Before, + the exported field `_mrifReplacements :: [Replacement]` named a type + that could not be imported. + +### Fixed + +- `grep` reads at most `maxSearchBytes` (64 MiB) of files per call. Only + the text files it searches count; binary files and files over the 1 MiB + per-file cap are not read and do not count. A search that runs out + stops before the next file and says so in the header ("stopped after + reading N bytes; narrow the path or pattern"), with the file count shown + as `N+`. Before, a search with few matches could read 20,000 files of + 1 MiB, about 20 GiB, in one call. Each file is now read only up to the + size charged for it, so a file that grows during the search is not read + past the cap. **Behaviour change** for searches over more than 64 MiB. +- A `get_history` page is at most `maxOutputBytes` (100 KiB). Before, + user and assistant text was rendered in full, so one page could be + larger than the context window that `agContextWindow` protects. A page + over the cap keeps its newest turns and ends with a line saying how many + older turns were left out; a newest turn that is over the cap on its own + is cut and marked (`[turn cut: N bytes, first M shown]`). The paging is + unchanged: the turns left out are not moved to the next chunk. + **Behaviour change** for pages over 100 KiB. +- A streamed response is at most 32 MiB in total, the same cap as a + non-streaming body. Before, only one event (8 MiB) was bounded, so an + endpoint sending small events forever kept the call reading, and the + parsers keeping every delta, without end. Past the cap the stream ends + with `ParseError "response body exceeds 33554432 bytes"`, as for an + oversized event. **Behaviour change** for streams over 32 MiB. +- Every generation entry point also checks the tool-call structure of the + request's messages (`validateConversation`): a tool turn that does not + directly follow an assistant turn with tool calls, tool calls not + followed by a tool turn, a call without a result, a call answered twice, + a result for an unknown call ID, or a call ID used twice in one turn. + Each problem is logged like an invalid turn + (`onLog Warn "turn : "`, in turn order) and the request is + still sent. Providers rejected such histories or dropped parts of them, + each differently. **0.5 will reject them** with an error. +- An agent run (`LLM.Agent.generateText`, `streamText`) checks the input + conversation once, at the start of the run, instead of on every model + call, so an invalid turn is logged once per run instead of once per + round. `n` in `turn : ` now counts in the input conversation, + also when `agContextWindow` sends only part of it. The turns the agent + adds come from providers and are not checked. + +- The source distribution (the Hackage tarball) includes the test + fixtures (`test/fixtures/*.json`, `test/fixtures/*.sse`). Before, the + test suite run from it failed 41 tests on missing files. CI now also + runs the tests from the unpacked tarball. +- Structured output with OpenAI (and Ollama, which shares the schema): + strict mode lists every property as required, so a field from + `optionalField`, `optionalFieldWithDefault` or + `optionalFieldWithOmittedDefault` had to be filled in by the model. + `normalizeSchemaOpenAI` now makes such a property nullable + (`anyOf: [, {"type": "null"}]`), in nested objects, array + items, union branches and `$defs` too. `genObject` and + `LLM.Agent.generateObject` drop a `null` for a property the codec has + optional and not nullable before decoding, for every provider, so the + field gets its default; before, that `null` was a decode error. + Nullable fields (`Maybe` with `requiredField`, `optionalFieldOrNull`) + are unchanged. `genObjectUntyped` and `generateObjectUntyped` return + the provider's JSON as before, `null`s included. **Behaviour change** + for structured-output types with optional fields. +- Structured output with OpenAI failed with 400 ("'oneOf' is not + permitted") for any type with an `Either` field or another disjoint + union, since strict mode refuses `oneOf`. `normalizeSchemaOpenAI` now + turns `oneOf` into `anyOf`; the branches cannot both match, so the + meaning is the same. It also changes only schema keywords now: before, + it added `required` and `additionalProperties` to objects inside + literal values such as `default` or `enum`. +- Structured output with OpenAI or Ollama failed with 400 ("reference to + component '#/$defs/' which was not found in the schema") for any + type using a `named` codec below the root: autodocodec puts the + `$defs` next to the first use, but `#/$defs/` points from the + root. `normalizeSchemaOpenAI` now moves every `$defs` entry to the + root. +- The bundled `model-catalog.json` set `temperature: 0.5` for + `gpt_5_6_terra`, which the model refuses ("Only the default (1) value + is supported"), so every call with it failed with 400. The entry now + leaves the temperature unset. + ## [0.4.2.0] - 2026-11-03 ### Added @@ -740,6 +830,7 @@ DeepSeek), single-shot generation with fallbacks, agent tool loops, structured output, JSON model catalog loading, and workspace-scoped filesystem tools. +[0.4.3.0]: https://github.com/aische/llm-simple/compare/v0.4.2.0...v0.4.3.0 [0.4.2.0]: https://github.com/aische/llm-simple/compare/v0.4.1.0...v0.4.2.0 [0.4.1.0]: https://github.com/aische/llm-simple/compare/v0.4.0.1...v0.4.1.0 [0.4.0.1]: https://github.com/aische/llm-simple/compare/v0.4.0.0...v0.4.0.1 diff --git a/Readme.md b/Readme.md --- a/Readme.md +++ b/Readme.md @@ -227,7 +227,7 @@ Set `rtReadonly = True` to omit mutating tools from the schema sent to the model and to block them at execution time. -Every request's messages are checked with `validateTurn` (for example, no tool-call parts in a user turn); an invalid turn is logged as a warning and the request is still sent. From 0.5 it will be an error. +Every request's messages are checked with `validateTurn` (for example, no tool-call parts in a user turn) and `validateConversation` (tool results follow the assistant turn whose calls they answer, each call answered exactly once); each problem is logged as a warning and the request is still sent. From 0.5 it will be an error. Log messages of an agent run (`onLog`) start with `[] `; leave `agName` empty for no prefix. @@ -289,7 +289,7 @@ - The sandbox is **path-scoped**, not kernel-level isolation. Use a dedicated, disposable workspace directory — not your home directory or a production tree. - There is no denylist for sensitive in-workspace files (`.env`, `.git/config`, etc.). Anything inside the workspace is accessible to tools the model can call. - Read tools enforce byte caps: whole-file and edit tools use a 1 MiB limit; `read_file_paginated` accepts source files up to 10 MiB and returns at most 100 KiB per page. -- Search and listing tools bound their work: `find_files`, `grep` and `directory_tree` look at no more than 20,000 entries per call and say so when they stop; tool output is capped at 100 KiB. +- Search and listing tools bound their work: `find_files`, `grep` and `directory_tree` look at no more than 20,000 entries per call, `grep` reads no more than 64 MiB of files per call, and each says so when it stops; tool output is capped at 100 KiB. - `rtReadonly` omits mutating tools from the model schema and blocks them at execution time if they are invoked anyway. - `writefile` and the replace tools replace the file as a whole (a temporary file renamed over it), so a failed write leaves the old content. The file keeps its mode bits but gets a new inode, so hard links to it keep the old content. Concurrent edits of one file from the same process are serialized; edits from other processes are not. diff --git a/internal/LLM/Tools/Internal/TextFile.hs b/internal/LLM/Tools/Internal/TextFile.hs new file mode 100644 --- /dev/null +++ b/internal/LLM/Tools/Internal/TextFile.hs @@ -0,0 +1,164 @@ +-- | Text reads on one open 'Handle': the binary sniff, the size check and +-- the read itself all use the same open file, instead of reopening the +-- path for each step. Shared by 'LLM.Tools.FsLimits' and the tools. +module LLM.Tools.Internal.TextFile + ( Opener, + binarySniffBytes, + withTextHandle, + CappedRead (..), + readCapped, + readCappedWith, + readBoundedBytesWith, + readBoundedBytesWithSize, + readBoundedTextWith, + readBoundedUtf8With, + readLenient, + writeUtf8, + ) +where + +import Control.Exception (IOException, throwIO, try) +import GHC.IO.Exception (IOException (ioe_filename)) +import Data.ByteString qualified as BS +import Data.Maybe (fromMaybe) +import Data.Text (Text) +import Data.Text qualified as T +import Data.Text.Encoding (decodeUtf8', decodeUtf8With, encodeUtf8) +import Data.Text.Encoding.Error (lenientDecode) +import System.IO (Handle, SeekMode (AbsoluteSeek), hFileSize, hSeek) + +-- | Opens a file for reading and runs an action on it, e.g. +-- @withBinaryFile path ReadMode@. +type Opener = forall a. (Handle -> IO a) -> IO a + +-- | Number of leading bytes inspected when sniffing for binary content. +-- Matches the heuristic used by @git diff@ and @grep@: any NUL byte in +-- the prefix means binary. +binarySniffBytes :: Int +binarySniffBytes = 8192 + +-- | Open a file, sniff its first 'binarySniffBytes' for a NUL byte, and run +-- the action from the start of the file if there is none. +-- +-- 'Nothing' if the file looks binary, or cannot be opened or sniffed: the +-- caller fails closed rather than read an unreadable file as text. An +-- 'IOException' from the action itself is rethrown. +withTextHandle :: Opener -> (Handle -> IO r) -> IO (Maybe r) +withTextHandle open act = do + r <- try $ open $ \h -> do + sniff <- try (BS.hGet h binarySniffBytes) + case sniff of + Left (_ :: IOException) -> pure (Right Nothing) + Right bs + | BS.elem 0 bs -> pure (Right Nothing) + | otherwise -> do + hSeek h AbsoluteSeek 0 + fmap Just <$> try (act h) + case r of + Left (_ :: IOException) -> pure Nothing + Right (Left (e :: IOException)) -> throwIO e + Right (Right x) -> pure x + +-- | What 'readCapped' found. +data CappedRead + = -- | The file is over the cap at the size check; its size. + CappedTooLarge Integer + | -- | The file was within the cap at the size check but went past it + -- before the read ended. + CappedGrew + | -- | The size at the check, and the file's bytes. + CappedBytes Integer BS.ByteString + deriving (Eq, Show) + +-- | Check the size of an open file, then read it, refusing more than @cap@ +-- bytes. Reads one byte past the cap, so a file that grows past it after +-- the size check is reported as 'CappedGrew' instead of looking like it +-- ended at the cap. +readCapped :: Integer -> Handle -> IO CappedRead +readCapped = readCappedWith hFileSize + +-- | 'readCapped' with the size check given, so a test can stand in for a +-- file that grows between the check and the read. +readCappedWith :: (Handle -> IO Integer) -> Integer -> Handle -> IO CappedRead +readCappedWith sizeOf cap h = do + size <- sizeOf h + if size > cap + then pure (CappedTooLarge size) + else do + bytes <- BS.hGet h (fromIntegral cap + 1) + pure $ + if fromIntegral (BS.length bytes) > cap + then CappedGrew + else CappedBytes size bytes + +-- | Read a file up to @cap@ bytes. Refuses binary files, checks the size +-- before reading, and reads at most @cap + 1@ bytes so a file that grows +-- during the read is caught ('readCapped'). +readBoundedBytesWith :: Opener -> Text -> Int -> IO (Either Text BS.ByteString) +readBoundedBytesWith = readBoundedBytesWithSize hFileSize + +-- | 'readBoundedBytesWith' with the size check given, so a test can stand +-- in for a file that grows between the check and the read. An +-- 'IOException' from the size check or the read becomes an error naming +-- that step. +readBoundedBytesWithSize :: (Handle -> IO Integer) -> Opener -> Text -> Int -> IO (Either Text BS.ByteString) +readBoundedBytesWithSize sizeOf open display cap = do + r <- withTextHandle open $ \h -> do + sizeRes <- try (sizeOf h) + case sizeRes of + Left (e :: IOException) -> + pure $ Left $ "could not stat " <> display <> ": " <> describeIOError e + Right sz -> do + readRes <- try (readCappedWith (const (pure sz)) (fromIntegral cap) h) + pure $ case readRes of + Left (e :: IOException) -> + Left $ "could not read " <> display <> ": " <> describeIOError e + Right (CappedTooLarge _) -> + Left $ + display + <> " is " + <> T.pack (show sz) + <> " bytes, which exceeds the " + <> T.pack (show cap) + <> "-byte cap." + Right CappedGrew -> + Left $ + display + <> " grew past the " + <> T.pack (show cap) + <> "-byte cap during the read." + Right (CappedBytes _ bs) -> Right bs + pure $ + fromMaybe + (Left (display <> " appears to be a binary file; only text files are supported.")) + r + +-- | An 'IOException' without its file name, which for a handle is the +-- absolute path it was opened with. Callers name the file themselves. +describeIOError :: IOException -> Text +describeIOError e = T.pack (show e {ioe_filename = Nothing}) + +-- | 'readBoundedBytesWith', decoded for display. Invalid UTF-8 bytes become +-- U+FFFD, so never write the result back. +readBoundedTextWith :: Opener -> Text -> Int -> IO (Either Text Text) +readBoundedTextWith open display cap = + fmap (decodeUtf8With lenientDecode) <$> readBoundedBytesWith open display cap + +-- | 'readBoundedBytesWith', for editing. Refuses files that are not valid +-- UTF-8, since writing them back would destroy their bytes. +readBoundedUtf8With :: Opener -> Text -> Int -> IO (Either Text Text) +readBoundedUtf8With open display cap = do + r <- readBoundedBytesWith open display cap + pure $ + r >>= \bs -> case decodeUtf8' bs of + Right t -> Right t + Left _ -> Left $ display <> " is not valid UTF-8; refusing to edit it." + +-- | Read the rest of the handle as UTF-8, replacing invalid bytes with +-- U+FFFD. Independent of the locale. +readLenient :: Handle -> IO Text +readLenient h = decodeUtf8With lenientDecode <$> BS.hGetContents h + +-- | Write text as UTF-8, independent of the locale. +writeUtf8 :: Handle -> Text -> IO () +writeUtf8 h = BS.hPut h . encodeUtf8 diff --git a/llm-simple.cabal b/llm-simple.cabal --- a/llm-simple.cabal +++ b/llm-simple.cabal @@ -2,7 +2,7 @@ name: llm-simple -version: 0.4.2.0 +version: 0.4.3.0 synopsis: Multi-provider LLM library with agent tool loops and filesystem tools @@ -41,6 +41,8 @@ extra-source-files: model-catalog.json providers.json + test/fixtures/*.json + test/fixtures/*.sse tested-with: GHC ==9.6.7 @@ -75,6 +77,18 @@ -Wunused-imports -Werror=missing-fields +-- Leaf modules shared by the library and the test suite. Private: other +-- packages cannot depend on it. +library internal + import: opts + exposed-modules: + LLM.Tools.Internal.TextFile + hs-source-dirs: internal + build-depends: + base >=4.18 && <4.22, + bytestring >=0.11 && <0.13, + text >=1.2 && <2.2 + library import: opts exposed-modules: @@ -140,13 +154,17 @@ other-modules: LLM.Core.Internal.Body + LLM.Core.Internal.Schema + LLM.Core.Internal.Transport LLM.Core.SSE + LLM.Generate.Internal.Generate + LLM.Generate.Internal.GenerateUtils LLM.Tools.Internal.PathLock LLM.Tools.Internal.TextEdit - LLM.Tools.Internal.TextFile LLM.Tools.Internal.WalkBudget hs-source-dirs: src build-depends: + llm-simple:internal, aeson >=2.0 && <2.3, aeson-pretty >=0.8 && <0.9, autodocodec >=0.5 && < 0.6, @@ -241,6 +259,8 @@ LLM.HttpConfigSpec LLM.LLMProviderSpec LLM.GenerateObjectSpec + LLM.OptionalFields + LLM.TextFileSpec LLM.ToolUtilsSpec LLM.CrossProviderSpec LLM.ScaleSpec @@ -248,6 +268,7 @@ LLM.WeatherTool hs-source-dirs: test build-depends: + llm-simple:internal, base >=4.18 && <4.22, containers >=0.6.7 && <0.8, directory >=1.3 && <1.4, @@ -257,6 +278,7 @@ aeson >=2.0 && <2.3, aeson-pretty >=0.8 && <0.9, autodocodec >=0.5 && < 0.6, + autodocodec-schema >=0.2 && <0.3, bytestring >=0.11 && <0.13, hspec >=2.10 && <2.12, heptapod >=1.1 && <1.2, diff --git a/model-catalog.json b/model-catalog.json --- a/model-catalog.json +++ b/model-catalog.json @@ -26,7 +26,6 @@ "pricePerMillionOutput": 12.0 }, "maxTokens": 4096, - "temperature": 0.5, "requestTimeout": 10000, "throttleDelay": 1000, "retryCount": 3, diff --git a/src/LLM.hs b/src/LLM.hs --- a/src/LLM.hs +++ b/src/LLM.hs @@ -5,9 +5,9 @@ -- A typical agent application wires four pieces together: -- -- 1. __Models__ — load a 'ModelConfig' (and optional fallbacks) from a JSON --- catalog via 'loadModelOrThrow' or 'loadModelsOrThrow'. See 'LLM.Load' for +-- catalog via 'loadModelOrThrow' or 'loadModelsOrThrow'. See "LLM.Load" for -- catalog format, provider names, and environment variables. --- 2. __Tools__ — build a 'ToolMap' of implementations the model may call. Use +-- 2. __Tools__ — build a t'LLM.Agent.Types.ToolMap' of implementations the model may call. Use -- 'fsTools' for the bundled workspace-scoped filesystem tools, or define -- custom tools with 'TypedTool' and 'toTool'. -- 3. __Agent__ — configure an 'Agent' (system prompt, tool allow-list, optional @@ -37,7 +37,7 @@ -- * __Agent layer__ ('LLM.Agent') — multi-round tool loop, context windowing, -- automatic @get_history@ injection, lifecycle events, and readonly enforcement. -- Prefer this for chat agents and coding assistants. --- * __Generate layer__ ('LLM.Generate') — single provider request with retries +-- * __Generate layer__ ("LLM.Generate") — single provider request with retries -- and model fallbacks ('generateTextWithFallbacks', 'streamTextWithFallbacks', -- 'genObject'). Use directly when you control the conversation loop yourself, -- or via 'createGenRequest' to share agent configuration. diff --git a/src/LLM/Agent.hs b/src/LLM/Agent.hs --- a/src/LLM/Agent.hs +++ b/src/LLM/Agent.hs @@ -1,11 +1,11 @@ -- | Multi-round agent loop with tool execution. -- -- 'generateText' and 'streamText' call the model, execute any returned tool --- calls, append 'ToolTurn' results, and repeat until the model replies with --- plain text or a limit is hit ('Agent.agMaxToolRounds', abort signal). +-- calls, append v'LLM.Core.Types.ToolTurn' results, and repeat until the model replies with +-- plain text or a limit is hit ('LLM.Agent.Types.agMaxToolRounds', abort signal). -- --- Tool implementations live in a separate 'ToolMap'; 'Agent.agTools' is the --- allow-list of names exposed to the model. When 'Agent.agContextWindow' is +-- Tool implementations live in a separate 'ToolMap'; 'LLM.Agent.Types.agTools' is the +-- allow-list of names exposed to the model. When 'LLM.Agent.Types.agContextWindow' is -- set, older turns are dropped from the provider request and a @get_history@ -- tool is injected automatically so the model can page through hidden history. -- diff --git a/src/LLM/Agent/Events.hs b/src/LLM/Agent/Events.hs --- a/src/LLM/Agent/Events.hs +++ b/src/LLM/Agent/Events.hs @@ -13,7 +13,7 @@ ) import LLM.Core.Utils (trySync) --- | Emit a lifecycle event, attaching 'rtGenerationId' from runtime. +-- | Emit a lifecycle event, attaching 'LLM.Agent.Types.rtGenerationId' from runtime. -- Synchronous observer exceptions are swallowed so observers cannot abort -- the loop; asynchronous ones (timeout, cancellation) propagate. emitEvent :: RuntimeArgs -> GenerateEventDetail -> IO () diff --git a/src/LLM/Agent/Generate.hs b/src/LLM/Agent/Generate.hs --- a/src/LLM/Agent/Generate.hs +++ b/src/LLM/Agent/Generate.hs @@ -31,10 +31,11 @@ ) import LLM.Core.Usage (Usage (..), emptyUsage) import LLM.Core.Utils (getToolCalls, toolResult) -import LLM.Generate.Generate - ( generateTextWithFallbacksRotated, - streamTextWithFallbacksRotated, +import LLM.Generate.Internal.Generate + ( generateTextWithFallbacksRotatedWith, + streamTextWithFallbacksRotatedWith, ) +import LLM.Generate.Internal.GenerateUtils (TurnCheck (SkipTurnCheck), logInvalidTurns) import LLM.Generate.Logger (Hooks (..), LogLevel (Warn)) import LLM.Generate.ModelConfig ( ModelWithFallbacks (..), @@ -49,15 +50,18 @@ -- | Run the agent loop until the model returns final text or the run fails. -- --- On success, 'gtrNewMessages' contains only the turns produced during this +-- On success, 'LLM.Generate.Types.gtrNewMessages' contains only the turns produced during this -- call (assistant and tool turns), not the input conversation. -- Tool rounds continue until the model responds without tool calls or --- 'Agent.agMaxToolRounds' rounds have run. Then one final model call asks +-- 'LLM.Agent.Types.agMaxToolRounds' rounds have run. Then one final model call asks -- for a text answer ('toolBudgetUsedUpPrompt'); tool calls in that reply -- are not run, and the run ends with 'GErrToolExceeded'. -- --- The warnings of 'LLM.Agent.ToolUtils.validateAgent' go to 'onLog' at --- the start of the run. +-- The warnings of 'LLM.Agent.ToolUtils.validateAgent' go to 'LLM.Generate.Logger.onLog' at +-- the start of the run, and so do the problems of the input conversation +-- ('LLM.Core.Types.validateTurns', 'LLM.Core.Types.validateConversation'), +-- once per run, as @turn : @ with @n@ counting from 0 in the +-- input conversation. generateText :: Agent -> ModelWithFallbacks -> @@ -67,7 +71,7 @@ IO (Either GenerateErrorResult GenerateTextResult) generateText agent models toolMap rt initialTurns = agentLoop - (\a m r t -> generateTextWithFallbacksRotated (createGenRequest id a toolMap r t) m) + (\a m r t -> generateTextWithFallbacksRotatedWith SkipTurnCheck (createGenRequest id a toolMap r t) m) agent models toolMap @@ -89,7 +93,7 @@ IO (Either GenerateErrorResult GenerateTextResult) streamText onChunk agent models toolMap rt initialTurns = agentLoop - (\a m r t -> streamTextWithFallbacksRotated onChunk (createGenRequest id a toolMap r t) m) + (\a m r t -> streamTextWithFallbacksRotatedWith SkipTurnCheck onChunk (createGenRequest id a toolMap r t) m) agent models toolMap @@ -97,7 +101,7 @@ initialTurns -- | The line appended to the system prompt for the final model call, made --- once 'Agent.agMaxToolRounds' tool rounds have run. +-- once 'LLM.Agent.Types.agMaxToolRounds' tool rounds have run. toolBudgetUsedUpPrompt :: Text toolBudgetUsedUpPrompt = "The tool budget for this run is used up. Answer in text; do not call tools." @@ -106,7 +110,9 @@ toolNotRunText :: Text toolNotRunText = "Not run: tool round limit reached" --- | The tool loop. @call@ returns the model chain rotated so the model that +-- | The tool loop. The caller's turns are checked once, here; @call@ must +-- not check them again (the turns the loop adds come from providers). +-- @call@ returns the model chain rotated so the model that -- answered comes first; the next round starts with that chain, so a run that -- fell back stays on the fallback instead of retrying a failing primary every -- round. The next 'generateText' / 'streamText' call starts from the caller's @@ -121,6 +127,7 @@ IO (Either GenerateErrorResult GenerateTextResult) agentLoop call agent models toolMap rt0 initialTurns = do mapM_ (rt.rtHooks.onLog Warn) (validateAgent agent toolMap) + logInvalidTurns rt.rtHooks initialTurns emitEvent rt GenerationStarted go models initialTurns [] emptyUsage 0 where diff --git a/src/LLM/Agent/GenerateObject.hs b/src/LLM/Agent/GenerateObject.hs --- a/src/LLM/Agent/GenerateObject.hs +++ b/src/LLM/Agent/GenerateObject.hs @@ -22,7 +22,7 @@ -- | Generate a typed Haskell value from the model via Autodocodec. -- --- Tools are never sent: @grTools@ is always empty. 'agContextWindow' still +-- Tools are never sent: @grTools@ is always empty. 'LLM.Agent.Types.agContextWindow' still -- truncates messages, but does not inject @get_history@. generateObject :: (GeneratableObject t) => diff --git a/src/LLM/Agent/ToolUtils.hs b/src/LLM/Agent/ToolUtils.hs --- a/src/LLM/Agent/ToolUtils.hs +++ b/src/LLM/Agent/ToolUtils.hs @@ -61,7 +61,7 @@ fromMaybe (toolResult tc "Aborted") <$> executeToolInterruptible Nothing hooks ctx tools tc -- | Like 'executeTool', but the tool is interrupted when the signal fires. --- Returns 'Nothing' if it was; 'onToolError' then reports @\"Aborted\"@. +-- Returns 'Nothing' if it was; 'LLM.Generate.Logger.onToolError' then reports @\"Aborted\"@. executeToolInterruptible :: Maybe AbortSignal -> Hooks -> ToolContext -> [Tool Text] -> ToolCall -> IO (Maybe ToolResult) executeToolInterruptible mSig hooks ctx tools tc = case lookup tc.tcName toolMap of Nothing -> do @@ -172,7 +172,7 @@ windowOffset Nothing _ = 0 windowOffset (Just n) conv = findNthUserFromEnd n conv --- | Find the index of the Nth 'UserTurn' from the end of a conversation. +-- | Find the index of the Nth v'LLM.Core.Types.UserTurn' from the end of a conversation. -- Returns 0 if there are fewer than @n@ user messages, or if @n <= 0@. findNthUserFromEnd :: Int -> [Turn] -> Int findNthUserFromEnd n conv @@ -183,7 +183,7 @@ -- -- Applies context windowing ('windowOffset'), resolves tools from the -- 'ToolMap' (including auto-injected @get_history@ when windowed), and --- respects 'RuntimeArgs.rtReadonly' when selecting tool definitions. +-- respects 'LLM.Agent.Types.rtReadonly' when selecting tool definitions. createGenRequest :: (Text -> result) -> Agent -> ToolMap result -> RuntimeArgs -> [Turn] -> GenRequest createGenRequest embed agent toolMap rt messages = let offset = windowOffset agent.agContextWindow messages @@ -199,8 +199,8 @@ -- | Like 'createGenRequest', but never attaches tool definitions. -- --- Still applies 'agContextWindow' truncation to 'grMessages'. Use this for --- structured-output paths ('generateObject' / 'generateObjectUntyped'), which +-- Still applies 'LLM.Agent.Types.agContextWindow' truncation to 'LLM.Generate.Types.grMessages'. Use this for +-- structured-output paths ('LLM.Agent.GenerateObject.generateObject' / 'LLM.Agent.GenerateObject.generateObjectUntyped'), which -- cannot execute tools and must not advertise @get_history@. createGenRequestNoTools :: Agent -> RuntimeArgs -> [Turn] -> GenRequest createGenRequestNoTools agent rt messages = @@ -214,9 +214,9 @@ grHooks = rt.rtHooks } --- | The tools sent to the model: the 'Agent.agTools' names found in the +-- | The tools sent to the model: the 'LLM.Agent.Types.agTools' names found in the -- 'ToolMap', each once (the first occurrence counts), read-only ones only --- when 'rtReadonly' is set, followed by the built-in @get_history@ when +-- when 'LLM.Agent.Types.rtReadonly' is set, followed by the built-in @get_history@ when -- windowing is on. A caller tool named @get_history@ is dropped while -- windowing is on. Names missing from the map are skipped; -- 'validateAgent' reports them. @@ -252,14 +252,14 @@ -- | Warnings about an agent configuration that the agent loop works around: -- --- * a name listed more than once in 'agTools' (only the first counts); --- * a name in 'agTools' that the 'ToolMap' lacks (it is not sent); +-- * a name listed more than once in 'LLM.Agent.Types.agTools' (only the first counts); +-- * a name in 'LLM.Agent.Types.agTools' that the 'ToolMap' lacks (it is not sent); -- * a caller tool named @get_history@ while windowing is on (the built-in -- one is sent instead); --- * @'agContextWindow' = 'Just' n@ with @n <= 0@ (no windowing). +-- * @'LLM.Agent.Types.agContextWindow' = 'Just' n@ with @n <= 0@ (no windowing). -- -- 'LLM.Agent.generateText' and 'LLM.Agent.streamText' send each warning --- to 'onLog' at 'LLM.Generate.Logger.Warn' once at the start of a run. +-- to 'LLM.Generate.Logger.onLog' at 'LLM.Generate.Logger.Warn' once at the start of a run. -- An empty list means none apply. validateAgent :: Agent -> ToolMap result -> [Text] validateAgent agent toolMap = @@ -320,7 +320,7 @@ -- | The runtime arguments the agent entry points ('LLM.Agent.generateText', -- 'LLM.Agent.streamText', 'LLM.Agent.generateObject', -- 'LLM.Agent.generateObjectUntyped') run with: 'safeRuntimeArgs', and every --- 'onLog' message prefixed with @[\] @. An empty 'agName' adds +-- 'LLM.Generate.Logger.onLog' message prefixed with @[\] @. An empty 'LLM.Agent.Types.agName' adds -- no prefix. agentRuntimeArgs :: Agent -> RuntimeArgs -> RuntimeArgs agentRuntimeArgs agent rt = safeRuntimeArgs rt {rtHooks = named rt.rtHooks} diff --git a/src/LLM/Agent/Tools/HistoryTool.hs b/src/LLM/Agent/Tools/HistoryTool.hs --- a/src/LLM/Agent/Tools/HistoryTool.hs +++ b/src/LLM/Agent/Tools/HistoryTool.hs @@ -18,6 +18,8 @@ import Data.Text qualified as T import GHC.Generics (Generic) import LLM.Agent.Types (ToolContext (..)) +import LLM.Tools.FsLimits (maxOutputBytes) +import LLM.Tools.Internal.WalkBudget (textPrefixBytes, utf8Length) import LLM.Core.Types ( ToolCall (tcName), ToolResult (trContent, trName), @@ -53,7 +55,7 @@ chunks = chunkBackward nUserMessages hidden if chunkIdx < 0 || chunkIdx >= length chunks then pure "(no more history)" - else pure $ formatChunk (chunks !! chunkIdx) + else pure $ formatChunkWithin maxOutputBytes (chunks !! chunkIdx) historyToolTyped :: TypedTool ToolContext HistoryToolArgs historyToolTyped = @@ -63,12 +65,15 @@ "Retrieve earlier conversation history that is not in your current context window. " <> "Pass chunk=0 for the most recent hidden history, chunk=1 for the one before that, etc. " <> "Returns \"(no earlier history)\" when nothing is hidden, and \"(no more history)\" " - <> "when the chunk index is out of range.", + <> "when the chunk index is out of range. A page is at most " + <> T.pack (show maxOutputBytes) + <> " bytes: past that its oldest turns are left out (the last line says how many), " + <> "and a single turn over the cap is cut and marked.", ttoolReadonly = True, ttoolExecute = getHistoryExecTyped } --- | Count the number of 'UserTurn's in a conversation. +-- | Count the number of v'LLM.Core.Types.UserTurn's in a conversation. countUserTurns :: [Turn] -> Int countUserTurns = length . filter isUserTurn @@ -77,7 +82,7 @@ isUserTurn _ = False -- | Split a conversation into pages of @n@ user messages each, working --- backward from the end. Each page starts at a 'UserTurn'. +-- backward from the end. Each page starts at a v'LLM.Core.Types.UserTurn'. -- Chunk 0 is the most recent page, chunk 1 the one before, etc. -- The oldest chunk (highest index) may contain fewer than @n@ user messages. -- @@ -124,6 +129,45 @@ -- | Format a chunk of conversation turns as readable text. formatChunk :: [Turn] -> Text formatChunk = T.intercalate "\n" . map formatTurn + +-- | 'formatChunk' within @cap@ UTF-8 bytes. A page over the cap keeps +-- its newest turns and ends with a line saying how many older turns of +-- the page were left out. If the newest turn alone is over the cap, its +-- text is cut and marked instead, so the page is never empty. +formatChunkWithin :: Int -> [Turn] -> Text +formatChunkWithin cap turns + | utf8Length full <= cap = full + | otherwise = T.intercalate "\n" (reverse keptNewestFirst ++ note) + where + full = formatChunk turns + rendered = map formatTurn turns + -- Room for the note and the cut marker. + budget = max 0 (cap - 256) + keptNewestFirst = case keepNewest 0 (reverse rendered) of + [] -> [cutTurn (last rendered)] + kept -> kept + dropped = length turns - length keptNewestFirst + note = + [ "[" + <> T.pack (show dropped) + <> " earlier turn(s) of this page left out: the page is over " + <> T.pack (show cap) + <> " bytes]" + | dropped > 0 + ] + keepNewest _ [] = [] + keepNewest used (l : ls) + | used + size > budget = [] + | otherwise = l : keepNewest (used + size) ls + where + size = utf8Length l + 1 + cutTurn l = + textPrefixBytes budget l + <> " [turn cut: " + <> T.pack (show (utf8Length l)) + <> " bytes, first " + <> T.pack (show budget) + <> " shown]" formatTurn :: Turn -> Text formatTurn (UserMessage parts) = diff --git a/src/LLM/Agent/Types.hs b/src/LLM/Agent/Types.hs --- a/src/LLM/Agent/Types.hs +++ b/src/LLM/Agent/Types.hs @@ -26,7 +26,7 @@ -- | Static agent configuration shared across requests. data Agent = Agent - { -- | Name of the agent. The agent entry points prefix every 'onLog' + { -- | Name of the agent. The agent entry points prefix every 'LLM.Generate.Logger.onLog' -- message of a run with @[\] @; an empty name adds no prefix. -- It is not sent to the model. agName :: Text, @@ -43,7 +43,7 @@ -- answer ('LLM.Agent.Generate.toolBudgetUsedUpPrompt'). A text reply -- ends the run successfully. Tool calls in that reply are not run: each -- gets the result @\"Not run: tool round limit reached\"@ and the run - -- returns 'GErrToolExceeded'. So every tool that runs has its result + -- returns v'LLM.Generate.Types.GErrToolExceeded'. So every tool that runs has its result -- seen by the model, and a run makes at most @agMaxToolRounds + 1@ -- model calls. A value @<= 0@ makes only the final call. agMaxToolRounds :: Int, @@ -69,20 +69,20 @@ -- | Observer for 'GenerateEvent' lifecycle notifications (UI, telemetry). rtOnEvent :: EventObserver, -- | When 'True', mutating tools are rejected at execution time even if - -- they appear in 'Agent.agTools'. Read-only tools still run. + -- they appear in 'LLM.Agent.Types.agTools'. Read-only tools still run. rtReadonly :: Bool } -- | A tool: its definition (sent to the model) paired with its implementation. -- --- 'toolExecute' receives a 'ToolContext' (full conversation + usage) and +-- 'LLM.Agent.Types.toolExecute' receives a 'ToolContext' (full conversation + usage) and -- the JSON arguments from the model. data Tool result = Tool { toolDef :: ToolDef, toolExecute :: ToolContext -> Value -> IO result } --- | Map from tool name to implementation. Built manually or via 'fsTools'. +-- | Map from tool name to implementation. Built manually or via 'LLM.Load.FsTools.fsTools'. type ToolMap result = Map Text (Tool result) -- | Context passed to tool implementations during execution. @@ -91,14 +91,14 @@ tcConversation :: [Turn], -- | Accumulated token usage so far tcUsage :: Usage, - -- | Index into 'tcConversation' where the visible window starts. + -- | Index into 'LLM.Agent.Types.tcConversation' where the visible window starts. -- Everything before this index is hidden from the model. -- The @get_history@ tool uses this to serve paginated history. tcWindowOffset :: Int, tcRuntimeArgs :: RuntimeArgs } --- | Generation lifecycle event tagged with the run's 'rtGenerationId'. +-- | Generation lifecycle event tagged with the run's 'LLM.Agent.Types.rtGenerationId'. data GenerateEvent = GenerateEvent { geGenerationId :: UUID, geDetail :: GenerateEventDetail diff --git a/src/LLM/Core.hs b/src/LLM/Core.hs --- a/src/LLM/Core.hs +++ b/src/LLM/Core.hs @@ -33,6 +33,7 @@ coalesceAdjacentTextParts, validateTurn, validateTurns, + validateConversation, ToolCall (..), ToolResult (..), LLMHooks (..), diff --git a/src/LLM/Core/Abort.hs b/src/LLM/Core/Abort.hs --- a/src/LLM/Core/Abort.hs +++ b/src/LLM/Core/Abort.hs @@ -11,7 +11,7 @@ import Control.Concurrent.STM (TVar, atomically, check, newTVarIO, readTVar, readTVarIO, writeTVar) -- | A cancellation signal. --- Create one with 'newAbortSignal', pass it into 'RuntimeArgs' or 'GenRequest', and call +-- Create one with 'newAbortSignal', pass it into t'LLM.Agent.Types.RuntimeArgs' or t'LLM.Generate.Types.GenRequest', and call -- 'abort' from any thread. A model call or tool that is running when the signal -- fires is interrupted. newtype AbortSignal = AbortSignal (TVar Bool) diff --git a/src/LLM/Core/Internal/Body.hs b/src/LLM/Core/Internal/Body.hs --- a/src/LLM/Core/Internal/Body.hs +++ b/src/LLM/Core/Internal/Body.hs @@ -1,9 +1,10 @@ -- | Bounded reading of provider response bodies. -- --- A 200 body larger than 'maxResponseBodyBytes' (or a stream event larger --- than 'maxStreamEventBytes') is signalled with 'ResponseTooLarge', which +-- A 200 body larger than 'maxResponseBodyBytes' (a stream larger than it +-- in total, or a stream event larger than 'maxStreamEventBytes') is +-- signalled with 'ResponseTooLarge', which -- is not exported from any public module. The @generic*@ functions in --- 'LLM.Core.LLMProvider' catch it and return a 'ParseError'; the type of +-- 'LLM.Core.LLMProvider' catch it and return a v'LLM.Core.Types.ParseError'; the type of -- 'LLM.Core.LLMProvider.sendRequest' stays unchanged. module LLM.Core.Internal.Body ( ResponseTooLarge (..), @@ -28,16 +29,17 @@ instance Exception ResponseTooLarge --- | The 'ParseError' text for 'ResponseTooLarge'. +-- | The v'LLM.Core.Types.ParseError' text for 'ResponseTooLarge'. responseTooLargeMessage :: ResponseTooLarge -> Text responseTooLargeMessage (ResponseTooLarge n) = "response body exceeds " <> T.pack (show n) <> " bytes" --- | Bytes of a non-200 body kept for the 'HttpError' message. +-- | Bytes of a non-200 body kept for the v'LLM.Core.Types.HttpError' message. maxErrorBodyBytes :: Int maxErrorBodyBytes = 64 * 1024 --- | Largest 200 body a non-streaming call accepts. +-- | Largest 200 body a non-streaming call accepts, and the largest total +-- size of a stream. maxResponseBodyBytes :: Int maxResponseBodyBytes = 32 * 1024 * 1024 diff --git a/src/LLM/Core/Internal/Schema.hs b/src/LLM/Core/Internal/Schema.hs new file mode 100644 --- /dev/null +++ b/src/LLM/Core/Internal/Schema.hs @@ -0,0 +1,237 @@ +-- | JSON Schema helpers for structured output. +-- +-- OpenAI's strict mode needs every property listed in @required@; an +-- optional property is emulated by making it nullable +-- ('LLM.Core.ProviderUtils.normalizeSchemaOpenAI'). The model then answers +-- @null@ for a field it leaves out, which the decoder of an +-- @optionalField*@ codec does not accept. 'dropOptionalNulls' removes those +-- @null@s again before decoding, using the schema the codec emitted. +module LLM.Core.Internal.Schema + ( SchemaDefs, + schemaDefs, + isNullableSchema, + optionalProperties, + dropOptionalNulls, + mapSubschemas, + hoistDefs, + ) +where + +import Data.Aeson (Value (..)) +import Data.Aeson.Key qualified as K +import Data.Aeson.KeyMap qualified as KM +import Data.Foldable (toList) +import Data.Maybe (mapMaybe) +import Data.Text (Text) +import Data.Text qualified as T + +-- | Named sub-schemas (@$defs@, @definitions@), and the root for @#@. +data SchemaDefs = SchemaDefs + { sdRoot :: Value, + sdNamed :: KM.KeyMap Value + } + +-- | The definitions anywhere in a schema. autodocodec puts the @$defs@ of a +-- named codec next to its first use, not at the root, so the whole schema +-- is searched. On a name clash the first one found wins. +schemaDefs :: Value -> SchemaDefs +schemaDefs root = SchemaDefs root (collectDefs "$defs" root `KM.union` collectDefs "definitions" root) + +-- | The entries of one definitions keyword (@$defs@ or @definitions@) +-- anywhere in a schema, following sub-schema positions only; the node's +-- own entries first, then those below it in order. +collectDefs :: K.Key -> Value -> KM.KeyMap Value +collectDefs k (Object o) = + foldr KM.union KM.empty $ + [d | Just (Object d) <- [KM.lookup k o]] <> map (collectDefs k) (subschemas o) +collectDefs _ _ = KM.empty + +-- | Move every @$defs@ and @definitions@ entry to the root of the schema. +-- References like @#/$defs/@ are pointers from the root, but +-- autodocodec puts the @$defs@ of a named codec next to its first use, +-- where OpenAI and Ollama do not find them. On a name clash the first one +-- found wins, as in 'schemaDefs'. +hoistDefs :: Value -> Value +hoistDefs root@(Object o) = + Object (put "$defs" (put "definitions" (mapSubschemas strip o))) + where + put k obj + | KM.null defs = KM.delete k obj + | otherwise = KM.insert k (Object defs) obj + where + defs = fmap strip (collectDefs k root) + -- A sub-schema without its definitions, at every depth. + strip (Object x) = Object (mapSubschemas strip (KM.delete "$defs" (KM.delete "definitions" x))) + strip v = v +hoistDefs v = v + +-- | The schema a @$ref@ points to, for @#@, @#/$defs/@ and +-- @#/definitions/@. Other references are not resolved. +resolveRef :: SchemaDefs -> Text -> Maybe Value +resolveRef defs ref + | ref == "#" = Just defs.sdRoot + | Just name <- T.stripPrefix "#/$defs/" ref = named name + | Just name <- T.stripPrefix "#/definitions/" ref = named name + | otherwise = Nothing + where + named name = KM.lookup (K.fromText name) defs.sdNamed + +-- | Whether a schema accepts @null@. A schema we cannot judge (an +-- unresolved @$ref@, a schema with no type constraint) counts as nullable, +-- so it is left alone. +isNullableSchema :: SchemaDefs -> Value -> Bool +isNullableSchema defs = go [] + where + go seen (Object o) + | Just (String ref) <- KM.lookup "$ref" o = + ref `elem` seen || maybe True (go (ref : seen)) (resolveRef defs ref) + | hasType "null" o = True + | Just (Array vs) <- KM.lookup "enum" o = Null `elem` vs + | Just c <- KM.lookup "const" o = c == Null + | any (branches o) ["anyOf", "oneOf"] = True + | Just (Array bs) <- KM.lookup "allOf" o = all (go seen) bs + | any (`KM.member` o) constraining = False + | otherwise = True + where + branches obj k = case KM.lookup k obj of + Just (Array bs) -> any (go seen) bs + _ -> False + go _ (Bool b) = b + go _ _ = True + constraining = ["type", "anyOf", "oneOf", "allOf", "not", "properties", "items"] + +-- | The properties of an object schema that are not in its @required@ list. +optionalProperties :: KM.KeyMap Value -> [K.Key] +optionalProperties o = case KM.lookup "properties" o of + Just (Object props) -> filter (`notElem` requiredKeys o) (KM.keys props) + _ -> [] + +requiredKeys :: KM.KeyMap Value -> [K.Key] +requiredKeys o = case KM.lookup "required" o of + Just (Array ks) -> [K.fromText k | String k <- toList ks] + _ -> [] + +hasType :: Text -> KM.KeyMap Value -> Bool +hasType t o = case KM.lookup "type" o of + Just (String s) -> s == t + Just (Array ts) -> String t `elem` ts + _ -> False + +-- | Remove the @null@ values of properties that the schema has optional and +-- not nullable, in nested objects and arrays too, so that the codec's +-- decoder sees them as absent. Everything else is kept as it is. +-- +-- In a union (@anyOf@, @oneOf@) the walk continues only into the one branch +-- that fits the value (an object with the branch's required keys and no +-- unknown ones, or an array); with no or several such branches the value +-- is left alone. +dropOptionalNulls :: Value -> Value -> Value +dropOptionalNulls schema = walk [] schema + where + defs = schemaDefs schema + + -- @seen@: references followed since the last step into the value, so a + -- reference cycle that does not consume the value stops. + walk :: [Text] -> Value -> Value -> Value + walk seen (Object s) v + | Just (String ref) <- KM.lookup "$ref" s = + case resolveRef defs ref of + Just target | ref `notElem` seen -> walk (ref : seen) target v + _ -> v + | Just bs <- unionBranches s = case filter (fits seen v) bs of + [b] -> walk seen b v + _ -> v + walk _ (Object s) (Object v) = Object (KM.mapMaybeWithKey (field s) v) + walk _ (Object s) (Array vs) + | Just items@(Object _) <- KM.lookup "items" s = Array (fmap (walk [] items) vs) + walk _ _ v = v + + field s k x = case propertySchema s k of + Just ps + | x == Null, + k `notElem` requiredKeys s, + not (isNullableSchema defs ps) -> + Nothing + | otherwise -> Just (walk [] ps x) + Nothing -> case KM.lookup "additionalProperties" s of + Just aps@(Object _) -> Just (walk [] aps x) + _ -> Just x + + propertySchema s k = case KM.lookup "properties" s of + Just (Object props) -> KM.lookup k props + _ -> Nothing + + unionBranches s = + case mapMaybe (`KM.lookup` s) ["anyOf", "oneOf"] of + [Array bs] -> Just (toList bs) + _ -> Nothing + + -- Whether the value has the shape of this branch; only objects and + -- arrays matter, since nothing is dropped from other values. + fits seen v (Object b) + | Just (String ref) <- KM.lookup "$ref" b = + case resolveRef defs ref of + Just target | ref `notElem` seen -> fits (ref : seen) v target + _ -> False + fits _ (Object v) (Object b) = + (hasType "object" b || KM.member "properties" b) + && all (`KM.member` v) (requiredKeys b) + && ( case (KM.lookup "properties" b, KM.lookup "additionalProperties" b) of + (_, Just (Object _)) -> True + (_, Just (Bool True)) -> True + (Just (Object props), _) -> all (`KM.member` props) (KM.keys v) + _ -> KM.null v + ) + fits _ (Array _) (Object b) = hasType "array" b || KM.member "items" b + fits _ _ _ = False + +-- | Apply a function to every direct sub-schema of a schema node: the +-- values of the keywords that hold a sub-schema, a list of them, or a map +-- of them. Everything else (property names, @enum@, @const@, @default@ +-- and other literal values) is kept as it is. +mapSubschemas :: (Value -> Value) -> KM.KeyMap Value -> KM.KeyMap Value +mapSubschemas f = KM.mapWithKey sub + where + sub k v + | k `elem` schemaMapKeywords = mapObject v + | k `elem` schemaKeywords = schemaOrList v + | otherwise = v + mapObject (Object o) = Object (fmap f o) + mapObject v = v + schemaOrList (Array a) = Array (fmap f a) + schemaOrList v = f v + +-- | The direct sub-schemas of a schema node, in the positions +-- 'mapSubschemas' visits. +subschemas :: KM.KeyMap Value -> [Value] +subschemas o = concat [sub k v | (k, v) <- KM.toList o] + where + sub k v + | k `elem` schemaMapKeywords, Object m <- v = KM.elems m + | k `elem` schemaKeywords, Array a <- v = toList a + | k `elem` schemaKeywords = [v] + | otherwise = [] + +-- | Keywords whose value maps names to sub-schemas. +schemaMapKeywords :: [KM.Key] +schemaMapKeywords = ["properties", "patternProperties", "$defs", "definitions", "dependentSchemas"] + +-- | Keywords whose value is a sub-schema or a list of sub-schemas. +schemaKeywords :: [KM.Key] +schemaKeywords = + [ "items", + "prefixItems", + "additionalItems", + "additionalProperties", + "unevaluatedProperties", + "unevaluatedItems", + "propertyNames", + "contains", + "anyOf", + "oneOf", + "allOf", + "not", + "if", + "then", + "else" + ] diff --git a/src/LLM/Core/Internal/Transport.hs b/src/LLM/Core/Internal/Transport.hs new file mode 100644 --- /dev/null +++ b/src/LLM/Core/Internal/Transport.hs @@ -0,0 +1,36 @@ +-- | The HTTP calls every built-in provider makes: a JSON @POST@ with +-- 'lenientConfig', read with 'readResponseBody' or handed to a stream +-- parser through 'handleStreamResponse'. Each provider keeps its own URL, +-- auth options and body preparation. +module LLM.Core.Internal.Transport + ( postJson, + postJsonStream, + ) +where + +import Data.Aeson (Value) +import Data.ByteString (ByteString) +import LLM.Core.ProviderUtils (handleStreamResponse, lenientConfig, readResponseBody) +import LLM.Core.Types (LLMTextResult) +import Network.HTTP.Req (Option, POST (POST), ReqBodyJson (ReqBodyJson), Url, reqBr, runReq) + +-- | Send a non-streaming request; the result is +-- 'LLM.Core.LLMProvider.sendRequest''s @(status, body)@. +postJson :: Url scheme -> Option scheme -> Value -> IO (Int, ByteString) +postJson url opts body = + runReq lenientConfig $ + reqBr POST url (ReqBodyJson body) opts readResponseBody + +-- | Send a streaming request, for 'LLM.Core.LLMProvider.openStream'. On a +-- 200 response the continuation gets the body reader; any other status +-- becomes an 'LLM.Core.Types.HttpError' without calling it. +postJsonStream :: + Url scheme -> + Option scheme -> + Value -> + (IO ByteString -> IO LLMTextResult) -> + IO LLMTextResult +postJsonStream url opts body k = + runReq lenientConfig $ + reqBr POST url (ReqBodyJson body) opts $ \resp -> + handleStreamResponse resp k diff --git a/src/LLM/Core/LLMProvider.hs b/src/LLM/Core/LLMProvider.hs --- a/src/LLM/Core/LLMProvider.hs +++ b/src/LLM/Core/LLMProvider.hs @@ -32,15 +32,15 @@ data LLMProvider = LLMProvider { -- | Provider name for logging/hooks (e.g. "claude", "gemini", "openai") providerName :: Text, - -- | MIME types of inline images the provider accepts (see 'gwImageMediaTypes'). + -- | MIME types of inline images the provider accepts (see 'LLM.Core.Types.gwImageMediaTypes'). providerImageMediaTypes :: [Text], - -- | Whether the provider accepts URL images (see 'gwImageUrlSupport'). + -- | Whether the provider accepts URL images (see 'LLM.Core.Types.gwImageUrlSupport'). providerImageUrlSupport :: Bool, -- | Build the JSON request body. Bool indicates whether streaming is -- requested. A top-level @_model@ key is routing data for - -- 'sendRequest' and 'openStream' (the Gemini provider puts the model + -- 'LLM.Core.LLMProvider.sendRequest' and 'LLM.Core.LLMProvider.openStream' (the Gemini provider puts the model -- name there, since its URL contains it); they must remove it before - -- sending, and 'onLLMRequest' sees the body without it + -- sending, and 'LLM.Core.Types.onLLMRequest' sees the body without it -- ('withoutRoutingFields'). buildBody :: Bool -> ChatRequest -> Value, -- | Make a non-streaming HTTP call, returning (status code, raw response @@ -108,7 +108,7 @@ Right resp -> handleResponse p hooks p.parseObjectResponse resp -- | The request body as it is sent: without the top-level @_model@ routing --- key (see 'buildBody'). Other values are returned unchanged. +-- key (see 'LLM.Core.LLMProvider.buildBody'). Other values are returned unchanged. withoutRoutingFields :: Value -> Value withoutRoutingFields (Object o) = Object (KM.delete "_model" o) withoutRoutingFields v = v @@ -140,7 +140,7 @@ -- | Run a provider's HTTP call. An 'HttpException' becomes a 'NetworkError' -- and a response body over the size cap a 'ParseError'; both are reported --- to 'onLLMResponseError'. Other exceptions propagate. +-- to 'LLM.Core.Types.onLLMResponseError'. Other exceptions propagate. providerCall :: LLMProvider -> LLMHooks -> IO a -> IO (Either LLMError a) providerCall p hooks act = do r <- @@ -160,7 +160,7 @@ -- | Turn a non-streaming (status, raw body) into a result. A non-200 status -- is an 'HttpError' carrying the body as text, whether or not it is JSON -- (an HTML 502 from a proxy keeps its status). A 200 body that is not JSON --- is a 'ParseError'. 'onLLMResponse' sees every body that decodes as JSON. +-- is a 'ParseError'. 'LLM.Core.Types.onLLMResponse' sees every body that decodes as JSON. handleResponse :: LLMProvider -> LLMHooks -> (Value -> IO (LLMResult a)) -> (Int, ByteString) -> IO (LLMResult a) handleResponse p hooks parse (status, raw) = do let decoded = eitherDecodeStrict' raw diff --git a/src/LLM/Core/ProviderUtils.hs b/src/LLM/Core/ProviderUtils.hs --- a/src/LLM/Core/ProviderUtils.hs +++ b/src/LLM/Core/ProviderUtils.hs @@ -16,7 +16,7 @@ import Control.Exception (throwIO) import Control.Retry (limitRetries) -import Data.Aeson (ToJSON (toJSON), Value (..), decodeStrict', eitherDecodeStrict', encode, object) +import Data.Aeson (ToJSON (toJSON), Value (..), decodeStrict', eitherDecodeStrict', encode, object, (.=)) import Data.Aeson.KeyMap qualified as KM import Data.ByteString qualified as BS import Data.Text (Text) @@ -31,6 +31,7 @@ readBodyPrefix, readErrorBody, ) +import LLM.Core.Internal.Schema (hoistDefs, isNullableSchema, mapSubschemas, optionalProperties, schemaDefs) import LLM.Core.Types (LLMError (EmptyResponse, HttpError, ParseError), LLMResult, LLMTextResult) import Network.HTTP.Client qualified as HC import Network.HTTP.Client.TLS (tlsManagerSettings) @@ -59,7 +60,7 @@ -- | Manager shared by all provider requests. A non-streaming call receives -- its response headers only after generation finishes, so http-client's -- default 30 s response timeout would cut off long calls. The only response --- deadline is the one the application configures ('mcRequestTimeout'). +-- deadline is the one the application configures ('LLM.Generate.ModelConfig.mcRequestTimeout'). sharedManager :: HC.Manager sharedManager = unsafePerformIO $ @@ -126,23 +127,53 @@ pure $ Left $ HttpError status (decodeUtf8With lenientDecode (BS.concat chunks)) else handler (HC.responseBody resp) --- | Recursively enforce OpenAI structured output constraints: --- every object node gets additionalProperties:false and all keys in required. +-- | Recursively enforce OpenAI structured output constraints: every object +-- node gets @additionalProperties: false@ and all its properties in +-- @required@. Strict mode allows no optional properties, so a property +-- that was optional and does not accept @null@ becomes nullable +-- (@anyOf: [, {"type": "null"}]@); the model answers @null@ to +-- leave it out, and 'LLM.Generate.genObject' removes that @null@ before +-- decoding. +-- +-- Strict mode refuses @oneOf@, so it becomes @anyOf@ (unless the node has +-- an @anyOf@ already). autodocodec emits @oneOf@ for disjoint unions such +-- as 'Either', whose branches cannot both match, so the meaning is the +-- same. +-- +-- Every @$defs@ entry moves to the root: autodocodec puts the @$defs@ of a +-- named codec next to its first use, but @#/$defs/@ points from the +-- root, and strict mode refuses a reference it cannot find there. +-- +-- Only schema keywords are changed: the walk follows sub-schema positions, +-- so a property called @oneOf@ or a literal value is kept. normalizeSchemaOpenAI :: Value -> Value -normalizeSchemaOpenAI (Object o) = - let o' = fmap normalizeSchemaOpenAI o - fixed = case KM.lookup "type" o' of - Just (String "object") -> - let propKeys = case KM.lookup "properties" o' of - Just (Object p) -> KM.keys p - _ -> [] - withAP = KM.insert "additionalProperties" (Bool False) o' - withReq = KM.insert "required" (toJSON propKeys) withAP - in withReq - _ -> o' - in Object fixed -normalizeSchemaOpenAI (Array a) = Array (fmap normalizeSchemaOpenAI a) -normalizeSchemaOpenAI v = v +normalizeSchemaOpenAI schema = hoistDefs (go schema) + where + defs = schemaDefs schema + go (Object o) = + let o' = oneOfToAnyOf (mapSubschemas go o) + fixed = case KM.lookup "type" o' of + Just (String "object") -> + let propKeys = case KM.lookup "properties" o' of + Just (Object p) -> KM.keys p + _ -> [] + optional = optionalProperties o + withNull k p + | k `elem` optional && not (isNullableSchema defs p) = + object ["anyOf" .= [p, object ["type" .= ("null" :: Text)]]] + | otherwise = p + withNullable = case KM.lookup "properties" o' of + Just (Object p) -> KM.insert "properties" (Object (KM.mapWithKey withNull p)) o' + _ -> o' + withAP = KM.insert "additionalProperties" (Bool False) withNullable + withReq = KM.insert "required" (toJSON propKeys) withAP + in withReq + _ -> o' + in Object fixed + go v = v + oneOfToAnyOf o = case KM.lookup "oneOf" o of + Just bs | not (KM.member "anyOf" o) -> KM.insert "anyOf" bs (KM.delete "oneOf" o) + _ -> o stripJsonFences :: Text -> Text stripJsonFences t = @@ -195,42 +226,6 @@ stripBoundsAndComments :: Value -> Value stripBoundsAndComments (Object obj) = Object $ - KM.fromList - [ (k, stripKeyword k v) - | (k, v) <- KM.toList obj, - k `notElem` ["minimum", "maximum", "$comment"] - ] - where - stripKeyword k v - | k `elem` schemaMapKeywords = mapObject stripBoundsAndComments v - | k `elem` schemaKeywords = stripSchemaOrList v - | otherwise = v - mapObject f (Object o) = Object (fmap f o) - mapObject _ v = v - stripSchemaOrList (Array a) = Array (fmap stripBoundsAndComments a) - stripSchemaOrList v = stripBoundsAndComments v + mapSubschemas stripBoundsAndComments $ + KM.filterWithKey (\k _ -> k `notElem` ["minimum", "maximum", "$comment"]) obj stripBoundsAndComments other = other - --- | Keywords whose value maps names to sub-schemas. -schemaMapKeywords :: [KM.Key] -schemaMapKeywords = ["properties", "patternProperties", "$defs", "definitions", "dependentSchemas"] - --- | Keywords whose value is a sub-schema or a list of sub-schemas. -schemaKeywords :: [KM.Key] -schemaKeywords = - [ "items", - "prefixItems", - "additionalItems", - "additionalProperties", - "unevaluatedProperties", - "unevaluatedItems", - "propertyNames", - "contains", - "anyOf", - "oneOf", - "allOf", - "not", - "if", - "then", - "else" - ] diff --git a/src/LLM/Core/SSE.hs b/src/LLM/Core/SSE.hs --- a/src/LLM/Core/SSE.hs +++ b/src/LLM/Core/SSE.hs @@ -16,7 +16,7 @@ import Data.Text (Text) import Data.Text qualified as T import Data.Text.Encoding qualified as TE -import LLM.Core.Internal.Body (ResponseTooLarge (..), maxStreamEventBytes) +import LLM.Core.Internal.Body (ResponseTooLarge (..), maxResponseBodyBytes, maxStreamEventBytes) import LLM.Core.Types (LLMError (..)) data SSEEvent = SSEEvent @@ -32,13 +32,24 @@ -- continues so the caller can still tell whether the stream ended properly. -- The result describes the first such event, if any. -- --- An event (or a line) larger than 'maxStreamEventBytes' throws --- 'ResponseTooLarge', which the @generic*@ functions turn into a --- 'ParseError'. +-- An event (or a line) larger than 'maxStreamEventBytes', or a stream +-- longer than 'maxResponseBodyBytes' in total, throws 'ResponseTooLarge', +-- which the @generic*@ functions turn into a 'ParseError'. readSSEEvents :: IO ByteString -> (SSEEvent -> IO ()) -> IO (Maybe Text) -readSSEEvents readChunk callback = do +readSSEEvents readRaw callback = do bufRef <- newIORef BS.empty badRef <- newIORef Nothing + totalRef <- newIORef (0 :: Int) + let -- Count every byte read, so a stream of small events cannot go on + -- forever. + readChunk :: IO ByteString + readChunk = do + chunk <- readRaw + total <- (+ BS.length chunk) <$> readIORef totalRef + when (total > maxResponseBodyBytes) $ + throwIO (ResponseTooLarge maxResponseBodyBytes) + writeIORef totalRef total + pure chunk let readLine :: IO (Maybe ByteString) readLine = do buf <- readIORef bufRef diff --git a/src/LLM/Core/Types.hs b/src/LLM/Core/Types.hs --- a/src/LLM/Core/Types.hs +++ b/src/LLM/Core/Types.hs @@ -28,6 +28,7 @@ coalesceAdjacentTextParts, validateTurn, validateTurns, + validateConversation, opaqueForProvider, stripForeignOpaque, @@ -71,6 +72,7 @@ ) import Data.Aeson.Types (Parser) import Data.Char (isSpace, isAsciiUpper, isAsciiLower, isDigit) +import Data.List (nub, sortOn, zip4) import Data.Maybe (mapMaybe) import Data.Text (Text) import Data.Text qualified as T @@ -80,7 +82,7 @@ -- | Provider adapter for one LLM backend. -- -- Implement the three callbacks to add a custom provider, or use the --- bundled constructors in 'LLM.Providers'. +-- bundled constructors in "LLM.Providers". data LLMGateway = LLMGateway { -- | Short name used in logs and hooks (e.g. @openai@). gwName :: Text, @@ -108,7 +110,7 @@ -- | Hooks for observing raw provider request and response JSON on the wire. -- -- Distinct from 'LLM.Generate.Logger.Hooks', which covers application-level --- logging and tool tracing. Bridge to this type via 'llmHooks'. +-- logging and tool tracing. Bridge to this type via 'LLM.Generate.GenerateUtils.llmHooks'. data LLMHooks = LLMHooks { onLLMRequest :: Text -> Value -> IO (), onLLMResponse :: Text -> Value -> IO (), @@ -128,7 +130,7 @@ meoIncludeReasoning :: Bool, -- | Send @reasoning_content@ on every assistant message, @""@ when the -- turn has no thinking text (DeepSeek thinking mode rejects assistant - -- messages without it). Only takes effect with 'meoIncludeReasoning'. + -- messages without it). Only takes effect with 'LLM.Core.Types.meoIncludeReasoning'. meoRequireReasoning :: Bool } deriving (Show, Eq) @@ -137,8 +139,8 @@ defaultMessageEncodeOptions = MessageEncodeOptions {meoIncludeReasoning = False, meoRequireReasoning = False} --- | DeepSeek encoding with thinking off. 'deepSeekBuildBodyPairs' sets --- 'meoRequireReasoning' when the request enables thinking. +-- | DeepSeek encoding with thinking off. @deepSeekBuildBodyPairs@ (in "LLM.Providers.DeepSeek") sets +-- 'LLM.Core.Types.meoRequireReasoning' when the request enables thinking. deepSeekMessageEncodeOptions :: MessageEncodeOptions deepSeekMessageEncodeOptions = MessageEncodeOptions {meoIncludeReasoning = True, meoRequireReasoning = False} @@ -146,7 +148,7 @@ -- | Opaque, provider-owned payload that must round-trip for replay. -- -- Used for Claude thinking signatures, Gemini thought signatures, and any --- similar provider-bound state. Consumers must treat 'poPayload' as opaque. +-- similar provider-bound state. Consumers must treat 'LLM.Core.Types.poPayload' as opaque. data ProviderOpaque = ProviderOpaque { poProvider :: Text, poModel :: Maybe Text, @@ -214,7 +216,7 @@ ] -- | MIME types accepted by 'mkImageBase64' / 'imageBase64Part'. Not every --- provider accepts all of them; see 'gwImageMediaTypes'. +-- provider accepts all of them; see 'LLM.Core.Types.gwImageMediaTypes'. supportedImageMediaTypes :: [Text] supportedImageMediaTypes = commonImageMediaTypes ++ ["image/heic", "image/heif"] @@ -393,7 +395,7 @@ -- | Migration helper: emit thinking (if any), then text, then tool calls. -- -- Does not recover arbitrary provider block order; use 'AssistantMessage' --- with 'respContent' when replaying authoritative ordered parts. +-- with 'LLM.Core.Types.respContent' when replaying authoritative ordered parts. assistantTurn :: Text -> Maybe Text -> [ToolCall] -> Turn assistantTurn text mReasoning calls = AssistantMessage $ @@ -499,6 +501,51 @@ validateTurns :: [Turn] -> [(Int, Text)] validateTurns turns = [(i, reason) | (i, turn) <- zip [0 ..] turns, Left reason <- [validateTurn turn]] +-- | Problems in the tool-call structure of a conversation, as (index from +-- 0, reason), in index order. Empty when the structure is valid: +-- +-- * a 'ToolTurn' directly follows an assistant turn with tool calls; +-- * the call IDs of an assistant turn are distinct, and that turn is +-- directly followed by a 'ToolTurn'; +-- * the 'ToolTurn' answers each of those calls exactly once, and answers +-- no other call ID. +-- +-- Providers reject such conversations or drop parts of them, each in its +-- own way. Every generation entry point logs these problems as warnings, +-- like 'validateTurns', and sends the request unchanged. From 0.5 they +-- will be errors. +-- +-- @since 0.4.3.0 +validateConversation :: [Turn] -> [(Int, Text)] +validateConversation turns = + sortOn + fst + [ problem + | (i, prev, turn, next) <- zip4 [0 ..] (Nothing : map Just turns) turns (map Just (drop 1 turns) ++ [Nothing]), + problem <- check i prev turn next + ] + where + check :: Int -> Maybe Turn -> Turn -> Maybe Turn -> [(Int, Text)] + check i prev turn next = case turn of + ToolTurn results -> case prev of + Just (AssistantMessage parts) + | calls@(_ : _) <- turnToolCalls parts -> + answers i (map (.tcId) calls) (map (.trCallId) results) + _ -> [(i, "tool results do not follow an assistant turn with tool calls")] + AssistantMessage parts + | ids@(_ : _) <- map (.tcId) (turnToolCalls parts) -> + [(i, "duplicate tool call id " <> quote d) | d <- duplicates ids] + ++ case next of + Just (ToolTurn _) -> [] + _ -> [(i, "tool calls are not followed by tool results")] + _ -> [] + answers i callIds resultIds = + [(i, "no result for tool call " <> quote c) | c <- nub callIds, c `notElem` resultIds] + ++ [(i, "tool call " <> quote r <> " is answered more than once") | r <- duplicates resultIds, r `elem` callIds] + ++ [(i, "tool result for unknown call id " <> quote r) | r <- nub resultIds, r `notElem` callIds] + duplicates xs = nub [x | x <- xs, length (filter (== x) xs) > 1] + quote = T.pack . show + -- | Keep opaque metadata only when it belongs to @provider@. opaqueForProvider :: Text -> Maybe ProviderOpaque -> Maybe ProviderOpaque opaqueForProvider provider (Just o) @@ -534,15 +581,15 @@ toolDescription :: Text, toolParameters :: Value, -- JSON Schema object -- | When 'True', the tool is advertised as read-only and remains available - -- when 'RuntimeArgs.rtReadonly' is set. Mutating tools should use 'False'. + -- when 'LLM.Agent.Types.rtReadonly' is set. Mutating tools should use 'False'. toolReadonly :: Bool } deriving (Show, Eq) --- | Typed tool definition before conversion to 'Tool' via 'toTool'. +-- | Typed tool definition before conversion to t'LLM.Agent.Types.Tool' via 'LLM.Agent.ToolUtils.toTool'. -- --- Define one with an Autodocodec argument type; 'toTool' derives the JSON --- Schema for 'ToolDef.toolParameters' automatically. +-- Define one with an Autodocodec argument type; 'LLM.Agent.ToolUtils.toTool' derives the JSON +-- Schema for 'LLM.Core.Types.toolParameters' automatically. data TypedTool c a = TypedTool { ttoolName :: Text, ttoolDescription :: Text, @@ -552,7 +599,7 @@ -- | A tool invocation returned by the model. -- --- 'tcProviderMeta' is opaque, provider-owned state that must round-trip back +-- 'LLM.Core.Types.tcProviderMeta' is opaque, provider-owned state that must round-trip back -- to the same provider (and usually the same model) when this tool call is -- replayed. Gemini thought signatures are the current example. data ToolCall = ToolCall @@ -622,8 +669,8 @@ -- | A response from an LLM provider. -- --- 'respContent' is authoritative ordered content. 'respText' and --- 'respReasoning' are convenience projections and may be lossy. +-- 'LLM.Core.Types.respContent' is authoritative ordered content. 'LLM.Core.Types.respText' and +-- 'LLM.Core.Types.respReasoning' are convenience projections and may be lossy. data ChatResponse = ChatResponse { respText :: Text, respContent :: [ContentPart], diff --git a/src/LLM/Core/Usage.hs b/src/LLM/Core/Usage.hs --- a/src/LLM/Core/Usage.hs +++ b/src/LLM/Core/Usage.hs @@ -17,7 +17,7 @@ -- | Token usage from a single API call. -- --- 'usageInputTokens' is the total input token count, including tokens that were +-- 'LLM.Core.Usage.usageInputTokens' is the total input token count, including tokens that were -- cache reads or cache writes. Provider parsers normalize differing wire -- semantics so this total is never double-counted. -- @@ -100,7 +100,7 @@ -- | Pricing in dollars per million tokens. -- --- Optional cache rates fall back to 'pricePerMillionInput' when absent. +-- Optional cache rates fall back to 'LLM.Core.Usage.pricePerMillionInput' when absent. data PricingInfo = PricingInfo { pricePerMillionInput :: Double, pricePerMillionOutput :: Double, diff --git a/src/LLM/Core/Utils.hs b/src/LLM/Core/Utils.hs --- a/src/LLM/Core/Utils.hs +++ b/src/LLM/Core/Utils.hs @@ -187,7 +187,7 @@ (const action) -- | Build a synthetic JSON summary from a streamed ChatResponse, --- used by providers to fire 'onResponse' after streaming completes. +-- used by providers to fire 'LLM.Generate.Logger.onResponse' after streaming completes. streamResponseJson :: ChatResponse -> Value streamResponseJson r = object diff --git a/src/LLM/Generate/Generate.hs b/src/LLM/Generate/Generate.hs --- a/src/LLM/Generate/Generate.hs +++ b/src/LLM/Generate/Generate.hs @@ -9,193 +9,12 @@ ) where -import Control.Exception (Exception, handle, throwIO) -import Control.Monad (unless, when) -import Data.IORef (IORef, newIORef, readIORef, writeIORef) -import Data.Maybe (fromMaybe) -import LLM.Core.Abort (isAbortedMaybe) -import LLM.Core.Types - ( ChatResponse (..), - LLMError (Aborted), - LLMGateway (..), - LLMTextResult, - StreamEvent (..), - Turn (..), - ) -import LLM.Core.Usage (emptyUsage) -import LLM.Core.Utils (withFirstEventTimeout) -import LLM.Generate.GenerateUtils - ( callWithRetry, - callWithRetryTimeout, - mkRequest, +import LLM.Generate.Internal.Generate + ( generateTextLLM, + generateTextWithFallbacks, + generateTextWithFallbacksRotated, + streamTextLLM, + streamTextWithFallbacks, + streamTextWithFallbacksRotated, usageWithModelCost, - safeGenRequest, - withModelFallbacksRotated, ) -import LLM.Generate.ModelConfig - ( ModelConfig (..), - ModelWithFallbacks (..), - ) -import LLM.Generate.Types - ( GenRequest (..), - GenerateResult, - RoundTextRole (..), - StreamChunk (..), - ) - --- | Generate text with retries and model fallbacks. --- --- Tries 'mwfModel' first, then each fallback in order. Returns a provider --- 'ChatResponse' on success; does not execute tools. -generateTextWithFallbacks :: - GenRequest -> - ModelWithFallbacks -> - IO (GenerateResult ChatResponse) -generateTextWithFallbacks gr models = - fmap snd <$> generateTextWithFallbacksRotated (safeGenRequest gr) models - --- | Like 'generateTextWithFallbacks', but also returns the chain rotated so --- the model that answered comes first (see 'withModelFallbacksRotated'). -generateTextWithFallbacksRotated :: - GenRequest -> - ModelWithFallbacks -> - IO (GenerateResult (ModelWithFallbacks, ChatResponse)) -generateTextWithFallbacksRotated gr0 models = - withModelFallbacksRotated gr models $ \mc -> do - r <- generateTextLLM gr mc - case r of - Left err -> pure $ Left err - Right resp -> - let usage = usageWithModelCost mc (fromMaybe emptyUsage resp.respUsage) - in pure $ Right resp {respUsage = Just usage} - where - gr = safeGenRequest gr0 - --- | Like 'generateTextWithFallbacks', but streams 'StreamChunk' values --- through @onChunk@ as the provider responds. -streamTextWithFallbacks :: - (StreamChunk -> IO ()) -> - GenRequest -> - ModelWithFallbacks -> - IO (GenerateResult ChatResponse) -streamTextWithFallbacks onChunk gr models = - fmap snd <$> streamTextWithFallbacksRotated onChunk (safeGenRequest gr) models - --- | Like 'streamTextWithFallbacks', but also returns the rotated chain --- (see 'withModelFallbacksRotated'). -streamTextWithFallbacksRotated :: - (StreamChunk -> IO ()) -> - GenRequest -> - ModelWithFallbacks -> - IO (GenerateResult (ModelWithFallbacks, ChatResponse)) -streamTextWithFallbacksRotated onChunk gr0 models = do - let gr = safeGenRequest gr0 - emittedRef <- newIORef False - withModelFallbacksRotated gr models $ \mc -> do - r <- streamTextAttempts emittedRef onChunk gr mc - case r of - Left err -> pure $ Left err - Right resp -> - let usage = usageWithModelCost mc (fromMaybe emptyUsage resp.respUsage) - in pure $ Right resp {respUsage = Just usage} - -generateTextLLM :: GenRequest -> ModelConfig -> IO LLMTextResult -generateTextLLM gr0 mc = - callWithRetryTimeout gr mc $ - let request = mkRequest gr mc - in mc.mcGateway.gwGenerateText gr.grLLMHooks request - where - gr = safeGenRequest gr0 - --- | Stream one model call with retries. 'mcRequestTimeout' bounds the time --- until the first stream event of each attempt, not the whole stream. --- --- If an attempt fails after streaming chunks, the next attempt first emits --- 'StreamAttemptReset'. -streamTextLLM :: (StreamChunk -> IO ()) -> GenRequest -> ModelConfig -> IO LLMTextResult -streamTextLLM onChunk gr mc = do - emittedRef <- newIORef False - streamTextAttempts emittedRef onChunk (safeGenRequest gr) mc - --- | 'streamTextLLM' with the \"chunks emitted\" flag supplied by the caller, --- so it is shared across retries and model fallbacks of one model call. -streamTextAttempts :: - IORef Bool -> - (StreamChunk -> IO ()) -> - GenRequest -> - ModelConfig -> - IO LLMTextResult -streamTextAttempts emittedRef onChunk gr mc = - callWithRetry gr mc $ - let request = mkRequest gr mc - onChunk' c = writeIORef emittedRef True >> onChunk c - in do - emitted <- readIORef emittedRef - when emitted $ do - onChunk StreamAttemptReset - writeIORef emittedRef False - ProviderStreamCallback {pscOnEvent, pscFinalize} <- - mkProviderStreamCallback gr onChunk' - -- Deliver no event after the abort signal fired. ('callWithRetry' - -- also interrupts the attempt, but asynchronously, so an event - -- could otherwise slip through first.) - let onEvent e = do - aborted <- isAbortedMaybe gr.grAbortSignal - if aborted then throwIO StreamAborted else pscOnEvent e - result <- - handle (\StreamAborted -> pure (Left Aborted)) $ - withFirstEventTimeout - mc.mcRequestTimeout - (mc.mcGateway.gwStreamText gr.grLLMHooks request) - onEvent - case result of - Right _ -> pscFinalize >> pure result - Left err -> pure (Left err) - --- | Thrown from the stream callback once the abort signal has fired. -data StreamAborted = StreamAborted - deriving (Show) - -instance Exception StreamAborted - -data RoundPhase = Unclassified | Preamble | Answer - deriving (Eq) - -data ProviderStreamCallback = ProviderStreamCallback - { pscOnEvent :: StreamEvent -> IO (), - pscFinalize :: IO () - } - -initialPhase :: [Turn] -> RoundPhase -initialPhase turns = - case reverse turns of - ToolTurn _ : _ -> Answer - _ -> Unclassified - -mkProviderStreamCallback :: - GenRequest -> - (StreamChunk -> IO ()) -> - IO ProviderStreamCallback -mkProviderStreamCallback gr onChunk = do - phaseRef <- newIORef (initialPhase gr.grMessages) - pure - ProviderStreamCallback - { pscOnEvent = \case - StreamReasoningDelta txt -> onChunk (ReasoningDelta txt) - StreamDelta txt -> do - phase <- readIORef phaseRef - case phase of - Answer -> onChunk (AnswerDelta txt) - Preamble -> onChunk (PreambleDelta txt) - Unclassified -> onChunk (TextDelta txt) - StreamToolCall tc -> do - phase <- readIORef phaseRef - unless (phase == Preamble) $ - onChunk (RoundTextRoleCommitted PreambleRole) - writeIORef phaseRef Preamble - onChunk (StreamToolCallChunk tc), - pscFinalize = do - phase <- readIORef phaseRef - when (phase == Unclassified) $ - onChunk (RoundTextRoleCommitted AnswerRole) - } diff --git a/src/LLM/Generate/GenerateObject.hs b/src/LLM/Generate/GenerateObject.hs --- a/src/LLM/Generate/GenerateObject.hs +++ b/src/LLM/Generate/GenerateObject.hs @@ -11,6 +11,7 @@ import Data.Maybe (fromMaybe) import Data.Text qualified as T import LLM.Core.Abort (isAbortedMaybe) +import LLM.Core.Internal.Schema (dropOptionalNulls) import LLM.Core.ProviderUtils (stripBoundsAndComments) import LLM.Core.Types (LLMGateway (..), LLMObjectResult) import LLM.Core.Usage (Usage (..), emptyUsage) @@ -52,7 +53,10 @@ case res of Left e -> pure (Left e) Right (v, u) -> do - case AE.fromJSON v of + -- A provider in strict mode answers null for an optional field + -- (see normalizeSchemaOpenAI); the codec reads it as absent only + -- without the null. + case AE.fromJSON (dropOptionalNulls jsonschema v) of AE.Error e -> pure $ Left $ diff --git a/src/LLM/Generate/GenerateUtils.hs b/src/LLM/Generate/GenerateUtils.hs --- a/src/LLM/Generate/GenerateUtils.hs +++ b/src/LLM/Generate/GenerateUtils.hs @@ -12,290 +12,15 @@ ) where -import Control.Concurrent (threadDelay) -import Control.Monad (when) -import Data.Text (Text) -import Data.Text qualified as T -import Data.List (nub) -import Data.Maybe (fromMaybe) -import LLM.Core.Abort (isAbortedMaybe) -import LLM.Core.Types - ( ChatRequest (..), - ContentPart (..), - ImageSource (..), - LLMError (..), - LLMGateway (gwImageMediaTypes, gwImageUrlSupport, gwName), - PartBody (..), - LLMHooks (..), - ThinkingMode (..), - Turn (..), - conversationHasImages, - validateTurns, - ) -import LLM.Core.Usage (Usage (..), estimateCost) -import LLM.Core.Utils (interruptOnAbort, withRetry, withTimeout) -import LLM.Generate.Logger (Hooks (..), LogLevel (..), safeHooks, safeLLMHooks) -import LLM.Generate.ModelConfig - ( ModelCapabilities (..), - ModelConfig (..), - ModelWithFallbacks (..), - mfwToModelConfigs, - modelRetryPolicy, +import LLM.Generate.Internal.GenerateUtils + ( callWithRetry, + callWithRetryTimeout, + llmHooks, + maybeThrottle, + mkRequest, + safeGenRequest, + usageWithModelCost, + validateModelCapabilities, + withModelFallbacks, + withModelFallbacksRotated, ) -import LLM.Generate.Types (GenRequest (..), GenerateError (..), GenerateResult) - -maybeThrottle :: Maybe Int -> IO a -> IO a -maybeThrottle Nothing io = io -maybeThrottle (Just ms) io = threadDelay (ms * 1000) >> io - -mkRequest :: GenRequest -> ModelConfig -> ChatRequest -mkRequest gr mc = - ChatRequest - { reqModel = mc.mcModel, - reqConversation = gr.grMessages, - reqSystem = gr.grSystemPrompt, - reqMaxTokens = mc.mcMaxTokens, - reqTemperature = mc.mcTemperature, - reqTools = gr.grTools, - reqThinking = mc.mcThinking - } - -usageWithModelCost :: ModelConfig -> Usage -> Usage -usageWithModelCost mc u = u {usageTotalCost = estimateCost mc.mcPricing u} - --- | Check that a candidate model declares every capability the request needs. --- --- Unsupported vision is a candidate failure (do not drop images), and so is --- an inline image whose media type the gateway does not accept --- ('gwImageMediaTypes') or a URL image on a gateway without --- 'gwImageUrlSupport'. URL images have no known media type, so their type --- is not checked. --- Thinking configuration requires a declared thinking capability. Cache --- hints are not validated here; unsupported providers ignore them. -validateModelCapabilities :: ModelConfig -> [Turn] -> Either LLMError () -validateModelCapabilities mc turns = do - whenNeedsVision - whenNeedsMediaTypes - whenNeedsImageUrls - whenNeedsThinking - where - caps = mc.mcCapabilities - whenNeedsVision - | conversationHasImages turns, - not caps.capVision = - Left $ - UnsupportedCapability $ - "model " - <> mc.mcModel - <> " does not support vision" - | otherwise = Right () - whenNeedsMediaTypes = - case filter (`notElem` mc.mcGateway.gwImageMediaTypes) (inlineImageMediaTypes turns) of - [] -> Right () - unsupported -> - Left $ - UnsupportedCapability $ - "model " - <> mc.mcModel - <> " via " - <> mc.mcGateway.gwName - <> " does not accept image media type " - <> T.intercalate ", " unsupported - whenNeedsImageUrls - | conversationHasImageUrls turns, - not mc.mcGateway.gwImageUrlSupport = - Left $ - UnsupportedCapability $ - "model " - <> mc.mcModel - <> " via " - <> mc.mcGateway.gwName - <> " does not accept image URLs" - | otherwise = Right () - whenNeedsThinking = - case mc.mcThinking of - Just ThinkingMode {tmEnabled = True} - | not caps.capThinking -> - Left $ - UnsupportedCapability $ - "model " - <> mc.mcModel - <> " does not support thinking" - _ -> Right () - --- | Distinct media types of the inline ('ImageBase64') images in a conversation. -inlineImageMediaTypes :: [Turn] -> [Text] -inlineImageMediaTypes turns = - nub [mt | ImageBase64 mt _ <- imageSources turns] - -conversationHasImageUrls :: [Turn] -> Bool -conversationHasImageUrls turns = - not (null [() | ImageUrl _ <- imageSources turns]) - -imageSources :: [Turn] -> [ImageSource] -imageSources turns = - [src | parts <- map turnParts turns, ContentPart (ImagePart src) _ <- parts] - where - turnParts (UserMessage parts) = parts - turnParts (AssistantMessage parts) = parts - turnParts (ToolTurn _) = [] - --- | Run one model call with the model's throttle delay and retry policy. --- @attempt@ is run once per try; any per-attempt timeout must be inside it. --- --- 'mcThrottleDelay' is slept before every try, retries included. Before a --- retry it comes after the retry backoff, so the two delays add up. --- --- When 'grAbortSignal' fires, the call is interrupted wherever it is (in an --- attempt, or in the throttle or backoff sleep) and returns 'Aborted'. An --- interrupted attempt is cancelled, which closes its HTTP connection. -callWithRetry :: - GenRequest -> - ModelConfig -> - IO (Either LLMError a) -> - IO (Either LLMError a) -callWithRetry gr mc attempt = - fmap (fromMaybe (Left Aborted)) $ - interruptOnAbort gr.grAbortSignal $ - withRetry - (modelRetryPolicy mc) - (gr.grHooks.onLog Warn) - (maybeThrottle mc.mcThrottleDelay (unlessAborted gr attempt)) - --- | Run @act@ unless 'grAbortSignal' has fired, else return 'Aborted' --- (which is not retried). Checked before every attempt, so also after --- throttle and backoff sleeps. -unlessAborted :: GenRequest -> IO (Either LLMError a) -> IO (Either LLMError a) -unlessAborted gr act = do - aborted <- isAbortedMaybe gr.grAbortSignal - if aborted then pure (Left Aborted) else act - --- | 'callWithRetry' where each attempt is limited by 'mcRequestTimeout'. --- --- The timeout applies to each attempt separately, not to the whole --- sequence of retries and backoff sleeps. A timed-out attempt yields --- 'TimeoutError', which is retryable. For streaming, see --- 'LLM.Core.Utils.withFirstEventTimeout' (used by 'streamTextLLM'). -callWithRetryTimeout :: - GenRequest -> - ModelConfig -> - IO (Either LLMError a) -> - IO (Either LLMError a) -callWithRetryTimeout gr mc invoke = - callWithRetry gr mc (withTimeout mc.mcRequestTimeout invoke) - -withModelFallbacks :: - GenRequest -> - ModelWithFallbacks -> - (ModelConfig -> IO (Either LLMError a)) -> - IO (GenerateResult a) -withModelFallbacks gr models invokePerModel = - fmap (\(_, _, a) -> a) <$> runModelFallbacks gr models invokePerModel - --- | Like 'withModelFallbacks', but also returns the chain rotated so that the --- model that answered comes first. The models before it (the ones that --- failed) move to the end, so they stay a last resort: --- @xs ++ [m] ++ ys@ becomes @ModelWithFallbacks m (ys ++ xs)@. --- --- Used to keep an agent run on the model that answered. Logs at 'Info' when --- the chain changes. -withModelFallbacksRotated :: - GenRequest -> - ModelWithFallbacks -> - (ModelConfig -> IO (Either LLMError a)) -> - IO (GenerateResult (ModelWithFallbacks, a)) -withModelFallbacksRotated gr models invokePerModel = do - r <- runModelFallbacks gr models invokePerModel - case r of - Left err -> pure $ Left err - Right (rotated, chain, a) -> do - when rotated $ - gr.grHooks.onLog Info (formatContinuingOn chain.mwfModel) - pure $ Right (chain, a) - --- | Shared loop of 'withModelFallbacks' and 'withModelFallbacksRotated'. --- On success, returns whether the chain was rotated, the rotated chain and --- the result. --- --- Before the first model is tried, each invalid turn in 'grMessages' --- ('validateTurns') is logged at 'Warn' as @turn : @ (@n@ --- counts from 0). The request is sent unchanged. -runModelFallbacks :: - GenRequest -> - ModelWithFallbacks -> - (ModelConfig -> IO (Either LLMError a)) -> - IO (GenerateResult (Bool, ModelWithFallbacks, a)) -runModelFallbacks gr models invokePerModel = do - mapM_ - (\(i, reason) -> gr.grHooks.onLog Warn ("turn " <> T.pack (show i) <> ": " <> reason)) - (validateTurns gr.grMessages) - case mfwToModelConfigs models of - [] -> pure $ Left GErrAllModelsFailed - modelConfigs -> loop [] modelConfigs Nothing - where - loop _ [] mLast = - pure $ - Left $ - maybe GErrAllModelsFailed GErrLLM mLast - loop failed (mc : rest) mLast = do - aborted <- isAbortedMaybe gr.grAbortSignal - if aborted then pure (Left GErrAborted) else tryModel failed mc rest mLast - tryModel failed mc rest _ = do - gr.grHooks.onLog Info (formatTryingModel mc) - case validateModelCapabilities mc gr.grMessages of - Left err -> - case rest of - [] -> pure $ Left (GErrLLM err) - _ -> do - gr.grHooks.onLog Warn (formatModelFallback mc err) - loop (failed ++ [mc]) rest (Just err) - Right () -> do - r <- invokePerModel mc - case r of - Left Aborted -> pure $ Left GErrAborted - Left err -> - case rest of - [] -> pure $ Left (GErrLLM err) - _ -> do - gr.grHooks.onLog Warn (formatModelFallback mc err) - loop (failed ++ [mc]) rest (Just err) - Right a -> pure $ Right (not (null failed), ModelWithFallbacks mc (rest ++ failed), a) - -formatTryingModel :: ModelConfig -> Text -formatTryingModel mc = - "Trying model: " - <> mc.mcModel - <> " via " - <> mc.mcGateway.gwName - -formatContinuingOn :: ModelConfig -> Text -formatContinuingOn mc = - "Continuing run on " - <> mc.mcModel - <> " via " - <> mc.mcGateway.gwName - -formatModelFallback :: ModelConfig -> LLMError -> Text -formatModelFallback mc err = - "Falling back from " - <> mc.mcModel - <> ": " - <> T.pack (show err) - -llmHooks :: Hooks -> LLMHooks -llmHooks hooks = - LLMHooks - { onLLMRequest = hooks.onRequest, - onLLMResponse = hooks.onResponse, - onLLMResponseError = hooks.onResponseError - } - --- | The request with its hooks wrapped by 'safeHooks' and 'safeLLMHooks', --- so an exception thrown by a hook cannot fail a call whose side effects --- already happened. Applied by every public generation entry point. -safeGenRequest :: GenRequest -> GenRequest -safeGenRequest gr = - gr - { grHooks = safeHooks gr.grHooks, - grLLMHooks = safeLLMHooks gr.grLLMHooks - } diff --git a/src/LLM/Generate/Internal/Generate.hs b/src/LLM/Generate/Internal/Generate.hs new file mode 100644 --- /dev/null +++ b/src/LLM/Generate/Internal/Generate.hs @@ -0,0 +1,223 @@ +-- | Implementation of "LLM.Generate.Generate", plus the variants that skip +-- the turn check ('TurnCheck') for the agent loop. +module LLM.Generate.Internal.Generate + ( usageWithModelCost, + generateTextLLM, + streamTextLLM, + generateTextWithFallbacks, + streamTextWithFallbacks, + generateTextWithFallbacksRotated, + streamTextWithFallbacksRotated, + generateTextWithFallbacksRotatedWith, + streamTextWithFallbacksRotatedWith, + ) +where + +import Control.Exception (Exception, handle, throwIO) +import Control.Monad (unless, when) +import Data.IORef (IORef, newIORef, readIORef, writeIORef) +import Data.Maybe (fromMaybe) +import LLM.Core.Abort (isAbortedMaybe) +import LLM.Core.Types + ( ChatResponse (..), + LLMError (Aborted), + LLMGateway (..), + LLMTextResult, + StreamEvent (..), + Turn (..), + ) +import LLM.Core.Usage (emptyUsage) +import LLM.Core.Utils (withFirstEventTimeout) +import LLM.Generate.Internal.GenerateUtils + ( callWithRetry, + callWithRetryTimeout, + mkRequest, + usageWithModelCost, + safeGenRequest, + TurnCheck (..), + withModelFallbacksRotatedWith, + ) +import LLM.Generate.ModelConfig + ( ModelConfig (..), + ModelWithFallbacks (..), + ) +import LLM.Generate.Types + ( GenRequest (..), + GenerateResult, + RoundTextRole (..), + StreamChunk (..), + ) + +-- | Generate text with retries and model fallbacks. +-- +-- Tries 'LLM.Generate.ModelConfig.mwfModel' first, then each fallback in order. Returns a provider +-- 'ChatResponse' on success; does not execute tools. +generateTextWithFallbacks :: + GenRequest -> + ModelWithFallbacks -> + IO (GenerateResult ChatResponse) +generateTextWithFallbacks gr models = + fmap snd <$> generateTextWithFallbacksRotated (safeGenRequest gr) models + +-- | Like 'generateTextWithFallbacks', but also returns the chain rotated so +-- the model that answered comes first (see 'LLM.Generate.GenerateUtils.withModelFallbacksRotated'). +generateTextWithFallbacksRotated :: + GenRequest -> + ModelWithFallbacks -> + IO (GenerateResult (ModelWithFallbacks, ChatResponse)) +generateTextWithFallbacksRotated = generateTextWithFallbacksRotatedWith CheckTurns + +-- | 'generateTextWithFallbacksRotated' with the choice of 'TurnCheck'. +generateTextWithFallbacksRotatedWith :: + TurnCheck -> + GenRequest -> + ModelWithFallbacks -> + IO (GenerateResult (ModelWithFallbacks, ChatResponse)) +generateTextWithFallbacksRotatedWith check gr0 models = + withModelFallbacksRotatedWith check gr models $ \mc -> do + r <- generateTextLLM gr mc + case r of + Left err -> pure $ Left err + Right resp -> + let usage = usageWithModelCost mc (fromMaybe emptyUsage resp.respUsage) + in pure $ Right resp {respUsage = Just usage} + where + gr = safeGenRequest gr0 + +-- | Like 'generateTextWithFallbacks', but streams 'StreamChunk' values +-- through @onChunk@ as the provider responds. +streamTextWithFallbacks :: + (StreamChunk -> IO ()) -> + GenRequest -> + ModelWithFallbacks -> + IO (GenerateResult ChatResponse) +streamTextWithFallbacks onChunk gr models = + fmap snd <$> streamTextWithFallbacksRotated onChunk (safeGenRequest gr) models + +-- | Like 'streamTextWithFallbacks', but also returns the rotated chain +-- (see 'LLM.Generate.GenerateUtils.withModelFallbacksRotated'). +streamTextWithFallbacksRotated :: + (StreamChunk -> IO ()) -> + GenRequest -> + ModelWithFallbacks -> + IO (GenerateResult (ModelWithFallbacks, ChatResponse)) +streamTextWithFallbacksRotated = streamTextWithFallbacksRotatedWith CheckTurns + +-- | 'streamTextWithFallbacksRotated' with the choice of 'TurnCheck'. +streamTextWithFallbacksRotatedWith :: + TurnCheck -> + (StreamChunk -> IO ()) -> + GenRequest -> + ModelWithFallbacks -> + IO (GenerateResult (ModelWithFallbacks, ChatResponse)) +streamTextWithFallbacksRotatedWith check onChunk gr0 models = do + let gr = safeGenRequest gr0 + emittedRef <- newIORef False + withModelFallbacksRotatedWith check gr models $ \mc -> do + r <- streamTextAttempts emittedRef onChunk gr mc + case r of + Left err -> pure $ Left err + Right resp -> + let usage = usageWithModelCost mc (fromMaybe emptyUsage resp.respUsage) + in pure $ Right resp {respUsage = Just usage} + +generateTextLLM :: GenRequest -> ModelConfig -> IO LLMTextResult +generateTextLLM gr0 mc = + callWithRetryTimeout gr mc $ + let request = mkRequest gr mc + in mc.mcGateway.gwGenerateText gr.grLLMHooks request + where + gr = safeGenRequest gr0 + +-- | Stream one model call with retries. 'LLM.Generate.ModelConfig.mcRequestTimeout' bounds the time +-- until the first stream event of each attempt, not the whole stream. +-- +-- If an attempt fails after streaming chunks, the next attempt first emits +-- 'StreamAttemptReset'. +streamTextLLM :: (StreamChunk -> IO ()) -> GenRequest -> ModelConfig -> IO LLMTextResult +streamTextLLM onChunk gr mc = do + emittedRef <- newIORef False + streamTextAttempts emittedRef onChunk (safeGenRequest gr) mc + +-- | 'streamTextLLM' with the \"chunks emitted\" flag supplied by the caller, +-- so it is shared across retries and model fallbacks of one model call. +streamTextAttempts :: + IORef Bool -> + (StreamChunk -> IO ()) -> + GenRequest -> + ModelConfig -> + IO LLMTextResult +streamTextAttempts emittedRef onChunk gr mc = + callWithRetry gr mc $ + let request = mkRequest gr mc + onChunk' c = writeIORef emittedRef True >> onChunk c + in do + emitted <- readIORef emittedRef + when emitted $ do + onChunk StreamAttemptReset + writeIORef emittedRef False + ProviderStreamCallback {pscOnEvent, pscFinalize} <- + mkProviderStreamCallback gr onChunk' + -- Deliver no event after the abort signal fired. ('callWithRetry' + -- also interrupts the attempt, but asynchronously, so an event + -- could otherwise slip through first.) + let onEvent e = do + aborted <- isAbortedMaybe gr.grAbortSignal + if aborted then throwIO StreamAborted else pscOnEvent e + result <- + handle (\StreamAborted -> pure (Left Aborted)) $ + withFirstEventTimeout + mc.mcRequestTimeout + (mc.mcGateway.gwStreamText gr.grLLMHooks request) + onEvent + case result of + Right _ -> pscFinalize >> pure result + Left err -> pure (Left err) + +-- | Thrown from the stream callback once the abort signal has fired. +data StreamAborted = StreamAborted + deriving (Show) + +instance Exception StreamAborted + +data RoundPhase = Unclassified | Preamble | Answer + deriving (Eq) + +data ProviderStreamCallback = ProviderStreamCallback + { pscOnEvent :: StreamEvent -> IO (), + pscFinalize :: IO () + } + +initialPhase :: [Turn] -> RoundPhase +initialPhase turns = + case reverse turns of + ToolTurn _ : _ -> Answer + _ -> Unclassified + +mkProviderStreamCallback :: + GenRequest -> + (StreamChunk -> IO ()) -> + IO ProviderStreamCallback +mkProviderStreamCallback gr onChunk = do + phaseRef <- newIORef (initialPhase gr.grMessages) + pure + ProviderStreamCallback + { pscOnEvent = \case + StreamReasoningDelta txt -> onChunk (ReasoningDelta txt) + StreamDelta txt -> do + phase <- readIORef phaseRef + case phase of + Answer -> onChunk (AnswerDelta txt) + Preamble -> onChunk (PreambleDelta txt) + Unclassified -> onChunk (TextDelta txt) + StreamToolCall tc -> do + phase <- readIORef phaseRef + unless (phase == Preamble) $ + onChunk (RoundTextRoleCommitted PreambleRole) + writeIORef phaseRef Preamble + onChunk (StreamToolCallChunk tc), + pscFinalize = do + phase <- readIORef phaseRef + when (phase == Unclassified) $ + onChunk (RoundTextRoleCommitted AnswerRole) + } diff --git a/src/LLM/Generate/Internal/GenerateUtils.hs b/src/LLM/Generate/Internal/GenerateUtils.hs new file mode 100644 --- /dev/null +++ b/src/LLM/Generate/Internal/GenerateUtils.hs @@ -0,0 +1,334 @@ +-- | Implementation of "LLM.Generate.GenerateUtils", plus the variants of +-- the fallback loop that the agent loop uses ('TurnCheck'). +module LLM.Generate.Internal.GenerateUtils + ( maybeThrottle, + mkRequest, + usageWithModelCost, + callWithRetry, + callWithRetryTimeout, + withModelFallbacks, + withModelFallbacksRotated, + TurnCheck (..), + withModelFallbacksRotatedWith, + logInvalidTurns, + validateModelCapabilities, + llmHooks, + safeGenRequest, + ) +where + +import Control.Concurrent (threadDelay) +import Control.Monad (when) +import Data.Text (Text) +import Data.Text qualified as T +import Data.List (nub, sortOn) +import Data.Maybe (fromMaybe) +import LLM.Core.Abort (isAbortedMaybe) +import LLM.Core.Types + ( ChatRequest (..), + ContentPart (..), + ImageSource (..), + LLMError (..), + LLMGateway (gwImageMediaTypes, gwImageUrlSupport, gwName), + PartBody (..), + LLMHooks (..), + ThinkingMode (..), + Turn (..), + conversationHasImages, + validateConversation, + validateTurns, + ) +import LLM.Core.Usage (Usage (..), estimateCost) +import LLM.Core.Utils (interruptOnAbort, withRetry, withTimeout) +import LLM.Generate.Logger (Hooks (..), LogLevel (..), safeHooks, safeLLMHooks) +import LLM.Generate.ModelConfig + ( ModelCapabilities (..), + ModelConfig (..), + ModelWithFallbacks (..), + mfwToModelConfigs, + modelRetryPolicy, + ) +import LLM.Generate.Types (GenRequest (..), GenerateError (..), GenerateResult) + +maybeThrottle :: Maybe Int -> IO a -> IO a +maybeThrottle Nothing io = io +maybeThrottle (Just ms) io = threadDelay (ms * 1000) >> io + +mkRequest :: GenRequest -> ModelConfig -> ChatRequest +mkRequest gr mc = + ChatRequest + { reqModel = mc.mcModel, + reqConversation = gr.grMessages, + reqSystem = gr.grSystemPrompt, + reqMaxTokens = mc.mcMaxTokens, + reqTemperature = mc.mcTemperature, + reqTools = gr.grTools, + reqThinking = mc.mcThinking + } + +usageWithModelCost :: ModelConfig -> Usage -> Usage +usageWithModelCost mc u = u {usageTotalCost = estimateCost mc.mcPricing u} + +-- | Check that a candidate model declares every capability the request needs. +-- +-- Unsupported vision is a candidate failure (do not drop images), and so is +-- an inline image whose media type the gateway does not accept +-- ('LLM.Core.Types.gwImageMediaTypes') or a URL image on a gateway without +-- 'LLM.Core.Types.gwImageUrlSupport'. URL images have no known media type, so their type +-- is not checked. +-- Thinking configuration requires a declared thinking capability. Cache +-- hints are not validated here; unsupported providers ignore them. +validateModelCapabilities :: ModelConfig -> [Turn] -> Either LLMError () +validateModelCapabilities mc turns = do + whenNeedsVision + whenNeedsMediaTypes + whenNeedsImageUrls + whenNeedsThinking + where + caps = mc.mcCapabilities + whenNeedsVision + | conversationHasImages turns, + not caps.capVision = + Left $ + UnsupportedCapability $ + "model " + <> mc.mcModel + <> " does not support vision" + | otherwise = Right () + whenNeedsMediaTypes = + case filter (`notElem` mc.mcGateway.gwImageMediaTypes) (inlineImageMediaTypes turns) of + [] -> Right () + unsupported -> + Left $ + UnsupportedCapability $ + "model " + <> mc.mcModel + <> " via " + <> mc.mcGateway.gwName + <> " does not accept image media type " + <> T.intercalate ", " unsupported + whenNeedsImageUrls + | conversationHasImageUrls turns, + not mc.mcGateway.gwImageUrlSupport = + Left $ + UnsupportedCapability $ + "model " + <> mc.mcModel + <> " via " + <> mc.mcGateway.gwName + <> " does not accept image URLs" + | otherwise = Right () + whenNeedsThinking = + case mc.mcThinking of + Just ThinkingMode {tmEnabled = True} + | not caps.capThinking -> + Left $ + UnsupportedCapability $ + "model " + <> mc.mcModel + <> " does not support thinking" + _ -> Right () + +-- | Distinct media types of the inline ('ImageBase64') images in a conversation. +inlineImageMediaTypes :: [Turn] -> [Text] +inlineImageMediaTypes turns = + nub [mt | ImageBase64 mt _ <- imageSources turns] + +conversationHasImageUrls :: [Turn] -> Bool +conversationHasImageUrls turns = + not (null [() | ImageUrl _ <- imageSources turns]) + +imageSources :: [Turn] -> [ImageSource] +imageSources turns = + [src | parts <- map turnParts turns, ContentPart (ImagePart src) _ <- parts] + where + turnParts (UserMessage parts) = parts + turnParts (AssistantMessage parts) = parts + turnParts (ToolTurn _) = [] + +-- | Run one model call with the model's throttle delay and retry policy. +-- @attempt@ is run once per try; any per-attempt timeout must be inside it. +-- +-- 'LLM.Generate.ModelConfig.mcThrottleDelay' is slept before every try, retries included. Before a +-- retry it comes after the retry backoff, so the two delays add up. +-- +-- When 'LLM.Generate.Types.grAbortSignal' fires, the call is interrupted wherever it is (in an +-- attempt, or in the throttle or backoff sleep) and returns 'Aborted'. An +-- interrupted attempt is cancelled, which closes its HTTP connection. +callWithRetry :: + GenRequest -> + ModelConfig -> + IO (Either LLMError a) -> + IO (Either LLMError a) +callWithRetry gr mc attempt = + fmap (fromMaybe (Left Aborted)) $ + interruptOnAbort gr.grAbortSignal $ + withRetry + (modelRetryPolicy mc) + (gr.grHooks.onLog Warn) + (maybeThrottle mc.mcThrottleDelay (unlessAborted gr attempt)) + +-- | Run @act@ unless 'LLM.Generate.Types.grAbortSignal' has fired, else return 'Aborted' +-- (which is not retried). Checked before every attempt, so also after +-- throttle and backoff sleeps. +unlessAborted :: GenRequest -> IO (Either LLMError a) -> IO (Either LLMError a) +unlessAborted gr act = do + aborted <- isAbortedMaybe gr.grAbortSignal + if aborted then pure (Left Aborted) else act + +-- | 'callWithRetry' where each attempt is limited by 'LLM.Generate.ModelConfig.mcRequestTimeout'. +-- +-- The timeout applies to each attempt separately, not to the whole +-- sequence of retries and backoff sleeps. A timed-out attempt yields +-- 'TimeoutError', which is retryable. For streaming, see +-- 'LLM.Core.Utils.withFirstEventTimeout' (used by 'LLM.Generate.Generate.streamTextLLM'). +callWithRetryTimeout :: + GenRequest -> + ModelConfig -> + IO (Either LLMError a) -> + IO (Either LLMError a) +callWithRetryTimeout gr mc invoke = + callWithRetry gr mc (withTimeout mc.mcRequestTimeout invoke) + +withModelFallbacks :: + GenRequest -> + ModelWithFallbacks -> + (ModelConfig -> IO (Either LLMError a)) -> + IO (GenerateResult a) +withModelFallbacks gr models invokePerModel = + fmap (\(_, _, a) -> a) <$> runModelFallbacks CheckTurns gr models invokePerModel + +-- | Like 'withModelFallbacks', but also returns the chain rotated so that the +-- model that answered comes first. The models before it (the ones that +-- failed) move to the end, so they stay a last resort: +-- @xs ++ [m] ++ ys@ becomes @ModelWithFallbacks m (ys ++ xs)@. +-- +-- Used to keep an agent run on the model that answered. Logs at 'Info' when +-- the chain changes. +withModelFallbacksRotated :: + GenRequest -> + ModelWithFallbacks -> + (ModelConfig -> IO (Either LLMError a)) -> + IO (GenerateResult (ModelWithFallbacks, a)) +withModelFallbacksRotated = withModelFallbacksRotatedWith CheckTurns + +-- | Whether the fallback loop checks the request's messages first. +data TurnCheck + = -- | Log the problems of @grMessages@ before the first model + -- ('logInvalidTurns'). What every public entry point does. + CheckTurns + | -- | Do not check. For the agent loop, which checks the caller's turns + -- once per run; the turns it adds come from providers. + SkipTurnCheck + +-- | 'withModelFallbacksRotated' with the choice of 'TurnCheck'. +withModelFallbacksRotatedWith :: + TurnCheck -> + GenRequest -> + ModelWithFallbacks -> + (ModelConfig -> IO (Either LLMError a)) -> + IO (GenerateResult (ModelWithFallbacks, a)) +withModelFallbacksRotatedWith check gr models invokePerModel = do + r <- runModelFallbacks check gr models invokePerModel + case r of + Left err -> pure $ Left err + Right (rotated, chain, a) -> do + when rotated $ + gr.grHooks.onLog Info (formatContinuingOn chain.mwfModel) + pure $ Right (chain, a) + +-- | Shared loop of 'withModelFallbacks' and 'withModelFallbacksRotated'. +-- On success, returns whether the chain was rotated, the rotated chain and +-- the result. +-- +-- With 'CheckTurns', 'logInvalidTurns' runs before the first model is +-- tried. The request is sent unchanged. +runModelFallbacks :: + TurnCheck -> + GenRequest -> + ModelWithFallbacks -> + (ModelConfig -> IO (Either LLMError a)) -> + IO (GenerateResult (Bool, ModelWithFallbacks, a)) +runModelFallbacks check gr models invokePerModel = do + case check of + CheckTurns -> logInvalidTurns gr.grHooks gr.grMessages + SkipTurnCheck -> pure () + case mfwToModelConfigs models of + [] -> pure $ Left GErrAllModelsFailed + modelConfigs -> loop [] modelConfigs Nothing + where + loop _ [] mLast = + pure $ + Left $ + maybe GErrAllModelsFailed GErrLLM mLast + loop failed (mc : rest) mLast = do + aborted <- isAbortedMaybe gr.grAbortSignal + if aborted then pure (Left GErrAborted) else tryModel failed mc rest mLast + tryModel failed mc rest _ = do + gr.grHooks.onLog Info (formatTryingModel mc) + case validateModelCapabilities mc gr.grMessages of + Left err -> + case rest of + [] -> pure $ Left (GErrLLM err) + _ -> do + gr.grHooks.onLog Warn (formatModelFallback mc err) + loop (failed ++ [mc]) rest (Just err) + Right () -> do + r <- invokePerModel mc + case r of + Left Aborted -> pure $ Left GErrAborted + Left err -> + case rest of + [] -> pure $ Left (GErrLLM err) + _ -> do + gr.grHooks.onLog Warn (formatModelFallback mc err) + loop (failed ++ [mc]) rest (Just err) + Right a -> pure $ Right (not (null failed), ModelWithFallbacks mc (rest ++ failed), a) + +-- | Log each invalid turn ('validateTurns') and each problem in the +-- tool-call structure ('validateConversation') at 'Warn' as +-- @turn : @ (@n@ counts from 0), in turn order. +logInvalidTurns :: Hooks -> [Turn] -> IO () +logInvalidTurns hooks turns = + mapM_ + (\(i, reason) -> hooks.onLog Warn ("turn " <> T.pack (show i) <> ": " <> reason)) + (sortOn fst (validateTurns turns ++ validateConversation turns)) + +formatTryingModel :: ModelConfig -> Text +formatTryingModel mc = + "Trying model: " + <> mc.mcModel + <> " via " + <> mc.mcGateway.gwName + +formatContinuingOn :: ModelConfig -> Text +formatContinuingOn mc = + "Continuing run on " + <> mc.mcModel + <> " via " + <> mc.mcGateway.gwName + +formatModelFallback :: ModelConfig -> LLMError -> Text +formatModelFallback mc err = + "Falling back from " + <> mc.mcModel + <> ": " + <> T.pack (show err) + +llmHooks :: Hooks -> LLMHooks +llmHooks hooks = + LLMHooks + { onLLMRequest = hooks.onRequest, + onLLMResponse = hooks.onResponse, + onLLMResponseError = hooks.onResponseError + } + +-- | The request with its hooks wrapped by 'safeHooks' and 'safeLLMHooks', +-- so an exception thrown by a hook cannot fail a call whose side effects +-- already happened. Applied by every public generation entry point. +safeGenRequest :: GenRequest -> GenRequest +safeGenRequest gr = + gr + { grHooks = safeHooks gr.grHooks, + grLLMHooks = safeLLMHooks gr.grLLMHooks + } diff --git a/src/LLM/Generate/ModelConfig.hs b/src/LLM/Generate/ModelConfig.hs --- a/src/LLM/Generate/ModelConfig.hs +++ b/src/LLM/Generate/ModelConfig.hs @@ -58,7 +58,7 @@ mcGateway :: LLMGateway, -- | Provider-specific model identifier (e.g. @gpt-4.1-2025-04-14@). mcModel :: Text, - -- | Per-million-token pricing used to populate 'Usage.usageTotalCost'. + -- | Per-million-token pricing used to populate 'LLM.Core.Usage.usageTotalCost'. mcPricing :: PricingInfo, -- | Maximum tokens requested from the provider. mcMaxTokens :: Int, @@ -75,7 +75,7 @@ -- | Delay in milliseconds before each API call attempt, retries -- included (rate limiting). Before a retry it is added to the backoff. mcThrottleDelay :: Maybe Int, - -- | Number of retries on retryable 'LLMError's before giving up on this model. + -- | Number of retries on retryable t'LLM.Core.Types.LLMError's before giving up on this model. mcRetryCount :: Int, -- | Base backoff in milliseconds between retries (jittered). The -- server's @Retry-After@ header is not honoured, so for a provider that @@ -85,8 +85,8 @@ -- | Primary model plus an ordered fallback chain. -- --- On failure the library tries 'mwfModel' first, then each entry in --- 'mwfFallbacks' in order. Aborts ('GErrAborted') do not fall through. +-- On failure the library tries 'LLM.Generate.ModelConfig.mwfModel' first, then each entry in +-- 'LLM.Generate.ModelConfig.mwfFallbacks' in order. Aborts (v'LLM.Generate.Types.GErrAborted') do not fall through. data ModelWithFallbacks = ModelWithFallbacks { mwfModel :: ModelConfig, mwfFallbacks :: [ModelConfig] @@ -95,8 +95,8 @@ mfwToModelConfigs :: ModelWithFallbacks -> [ModelConfig] mfwToModelConfigs mwf = mwf.mwfModel : mwf.mwfFallbacks --- | 'mcRetryCount' retries with full-jitter exponential backoff from --- 'mcJitterBackoff', each delay capped at 30 s. @Retry-After@ is ignored. +-- | 'LLM.Generate.ModelConfig.mcRetryCount' retries with full-jitter exponential backoff from +-- 'LLM.Generate.ModelConfig.mcJitterBackoff', each delay capped at 30 s. @Retry-After@ is ignored. modelRetryPolicy :: ModelConfig -> RetryPolicyM IO modelRetryPolicy mc = capDelay 30_000_000 $ diff --git a/src/LLM/Generate/Types.hs b/src/LLM/Generate/Types.hs --- a/src/LLM/Generate/Types.hs +++ b/src/LLM/Generate/Types.hs @@ -25,11 +25,11 @@ import LLM.Core.Usage (Usage) import LLM.Generate.Logger (Hooks) --- | A single provider request. Built manually or via 'createGenRequest'. +-- | A single provider request. Built manually or via 'LLM.Agent.ToolUtils.createGenRequest'. -- --- For structured output ('genObject', 'genObjectUntyped'), leave 'grTools' empty. +-- For structured output ('LLM.Generate.GenerateObject.genObject', 'LLM.Generate.GenerateObject.genObjectUntyped'), leave 'LLM.Generate.Types.grTools' empty. data GenRequest = GenRequest - { -- | System prompt override (also available on 'Agent.agSystemPrompt'). + { -- | System prompt override (also available on 'LLM.Agent.Types.agSystemPrompt'). grSystemPrompt :: Maybe Text, -- | Conversation turns sent to the provider (may already be windowed). grMessages :: [Turn], @@ -51,7 +51,7 @@ -- | Successful result from the agent loop or a single-shot text generation. data GenerateTextResult = GenerateTextResult - { -- | Matches 'RuntimeArgs.rtGenerationId' for this run. + { -- | Matches 'LLM.Agent.Types.rtGenerationId' for this run. gtrGenerationId :: UUID, -- | New turns produced during this call (assistant and tool turns only). gtrNewMessages :: [Turn], @@ -79,9 +79,9 @@ data GenerateError = -- | Provider-level failure (HTTP, network, parse, etc.). GErrLLM LLMError - | -- | 'Agent.agMaxToolRounds' exhausted, and the model still called tools + | -- | 'LLM.Agent.Types.agMaxToolRounds' exhausted, and the model still called tools -- in the final call that followed. Those calls were not run; - -- 'gerPartialNewMessages' ends with that assistant turn and a tool turn + -- 'LLM.Generate.Types.gerPartialNewMessages' ends with that assistant turn and a tool turn -- answering each call with a placeholder, so it can be sent again as is. GErrToolExceeded | -- | Every model in the fallback chain failed. diff --git a/src/LLM/Load/FsTools.hs b/src/LLM/Load/FsTools.hs --- a/src/LLM/Load/FsTools.hs +++ b/src/LLM/Load/FsTools.hs @@ -29,7 +29,7 @@ -- All paths are resolved inside a canonical sandbox ('LLM.Tools.FsConfig'). -- Mutating tools refuse paths that escape the workspace; reads are bounded -- (1 MiB for edits, 10 MiB for paginated reads). Include desired names in --- 'Agent.agTools' to expose them to the model. +-- 'LLM.Agent.Types.agTools' to expose them to the model. fsTools :: FilePath -> IO (ToolMap Text) fsTools filePath = do cfg <- mkFsConfig filePath diff --git a/src/LLM/Load/LoadGateways.hs b/src/LLM/Load/LoadGateways.hs --- a/src/LLM/Load/LoadGateways.hs +++ b/src/LLM/Load/LoadGateways.hs @@ -209,7 +209,7 @@ -- yourself or use 'loadGatewaysWithDotenv'. -- -- For custom providers, place a @providers.json@ next to your model catalog; --- 'loadModelOrThrow' loads that file automatically. +-- 'LLM.Load.LoadModels.loadModelOrThrow' loads that file automatically. loadGateways :: IO GatewayMap loadGateways = loadGatewaysFromCatalog defaultProviderCatalogMap diff --git a/src/LLM/Providers/Claude.hs b/src/LLM/Providers/Claude.hs --- a/src/LLM/Providers/Claude.hs +++ b/src/LLM/Providers/Claude.hs @@ -39,8 +39,9 @@ import Data.Text.Encoding (encodeUtf8) import Data.Text.Lazy qualified as TL import Data.Text.Lazy.Encoding (decodeUtf8) +import LLM.Core.Internal.Transport (postJson, postJsonStream) import LLM.Core.LLMProvider (LLMProvider (..), toGateway) -import LLM.Core.ProviderUtils (decodeObjectText, decodeToolArguments, handleStreamResponse, invalidToolArgumentsError, lenientConfig, readResponseBody, unexpectedResponse) +import LLM.Core.ProviderUtils (decodeObjectText, decodeToolArguments, invalidToolArgumentsError, unexpectedResponse) import LLM.Core.SSE (SSEEvent (sseData, sseEvent), malformedEvent, readSSEEvents, streamFailure) import LLM.Core.Types ( commonImageMediaTypes, @@ -78,16 +79,11 @@ pattern UserTurn, ) import LLM.Core.Usage (Usage (..), emptyUsage) -import Network.HTTP.Client qualified as HC import Network.HTTP.Req ( Option, - POST (POST), - ReqBodyJson (ReqBodyJson), Url, header, https, - reqBr, - runReq, (/:), ) @@ -132,12 +128,7 @@ providerImageUrlSupport = True, buildBody = claudeBuildBody, sendRequest = sendRequest, - openStream = \body k -> - runReq lenientConfig $ do - let url = baseUrl /: "v1" /: "messages" - opts = baseOpts <> claudeAuthOpts apiKey - reqBr POST url (ReqBodyJson body) opts $ \resp -> - handleStreamResponse resp (k . HC.brRead), + openStream = postJsonStream url opts, parseStream = \body readChunk callback -> let modelHint = fromMaybe "" (parseMaybe (withObject "body" (.: "model")) body) in parseClaudeStream modelHint readChunk callback, @@ -151,11 +142,9 @@ parseObjectResponse = parseClaudeObjectResponse } where - sendRequest body = - runReq lenientConfig $ do - let url = baseUrl /: "v1" /: "messages" - opts = baseOpts <> claudeAuthOpts apiKey - reqBr POST url (ReqBodyJson body) opts readResponseBody + sendRequest = postJson url opts + url = baseUrl /: "v1" /: "messages" + opts = baseOpts <> claudeAuthOpts apiKey claudeAuthOpts :: Text -> Option scheme claudeAuthOpts apiKey = @@ -447,7 +436,7 @@ -- | The JSON text of the first text block. No text block is -- 'EmptyResponse'; invalid JSON text or a body without @content@ is a --- 'ParseError' (see 'decodeObjectText' and 'unexpectedResponse'). +-- v'LLM.Core.Types.ParseError' (see 'decodeObjectText' and 'unexpectedResponse'). parseClaudeObjectResponse :: Value -> IO (LLMResult (Value, Maybe Usage)) parseClaudeObjectResponse v = pure $ case parseEither go v of Left err -> Left (unexpectedResponse err v) diff --git a/src/LLM/Providers/DeepSeek.hs b/src/LLM/Providers/DeepSeek.hs --- a/src/LLM/Providers/DeepSeek.hs +++ b/src/LLM/Providers/DeepSeek.hs @@ -17,8 +17,8 @@ import Data.Text (Text) import Data.Text.Lazy qualified as TL import Data.Text.Lazy.Encoding qualified as TLE +import LLM.Core.Internal.Transport (postJson, postJsonStream) import LLM.Core.LLMProvider (LLMProvider (..), toGateway) -import LLM.Core.ProviderUtils (handleStreamResponse, lenientConfig, readResponseBody) import LLM.Core.Types ( commonImageMediaTypes, ChatRequest @@ -45,12 +45,8 @@ ) import Network.HTTP.Req ( Option, - POST (POST), - ReqBodyJson (ReqBodyJson), Url, https, - reqBr, - runReq, (/:), ) @@ -75,12 +71,7 @@ providerImageUrlSupport = True, buildBody = deepSeekBuildBody, sendRequest = sendRequest, - openStream = \body k -> - runReq lenientConfig $ do - let url = baseUrl /: "v1" /: "chat" /: "completions" - opts = baseOpts <> authHeader apiKey - reqBr POST url (ReqBodyJson body) opts $ \resp -> - handleStreamResponse resp k, + openStream = postJsonStream url opts, parseStream = const parseOpenAIStream, parseResponse = pure . parseOpenAIResponse, buildObjectBody = \r schema -> @@ -97,11 +88,9 @@ parseObjectResponse = pure . parseOpenAIObjectResponse } where - sendRequest body = - runReq lenientConfig $ do - let url = baseUrl /: "v1" /: "chat" /: "completions" - opts = baseOpts <> authHeader apiKey - reqBr POST url (ReqBodyJson body) opts readResponseBody + sendRequest = postJson url opts + url = baseUrl /: "v1" /: "chat" /: "completions" + opts = baseOpts <> authHeader apiKey -- | OpenAI-compatible body; DeepSeek uses @max_tokens@ (not @max_completion_tokens@). deepSeekBuildBody :: Bool -> ChatRequest -> Value diff --git a/src/LLM/Providers/Gemini.hs b/src/LLM/Providers/Gemini.hs --- a/src/LLM/Providers/Gemini.hs +++ b/src/LLM/Providers/Gemini.hs @@ -35,8 +35,9 @@ import Data.Text.Encoding (encodeUtf8) import Data.UUID.Types (UUID) import Data.UUID.Types qualified as UUID +import LLM.Core.Internal.Transport (postJson, postJsonStream) import LLM.Core.LLMProvider (LLMProvider (..), toGateway, withoutRoutingFields) -import LLM.Core.ProviderUtils (decodeObjectText, handleStreamResponse, lenientConfig, readResponseBody, stripBoundsAndComments) +import LLM.Core.ProviderUtils (decodeObjectText, stripBoundsAndComments) import LLM.Core.SSE (SSEEvent (sseData), malformedEvent, readSSEEvents, streamFailure) import LLM.Core.Types ( supportedImageMediaTypes, @@ -75,13 +76,9 @@ import Network.HTTP.Client qualified as HC import Network.HTTP.Req ( Option, - POST (POST), - ReqBodyJson (ReqBodyJson), Url, header, https, - reqBr, - runReq, (/:), (=:), ) @@ -113,17 +110,11 @@ providerImageUrlSupport = True, buildBody = const geminiBuildBody, sendRequest = sendRequest, - openStream = \body k -> - runReq lenientConfig $ do - let model = extractModel body - url = - baseUrl - /: "v1beta" - /: "models" - /: (model <> ":streamGenerateContent") - opts = baseOpts <> geminiAuthOpts apiKey <> "alt" =: ("sse" :: Text) - reqBr POST url (ReqBodyJson (withoutRoutingFields body)) opts $ \resp -> - handleStreamResponse resp k, + openStream = \body -> + postJsonStream + (modelUrl body ":streamGenerateContent") + (opts <> "alt" =: ("sse" :: Text)) + (withoutRoutingFields body), parseStream = const parseGeminiStream, parseResponse = parseGeminiResponse, buildObjectBody = \r schema -> @@ -153,15 +144,10 @@ } where sendRequest body = - runReq lenientConfig $ do - let model = extractModel body - url = - baseUrl - /: "v1beta" - /: "models" - /: (model <> ":generateContent") - opts = baseOpts <> geminiAuthOpts apiKey - reqBr POST url (ReqBodyJson (withoutRoutingFields body)) opts readResponseBody + postJson (modelUrl body ":generateContent") opts (withoutRoutingFields body) + -- The model is in the path, taken from the body's routing field. + modelUrl body method = baseUrl /: "v1beta" /: "models" /: (extractModel body <> method) + opts = baseOpts <> geminiAuthOpts apiKey geminiAuthOpts :: Text -> Option scheme geminiAuthOpts apiKey = header "x-goog-api-key" (encodeUtf8 apiKey) @@ -562,7 +548,7 @@ } } --- | Pull a thought signature out of 'tcProviderMeta' iff it was emitted by +-- | Pull a thought signature out of 'LLM.Core.Types.tcProviderMeta' iff it was emitted by -- the model we're currently calling. signatureForModel :: Text -> Maybe ProviderOpaque -> Maybe Text signatureForModel currentModel (Just o) diff --git a/src/LLM/Providers/Ollama.hs b/src/LLM/Providers/Ollama.hs --- a/src/LLM/Providers/Ollama.hs +++ b/src/LLM/Providers/Ollama.hs @@ -12,8 +12,9 @@ ) import Data.Aeson.Types (Pair) import Data.Text (Text) +import LLM.Core.Internal.Transport (postJson, postJsonStream) import LLM.Core.LLMProvider (LLMProvider (..), toGateway) -import LLM.Core.ProviderUtils (handleStreamResponse, lenientConfig, readResponseBody, normalizeSchemaOpenAI) +import LLM.Core.ProviderUtils (normalizeSchemaOpenAI) import LLM.Core.Types ( commonImageMediaTypes, ChatRequest @@ -28,13 +29,9 @@ import LLM.Providers.OpenAI (buildMessages, encodeToolDef, parseOpenAIObjectResponse, parseOpenAIResponse, parseOpenAIStream) import Network.HTTP.Req ( Option, - POST (POST), - ReqBodyJson (ReqBodyJson), Url, http, port, - reqBr, - runReq, (/:), ) @@ -60,11 +57,7 @@ providerImageUrlSupport = False, buildBody = \stream r -> object (ollamaBuildBodyPairs stream r), sendRequest = sendRequest, - openStream = \body k -> - runReq lenientConfig $ do - let url = baseUrl /: "v1" /: "chat" /: "completions" - reqBr POST url (ReqBodyJson body) baseOpts $ \resp -> - handleStreamResponse resp k, + openStream = postJsonStream url baseOpts, parseStream = const parseOpenAIStream, parseResponse = pure . parseOpenAIResponse, buildObjectBody = \r schema -> @@ -85,10 +78,8 @@ parseObjectResponse = pure . parseOpenAIObjectResponse } where - sendRequest body = - runReq lenientConfig $ do - let url = baseUrl /: "v1" /: "chat" /: "completions" - reqBr POST url (ReqBodyJson body) baseOpts readResponseBody + sendRequest = postJson url baseOpts + url = baseUrl /: "v1" /: "chat" /: "completions" -- | Request body for Ollama's OpenAI-compatible endpoint. Like OpenAI's, but -- the token limit is @max_tokens@: Ollama ignores @max_completion_tokens@ diff --git a/src/LLM/Providers/OpenAI.hs b/src/LLM/Providers/OpenAI.hs --- a/src/LLM/Providers/OpenAI.hs +++ b/src/LLM/Providers/OpenAI.hs @@ -44,8 +44,9 @@ import Data.Text (Text) import Data.Text qualified as T import Data.Text.Encoding (decodeUtf8, encodeUtf8) +import LLM.Core.Internal.Transport (postJson, postJsonStream) import LLM.Core.LLMProvider (LLMProvider (..), toGateway) -import LLM.Core.ProviderUtils (decodeObjectText, decodeToolArguments, handleStreamResponse, invalidToolArgumentsError, lenientConfig, normalizeSchemaOpenAI, readResponseBody, unexpectedResponse) +import LLM.Core.ProviderUtils (decodeObjectText, decodeToolArguments, invalidToolArgumentsError, normalizeSchemaOpenAI, unexpectedResponse) import LLM.Core.SSE (SSEEvent (sseData), malformedEvent, readSSEEvents, streamFailure) import LLM.Core.Types ( commonImageMediaTypes, @@ -86,13 +87,9 @@ import Network.HTTP.Client qualified as HC import Network.HTTP.Req ( Option, - POST (POST), - ReqBodyJson (ReqBodyJson), Url, header, https, - reqBr, - runReq, (/:), ) @@ -126,12 +123,7 @@ providerImageUrlSupport = True, buildBody = openAIBuildBody, sendRequest = sendRequest, - openStream = \body k -> - runReq lenientConfig $ do - let url = baseUrl /: "v1" /: "chat" /: "completions" - opts = baseOpts <> authHeader apiKey - reqBr POST url (ReqBodyJson body) opts $ \resp -> - handleStreamResponse resp k, + openStream = postJsonStream url opts, parseStream = const parseOpenAIStream, parseResponse = pure . parseOpenAIResponse, buildObjectBody = \r schema -> @@ -152,11 +144,9 @@ parseObjectResponse = pure . parseOpenAIObjectResponse } where - sendRequest body = - runReq lenientConfig $ do - let url = baseUrl /: "v1" /: "chat" /: "completions" - opts = baseOpts <> authHeader apiKey - reqBr POST url (ReqBodyJson body) opts readResponseBody + sendRequest = postJson url opts + url = baseUrl /: "v1" /: "chat" /: "completions" + opts = baseOpts <> authHeader apiKey authHeader :: Text -> Option scheme authHeader apiKey @@ -352,7 +342,7 @@ -- -- No choice, or a missing or null content, is 'EmptyResponse'; so is -- content that is empty once code fences are stripped. Content that is not --- valid JSON, and a body without the expected fields, are 'ParseError's +-- valid JSON, and a body without the expected fields, are v'LLM.Core.Types.ParseError's -- that quote the decoder's message and the start of the text or body. parseOpenAIObjectResponse :: Value -> LLMResult (Value, Maybe Usage) parseOpenAIObjectResponse v = case parseEither content v of diff --git a/src/LLM/Tools/FindFiles.hs b/src/LLM/Tools/FindFiles.hs --- a/src/LLM/Tools/FindFiles.hs +++ b/src/LLM/Tools/FindFiles.hs @@ -229,8 +229,8 @@ matchGlob (p : ps) (s : ss) = matchSegment (T.unpack p) (T.unpack s) && matchGlob ps ss -- | Match a single glob segment (no '/' allowed in either side). --- Supports '*' (any run of chars, possibly empty) and '?' (one char). --- On a mismatch only the most recent '*' takes one more char, which is +-- Supports @*@ (any run of chars, possibly empty) and @?@ (one char). +-- On a mismatch only the most recent @*@ takes one more char, which is -- enough for this glob syntax and keeps matching O(pattern * name) instead -- of exponential in the number of stars. matchSegment :: String -> String -> Bool diff --git a/src/LLM/Tools/FsConfig.hs b/src/LLM/Tools/FsConfig.hs --- a/src/LLM/Tools/FsConfig.hs +++ b/src/LLM/Tools/FsConfig.hs @@ -28,7 +28,7 @@ import System.FilePath (addTrailingPathSeparator, dropTrailingPathSeparator, equalFilePath, isAbsolute, isRelative, joinPath, normalise, splitDirectories, splitFileName, takeDirectory, ()) -- | Configuration for file-system tools. --- 'fsBasePath' must be a canonical absolute path (use 'mkFsConfig'). +-- 'LLM.Tools.FsConfig.fsBasePath' must be a canonical absolute path (use 'mkFsConfig'). -- -- The @FsConfig@ data constructor is deprecated: build the configuration -- with 'mkFsConfig', which canonicalizes the path. From 0.5 the constructor @@ -99,7 +99,7 @@ -- dangling link is resolved to that target and judged by where it -- points rather than appended as if it were an ordinary new file. -- Throws 'SandboxViolation' on escape attempts, and an 'IOException' for a --- @..@ that would step back out of a symbolic link (see 'collapseDots'). +-- @..@ that would step back out of a symbolic link (see @collapseDots@). sandboxPath :: FsConfig -> FilePath -> IO FilePath sandboxPath cfg relPath = do let base = cfg.fsBasePath diff --git a/src/LLM/Tools/FsLimits.hs b/src/LLM/Tools/FsLimits.hs --- a/src/LLM/Tools/FsLimits.hs +++ b/src/LLM/Tools/FsLimits.hs @@ -23,6 +23,7 @@ maxTreeEntries, maxVisitedEntries, maxOutputBytes, + maxSearchBytes, binarySniffBytes, detectBinary, readBoundedTextFile, @@ -40,10 +41,10 @@ import Data.Text.Encoding (decodeUtf8With, encodeUtf8) import Data.Text.Encoding.Error (lenientDecode) import LLM.Tools.Internal.TextFile - ( binarySniffBytes, - readBoundedTextWith, + ( readBoundedTextWith, readBoundedUtf8With, ) +import LLM.Tools.Internal.TextFile qualified as TextFile import System.Directory (copyPermissions, doesFileExist, removeFile, renameFile) import System.FilePath (splitFileName) import System.IO @@ -94,6 +95,21 @@ -- is cut and the cut is reported. maxOutputBytes :: Int maxOutputBytes = 100 * 1024 -- 100 KiB + +-- | Budget of file bytes one @grep@ call may read. Only the text files it +-- searches count; binary files and files over the per-file size cap are +-- skipped without being read. A search that runs out stops before the +-- next file and says so in its output. +-- +-- @since 0.4.3.0 +maxSearchBytes :: Int +maxSearchBytes = 64 * 1024 * 1024 -- 64 MiB + +-- | Number of leading bytes inspected when sniffing for binary content. +-- Matches the heuristic used by @git diff@ and @grep@: any NUL byte in +-- the prefix means binary. +binarySniffBytes :: Int +binarySniffBytes = TextFile.binarySniffBytes -- | A NUL byte in the first 'binarySniffBytes' is a robust, cheap binary -- heuristic. On I/O failure the file is treated as binary (the caller diff --git a/src/LLM/Tools/Grep.hs b/src/LLM/Tools/Grep.hs --- a/src/LLM/Tools/Grep.hs +++ b/src/LLM/Tools/Grep.hs @@ -7,16 +7,20 @@ import Autodocodec qualified as AC import Control.Exception (IOException, try) import Control.Monad (unless) -import Data.IORef (IORef, modifyIORef', newIORef, readIORef) +import Data.IORef (IORef, atomicModifyIORef', modifyIORef', newIORef, readIORef) import Data.Aeson (FromJSON) import Data.List (sort) import Data.Text (Text) +import Data.ByteString qualified as BS +import Data.Maybe (catMaybes) import Data.Text qualified as T +import Data.Text.Encoding (decodeUtf8With) +import Data.Text.Encoding.Error (lenientDecode) import GHC.Generics (Generic) import LLM.Core.Types (TypedTool (..)) import LLM.Tools.FsConfig (FsConfig, isFileHidden, isSymlink, sandboxPath, withWorkspacePaths) -import LLM.Tools.FsLimits (expensiveWalkDepth, maxOutputBytes, maxVisitedEntries) -import LLM.Tools.Internal.TextFile (readLenient, withTextHandle) +import LLM.Tools.FsLimits (expensiveWalkDepth, maxOutputBytes, maxSearchBytes, maxVisitedEntries) +import LLM.Tools.Internal.TextFile (withTextHandle) import LLM.Tools.Internal.WalkBudget (WalkBudget, budgetExhausted, budgetNote, capLines, newWalkBudget, takeEntries) import System.Directory ( doesDirectoryExist, @@ -27,6 +31,7 @@ import System.FilePath (takeExtension, takeFileName, ()) import System.IO ( IOMode (ReadMode), + hFileSize, withBinaryFile, ) @@ -104,7 +109,9 @@ <> "the header reports whether results were truncated and how many files were scanned. " <> "A search looks at most " <> T.pack (show maxVisitedEntries) - <> " entries and stops at maxResults; then the file count is shown as 'N+'.", + <> " entries, reads at most " + <> T.pack (show (maxSearchBytes `div` (1024 * 1024))) + <> " MiB of files, and stops at maxResults; then the file count is shown as 'N+'.", ttoolReadonly = True, ttoolExecute = const (withWorkspacePaths cfg . execute cfg) } @@ -139,34 +146,49 @@ <> "-byte cap for grep." else do budget <- newWalkBudget maxVisitedEntries - st <- newIORef (GrepState [] 0 0 False) + st <- newIORef (GrepState [] 0 0 False 0 False) let search = searchFile st pat ci cap if isFile then search resolved (takeFileName resolved) else walkDir budget st search exts resolved "" 0 - GrepState matches _ scanned capHit <- readIORef st + GrepState matches _ scanned capHit bytesRead bytesOut <- readIORef st stopped <- budgetExhausted budget + let notes = + catMaybes + [ if stopped then Just (budgetNote budget) else Nothing, + if bytesOut then Just (bytesNote bytesRead) else Nothing + ] pure $ renderResults args._grepPath pat (reverse matches) scanned - ((capHit && not isFile) || stopped) + ((capHit && not isFile) || stopped || bytesOut) capHit - (if stopped then Just (budgetNote budget) else Nothing) + (if null notes then Nothing else Just (T.intercalate " | " notes)) -- | Lowercase and strip leading dots so callers can pass "hs" or ".hs". normalizeExtensions :: [Text] -> [Text] normalizeExtensions = map (T.toLower . T.dropWhile (== '.')) --- | Matches so far (newest first), their count, files searched, and --- whether the walk stopped because 'maxResults' was reached. -data GrepState = GrepState [Match] Int Int Bool +-- | Matches so far (newest first), their count, files searched, whether +-- the walk stopped because @maxResults@ was reached, the file bytes read, +-- and whether the walk stopped because 'maxSearchBytes' ran out. +data GrepState = GrepState [Match] Int Int Bool Int Bool +-- | True once the search should look at no more files. +searchDone :: GrepState -> Bool +searchDone (GrepState _ _ _ capHit _ bytesOut) = capHit || bytesOut + +-- | The note added to the output when 'maxSearchBytes' ran out. +bytesNote :: Int -> Text +bytesNote n = "stopped after reading " <> T.pack (show n) <> " bytes; narrow the path or pattern" + -- | Walk a directory and search each candidate file as it is found, -- respecting depth, hidden-file, extension and size limits and the entry --- budget. Stops once 'maxResults' is reached. Order is stable (entries +-- budget. Stops once @maxResults@ is reached or 'maxSearchBytes' runs +-- out. Order is stable (entries -- sorted) so output is deterministic. Each file is shown by its path -- relative to the search root, built from the names walked, so no host -- path is shown. @@ -188,8 +210,8 @@ allowed <- takeEntries budget (filter (not . isFileHidden) entries) mapM_ (visit dir display depth) allowed visit parent display depth name = do - GrepState _ _ _ capHit <- readIORef st - unless capHit $ do + done <- searchDone <$> readIORef st + unless done $ do let full = parent name fullDisplay = display name -- Skip symlinks to prevent traversal out of the sandbox. @@ -230,22 +252,40 @@ Left _ -> pure [] -- | Search one file and add its matches, up to the cap. Binary and --- unreadable files count as searched but give no matches. +-- unreadable files count as searched but give no matches. A text file is +-- charged to 'maxSearchBytes' before it is read; a file that does not fit +-- is not read or counted, and ends the search. searchFile :: IORef GrepState -> Text -> Bool -> Int -> FilePath -> FilePath -> IO () searchFile st pat ci cap f display = do r <- - try (withTextHandle (withBinaryFile f ReadMode) readLenient) :: - IO (Either IOException (Maybe Text)) - let ms = case r of - Right (Just content) -> matchLines display content needle ci - _ -> [] - modifyIORef' st $ \(GrepState acc n scanned _) -> - let remaining = cap - n - taken = take remaining ms - n' = n + length taken - in GrepState (reverse taken ++ acc) n' (scanned + 1) (n' >= cap) + try (withTextHandle (withBinaryFile f ReadMode) readCharged) :: + IO (Either IOException (Maybe (Maybe Text))) + case r of + Right (Just Nothing) -> pure () + _ -> do + let ms = case r of + Right (Just (Just content)) -> matchLines display content needle ci + _ -> [] + modifyIORef' st $ \(GrepState acc n scanned _ used out) -> + let remaining = cap - n + taken = take remaining ms + n' = n + length taken + in GrepState (reverse taken ++ acc) n' (scanned + 1) (n' >= cap) used out where needle = if ci then T.toLower pat else pat + -- Charge the file's size (at most the per-file cap) and read that + -- many bytes, or refuse it when the budget cannot cover it. + readCharged h = do + sz <- min maxFileSizeBytes <$> hFileSize h + let n = fromIntegral sz + fits <- charge n + if fits + then Just . decodeUtf8With lenientDecode <$> BS.hGet h n + else pure Nothing + charge n = atomicModifyIORef' st $ \(GrepState acc k scanned capHit used out) -> + if used + n <= maxSearchBytes + then (GrepState acc k scanned capHit (used + n) out, True) + else (GrepState acc k scanned capHit used True, False) data Match = Match { mPath :: FilePath, @@ -267,7 +307,7 @@ | otherwise = T.take maxLineLength t <> "…" -- | Render the results. @partial@: the walk stopped early, so the file --- count is a lower bound. @capHit@: 'maxResults' was reached. +-- count is a lower bound. @capHit@: @maxResults@ was reached. renderResults :: Text -> Text -> [Match] -> Int -> Bool -> Bool -> Maybe Text -> Text renderResults rootDisplay pat allMatches scanned partial capHit note = let (body, cut) = capLines maxOutputBytes (map fmtMatch allMatches) diff --git a/src/LLM/Tools/Internal/TextFile.hs b/src/LLM/Tools/Internal/TextFile.hs deleted file mode 100644 --- a/src/LLM/Tools/Internal/TextFile.hs +++ /dev/null @@ -1,124 +0,0 @@ --- | Text reads on one open 'Handle': the binary sniff, the size check and --- the read itself all use the same open file, instead of reopening the --- path for each step. Shared by 'LLM.Tools.FsLimits' and the tools. -module LLM.Tools.Internal.TextFile - ( Opener, - binarySniffBytes, - withTextHandle, - readBoundedBytesWith, - readBoundedTextWith, - readBoundedUtf8With, - readLenient, - writeUtf8, - ) -where - -import Control.Exception (IOException, throwIO, try) -import GHC.IO.Exception (IOException (ioe_filename)) -import Data.ByteString qualified as BS -import Data.Maybe (fromMaybe) -import Data.Text (Text) -import Data.Text qualified as T -import Data.Text.Encoding (decodeUtf8', decodeUtf8With, encodeUtf8) -import Data.Text.Encoding.Error (lenientDecode) -import System.IO (Handle, SeekMode (AbsoluteSeek), hFileSize, hSeek) - --- | Opens a file for reading and runs an action on it, e.g. --- @withBinaryFile path ReadMode@. -type Opener = forall a. (Handle -> IO a) -> IO a - --- | Number of leading bytes inspected when sniffing for binary content. --- Matches the heuristic used by @git diff@ and @grep@: any NUL byte in --- the prefix means binary. -binarySniffBytes :: Int -binarySniffBytes = 8192 - --- | Open a file, sniff its first 'binarySniffBytes' for a NUL byte, and run --- the action from the start of the file if there is none. --- --- 'Nothing' if the file looks binary, or cannot be opened or sniffed: the --- caller fails closed rather than read an unreadable file as text. An --- 'IOException' from the action itself is rethrown. -withTextHandle :: Opener -> (Handle -> IO r) -> IO (Maybe r) -withTextHandle open act = do - r <- try $ open $ \h -> do - sniff <- try (BS.hGet h binarySniffBytes) - case sniff of - Left (_ :: IOException) -> pure (Right Nothing) - Right bs - | BS.elem 0 bs -> pure (Right Nothing) - | otherwise -> do - hSeek h AbsoluteSeek 0 - fmap Just <$> try (act h) - case r of - Left (_ :: IOException) -> pure Nothing - Right (Left (e :: IOException)) -> throwIO e - Right (Right x) -> pure x - --- | Read a file up to @cap@ bytes. Refuses binary files, checks the size --- before reading, and reads at most @cap + 1@ bytes so a file that grows --- during the read is caught. -readBoundedBytesWith :: Opener -> Text -> Int -> IO (Either Text BS.ByteString) -readBoundedBytesWith open display cap = do - r <- withTextHandle open $ \h -> do - sizeRes <- try (hFileSize h) - case sizeRes of - Left (e :: IOException) -> - pure $ Left $ "could not stat " <> display <> ": " <> describeIOError e - Right sz - | sz > fromIntegral cap -> - pure $ - Left $ - display - <> " is " - <> T.pack (show sz) - <> " bytes, which exceeds the " - <> T.pack (show cap) - <> "-byte cap." - | otherwise -> do - bsRes <- try (BS.hGet h (cap + 1)) - pure $ case bsRes of - Left (e :: IOException) -> - Left $ "could not read " <> display <> ": " <> describeIOError e - Right bs - | BS.length bs > cap -> - Left $ - display - <> " grew past the " - <> T.pack (show cap) - <> "-byte cap during the read." - | otherwise -> Right bs - pure $ - fromMaybe - (Left (display <> " appears to be a binary file; only text files are supported.")) - r - --- | An 'IOException' without its file name, which for a handle is the --- absolute path it was opened with. Callers name the file themselves. -describeIOError :: IOException -> Text -describeIOError e = T.pack (show e {ioe_filename = Nothing}) - --- | 'readBoundedBytesWith', decoded for display. Invalid UTF-8 bytes become --- U+FFFD, so never write the result back. -readBoundedTextWith :: Opener -> Text -> Int -> IO (Either Text Text) -readBoundedTextWith open display cap = - fmap (decodeUtf8With lenientDecode) <$> readBoundedBytesWith open display cap - --- | 'readBoundedBytesWith', for editing. Refuses files that are not valid --- UTF-8, since writing them back would destroy their bytes. -readBoundedUtf8With :: Opener -> Text -> Int -> IO (Either Text Text) -readBoundedUtf8With open display cap = do - r <- readBoundedBytesWith open display cap - pure $ - r >>= \bs -> case decodeUtf8' bs of - Right t -> Right t - Left _ -> Left $ display <> " is not valid UTF-8; refusing to edit it." - --- | Read the rest of the handle as UTF-8, replacing invalid bytes with --- U+FFFD. Independent of the locale. -readLenient :: Handle -> IO Text -readLenient h = decodeUtf8With lenientDecode <$> BS.hGetContents h - --- | Write text as UTF-8, independent of the locale. -writeUtf8 :: Handle -> Text -> IO () -writeUtf8 h = BS.hPut h . encodeUtf8 diff --git a/src/LLM/Tools/Internal/WalkBudget.hs b/src/LLM/Tools/Internal/WalkBudget.hs --- a/src/LLM/Tools/Internal/WalkBudget.hs +++ b/src/LLM/Tools/Internal/WalkBudget.hs @@ -9,6 +9,8 @@ budgetNote, capLines, utf8Length, + utf8Prefix, + textPrefixBytes, ) where @@ -16,7 +18,7 @@ import Data.IORef (IORef, atomicModifyIORef', newIORef, readIORef) import Data.Text (Text) import Data.Text qualified as T -import Data.Text.Encoding (encodeUtf8) +import Data.Text.Encoding (decodeUtf8, encodeUtf8) -- | How many more directory entries a walk may look at, and whether it was -- denied any. @@ -60,3 +62,20 @@ -- | Size of a text in UTF-8 bytes. utf8Length :: Text -> Int utf8Length = BS.length . encodeUtf8 + +-- | At most @n@ bytes from the start, without splitting a UTF-8 sequence. +utf8Prefix :: Int -> BS.ByteString -> BS.ByteString +utf8Prefix n bs + | BS.length bs <= n = bs + | otherwise = BS.take (boundary n) bs + where + -- Back up while the byte at the cut is a continuation byte (10xxxxxx), + -- so the cut falls before the lead byte of its sequence. + boundary i + | i > 0 && BS.index bs i >= 0x80 && BS.index bs i < 0xC0 = boundary (i - 1) + | otherwise = i + +-- | At most @n@ UTF-8 bytes from the start of a text, ending on a whole +-- character. +textPrefixBytes :: Int -> Text -> Text +textPrefixBytes n = decodeUtf8 . utf8Prefix n . encodeUtf8 diff --git a/src/LLM/Tools/MultiReplaceInFile.hs b/src/LLM/Tools/MultiReplaceInFile.hs --- a/src/LLM/Tools/MultiReplaceInFile.hs +++ b/src/LLM/Tools/MultiReplaceInFile.hs @@ -1,4 +1,4 @@ -module LLM.Tools.MultiReplaceInFile (multiReplaceInFileToolTyped, MultiReplaceInFileToolArgs (..)) where +module LLM.Tools.MultiReplaceInFile (multiReplaceInFileToolTyped, MultiReplaceInFileToolArgs (..), Replacement) where import Autodocodec qualified as AC import Data.Aeson (FromJSON) @@ -11,6 +11,9 @@ import LLM.Tools.Internal.PathLock (withPathLock) import LLM.Tools.Internal.TextEdit (countOccurrences, replaceFirst) +-- | One edit of @multi_replace_in_file@: the @old@ text, which must occur +-- exactly once, and the @new@ text that replaces it. Decoded from the +-- tool's JSON arguments. data Replacement = Replacement { _repOld :: Text, _repNew :: Text diff --git a/src/LLM/Tools/ReadFilePaginated.hs b/src/LLM/Tools/ReadFilePaginated.hs --- a/src/LLM/Tools/ReadFilePaginated.hs +++ b/src/LLM/Tools/ReadFilePaginated.hs @@ -15,14 +15,14 @@ import LLM.Core.Types (TypedTool (..)) import LLM.Tools.FsConfig (FsConfig, sandboxPath, withWorkspacePaths) import LLM.Tools.FsLimits (maxOutputBytes, maxPaginatedFileBytes) -import LLM.Tools.Internal.TextFile (withTextHandle) +import LLM.Tools.Internal.TextFile (CappedRead (..), readCapped, withTextHandle) +import LLM.Tools.Internal.WalkBudget (utf8Prefix) import System.IO ( IOMode (ReadMode), - hFileSize, withBinaryFile, ) --- | Default number of lines returned per page when the caller omits 'limit'. +-- | Default number of lines returned per page when the caller omits @limit@. defaultLimit :: Int defaultLimit = 200 @@ -89,31 +89,25 @@ capText = T.pack (show maxPaginatedFileBytes) resolved <- sandboxPath cfg rel result <- withTextHandle (withBinaryFile resolved ReadMode) $ \h -> do - size <- hFileSize h - if size > maxPaginatedFileBytes - then - pure . Left $ + capped <- readCapped maxPaginatedFileBytes h + pure $ case capped of + CappedTooLarge size -> + Left $ path <> " is " <> T.pack (show size) <> " bytes, which exceeds the " <> capText <> "-byte cap for read_file_paginated." - else do - -- Read one byte past the cap, so a file that grew past it since - -- the size check is reported instead of looking like it ended. - bytes <- BS.hGet h (fromIntegral maxPaginatedFileBytes + 1) - pure $ - if fromIntegral (BS.length bytes) > maxPaginatedFileBytes - then - Left $ - path - <> " grew past the " - <> capText - <> "-byte cap for read_file_paginated during the read." - else - let (lns, hasMore) = takePage offset limit bytes - in Right (size, lns, hasMore) + CappedGrew -> + Left $ + path + <> " grew past the " + <> capText + <> "-byte cap for read_file_paginated during the read." + CappedBytes size bytes -> + let (lns, hasMore) = takePage offset limit bytes + in Right (size, lns, hasMore) pure $ case result of Nothing -> "Error: " @@ -168,18 +162,6 @@ decode = decodeUtf8With lenientDecode newline = 10 - --- | At most @n@ bytes from the start, without splitting a UTF-8 sequence. -utf8Prefix :: Int -> BS.ByteString -> BS.ByteString -utf8Prefix n bs - | BS.length bs <= n = bs - | otherwise = BS.take (boundary n) bs - where - -- Back up while the byte at the cut is a continuation byte (10xxxxxx), - -- so the cut falls before the lead byte of its sequence. - boundary i - | i > 0 && BS.index bs i >= 0x80 && BS.index bs i < 0xC0 = boundary (i - 1) - | otherwise = i renderPage :: Text -> Integer -> Int -> [Text] -> Bool -> Text renderPage path size offset lns hasMore = diff --git a/src/LLM/Tools/Readdir.hs b/src/LLM/Tools/Readdir.hs --- a/src/LLM/Tools/Readdir.hs +++ b/src/LLM/Tools/Readdir.hs @@ -62,7 +62,7 @@ pure $ T.intercalate "\n" (shown ++ note) -- | Annotate a directory entry without following symlinks. --- Symbolic links are tagged with '@' and never treated as directories, +-- Symbolic links are tagged with @\@@ and never treated as directories, -- even if they happen to point at one (possibly outside the sandbox). annotateEntry :: FilePath -> FilePath -> IO Text annotateEntry parent name = do diff --git a/test/LLM/ChatSpec.hs b/test/LLM/ChatSpec.hs --- a/test/LLM/ChatSpec.hs +++ b/test/LLM/ChatSpec.hs @@ -791,6 +791,24 @@ filter ("no_such_tool" `T.isInfixOf`) ws `shouldBe` ["[test] agTools lists no_such_tool, which is not in the tool map; it is not sent to the model"] + it "logs an invalid caller turn once per run, not once per round" $ do + reqs <- newIORef [] + (rt, logs, _) <- recordingRuntime + let -- Two tool rounds, then text: three model calls. + twoRounds req = + let toolTurns = length [() | ToolTurn _ <- req.reqConversation] + tc = mkToolCall ("call_" <> T.pack (show toolTurns)) "get_weather" (object ["location" .= ("London" :: Text)]) + in pure . Right $ + if toolTurns >= 2 + then ChatResponse "done" [textPart "done"] Nothing Nothing + else ChatResponse "" [toolCallPart tc] Nothing Nothing + history = [UserMessage [textPart "go", toolCallPart (mkToolCall "c1" "get_weather" (object []))]] + result <- generateText defaultAgent {agTools = ["get_weather"]} (ModelWithFallbacks (mockModel (recordingGateway reqs twoRounds)) []) toolMap rt history + fmap (.gtrText) result `shouldBe` Right "done" + length <$> readIORef reqs `shouldReturn` 3 + ws <- warnings <$> readIORef logs + filter ("turn " `T.isInfixOf`) ws `shouldBe` ["[test] turn 0: user messages may not contain tool-call parts"] + it "uses the built-in get_history over a caller tool of that name when windowing is on" $ do reqs <- newIORef [] (rt, logs, _) <- recordingRuntime @@ -880,6 +898,22 @@ readIORef calls `shouldReturn` ["a", "b"] ws <- (\ls -> [m | (Warn, m) <- ls, "turn " `T.isPrefixOf` m]) <$> readIORef logs ws `shouldBe` ["turn 2: user messages may not contain tool-call parts"] + + it "logs problems in the tool-call structure in turn order" $ do + logs <- newIORef [] + calls <- newIORef [] + let history = + invalidHistory + ++ [ ToolTurn [ToolResult "c9" "get_weather" "sunny"], + AssistantMessage [textPart "done"] + ] + _ <- generateTextWithFallbacks (request logs history) (ModelWithFallbacks (mockModel (loggingGateway calls "m" answer)) []) + readIORef calls `shouldReturn` ["m"] + ws <- (\ls -> [m | (Warn, m) <- ls]) <$> readIORef logs + ws + `shouldBe` [ "turn 2: user messages may not contain tool-call parts", + "turn 3: tool results do not follow an assistant turn with tool calls" + ] it "logs nothing for a valid history" $ do logs <- newIORef [] diff --git a/test/LLM/FsToolsSpec.hs b/test/LLM/FsToolsSpec.hs --- a/test/LLM/FsToolsSpec.hs +++ b/test/LLM/FsToolsSpec.hs @@ -4,7 +4,7 @@ import Control.Concurrent.MVar (newEmptyMVar, putMVar, takeMVar) import Control.Exception (IOException, bracket_, evaluate, try) import Control.Monad (forM, forM_) -import Data.Aeson (Value, object, (.=)) +import Data.Aeson (Result (..), Value, fromJSON, object, (.=)) import Data.ByteString qualified as BS import Data.Map qualified as Map import Data.Text (Text) @@ -19,7 +19,8 @@ import LLM.Generate.Logger (noHooks) import LLM.Load.FsTools (fsTools) import LLM.Tools.FsConfig (SandboxViolation (..)) -import LLM.Tools.FsLimits (maxOutputBytes, maxTreeEntries, maxVisitedEntries) +import LLM.Tools.FsLimits (maxOutputBytes, maxSearchBytes, maxTreeEntries, maxVisitedEntries) +import LLM.Tools.MultiReplaceInFile (MultiReplaceInFileToolArgs (..), Replacement) import System.Directory ( createDirectory, createDirectoryLink, @@ -218,6 +219,16 @@ readBack <- runTool tm "read_file_paginated" (object ["path" .= ("multi.txt" :: Text)]) T.unpack readBack `shouldContain` "AAA BBB" + it "exports the type of the replacements field" $ do + let args = + object + [ "path" .= ("multi.txt" :: Text), + "replacements" .= [object ["old" .= ("a" :: Text), "new" .= ("b" :: Text)]] + ] + case fromJSON args :: Result MultiReplaceInFileToolArgs of + Success decoded -> length (decoded._mrifReplacements :: [Replacement]) `shouldBe` 1 + Error err -> expectationFailure err + describe "create_directory" $ do it "creates nested directories" $ withSandbox $ \sb -> do tm <- fsTools sb @@ -463,6 +474,25 @@ result <- runTool tm "grep" (object ["pattern" .= ("needle" :: Text), "maxResults" .= (2 :: Int)]) T.unpack result `shouldContain` "2 match(es) across 2+ file(s)" T.unpack result `shouldContain` "stopped at maxResults" + + it "grep stops before the file that would exceed maxSearchBytes" $ withSandbox $ \sb -> do + tm <- fsTools sb + let mib = 1024 * 1024 + perFile = mib -- the per-file cap, so each file is searched + filesInBudget = maxSearchBytes `div` perFile + name i = sb ("t" <> replicate (3 - length (show i)) '0' <> show i <> ".txt") + withNeedle = BS.replicate (perFile - 7) 0x61 <> "needle\n" + -- A binary file is not read, so it does not count against the budget. + BS.writeFile (sb "a-binary.bin") (BS.cons 0 (BS.replicate (perFile - 1) 0x61)) + mapM_ (\i -> BS.writeFile (name i) (BS.replicate perFile 0x61)) [1 .. filesInBudget - 1] + -- The last file that fits, and one past the budget. + BS.writeFile (name filesInBudget) withNeedle + BS.writeFile (name (filesInBudget + 1)) withNeedle + result <- runTool tm "grep" (object ["pattern" .= ("needle" :: Text)]) + T.unpack result `shouldContain` ("1 match(es) across " <> show (filesInBudget + 1) <> "+ file(s)") + T.unpack result `shouldContain` ("stopped after reading " <> show maxSearchBytes <> " bytes") + T.unpack result `shouldContain` (drop (length sb + 1) (name filesInBudget) <> ":1:") + T.unpack result `shouldNotContain` drop (length sb + 1) (name (filesInBudget + 1)) it "grep reports an exact file count when it searched everything" $ withSandbox $ \sb -> do tm <- fsTools sb diff --git a/test/LLM/GeminiSpec.hs b/test/LLM/GeminiSpec.hs --- a/test/LLM/GeminiSpec.hs +++ b/test/LLM/GeminiSpec.hs @@ -102,7 +102,8 @@ let schema = object [ "type" .= ("object" :: Text), - "properties" .= object ["type" .= object ["type" .= ("string" :: Text)]] + "properties" .= object ["type" .= object ["type" .= ("string" :: Text)]], + "required" .= (["type"] :: [Text]) ] normalizeSchemaOpenAI schema `shouldBe` object diff --git a/test/LLM/GenerateObjectSpec.hs b/test/LLM/GenerateObjectSpec.hs --- a/test/LLM/GenerateObjectSpec.hs +++ b/test/LLM/GenerateObjectSpec.hs @@ -1,8 +1,9 @@ +{-# LANGUAGE TypeApplications #-} {-# OPTIONS_GHC -Wno-missing-fields #-} module LLM.GenerateObjectSpec (spec) where -import Data.Aeson (Value, object, (.=)) +import Data.Aeson (Value (Null), object, (.=)) import Data.IORef (IORef, newIORef, readIORef, writeIORef) import Data.Text (Text) import Data.Text qualified as T @@ -18,7 +19,8 @@ ModelWithFallbacks (..), defaultModelCapabilities, ) -import LLM.Generate.Types (GenerateError (..), GenerateErrorResult (..)) +import LLM.Generate.Types (GenerateError (..), GenerateErrorResult (..), GeneratableObject) +import LLM.OptionalFields (Inner (..), Node (..), Optionals (..), innerDefaults) import LLM.WeatherTool (WeatherToolArgs (..)) import Test.Hspec ( Spec, @@ -110,6 +112,62 @@ Just req -> req.reqTools `shouldBe` [] Left err -> expectationFailure $ show err + describe "generateObject with optional fields answered null (strict mode)" $ do + let nullInner = object ["a" .= Null, "b" .= Null] + answer = + object + [ "req_maybe" .= Null, + "or_null" .= Null, + "opt" .= Null, + "def" .= Null, + "omit" .= Null, + "nested" .= nullInner, + "named" .= nullInner, + "list" .= [nullInner, object ["a" .= ("x" :: Text), "b" .= (7 :: Int)]], + "union" .= object ["Left" .= nullInner], + "plain" .= ("p" :: Text) + ] + + it "reads null as absent for every optional field, in nested objects, arrays, unions and $refs" $ do + result <- runTypedWith (objectGateway answer (mkUsage 1 1)) + case result of + Right (v, _) -> + v + `shouldBe` Optionals + { reqMaybe = Nothing, + orNull = Nothing, + opt = Nothing, + withDefault = 5, + omitted = 5, + nested = Just innerDefaults, + named = Just innerDefaults, + list = [innerDefaults, Inner (Just "x") 7], + union = Left innerDefaults, + plain = "p" + } + Left err -> expectationFailure $ show err + + it "reads a null optional object as absent" $ do + let answer' = object ["req_maybe" .= Null, "nested" .= Null, "named" .= Null, "list" .= ([] :: [Value]), "union" .= object ["Right" .= (1 :: Int)], "plain" .= ("p" :: Text)] + result <- runTypedWith @Optionals (objectGateway answer' (mkUsage 1 1)) + case result of + Right (v, _) -> (v.nested, v.named, v.union) `shouldBe` (Nothing, Nothing, Right 1) + Left err -> expectationFailure $ show err + + it "still reports null for a required field that is not nullable" $ do + let answer' = object ["req_maybe" .= Null, "list" .= ([] :: [Value]), "union" .= object ["Right" .= (1 :: Int)], "plain" .= Null] + result <- runTypedWith @Optionals (objectGateway answer' (mkUsage 1 1)) + case result of + Left GenerateErrorResult {gerError = GErrParseObjectError _} -> pure () + other -> expectationFailure $ "expected GErrParseObjectError, got: " <> show (fmap fst other) + + it "reads null as absent in a recursive type" $ do + let answer' = object ["label" .= ("a" :: Text), "next" .= object ["label" .= ("b" :: Text), "next" .= Null]] + result <- runTypedWith (objectGateway answer' (mkUsage 1 1)) + case result of + Right (v, _) -> v `shouldBe` Node "a" (Just (Node "b" Nothing)) + Left err -> expectationFailure $ show err + objectGateway :: Value -> Usage -> LLMGateway objectGateway value usage = LLMGateway @@ -197,3 +255,8 @@ runTyped models = do rt <- mkRuntime Nothing generateObject defaultAgent models rt [UserTurn "weather?"] + +runTypedWith :: (GeneratableObject t) => LLMGateway -> IO (Either GenerateErrorResult (t, Usage)) +runTypedWith gw = do + rt <- mkRuntime Nothing + generateObject defaultAgent (ModelWithFallbacks (mockModel gw) []) rt [UserTurn "extract"] diff --git a/test/LLM/HistoryToolSpec.hs b/test/LLM/HistoryToolSpec.hs --- a/test/LLM/HistoryToolSpec.hs +++ b/test/LLM/HistoryToolSpec.hs @@ -4,7 +4,9 @@ import Data.Aeson (object, (.=)) import Data.Text (Text) +import Data.ByteString qualified as BS import Data.Text qualified as T +import Data.Text.Encoding (encodeUtf8) import Heptapod (generate) import LLM.Agent.ToolUtils (toTool, windowOffset) import LLM.Agent.Tools.HistoryTool (historyToolTyped) @@ -12,6 +14,7 @@ import LLM.Core.Types (Turn (..), assistantTurn, pattern UserTurn) import LLM.Generate.GenerateUtils (llmHooks) import LLM.Generate.Logger (noHooks) +import LLM.Tools.FsLimits (maxOutputBytes) import Test.Hspec ( Spec, describe, @@ -19,6 +22,7 @@ shouldBe, shouldContain, shouldNotContain, + shouldSatisfy, ) spec :: Spec @@ -72,6 +76,39 @@ T.unpack result `shouldContain` "[User] hidden question" T.unpack result `shouldContain` "[Assistant] hidden answer" + describe "get_history page size" $ do + it "leaves out the oldest turns of a page over maxOutputBytes and says how many" $ do + let answer i = assistantTurn ("answer-" <> T.pack (show i) <> " " <> T.replicate 5000 "x") Nothing [] + conv = + [UserTurn "old question"] + ++ map answer [0 .. 39 :: Int] + ++ [UserTurn "visible question"] + result <- runHistory windowedAgent conv 0 + utf8Size result `shouldSatisfy` (<= maxOutputBytes) + T.unpack result `shouldContain` "answer-39 " + T.unpack result `shouldNotContain` "answer-0 " + T.unpack result `shouldNotContain` "old question" + let keptAnswers = length (filter ("[Assistant] answer-" `T.isPrefixOf`) (T.lines result)) + last (T.lines result) + `shouldBe` "[" + <> T.pack (show (41 - keptAnswers)) + <> " earlier turn(s) of this page left out: the page is over " + <> T.pack (show maxOutputBytes) + <> " bytes]" + + it "cuts the newest turn when it alone is over maxOutputBytes" $ do + let conv = + [ UserTurn "old question", + assistantTurn ("huge " <> T.replicate 200000 "y") Nothing [], + UserTurn "visible question" + ] + result <- runHistory windowedAgent conv 0 + utf8Size result `shouldSatisfy` (<= maxOutputBytes) + T.unpack result `shouldContain` "[Assistant] huge yyy" + T.unpack result `shouldContain` "[turn cut: " + T.unpack result `shouldNotContain` "old question" + T.unpack result `shouldContain` "[1 earlier turn(s) of this page left out" + sampleConversation :: [Turn] sampleConversation = [ UserTurn "first question", @@ -118,3 +155,6 @@ } tool = toTool historyToolTyped tool.toolExecute ctx (object ["chunk" .= chunk]) + +utf8Size :: Text -> Int +utf8Size = BS.length . encodeUtf8 diff --git a/test/LLM/HttpConfigSpec.hs b/test/LLM/HttpConfigSpec.hs --- a/test/LLM/HttpConfigSpec.hs +++ b/test/LLM/HttpConfigSpec.hs @@ -5,6 +5,7 @@ import Control.Monad (forever, void) import Data.Aeson (Value, object) import Data.ByteString qualified as BS +import Data.ByteString.Char8 qualified as BC import Data.IORef (atomicModifyIORef', newIORef, readIORef) import Data.Text qualified as T import System.Timeout (timeout) @@ -67,6 +68,31 @@ Left (ParseError msg) -> T.unpack msg `shouldBe` "response body exceeds 8388608 bytes" other -> expectationFailure ("expected ParseError, got " <> take 200 (show other)) + describe "stream total size" $ do + it "turns an endless stream of small events into a ParseError" $ + withRepeatingServer "200 OK" "" (BS.concat (replicate 1000 textEvent)) $ \p -> do + r <- within $ genericStreamText (ollamaProviderWith (http "127.0.0.1") (port p)) (llmHooks noHooks) request (const (pure ())) + case r of + Left (ParseError msg) -> T.unpack msg `shouldBe` "response body exceeds 33554432 bytes" + other -> expectationFailure ("expected ParseError, got " <> take 200 (show other)) + + it "parses a stream just under the total cap" $ do + -- 31 comment lines of 1 MiB (each under the per-line cap), then a + -- short reply: about 31 MiB in all. + let padding = BS.concat (replicate 31 (": " <> BS.replicate (1024 * 1024) 0x70 <> "\n")) + body = padding <> textEvent <> finishEvent <> "data: [DONE]\n\n" + BS.length body `shouldSatisfy` (< 32 * 1024 * 1024) + withFixedServer "200 OK" body $ \p -> do + r <- within $ genericStreamText (ollamaProviderWith (http "127.0.0.1") (port p)) (llmHooks noHooks) request (const (pure ())) + case r of + Right _ -> pure () + Left err -> expectationFailure ("expected a parsed stream, got " <> take 200 (show err)) + +-- | One OpenAI-style stream event with a text delta, and the final one. +textEvent, finishEvent :: BS.ByteString +textEvent = "data: {\"choices\":[{\"index\":0,\"delta\":{\"content\":\"x\"},\"finish_reason\":null}]}\n\n" +finishEvent = "data: {\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\"}]}\n\n" + -- | Fail instead of hanging when a body is read without a bound. within :: IO a -> IO a within act = timeout 20_000_000 act >>= maybe (fail "timed out: the body was read without a bound") pure @@ -87,19 +113,34 @@ -- given status line, then the prefix, then @x@ bytes until the client -- hangs up. withEndlessServer :: BS.ByteString -> BS.ByteString -> (Int -> IO a) -> IO a -withEndlessServer status prefix action = +withEndlessServer status prefix = withRepeatingServer status prefix (BS.replicate 65536 0x78) + +-- | Like 'withEndlessServer', repeating the given chunk. +withRepeatingServer :: BS.ByteString -> BS.ByteString -> BS.ByteString -> (Int -> IO a) -> IO a +withRepeatingServer status prefix chunk = + withServer $ \conn -> do + NB.sendAll conn ("HTTP/1.1 " <> status <> "\r\nContent-Length: 1000000000000\r\n\r\n" <> prefix) + -- Stops when the client closes the connection and a send fails. + void (try @SomeException (forever (NB.sendAll conn chunk))) + +-- | A TCP server on a free local port that answers every POST with the +-- given status line and body. +withFixedServer :: BS.ByteString -> BS.ByteString -> (Int -> IO a) -> IO a +withFixedServer status body = + withServer $ \conn -> + void . try @SomeException $ + NB.sendAll conn ("HTTP/1.1 " <> status <> "\r\nContent-Length: " <> BC.pack (show (BS.length body)) <> "\r\n\r\n" <> body) + +-- | A TCP server on a free local port that runs the handler for each POST. +withServer :: (N.Socket -> IO ()) -> (Int -> IO a) -> IO a +withServer handler action = bracket open N.close $ \sock -> do p <- fromIntegral <$> N.socketPort sock tid <- forkIO $ forever $ do (conn, _) <- N.accept sock void $ forkIO $ flip finally (N.close conn) $ do chunk <- NB.recv conn 4096 - if "POST" `BS.isPrefixOf` chunk - then do - NB.sendAll conn ("HTTP/1.1 " <> status <> "\r\nContent-Length: 1000000000000\r\n\r\n" <> prefix) - -- Stops when the client closes the connection and a send fails. - void (try @SomeException (forever (NB.sendAll conn (BS.replicate 65536 0x78)))) - else pure () + if "POST" `BS.isPrefixOf` chunk then handler conn else pure () action p `finally` killThread tid where open = do diff --git a/test/LLM/LoadSpec.hs b/test/LLM/LoadSpec.hs --- a/test/LLM/LoadSpec.hs +++ b/test/LLM/LoadSpec.hs @@ -88,6 +88,16 @@ ) Left err -> expectationFailure err + -- gpt-5.6-terra refuses any temperature but the default (1): with + -- 0.5 every call failed with 400. + it "sends no temperature for the bundled gpt_5_6_terra" $ do + result <- eitherDecodeFileStrict' "./model-catalog.json" + case result of + Right (items :: [ModelCatalogItem]) -> + fmap (.temperature) (find (\i -> i.modelConfigName == "gpt_5_6_terra") items) + `shouldBe` Just Nothing + Left err -> expectationFailure err + it "loads catalogs without capabilities (defaults false)" $ do cfg <- loadModelOrThrow ollamaCatalog "mistral" cfg.mcCapabilities `shouldBe` defaultModelCapabilities diff --git a/test/LLM/OpenAISpec.hs b/test/LLM/OpenAISpec.hs --- a/test/LLM/OpenAISpec.hs +++ b/test/LLM/OpenAISpec.hs @@ -2,10 +2,13 @@ module LLM.OpenAISpec (spec) where +import Autodocodec qualified as AC +import Autodocodec.Schema (jsonSchemaVia) import Data.Aeson ( Value (Array, Object, String), eitherDecodeFileStrict', object, + toJSON, (.=), ) import Data.Aeson.Key qualified as K @@ -32,8 +35,10 @@ thinkingPart, toolCallPart, ) +import LLM.Core.ProviderUtils (normalizeSchemaOpenAI, stripBoundsAndComments) import LLM.Core.Usage (Usage (..), mkUsage) import LLM.Core.Utils (getToolCalls, hasToolCalls, isRetryable) +import LLM.OptionalFields (Node, Optionals) import LLM.Providers.OpenAI (encodeTurn, parseOpenAIResponse, parseOpenAIStream, parseOpenAIUsage) import Test.Hspec ( Spec, @@ -200,6 +205,94 @@ contentTypes parts `shouldBe` ["text", "image_url", "image_url"] mapM_ (\p -> lookupKey "cache_control" p `shouldBe` Nothing) parts + describe "normalizeSchemaOpenAI optional properties (strict mode)" $ do + let schema = strictSchema (AC.codec :: AC.JSONCodec Optionals) + prop k = schemaAt ["properties", k] schema + + it "lists every property as required" $ + schemaAt ["required"] schema + `shouldBe` Just + (toJSON (["def", "list", "named", "nested", "omit", "opt", "or_null", "plain", "req_maybe", "union"] :: [Text])) + + it "makes optionalField, optionalFieldWithDefault and optionalFieldWithOmittedDefault nullable" $ do + prop "opt" `shouldBe` Just (nullable (typed "string")) + prop "def" `shouldBe` Just (nullable (typed "integer")) + prop "omit" `shouldBe` Just (nullable (typed "integer")) + + it "makes an optional nested object nullable, inline or by $ref" $ do + fmap (const ()) (schemaAt ["properties", "nested", "anyOf", "0", "properties"] schema) `shouldBe` Just () + schemaAt ["properties", "nested", "anyOf", "1"] schema `shouldBe` Just (typed "null") + schemaAt ["properties", "named", "anyOf", "0", "$ref"] schema `shouldBe` Just "#/$defs/NamedInner" + schemaAt ["properties", "named", "anyOf", "1"] schema `shouldBe` Just (typed "null") + + it "leaves nullable and required properties as they are" $ do + let nullOrString = object ["anyOf" .= [typed "null", typed "string"]] + prop "req_maybe" `shouldBe` Just nullOrString + prop "or_null" `shouldBe` Just nullOrString + prop "plain" `shouldBe` Just (typed "string") + schemaAt ["properties", "list", "type"] schema `shouldBe` Just "array" + + it "does the same inside nested objects, array items, union branches and definitions" $ do + let inner path = do + schemaAt (path <> ["properties", "a"]) schema `shouldBe` Just (nullable (typed "string")) + schemaAt (path <> ["properties", "b"]) schema `shouldBe` Just (nullable (typed "integer")) + schemaAt (path <> ["required"]) schema `shouldBe` Just (toJSON (["a", "b"] :: [Text])) + inner ["properties", "nested", "anyOf", "0"] + inner ["properties", "list", "items"] + inner ["properties", "union", "anyOf", "0", "properties", "Left"] + inner ["$defs", "NamedInner"] + + it "turns the oneOf of an Either into anyOf" $ do + schemaAt ["properties", "union", "oneOf"] schema `shouldBe` Nothing + schemaAt ["properties", "union", "anyOf", "0", "required"] schema `shouldBe` Just (toJSON (["Left"] :: [Text])) + schemaAt ["properties", "union", "anyOf", "1", "required"] schema `shouldBe` Just (toJSON (["Right"] :: [Text])) + + it "changes only schema keywords, not property names or literal values" $ do + let oneOfSchema = object ["oneOf" .= [typed "string", typed "integer"]] + literal = object ["type" .= ("object" :: Text), "oneOf" .= [typed "string"]] + input = + object + [ "type" .= ("object" :: Text), + "properties" .= object ["oneOf" .= oneOfSchema], + "required" .= (["oneOf"] :: [Text]), + "default" .= object ["oneOf" .= literal] + ] + normalizeSchemaOpenAI input + `shouldBe` object + [ "type" .= ("object" :: Text), + "properties" .= object ["oneOf" .= object ["anyOf" .= [typed "string", typed "integer"]]], + "required" .= (["oneOf"] :: [Text]), + "default" .= object ["oneOf" .= literal], + "additionalProperties" .= False + ] + + it "moves the $defs of a named codec to the root" $ do + schemaAt ["properties", "named", "anyOf", "0"] schema + `shouldBe` Just (object ["$ref" .= ("#/$defs/NamedInner" :: Text)]) + fmap (const ()) (schemaAt ["$defs", "NamedInner", "properties"] schema) `shouldBe` Just () + + it "keeps $defs that are at the root already, and a property called $defs" $ do + let node = strictSchema (AC.codec :: AC.JSONCodec Node) + schemaAt ["$ref"] node `shouldBe` Just "#/$defs/Node" + fmap (const ()) (schemaAt ["$defs", "Node", "properties"] node) `shouldBe` Just () + let input = + object + [ "type" .= ("object" :: Text), + "properties" .= object ["$defs" .= typed "string"], + "required" .= (["$defs"] :: [Text]) + ] + normalizeSchemaOpenAI input + `shouldBe` object + [ "type" .= ("object" :: Text), + "properties" .= object ["$defs" .= typed "string"], + "required" .= (["$defs"] :: [Text]), + "additionalProperties" .= False + ] + + it "makes a recursive optional property nullable" $ + schemaAt ["$defs", "Node", "properties", "next"] (strictSchema (AC.codec :: AC.JSONCodec Node)) + `shouldBe` Just (nullable (object ["$ref" .= ("#/$defs/Node" :: Text)])) + describe "parseOpenAIUsage" $ do it "extracts token counts" $ do Right val <- eitherDecodeFileStrict' "test/fixtures/openai-text.json" @@ -245,6 +338,25 @@ _ -> expectationFailure "missing response" _ -> expectationFailure "empty conversation array" _ -> expectationFailure "expected conversation array" + +-- | The schema 'genObject' sends for a codec, as OpenAI receives it. +strictSchema :: AC.JSONCodec a -> Value +strictSchema c = normalizeSchemaOpenAI (stripBoundsAndComments (toJSON (jsonSchemaVia c))) + +-- | The sub-schema at a path of keys; a number indexes an array. +schemaAt :: [Text] -> Value -> Maybe Value +schemaAt [] v = Just v +schemaAt (k : rest) (Object o) = KM.lookup (K.fromText k) o >>= schemaAt rest +schemaAt (k : rest) (Array a) = case reads (T.unpack k) of + [(i, "")] -> a V.!? i >>= schemaAt rest + _ -> Nothing +schemaAt _ _ = Nothing + +nullable :: Value -> Value +nullable s = object ["anyOf" .= [s, object ["type" .= ("null" :: Text)]]] + +typed :: Text -> Value +typed t = object ["type" .= t] userContent :: Value -> [Value] userContent (Object o) = diff --git a/test/LLM/OptionalFields.hs b/test/LLM/OptionalFields.hs new file mode 100644 --- /dev/null +++ b/test/LLM/OptionalFields.hs @@ -0,0 +1,76 @@ +-- | A structured-output type with every kind of optional field +-- autodocodec offers, for the strict-schema tests (P3 in fixes-043). +module LLM.OptionalFields + ( Optionals (..), + Inner (..), + Node (..), + innerDefaults, + ) +where + +import Autodocodec qualified as AC +import Data.Aeson (FromJSON) +import Data.Text (Text) + +data Inner = Inner + { innerA :: Maybe Text, + innerB :: Int + } + deriving (Eq, Show) + +-- | The value of an 'Inner' whose fields were all left out. +innerDefaults :: Inner +innerDefaults = Inner Nothing 3 + +instance AC.HasCodec Inner where + codec = + AC.object "Inner" $ + Inner + <$> AC.optionalField "a" "optional" AC..= (.innerA) + <*> AC.optionalFieldWithDefault "b" 3 "with default" AC..= (.innerB) + +data Optionals = Optionals + { reqMaybe :: Maybe Text, + orNull :: Maybe Text, + opt :: Maybe Text, + withDefault :: Int, + omitted :: Int, + nested :: Maybe Inner, + named :: Maybe Inner, + list :: [Inner], + union :: Either Inner Int, + plain :: Text + } + deriving (Eq, Show) + deriving (FromJSON) via (AC.Autodocodec Optionals) + +instance AC.HasCodec Optionals where + codec = + AC.object "Optionals" $ + Optionals + <$> AC.requiredField "req_maybe" "required, Maybe" AC..= (.reqMaybe) + <*> AC.optionalFieldOrNull "or_null" "optionalFieldOrNull" AC..= (.orNull) + <*> AC.optionalField "opt" "optionalField" AC..= (.opt) + <*> AC.optionalFieldWithDefault "def" 5 "optionalFieldWithDefault" AC..= (.withDefault) + <*> AC.optionalFieldWithOmittedDefault "omit" 5 "optionalFieldWithOmittedDefault" AC..= (.omitted) + <*> AC.optionalField "nested" "nested object" AC..= (.nested) + <*> AC.optionalFieldWith "named" (AC.named "NamedInner" AC.codec) "named object ($ref)" AC..= (.named) + <*> AC.requiredField "list" "objects in an array" AC..= (.list) + <*> AC.requiredField "union" "Either (oneOf)" AC..= (.union) + <*> AC.requiredField "plain" "required, not nullable" AC..= (.plain) + +-- | A recursive type: its schema refers to itself through @$ref@. +data Node = Node + { label :: Text, + next :: Maybe Node + } + deriving (Eq, Show) + deriving (FromJSON) via (AC.Autodocodec Node) + +instance AC.HasCodec Node where + codec = + AC.named "Node" $ + AC.object "Node" $ + Node + <$> AC.requiredField "label" "label" AC..= (.label) + <*> AC.optionalField "next" "next node" AC..= (.next) diff --git a/test/LLM/TextFileSpec.hs b/test/LLM/TextFileSpec.hs new file mode 100644 --- /dev/null +++ b/test/LLM/TextFileSpec.hs @@ -0,0 +1,86 @@ +module LLM.TextFileSpec (spec) where + +import Control.Exception (throwIO) +import Data.ByteString qualified as BS +import Data.Text (Text) +import LLM.Tools.Internal.TextFile (CappedRead (..), readBoundedBytesWith, readBoundedBytesWithSize, readCapped, readCappedWith) +import System.FilePath (()) +import System.IO (Handle, IOMode (ReadMode), hFileSize, withBinaryFile) +import System.IO.Temp (withSystemTempDirectory) +import Test.Hspec (Spec, describe, it, shouldReturn) + +spec :: Spec +spec = do + readCappedSpec + readBoundedBytesSpec + +readCappedSpec :: Spec +readCappedSpec = describe "TextFile.readCapped" $ do + it "reads a file of exactly the cap" $ + withFile (BS.replicate cap 97) $ \path -> + withBinaryFile path ReadMode (readCapped (fromIntegral cap)) + `shouldReturn` CappedBytes (fromIntegral cap) (BS.replicate cap 97) + + it "refuses a file over the cap at the size check" $ + withFile (BS.replicate (cap + 1) 97) $ \path -> + withBinaryFile path ReadMode (readCapped (fromIntegral cap)) + `shouldReturn` CappedTooLarge (fromIntegral cap + 1) + + -- C23: a file within the cap at the size check that grows past it + -- before the read must be an error, not a page that ends at the cap. + it "reports a file that grows past the cap after the size check" $ + withFile (BS.replicate (cap + 2) 97) $ \path -> + withBinaryFile path ReadMode (readCappedWith (sizeBeforeGrowth 2) (fromIntegral cap)) + `shouldReturn` CappedGrew + + it "reads a file that grows but stays within the cap" $ + withFile (BS.replicate cap 97) $ \path -> + withBinaryFile path ReadMode (readCappedWith (sizeBeforeGrowth 2) (fromIntegral cap)) + `shouldReturn` CappedBytes (fromIntegral cap - 2) (BS.replicate cap 97) + where + cap = 1000 + +readBoundedBytesSpec :: Spec +readBoundedBytesSpec = describe "TextFile.readBoundedBytesWith" $ do + it "reads a file of exactly the cap" $ + withFile (BS.replicate cap 97) $ \path -> + readBoundedBytesWith (withBinaryFile path ReadMode) name cap + `shouldReturn` Right (BS.replicate cap 97) + + it "refuses a file over the cap at the size check" $ + withFile (BS.replicate (cap + 1) 97) $ \path -> + readBoundedBytesWith (withBinaryFile path ReadMode) name cap + `shouldReturn` Left "f.txt is 1001 bytes, which exceeds the 1000-byte cap." + + it "reports a file that grows past the cap after the size check" $ + withFile (BS.replicate (cap + 2) 97) $ \path -> + readBoundedBytesWithSize (sizeBeforeGrowth 2) (withBinaryFile path ReadMode) name cap + `shouldReturn` Left "f.txt grew past the 1000-byte cap during the read." + + it "reads a file that grows but stays within the cap" $ + withFile (BS.replicate cap 97) $ \path -> + readBoundedBytesWithSize (sizeBeforeGrowth 2) (withBinaryFile path ReadMode) name cap + `shouldReturn` Right (BS.replicate cap 97) + + it "names the size check when it fails" $ + withFile "text" $ \path -> + readBoundedBytesWithSize (\_ -> throwIO (userError "no size")) (withBinaryFile path ReadMode) name cap + `shouldReturn` Left "could not stat f.txt: user error (no size)" + where + cap = 1000 + name :: Text + name = "f.txt" + +-- | A size check that sees the file as it was @n@ bytes earlier: the file +-- grew by @n@ bytes between the check and the read. (Appending for real +-- from the check does not work: GHC's file locking refuses a write handle +-- while the read handle is open.) +sizeBeforeGrowth :: Integer -> Handle -> IO Integer +sizeBeforeGrowth n h = subtract n <$> hFileSize h + +withFile :: BS.ByteString -> (FilePath -> IO a) -> IO a +withFile content act = + withSystemTempDirectory "llm-simple-textfile" $ \dir -> do + let path = dir "f.txt" + BS.writeFile path content + act path diff --git a/test/LLM/TypesSpec.hs b/test/LLM/TypesSpec.hs --- a/test/LLM/TypesSpec.hs +++ b/test/LLM/TypesSpec.hs @@ -19,6 +19,8 @@ textPart, thinkingPart, toolCallPart, + ToolResult (..), + validateConversation, validateTurn, ) import LLM.Core.Usage @@ -196,6 +198,47 @@ validateTurn (AssistantMessage [imageUrlPart "https://example.com/a.png"]) `shouldBe` Left "assistant messages may not contain image parts" + + describe "validateConversation" $ do + let call i = toolCallPart (mkToolCall i "get_weather" (object [])) + result i = ToolResult i "get_weather" "sunny" + it "accepts a valid multi-round history" $ + validateConversation + [ UserTurn "weather?", + AssistantMessage [textPart "checking", call "c1", call "c2"], + ToolTurn [result "c2", result "c1"], + AssistantMessage [call "c3"], + ToolTurn [result "c3"], + AssistantMessage [textPart "sunny"], + UserTurn "thanks" + ] + `shouldBe` [] + + it "reports tool results that do not follow tool calls" $ do + validateConversation [UserTurn "hi", ToolTurn [result "c1"]] + `shouldBe` [(1, "tool results do not follow an assistant turn with tool calls")] + validateConversation [UserTurn "hi", AssistantMessage [textPart "no calls"], ToolTurn [result "c1"]] + `shouldBe` [(2, "tool results do not follow an assistant turn with tool calls")] + + it "reports tool calls without a following tool turn" $ + validateConversation [UserTurn "hi", AssistantMessage [call "c1"], UserTurn "again"] + `shouldBe` [(1, "tool calls are not followed by tool results")] + + it "reports a call without a result" $ + validateConversation [UserTurn "hi", AssistantMessage [call "c1", call "c2"], ToolTurn [result "c1"]] + `shouldBe` [(2, "no result for tool call \"c2\"")] + + it "reports a call answered twice" $ + validateConversation [UserTurn "hi", AssistantMessage [call "c1"], ToolTurn [result "c1", result "c1"]] + `shouldBe` [(2, "tool call \"c1\" is answered more than once")] + + it "reports a result for an unknown call id" $ + validateConversation [UserTurn "hi", AssistantMessage [call "c1"], ToolTurn [result "c1", result "zz"]] + `shouldBe` [(2, "tool result for unknown call id \"zz\"")] + + it "reports duplicate call ids in one assistant turn" $ + validateConversation [UserTurn "hi", AssistantMessage [call "c1", call "c1"], ToolTurn [result "c1"]] + `shouldBe` [(1, "duplicate tool call id \"c1\"")] describe "imageBase64Part" $ do it "accepts valid png base64" $ do diff --git a/test/fixtures/claude-conversation-generated.json b/test/fixtures/claude-conversation-generated.json new file mode 100644 --- /dev/null +++ b/test/fixtures/claude-conversation-generated.json @@ -0,0 +1,364 @@ +[ + { + "prompt": "how's the weather in london?", + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "how's the weather in london?", + "role": "user" + } + ], + "model": "claude-haiku-4-5-20251001", + "system": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "tools": [ + { + "description": "Get the current weather for a given location", + "input_schema": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": ["location"], + "type": "object" + }, + "name": "get_weather" + } + ] + }, + "response": { + "content": [ + { + "caller": { + "type": "direct" + }, + "id": "toolu_01DDtP6JvBHoWtxVLcVsz1JJ", + "input": { + "location": "London" + }, + "name": "get_weather", + "type": "tool_use" + } + ], + "id": "msg_01JZsXgNBYvVB4gFbTAEHFzc", + "model": "claude-haiku-4-5-20251001", + "role": "assistant", + "stop_details": null, + "stop_reason": "tool_use", + "stop_sequence": null, + "type": "message", + "usage": { + "cache_creation": { + "ephemeral_1h_input_tokens": 0, + "ephemeral_5m_input_tokens": 0 + }, + "cache_creation_input_tokens": 0, + "cache_read_input_tokens": 0, + "inference_geo": "not_available", + "input_tokens": 623, + "output_tokens": 54, + "service_tier": "standard" + } + } + }, + { + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "content": [ + { + "id": "toolu_01DDtP6JvBHoWtxVLcVsz1JJ", + "input": { + "location": "London" + }, + "name": "get_weather", + "type": "tool_use" + } + ], + "role": "assistant" + }, + { + "content": [ + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "tool_use_id": "toolu_01DDtP6JvBHoWtxVLcVsz1JJ", + "type": "tool_result" + } + ], + "role": "user" + } + ], + "model": "claude-haiku-4-5-20251001", + "system": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "tools": [ + { + "description": "Get the current weather for a given location", + "input_schema": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": ["location"], + "type": "object" + }, + "name": "get_weather" + } + ] + }, + "response": { + "content": [ + { + "text": "The weather in London is currently **partly cloudy** with a temperature of **18°C** and a **light breeze from the west**. It's pleasant weather overall!", + "type": "text" + } + ], + "id": "msg_01UXKDjyeAJFHXJXzsvVBNtE", + "model": "claude-haiku-4-5-20251001", + "role": "assistant", + "stop_details": null, + "stop_reason": "end_turn", + "stop_sequence": null, + "type": "message", + "usage": { + "cache_creation": { + "ephemeral_1h_input_tokens": 0, + "ephemeral_5m_input_tokens": 0 + }, + "cache_creation_input_tokens": 0, + "cache_read_input_tokens": 0, + "inference_geo": "not_available", + "input_tokens": 708, + "output_tokens": 40, + "service_tier": "standard" + } + } + }, + { + "prompt": "and in paris?", + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "content": [ + { + "id": "toolu_01DDtP6JvBHoWtxVLcVsz1JJ", + "input": { + "location": "London" + }, + "name": "get_weather", + "type": "tool_use" + } + ], + "role": "assistant" + }, + { + "content": [ + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "tool_use_id": "toolu_01DDtP6JvBHoWtxVLcVsz1JJ", + "type": "tool_result" + } + ], + "role": "user" + }, + { + "content": [ + { + "text": "The weather in London is currently **partly cloudy** with a temperature of **18°C** and a **light breeze from the west**. It's pleasant weather overall!", + "type": "text" + } + ], + "role": "assistant" + }, + { + "content": "and in paris?", + "role": "user" + } + ], + "model": "claude-haiku-4-5-20251001", + "system": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "tools": [ + { + "description": "Get the current weather for a given location", + "input_schema": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": ["location"], + "type": "object" + }, + "name": "get_weather" + } + ] + }, + "response": { + "content": [ + { + "caller": { + "type": "direct" + }, + "id": "toolu_01Q5J5oAGkvBUfM52fFfshJk", + "input": { + "location": "Paris" + }, + "name": "get_weather", + "type": "tool_use" + } + ], + "id": "msg_019hKACjpVyumUCpqRWHAcb8", + "model": "claude-haiku-4-5-20251001", + "role": "assistant", + "stop_details": null, + "stop_reason": "tool_use", + "stop_sequence": null, + "type": "message", + "usage": { + "cache_creation": { + "ephemeral_1h_input_tokens": 0, + "ephemeral_5m_input_tokens": 0 + }, + "cache_creation_input_tokens": 0, + "cache_read_input_tokens": 0, + "inference_geo": "not_available", + "input_tokens": 755, + "output_tokens": 54, + "service_tier": "standard" + } + } + }, + { + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "content": [ + { + "id": "toolu_01DDtP6JvBHoWtxVLcVsz1JJ", + "input": { + "location": "London" + }, + "name": "get_weather", + "type": "tool_use" + } + ], + "role": "assistant" + }, + { + "content": [ + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "tool_use_id": "toolu_01DDtP6JvBHoWtxVLcVsz1JJ", + "type": "tool_result" + } + ], + "role": "user" + }, + { + "content": [ + { + "text": "The weather in London is currently **partly cloudy** with a temperature of **18°C** and a **light breeze from the west**. It's pleasant weather overall!", + "type": "text" + } + ], + "role": "assistant" + }, + { + "content": "and in paris?", + "role": "user" + }, + { + "content": [ + { + "id": "toolu_01Q5J5oAGkvBUfM52fFfshJk", + "input": { + "location": "Paris" + }, + "name": "get_weather", + "type": "tool_use" + } + ], + "role": "assistant" + }, + { + "content": [ + { + "content": "Weather in Paris is sunny, 23°C, no wind.", + "tool_use_id": "toolu_01Q5J5oAGkvBUfM52fFfshJk", + "type": "tool_result" + } + ], + "role": "user" + } + ], + "model": "claude-haiku-4-5-20251001", + "system": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "tools": [ + { + "description": "Get the current weather for a given location", + "input_schema": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": ["location"], + "type": "object" + }, + "name": "get_weather" + } + ] + }, + "response": { + "content": [ + { + "text": "The weather in Paris is **sunny** with a temperature of **23°C** and **no wind**. It's warmer and nicer than London at the moment!", + "type": "text" + } + ], + "id": "msg_01KwfBSA7mE7KUiq6gRk19go", + "model": "claude-haiku-4-5-20251001", + "role": "assistant", + "stop_details": null, + "stop_reason": "end_turn", + "stop_sequence": null, + "type": "message", + "usage": { + "cache_creation": { + "ephemeral_1h_input_tokens": 0, + "ephemeral_5m_input_tokens": 0 + }, + "cache_creation_input_tokens": 0, + "cache_read_input_tokens": 0, + "inference_geo": "not_available", + "input_tokens": 834, + "output_tokens": 39, + "service_tier": "standard" + } + } + } +] diff --git a/test/fixtures/claude-conversation-streamed.json b/test/fixtures/claude-conversation-streamed.json new file mode 100644 --- /dev/null +++ b/test/fixtures/claude-conversation-streamed.json @@ -0,0 +1,332 @@ +[ + { + "prompt": "how's the weather in london?", + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "how's the weather in london?", + "role": "user" + } + ], + "model": "claude-haiku-4-5-20251001", + "stream": true, + "system": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "tools": [ + { + "description": "Get the current weather for a given location", + "input_schema": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + }, + "name": "get_weather" + } + ] + }, + "response": { + "content": [ + { + "arguments": { + "location": "London" + }, + "id": "toolu_01GvM5JAsPjxKLnk7qohqRSR", + "name": "get_weather", + "type": "tool_call" + } + ], + "reasoning": null, + "text": "", + "usage": { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "input_tokens": 623, + "output_tokens": 54 + } + }, + "sse": "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"model\":\"claude-haiku-4-5-20251001\",\"id\":\"msg_011CfxsaAbH28pQFPzpsK9Bc\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[],\"container\":null,\"stop_reason\":null,\"stop_sequence\":null,\"stop_details\":null,\"usage\":{\"input_tokens\":623,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":22,\"service_tier\":\"standard\",\"inference_geo\":\"not_available\"},\"diagnostics\":null} }\n\nevent: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"tool_use\",\"id\":\"toolu_01GvM5JAsPjxKLnk7qohqRSR\",\"name\":\"get_weather\",\"input\":{},\"caller\":{\"type\":\"direct\"}} }\n\nevent: ping\ndata: {\"type\": \"ping\"}\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"{\\\"locat\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"ion\\\": \\\"\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"London\\\"}\"} }\n\nevent: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0 }\n\nevent: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"tool_use\",\"stop_sequence\":null,\"stop_details\":null,\"container\":null},\"usage\":{\"input_tokens\":623,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"output_tokens\":54} }\n\nevent: message_stop\ndata: {\"type\":\"message_stop\" }\n\n" + }, + { + "prompt": null, + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "content": [ + { + "id": "toolu_01GvM5JAsPjxKLnk7qohqRSR", + "input": { + "location": "London" + }, + "name": "get_weather", + "type": "tool_use" + } + ], + "role": "assistant" + }, + { + "content": [ + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "tool_use_id": "toolu_01GvM5JAsPjxKLnk7qohqRSR", + "type": "tool_result" + } + ], + "role": "user" + } + ], + "model": "claude-haiku-4-5-20251001", + "stream": true, + "system": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "tools": [ + { + "description": "Get the current weather for a given location", + "input_schema": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + }, + "name": "get_weather" + } + ] + }, + "response": { + "content": [ + { + "text": "The weather in London is currently **partly cloudy** with a temperature of **18°C** and a **light breeze from the west**. It's a mild day, so you might want to bring a light jacket if you're heading out!", + "type": "text" + } + ], + "reasoning": null, + "text": "The weather in London is currently **partly cloudy** with a temperature of **18°C** and a **light breeze from the west**. It's a mild day, so you might want to bring a light jacket if you're heading out!", + "usage": { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "input_tokens": 708, + "output_tokens": 55 + } + }, + "sse": "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"model\":\"claude-haiku-4-5-20251001\",\"id\":\"msg_011CfxsaDxCHnwCSKMbSgBYN\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[],\"container\":null,\"stop_reason\":null,\"stop_sequence\":null,\"stop_details\":null,\"usage\":{\"input_tokens\":708,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":1,\"service_tier\":\"standard\",\"inference_geo\":\"not_available\"},\"diagnostics\":null} }\n\nevent: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"text\",\"text\":\"\"} }\n\nevent: ping\ndata: {\"type\": \"ping\"}\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"The\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" weather in London is\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" currently **\"}}\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"partly cloudy**\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" with a temperature of\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" **\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"18°C**\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" and a **\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"light breeze from\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" the west**.\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" It\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"'s a mild\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" day\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\",\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" so\"}}\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" you might\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" want to bring\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" a light jacket if\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" you're heading\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" out!\"} }\n\nevent: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0 }\n\nevent: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"end_turn\",\"stop_sequence\":null,\"stop_details\":null,\"container\":null},\"usage\":{\"input_tokens\":708,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"output_tokens\":55} }\n\nevent: message_stop\ndata: {\"type\":\"message_stop\" }\n\n" + }, + { + "prompt": "and in paris?", + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "content": [ + { + "id": "toolu_01GvM5JAsPjxKLnk7qohqRSR", + "input": { + "location": "London" + }, + "name": "get_weather", + "type": "tool_use" + } + ], + "role": "assistant" + }, + { + "content": [ + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "tool_use_id": "toolu_01GvM5JAsPjxKLnk7qohqRSR", + "type": "tool_result" + } + ], + "role": "user" + }, + { + "content": [ + { + "text": "The weather in London is currently **partly cloudy** with a temperature of **18°C** and a **light breeze from the west**. It's a mild day, so you might want to bring a light jacket if you're heading out!", + "type": "text" + } + ], + "role": "assistant" + }, + { + "content": "and in paris?", + "role": "user" + } + ], + "model": "claude-haiku-4-5-20251001", + "stream": true, + "system": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "tools": [ + { + "description": "Get the current weather for a given location", + "input_schema": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + }, + "name": "get_weather" + } + ] + }, + "response": { + "content": [ + { + "arguments": { + "location": "Paris" + }, + "id": "toolu_01Ut7JGiytus1NwfjxXuni3Z", + "name": "get_weather", + "type": "tool_call" + } + ], + "reasoning": null, + "text": "", + "usage": { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "input_tokens": 770, + "output_tokens": 54 + } + }, + "sse": "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"model\":\"claude-haiku-4-5-20251001\",\"id\":\"msg_011CfxsaHwoLbCoepii2Rhw4\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[],\"container\":null,\"stop_reason\":null,\"stop_sequence\":null,\"stop_details\":null,\"usage\":{\"input_tokens\":770,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":21,\"service_tier\":\"standard\",\"inference_geo\":\"not_available\"},\"diagnostics\":null} }\n\nevent: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"tool_use\",\"id\":\"toolu_01Ut7JGiytus1NwfjxXuni3Z\",\"name\":\"get_weather\",\"input\":{},\"caller\":{\"type\":\"direct\"}} }\n\nevent: ping\ndata: {\"type\": \"ping\"}\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"{\\\"locat\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"ion\\\": \\\"\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"input_json_delta\",\"partial_json\":\"Paris\\\"}\"} }\n\nevent: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0 }\n\nevent: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"tool_use\",\"stop_sequence\":null,\"stop_details\":null,\"container\":null},\"usage\":{\"input_tokens\":770,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"output_tokens\":54} }\n\nevent: message_stop\ndata: {\"type\":\"message_stop\" }\n\n" + }, + { + "prompt": null, + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "content": [ + { + "id": "toolu_01GvM5JAsPjxKLnk7qohqRSR", + "input": { + "location": "London" + }, + "name": "get_weather", + "type": "tool_use" + } + ], + "role": "assistant" + }, + { + "content": [ + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "tool_use_id": "toolu_01GvM5JAsPjxKLnk7qohqRSR", + "type": "tool_result" + } + ], + "role": "user" + }, + { + "content": [ + { + "text": "The weather in London is currently **partly cloudy** with a temperature of **18°C** and a **light breeze from the west**. It's a mild day, so you might want to bring a light jacket if you're heading out!", + "type": "text" + } + ], + "role": "assistant" + }, + { + "content": "and in paris?", + "role": "user" + }, + { + "content": [ + { + "id": "toolu_01Ut7JGiytus1NwfjxXuni3Z", + "input": { + "location": "Paris" + }, + "name": "get_weather", + "type": "tool_use" + } + ], + "role": "assistant" + }, + { + "content": [ + { + "content": "Weather in Paris is sunny, 23°C, no wind.", + "tool_use_id": "toolu_01Ut7JGiytus1NwfjxXuni3Z", + "type": "tool_result" + } + ], + "role": "user" + } + ], + "model": "claude-haiku-4-5-20251001", + "stream": true, + "system": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "tools": [ + { + "description": "Get the current weather for a given location", + "input_schema": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + }, + "name": "get_weather" + } + ] + }, + "response": { + "content": [ + { + "text": "The weather in Paris is **sunny** with a temperature of **23°C** and **no wind**. It's a beautiful day in Paris - warmer and sunnier than London!", + "type": "text" + } + ], + "reasoning": null, + "text": "The weather in Paris is **sunny** with a temperature of **23°C** and **no wind**. It's a beautiful day in Paris - warmer and sunnier than London!", + "usage": { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "input_tokens": 849, + "output_tokens": 42 + } + }, + "sse": "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"model\":\"claude-haiku-4-5-20251001\",\"id\":\"msg_011CfxsaM9HegK5K5qdRsJvE\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[],\"container\":null,\"stop_reason\":null,\"stop_sequence\":null,\"stop_details\":null,\"usage\":{\"input_tokens\":849,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"cache_creation\":{\"ephemeral_5m_input_tokens\":0,\"ephemeral_1h_input_tokens\":0},\"output_tokens\":4,\"service_tier\":\"standard\",\"inference_geo\":\"not_available\"},\"diagnostics\":null} }\n\nevent: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"text\",\"text\":\"\"} }\n\nevent: ping\ndata: {\"type\": \"ping\"}\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"The weather in Paris\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" is **\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"sunny** with a\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" temperature of **23\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"°C** and\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" **\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"no wind**.\"}}\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" It's a beautiful\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" day in Paris\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" -\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" war\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"mer and\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\" sunnier than London\"} }\n\nevent: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"!\"} }\n\nevent: content_block_stop\ndata: {\"type\":\"content_block_stop\",\"index\":0 }\n\nevent: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"end_turn\",\"stop_sequence\":null,\"stop_details\":null,\"container\":null},\"usage\":{\"input_tokens\":849,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"output_tokens\":42} }\n\nevent: message_stop\ndata: {\"type\":\"message_stop\" }\n\n" + } +] diff --git a/test/fixtures/claude-max-tokens-tool.sse b/test/fixtures/claude-max-tokens-tool.sse new file mode 100644 --- /dev/null +++ b/test/fixtures/claude-max-tokens-tool.sse @@ -0,0 +1,18 @@ +event: message_start +data: {"type":"message_start","message":{"id":"msg_trunc","type":"message","role":"assistant","model":"claude-haiku-4-5-20251001","content":[],"stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":10,"output_tokens":0}}} + +event: content_block_start +data: {"type":"content_block_start","index":0,"content_block":{"type":"tool_use","id":"toolu_trunc_1","name":"get_weather","input":{}}} + +event: content_block_delta +data: {"type":"content_block_delta","index":0,"delta":{"type":"input_json_delta","partial_json":"{\"location\": \"Lon"}} + +event: content_block_stop +data: {"type":"content_block_stop","index":0} + +event: message_delta +data: {"type":"message_delta","delta":{"stop_reason":"max_tokens","stop_sequence":null},"usage":{"output_tokens":16}} + +event: message_stop +data: {"type":"message_stop"} + diff --git a/test/fixtures/claude-midstream-error.sse b/test/fixtures/claude-midstream-error.sse new file mode 100644 --- /dev/null +++ b/test/fixtures/claude-midstream-error.sse @@ -0,0 +1,12 @@ +event: message_start +data: {"type":"message_start","message":{"id":"msg_err","type":"message","role":"assistant","model":"claude-haiku-4-5-20251001","content":[],"stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":10,"output_tokens":0}}} + +event: content_block_start +data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}} + +event: content_block_delta +data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Partial answer"}} + +event: error +data: {"type":"error","error":{"type":"overloaded_error","message":"Overloaded"}} + diff --git a/test/fixtures/claude-text.json b/test/fixtures/claude-text.json new file mode 100644 --- /dev/null +++ b/test/fixtures/claude-text.json @@ -0,0 +1,17 @@ +{ + "id": "msg_01XFDUDYJgAACzvnptvVoYEL", + "type": "message", + "role": "assistant", + "content": [ + { + "type": "text", + "text": "Hello! How can I help you today?" + } + ], + "model": "claude-haiku-4-5-20251001", + "stop_reason": "end_turn", + "usage": { + "input_tokens": 25, + "output_tokens": 10 + } +} diff --git a/test/fixtures/claude-thinking-tool-use.json b/test/fixtures/claude-thinking-tool-use.json new file mode 100644 --- /dev/null +++ b/test/fixtures/claude-thinking-tool-use.json @@ -0,0 +1,28 @@ +{ + "id": "msg_thinking_tool", + "type": "message", + "role": "assistant", + "model": "claude-haiku-4-5-20251001", + "content": [ + { + "type": "thinking", + "thinking": "I should call the weather tool.", + "signature": "sig_thinking_abc123" + }, + { + "type": "text", + "text": "Checking the weather." + }, + { + "type": "tool_use", + "id": "toolu_weather_1", + "name": "get_weather", + "input": { "location": "London" } + } + ], + "stop_reason": "tool_use", + "usage": { + "input_tokens": 40, + "output_tokens": 60 + } +} diff --git a/test/fixtures/claude-thinking-tool-use.sse b/test/fixtures/claude-thinking-tool-use.sse new file mode 100644 --- /dev/null +++ b/test/fixtures/claude-thinking-tool-use.sse @@ -0,0 +1,38 @@ +event: message_start +data: {"type":"message_start","message":{"id":"msg_thinking_tool","type":"message","role":"assistant","model":"claude-haiku-4-5-20251001","content":[],"stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":40,"output_tokens":0,"cache_read_input_tokens":0,"cache_creation_input_tokens":0}}} + +event: content_block_start +data: {"type":"content_block_start","index":0,"content_block":{"type":"thinking","thinking":""}} + +event: content_block_delta +data: {"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":"I should call the weather tool."}} + +event: content_block_delta +data: {"type":"content_block_delta","index":0,"delta":{"type":"signature_delta","signature":"sig_thinking_abc123"}} + +event: content_block_stop +data: {"type":"content_block_stop","index":0} + +event: content_block_start +data: {"type":"content_block_start","index":1,"content_block":{"type":"text","text":""}} + +event: content_block_delta +data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":"Checking the weather."}} + +event: content_block_stop +data: {"type":"content_block_stop","index":1} + +event: content_block_start +data: {"type":"content_block_start","index":2,"content_block":{"type":"tool_use","id":"toolu_weather_1","name":"get_weather","input":{}}} + +event: content_block_delta +data: {"type":"content_block_delta","index":2,"delta":{"type":"input_json_delta","partial_json":"{\"location\": \"London\"}"}} + +event: content_block_stop +data: {"type":"content_block_stop","index":2} + +event: message_delta +data: {"type":"message_delta","delta":{"stop_reason":"tool_use","stop_sequence":null},"usage":{"output_tokens":60}} + +event: message_stop +data: {"type":"message_stop"} diff --git a/test/fixtures/claude-tool-use.json b/test/fixtures/claude-tool-use.json new file mode 100644 --- /dev/null +++ b/test/fixtures/claude-tool-use.json @@ -0,0 +1,23 @@ +{ + "id": "msg_01Aq9w938a90dw8q", + "type": "message", + "role": "assistant", + "content": [ + { + "type": "text", + "text": "Let me check the weather for you." + }, + { + "type": "tool_use", + "id": "toolu_01A09q90qw90lq917835lq9", + "name": "get_weather", + "input": { "location": "London" } + } + ], + "model": "claude-haiku-4-5-20251001", + "stop_reason": "tool_use", + "usage": { + "input_tokens": 50, + "output_tokens": 35 + } +} diff --git a/test/fixtures/deepseek-conversation-generated.json b/test/fixtures/deepseek-conversation-generated.json new file mode 100644 --- /dev/null +++ b/test/fixtures/deepseek-conversation-generated.json @@ -0,0 +1,414 @@ +[ + { + "prompt": "how's the weather in london?", + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + } + ], + "model": "deepseek-v4-flash", + "thinking": { + "type": "enabled" + }, + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "choices": [ + { + "finish_reason": "tool_calls", + "index": 0, + "logprobs": null, + "message": { + "content": "", + "reasoning_content": "The user is asking about the weather in London. I can use the get_weather tool to answer this question.", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\": \"London\"}", + "name": "get_weather" + }, + "id": "call_00_Rx8k7TxPUlsbOoGFTqVR6618", + "index": 0, + "type": "function" + } + ] + } + } + ], + "created": 1783771054, + "id": "86881337-9981-4093-b08b-c6bb828f51ef", + "model": "deepseek-v4-flash", + "object": "chat.completion", + "system_fingerprint": "fp_8b330d02d0_prod0820_fp8_kvcache_20260402", + "usage": { + "completion_tokens": 68, + "completion_tokens_details": { + "reasoning_tokens": 23 + }, + "prompt_cache_hit_tokens": 256, + "prompt_cache_miss_tokens": 80, + "prompt_tokens": 336, + "prompt_tokens_details": { + "cached_tokens": 256 + }, + "total_tokens": 404 + } + } + }, + { + "prompt": null, + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "reasoning_content": "The user is asking about the weather in London. I can use the get_weather tool to answer this question.", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"London\"}", + "name": "get_weather" + }, + "id": "call_00_Rx8k7TxPUlsbOoGFTqVR6618", + "type": "function" + } + ] + }, + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "role": "tool", + "tool_call_id": "call_00_Rx8k7TxPUlsbOoGFTqVR6618" + } + ], + "model": "deepseek-v4-flash", + "thinking": { + "type": "enabled" + }, + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "choices": [ + { + "finish_reason": "stop", + "index": 0, + "logprobs": null, + "message": { + "content": "The weather in London is currently **partly cloudy** with a temperature of **18°C**. There's a **light breeze** coming from the west. Not too bad! 🌤️", + "reasoning_content": "I have the weather information for London. Let me share it with the user.", + "role": "assistant" + } + } + ], + "created": 1783771055, + "id": "aa2e01d5-7e2f-441d-a51e-67dba6def0f5", + "model": "deepseek-v4-flash", + "object": "chat.completion", + "system_fingerprint": "fp_8b330d02d0_prod0820_fp8_kvcache_20260402", + "usage": { + "completion_tokens": 56, + "completion_tokens_details": { + "reasoning_tokens": 16 + }, + "prompt_cache_hit_tokens": 384, + "prompt_cache_miss_tokens": 49, + "prompt_tokens": 433, + "prompt_tokens_details": { + "cached_tokens": 384 + }, + "total_tokens": 489 + } + } + }, + { + "prompt": "and in paris?", + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "reasoning_content": "The user is asking about the weather in London. I can use the get_weather tool to answer this question.", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"London\"}", + "name": "get_weather" + }, + "id": "call_00_Rx8k7TxPUlsbOoGFTqVR6618", + "type": "function" + } + ] + }, + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "role": "tool", + "tool_call_id": "call_00_Rx8k7TxPUlsbOoGFTqVR6618" + }, + { + "content": "The weather in London is currently **partly cloudy** with a temperature of **18°C**. There's a **light breeze** coming from the west. Not too bad! 🌤️", + "reasoning_content": "I have the weather information for London. Let me share it with the user.", + "role": "assistant" + }, + { + "content": "and in paris?", + "role": "user" + } + ], + "model": "deepseek-v4-flash", + "thinking": { + "type": "enabled" + }, + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "choices": [ + { + "finish_reason": "tool_calls", + "index": 0, + "logprobs": null, + "message": { + "content": "", + "reasoning_content": "The user is asking about the weather in Paris. Let me get that information.", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\": \"Paris\"}", + "name": "get_weather" + }, + "id": "call_00_1PYplGFNtKGlcxKaWG299040", + "index": 0, + "type": "function" + } + ] + } + } + ], + "created": 1783771056, + "id": "87b805ea-a0bb-4a9d-8c98-5b2136a56fae", + "model": "deepseek-v4-flash", + "object": "chat.completion", + "system_fingerprint": "fp_8b330d02d0_prod0820_fp8_kvcache_20260402", + "usage": { + "completion_tokens": 61, + "completion_tokens_details": { + "reasoning_tokens": 16 + }, + "prompt_cache_hit_tokens": 384, + "prompt_cache_miss_tokens": 114, + "prompt_tokens": 498, + "prompt_tokens_details": { + "cached_tokens": 384 + }, + "total_tokens": 559 + } + } + }, + { + "prompt": null, + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "reasoning_content": "The user is asking about the weather in London. I can use the get_weather tool to answer this question.", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"London\"}", + "name": "get_weather" + }, + "id": "call_00_Rx8k7TxPUlsbOoGFTqVR6618", + "type": "function" + } + ] + }, + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "role": "tool", + "tool_call_id": "call_00_Rx8k7TxPUlsbOoGFTqVR6618" + }, + { + "content": "The weather in London is currently **partly cloudy** with a temperature of **18°C**. There's a **light breeze** coming from the west. Not too bad! 🌤️", + "reasoning_content": "I have the weather information for London. Let me share it with the user.", + "role": "assistant" + }, + { + "content": "and in paris?", + "role": "user" + }, + { + "reasoning_content": "The user is asking about the weather in Paris. Let me get that information.", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"Paris\"}", + "name": "get_weather" + }, + "id": "call_00_1PYplGFNtKGlcxKaWG299040", + "type": "function" + } + ] + }, + { + "content": "Weather in Paris is sunny, 23°C, no wind.", + "role": "tool", + "tool_call_id": "call_00_1PYplGFNtKGlcxKaWG299040" + } + ], + "model": "deepseek-v4-flash", + "thinking": { + "type": "enabled" + }, + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "choices": [ + { + "finish_reason": "stop", + "index": 0, + "logprobs": null, + "message": { + "content": "The weather in Paris is **sunny** and **23°C** with no wind. ☀️ It's a bit warmer than London and a lovely day there!", + "reasoning_content": "Let me share the Paris weather info with the user.", + "role": "assistant" + } + } + ], + "created": 1783771057, + "id": "a40d09ab-3803-4018-b25b-b92f7938c3cb", + "model": "deepseek-v4-flash", + "object": "chat.completion", + "system_fingerprint": "fp_8b330d02d0_prod0820_fp8_kvcache_20260402", + "usage": { + "completion_tokens": 47, + "completion_tokens_details": { + "reasoning_tokens": 11 + }, + "prompt_cache_hit_tokens": 512, + "prompt_cache_miss_tokens": 72, + "prompt_tokens": 584, + "prompt_tokens_details": { + "cached_tokens": 512 + }, + "total_tokens": 631 + } + } + } +] diff --git a/test/fixtures/deepseek-conversation-streamed.json b/test/fixtures/deepseek-conversation-streamed.json new file mode 100644 --- /dev/null +++ b/test/fixtures/deepseek-conversation-streamed.json @@ -0,0 +1,367 @@ +[ + { + "prompt": "how's the weather in london?", + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + } + ], + "model": "deepseek-v4-flash", + "stream": true, + "stream_options": { + "include_usage": true + }, + "thinking": { + "type": "enabled" + }, + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "content": [ + { + "text": "I'll check the current weather in London for you.", + "type": "text" + }, + { + "arguments": { + "location": "London" + }, + "id": "call_00_ET_untwD9piFDtwetSJEIiB1629", + "name": "get_weather", + "type": "tool_call" + } + ], + "reasoning": null, + "text": "I'll check the current weather in London for you.", + "usage": { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "input_tokens": 352, + "output_tokens": 49 + } + }, + "sse": "data: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":null,\"reasoning_content\":\"\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"I\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"'ll\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" check\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" the\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" current\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" weather\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" in\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" London\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" for\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" you\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\".\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"id\":\"call_00_ET_untwD9piFDtwetSJEIiB1629\",\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"arguments\":\"\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"{\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\\\"\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"location\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\\\"\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\": \"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\\\"\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"London\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\\\"\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"}\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"bf126b7a-907f-4430-9507-c4d94db6891a\",\"object\":\"chat.completion.chunk\",\"created\":1791831612,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":\"tool_calls\"}],\"usage\":{\"prompt_tokens\":352,\"completion_tokens\":49,\"total_tokens\":401,\"prompt_tokens_details\":{\"cached_tokens\":0},\"completion_tokens_details\":{\"reasoning_tokens\":0},\"prompt_cache_hit_tokens\":0,\"prompt_cache_miss_tokens\":352}}\n\ndata: [DONE]\n\n" + }, + { + "prompt": null, + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "content": "I'll check the current weather in London for you.", + "reasoning_content": "", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"London\"}", + "name": "get_weather" + }, + "id": "call_00_ET_untwD9piFDtwetSJEIiB1629", + "type": "function" + } + ] + }, + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "role": "tool", + "tool_call_id": "call_00_ET_untwD9piFDtwetSJEIiB1629" + } + ], + "model": "deepseek-v4-flash", + "stream": true, + "stream_options": { + "include_usage": true + }, + "thinking": { + "type": "enabled" + }, + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "content": [ + { + "text": "Here's the current weather in London:\n\n- **Conditions:** Partly cloudy\n- **Temperature:** 18°C (about 64°F)\n- **Wind:** Light breeze from the west\n\nFairly mild and pleasant — a good day to be out and about, though you might want a light layer just in case. Let me know if you'd like the forecast for another city!", + "type": "text" + } + ], + "reasoning": null, + "text": "Here's the current weather in London:\n\n- **Conditions:** Partly cloudy\n- **Temperature:** 18°C (about 64°F)\n- **Wind:** Light breeze from the west\n\nFairly mild and pleasant — a good day to be out and about, though you might want a light layer just in case. Let me know if you'd like the forecast for another city!", + "usage": { + "cache_creation_tokens": 0, + "cache_read_tokens": 256, + "input_tokens": 430, + "output_tokens": 81 + } + }, + "sse": "data: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":null,\"reasoning_content\":\"\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Here\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"'s\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" the\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" current\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" weather\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" in\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" London\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\":\\n\\n\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"-\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" **\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Conditions\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\":**\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" Part\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"ly\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" cloudy\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"\\n\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"-\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" **\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Temperature\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\":**\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" \",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"18\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"°\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"C\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" (\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"about\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" \",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"64\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"°\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"F\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\")\\n\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"-\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" **\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Wind\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\":**\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" Light\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" breeze\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" from\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" the\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" west\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"\\n\\n\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Fair\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"ly\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" mild\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" and\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" pleasant\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" —\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" a\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" good\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" day\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" to\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" be\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" out\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" and\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" about\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\",\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" though\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" you\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" might\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" want\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" a\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" light\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" layer\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" just\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" in\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" case\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\".\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" Let\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" me\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" know\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" if\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" you\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"'d\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" like\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" the\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" forecast\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" for\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" another\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" city\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"!\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"07fdc257-1774-4c36-a6ff-fcf2de1d950c\",\"object\":\"chat.completion.chunk\",\"created\":1791831613,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":430,\"completion_tokens\":81,\"total_tokens\":511,\"prompt_tokens_details\":{\"cached_tokens\":256},\"completion_tokens_details\":{\"reasoning_tokens\":0},\"prompt_cache_hit_tokens\":256,\"prompt_cache_miss_tokens\":174}}\n\ndata: [DONE]\n\n" + }, + { + "prompt": "and in paris?", + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "content": "I'll check the current weather in London for you.", + "reasoning_content": "", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"London\"}", + "name": "get_weather" + }, + "id": "call_00_ET_untwD9piFDtwetSJEIiB1629", + "type": "function" + } + ] + }, + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "role": "tool", + "tool_call_id": "call_00_ET_untwD9piFDtwetSJEIiB1629" + }, + { + "content": "Here's the current weather in London:\n\n- **Conditions:** Partly cloudy\n- **Temperature:** 18°C (about 64°F)\n- **Wind:** Light breeze from the west\n\nFairly mild and pleasant — a good day to be out and about, though you might want a light layer just in case. Let me know if you'd like the forecast for another city!", + "reasoning_content": "", + "role": "assistant" + }, + { + "content": "and in paris?", + "role": "user" + } + ], + "model": "deepseek-v4-flash", + "stream": true, + "stream_options": { + "include_usage": true + }, + "thinking": { + "type": "enabled" + }, + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "content": [ + { + "text": "The user wants weather in Paris. Call the tool.", + "type": "thinking" + }, + { + "arguments": { + "location": "Paris" + }, + "id": "call_00_e8BJvsWyP3oJO3v2tKQx8381", + "name": "get_weather", + "type": "tool_call" + } + ], + "reasoning": "The user wants weather in Paris. Call the tool.", + "text": "", + "usage": { + "cache_creation_tokens": 0, + "cache_read_tokens": 384, + "input_tokens": 520, + "output_tokens": 50 + } + }, + "sse": "data: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":null,\"reasoning_content\":\"\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":null,\"reasoning_content\":\"The\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":null,\"reasoning_content\":\" user\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":null,\"reasoning_content\":\" wants\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":null,\"reasoning_content\":\" weather\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":null,\"reasoning_content\":\" in\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":null,\"reasoning_content\":\" Paris\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":null,\"reasoning_content\":\".\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":null,\"reasoning_content\":\" Call\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":null,\"reasoning_content\":\" the\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":null,\"reasoning_content\":\" tool\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":null,\"reasoning_content\":\".\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"id\":\"call_00_e8BJvsWyP3oJO3v2tKQx8381\",\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"arguments\":\"\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"{\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\\\"\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"location\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\\\"\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\": \"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\\\"\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"Paris\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\\\"\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"}\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"d68e6d85-0ef2-44d4-bfe2-5955338a7ad9\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":\"tool_calls\"}],\"usage\":{\"prompt_tokens\":520,\"completion_tokens\":50,\"total_tokens\":570,\"prompt_tokens_details\":{\"cached_tokens\":384},\"completion_tokens_details\":{\"reasoning_tokens\":11},\"prompt_cache_hit_tokens\":384,\"prompt_cache_miss_tokens\":136}}\n\ndata: [DONE]\n\n" + }, + { + "prompt": null, + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "content": "I'll check the current weather in London for you.", + "reasoning_content": "", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"London\"}", + "name": "get_weather" + }, + "id": "call_00_ET_untwD9piFDtwetSJEIiB1629", + "type": "function" + } + ] + }, + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "role": "tool", + "tool_call_id": "call_00_ET_untwD9piFDtwetSJEIiB1629" + }, + { + "content": "Here's the current weather in London:\n\n- **Conditions:** Partly cloudy\n- **Temperature:** 18°C (about 64°F)\n- **Wind:** Light breeze from the west\n\nFairly mild and pleasant — a good day to be out and about, though you might want a light layer just in case. Let me know if you'd like the forecast for another city!", + "reasoning_content": "", + "role": "assistant" + }, + { + "content": "and in paris?", + "role": "user" + }, + { + "reasoning_content": "The user wants weather in Paris. Call the tool.", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"Paris\"}", + "name": "get_weather" + }, + "id": "call_00_e8BJvsWyP3oJO3v2tKQx8381", + "type": "function" + } + ] + }, + { + "content": "Weather in Paris is sunny, 23°C, no wind.", + "role": "tool", + "tool_call_id": "call_00_e8BJvsWyP3oJO3v2tKQx8381" + } + ], + "model": "deepseek-v4-flash", + "stream": true, + "stream_options": { + "include_usage": true + }, + "thinking": { + "type": "enabled" + }, + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "content": [ + { + "text": "Here's the current weather in Paris:\n\n- **Conditions:** Sunny\n- **Temperature:** 23°C (about 73°F)\n- **Wind:** None\n\nParis is warmer and brighter than London right now — a great day for a walk along the Seine or a café terrace. Anything else you'd like to check?", + "type": "text" + } + ], + "reasoning": null, + "text": "Here's the current weather in Paris:\n\n- **Conditions:** Sunny\n- **Temperature:** 23°C (about 73°F)\n- **Wind:** None\n\nParis is warmer and brighter than London right now — a great day for a walk along the Seine or a café terrace. Anything else you'd like to check?", + "usage": { + "cache_creation_tokens": 0, + "cache_read_tokens": 384, + "input_tokens": 595, + "output_tokens": 68 + } + }, + "sse": "data: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":null,\"reasoning_content\":\"\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Here\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"'s\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" the\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" current\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" weather\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" in\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" Paris\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\":\\n\\n\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"-\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" **\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Conditions\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\":**\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" Sunny\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"\\n\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"-\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" **\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Temperature\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\":**\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" \",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"23\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"°\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"C\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" (\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"about\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" \",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"73\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"°\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"F\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\")\\n\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"-\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" **\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Wind\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\":**\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" None\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"\\n\\n\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Paris\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" is\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" warmer\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" and\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" brighter\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" than\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" London\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" right\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" now\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" —\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" a\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" great\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" day\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" for\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" a\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" walk\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" along\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" the\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" Seine\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" or\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" a\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" café\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" terrace\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\".\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" Anything\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" else\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" you\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"'d\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" like\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" to\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" check\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"?\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null}\n\ndata: {\"id\":\"3b6fbf1c-520b-4047-bc01-db9637f3b8fd\",\"object\":\"chat.completion.chunk\",\"created\":1791831615,\"model\":\"deepseek-flash\",\"system_fingerprint\":\"aeb56401ca74e127821c4f9126dcb669\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"\",\"reasoning_content\":null},\"logprobs\":null,\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":595,\"completion_tokens\":68,\"total_tokens\":663,\"prompt_tokens_details\":{\"cached_tokens\":384},\"completion_tokens_details\":{\"reasoning_tokens\":0},\"prompt_cache_hit_tokens\":384,\"prompt_cache_miss_tokens\":211}}\n\ndata: [DONE]\n\n" + } +] diff --git a/test/fixtures/gemini-conversation-generated.json b/test/fixtures/gemini-conversation-generated.json new file mode 100644 --- /dev/null +++ b/test/fixtures/gemini-conversation-generated.json @@ -0,0 +1,466 @@ +[ + { + "prompt": "how's the weather in london?", + "request": { + "_model": "gemini-2.5-flash", + "contents": [ + { + "parts": [ + { + "text": "how's the weather in london?" + } + ], + "role": "user" + } + ], + "generationConfig": { + "maxOutputTokens": 1024 + }, + "system_instruction": { + "parts": [ + { + "text": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available." + } + ] + }, + "tools": [ + { + "function_declarations": [ + { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "properties": { + "location": { + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + } + ] + } + ] + }, + "response": { + "candidates": [ + { + "content": { + "parts": [ + { + "functionCall": { + "args": { + "location": "london" + }, + "name": "get_weather" + }, + "thoughtSignature": "CrYCAQw51sfPxRG9vnimuR22EaIvtoyyucxEJBQJ/X4Psewnt2wdz1be27Atf9YdHnB6/PBdAbq5RCk1Vh+08JCYgPOVpXLk9K7Bsou6wpFSyb3kyw7XWFQtw1R6hU5Igo3j/Fx+1R6qWkohByimQSOOFLpTZSsf/op6GzkeCC4vjamvIylc08JpA9/eBmJmt+54uU7JSaIzMVMHTs7Gyy8myV1cuJ4vLozakJf0ZwgC211Bcj4/YECr/z2B+mwGOtRlrNp6x1h/xKB2zbYlfqxi8UBG2VKCVPnFFJhlRwsGtoCyFtPHiyBYAp/g5Oa1g1TH1htteqR7DA8D2plfzcVpz1YP6YArNkCDuIX9Fo8diuDF558XLebgD3tibSLtoNP+mYR7Mz9Grelzx01YtCSlxKI6Y6auZQ==" + } + ], + "role": "model" + }, + "finishMessage": "Model generated function call(s).", + "finishReason": "STOP", + "index": 0 + } + ], + "modelVersion": "gemini-2.5-flash", + "responseId": "MQf_aamWB5qDkdUPxoCDuAg", + "usageMetadata": { + "candidatesTokenCount": 15, + "promptTokenCount": 80, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 80 + } + ], + "serviceTier": "standard", + "thoughtsTokenCount": 67, + "totalTokenCount": 162 + } + } + }, + { + "request": { + "_model": "gemini-2.5-flash", + "contents": [ + { + "parts": [ + { + "text": "how's the weather in london?" + } + ], + "role": "user" + }, + { + "parts": [ + { + "functionCall": { + "args": { + "location": "london" + }, + "name": "get_weather" + }, + "thoughtSignature": "CrYCAQw51sfPxRG9vnimuR22EaIvtoyyucxEJBQJ/X4Psewnt2wdz1be27Atf9YdHnB6/PBdAbq5RCk1Vh+08JCYgPOVpXLk9K7Bsou6wpFSyb3kyw7XWFQtw1R6hU5Igo3j/Fx+1R6qWkohByimQSOOFLpTZSsf/op6GzkeCC4vjamvIylc08JpA9/eBmJmt+54uU7JSaIzMVMHTs7Gyy8myV1cuJ4vLozakJf0ZwgC211Bcj4/YECr/z2B+mwGOtRlrNp6x1h/xKB2zbYlfqxi8UBG2VKCVPnFFJhlRwsGtoCyFtPHiyBYAp/g5Oa1g1TH1htteqR7DA8D2plfzcVpz1YP6YArNkCDuIX9Fo8diuDF558XLebgD3tibSLtoNP+mYR7Mz9Grelzx01YtCSlxKI6Y6auZQ==" + } + ], + "role": "model" + }, + { + "parts": [ + { + "functionResponse": { + "name": "get_weather", + "response": { + "result": "Weather in London is partly cloudy, 18°C, light breeze from the west." + } + } + } + ], + "role": "user" + } + ], + "generationConfig": { + "maxOutputTokens": 1024 + }, + "system_instruction": { + "parts": [ + { + "text": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available." + } + ] + }, + "tools": [ + { + "function_declarations": [ + { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "properties": { + "location": { + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + } + ] + } + ] + }, + "response": { + "candidates": [ + { + "content": { + "parts": [ + { + "text": "The weather in London is partly cloudy, 18°C, with a light breeze from the west. \n" + } + ], + "role": "model" + }, + "finishReason": "STOP", + "index": 0 + } + ], + "modelVersion": "gemini-2.5-flash", + "responseId": "Mgf_adrRNMW0nsEP0vmmiQ4", + "usageMetadata": { + "candidatesTokenCount": 22, + "promptTokenCount": 128, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 128 + } + ], + "serviceTier": "standard", + "totalTokenCount": 150 + } + } + }, + { + "prompt": "and in paris?", + "request": { + "_model": "gemini-2.5-flash", + "contents": [ + { + "parts": [ + { + "text": "how's the weather in london?" + } + ], + "role": "user" + }, + { + "parts": [ + { + "functionCall": { + "args": { + "location": "london" + }, + "name": "get_weather" + }, + "thoughtSignature": "CrYCAQw51sfPxRG9vnimuR22EaIvtoyyucxEJBQJ/X4Psewnt2wdz1be27Atf9YdHnB6/PBdAbq5RCk1Vh+08JCYgPOVpXLk9K7Bsou6wpFSyb3kyw7XWFQtw1R6hU5Igo3j/Fx+1R6qWkohByimQSOOFLpTZSsf/op6GzkeCC4vjamvIylc08JpA9/eBmJmt+54uU7JSaIzMVMHTs7Gyy8myV1cuJ4vLozakJf0ZwgC211Bcj4/YECr/z2B+mwGOtRlrNp6x1h/xKB2zbYlfqxi8UBG2VKCVPnFFJhlRwsGtoCyFtPHiyBYAp/g5Oa1g1TH1htteqR7DA8D2plfzcVpz1YP6YArNkCDuIX9Fo8diuDF558XLebgD3tibSLtoNP+mYR7Mz9Grelzx01YtCSlxKI6Y6auZQ==" + } + ], + "role": "model" + }, + { + "parts": [ + { + "functionResponse": { + "name": "get_weather", + "response": { + "result": "Weather in London is partly cloudy, 18°C, light breeze from the west." + } + } + } + ], + "role": "user" + }, + { + "parts": [ + { + "text": "The weather in London is partly cloudy, 18°C, with a light breeze from the west. \n" + } + ], + "role": "model" + }, + { + "parts": [ + { + "text": "and in paris?" + } + ], + "role": "user" + } + ], + "generationConfig": { + "maxOutputTokens": 1024 + }, + "system_instruction": { + "parts": [ + { + "text": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available." + } + ] + }, + "tools": [ + { + "function_declarations": [ + { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "properties": { + "location": { + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + } + ] + } + ] + }, + "response": { + "candidates": [ + { + "content": { + "parts": [ + { + "functionCall": { + "args": { + "location": "paris" + }, + "name": "get_weather" + }, + "thoughtSignature": "CrQBAQw51sevrdIM6Bou7D32/R36TgB5hj/7FN/LL7N1gwPXNi8IJ4I8iTtpuFuP+nroxiSvtr35+QmtIQvPMcaEZZJHYC7shC9rSa2S2xzqQMrlU71f2I7guOxzBvT62Z/RW/M1423X5dcLBKBkJz+n0M219tR/6eJepndHtJLP2nsiPDlhTS4/QulhjP3iMN1GzEg20HhsY1Ri6GMS3BFyu+udYByLXiaCwaNyyquNwdApTNml" + } + ], + "role": "model" + }, + "finishMessage": "Model generated function call(s).", + "finishReason": "STOP", + "index": 0 + } + ], + "modelVersion": "gemini-2.5-flash", + "responseId": "NAf_aaO0FsTknsEP3MvJqA0", + "usageMetadata": { + "candidatesTokenCount": 15, + "promptTokenCount": 158, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 158 + } + ], + "serviceTier": "standard", + "thoughtsTokenCount": 36, + "totalTokenCount": 209 + } + } + }, + { + "request": { + "_model": "gemini-2.5-flash", + "contents": [ + { + "parts": [ + { + "text": "how's the weather in london?" + } + ], + "role": "user" + }, + { + "parts": [ + { + "functionCall": { + "args": { + "location": "london" + }, + "name": "get_weather" + }, + "thoughtSignature": "CrYCAQw51sfPxRG9vnimuR22EaIvtoyyucxEJBQJ/X4Psewnt2wdz1be27Atf9YdHnB6/PBdAbq5RCk1Vh+08JCYgPOVpXLk9K7Bsou6wpFSyb3kyw7XWFQtw1R6hU5Igo3j/Fx+1R6qWkohByimQSOOFLpTZSsf/op6GzkeCC4vjamvIylc08JpA9/eBmJmt+54uU7JSaIzMVMHTs7Gyy8myV1cuJ4vLozakJf0ZwgC211Bcj4/YECr/z2B+mwGOtRlrNp6x1h/xKB2zbYlfqxi8UBG2VKCVPnFFJhlRwsGtoCyFtPHiyBYAp/g5Oa1g1TH1htteqR7DA8D2plfzcVpz1YP6YArNkCDuIX9Fo8diuDF558XLebgD3tibSLtoNP+mYR7Mz9Grelzx01YtCSlxKI6Y6auZQ==" + } + ], + "role": "model" + }, + { + "parts": [ + { + "functionResponse": { + "name": "get_weather", + "response": { + "result": "Weather in London is partly cloudy, 18°C, light breeze from the west." + } + } + } + ], + "role": "user" + }, + { + "parts": [ + { + "text": "The weather in London is partly cloudy, 18°C, with a light breeze from the west. \n" + } + ], + "role": "model" + }, + { + "parts": [ + { + "text": "and in paris?" + } + ], + "role": "user" + }, + { + "parts": [ + { + "functionCall": { + "args": { + "location": "paris" + }, + "name": "get_weather" + }, + "thoughtSignature": "CrQBAQw51sevrdIM6Bou7D32/R36TgB5hj/7FN/LL7N1gwPXNi8IJ4I8iTtpuFuP+nroxiSvtr35+QmtIQvPMcaEZZJHYC7shC9rSa2S2xzqQMrlU71f2I7guOxzBvT62Z/RW/M1423X5dcLBKBkJz+n0M219tR/6eJepndHtJLP2nsiPDlhTS4/QulhjP3iMN1GzEg20HhsY1Ri6GMS3BFyu+udYByLXiaCwaNyyquNwdApTNml" + } + ], + "role": "model" + }, + { + "parts": [ + { + "functionResponse": { + "name": "get_weather", + "response": { + "result": "Weather in Paris is sunny, 23°C, no wind." + } + } + } + ], + "role": "user" + } + ], + "generationConfig": { + "maxOutputTokens": 1024 + }, + "system_instruction": { + "parts": [ + { + "text": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available." + } + ] + }, + "tools": [ + { + "function_declarations": [ + { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "properties": { + "location": { + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + } + ] + } + ] + }, + "response": { + "candidates": [ + { + "content": { + "parts": [ + { + "text": "The weather in Paris is sunny, 23°C, with no wind.", + "thoughtSignature": "CqgCAQw51seXOw7QjuKe6rjxSd4E2U2OFa1doN8TM5udn1WvUSWsA1vIVh7IoJf5CcegjMFjQMv2lpofnTtuSAWUwJcQpBbL9VR629g50rsReSvUqWXVTxahct+x/VthG9sP0VG2P2dxg037Ydzz7v1pBN4tXJUBM0qlA/Up3wG2mlhZf7cIQesxPoqBXww3t/XmUzvbjZmVlzCns6/gKPGcEVGkhu6P4F7jjf5V4MomL41paTZ4OwrK871YFKBzpWCbgzUnUBAMPWutw4D6DEJGhS6DcW6r/QqI0fvHXKTeGPkuU0iw0lje0+6i0sBwLn1qcej9yKjh4eJl1UMb1SNGnUmS1SPx16+NKkrcLrKRWLlZ31SvfEdrxyMbOTB89ware1RExK9E8hQ=" + } + ], + "role": "model" + }, + "finishReason": "STOP", + "index": 0 + } + ], + "modelVersion": "gemini-2.5-flash", + "responseId": "Ngf_ac2NDPPknsEPo5al8Q8", + "usageMetadata": { + "candidatesTokenCount": 17, + "promptTokenCount": 202, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 202 + } + ], + "serviceTier": "standard", + "thoughtsTokenCount": 60, + "totalTokenCount": 279 + } + } + } +] diff --git a/test/fixtures/gemini-conversation-streamed.json b/test/fixtures/gemini-conversation-streamed.json new file mode 100644 --- /dev/null +++ b/test/fixtures/gemini-conversation-streamed.json @@ -0,0 +1,420 @@ +[ + { + "prompt": "how's the weather in london?", + "request": { + "_model": "gemini-2.5-flash", + "contents": [ + { + "parts": [ + { + "text": "how's the weather in london?" + } + ], + "role": "user" + } + ], + "generationConfig": { + "maxOutputTokens": 1024 + }, + "system_instruction": { + "parts": [ + { + "text": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available." + } + ] + }, + "tools": [ + { + "function_declarations": [ + { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "properties": { + "location": { + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + } + ] + } + ] + }, + "response": { + "content": [ + { + "arguments": { + "location": "london" + }, + "id": "call_3", + "name": "get_weather", + "provider_meta": { + "model": "gemini-2.5-flash", + "payload": { + "thoughtSignature": "CiQBaRR9E1bQKFDb1IdCRw/EwdzLL8d3nA9HW1uaE6faGtp/yJkKbgFpFH0THbbSxOnGPDEv9ClVdoqXwmLCFArKL60PLMXEWPW7sNdcYw8NpJ+AYt2PrSUqTUa3/CycUwxkaatMAZH+XwtkGk9fWK82f1kIyp/849p37fZlAGNVYetP7OoHqwzVtMfSVS92pTGO7Zy7CskBAWkUfRMyfjq19zfhRokIcvIu7DY+jDwoabfNpANlgtBuXPnDnKlCy2kvjMd9lQCAF9UC4Awi9r9NClqa7D4W5tbKdNswYMlvXfTbcRRcuSJODFQKgLqXtLAS0NLLs1j96KjQrGJKWeGHvUxiQ1NTIGnOk0DimDdsZ9sSm94RYNkPmCsQSbOZJH5/8GO20HhJA90WsqRtufz8KFenR9b+N1yBpEwGVNdzRWvQYNQ40z14yJ8ue0ZHTJFCGL1/k1Rcq798v6ZhKdHR" + }, + "provider": "gemini" + }, + "type": "tool_call" + } + ], + "reasoning": null, + "text": "", + "usage": { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "input_tokens": 80, + "output_tokens": 15 + } + }, + "sse": "data: {\"candidates\": [{\"content\": {\"parts\": [{\"functionCall\": {\"name\": \"get_weather\",\"args\": {\"location\": \"london\"}},\"thoughtSignature\": \"CiQBaRR9E1bQKFDb1IdCRw/EwdzLL8d3nA9HW1uaE6faGtp/yJkKbgFpFH0THbbSxOnGPDEv9ClVdoqXwmLCFArKL60PLMXEWPW7sNdcYw8NpJ+AYt2PrSUqTUa3/CycUwxkaatMAZH+XwtkGk9fWK82f1kIyp/849p37fZlAGNVYetP7OoHqwzVtMfSVS92pTGO7Zy7CskBAWkUfRMyfjq19zfhRokIcvIu7DY+jDwoabfNpANlgtBuXPnDnKlCy2kvjMd9lQCAF9UC4Awi9r9NClqa7D4W5tbKdNswYMlvXfTbcRRcuSJODFQKgLqXtLAS0NLLs1j96KjQrGJKWeGHvUxiQ1NTIGnOk0DimDdsZ9sSm94RYNkPmCsQSbOZJH5/8GO20HhJA90WsqRtufz8KFenR9b+N1yBpEwGVNdzRWvQYNQ40z14yJ8ue0ZHTJFCGL1/k1Rcq798v6ZhKdHR\"}],\"role\": \"model\"},\"finishReason\": \"STOP\",\"index\": 0,\"finishMessage\": \"Model generated function call(s).\"}],\"usageMetadata\": {\"promptTokenCount\": 80,\"candidatesTokenCount\": 15,\"totalTokenCount\": 154,\"promptTokensDetails\": [{\"modality\": \"TEXT\",\"tokenCount\": 80}],\"thoughtsTokenCount\": 59,\"serviceTier\": \"standard\"},\"modelVersion\": \"gemini-2.5-flash\",\"responseId\": \"OS7Navm-Dv7_vdIPqLDGgAk\"}\r\n\r\n" + }, + { + "prompt": null, + "request": { + "_model": "gemini-2.5-flash", + "contents": [ + { + "parts": [ + { + "text": "how's the weather in london?" + } + ], + "role": "user" + }, + { + "parts": [ + { + "functionCall": { + "args": { + "location": "london" + }, + "name": "get_weather" + }, + "thoughtSignature": "CiQBaRR9E1bQKFDb1IdCRw/EwdzLL8d3nA9HW1uaE6faGtp/yJkKbgFpFH0THbbSxOnGPDEv9ClVdoqXwmLCFArKL60PLMXEWPW7sNdcYw8NpJ+AYt2PrSUqTUa3/CycUwxkaatMAZH+XwtkGk9fWK82f1kIyp/849p37fZlAGNVYetP7OoHqwzVtMfSVS92pTGO7Zy7CskBAWkUfRMyfjq19zfhRokIcvIu7DY+jDwoabfNpANlgtBuXPnDnKlCy2kvjMd9lQCAF9UC4Awi9r9NClqa7D4W5tbKdNswYMlvXfTbcRRcuSJODFQKgLqXtLAS0NLLs1j96KjQrGJKWeGHvUxiQ1NTIGnOk0DimDdsZ9sSm94RYNkPmCsQSbOZJH5/8GO20HhJA90WsqRtufz8KFenR9b+N1yBpEwGVNdzRWvQYNQ40z14yJ8ue0ZHTJFCGL1/k1Rcq798v6ZhKdHR" + } + ], + "role": "model" + }, + { + "parts": [ + { + "functionResponse": { + "name": "get_weather", + "response": { + "result": "Weather in London is partly cloudy, 18°C, light breeze from the west." + } + } + } + ], + "role": "user" + } + ], + "generationConfig": { + "maxOutputTokens": 1024 + }, + "system_instruction": { + "parts": [ + { + "text": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available." + } + ] + }, + "tools": [ + { + "function_declarations": [ + { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "properties": { + "location": { + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + } + ] + } + ] + }, + "response": { + "content": [ + { + "text": "The weather in London is partly cloudy, 18°C, with a light breeze from the west.", + "type": "text" + } + ], + "reasoning": null, + "text": "The weather in London is partly cloudy, 18°C, with a light breeze from the west.", + "usage": { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "input_tokens": 187, + "output_tokens": 22 + } + }, + "sse": "data: {\"candidates\": [{\"content\": {\"parts\": [{\"text\": \"The weather in London\"}],\"role\": \"model\"},\"index\": 0}],\"usageMetadata\": {\"promptTokenCount\": 187,\"candidatesTokenCount\": 4,\"totalTokenCount\": 191,\"promptTokensDetails\": [{\"modality\": \"TEXT\",\"tokenCount\": 187}],\"serviceTier\": \"standard\"},\"modelVersion\": \"gemini-2.5-flash\",\"responseId\": \"Oi7NaujWAa2ExN8P89zcgQg\"}\r\n\r\ndata: {\"candidates\": [{\"content\": {\"parts\": [{\"text\": \" is partly cloudy, 18°C, with a light breeze from the west.\"}],\"role\": \"model\"},\"finishReason\": \"STOP\",\"index\": 0}],\"usageMetadata\": {\"promptTokenCount\": 187,\"candidatesTokenCount\": 22,\"totalTokenCount\": 209,\"promptTokensDetails\": [{\"modality\": \"TEXT\",\"tokenCount\": 187}],\"serviceTier\": \"standard\"},\"modelVersion\": \"gemini-2.5-flash\",\"responseId\": \"Oi7NaujWAa2ExN8P89zcgQg\"}\r\n\r\n" + }, + { + "prompt": "and in paris?", + "request": { + "_model": "gemini-2.5-flash", + "contents": [ + { + "parts": [ + { + "text": "how's the weather in london?" + } + ], + "role": "user" + }, + { + "parts": [ + { + "functionCall": { + "args": { + "location": "london" + }, + "name": "get_weather" + }, + "thoughtSignature": "CiQBaRR9E1bQKFDb1IdCRw/EwdzLL8d3nA9HW1uaE6faGtp/yJkKbgFpFH0THbbSxOnGPDEv9ClVdoqXwmLCFArKL60PLMXEWPW7sNdcYw8NpJ+AYt2PrSUqTUa3/CycUwxkaatMAZH+XwtkGk9fWK82f1kIyp/849p37fZlAGNVYetP7OoHqwzVtMfSVS92pTGO7Zy7CskBAWkUfRMyfjq19zfhRokIcvIu7DY+jDwoabfNpANlgtBuXPnDnKlCy2kvjMd9lQCAF9UC4Awi9r9NClqa7D4W5tbKdNswYMlvXfTbcRRcuSJODFQKgLqXtLAS0NLLs1j96KjQrGJKWeGHvUxiQ1NTIGnOk0DimDdsZ9sSm94RYNkPmCsQSbOZJH5/8GO20HhJA90WsqRtufz8KFenR9b+N1yBpEwGVNdzRWvQYNQ40z14yJ8ue0ZHTJFCGL1/k1Rcq798v6ZhKdHR" + } + ], + "role": "model" + }, + { + "parts": [ + { + "functionResponse": { + "name": "get_weather", + "response": { + "result": "Weather in London is partly cloudy, 18°C, light breeze from the west." + } + } + } + ], + "role": "user" + }, + { + "parts": [ + { + "text": "The weather in London is partly cloudy, 18°C, with a light breeze from the west." + } + ], + "role": "model" + }, + { + "parts": [ + { + "text": "and in paris?" + } + ], + "role": "user" + } + ], + "generationConfig": { + "maxOutputTokens": 1024 + }, + "system_instruction": { + "parts": [ + { + "text": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available." + } + ] + }, + "tools": [ + { + "function_declarations": [ + { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "properties": { + "location": { + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + } + ] + } + ] + }, + "response": { + "content": [ + { + "arguments": { + "location": "paris" + }, + "id": "call_8", + "name": "get_weather", + "provider_meta": { + "model": "gemini-2.5-flash", + "payload": { + "thoughtSignature": "CiQBaRR9EyXvkCduORckP6Gd3AwN7JRlQTOFepCIqTV3p1bf5YoKYQFpFH0T6odim/izeA1QBHQ4ooFrwCGXlJ8JzWewHhM6MS5qBnQjM9bUB2pJeoVGWC6wqJTNHp0kh2QP2P9kBvUWsccL/6IAgVZmVR9qOeWhun5Ye7XIo66bHi6ozVag5+sKMAFpFH0TFr+bKrK6F+Q0knAs07pXDXdHPATItqO1PJBM9pvJ9Y/mVOajK8JmoYv5Pgp3AWkUfRPFRL2DVEVug3eX8/UCTche9nghcBzrDhj0FLlEa+vfDqLiV7+xtEjv5ICphg3cq39asZClLQ/qAqIHzNo1XVrpxCQKaLVVBkB3YXJohSex+FQYMsowW1zOwKUycx2akiiaaWR2bCE44y4zJ8YW453nccMKVwFpFH0T4bYsGa/XUdVDKILzrvMYuqgai6XQ3yoOU6fwGnkWOaoqUNTXWOdCsxqE22HL1YsnBzuDlk9yc8QOBfcVcA3kmrJkZbzpdwrVdOLIp6x9FWN23Q==" + }, + "provider": "gemini" + }, + "type": "tool_call" + } + ], + "reasoning": null, + "text": "", + "usage": { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "input_tokens": 156, + "output_tokens": 15 + } + }, + "sse": "data: {\"candidates\": [{\"content\": {\"parts\": [{\"functionCall\": {\"name\": \"get_weather\",\"args\": {\"location\": \"paris\"}},\"thoughtSignature\": \"CiQBaRR9EyXvkCduORckP6Gd3AwN7JRlQTOFepCIqTV3p1bf5YoKYQFpFH0T6odim/izeA1QBHQ4ooFrwCGXlJ8JzWewHhM6MS5qBnQjM9bUB2pJeoVGWC6wqJTNHp0kh2QP2P9kBvUWsccL/6IAgVZmVR9qOeWhun5Ye7XIo66bHi6ozVag5+sKMAFpFH0TFr+bKrK6F+Q0knAs07pXDXdHPATItqO1PJBM9pvJ9Y/mVOajK8JmoYv5Pgp3AWkUfRPFRL2DVEVug3eX8/UCTche9nghcBzrDhj0FLlEa+vfDqLiV7+xtEjv5ICphg3cq39asZClLQ/qAqIHzNo1XVrpxCQKaLVVBkB3YXJohSex+FQYMsowW1zOwKUycx2akiiaaWR2bCE44y4zJ8YW453nccMKVwFpFH0T4bYsGa/XUdVDKILzrvMYuqgai6XQ3yoOU6fwGnkWOaoqUNTXWOdCsxqE22HL1YsnBzuDlk9yc8QOBfcVcA3kmrJkZbzpdwrVdOLIp6x9FWN23Q==\"}],\"role\": \"model\"},\"finishReason\": \"STOP\",\"index\": 0,\"finishMessage\": \"Model generated function call(s).\"}],\"usageMetadata\": {\"promptTokenCount\": 156,\"candidatesTokenCount\": 15,\"totalTokenCount\": 223,\"promptTokensDetails\": [{\"modality\": \"TEXT\",\"tokenCount\": 156}],\"thoughtsTokenCount\": 52,\"serviceTier\": \"standard\"},\"modelVersion\": \"gemini-2.5-flash\",\"responseId\": \"Oi7NauCwGuTR28oPquKmuAk\"}\r\n\r\n" + }, + { + "prompt": null, + "request": { + "_model": "gemini-2.5-flash", + "contents": [ + { + "parts": [ + { + "text": "how's the weather in london?" + } + ], + "role": "user" + }, + { + "parts": [ + { + "functionCall": { + "args": { + "location": "london" + }, + "name": "get_weather" + }, + "thoughtSignature": "CiQBaRR9E1bQKFDb1IdCRw/EwdzLL8d3nA9HW1uaE6faGtp/yJkKbgFpFH0THbbSxOnGPDEv9ClVdoqXwmLCFArKL60PLMXEWPW7sNdcYw8NpJ+AYt2PrSUqTUa3/CycUwxkaatMAZH+XwtkGk9fWK82f1kIyp/849p37fZlAGNVYetP7OoHqwzVtMfSVS92pTGO7Zy7CskBAWkUfRMyfjq19zfhRokIcvIu7DY+jDwoabfNpANlgtBuXPnDnKlCy2kvjMd9lQCAF9UC4Awi9r9NClqa7D4W5tbKdNswYMlvXfTbcRRcuSJODFQKgLqXtLAS0NLLs1j96KjQrGJKWeGHvUxiQ1NTIGnOk0DimDdsZ9sSm94RYNkPmCsQSbOZJH5/8GO20HhJA90WsqRtufz8KFenR9b+N1yBpEwGVNdzRWvQYNQ40z14yJ8ue0ZHTJFCGL1/k1Rcq798v6ZhKdHR" + } + ], + "role": "model" + }, + { + "parts": [ + { + "functionResponse": { + "name": "get_weather", + "response": { + "result": "Weather in London is partly cloudy, 18°C, light breeze from the west." + } + } + } + ], + "role": "user" + }, + { + "parts": [ + { + "text": "The weather in London is partly cloudy, 18°C, with a light breeze from the west." + } + ], + "role": "model" + }, + { + "parts": [ + { + "text": "and in paris?" + } + ], + "role": "user" + }, + { + "parts": [ + { + "functionCall": { + "args": { + "location": "paris" + }, + "name": "get_weather" + }, + "thoughtSignature": "CiQBaRR9EyXvkCduORckP6Gd3AwN7JRlQTOFepCIqTV3p1bf5YoKYQFpFH0T6odim/izeA1QBHQ4ooFrwCGXlJ8JzWewHhM6MS5qBnQjM9bUB2pJeoVGWC6wqJTNHp0kh2QP2P9kBvUWsccL/6IAgVZmVR9qOeWhun5Ye7XIo66bHi6ozVag5+sKMAFpFH0TFr+bKrK6F+Q0knAs07pXDXdHPATItqO1PJBM9pvJ9Y/mVOajK8JmoYv5Pgp3AWkUfRPFRL2DVEVug3eX8/UCTche9nghcBzrDhj0FLlEa+vfDqLiV7+xtEjv5ICphg3cq39asZClLQ/qAqIHzNo1XVrpxCQKaLVVBkB3YXJohSex+FQYMsowW1zOwKUycx2akiiaaWR2bCE44y4zJ8YW453nccMKVwFpFH0T4bYsGa/XUdVDKILzrvMYuqgai6XQ3yoOU6fwGnkWOaoqUNTXWOdCsxqE22HL1YsnBzuDlk9yc8QOBfcVcA3kmrJkZbzpdwrVdOLIp6x9FWN23Q==" + } + ], + "role": "model" + }, + { + "parts": [ + { + "functionResponse": { + "name": "get_weather", + "response": { + "result": "Weather in Paris is sunny, 23°C, no wind." + } + } + } + ], + "role": "user" + } + ], + "generationConfig": { + "maxOutputTokens": 1024 + }, + "system_instruction": { + "parts": [ + { + "text": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available." + } + ] + }, + "tools": [ + { + "function_declarations": [ + { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "properties": { + "location": { + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + } + ] + } + ] + }, + "response": { + "content": [ + { + "text": "The weather in Paris is sunny, 23°C, with no wind.", + "type": "text" + } + ], + "reasoning": null, + "text": "The weather in Paris is sunny, 23°C, with no wind.", + "usage": { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "input_tokens": 252, + "output_tokens": 16 + } + }, + "sse": "data: {\"candidates\": [{\"content\": {\"parts\": [{\"text\": \"The weather in Paris\"}],\"role\": \"model\"},\"index\": 0}],\"usageMetadata\": {\"promptTokenCount\": 252,\"candidatesTokenCount\": 4,\"totalTokenCount\": 256,\"promptTokensDetails\": [{\"modality\": \"TEXT\",\"tokenCount\": 252}],\"serviceTier\": \"standard\"},\"modelVersion\": \"gemini-2.5-flash\",\"responseId\": \"Oy7NauqyJtzAxN8Pho7W0A0\"}\r\n\r\ndata: {\"candidates\": [{\"content\": {\"parts\": [{\"text\": \" is sunny, \"}],\"role\": \"model\"},\"index\": 0}],\"usageMetadata\": {\"promptTokenCount\": 252,\"candidatesTokenCount\": 7,\"totalTokenCount\": 259,\"promptTokensDetails\": [{\"modality\": \"TEXT\",\"tokenCount\": 252}],\"serviceTier\": \"standard\"},\"modelVersion\": \"gemini-2.5-flash\",\"responseId\": \"Oy7NauqyJtzAxN8Pho7W0A0\"}\r\n\r\ndata: {\"candidates\": [{\"content\": {\"parts\": [{\"text\": \"23°C, with no wind.\"}],\"role\": \"model\"},\"finishReason\": \"STOP\",\"index\": 0}],\"usageMetadata\": {\"promptTokenCount\": 252,\"candidatesTokenCount\": 16,\"totalTokenCount\": 268,\"promptTokensDetails\": [{\"modality\": \"TEXT\",\"tokenCount\": 252}],\"serviceTier\": \"standard\"},\"modelVersion\": \"gemini-2.5-flash\",\"responseId\": \"Oy7NauqyJtzAxN8Pho7W0A0\"}\r\n\r\n" + } +] diff --git a/test/fixtures/gemini-text.json b/test/fixtures/gemini-text.json new file mode 100644 --- /dev/null +++ b/test/fixtures/gemini-text.json @@ -0,0 +1,20 @@ +{ + "candidates": [ + { + "content": { + "parts": [ + { + "text": "Hello! How can I help you today?" + } + ], + "role": "model" + }, + "finishReason": "STOP" + } + ], + "usageMetadata": { + "promptTokenCount": 20, + "candidatesTokenCount": 8, + "totalTokenCount": 28 + } +} diff --git a/test/fixtures/gemini-tool-use.json b/test/fixtures/gemini-tool-use.json new file mode 100644 --- /dev/null +++ b/test/fixtures/gemini-tool-use.json @@ -0,0 +1,23 @@ +{ + "candidates": [ + { + "content": { + "parts": [ + { + "functionCall": { + "name": "get_weather", + "args": { "location": "London" } + } + } + ], + "role": "model" + }, + "finishReason": "STOP" + } + ], + "usageMetadata": { + "promptTokenCount": 40, + "candidatesTokenCount": 12, + "totalTokenCount": 52 + } +} diff --git a/test/fixtures/ollama-catalog.json b/test/fixtures/ollama-catalog.json new file mode 100644 --- /dev/null +++ b/test/fixtures/ollama-catalog.json @@ -0,0 +1,32 @@ +[ + { + "modelConfigName": "llama_3_2", + "providerName": "ollama", + "modelName": "llama3.2:latest", + "pricing": { + "pricePerMillionInput": 0.0, + "pricePerMillionOutput": 0.0 + }, + "maxTokens": 1024, + "temperature": 0.5, + "requestTimeout": 3000, + "throttleDelay": 1000, + "retryCount": 3, + "jitterBackoff": 1000 + }, + { + "modelConfigName": "mistral", + "providerName": "ollama", + "modelName": "mistral:latest", + "pricing": { + "pricePerMillionInput": 0.0, + "pricePerMillionOutput": 0.0 + }, + "maxTokens": 1024, + "temperature": 0.5, + "requestTimeout": 3000, + "throttleDelay": 1000, + "retryCount": 3, + "jitterBackoff": 1000 + } +] diff --git a/test/fixtures/ollama-conversation-generated.json b/test/fixtures/ollama-conversation-generated.json new file mode 100644 --- /dev/null +++ b/test/fixtures/ollama-conversation-generated.json @@ -0,0 +1,360 @@ +[ + { + "prompt": "how's the weather in london?", + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + } + ], + "model": "llama3.2:latest", + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "choices": [ + { + "finish_reason": "tool_calls", + "index": 0, + "message": { + "content": "", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"London\"}", + "name": "get_weather" + }, + "id": "call_4ysdmo61", + "index": 0, + "type": "function" + } + ] + } + } + ], + "created": 1792089544, + "id": "chatcmpl-279", + "model": "llama3.2:latest", + "object": "chat.completion", + "system_fingerprint": "fp_ollama", + "usage": { + "completion_tokens": 17, + "prompt_tokens": 188, + "total_tokens": 205 + } + }, + "sse": null + }, + { + "prompt": null, + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"London\"}", + "name": "get_weather" + }, + "id": "call_4ysdmo61", + "type": "function" + } + ] + }, + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "role": "tool", + "tool_call_id": "call_4ysdmo61" + } + ], + "model": "llama3.2:latest", + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "choices": [ + { + "finish_reason": "stop", + "index": 0, + "message": { + "content": "The current weather conditions in London are mainly sunny with some cloudiness, and the temperature is around 18 degrees Celsius (64°F). There's a gentle breeze coming from the west at about 19 km/h (12 mph). Would you like to know more or check the forecast for a specific date?", + "role": "assistant" + } + } + ], + "created": 1792089545, + "id": "chatcmpl-901", + "model": "llama3.2:latest", + "object": "chat.completion", + "system_fingerprint": "fp_ollama", + "usage": { + "completion_tokens": 62, + "prompt_tokens": 137, + "total_tokens": 199 + } + }, + "sse": null + }, + { + "prompt": "and in paris?", + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"London\"}", + "name": "get_weather" + }, + "id": "call_4ysdmo61", + "type": "function" + } + ] + }, + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "role": "tool", + "tool_call_id": "call_4ysdmo61" + }, + { + "content": "The current weather conditions in London are mainly sunny with some cloudiness, and the temperature is around 18 degrees Celsius (64°F). There's a gentle breeze coming from the west at about 19 km/h (12 mph). Would you like to know more or check the forecast for a specific date?", + "role": "assistant" + }, + { + "content": "and in paris?", + "role": "user" + } + ], + "model": "llama3.2:latest", + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "choices": [ + { + "finish_reason": "tool_calls", + "index": 0, + "message": { + "content": "", + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"Paris\"}", + "name": "get_weather" + }, + "id": "call_sahmucf9", + "index": 0, + "type": "function" + } + ] + } + } + ], + "created": 1792089546, + "id": "chatcmpl-768", + "model": "llama3.2:latest", + "object": "chat.completion", + "system_fingerprint": "fp_ollama", + "usage": { + "completion_tokens": 17, + "prompt_tokens": 306, + "total_tokens": 323 + } + }, + "sse": null + }, + { + "prompt": null, + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"London\"}", + "name": "get_weather" + }, + "id": "call_4ysdmo61", + "type": "function" + } + ] + }, + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "role": "tool", + "tool_call_id": "call_4ysdmo61" + }, + { + "content": "The current weather conditions in London are mainly sunny with some cloudiness, and the temperature is around 18 degrees Celsius (64°F). There's a gentle breeze coming from the west at about 19 km/h (12 mph). Would you like to know more or check the forecast for a specific date?", + "role": "assistant" + }, + { + "content": "and in paris?", + "role": "user" + }, + { + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"Paris\"}", + "name": "get_weather" + }, + "id": "call_sahmucf9", + "type": "function" + } + ] + }, + { + "content": "Weather in Paris is sunny, 23°C, no wind.", + "role": "tool", + "tool_call_id": "call_sahmucf9" + } + ], + "model": "llama3.2:latest", + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "choices": [ + { + "finish_reason": "stop", + "index": 0, + "message": { + "content": "The current weather conditions in Paris are perfect for a warm day outdoors! It's a beautiful, sunny day with a high temperature of 23 degrees Celsius (73°F) and not a single cloud in the sky. There's no wind to speak of, making it an ideal day to enjoy the city's famous landmarks, cafes, or simply stroll along the Seine River.", + "role": "assistant" + } + } + ], + "created": 1792089548, + "id": "chatcmpl-34", + "model": "llama3.2:latest", + "object": "chat.completion", + "system_fingerprint": "fp_ollama", + "usage": { + "completion_tokens": 76, + "prompt_tokens": 251, + "total_tokens": 327 + } + }, + "sse": null + } +] diff --git a/test/fixtures/ollama-conversation-streamed.json b/test/fixtures/ollama-conversation-streamed.json new file mode 100644 --- /dev/null +++ b/test/fixtures/ollama-conversation-streamed.json @@ -0,0 +1,338 @@ +[ + { + "prompt": "how's the weather in london?", + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + } + ], + "model": "llama3.2:latest", + "stream": true, + "stream_options": { + "include_usage": true + }, + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "content": [ + { + "arguments": { + "location": "London" + }, + "id": "call_wucefmo8", + "name": "get_weather", + "type": "tool_call" + } + ], + "reasoning": null, + "text": "", + "usage": { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "input_tokens": 188, + "output_tokens": 17 + } + }, + "sse": "data: {\"id\":\"chatcmpl-793\",\"object\":\"chat.completion.chunk\",\"created\":1792089548,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"\",\"tool_calls\":[{\"id\":\"call_wucefmo8\",\"index\":0,\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"arguments\":\"{\\\"location\\\":\\\"London\\\"}\"}}]},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-793\",\"object\":\"chat.completion.chunk\",\"created\":1792089548,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"\"},\"finish_reason\":\"tool_calls\"}]}\n\ndata: {\"id\":\"chatcmpl-793\",\"object\":\"chat.completion.chunk\",\"created\":1792089548,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[],\"usage\":{\"prompt_tokens\":188,\"completion_tokens\":17,\"total_tokens\":205}}\n\ndata: [DONE]\n\n" + }, + { + "prompt": null, + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"London\"}", + "name": "get_weather" + }, + "id": "call_wucefmo8", + "type": "function" + } + ] + }, + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "role": "tool", + "tool_call_id": "call_wucefmo8" + } + ], + "model": "llama3.2:latest", + "stream": true, + "stream_options": { + "include_usage": true + }, + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "content": [ + { + "text": "Currently, the weather in London is partly cloudy with a temperature of 18°C and a light breeze blowing from the west.", + "type": "text" + } + ], + "reasoning": null, + "text": "Currently, the weather in London is partly cloudy with a temperature of 18°C and a light breeze blowing from the west.", + "usage": { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "input_tokens": 137, + "output_tokens": 26 + } + }, + "sse": "data: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089548,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"Currently\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089548,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\",\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089548,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" the\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" weather\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" in\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" London\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" is\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" partly\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" cloudy\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" with\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" a\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" temperature\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" of\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" \"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"18\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"°C\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" and\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" a\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" light\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" breeze\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" blowing\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" from\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" the\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" west\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\".\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"\"},\"finish_reason\":\"stop\"}]}\n\ndata: {\"id\":\"chatcmpl-502\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[],\"usage\":{\"prompt_tokens\":137,\"completion_tokens\":26,\"total_tokens\":163}}\n\ndata: [DONE]\n\n" + }, + { + "prompt": "and in paris?", + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"London\"}", + "name": "get_weather" + }, + "id": "call_wucefmo8", + "type": "function" + } + ] + }, + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "role": "tool", + "tool_call_id": "call_wucefmo8" + }, + { + "content": "Currently, the weather in London is partly cloudy with a temperature of 18°C and a light breeze blowing from the west.", + "role": "assistant" + }, + { + "content": "and in paris?", + "role": "user" + } + ], + "model": "llama3.2:latest", + "stream": true, + "stream_options": { + "include_usage": true + }, + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "content": [ + { + "arguments": { + "location": "Paris" + }, + "id": "call_9243whne", + "name": "get_weather", + "type": "tool_call" + } + ], + "reasoning": null, + "text": "", + "usage": { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "input_tokens": 270, + "output_tokens": 17 + } + }, + "sse": "data: {\"id\":\"chatcmpl-72\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"\",\"tool_calls\":[{\"id\":\"call_9243whne\",\"index\":0,\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"arguments\":\"{\\\"location\\\":\\\"Paris\\\"}\"}}]},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-72\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"\"},\"finish_reason\":\"tool_calls\"}]}\n\ndata: {\"id\":\"chatcmpl-72\",\"object\":\"chat.completion.chunk\",\"created\":1792089549,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[],\"usage\":{\"prompt_tokens\":270,\"completion_tokens\":17,\"total_tokens\":287}}\n\ndata: [DONE]\n\n" + }, + { + "prompt": null, + "request": { + "max_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"London\"}", + "name": "get_weather" + }, + "id": "call_wucefmo8", + "type": "function" + } + ] + }, + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "role": "tool", + "tool_call_id": "call_wucefmo8" + }, + { + "content": "Currently, the weather in London is partly cloudy with a temperature of 18°C and a light breeze blowing from the west.", + "role": "assistant" + }, + { + "content": "and in paris?", + "role": "user" + }, + { + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"Paris\"}", + "name": "get_weather" + }, + "id": "call_9243whne", + "type": "function" + } + ] + }, + { + "content": "Weather in Paris is sunny, 23°C, no wind.", + "role": "tool", + "tool_call_id": "call_9243whne" + } + ], + "model": "llama3.2:latest", + "stream": true, + "stream_options": { + "include_usage": true + }, + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "content": [ + { + "text": "Currently, the weather in Paris is sunny with a temperature of 23°C and no wind blowing.", + "type": "text" + } + ], + "reasoning": null, + "text": "Currently, the weather in Paris is sunny with a temperature of 23°C and no wind blowing.", + "usage": { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "input_tokens": 215, + "output_tokens": 21 + } + }, + "sse": "data: {\"id\":\"chatcmpl-480\",\"object\":\"chat.completion.chunk\",\"created\":1792089550,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"Currently\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-480\",\"object\":\"chat.completion.chunk\",\"created\":1792089550,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\",\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-480\",\"object\":\"chat.completion.chunk\",\"created\":1792089550,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" the\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-480\",\"object\":\"chat.completion.chunk\",\"created\":1792089550,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" weather\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-480\",\"object\":\"chat.completion.chunk\",\"created\":1792089550,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" in\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-480\",\"object\":\"chat.completion.chunk\",\"created\":1792089550,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" Paris\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-480\",\"object\":\"chat.completion.chunk\",\"created\":1792089550,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" is\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-480\",\"object\":\"chat.completion.chunk\",\"created\":1792089550,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" sunny\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-480\",\"object\":\"chat.completion.chunk\",\"created\":1792089550,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" with\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-480\",\"object\":\"chat.completion.chunk\",\"created\":1792089550,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" a\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-480\",\"object\":\"chat.completion.chunk\",\"created\":1792089550,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" temperature\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-480\",\"object\":\"chat.completion.chunk\",\"created\":1792089550,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" of\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-480\",\"object\":\"chat.completion.chunk\",\"created\":1792089550,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" \"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-480\",\"object\":\"chat.completion.chunk\",\"created\":1792089550,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"23\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-480\",\"object\":\"chat.completion.chunk\",\"created\":1792089550,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"°C\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-480\",\"object\":\"chat.completion.chunk\",\"created\":1792089550,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" and\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-480\",\"object\":\"chat.completion.chunk\",\"created\":1792089550,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" no\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-480\",\"object\":\"chat.completion.chunk\",\"created\":1792089550,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" wind\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-480\",\"object\":\"chat.completion.chunk\",\"created\":1792089550,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\" blowing\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-480\",\"object\":\"chat.completion.chunk\",\"created\":1792089550,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\".\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"chatcmpl-480\",\"object\":\"chat.completion.chunk\",\"created\":1792089550,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"\"},\"finish_reason\":\"stop\"}]}\n\ndata: {\"id\":\"chatcmpl-480\",\"object\":\"chat.completion.chunk\",\"created\":1792089550,\"model\":\"llama3.2:latest\",\"system_fingerprint\":\"fp_ollama\",\"choices\":[],\"usage\":{\"prompt_tokens\":215,\"completion_tokens\":21,\"total_tokens\":236}}\n\ndata: [DONE]\n\n" + } +] diff --git a/test/fixtures/ollama-multi-tool-chunk.sse b/test/fixtures/ollama-multi-tool-chunk.sse new file mode 100644 --- /dev/null +++ b/test/fixtures/ollama-multi-tool-chunk.sse @@ -0,0 +1,6 @@ +data: {"id":"chatcmpl-1","object":"chat.completion.chunk","model":"llama3.2","choices":[{"index":0,"delta":{"role":"assistant","content":"","tool_calls":[{"id":"call_x","type":"function","function":{"name":"get_weather","arguments":"{\"location\":\"London\"}"}},{"id":"call_y","type":"function","function":{"name":"get_weather","arguments":"{\"location\":\"Paris\"}"}}]},"finish_reason":null}]} + +data: {"id":"chatcmpl-1","object":"chat.completion.chunk","model":"llama3.2","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":"stop"}]} + +data: [DONE] + diff --git a/test/fixtures/openai-conversation-generated.json b/test/fixtures/openai-conversation-generated.json new file mode 100644 --- /dev/null +++ b/test/fixtures/openai-conversation-generated.json @@ -0,0 +1,400 @@ +[ + { + "prompt": "how's the weather in london?", + "request": { + "max_completion_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + } + ], + "model": "gpt-4.1-2025-04-14", + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": ["location"], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "choices": [ + { + "finish_reason": "tool_calls", + "index": 0, + "logprobs": null, + "message": { + "annotations": [], + "content": null, + "refusal": null, + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"London\"}", + "name": "get_weather" + }, + "id": "call_iEUnimmOnAOY9s9IVbLEAFzJ", + "type": "function" + } + ] + } + } + ], + "created": 1778258479, + "id": "chatcmpl-DdIYJq6wNZp4OzHKLO6FRfALkcbZE", + "model": "gpt-4.1-2025-04-14", + "object": "chat.completion", + "service_tier": "default", + "system_fingerprint": "fp_32787711d0", + "usage": { + "completion_tokens": 14, + "completion_tokens_details": { + "accepted_prediction_tokens": 0, + "audio_tokens": 0, + "reasoning_tokens": 0, + "rejected_prediction_tokens": 0 + }, + "prompt_tokens": 84, + "prompt_tokens_details": { + "audio_tokens": 0, + "cached_tokens": 0 + }, + "total_tokens": 98 + } + } + }, + { + "request": { + "max_completion_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"London\"}", + "name": "get_weather" + }, + "id": "call_iEUnimmOnAOY9s9IVbLEAFzJ", + "type": "function" + } + ] + }, + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "role": "tool", + "tool_call_id": "call_iEUnimmOnAOY9s9IVbLEAFzJ" + } + ], + "model": "gpt-4.1-2025-04-14", + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": ["location"], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "choices": [ + { + "finish_reason": "stop", + "index": 0, + "logprobs": null, + "message": { + "annotations": [], + "content": "The weather in London is partly cloudy with a temperature of 18°C. There's a light breeze coming from the west.", + "refusal": null, + "role": "assistant" + } + } + ], + "created": 1778258482, + "id": "chatcmpl-DdIYM5PzqXzTq0bilZwSUsbnHCxFX", + "model": "gpt-4.1-2025-04-14", + "object": "chat.completion", + "service_tier": "default", + "system_fingerprint": "fp_32787711d0", + "usage": { + "completion_tokens": 25, + "completion_tokens_details": { + "accepted_prediction_tokens": 0, + "audio_tokens": 0, + "reasoning_tokens": 0, + "rejected_prediction_tokens": 0 + }, + "prompt_tokens": 123, + "prompt_tokens_details": { + "audio_tokens": 0, + "cached_tokens": 0 + }, + "total_tokens": 148 + } + } + }, + { + "prompt": "and in paris?", + "request": { + "max_completion_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"London\"}", + "name": "get_weather" + }, + "id": "call_iEUnimmOnAOY9s9IVbLEAFzJ", + "type": "function" + } + ] + }, + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "role": "tool", + "tool_call_id": "call_iEUnimmOnAOY9s9IVbLEAFzJ" + }, + { + "content": "The weather in London is partly cloudy with a temperature of 18°C. There's a light breeze coming from the west.", + "role": "assistant" + }, + { + "content": "and in paris?", + "role": "user" + } + ], + "model": "gpt-4.1-2025-04-14", + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": ["location"], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "choices": [ + { + "finish_reason": "tool_calls", + "index": 0, + "logprobs": null, + "message": { + "annotations": [], + "content": null, + "refusal": null, + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"Paris\"}", + "name": "get_weather" + }, + "id": "call_BjIIjpMKAjecx0iXj6sFzyz8", + "type": "function" + } + ] + } + } + ], + "created": 1778258484, + "id": "chatcmpl-DdIYOrExHpeG6IO8eZcrLSAVBiZ8s", + "model": "gpt-4.1-2025-04-14", + "object": "chat.completion", + "service_tier": "default", + "system_fingerprint": "fp_32787711d0", + "usage": { + "completion_tokens": 14, + "completion_tokens_details": { + "accepted_prediction_tokens": 0, + "audio_tokens": 0, + "reasoning_tokens": 0, + "rejected_prediction_tokens": 0 + }, + "prompt_tokens": 159, + "prompt_tokens_details": { + "audio_tokens": 0, + "cached_tokens": 0 + }, + "total_tokens": 173 + } + } + }, + { + "request": { + "max_completion_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"London\"}", + "name": "get_weather" + }, + "id": "call_iEUnimmOnAOY9s9IVbLEAFzJ", + "type": "function" + } + ] + }, + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "role": "tool", + "tool_call_id": "call_iEUnimmOnAOY9s9IVbLEAFzJ" + }, + { + "content": "The weather in London is partly cloudy with a temperature of 18°C. There's a light breeze coming from the west.", + "role": "assistant" + }, + { + "content": "and in paris?", + "role": "user" + }, + { + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"Paris\"}", + "name": "get_weather" + }, + "id": "call_BjIIjpMKAjecx0iXj6sFzyz8", + "type": "function" + } + ] + }, + { + "content": "Weather in Paris is sunny, 23°C, no wind.", + "role": "tool", + "tool_call_id": "call_BjIIjpMKAjecx0iXj6sFzyz8" + } + ], + "model": "gpt-4.1-2025-04-14", + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": ["location"], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "choices": [ + { + "finish_reason": "stop", + "index": 0, + "logprobs": null, + "message": { + "annotations": [], + "content": "The weather in Paris is sunny with a temperature of 23°C and no wind.", + "refusal": null, + "role": "assistant" + } + } + ], + "created": 1778258487, + "id": "chatcmpl-DdIYROAfAokkKxQSrRIrwAc0sluGR", + "model": "gpt-4.1-2025-04-14", + "object": "chat.completion", + "service_tier": "default", + "system_fingerprint": "fp_32787711d0", + "usage": { + "completion_tokens": 18, + "completion_tokens_details": { + "accepted_prediction_tokens": 0, + "audio_tokens": 0, + "reasoning_tokens": 0, + "rejected_prediction_tokens": 0 + }, + "prompt_tokens": 194, + "prompt_tokens_details": { + "audio_tokens": 0, + "cached_tokens": 0 + }, + "total_tokens": 212 + } + } + } +] diff --git a/test/fixtures/openai-conversation-streamed.json b/test/fixtures/openai-conversation-streamed.json new file mode 100644 --- /dev/null +++ b/test/fixtures/openai-conversation-streamed.json @@ -0,0 +1,338 @@ +[ + { + "prompt": "how's the weather in london?", + "request": { + "max_completion_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + } + ], + "model": "gpt-4.1-2025-04-14", + "stream": true, + "stream_options": { + "include_usage": true + }, + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "content": [ + { + "arguments": { + "location": "London" + }, + "id": "call_rolc2qOqqIpkgBi9c3AjGJdp", + "name": "get_weather", + "type": "tool_call" + } + ], + "reasoning": null, + "text": "", + "usage": { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "input_tokens": 84, + "output_tokens": 14 + } + }, + "sse": "data: {\"id\":\"chatcmpl-EYFXgnXPiWvhYoXnGEs3qsPAwkkzH\",\"object\":\"chat.completion.chunk\",\"created\":1791831604,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":null,\"tool_calls\":[{\"index\":0,\"id\":\"call_rolc2qOqqIpkgBi9c3AjGJdp\",\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"arguments\":\"\"}}],\"refusal\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"va0ZsDC5\"}\n\ndata: {\"id\":\"chatcmpl-EYFXgnXPiWvhYoXnGEs3qsPAwkkzH\",\"object\":\"chat.completion.chunk\",\"created\":1791831604,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"{\\\"\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"KI\"}\n\ndata: {\"id\":\"chatcmpl-EYFXgnXPiWvhYoXnGEs3qsPAwkkzH\",\"object\":\"chat.completion.chunk\",\"created\":1791831604,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"location\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"xoXvkYjMN1CWu\"}\n\ndata: {\"id\":\"chatcmpl-EYFXgnXPiWvhYoXnGEs3qsPAwkkzH\",\"object\":\"chat.completion.chunk\",\"created\":1791831604,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\\\":\\\"\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"\"}\n\ndata: {\"id\":\"chatcmpl-EYFXgnXPiWvhYoXnGEs3qsPAwkkzH\",\"object\":\"chat.completion.chunk\",\"created\":1791831604,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"London\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"bSKKjNvByhoAiII\"}\n\ndata: {\"id\":\"chatcmpl-EYFXgnXPiWvhYoXnGEs3qsPAwkkzH\",\"object\":\"chat.completion.chunk\",\"created\":1791831604,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\\\"}\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"i7\"}\n\ndata: {\"id\":\"chatcmpl-EYFXgnXPiWvhYoXnGEs3qsPAwkkzH\",\"object\":\"chat.completion.chunk\",\"created\":1791831604,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{},\"logprobs\":null,\"finish_reason\":\"tool_calls\"}],\"usage\":null,\"obfuscation\":\"Ycm\"}\n\ndata: {\"id\":\"chatcmpl-EYFXgnXPiWvhYoXnGEs3qsPAwkkzH\",\"object\":\"chat.completion.chunk\",\"created\":1791831604,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[],\"usage\":{\"prompt_tokens\":84,\"completion_tokens\":14,\"total_tokens\":98,\"prompt_tokens_details\":{\"cached_tokens\":0,\"audio_tokens\":0},\"completion_tokens_details\":{\"reasoning_tokens\":0,\"audio_tokens\":0,\"accepted_prediction_tokens\":0,\"rejected_prediction_tokens\":0}},\"obfuscation\":\"D1Zx8xfEzl9CR8\"}\n\ndata: [DONE]\n\n" + }, + { + "prompt": null, + "request": { + "max_completion_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"London\"}", + "name": "get_weather" + }, + "id": "call_rolc2qOqqIpkgBi9c3AjGJdp", + "type": "function" + } + ] + }, + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "role": "tool", + "tool_call_id": "call_rolc2qOqqIpkgBi9c3AjGJdp" + } + ], + "model": "gpt-4.1-2025-04-14", + "stream": true, + "stream_options": { + "include_usage": true + }, + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "content": [ + { + "text": "The weather in London is partly cloudy, with a temperature of 18°C and a light breeze coming from the west.", + "type": "text" + } + ], + "reasoning": null, + "text": "The weather in London is partly cloudy, with a temperature of 18°C and a light breeze coming from the west.", + "usage": { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "input_tokens": 123, + "output_tokens": 25 + } + }, + "sse": "data: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"\",\"refusal\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"Rg2310DL4NDTt\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"The\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"FLX8yAJ2iTL8\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" weather\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"5OeA6mU\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" in\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"o0zkOIiJPVD0\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" London\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"DriFrE1n\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" is\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"boexk7KlBxvY\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" partly\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"Ir4rXbYv\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" cloudy\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"UDXQldlH\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\",\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"TKACdWqZD0437K\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" with\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"a5nYEZRMg7\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" a\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"E8tWufncVyTR6\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" temperature\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"YKC\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" of\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"9mPYJaLgHI9b\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" \"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"40WgCSgs89AvxD\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"18\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"w33nHLLkiMakd\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"°C\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"z5Zs5ASmVpgca\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" and\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"oDeYj4S96PN\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" a\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"QwphErMf4ICfi\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" light\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"BDJ9SB0aV\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" breeze\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"MEeYj2C5\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" coming\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"ILYESqZy\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" from\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"JPAK9v887j\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" the\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"7GBrKpekfjg\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" west\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"JhxHG9IHVF\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\".\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"Qq8N3mfgEKTmh6\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{},\"logprobs\":null,\"finish_reason\":\"stop\"}],\"usage\":null,\"obfuscation\":\"cDU785Xq7\"}\n\ndata: {\"id\":\"chatcmpl-EYFXhFwnrBsOI7NC135l7gX10dK7g\",\"object\":\"chat.completion.chunk\",\"created\":1791831605,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[],\"usage\":{\"prompt_tokens\":123,\"completion_tokens\":25,\"total_tokens\":148,\"prompt_tokens_details\":{\"cached_tokens\":0,\"audio_tokens\":0},\"completion_tokens_details\":{\"reasoning_tokens\":0,\"audio_tokens\":0,\"accepted_prediction_tokens\":0,\"rejected_prediction_tokens\":0}},\"obfuscation\":\"reJ4krfYXryt\"}\n\ndata: [DONE]\n\n" + }, + { + "prompt": "and in paris?", + "request": { + "max_completion_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"London\"}", + "name": "get_weather" + }, + "id": "call_rolc2qOqqIpkgBi9c3AjGJdp", + "type": "function" + } + ] + }, + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "role": "tool", + "tool_call_id": "call_rolc2qOqqIpkgBi9c3AjGJdp" + }, + { + "content": "The weather in London is partly cloudy, with a temperature of 18°C and a light breeze coming from the west.", + "role": "assistant" + }, + { + "content": "and in paris?", + "role": "user" + } + ], + "model": "gpt-4.1-2025-04-14", + "stream": true, + "stream_options": { + "include_usage": true + }, + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "content": [ + { + "arguments": { + "location": "Paris" + }, + "id": "call_mZiezYE31U7yku40mf9oVc7B", + "name": "get_weather", + "type": "tool_call" + } + ], + "reasoning": null, + "text": "", + "usage": { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "input_tokens": 159, + "output_tokens": 14 + } + }, + "sse": "data: {\"id\":\"chatcmpl-EYFXjUquzNtVFbuJ6QX9Ros0rrMqU\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":null,\"tool_calls\":[{\"index\":0,\"id\":\"call_mZiezYE31U7yku40mf9oVc7B\",\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"arguments\":\"\"}}],\"refusal\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"0JpuBFff\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjUquzNtVFbuJ6QX9Ros0rrMqU\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"{\\\"\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"j5\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjUquzNtVFbuJ6QX9Ros0rrMqU\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"location\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"WB5D1oeSiS4DZ\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjUquzNtVFbuJ6QX9Ros0rrMqU\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\\\":\\\"\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjUquzNtVFbuJ6QX9Ros0rrMqU\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"Paris\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjUquzNtVFbuJ6QX9Ros0rrMqU\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\\\"}\"}}]},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"69\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjUquzNtVFbuJ6QX9Ros0rrMqU\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{},\"logprobs\":null,\"finish_reason\":\"tool_calls\"}],\"usage\":null,\"obfuscation\":\"4PH\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjUquzNtVFbuJ6QX9Ros0rrMqU\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[],\"usage\":{\"prompt_tokens\":159,\"completion_tokens\":14,\"total_tokens\":173,\"prompt_tokens_details\":{\"cached_tokens\":0,\"audio_tokens\":0},\"completion_tokens_details\":{\"reasoning_tokens\":0,\"audio_tokens\":0,\"accepted_prediction_tokens\":0,\"rejected_prediction_tokens\":0}},\"obfuscation\":\"yL2QUw8rFx9D\"}\n\ndata: [DONE]\n\n" + }, + { + "prompt": null, + "request": { + "max_completion_tokens": 1024, + "messages": [ + { + "content": "You are a helpful assistant who answers questions and executes tools for the user. Always use tools when asked to, but use only the tools that are available.", + "role": "system" + }, + { + "content": "how's the weather in london?", + "role": "user" + }, + { + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"London\"}", + "name": "get_weather" + }, + "id": "call_rolc2qOqqIpkgBi9c3AjGJdp", + "type": "function" + } + ] + }, + { + "content": "Weather in London is partly cloudy, 18°C, light breeze from the west.", + "role": "tool", + "tool_call_id": "call_rolc2qOqqIpkgBi9c3AjGJdp" + }, + { + "content": "The weather in London is partly cloudy, with a temperature of 18°C and a light breeze coming from the west.", + "role": "assistant" + }, + { + "content": "and in paris?", + "role": "user" + }, + { + "role": "assistant", + "tool_calls": [ + { + "function": { + "arguments": "{\"location\":\"Paris\"}", + "name": "get_weather" + }, + "id": "call_mZiezYE31U7yku40mf9oVc7B", + "type": "function" + } + ] + }, + { + "content": "Weather in Paris is sunny, 23°C, no wind.", + "role": "tool", + "tool_call_id": "call_mZiezYE31U7yku40mf9oVc7B" + } + ], + "model": "gpt-4.1-2025-04-14", + "stream": true, + "stream_options": { + "include_usage": true + }, + "tools": [ + { + "function": { + "description": "Get the current weather for a given location", + "name": "get_weather", + "parameters": { + "$comment": "WeatherToolArgs", + "properties": { + "location": { + "$comment": "City name, e.g. London", + "type": "string" + } + }, + "required": [ + "location" + ], + "type": "object" + } + }, + "type": "function" + } + ] + }, + "response": { + "content": [ + { + "text": "The weather in Paris is sunny, with a temperature of 23°C and no wind.", + "type": "text" + } + ], + "reasoning": null, + "text": "The weather in Paris is sunny, with a temperature of 23°C and no wind.", + "usage": { + "cache_creation_tokens": 0, + "cache_read_tokens": 0, + "input_tokens": 194, + "output_tokens": 19 + } + }, + "sse": "data: {\"id\":\"chatcmpl-EYFXjaepQSO9zofmXZ8tduRkU3929\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"\",\"refusal\":null},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"AF0OZRnw8ytw5\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjaepQSO9zofmXZ8tduRkU3929\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"The\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"prrzYtM3NFgP\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjaepQSO9zofmXZ8tduRkU3929\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" weather\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"lahdur4\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjaepQSO9zofmXZ8tduRkU3929\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" in\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"oQD1nUYZce8T\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjaepQSO9zofmXZ8tduRkU3929\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" Paris\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"BLSE38u0g\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjaepQSO9zofmXZ8tduRkU3929\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" is\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"yXU9Ewor0nAR\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjaepQSO9zofmXZ8tduRkU3929\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" sunny\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"KCydgTmTm\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjaepQSO9zofmXZ8tduRkU3929\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\",\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"gyCuksQfSZKQdg\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjaepQSO9zofmXZ8tduRkU3929\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" with\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"DFVurpOU07\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjaepQSO9zofmXZ8tduRkU3929\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" a\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"kN54izS8DAiZO\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjaepQSO9zofmXZ8tduRkU3929\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" temperature\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"ig8\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjaepQSO9zofmXZ8tduRkU3929\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" of\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"37xdnUnN6QCw\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjaepQSO9zofmXZ8tduRkU3929\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" \"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"QU3gsc6sxaB4JJ\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjaepQSO9zofmXZ8tduRkU3929\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"23\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"w03JQcB4YDdiR\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjaepQSO9zofmXZ8tduRkU3929\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"°C\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"3m30a1OTU9Oze\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjaepQSO9zofmXZ8tduRkU3929\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" and\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"lxaH9sU7HU9\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjaepQSO9zofmXZ8tduRkU3929\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" no\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"c00kDsMHK4CT\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjaepQSO9zofmXZ8tduRkU3929\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" wind\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"abxw3dD7GL\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjaepQSO9zofmXZ8tduRkU3929\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\".\"},\"logprobs\":null,\"finish_reason\":null}],\"usage\":null,\"obfuscation\":\"epYTgq2pXsh1Lp\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjaepQSO9zofmXZ8tduRkU3929\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[{\"index\":0,\"delta\":{},\"logprobs\":null,\"finish_reason\":\"stop\"}],\"usage\":null,\"obfuscation\":\"nwFLoPGYk\"}\n\ndata: {\"id\":\"chatcmpl-EYFXjaepQSO9zofmXZ8tduRkU3929\",\"object\":\"chat.completion.chunk\",\"created\":1791831607,\"model\":\"gpt-4.1-2025-04-14\",\"service_tier\":\"default\",\"system_fingerprint\":\"fp_99c6909298\",\"choices\":[],\"usage\":{\"prompt_tokens\":194,\"completion_tokens\":19,\"total_tokens\":213,\"prompt_tokens_details\":{\"cached_tokens\":0,\"audio_tokens\":0},\"completion_tokens_details\":{\"reasoning_tokens\":0,\"audio_tokens\":0,\"accepted_prediction_tokens\":0,\"rejected_prediction_tokens\":0}},\"obfuscation\":\"WxxeqAvN1ywr\"}\n\ndata: [DONE]\n\n" + } +] diff --git a/test/fixtures/openai-length-tool.sse b/test/fixtures/openai-length-tool.sse new file mode 100644 --- /dev/null +++ b/test/fixtures/openai-length-tool.sse @@ -0,0 +1,8 @@ +data: {"id":"c2","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","tool_calls":[{"index":0,"id":"call_1","type":"function","function":{"name":"get_weather","arguments":""}}]},"finish_reason":null}]} + +data: {"id":"c2","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"location\": \"Lon"}}]},"finish_reason":null}]} + +data: {"id":"c2","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"length"}]} + +data: [DONE] + diff --git a/test/fixtures/openai-midstream-error.sse b/test/fixtures/openai-midstream-error.sse new file mode 100644 --- /dev/null +++ b/test/fixtures/openai-midstream-error.sse @@ -0,0 +1,6 @@ +data: {"id":"c1","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":"Partial answer"},"finish_reason":null}]} + +data: {"error":{"message":"Rate limit reached","type":"rate_limit_exceeded","code":null}} + +data: [DONE] + diff --git a/test/fixtures/openai-multi-tool-chunk.sse b/test/fixtures/openai-multi-tool-chunk.sse new file mode 100644 --- /dev/null +++ b/test/fixtures/openai-multi-tool-chunk.sse @@ -0,0 +1,8 @@ +data: {"id":"c3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","tool_calls":[{"index":0,"id":"call_a","type":"function","function":{"name":"get_weather","arguments":"{\"location\": "}},{"index":1,"id":"call_b","type":"function","function":{"name":"get_time","arguments":"{\"zone\": "}}]},"finish_reason":null}]} + +data: {"id":"c3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"function":{"arguments":"\"CET\"}"}},{"index":0,"function":{"arguments":"\"London\"}"}}]},"finish_reason":null}]} + +data: {"id":"c3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]} + +data: [DONE] + diff --git a/test/fixtures/openai-text.json b/test/fixtures/openai-text.json new file mode 100644 --- /dev/null +++ b/test/fixtures/openai-text.json @@ -0,0 +1,20 @@ +{ + "id": "chatcmpl-abc123", + "object": "chat.completion", + "model": "gpt-4o-mini", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "Hello! How can I help you today?" + }, + "finish_reason": "stop" + } + ], + "usage": { + "prompt_tokens": 15, + "completion_tokens": 9, + "total_tokens": 24 + } +} diff --git a/test/fixtures/openai-tool-use.json b/test/fixtures/openai-tool-use.json new file mode 100644 --- /dev/null +++ b/test/fixtures/openai-tool-use.json @@ -0,0 +1,30 @@ +{ + "id": "chatcmpl-abc456", + "object": "chat.completion", + "model": "gpt-4o-mini", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_abc123", + "type": "function", + "function": { + "name": "get_weather", + "arguments": "{\"location\":\"London\"}" + } + } + ] + }, + "finish_reason": "tool_calls" + } + ], + "usage": { + "prompt_tokens": 30, + "completion_tokens": 18, + "total_tokens": 48 + } +} diff --git a/test/fixtures/providers-ollama-only.json b/test/fixtures/providers-ollama-only.json new file mode 100644 --- /dev/null +++ b/test/fixtures/providers-ollama-only.json @@ -0,0 +1,7 @@ +[ + { + "providerName": "ollama", + "protocol": "ollama", + "baseUrl": "http://localhost:11434" + } +] diff --git a/test/fixtures/providers-openrouter-only.json b/test/fixtures/providers-openrouter-only.json new file mode 100644 --- /dev/null +++ b/test/fixtures/providers-openrouter-only.json @@ -0,0 +1,8 @@ +[ + { + "providerName": "openrouter", + "protocol": "openai", + "baseUrl": "https://openrouter.ai/api", + "apiKeyEnv": "OPENROUTER_API_KEY" + } +] diff --git a/test/fixtures/providers-openrouter.json b/test/fixtures/providers-openrouter.json new file mode 100644 --- /dev/null +++ b/test/fixtures/providers-openrouter.json @@ -0,0 +1,8 @@ +[ + { + "providerName": "openrouter", + "protocol": "openai", + "baseUrl": "https://openrouter.ai/api", + "apiKeyEnv": "OPENROUTER_API_KEY" + } +]