diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,206 @@ ## [Unreleased] +## [baikai 0.7.0.0] - 2026-09-08 + +### Added + +- `BaikaiError.refusalCategory` preserves an Anthropic refusal's + provider category. JSON adds `refusal_category`; older errors still decode. + Evidence schema 2.5 records the addition without changing digest inputs. + __Breaking__ to construct a `BaikaiError` from its full field list. + +- `Speed`, `Options.speed`, catalog-owned fast rates and + `computeCostAtSpeed`. Anthropic gates fast mode by model capability, adds the + beta header and records unsupported drops. Terminal pricing uses observed + speed, including cache duration; unreported speed is an explicit estimate. + Older `Model` JSON defaults the new fields safely. __Breaking__: public + records and sum types gain fields and constructors. + +- API usage now records observed service tiers, inference speed and server-tool + use in optional billing facts covered by evidence schema 2.2. Missing service + information and uncurated products produce explicit standard-rate estimates. + `computeCostForService` separates requested and observed service, while + `computeCostAtRates` prices a resolved rate set once for future speed policies. + Empty billing facts preserve legacy availability JSON; a CLI-reported zero + cost retains its reported-total source. __Breaking__: the public vocabulary + and records gain members. + +- Failed trace terminals now retain partial response token counts, cost basis, + usage availability and USD totals. Synthetic aborts leave unreported billing + absent; legacy failed trace JSON still decodes. __Breaking__: `TraceEvent`'s + `CallFailed` gains fields. See `baikai-trace-otel 0.4.0.1` for the export. + +- Successful trace terminals and call-log records carry optional cost basis and + usage availability; call logs also carry cache-write counts. Old JSON decodes + with absent metadata and empty additive-zero bases stay omitted from traces. + __Breaking__: `CallFinished` and the call-log record gain fields. + +- Optional `Usage.availability` and shared inclusive/exclusive billing + normalization, in the new `Baikai.Usage.Normalize`. OpenAI Chat/Responses and + Claude preserve missing cache counters as explicit estimation reasons, + distinguish reported zeroes, and merge cumulative usage without + double-counting. Schema 2.2 commits provider availability while preserving + legacy usage digests. __Breaking__: `Usage` gains a field. + +- Optional `Model.pricingPolicy`, exact whole-request context tiers, + and an explicit cache-duration rate resolver. Generated Astra pricing changes + above 272000 input tokens; Fable exposes its one-hour write price. `Cost.basis` + preserves calculation sources and estimation reasons when summed. Evidence + schema 2.2 serializes the local basis without including local pricing metadata + in provider commitments. __Breaking__: `Model` and `Cost` gain fields. + +- Separate `OpenAIResponses` dispatch and compatibility types, and + optional provider/model-scoped `ThinkingContent.replayState` with opaque + diagnostic output and backward-compatible JSON decoding. Evidence schema 2.2 + includes replay state and optional billing facts in commitments while preserving + legacy encodings when those fields are absent. __Breaking__ for a `case` over + `Api` that is exhaustive without a wildcard. + +- `Baikai.Evidence.ThinkingTranslation` gains `displayText` and + `ThinkingAdjustment` gains `ThinkingSummaryUnavailable`, so a transport can + record the thinking display setting it asked for and diagnose a successful + response whose thinking blocks carry no readable summary. `Baikai.Compat` + gains `supportsForcedToolChoice`; legacy JSON defaults it to True. + __Breaking__ for an exhaustive `case` over `ThinkingAdjustment`. + +- GPT-6 Astra and Claude Fable 5.1 catalog bindings, with verified + pricing, token limits, and Anthropic thinking/sampling compatibility. + +- Repository `update-models` skill for verifying provider releases and refreshing + the curated JSON and generated Haskell catalog. + +### Fixed + +- Preserve OpenAI endpoint capability facts through catalog refreshes. + +- Chat and Claude reject provider-scoped reasoning replay they cannot encode. + +- Widened the `http-client-tls` bound to admit 0.4 (carried forward from the + tagged but never-published 0.6.0.1). + +## [baikai-claude 0.7.0.0] - 2026-09-08 + +### Changed + +- Refusal messages include the reported category and explanation, + retaining the original message when neither exists. Classification remains + non-retryable `ContentFiltered`. Server-side fallbacks remain deliberately + unsupported, as recorded in ADR 0005. + +- Adaptive reasoning requests explicitly ask for summarized + thinking. Evidence schema 2.4 records the display setting and diagnoses + successful responses whose thinking blocks contain no readable summary. + Budget and absent-thinking request shapes, signed empty blocks, redacted + content and multi-turn replay are preserved. + +- Fast mode is gated by the generated model capability: it adds the Anthropic + beta header for a model that advertises it and records an evidence adjustment + for one that does not. + +### Fixed + +- Price Fable cache writes using the TTL in the shaped request, + including compatibility downgrades. Missing write-duration context is explicit + in the cost basis. + +- Reject forced tool choices locally on Fable 5.1, using the + generated `supportsForcedToolChoice` capability. Automatic tool rounds retain + signed empty/visible thinking, redacted blocks and prior-message order. + +- Widened the `http-client-tls` bound to admit 0.4 (carried forward from the + tagged but never-published 0.6.0.1). + +### Changed (dependencies) + +- Requires `baikai ^>=0.7.0`. + +## [baikai-openai 0.7.0.0] - 2026-09-08 + +### Added + +- Explicit `Baikai.Provider.OpenAI.Responses` registration and + stream/complete provider with stateless reasoning replay, function tool turns, + structured output and bounded worker cleanup, across the new + `Baikai.Provider.OpenAI.Responses.{Request,Stream,Assembler}` modules. Astra + now selects this provider through a per-model catalog override; callers must + register it explicitly. Cache writes, billing availability and context pricing + are integrated. + +- `Baikai.Provider.OpenAI.Internal.Usage`, the shared usage mapping both the + Chat Completions and Responses transports read. + +### Fixed + +- Reject tools locally for models whose Chat Completions endpoint + disallows them, including GPT-6 Astra. Respect generated effort policies and + sampling restrictions, with matching translation evidence and strict refusal. + +- Validate Responses terminals and enforce the stream contracts. + +- Widened the `http-client-tls` bound to admit 0.4 (carried forward from the + tagged but never-published 0.6.0.1). + +### Changed (dependencies) + +- Requires `baikai ^>=0.7.0`. + +## [baikai-trace-otel 0.4.0.1] - 2026-09-08 + +### Added + +- Successful and failed spans export `baikai.cost.basis` and + `baikai.usage.availability` as canonically encoded JSON. A failed span now + also carries the input/output token counts and USD total that + `baikai 0.7.0.0` retains on `CallFailed`, alongside its error status. + +### Changed (dependencies) + +- Requires `baikai ^>=0.7.0`, and now depends on `aeson ^>=2.2` to encode the + two new attributes. + +## [baikai-effectful 0.4.0.1] - 2026-09-08 + +### Changed (dependencies) + +- Requires `baikai ^>=0.7.0`. No API change. + +## [baikai-kit 0.2.0.1] - 2026-09-08 + +### Changed (dependencies) + +- Requires `baikai ^>=0.7.0`. No API change. + +## [baikai-agent 0.2.0.1] - 2026-09-08 + +### Changed (dependencies) + +- Requires `baikai ^>=0.7.0`, `baikai-claude ^>=0.7` and + `baikai-openai ^>=0.7`. No API change. + +## [baikai 0.6.0.1] - 2026-08-30 + +### Fixed + +- widened the `http-client-tls` bound to admit 0.4. The 0.4 API retains the + manager functions this package uses and belongs to the same TLS 2.x / Crypton + 1.1 dependency cohort as baikai 0.6; the old `^>=0.3` cap made baikai 0.6 + impossible to solve in applications that require Crypton 1.1. + +## [baikai-claude 0.6.0.1] - 2026-08-30 + +### Fixed + +- widened the `http-client-tls` bound to admit 0.4, allowing applications that + require Crypton 1.1 to solve the dependency set. + +## [baikai-openai 0.6.0.1] - 2026-08-30 + +### Fixed + +- widened the `http-client-tls` bound to admit 0.4, allowing applications that + require Crypton 1.1 to solve the dependency set. + ## [baikai 0.6.0.0] - 2026-08-28 ### Added diff --git a/baikai-trace-otel.cabal b/baikai-trace-otel.cabal --- a/baikai-trace-otel.cabal +++ b/baikai-trace-otel.cabal @@ -1,6 +1,6 @@ cabal-version: 3.4 name: baikai-trace-otel -version: 0.4.0.0 +version: 0.4.0.1 synopsis: OpenTelemetry TraceSink for baikai. description: Provides an opt-in OpenTelemetry adapter for the baikai 'TraceSink' @@ -52,7 +52,8 @@ hs-source-dirs: src exposed-modules: Baikai.Trace.Sink.OpenTelemetry build-depends: - , baikai ^>=0.6.0 + , aeson ^>=2.2 + , baikai ^>=0.7.0 , base >=4.20 && <5 , containers ^>=0.7 , hs-opentelemetry-api >=1.0 && <1.1 @@ -72,7 +73,7 @@ ghc-options: -threaded -with-rtsopts=-N build-depends: , aeson ^>=2.2 - , baikai ^>=0.6.0 + , baikai ^>=0.7.0 , baikai-trace-otel , base , generic-lens diff --git a/src/Baikai/Trace/Sink/OpenTelemetry.hs b/src/Baikai/Trace/Sink/OpenTelemetry.hs --- a/src/Baikai/Trace/Sink/OpenTelemetry.hs +++ b/src/Baikai/Trace/Sink/OpenTelemetry.hs @@ -33,12 +33,14 @@ import Baikai.Trace.Event (TraceEvent (..)) import Baikai.Trace.Sink (TraceSink (..)) import Control.Monad (forM_) +import Data.Aeson qualified as Aeson import Data.HashMap.Strict qualified as HashMap import Data.Int (Int64) import Data.Map.Strict qualified as Map import Data.Maybe (fromMaybe) import Data.Scientific qualified as Scientific import Data.Text (Text) +import Data.Text.Encoding qualified as Text import Data.Time (UTCTime) import Data.Time.Clock.POSIX (utcTimeToPOSIXSeconds) import Data.Word (Word64) @@ -144,7 +146,7 @@ -- evidence, and, because 'Otel.addAttributes' replaces an existing key -- and evidence is pushed before the terminal, overwrote the genuinely -- observed value on every call that did. - CallFinished {eventId, timestamp, latencyMs, inputTokens, outputTokens, usd} -> + CallFinished {eventId, timestamp, latencyMs, inputTokens, outputTokens, usd, costBasis, usageAvailability} -> case Map.lookup eventId m of -- A terminal without a live span is unreachable for normal withTraceStream -- usage because each traced call drives a fresh fold. Keep the silent drop so @@ -155,15 +157,17 @@ attrs = maybe id (\n -> AttrMap.insertByKey SC.genAi_usage_inputTokens (fromIntegral n :: Int64)) inputTokens $ maybe id (\n -> AttrMap.insertByKey SC.genAi_usage_outputTokens (fromIntegral n :: Int64)) outputTokens $ - maybe id (\s -> HashMap.insert "baikai.cost.usd" (Attr.toAttribute (Scientific.toRealFloat s :: Double))) usd $ - HashMap.fromList - [ ("baikai.latency_ms", Attr.toAttribute latencyMs) - ] + maybe id (\b -> HashMap.insert "baikai.cost.basis" (Attr.toAttribute (Text.decodeUtf8 (Ev.canonicalEncode (Aeson.toJSON b))))) costBasis $ + maybe id (\a -> HashMap.insert "baikai.usage.availability" (Attr.toAttribute (Text.decodeUtf8 (Ev.canonicalEncode (Aeson.toJSON a))))) usageAvailability $ + maybe id (\s -> HashMap.insert "baikai.cost.usd" (Attr.toAttribute (Scientific.toRealFloat s :: Double))) usd $ + HashMap.fromList + [ ("baikai.latency_ms", Attr.toAttribute latencyMs) + ] Otel.addAttributes sp attrs Otel.setStatus sp Otel.Ok Otel.endSpan sp (Just (utcToTimestamp timestamp)) pure (Map.delete eventId m) - CallFailed {eventId, timestamp, latencyMs, errorMessage} -> + CallFailed {eventId, timestamp, latencyMs, errorMessage, inputTokens, outputTokens, usd, costBasis, usageAvailability} -> case Map.lookup eventId m of -- A terminal without a live span is unreachable for normal withTraceStream -- usage because each traced call drives a fresh fold. Keep the silent drop so @@ -171,10 +175,15 @@ Nothing -> pure m Just sp -> do Otel.addAttributes sp $ - HashMap.fromList - [ ("baikai.latency_ms", Attr.toAttribute latencyMs), - ("baikai.error", Attr.toAttribute errorMessage) - ] + maybe id (\n -> AttrMap.insertByKey SC.genAi_usage_inputTokens (fromIntegral n :: Int64)) inputTokens $ + maybe id (\n -> AttrMap.insertByKey SC.genAi_usage_outputTokens (fromIntegral n :: Int64)) outputTokens $ + maybe id (\s -> HashMap.insert "baikai.cost.usd" (Attr.toAttribute (Scientific.toRealFloat s :: Double))) usd $ + maybe id (\b -> HashMap.insert "baikai.cost.basis" (Attr.toAttribute (Text.decodeUtf8 (Ev.canonicalEncode (Aeson.toJSON b))))) costBasis $ + maybe id (\a -> HashMap.insert "baikai.usage.availability" (Attr.toAttribute (Text.decodeUtf8 (Ev.canonicalEncode (Aeson.toJSON a))))) usageAvailability $ + HashMap.fromList + [ ("baikai.latency_ms", Attr.toAttribute latencyMs), + ("baikai.error", Attr.toAttribute errorMessage) + ] Otel.setStatus sp (Otel.Error errorMessage) Otel.endSpan sp (Just (utcToTimestamp timestamp)) pure (Map.delete eventId m) diff --git a/test/Main.hs b/test/Main.hs --- a/test/Main.hs +++ b/test/Main.hs @@ -5,12 +5,14 @@ import Baikai.Api (Api (..)) import Baikai.Content (AssistantContent (..), TextContent (..)) import Baikai.Context (Context (..), emptyContext) +import Baikai.Cost qualified as Cost import Baikai.Error (BaikaiError, providerError) import Baikai.Evidence ( CallStatus (..), EvidenceRequest, Observed (..), TransportKind (..), + canonicalEncode, evidenceRequest, noThinkingRequested, ) @@ -41,6 +43,7 @@ import Data.IORef (IORef, readIORef) import Data.Text (Text) import Data.Text qualified as Text +import Data.Text.Encoding qualified as TextEncoding import Data.Time (getCurrentTime) import Data.Vector qualified as V import OpenTelemetry.Attributes qualified as Attr @@ -62,6 +65,7 @@ [ successSpanTest, observedModelSpanTest, failureSpanTest, + partialBillingSpanTest, abortSpanTest, evidenceSpanTest, liveEvidenceSpanTest, @@ -93,6 +97,7 @@ & #inputTokens .~ 12 & #outputTokens .~ 3 & #totalTokens .~ 15 + & #cost .~ Cost.estimateCost [Cost.ServiceTierNotReported] Cost.zeroCost stubResponse :: Api -> Response stubResponse a = @@ -178,6 +183,7 @@ (not (HashMap.member "gen_ai.response.model" attrs)) assertBool "has gen_ai.usage.input_tokens" (HashMap.member "gen_ai.usage.input_tokens" attrs) assertBool "has gen_ai.usage.output_tokens" (HashMap.member "gen_ai.usage.output_tokens" attrs) + HashMap.lookup "baikai.cost.basis" attrs @?= Just (Attr.toAttribute (TextEncoding.decodeUtf8 (canonicalEncode (Aeson.toJSON (sampleUsage ^. #cost . #basis))))) assertBool "has baikai.event_id" (HashMap.member "baikai.event_id" attrs) assertBool "has baikai.latency_ms" (HashMap.member "baikai.latency_ms" attrs) assertBool "does not emit deprecated GenAI system key" (not (HashMap.member deprecatedGenAiSystemKey attrs)) @@ -244,6 +250,29 @@ "the call span records a parent" (maybe False (const True) (Otel.spanParent sp)) other -> assertFailure ("expected one baikai.call span, got " <> show (length other)) + +partialBillingSpanTest :: TestTree +partialBillingSpanTest = + testCase "failed span retains partial usage and calculation basis" $ do + let a = Custom "baikai-otel-partial-cost" + partial = stubResponse a & #message . #stopReason .~ ErrorReason & #message . #errorMessage .~ Just "connection reset" & #message . #usage . #cost . #usd .~ (3 / 1000000) & #message . #usage . #cost . #breakdown . #inputUsd .~ (3 / 1000000) + handler _ _ _ = pure partial + registerApiProvider (apiProviderWith a (liftCompleteToStream handler) handler) + (tracer, getSpans) <- newTracerWithInMemory + response <- withTrace (otelSink tracer) (stubModel a) stubContext stubOptions + spans <- getSpans + case spans of + [sp] -> do + hot <- spanHotSnapshot sp + let attrs = Attr.getAttributeMap (Otel.hotAttributes hot) + usage = response ^. #message . #usage + HashMap.lookup "baikai.cost.basis" attrs @?= Just (Attr.toAttribute (TextEncoding.decodeUtf8 (canonicalEncode (Aeson.toJSON (usage ^. #cost . #basis))))) + HashMap.lookup "baikai.cost.usd" attrs @?= Just (Attr.toAttribute (fromRational (usage ^. #cost . #usd) :: Double)) + assertBool "partial input count reaches span" (HashMap.member "gen_ai.usage.input_tokens" attrs) + case Otel.hotStatus hot of + Otel.Error _ -> pure () + other -> assertFailure ("expected error status: " <> show other) + other -> assertFailure ("expected one span: " <> show (length other)) failureSpanTest :: TestTree failureSpanTest =