packages feed

shikumi-compile (empty) → 0.1.0.0

raw patch · 13 files changed

+978/−0 lines, 13 filesdep +aesondep +baikaidep +base

Dependencies added: aeson, baikai, base, bytestring, effectful, generic-lens, lens, shikumi, shikumi-compile, tasty, tasty-hunit, text, vector

Files

+ CHANGELOG.md view
@@ -0,0 +1,10 @@+# Changelog++## Unreleased++## 0.1.0.0 - 2026-06-13++### Added++- Initial Hackage release of pure compiler transformations for shikumi programs.+- Zero-shot, few-shot, chain-of-thought, retrieval-augmented generation, retriever, compiled-program, and serialization APIs.
+ shikumi-compile.cabal view
@@ -0,0 +1,81 @@+cabal-version:   3.4+name:            shikumi-compile+version:         0.1.0.0+synopsis:        The compiler layer for shikumi LM programs (EP-9)+category:        AI+description:+  The compiler layer for shikumi: pure @Program -> Program@ transformations that+  bake a prompting strategy into a program's per-node parameters or structure+  (zero-shot, few-shot, chain-of-thought, retrieval-augmented), the owned type+  @CompiledProgram@ (MasterPlan integration point #6), a minimal @Retriever@+  interface with a trivial in-memory implementation, and parameter-state+  serialization mirroring DSPy's @dump_state@/@load_state@. A compiler never calls+  the LM to try variations and keep the best — that is the optimizer (EP-10). Every+  acceptance assertion is about the prompt the model would have seen, observed+  offline through a capturing stub, so the whole package tests with no network.++license:         BSD-3-Clause+author:          Nadeem Bitar+maintainer:      nadeem@gmail.com+build-type:      Simple+extra-doc-files: CHANGELOG.md++common common-options+  ghc-options:+    -Wall -Wcompat -Widentities -Wincomplete-uni-patterns+    -Wincomplete-record-updates -Wredundant-constraints+    -fhide-source-paths -Wmissing-export-lists -Wpartial-fields+    -Wmissing-deriving-strategies++  default-language:   GHC2024+  default-extensions:+    DeriveAnyClass+    DuplicateRecordFields+    OverloadedLabels+    OverloadedStrings++library+  import:          common-options+  hs-source-dirs:  src+  exposed-modules:+    Shikumi.Compile+    Shikumi.Compile.ChainOfThought+    Shikumi.Compile.FewShot+    Shikumi.Compile.RAG+    Shikumi.Compile.Retriever+    Shikumi.Compile.Serialize+    Shikumi.Compile.Types+    Shikumi.Compile.ZeroShot++  build-depends:+    , aeson+    , base        >=4.20     && <5+    , bytestring+    , effectful+    , shikumi     ^>=0.1.0.0+    , text        ^>=2.1++test-suite shikumi-compile-test+  import:         common-options+  type:           exitcode-stdio-1.0+  hs-source-dirs: test+  main-is:        Main.hs+  ghc-options:    -threaded -with-rtsopts=-N+  other-modules:+    Test.Capture+    Test.Fixtures++  build-depends:+    , aeson+    , baikai+    , base+    , bytestring+    , effectful+    , generic-lens+    , lens+    , shikumi          ^>=0.1.0.0+    , shikumi-compile  ^>=0.1.0.0+    , tasty+    , tasty-hunit+    , text+    , vector
+ src/Shikumi/Compile.hs view
@@ -0,0 +1,55 @@+-- | The shikumi compiler layer (EP-9): pure @Program -> Program@ transformations+-- that bake a prompting strategy into a program.+--+-- A /compiler/ (in the DSPy sense the user is porting) sets a program's per-node+-- parameters and/or rewrites its structure according to a fixed recipe — zero-shot,+-- few-shot, chain-of-thought, retrieval-augmented. It is pure and never calls the+-- LM; searching for better parameters is the optimizer's job (EP-10).+--+-- @+-- import Shikumi.Compile+--+-- qaFewShot :: CompiledProgram Question Answer+-- qaFewShot = compile (fewShotTyped [(Question \"2+2?\", Answer \"4\")]) qa+--+-- qaCoT :: CompiledProgram Question Answer+-- qaCoT = compile chainOfThoughtCompiler qa+-- @+--+-- This module re-exports the full public surface; see the per-strategy modules for+-- detail and caveats.+module Shikumi.Compile+  ( -- * Types+    Compiler (..),+    CompiledProgram (..),+    compile,+    runCompiled,+    identity,++    -- * Strategies+    zeroShot,+    zeroShotClear,+    fewShot,+    fewShotTyped,+    chainOfThoughtCompiler,+    rag,++    -- * Retrieval+    Retriever (..),+    Passage (..),+    inMemoryRetriever,+    formatPassages,++    -- * Serialization+    encodeCompiled,+    decodeCompiledOnto,+  )+where++import Shikumi.Compile.ChainOfThought (chainOfThoughtCompiler)+import Shikumi.Compile.FewShot (fewShot, fewShotTyped)+import Shikumi.Compile.RAG (formatPassages, rag)+import Shikumi.Compile.Retriever (Passage (..), Retriever (..), inMemoryRetriever)+import Shikumi.Compile.Serialize (decodeCompiledOnto, encodeCompiled)+import Shikumi.Compile.Types (CompiledProgram (..), Compiler (..), compile, identity, runCompiled)+import Shikumi.Compile.ZeroShot (zeroShot, zeroShotClear)
+ src/Shikumi/Compile/ChainOfThought.hs view
@@ -0,0 +1,78 @@+{-# LANGUAGE GADTs #-}+{-# LANGUAGE ScopedTypeVariables #-}++-- | The chain-of-thought compiler: ensure every LM-call node reasons step by step+-- before answering.+--+-- __Implementation: a structural rewrite (the plan's preferred form).__ EP-4 has+-- no @reasoning@ flag on 'Shikumi.Program.Params' to flip; instead, reasoning is a+-- /structural/ property — 'Shikumi.Module.chainOfThought' augments a signature's+-- output with a leading @reasoning@ field and amends its instruction to "think step+-- by step", then projects the answer back out with @FMap@. So this compiler walks+-- the program and replaces each @Predict sig ps@ leaf with exactly that shape:+-- @FMap value (chainOfThoughtRaw sig)@, carrying the node's existing 'Params'+-- across. The rewrite is type-preserving (a @Program i o@ stays a @Program i o@)+-- because the augmented @Program i (WithReasoning o)@ is mapped back through+-- 'Shikumi.Module.value'.+--+-- This is feasible — and faithful — only because EP-4 /exports the GADT+-- constructors/, so a downstream package can pattern-match and rebuild nodes. EP-4+-- does not ship a node-level @rewriteNodes@ helper, so the recursion is spelled out+-- here over every constructor (the EP-4/EP-5 rule: every function over the GADT+-- must match every constructor).+--+-- __Caveat (recorded in the plan's Decision Log).__ The node's existing 'Params'+-- are preserved verbatim. For the common case — compiling a base (uncompiled)+-- program whose nodes carry 'Shikumi.Program.emptyParams' — this reproduces+-- 'Shikumi.Module.chainOfThought' exactly. If a node already carries an+-- @instructionOverride@, that override still replaces the augmented instruction at+-- run time (so the cue is lost): apply chain-of-thought /before/ a zero-shot+-- instruction, or fold the cue into the instruction yourself. If a node already+-- carries demos, they must be chain-of-thought-shaped (include a @reasoning@ field)+-- to decode under the augmented output; otherwise compile chain-of-thought before+-- few-shot.+module Shikumi.Compile.ChainOfThought+  ( chainOfThoughtCompiler,+  )+where++import Shikumi.Compile.Types (Compiler (..))+import Shikumi.Module (chainOfThoughtRaw, value)+import Shikumi.Program+  ( Program+      ( Compose,+        Embed,+        Ensemble,+        FMap,+        MajorityVote,+        Map,+        Parallel,+        Predict,+        Retry,+        RetryWhen,+        Validate+      ),+    mapParams,+  )++-- | Rewrite every 'Predict' node into its chain-of-thought form, recursing through+-- every composite/combinator node so nodes nested arbitrarily deep are reached.+chainOfThoughtCompiler :: Compiler+chainOfThoughtCompiler = Compiler cot++-- | The structural rewrite, polymorphic in @i@/@o@ so it threads through the GADT's+-- existential children. At a 'Predict' leaf the constructor's captured dictionaries+-- ('Shikumi.Schema.FromModel' @i@/@o@, 'Shikumi.Schema.ToSchema' @o@, the+-- 'Shikumi.Adapter.ToPrompt's) are exactly what 'chainOfThoughtRaw' needs.+cot :: Program i o -> Program i o+cot (Predict sig ps) = FMap value (mapParams (const ps) (chainOfThoughtRaw sig))+cot (Compose a b) = Compose (cot a) (cot b)+cot (FMap k p) = FMap k (cot p)+cot (Map w p) = Map w (cot p)+cot (Parallel a b) = Parallel (cot a) (cot b)+cot (Retry n p) = Retry n (cot p)+cot (RetryWhen ok n p) = RetryWhen ok n (cot p)+cot (Validate v p) = Validate v (cot p)+cot (MajorityVote k sched p) = MajorityVote k sched (cot p)+cot (Ensemble ps reduce) = Ensemble (map cot ps) reduce+cot (Embed f) = Embed f -- an agent node has no 'Predict' to rewrite; pass through
+ src/Shikumi/Compile/FewShot.hs view
@@ -0,0 +1,35 @@+-- | The few-shot compiler: inject a /static/ list of demonstrations into every+-- LM-call node's parameters. This is DSPy's @LabeledFewShot@ and the headline+-- compiler of the plan.+--+-- The injected 'Demo's are stored as type-agnostic JSON (EP-4's+-- 'Shikumi.Program.Demo' carries @input@/@output@ as aeson @Value@s), so a+-- single demo pool attaches uniformly to every node regardless of its signature.+-- At run time each node's adapter (EP-3) renders the fields it recognizes; a demo+-- authored for one node's shape simply renders what overlaps at a node with a+-- different shape. Use 'Shikumi.Compile.FewShot.fewShotTyped' to build demos from+-- typed pairs.+module Shikumi.Compile.FewShot+  ( fewShot,+    fewShotTyped,+  )+where++import Data.Aeson (ToJSON, toJSON)+import Shikumi.Compile.Types (Compiler (..))+import Shikumi.Program (Demo (..), Params (..), mapParams)++-- | Inject the given demos at every node, /replacing/ (not appending) the node's+-- demo list so that re-compiling is idempotent — compiling twice yields the same+-- demos, never duplicates. (If append semantics are ever wanted that is a separate+-- combinator, deliberately out of scope here.)+fewShot :: [Demo] -> Compiler+fewShot ds = Compiler $ mapParams (\ps -> ps {demos = ds})++-- | Build a few-shot compiler from typed input/output pairs, serializing each to a+-- JSON 'Demo'. This is the recommended path when demos must line up with a node's+-- record fields: the JSON keys are the record field names, so the adapter renders+-- them precisely.+fewShotTyped :: (ToJSON i, ToJSON o) => [(i, o)] -> Compiler+fewShotTyped pairs =+  fewShot [Demo {input = toJSON i, output = toJSON o} | (i, o) <- pairs]
+ src/Shikumi/Compile/RAG.hs view
@@ -0,0 +1,90 @@+{-# LANGUAGE GADTs #-}+{-# LANGUAGE ScopedTypeVariables #-}++-- | The retrieval-augmented (RAG) compiler: install retrieved context so that the+-- rendered prompt at every LM-call node carries the passages a retriever found.+--+-- __Purity vs. retrieval (how the plan's question is resolved here).__ 'compile' is+-- pure, but retrieval fetches data. EP-4 ships /no/ effectful escape-hatch node+-- (no @embed@ / @Embed@ constructor that would let an @i -> Eff es o@ become a+-- @Program@ node), so the plan's "approach 1" (install a runtime retrieval step+-- keyed on the actual input) is not available. This is the plan's documented+-- /fallback/: retrieve at /compile time/ against a fixed sample query, then inject+-- the top passages into every node's signature instruction. 'compile' stays pure+-- because the trivial 'Shikumi.Compile.Retriever.inMemoryRetriever' performs no+-- effect — it is run via 'runPureEff'. The limitation is that retrieval is+-- query-independent of the actual program input; wiring true per-input retrieval+-- awaits an EP-4 embed node and is left as a TODO.+--+-- The injection is a /structural/ rewrite (like "Shikumi.Compile.ChainOfThought"):+-- each @Predict sig ps@ leaf becomes @Predict (setInstruction (base <> context)+-- sig) ps@, so the original task instruction is preserved /and/ the retrieved+-- context is added (it appears in the rendered system prompt). The node's existing+-- 'Shikumi.Program.Params' are untouched; the same @instructionOverride@-precedence+-- caveat as chain-of-thought applies (an override set on a node replaces the whole+-- instruction at run time, context included), so apply 'rag' before a zero-shot+-- instruction if both are wanted.+module Shikumi.Compile.RAG+  ( rag,+    formatPassages,+  )+where++import Data.Text (Text)+import Data.Text qualified as T+import Effectful (runPureEff)+import Shikumi.Compile.Retriever (Passage (..), Retriever (..))+import Shikumi.Compile.Types (Compiler (..))+import Shikumi.Program+  ( Program+      ( Compose,+        Embed,+        Ensemble,+        FMap,+        MajorityVote,+        Map,+        Parallel,+        Predict,+        Retry,+        RetryWhen,+        Validate+      ),+  )+import Shikumi.Signature (getInstruction, setInstruction)++-- | Install retrieved context at every node. Retrieval happens once, now, against+-- @query@ (the documented compile-time fallback); the formatted passages are then+-- prepended-as-context to every 'Shikumi.Program.Predict' node's instruction.+rag :: Retriever -> Text -> Compiler+rag r query = Compiler (install context)+  where+    passages = runPureEff (retrieve r query)+    context = formatPassages passages++-- | Render retrieved passages as a labeled context block for the prompt. Empty+-- input yields the empty string (no spurious header).+formatPassages :: [Passage] -> Text+formatPassages [] = ""+formatPassages ps =+  "Use the following retrieved context to answer:\n"+    <> T.unlines ["- " <> text p | p <- ps]++-- | Append the context block to every node's signature instruction, recursing+-- through every composite/combinator node. A no-op when @ctx@ is empty.+install :: Text -> Program i o -> Program i o+install ctx+  | T.null ctx = id+  | otherwise = go+  where+    go :: forall i o. Program i o -> Program i o+    go (Predict sig ps) = Predict (setInstruction (getInstruction sig <> "\n\n" <> ctx) sig) ps+    go (Compose a b) = Compose (go a) (go b)+    go (FMap k p) = FMap k (go p)+    go (Map w p) = Map w (go p)+    go (Parallel a b) = Parallel (go a) (go b)+    go (Retry n p) = Retry n (go p)+    go (RetryWhen ok n p) = RetryWhen ok n (go p)+    go (Validate v p) = Validate v (go p)+    go (MajorityVote k sched p) = MajorityVote k sched (go p)+    go (Ensemble ps reduce) = Ensemble (map go ps) reduce+    go (Embed f) = Embed f -- an agent node has no 'Predict' to rewrite; pass through
+ src/Shikumi/Compile/Retriever.hs view
@@ -0,0 +1,47 @@+{-# LANGUAGE RankNTypes #-}++-- | The minimal retrieval interface and the one trivial implementation shikumi+-- ships. Production retrievers (vector stores, search services) are explicitly out+-- of scope for shikumi (MasterPlan "Out of scope"); this defines the /interface/ a+-- real one would later satisfy.+module Shikumi.Compile.Retriever+  ( Passage (..),+    Retriever (..),+    inMemoryRetriever,+  )+where++import Data.List (sortOn)+import Data.Ord (Down (..))+import Data.Text (Text)+import Data.Text qualified as T+import Effectful (Eff)++-- | A unit of retrievable text.+data Passage = Passage+  { passageId :: !Text,+    text :: !Text+  }+  deriving stock (Eq, Show)++-- | Given a query, return relevant passages ranked best-first. Effect-polymorphic+-- (@forall es@) so the interface is honest about a /real/ retriever running under a+-- program's effect stack — though the trivial implementation below performs no+-- effect, and the RAG compiler runs retrieval purely at compile time (see+-- "Shikumi.Compile.RAG"). A real IO-backed retriever would define its own concrete+-- type with the effects it needs rather than satisfy this @forall es@ signature.+newtype Retriever = Retriever+  { retrieve :: forall es. Text -> Eff es [Passage]+  }++-- | A brute-force in-memory retriever: rank passages by the number of shared+-- (lowercased, whitespace-split) word tokens with the query, and return the top+-- @k@. Not production quality — no embeddings, no stemming, no scoring beyond raw+-- overlap — this is the documented placeholder.+inMemoryRetriever :: Int -> [Passage] -> Retriever+inMemoryRetriever k corpus = Retriever $ \query ->+  let qWords = tokens query+      score p = length (filter (`elem` qWords) (tokens (text p)))+   in pure (take k (sortOn (Down . score) corpus))+  where+    tokens = T.words . T.toLower
+ src/Shikumi/Compile/Serialize.hs view
@@ -0,0 +1,57 @@+-- | Serialization of a 'CompiledProgram'.+--+-- __Parameter-state, not whole-structure.__ A @Program@'s structure cannot be+-- serialized in general (its @FMap@ nodes hold opaque functions and its GADT type+-- indices are not reflected at runtime). What /can/ be saved — and is exactly what+-- the optimizer (EP-10) and CLI (EP-12) need — is each node's tunable+-- 'Shikumi.Program.Params', against a known program /template/. This mirrors DSPy's+-- @dump_state@/@load_state@: dump the parameters, load them back onto a program you+-- still hold as code.+--+-- This reuses EP-4's already-provided parameter interface verbatim:+-- 'Shikumi.Program.programParams' (the ordered @[Params]@ vector, in @foldParams@+-- order) and 'Shikumi.Program.setProgramParams' (re-apply a vector onto a matching+-- program, with a length-mismatch guard). So this module is a thin JSON wrapper —+-- the node-order contract and the count check are EP-4's, not re-invented here.+module Shikumi.Compile.Serialize+  ( encodeCompiled,+    decodeCompiledOnto,+  )+where++import Data.Aeson (eitherDecode, encode)+import Data.ByteString.Lazy (ByteString)+import Shikumi.Compile.Types (CompiledProgram (..))+import Shikumi.Program+  ( Params,+    Program,+    ProgramShapeError (..),+    programParams,+    setProgramParams,+  )++-- | Emit a compiled program's ordered parameter vector as JSON (a JSON array of+-- @Params@, in 'Shikumi.Program.foldParams' order).+encodeCompiled :: CompiledProgram i o -> ByteString+encodeCompiled (CompiledProgram p) = encode (programParams p)++-- | Re-apply a saved parameter vector onto a structural /template/ (the base+-- program, held as code), in node order. Fails with a descriptive message if the+-- JSON is malformed or if its node count does not match the template — which would+-- mean the template is not the program the parameters were saved from. The signature+-- mirrors DSPy's @load_state(program)@.+decodeCompiledOnto ::+  Program i o ->+  ByteString ->+  Either String (CompiledProgram i o)+decodeCompiledOnto template bs = do+  ps <- eitherDecode bs :: Either String [Params]+  case setProgramParams ps template of+    Left (ParamCountMismatch (expected, got)) ->+      Left+        ( "shikumi-compile: parameter count mismatch — the template has "+            <> show expected+            <> " node(s) but the saved state has "+            <> show got+        )+    Right prog -> Right (CompiledProgram prog)
+ src/Shikumi/Compile/Types.hs view
@@ -0,0 +1,75 @@+{-# LANGUAGE RankNTypes #-}++-- | The two types this plan (EP-9) owns and the verbs over them.+--+-- A 'Compiler' is a /pure/ @Program -> Program@ rewrite that bakes a prompting+-- strategy into a program — it sets per-node parameters (zero-shot, few-shot) or+-- rewrites structure (chain-of-thought, RAG). It is emphatically /not/ a search:+-- it never calls the LM to try variations and keep the best (that is the optimizer,+-- @docs/plans/10-optimizer-framework.md@).+--+-- A 'CompiledProgram' is the result of compilation — a phantom-marked 'Program'+-- that says "these parameters are intentional, this program is ready to+-- run/serialize/optimize". It is MasterPlan integration point #6: the optimizer+-- (EP-10) emits one and the CLI (EP-12) loads/runs/saves one.+--+-- __Why a newtype, not a side table of frozen parameters.__ EP-4+-- (@Shikumi.Program@) already stores each node's 'Shikumi.Program.Params' /inside/+-- the @Program@ GADT node. A compiler therefore produces a fully-formed @Program@+-- whose nodes already hold the baked-in parameters; there is nothing to keep on the+-- side. The newtype is a marker that prevents a half-built base program from being+-- silently treated as compiled, and gives the optimizer's return type and the CLI's+-- load/save type something precise to name.+module Shikumi.Compile.Types+  ( Compiler (..),+    CompiledProgram (..),+    compile,+    runCompiled,+    identity,+  )+where++import Effectful (Eff, (:>))+import Effectful.Error.Static (Error)+import Shikumi.Error (ShikumiError)+import Shikumi.LLM (LLM)+import Shikumi.Program (Program, runProgram)++-- | A pure rewrite of a program. The @forall i o@ is the key design move: a+-- compiler must apply to /any/ program regardless of its input/output types+-- (few-shot demos are type-agnostic JSON, an instruction string is type-agnostic,+-- the chain-of-thought rewrite is uniform), so a single 'Compiler' value+-- ('identity', 'Shikumi.Compile.ZeroShot.zeroShot', …) is usable everywhere.+-- @RankNTypes@ (on by default in GHC2024) makes the rank-2 field legal.+newtype Compiler = Compiler+  { runCompiler :: forall i o. Program i o -> Program i o+  }++-- | A program that has been through a 'Compiler'. A newtype over 'Program' — the+-- parameters live on the nodes (see the module header).+newtype CompiledProgram i o = CompiledProgram+  { compiledProgram :: Program i o+  }++-- | Apply a compiler to a base program, marking the result compiled. Pure: no+-- @Eff@, no @IO@. Every initial compiler is a deterministic rewrite given+-- statically-supplied data, so compilation is trivially testable and composable+-- (@compile c2 . CompiledProgram . runCompiler c1@).+compile :: Compiler -> Program i o -> CompiledProgram i o+compile (Compiler f) = CompiledProgram . f++-- | Run a compiled program. Identical to running the wrapped 'Program', so it+-- inherits EP-4's @runProgram@ constraint exactly (MasterPlan integration point+-- #4): @(LLM :> es, Error ShikumiError :> es)@ — the decode path throws typed+-- 'ShikumiError's.+runCompiled ::+  (LLM :> es, Error ShikumiError :> es) =>+  CompiledProgram i o ->+  i ->+  Eff es o+runCompiled (CompiledProgram p) = runProgram p++-- | The no-op compiler: returns the program unchanged. Useful as a baseline and as+-- the unit for composing compilers.+identity :: Compiler+identity = Compiler id
+ src/Shikumi/Compile/ZeroShot.hs view
@@ -0,0 +1,26 @@+-- | The zero-shot compiler: set a single instruction at every LM-call node and+-- clear all demonstrations. The simplest real compiler, and the first to exercise+-- EP-4's parameter traversal ('Shikumi.Program.mapParams').+module Shikumi.Compile.ZeroShot+  ( zeroShot,+    zeroShotClear,+  )+where++import Data.Text (Text)+import Shikumi.Compile.Types (Compiler (..))+import Shikumi.Program (Params (..), mapParams)++-- | Override every node's instruction with @instr@ and remove all demos. Reaches+-- every 'Shikumi.Program.Predict' node in the program — including nodes nested+-- inside @Compose@/@Parallel@/@Retry@/etc. — because 'mapParams' visits them all.+zeroShot :: Text -> Compiler+zeroShot instr =+  Compiler $+    mapParams (\ps -> ps {instructionOverride = Just instr, demos = []})++-- | Clear demos at every node but keep each node's existing instruction override+-- (i.e. fall back to the signature default where none was set). Useful to strip a+-- few-shot program back to zero-shot without choosing a new instruction.+zeroShotClear :: Compiler+zeroShotClear = Compiler $ mapParams (\ps -> ps {demos = []})
+ test/Main.hs view
@@ -0,0 +1,219 @@+-- | The EP-9 acceptance suite. Every assertion is about the /prompt the model would+-- have seen/ (captured offline via "Test.Capture") or the /parameters now stored on+-- the nodes/ (a pure @foldParams@ read) — never merely "a type was added".+module Main (main) where++import Data.Text (Text)+import Data.Text qualified as T+import Effectful (runPureEff)+import Shikumi.Compile+  ( chainOfThoughtCompiler,+    compile,+    compiledProgram,+    decodeCompiledOnto,+    encodeCompiled,+    fewShotTyped,+    identity,+    inMemoryRetriever,+    rag,+    retrieve,+    zeroShot,+  )+import Shikumi.Compile.Retriever (Passage (..))+import Shikumi.Program (Params (..), foldParams)+import Test.Capture (runWithCapture)+import Test.Fixtures+  ( Question (..),+    answerResponse,+    cotAnswerResponse,+    cotDraftResponse,+    demoPairs,+    draftResponse,+    qaBase,+    qaPipeline,+  )+import Test.Tasty (TestTree, defaultMain, testGroup)+import Test.Tasty.HUnit (assertBool, testCase, (@?=))++main :: IO ()+main = defaultMain tests++tests :: TestTree+tests =+  testGroup+    "shikumi-compile (EP-9)"+    [ m0_identity,+      m1_baseline,+      m2_zeroShot,+      m3_fewShot,+      m4_chainOfThought,+      m5_rag,+      m6_serialize+    ]++-- A question whose words overlap the target RAG passage.+q :: Question+q = Question "what is the shikumi mechanism"++-- | Assert a captured prompt contains a substring.+contains :: Text -> Text -> Bool+contains needle hay = needle `T.isInfixOf` hay++-- ---------------------------------------------------------------------------+-- M0 — the no-op compiler round-trips a program unchanged+-- ---------------------------------------------------------------------------++m0_identity :: TestTree+m0_identity =+  testCase "M0 identity: compiling with `identity` leaves the rendered prompt unchanged" $ do+    (basePrompts, _) <- runWithCapture [answerResponse] qaBase q+    (idPrompts, _) <- runWithCapture [answerResponse] (compiledProgram (compile identity qaBase)) q+    idPrompts @?= basePrompts++-- ---------------------------------------------------------------------------+-- M1 — baseline: the uncompiled program's prompt carries the signature default+-- instruction and no demonstrations+-- ---------------------------------------------------------------------------++m1_baseline :: TestTree+m1_baseline =+  testCase "M1 baseline: uncompiled qaBase carries its default instruction and no demos" $ do+    (prompts, _) <- runWithCapture [answerResponse] qaBase q+    case prompts of+      [p] -> do+        assertBool "default instruction present" (contains "Answer the question." p)+        -- No demo content: the distinctive demo answers must be absent.+        assertBool "no demo content" (not (contains "Paris" p))+      _ -> assertBool ("expected exactly one captured prompt, got " <> show (length prompts)) False++-- ---------------------------------------------------------------------------+-- M2 — zero-shot: set the instruction, clear demos, at every node+-- ---------------------------------------------------------------------------++m2_zeroShot :: TestTree+m2_zeroShot =+  testGroup+    "M2 zero-shot"+    [ testCase "instruction present, no demos (qaBase)" $ do+        let compiled = compile (zeroShot "Answer concisely.") qaBase+        (prompts, _) <- runWithCapture [answerResponse] (compiledProgram compiled) q+        case prompts of+          [p] -> assertBool "instruction present" (contains "Answer concisely." p)+          _ -> assertBool "one prompt" False+        -- pure assertion: every node's demo list is empty+        assertBool "all nodes have no demos" (all (null . demos) (foldParams (compiledProgram compiled))),+      testCase "instruction reaches BOTH nodes (qaPipeline)" $ do+        let compiled = compile (zeroShot "Answer concisely.") qaPipeline+        (prompts, _) <- runWithCapture [draftResponse, answerResponse] (compiledProgram compiled) q+        length prompts @?= 2+        assertBool "instruction in both prompts" (all (contains "Answer concisely.") prompts)+    ]++-- ---------------------------------------------------------------------------+-- M3 — few-shot: inject demos at every node+-- ---------------------------------------------------------------------------++m3_fewShot :: TestTree+m3_fewShot =+  testGroup+    "M3 few-shot"+    [ testCase "injected demos appear in the prompt (qaBase)" $ do+        let compiled = compile (fewShotTyped demoPairs) qaBase+        (prompts, _) <- runWithCapture [answerResponse] (compiledProgram compiled) q+        case prompts of+          [p] -> do+            assertBool "demo input present" (contains "capital of france" p)+            assertBool "demo output present" (contains "Paris" p)+            assertBool "second demo output present" (contains "blue" p)+          _ -> assertBool "one prompt" False,+      testCase "every node carries the injected demos (qaPipeline, pure)" $ do+        -- A pure, no-LM assertion that the parameters changed at every node.+        -- (Running the pipeline is not asserted here: the same demo pool is+        -- type-mismatched at the second node, which DSPy's LabeledFewShot also+        -- permits — see the plan. The reach guarantee is what matters.)+        let compiled = compile (fewShotTyped demoPairs) qaPipeline+        map (length . demos) (foldParams (compiledProgram compiled)) @?= [3, 3]+    ]++-- ---------------------------------------------------------------------------+-- M4 — chain-of-thought: every node reasons step by step first+-- ---------------------------------------------------------------------------++m4_chainOfThought :: TestTree+m4_chainOfThought =+  testGroup+    "M4 chain-of-thought"+    [ testCase "reasoning cue in the prompt (qaBase)" $ do+        let compiled = compile chainOfThoughtCompiler qaBase+        (prompts, _) <- runWithCapture [cotAnswerResponse] (compiledProgram compiled) q+        case prompts of+          [p] -> assertBool "reasoning cue present" (contains "step by step" p)+          _ -> assertBool "one prompt" False,+      testCase "reasoning cue reaches BOTH nodes (qaPipeline)" $ do+        let compiled = compile chainOfThoughtCompiler qaPipeline+        (prompts, _) <- runWithCapture [cotDraftResponse, cotAnswerResponse] (compiledProgram compiled) q+        length prompts @?= 2+        assertBool "reasoning cue in both prompts" (all (contains "step by step") prompts)+    ]++-- ---------------------------------------------------------------------------+-- M5 — RAG: a retrieved passage appears in the rendered prompt+-- ---------------------------------------------------------------------------++corpus :: [Passage]+corpus =+  [ Passage "p1" "Shikumi is the mechanism behind how a system works.",+    Passage "p2" "The Eiffel Tower is in Paris, the capital of France.",+    Passage "p3" "Photosynthesis converts light into chemical energy."+  ]++m5_rag :: TestTree+m5_rag =+  testGroup+    "M5 RAG"+    [ testCase "retriever ranking (pure)" $ do+        let top = runPureEff (retrieve (inMemoryRetriever 1 corpus) "shikumi mechanism")+        map passageId top @?= ["p1"],+      testCase "retrieved passage appears in the prompt, others do not" $ do+        let compiled = compile (rag (inMemoryRetriever 1 corpus) "shikumi mechanism") qaBase+        (prompts, _) <- runWithCapture [answerResponse] (compiledProgram compiled) q+        case prompts of+          [p] -> do+            assertBool "target passage present" (contains "mechanism behind how a system works" p)+            assertBool "non-matching passage absent" (not (contains "Photosynthesis" p))+          _ -> assertBool "one prompt" False+    ]++-- ---------------------------------------------------------------------------+-- M6 — serialization round-trips the parameter state+-- ---------------------------------------------------------------------------++m6_serialize :: TestTree+m6_serialize =+  testGroup+    "M6 serialize"+    [ testCase "round-trip renders an identical prompt (zeroShot over a pipeline)" $ do+        let compiled = compile (zeroShot "Be precise.") qaPipeline+            bytes = encodeCompiled compiled+        case decodeCompiledOnto qaPipeline bytes of+          Left e -> assertBool ("unexpected decode failure: " <> e) False+          Right roundTripped -> do+            (origPrompts, _) <- runWithCapture [draftResponse, answerResponse] (compiledProgram compiled) q+            (rtPrompts, _) <- runWithCapture [draftResponse, answerResponse] (compiledProgram roundTripped) q+            rtPrompts @?= origPrompts,+      testCase "round-trip preserves injected demos (fewShot over a single node)" $ do+        let compiled = compile (fewShotTyped demoPairs) qaBase+            bytes = encodeCompiled compiled+        case decodeCompiledOnto qaBase bytes of+          Left e -> assertBool ("unexpected decode failure: " <> e) False+          Right roundTripped -> do+            (origPrompts, _) <- runWithCapture [answerResponse] (compiledProgram compiled) q+            (rtPrompts, _) <- runWithCapture [answerResponse] (compiledProgram roundTripped) q+            rtPrompts @?= origPrompts,+      testCase "decoding onto a wrong-shaped template returns Left" $ do+        -- a 2-node payload onto a 1-node template+        let bytes = encodeCompiled (compile (zeroShot "x") qaPipeline)+        case decodeCompiledOnto qaBase bytes of+          Left _ -> pure ()+          Right _ -> assertBool "expected a count-mismatch Left" False+    ]
+ test/Test/Capture.hs view
@@ -0,0 +1,71 @@+-- | The offline prompt-inspection harness every milestone asserts against.+--+-- 'runWithCapture' runs a program through a /capturing stub/ — a fake @LLM@+-- interpreter that records each request's fully-rendered prompt (it intercepts at+-- the @LLM@ boundary, /after/ EP-3's adapter has turned the signature + params ++-- input into a baikai @Context@) and returns scripted canned responses. Because the+-- rendering happens before the @LLM@ call, the captured text is exactly the prompt+-- the model would have seen — which is what a compiler changes. No network, no API+-- key; the whole suite is a deterministic @cabal test@.+--+-- The captured text is the JSON encoding of the whole 'Baikai.Context' (system+-- prompt + messages + tools). The instruction/reasoning cue/retrieved context live+-- in the system prompt; few-shot demos render as user/assistant /messages/ — both+-- are present in the JSON, so a substring assertion finds either.+module Test.Capture+  ( runWithCapture,+  )+where++import Baikai (Context, Response)+import Data.Aeson (encode)+import Data.IORef (IORef, atomicModifyIORef', modifyIORef', newIORef, readIORef)+import Data.Text (Text)+import Data.Text.Lazy qualified as TL+import Data.Text.Lazy.Encoding qualified as TLE+import Effectful (liftIO, runEff)+import Effectful.Dispatch.Dynamic (interpret)+import Effectful.Error.Static (runErrorNoCallStack)+import Shikumi.Error (ShikumiError)+import Shikumi.LLM (LLM (..))+import Shikumi.Program (Program, runProgram)+import Test.Fixtures (markerBody, mkResponse)++-- | Run @prog@ on @input@, serving the given scripted responses (one per LM-call+-- node, in order), and return @(captured prompts in execution order, the program's+-- decoded result)@. The result is @Left@ if a node's response fails to decode — but+-- the prompt for that node is captured /before/ the response is returned, so prompt+-- assertions hold regardless of decode success.+runWithCapture ::+  [Response] ->+  Program i o ->+  i ->+  IO ([Text], Either ShikumiError o)+runWithCapture responses prog input = do+  captureRef <- newIORef []+  scriptRef <- newIORef responses+  result <-+    runEff+      . runErrorNoCallStack @ShikumiError+      . interpret+        ( \_ -> \case+            Complete _ ctx _ -> do+              liftIO (modifyIORef' captureRef (renderedText ctx :))+              liftIO (pop scriptRef)+            Stream {} -> pure []+        )+      $ runProgram prog input+  captured <- reverse <$> readIORef captureRef+  pure (captured, result)++-- | The full rendered request as text: the JSON encoding of the whole context.+renderedText :: Context -> Text+renderedText = TL.toStrict . TLE.decodeUtf8 . encode++-- | Pop the next scripted response; an exhausted script yields a trivially-empty+-- response (tests script enough that this is unreached on the success path).+pop :: IORef [Response] -> IO Response+pop ref = atomicModifyIORef' ref step+  where+    step (x : xs) = (xs, x)+    step [] = ([], mkResponse (markerBody []))
+ test/Test/Fixtures.hs view
@@ -0,0 +1,134 @@+-- | Shared fixtures for the EP-9 compiler tests: a minimal @Question -> Answer@+-- signature pair (plus an intermediate @Draft@ so a two-node pipeline can prove a+-- compiler reaches a nested node), the base programs every milestone rewrites, and+-- canned fallback-adapter 'Response's the capturing stub returns.+--+-- The record instances mirror @ProgramFixtures@ in the @shikumi@ test tree (which+-- lives there and is not importable here): each record derives 'Generic' and gets+-- empty 'ToSchema' / 'FromModel' / 'ToPrompt' / 'Validatable' bodies (Generic+-- defaults), plus 'ToJSON' / 'FromJSON' so 'Shikumi.Compile.fewShotTyped' can build+-- demos from typed pairs.+module Test.Fixtures+  ( -- * Records+    Question (..),+    Answer (..),+    Draft (..),++    -- * Base programs+    qaBase,+    qaPipeline,++    -- * Demo pool+    demoPairs,++    -- * Canned responses+    answerResponse,+    draftResponse,+    cotDraftResponse,+    cotAnswerResponse,+    markerBody,+    mkResponse,+  )+where++import Baikai+  ( AssistantContent (..),+    Response,+    _Response,+    _TextContent,+  )+import Control.Lens ((&), (.~))+import Data.Aeson (FromJSON, ToJSON)+import Data.Generics.Labels ()+import Data.Text (Text)+import Data.Text qualified as T+import Data.Vector qualified as V+import GHC.Generics (Generic)+import Shikumi.Adapter (ToPrompt)+import Shikumi.Module (predict)+import Shikumi.Program (Program, pipeline)+import Shikumi.Schema (FromModel, ToSchema, Validatable)+import Shikumi.Signature (Signature, mkSignature)++-- ---------------------------------------------------------------------------+-- Records+-- ---------------------------------------------------------------------------++newtype Question = Question {question :: Text}+  deriving stock (Generic, Show, Eq)+  deriving anyclass (ToJSON, FromJSON, ToSchema, FromModel, ToPrompt, Validatable)++newtype Answer = Answer {answer :: Text}+  deriving stock (Generic, Show, Eq)+  deriving anyclass (ToJSON, FromJSON, ToSchema, FromModel, ToPrompt, Validatable)++newtype Draft = Draft {draft :: Text}+  deriving stock (Generic, Show, Eq)+  deriving anyclass (ToJSON, FromJSON, ToSchema, FromModel, ToPrompt, Validatable)++-- ---------------------------------------------------------------------------+-- Signatures and base programs+-- ---------------------------------------------------------------------------++qaSig :: Signature Question Answer+qaSig = mkSignature "Answer the question."++questionToDraft :: Signature Question Draft+questionToDraft = mkSignature "Draft an answer to the question."++draftToAnswer :: Signature Draft Answer+draftToAnswer = mkSignature "Finalize the answer from the draft."++-- | A single-node base program (demos line up exactly for the strong few-shot+-- assertion).+qaBase :: Program Question Answer+qaBase = predict qaSig++-- | A two-node base program: @Question -> Draft -> Answer@, so a compiler can be+-- shown to reach a node nested inside a 'Shikumi.Program.Compose'.+qaPipeline :: Program Question Answer+qaPipeline = pipeline (predict questionToDraft) (predict draftToAnswer)++-- | A small demo pool, as typed pairs (used via 'Shikumi.Compile.fewShotTyped').+-- The answer texts are distinctive so a prompt assertion is unambiguous.+demoPairs :: [(Question, Answer)]+demoPairs =+  [ (Question "what is two plus two", Answer "four"),+    (Question "capital of france", Answer "Paris"),+    (Question "color of the clear sky", Answer "blue")+  ]++-- ---------------------------------------------------------------------------+-- Canned responses (rendered as the fallback adapter's [[ ## field ## ]] sections)+-- ---------------------------------------------------------------------------++-- | Build a fallback-style response body from @(field, value)@ sections, closing+-- with the @completed@ marker.+markerBody :: [(Text, Text)] -> Text+markerBody fields = T.unlines (concatMap sect fields ++ ["[[ ## completed ## ]]"])+  where+    sect (k, v) = ["[[ ## " <> k <> " ## ]]", v]++-- | An assistant 'Response' carrying @t@ as its single text block.+mkResponse :: Text -> Response+mkResponse t =+  _Response & #message . #content .~ V.singleton (AssistantText (_TextContent & #text .~ t))++-- | Decodes as an @Answer@ under the fallback adapter.+answerResponse :: Response+answerResponse = mkResponse (markerBody [("answer", "forty-two")])++-- | Decodes as a @Draft@ under the fallback adapter.+draftResponse :: Response+draftResponse = mkResponse (markerBody [("draft", "a rough answer")])++-- | Decodes as a @WithReasoning Draft@ (the chain-of-thought augmented output):+-- the @value@ section carries the nested @Draft@ JSON object.+cotDraftResponse :: Response+cotDraftResponse =+  mkResponse (markerBody [("reasoning", "thinking about it"), ("value", "{\"draft\":\"a rough answer\"}")])++-- | Decodes as a @WithReasoning Answer@.+cotAnswerResponse :: Response+cotAnswerResponse =+  mkResponse (markerBody [("reasoning", "thinking about it"), ("value", "{\"answer\":\"forty-two\"}")])