packages feed

kenshou-cli (empty) → 0.1.0.0

raw patch · 39 files changed

+6034/−0 lines, 39 filesdep +aesondep +basedep +bytestring

Dependencies added: aeson, base, bytestring, containers, directory, file-embed, filepath, githash, hasql, hspec, kenshou-check, kenshou-cli, kenshou-core, kenshou-diagnose, kenshou-evidence, kenshou-measure, kenshou-remote, kenshou-telemetry, optparse-applicative, process, random, settei, settei-env, settei-optparse-applicative, temporary, terminal-size, text, time, uuid

Files

+ CHANGELOG.md view
@@ -0,0 +1,12 @@+# Changelog++## 0.1.0.0 - 2026-10-28++### New Features++- First public release of the operator command library: `list`, `plan`, `run`, `execute`, `summarize`, `compare`, `diagnose`, `overhead`, `cell`, `record`, `attest`, `evidence`, `history`, `cohort`, and `perf`.+- The library takes its assembly from the executable: `mainWith`, `runWithArgsWith`, `standardCommands`, `standardTopics`, and `selfTestAssembly`.++### Other Changes++- Extracted from `mori://shinzui/keiro-runtime-kenshou` at commit `a1648293443ad88ff1695ed800ff44c99993574e`, with its package and module names unchanged.
+ LICENSE view
@@ -0,0 +1,28 @@+BSD 3-Clause License++Copyright (c) 2026, Nadeem Bitar++Redistribution and use in source and binary forms, with or without+modification, are permitted provided that the following conditions are met:++1. Redistributions of source code must retain the above copyright notice, this+   list of conditions and the following disclaimer.++2. Redistributions in binary form must reproduce the above copyright notice,+   this list of conditions and the following disclaimer in the documentation+   and/or other materials provided with the distribution.++3. Neither the name of the copyright holder nor the names of its contributors+   may be used to endorse or promote products derived from this software without+   specific prior written permission.++THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ help/cells.md view
@@ -0,0 +1,132 @@+# Leased verification cells++A cell runs a prepared verification plan on an exclusively leased machine set.+The payload identifies the Nix-built executable and the runtime cohort it+contains. Cell results are sealed in the results bucket and can be fetched and+verified again later.++Build and publish a payload descriptor:++```bash+kenshou cell payload publish --cohort released --out payloads/released.json+kenshou cell payload show payloads/released.json+```++`payload show` validates the descriptor and checks that the bundle object+exists with the recorded byte size before printing the descriptor as JSON.++Probe a cell after publishing a payload that contains the environment+scenario:++```bash+kenshou cell probe --cell alpha --payload payloads/head.json+```++The probe runs under a lease, verifies the sealed result, and writes+`.dev/cells/alpha.capabilities.json`. Route and submit use this cache to check+optional capabilities such as `pg_partman`. The cache is tied to the cell+descriptor and is refreshed by running the probe again.++Inspect a cell and run a plan in one process:++```bash+kenshou cell status --cell alpha+kenshou cell run --cell alpha --payload payloads/released.json \+  --plan plan.json --out cell-runs/alpha-1 --coerce-durable+```++The run command acquires a lease, prepares compatible runs for the cell,+submits them in reset-aware slices, follows the cell status, fetches and+verifies sealed output, then releases the lease. `--out` must name a new+directory. Keep that directory to resume an interrupted session:++```bash+kenshou cell resume --session cell-runs/alpha-1+```++Use `cell lease`, `cell submit`, `cell watch`, `cell fetch`, `cell verify`, and+`cell release` when the steps need separate processes. `cell fetch` needs the+results bucket and cell run ID; it writes a verified tree and an index beside+it. `cell verify` also accepts `gs://BUCKET/runs/CELL_RUN_ID` and checks a+sealed run without an existing session directory.++Compare a local run directory with a fetched nested cell run directory:++```bash+kenshou cell parity --local runs/LOCAL_RUN_ID \+  --cell cell-runs/CELL_RUN_ID/tree/output/NESTED_RUN_ID \+  --out parity.json+```++The report lists equal fields, expected placement differences, and unexpected+differences. The command exits 1 if a scenario input or verdict differs. Use+`--volatile result.summaries.verdicts.FIELD` only for a known path inside a+verdict summary.++Measure a benchmark with interleaved baseline and candidate trials on one+leased cell:++```bash+kenshou cell pair --cell alpha --start --baseline payloads/released.json \+  --candidate payloads/head.json --plan benchmark.plan.json \+  --policy policies/default.json --out cell-pairs/run-1+```++Each trial is a separate cell submission with a fresh reset. The command+keeps one lease across the schedule, replaces invalid pairs within its budget,+and writes `comparison.json`. Use the same payload for both arms as an A/A+control before comparing different cohorts.+`--start` starts stopped instances named by the verified cell descriptor; it is+also available on `cell lease` and `cell run`.++Measure telemetry overhead under one lease:++```bash+kenshou cell overhead selftest/telemetry/benchmark/arms-on-synthetic-service \+  --cell alpha --payload payloads/released.json \+  --arms tracing=off,sdk-otlp --trials 3 --otlp-sink file \+  --out cell-overhead/tracing+```++Each arm slot is a separate cold-reset submission. The command links verified+nested run directories into its `runs/` tree, retains the cell sessions,+reuses the overhead planner's retry and replacement-block rules, and writes+`overhead-report.json`. `--resume` continues the saved invocation under a+new lease; `--analyse-only` recomputes the report without submitting work.++Route a plan across cells with different PostgreSQL majors:++```bash+kenshou cell route --plan plan.json --cell beta --cell alpha \+  --payload payloads/released.json --out routed --coerce-durable+```++The output contains `plan.<cell>.json` for admitted runs, `plan.local.json`+for work requiring local server control, and `unroutable.json` for remaining+refusals. Cell order is preference order. Routing and submission read a+capability cache from `.dev/cells/<cell>.capabilities.json` when present;+`KENSHOU_CELL_CAPABILITIES_DIR` selects another directory. A cache for an old+descriptor is ignored. An unknown capability produces a warning, and an+explicitly unavailable capability prevents admission.++The control bucket defaults to `tan-nb-exp-cells-control`; pass+`--control-bucket` to select another. The client accepts only cells in the+projects named by `KENSHOU_GCP_ALLOWED_PROJECTS` (default `tan-nb-exp`).++For a running cell, use the owner IAP script through the verified descriptor:++```bash+kenshou cell debug --cell alpha ssh monitoring -- systemctl is-active opentelemetry-collector.service+kenshou cell debug --cell alpha journal+kenshou cell debug --cell alpha tunnel monitoring 8888 18888+```++`journal` reads the last 100 driver agent lines. `tunnel` holds an SSH port+forward until interrupted; while it runs, the collector metrics endpoint is+at `http://127.0.0.1:18888/metrics`. Keep a lease held during a longer debug+session so the cell does not idle off. The command locates the owner checkout+with Mori, or uses `KENSHOU_LTI_DIR` when set.++For profiling, publish a `profiled` payload and pass `--rts=-p` to `cell run`.+The cell includes the generated `.prof` files under `tree/output/profiles/` in+the sealed result, subject to its 128 MiB profile collection limit.
+ help/comparisons.md view
@@ -0,0 +1,20 @@+# Comparing benchmark runs++`kenshou compare` evaluates paired baseline and candidate runs. Run both arms in+the same environment and interleave them in ABBA or BAAB order so drift affects+each arm equally. Supply one `--baseline` and one `--candidate` directory per+pair.++The declared `--vary` axes are the only compatibility fields allowed to differ.+Use `cohort`, `dim:NAME`, or `knob:NAME`. A scenario, knob, dimension, schema, or+cohort difference outside those axes is a usage error. A machine-profile change+is an infrastructure failure.++Policies combine a relative limit with an absolute floor. A metric regresses+only when the confidence interval clears both limits. Wide or crossing intervals+are inconclusive. Hard health observations produce infrastructure failure and+soft observations produce an inconclusive result. Historical series can inform+future work, but they never determine a comparison verdict.++`kenshou summarize RUN_DIR --verify` recomputes measurements from retained+samples and series and checks the section stored in `run-result.json`.
+ help/diagnostics.md view
@@ -0,0 +1,24 @@+# Diagnostics++Re-evaluate the sampled series from a completed run without changing its sealed+artifacts:++    kenshou diagnose leak RUN_DIR++Render saved watchdog captures, rebuild their classification, or export the+PostgreSQL wait graph as Graphviz DOT:++    kenshou diagnose stall RUN_DIR+    kenshou diagnose stall RUN_DIR --reclassify --dot wait-graph.dot++Rerun a scenario under a bounded heap profile or GHC event log:++    kenshou diagnose profile SCENARIO --mode closure-type+    kenshou diagnose profile SCENARIO --mode info-table+    kenshou diagnose profile SCENARIO --mode eventlog --classes gu++Diagnosis uses the ordinary process exit contract: 0 means no finding, 1 means+a finding was confirmed, 2 is invalid usage, 3 means insufficient evidence, and+4 is an input, execution, or infrastructure error. See+`docs/guides/diagnosing-leaks-and-stalls.md` in the source distribution for the+full investigation workflow.
+ help/evidence.md view
@@ -0,0 +1,23 @@+# Verification evidence++`kenshou record RUN-DIR --data-base-uri gs://BUCKET/PREFIX --purpose release --project PROJECT` publishes a finished run's files and adds one record to the OKF verification bundle. The record names the run, cohort, outcome, and SHA-256-pinned object links. It contains no measurements; those remain in the linked run files.++`kenshou record --comparison FILE --data-base-uri gs://BUCKET/PREFIX --purpose release --project PROJECT` records a `kenshou.comparison/v1` document. Every baseline and candidate run named by the comparison must already have a run record in the bundle. The comparison record links the source document by digest and references those run concepts. Supply either `RUN-DIR` or `--comparison FILE`.++The default bundle is `docs/verification`. `--bundle DIR` selects another bundle. A real GCS upload requires `--project PROJECT`, and the command checks that the active gcloud project matches it. `--verify-only` checks that all objects already exist; `--deep-verify` downloads them and checks their bytes. Failed or inconclusive runs link logs automatically; `--link-logs` also links logs for a passing run.++`--config PATH` reads ordered YAML configuration. The supported keys are `evidence.bundle-root`, `gcp.project`, `evidence.data-base-uri`, `evidence.default-subject`, and `evidence.bundle-uri`; the corresponding environment variables are `KENSHOU_EVIDENCE_BUNDLE`, `KENSHOU_GCP_PROJECT`, `KENSHOU_EVIDENCE_DATA_BASE_URI`, `KENSHOU_EVIDENCE_SUBJECT`, and `KENSHOU_EVIDENCE_BUNDLE_URI`. Named flags take precedence over the environment and files. `kenshou evidence check` uses the same bundle default and configuration sources. Unknown keys are rejected.++A dirty harness requires `--allow-dirty` and is recorded with `purpose: investigation` regardless of the requested purpose. Use `--subject MORI-URI` with `--subject-kind project|package` when the default layer project is too broad. Repeat `--produced MORI-URI` for reports caused by this run.++For a local rehearsal, copy the verification bundle outside the repository and pass `--bundle COPY --store-root DIR`. The record still contains `gs://` URIs, while the bytes are stored under the scratch directory. The command refuses `--store-root` with a bundle inside this repository.++`--json` prints one `kenshou.record-result/v1` document and sends errors to standard error. A repeat with the same record content succeeds and reports `already recorded`; different content at the same record path is a conflict.++`kenshou evidence check` checks record paths, IDs, digest and revision shapes, textual fields, event-only keys, timestamps, required data links, comparison arm references and verdicts, and changes to committed records. `--base REF` limits the Git history span. `--network --project PROJECT` checks object size and SHA-256 metadata, and `--deep` downloads bytes to verify each digest. The command exits 0 when clean, 1 for findings, and 4 when it cannot check. `--json` prints one `kenshou.evidence-check/v1` document.++`kenshou history --scenario LAYER/COMPONENT/KIND/NAME --json` reads matching records in time order and emits `kenshou.evidence-history/v1`. Use `--cohort-component PACKAGE[=VERSION|REVISION]`, repeatable `--outcome`, `--since YYYY-MM-DD`, or `--confirmed-only` to narrow the series. Each entry includes linked attestations, a derived trust tier, and the latest earlier confirmed baseline with the same scenario and compatibility key. The history view does not modify the bundle.++`kenshou attest RECORD-OR-ID --project PROJECT` fetches the recorded run or comparison data and writes an immutable attestation with six named checks. For a comparison, it verifies both recorded arms and replays the paired algorithm from their samples under the stored policy. `--offline` skips revision resolution that needs network access; `--linked-only` checks only directly linked objects. It reports `refuted` when fetched data contradicts the record and `incomplete` when a check cannot establish the claim. A scenario oracle must be registered before `verdict-recomputed` can pass for an ordinary run, so generic runs currently remain incomplete. Exit codes are 0 for confirmed, 1 for refuted, 3 for incomplete, 2 for invalid input, and 4 for an operational failure. `--json` emits one `kenshou.attest-result/v1` result document.++A person may annotate an incomplete or refuted attestation with `--accept-anomaly --authority human:ID --reason TEXT` from an interactive terminal. The computed verdict does not change. The command rejects this action in CI or without a terminal.
+ help/exit-codes.md view
@@ -0,0 +1,3 @@+# Exit codes++Exit code 0 means passed, 1 failed, 2 usage error, 3 inconclusive, and 4 errored or infrastructure failure. A reproduced known defect exits 0 unless strict known-defect handling is requested.
+ help/outcomes.md view
@@ -0,0 +1,3 @@+# Outcomes++Runs finish as `passed`, `failed`, `errored`, `inconclusive`, or `infrastructure-failure`. A known defect never changes the true outcome; it can only make an explained failure non-blocking for automation.
+ help/planning.md view
@@ -0,0 +1,20 @@+# Planning and execution++`kenshou plan` maps named components, cohort differences, local changes, or+upstream revision ranges through the checked-in component graph. The transitive+dependent closure selects scenarios, and `--explain` shows why each one was+included. Unmapped inputs select everything rather than risk missing evidence.++Dimension policies are `default-only`, `telemetry-corners`, `pairwise`, and+`full`. Telemetry changes raise their minimum policy automatically. Benchmarks+always use durable PostgreSQL and are budgeted as complete interleaved trial+groups.++Use `--suite smoke`, `change`, `nightly`, `weekly-soak`, or `release` for the+checked-in policies. The generated run plan binds the graph digest, cohort,+seed, reasons, skips, and preassigned run IDs.++`kenshou execute --plan FILE --out DIR` runs entries sequentially and writes an+atomic plan summary. `--resume` requires the byte-identical plan, skips completed+entries, and gives interrupted entries fresh run IDs. See `docs/planning.md` for+the complete guide.
+ help/run-specs.md view
@@ -0,0 +1,3 @@+# Run specifications++A run specification is a versioned JSON request to execute one scenario. The effective document records every default explicitly, including dimensions, knobs, seed, phases, environment, and expected runtime cohort.
+ help/scenarios.md view
@@ -0,0 +1,5 @@+# Scenarios++A scenario is one named verification procedure. Its stable identifier has four lowercase segments: `layer/component/kind/name`. The layer identifies the runtime boundary under test, and the kind is `correctness`, `concurrency`, `soak`, or `benchmark`.++Use `kenshou list` to inspect registered scenarios. Scenario identifiers are versioned by a separate revision number so a changed workload is never silently compared with an older one.
+ help/selectors.md view
@@ -0,0 +1,3 @@+# Selectors++Selectors match scenario identifiers one segment at a time. `*` matches one segment and a final `**` matches the remainder. For example, `selftest/**` selects every self-test and `*/*/benchmark/*` selects all benchmark scenarios.
+ kenshou-cli.cabal view
@@ -0,0 +1,125 @@+cabal-version: 3.0+name: kenshou-cli+version: 0.1.0.0+synopsis: Operator command library for the Kenshou verification harness+description: Operator and automation interface for the Kenshou verification harness.+license: BSD-3-Clause+license-file: LICENSE+author: Nadeem Bitar+maintainer: nadeem@gmail.com+homepage: https://github.com/shinzui/kenshou+bug-reports: https://github.com/shinzui/kenshou/issues+category: Testing+build-type: Simple+extra-doc-files: CHANGELOG.md+tested-with: ghc >=9.12 && <9.13+extra-source-files:+  help/*.md+  test/fixtures/*.json+  test/golden/*.txt++common warnings+  ghc-options:+    -Wall+    -Wcompat+    -Widentities+    -Wincomplete-record-updates+    -Wincomplete-uni-patterns+    -Wpartial-fields+    -Wredundant-constraints++  default-language: GHC2024+  default-extensions:+    BlockArguments+    DuplicateRecordFields+    ImportQualifiedPost+    NoFieldSelectors+    OverloadedLabels+    OverloadedRecordDot+    OverloadedStrings++library+  import: warnings+  hs-source-dirs: src+  exposed-modules:+    Kenshou.Cli+    Kenshou.Cli.Attest+    Kenshou.Cli.Cohort+    Kenshou.Cli.Command.Cell+    Kenshou.Cli.Command.Cohort+    Kenshou.Cli.Command.Execute+    Kenshou.Cli.Command.List+    Kenshou.Cli.Command.Overhead+    Kenshou.Cli.Command.Perf+    Kenshou.Cli.Command.Perf.Session+    Kenshou.Cli.Command.Plan+    Kenshou.Cli.Command.Run+    Kenshou.Cli.Command.Worker+    Kenshou.Cli.Completions+    Kenshou.Cli.Config+    Kenshou.Cli.Diagnose+    Kenshou.Cli.Help+    Kenshou.Cli.Help.Width+    Kenshou.Cli.Main+    Kenshou.Cli.Options+    Kenshou.Cli.Registry+    Kenshou.Cli.Version++  other-modules: Paths_kenshou_cli+  autogen-modules: Paths_kenshou_cli+  build-depends:+    aeson >=2.2 && <2.3,+    base >=4.21 && <4.22,+    bytestring >=0.12 && <0.13,+    containers >=0.7 && <0.8,+    directory >=1.3 && <1.4,+    file-embed >=0.0.16 && <0.1,+    filepath >=1.5 && <1.6,+    githash >=0.1.7 && <0.2,+    hasql >=1.10 && <1.11,+    kenshou-check ^>=0.1.0.0,+    kenshou-core ^>=0.1.0.0,+    kenshou-diagnose ^>=0.1.0.0,+    kenshou-evidence ^>=0.1.0.0,+    kenshou-measure ^>=0.1.0.0,+    kenshou-remote ^>=0.1.0.0,+    kenshou-telemetry ^>=0.1.0.0,+    optparse-applicative >=0.19 && <0.20,+    process >=1.6 && <1.7,+    random >=1.2.1 && <1.4,+    settei ^>=0.2.0.0,+    settei-env ^>=0.2.0.0,+    settei-optparse-applicative ^>=0.2.0.0,+    temporary >=1.3 && <1.4,+    terminal-size >=0.3.4 && <0.4,+    text >=2.1 && <2.2,+    time >=1.14 && <1.15,+    uuid >=1.3 && <1.4,++test-suite kenshou-cli-test+  import: warnings+  type: exitcode-stdio-1.0+  hs-source-dirs: test+  main-is: Main.hs+  other-modules: PerfSessionSpec+  build-depends:+    aeson >=2.2 && <2.3,+    base >=4.21 && <4.22,+    bytestring,+    containers >=0.7 && <0.8,+    directory >=1.3 && <1.4,+    filepath,+    hspec >=2.11 && <2.12,+    kenshou-cli,+    kenshou-core ^>=0.1.0.0,+    kenshou-measure ^>=0.1.0.0,+    kenshou-remote ^>=0.1.0.0,+    kenshou-telemetry ^>=0.1.0.0,+    temporary >=1.3 && <1.4,+    text >=2.1 && <2.2,+    time >=1.14 && <1.15,+    uuid >=1.3 && <1.4,++source-repository head+  type: git+  location: https://github.com/shinzui/kenshou.git
+ src/Kenshou/Cli.hs view
@@ -0,0 +1,14 @@+module Kenshou.Cli+  ( mainWith,+    runWithArgsWith,+    CliAssembly,+    Program (..),+    selfTestBundles,+    standardCommands,+    standardTopics,+    selfTestAssembly,+  )+where++import Kenshou.Cli.Main (mainWith, runWithArgsWith)+import Kenshou.Cli.Registry (CliAssembly, Program (..), selfTestAssembly, selfTestBundles, standardCommands, standardTopics)
+ src/Kenshou/Cli/Attest.hs view
@@ -0,0 +1,101 @@+module Kenshou.Cli.Attest (runOutcomeRecomputer, measurementRecomputer, pairedComparisonRecomputer) where++import Control.Monad (forM)+import Data.Aeson (Result (..), Value (..), eitherDecodeFileStrict', fromJSON, toJSON)+import Data.Aeson.KeyMap qualified as KeyMap+import Data.List.NonEmpty qualified as NonEmpty+import Data.Text (Text)+import Data.Text qualified as Text+import Kenshou.Evidence.Attest (Recomputation (..), Recomputer (..), coreRecomputers)+import Kenshou.Measure.Compare (CompareError (..), compareRuns)+import Kenshou.Measure.Compare.Compatibility (VaryingAxis, parseVaryingAxis)+import Kenshou.Measure.Compare.Policy (Policy)+import Kenshou.Measure.Summary (SummaryError (..), summarizeRunDir)+import System.FilePath ((</>))++runOutcomeRecomputer :: Recomputer+runOutcomeRecomputer = Recomputer "run-outcome" 1 $ \root -> case coreRecomputers of+  (core : _) -> core.recompute root+  [] -> pure (Left "the core outcome oracle is unavailable")++measurementRecomputer :: Recomputer+measurementRecomputer = Recomputer "kenshou-summary" 1 $ \root -> do+  measured <- summarizeRunDir root+  case measured of+    Left (SummaryError message) -> pure (Left message)+    Right summary -> do+      stored <- eitherDecodeFileStrict' (root </> "run-result.json") :: IO (Either String Value)+      pure case stored of+        Left message -> Left (Text.pack message)+        Right document ->+          Right+            Recomputation+              { agreesWithDocuments = storedMeasurements document == Just (toJSON summary),+                outcome = Nothing,+                comparisonVerdict = Nothing,+                detail = "recomputed kenshou.measurements/v1 from the sealed samples and series"+              }++storedMeasurements :: Value -> Maybe Value+storedMeasurements (Object result) = do+  Object summaries <- KeyMap.lookup "summaries" result+  Object measurements <- KeyMap.lookup "measurements" summaries+  KeyMap.lookup "measurements" measurements+storedMeasurements _ = Nothing++pairedComparisonRecomputer :: Recomputer+pairedComparisonRecomputer = Recomputer "paired-bootstrap-t-envelope" 1 $ \root -> do+  stored <- eitherDecodeFileStrict' (root </> "comparison.json") :: IO (Either String Value)+  case stored of+    Left message -> pure (Left (Text.pack message))+    Right document -> case comparisonInputs document of+      Left message -> pure (Left message)+      Right (policy, axes, pairs) -> do+        let baselines = [root </> "baseline" </> show index | index <- [0 .. pairs - 1]]+            candidates = [root </> "candidate" </> show index | index <- [0 .. pairs - 1]]+        verified <- forM (baselines <> candidates) $ \path -> do+          measured <- summarizeRunDir path+          result <- eitherDecodeFileStrict' (path </> "run-result.json") :: IO (Either String Value)+          pure case (measured, result) of+            (Right summary, Right value) | storedMeasurements value == Just (toJSON summary) -> Right ()+            (Left (SummaryError message), _) -> Left message+            (_, Left message) -> Left (Text.pack message)+            _ -> Left "arm measurements differ from the sealed samples and series"+        case sequence verified of+          Left message -> pure (Left message)+          Right _ -> do+            replayed <- compareRuns policy axes baselines candidates+            pure case replayed of+              Left (CompareError message) -> Left message+              Right comparison ->+                let actual = toJSON comparison+                    stableKeys = ["design", "baselineRuns", "candidateRuns", "algorithm", "policy", "variedFactors", "pairCount", "metrics", "reasons", "verdict", "exitCode"]+                    agrees = all (\key -> field key actual == field key document) stableKeys+                 in Right+                      Recomputation+                        { agreesWithDocuments = agrees,+                          outcome = Nothing,+                          comparisonVerdict = case field "verdict" actual of Just (String verdict) -> Just verdict; _ -> Nothing,+                          detail = "recomputed arm summaries and the paired comparison from sealed data"+                        }+  where+    field name (Object value) = KeyMap.lookup name value+    field _ _ = Nothing++comparisonInputs :: Value -> Either Text (Policy, NonEmpty.NonEmpty VaryingAxis, Int)+comparisonInputs (Object document) = do+  rawPolicy <- maybe (Left "comparison has no policy") Right (KeyMap.lookup "policy" document)+  policy <- case fromJSON rawPolicy of+    Error message -> Left (Text.pack message)+    Success value -> Right value+  rawAxes <- maybe (Left "comparison has no varied factors") Right (KeyMap.lookup "variedFactors" document)+  axisNames <- case fromJSON rawAxes of+    Error message -> Left (Text.pack message)+    Success value -> Right value+  axes <- traverse parseVaryingAxis axisNames >>= maybe (Left "comparison has no varied factors") Right . NonEmpty.nonEmpty+  rawPairs <- maybe (Left "comparison has no pair count") Right (KeyMap.lookup "pairCount" document)+  pairs <- case fromJSON rawPairs of+    Error message -> Left (Text.pack message)+    Success value -> Right value+  if pairs > 0 then Right (policy, axes, pairs) else Left "comparison pair count must be positive"+comparisonInputs _ = Left "comparison document is not an object"
+ src/Kenshou/Cli/Cohort.hs view
@@ -0,0 +1,93 @@+{-# LANGUAGE FieldSelectors #-}++module Kenshou.Cli.Cohort+  ( runCohortCommand,+    resolveDefaultCohortIdentity,+  )+where++import Control.Applicative ((<|>))+import Data.Aeson qualified as Aeson+import Data.ByteString.Lazy.Char8 qualified as LazyByteString+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Text.IO qualified as Text.IO+import Kenshou.Cli.Options (CohortCommand (..))+import Kenshou.Core.Cohort+import System.Environment (lookupEnv)+import System.Exit (ExitCode (..))+import System.FilePath ((</>))+import System.IO (stderr)++resolveDefaultCohortIdentity :: IO (Either CohortError CohortIdentity)+resolveDefaultCohortIdentity = do+  selected <- lookupEnv "KENSHOU_COHORT_IDENTITY"+  resolveCohortIdentity (maybe (FromProject "." Nothing Nothing) FromIdentityFile selected)++runCohortCommand :: CohortCommand -> IO ExitCode+runCohortCommand (CohortShow json projectDir planJson identityOption) = do+  identityEnvironment <- lookupEnv "KENSHOU_COHORT_IDENTITY"+  let source =+        maybe+          (FromProject projectDir planJson Nothing)+          FromIdentityFile+          (identityOption <|> identityEnvironment)+  result <- resolveCohortIdentity source+  case result of+    Left err -> reportError err+    Right identity -> do+      if json+        then LazyByteString.putStrLn (Aeson.encode identity)+        else Text.IO.putStr (renderCohortIdentity identity)+      pure ExitSuccess+runCohortCommand (CohortCheck projectDir planJson descriptorOverride identityOption) = do+  identityEnvironment <- lookupEnv "KENSHOU_COHORT_IDENTITY"+  descriptorResult <- loadActiveDescriptor projectDir descriptorOverride+  identityResult <-+    resolveCohortIdentity $+      maybe+        (FromProject projectDir planJson descriptorOverride)+        FromIdentityFile+        (identityOption <|> identityEnvironment)+  case (descriptorResult, identityResult) of+    (Left err, _) -> reportError err+    (_, Left err) -> reportError err+    (Right descriptor, Right identity) ->+      case checkCohort descriptor identity of+        [] -> pure ExitSuccess+        mismatches -> do+          mapM_ (Text.IO.hPutStrLn stderr . renderMismatch) mismatches+          pure (ExitFailure 1)++loadActiveDescriptor :: FilePath -> Maybe FilePath -> IO (Either CohortError CohortDescriptor)+loadActiveDescriptor _ (Just path) = loadCohortDescriptor path+loadActiveDescriptor projectDir Nothing = do+  nameResult <- activeCohortName projectDir+  case nameResult of+    Left err -> pure (Left err)+    Right (CohortName name) -> loadCohortDescriptor (projectDir </> "cohort" </> Text.unpack name <> ".json")++reportError :: CohortError -> IO ExitCode+reportError err = do+  Text.IO.hPutStrLn stderr (renderError err)+  pure (ExitFailure 4)++renderError :: CohortError -> Text+renderError (CohortIoError message)+  | "dist-newstyle/cache/plan.json" `Text.isInfixOf` message = "no dist-newstyle/cache/plan.json; run cabal build all first"+  | otherwise = message+renderError (CohortDecodeError message) = message+renderError (CohortPlanError message) = message+renderError (CohortInvalidActive message) = message++renderMismatch :: CohortMismatch -> Text+renderMismatch (CohortNameMismatch (CohortName expected) (CohortName actual)) =+  "cohort: expected " <> expected <> ", resolved " <> actual+renderMismatch (CompilerMismatch expected actual) =+  "compiler: expected " <> expected <> ", resolved " <> actual+renderMismatch (MissingPackage packageName) = "missing package: " <> packageName+renderMismatch (VersionMismatch packageName expected actual) =+  packageName <> ": expected version " <> expected <> ", resolved " <> actual+renderMismatch (SourceMismatch packageName expected actual) =+  packageName <> ": expected source " <> Text.pack (show expected) <> ", resolved " <> Text.pack (show actual)+renderMismatch (LocalPathSource packageName path) = packageName <> ": resolved from forbidden local path " <> Text.pack path
+ src/Kenshou/Cli/Command/Cell.hs view
@@ -0,0 +1,1208 @@+module Kenshou.Cli.Command.Cell (cellCommand, startCellInstances) where++import Control.Concurrent (threadDelay)+import Control.Exception (SomeAsyncException, SomeException, displayException, finally, fromException, throwIO, try)+import Data.Aeson (FromJSON, Value (..), eitherDecode, eitherDecodeFileStrict', encode, object, (.=))+import Data.Aeson qualified as Aeson+import Data.Aeson.Key qualified as Key+import Data.Aeson.KeyMap qualified as KeyMap+import Data.ByteString qualified as ByteString+import Data.ByteString.Lazy.Char8 qualified as LazyByteString+import Data.Foldable (traverse_)+import Data.List (find, nub, sort)+import Data.List.NonEmpty (NonEmpty (..))+import Data.List.NonEmpty qualified as NonEmpty+import Data.Map.Strict qualified as Map+import Data.Maybe (fromMaybe, isNothing)+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Text.IO qualified as TextIO+import Data.Time (diffUTCTime, getCurrentTime)+import Kenshou.Cli.Command.Overhead qualified as LocalOverhead+import Kenshou.Core.Bundle (lookupScenario, registryRoutingRules)+import Kenshou.Core.Cli (CliCommand (..), CliEnv (..), CliGroup (..))+import Kenshou.Core.Cohort (CohortIdentity (..), PlanHash (..))+import Kenshou.Core.Id (RunId, mkSeed, newRunId, parseRunId, parseScenarioId, renderRunId, unSeed)+import Kenshou.Core.RunSpec (EnvironmentSpec (..), RunSpec (..), SpecPlacement (..))+import Kenshou.Core.Scenario (Scenario (..))+import Kenshou.Diagnose.Leak (analyseRunDirectory, defaultLeakSpec)+import Kenshou.Measure.Compare (CompareError (..), Comparison (..), compareRuns, verdictExitCode)+import Kenshou.Measure.Compare.Compatibility (VaryingAxis (..), compatibilityInputs, varyingValue)+import Kenshou.Measure.Compare.Ordering (Arm (..), PairedOrdering (..))+import Kenshou.Measure.Compare.Policy (Policy (..), decodePolicy)+import Kenshou.Plan.Change (Change (..), ChangeSource (..), Reason (..))+import Kenshou.Plan.Components (ComponentId (..), ComponentRef (..))+import Kenshou.Plan.Policy (PlanPolicy (..), defaultPlanPolicy)+import Kenshou.Plan.RunPlan (PlanContext (..), PlanInputs (PlanInputs), PlannedRun (..), RunPlan (..), TrialInfo (..))+import Kenshou.Plan.Selector (parseSelector)+import Kenshou.Remote.Cell.Control (CellSnapshot (..), readCellSnapshot)+import Kenshou.Remote.Cell.Debug (DebugCommand (..), runCellDebug)+import Kenshou.Remote.Cell.Docs (CachePolicy (..), CellBuckets (..), CellDescriptor (..), CellManifest (..), CellNodes (..), CellOutcome (..), CellStatus (..), Limits (..), Rejected (..), Submission (..))+import Kenshou.Remote.Cell.Exec (cellExec)+import Kenshou.Remote.Cell.Fetch (FetchError (..), fetchCellRun, verifyCellRun)+import Kenshou.Remote.Cell.Index (deriveCellRunIndex, writeCellRunIndex)+import Kenshou.Remote.Cell.Lease (AcquireOutcome (..), CellRef (..), Lease (..), LeaseHandle, LeaseRequest (..), Quarantine (..), acquireLease, leaseSnapshot, reattachLease, releaseLease, resizeLease, validCellName, withHeartbeat)+import Kenshou.Remote.Cell.Parity (ParityOptions (..), ParityReport (..), compareForParity)+import Kenshou.Remote.Cell.Prepare (Granularity (..), OtlpSink (..), PrepareOptions (..))+import Kenshou.Remote.Cell.RouteJson (RouteDocuments (..), routeWorkJson)+import Kenshou.Remote.Cell.RouteRules (CellCapabilities (..), descriptorDigest, loadCellCapabilities)+import Kenshou.Remote.Cell.Session (SessionTransition (..))+import Kenshou.Remote.Cell.Session.Build (BuildOptions (..), BuiltSession (..), buildSession)+import Kenshou.Remote.Cell.Session.Journal (LeaseMode (..), SessionJournal (..), SliceJournal (..), SliceState (..), applyTransition, readSessionJournal, writeSessionJournal)+import Kenshou.Remote.Cell.Session.Resume (resumeHeldSession, resumeObservedSlices)+import Kenshou.Remote.Cell.Session.Runner (runPlannedSlices)+import Kenshou.Remote.Cell.Submit (PublishOutcome (Submitted), publishSubmission)+import Kenshou.Remote.Cell.Watch (WatchEvent (..), WatchTerminal (..), watchCellRun)+import Kenshou.Remote.Cell.WorkJson (decodeWorkPlan)+import Kenshou.Remote.Pair (PairRequest (..), PairState (..), SliceResult (..), judgePairs, pairPlan)+import Kenshou.Remote.Payload (Bundle (..), CellPayload (..), PayloadDescriptor (..))+import Kenshou.Remote.Payload.Publisher (PublishError (..), PublishOptions (..), publishPayload)+import Kenshou.Remote.Store (Bucket (..), ObjectMeta (..), ObjectName (..), ObjectStore (..))+import Kenshou.Remote.Store.File (newFileStore)+import Kenshou.Remote.Store.Gcs (newGcsStore, newTokenProvider)+import Kenshou.Telemetry.Overhead (OverheadHooks (..), OverheadPlan (..), OverheadReport (..), OverheadRequest (..), OverheadState (..), UsageError (..), analyseOverhead, executeOverhead, loadOverheadState, overheadVerdictExitCode, planOverhead)+import Kenshou.Telemetry.Overhead.Policy (OverheadPolicy)+import Options.Applicative+import System.Directory (createDirectoryIfMissing, createDirectoryLink, doesDirectoryExist, doesPathExist, makeAbsolute, renameFile)+import System.Environment (getExecutablePath, lookupEnv)+import System.Exit (ExitCode (..))+import System.FilePath (takeDirectory, (</>))+import System.IO (hFlush, stderr, stdout)+import System.IO.Temp (withSystemTempDirectory)+import System.Process (readProcessWithExitCode)++data CellLocation = CellLocation !Text !(Maybe Text)++data LeaseCli = LeaseCli !CellLocation !Text !(Maybe Text) !Int !Int !Bool !Bool !Bool++data RouteCli = RouteCli ![Text] !(Maybe Text) !FilePath !(Maybe FilePath) !(Maybe FilePath) !FilePath !Bool !Bool++data SubmitSettings = SubmitSettings ![String] !FilePath !FilePath !Granularity !CachePolicy ![Text] !Bool !Bool !Bool !(Maybe FilePath) !OtlpSink !(Maybe Text) !Bool++data SubmitCli = SubmitCli !CellLocation !Text !SubmitSettings !Bool++data RunCli = RunCli !CellLocation !SubmitSettings !Int !Int !Bool !Bool++data ProbeCli = ProbeCli !CellLocation !FilePath !(Maybe FilePath)++data PairCli = PairCli !CellLocation !FilePath !FilePath !FilePath !(Maybe Int) !PairedOrdering !FilePath ![Text] !Int !Bool !FilePath++data PairObservation = PairObservation !PlannedRun !FilePath !Bool++data OverheadCli = OverheadCli !CellLocation !FilePath !OtlpSink ![Text] !Bool !LocalOverhead.OverheadOptions++data PayloadPublishCli = PayloadPublishCli !Text !Text !(Maybe Text) !FilePath !FilePath !Bool++data CellAction+  = Fetch !Text !Text !FilePath+  | Verify !FilePath+  | Parity !FilePath !FilePath !(Maybe FilePath) ![Text]+  | Status !CellLocation !Bool+  | LeaseCell !LeaseCli+  | Release !CellLocation !Text+  | Watch !CellLocation !Text+  | Debug !CellLocation !DebugCommand+  | Route !RouteCli+  | Submit !SubmitCli+  | Run !RunCli+  | Probe !ProbeCli+  | Pair !PairCli+  | Overhead !OverheadCli+  | Resume !FilePath !(Maybe Text) !Bool+  | PublishPayload !PayloadPublishCli+  | ShowPayload !FilePath+  | Exec !FilePath !FilePath++cellCommand :: CliCommand+cellCommand = CliCommand "cell" "Run and inspect leased verification cells" Execution False (runCell <$> cellParser)++cellParser :: Parser CellAction+cellParser =+  hsubparser $+    command "fetch" (info (fetchParser <**> helper) (progDesc "Fetch and verify one sealed cell run"))+      <> command "verify" (info (verifyParser <**> helper) (progDesc "Verify a fetched tree or a sealed GCS run"))+      <> command "parity" (info (parityParser <**> helper) (progDesc "Compare local and cell verdicts with named placement differences"))+      <> command "status" (info (statusParser <**> helper) (progDesc "Inspect cell descriptor and lease state"))+      <> command "lease" (info (leaseParser <**> helper) (progDesc "Acquire a cell lease"))+      <> command "release" (info (releaseParser <**> helper) (progDesc "Release an owned cell lease"))+      <> command "watch" (info (watchParser <**> helper) (progDesc "Follow a cell run to its terminal status"))+      <> command "debug" (info (debugParser <**> helper) (progDesc "Inspect a cell role through the owner IAP script"))+      <> command "route" (info (routeParser <**> helper) (progDesc "Split a plan across compatible cells"))+      <> command "submit" (info (submitParser <**> helper) (progDesc "Submit a prepared plan under an existing lease"))+      <> command "run" (info (runParser <**> helper) (progDesc "Lease, submit, verify and release a cell session"))+      <> command "probe" (info (probeParser <**> helper) (progDesc "Run a cell environment probe and cache capabilities"))+      <> command "pair" (info (pairParser <**> helper) (progDesc "Measure interleaved payload arms under one cell lease"))+      <> command "overhead" (info (overheadCellParser <**> helper) (progDesc "Measure telemetry overhead under one cell lease"))+      <> command "resume" (info (resumeParser <**> helper) (progDesc "Continue a saved cell session"))+      <> command "payload" (info (payloadParser <**> helper) (progDesc "Build and publish a checked cell payload"))+      <> (command "exec" (info (cellExecParser <**> helper) (progDesc "Execute prepared work on a cell driver")) <> internal)++payloadParser :: Parser CellAction+payloadParser =+  hsubparser $+    command "publish" (info (publishPayloadParser <**> helper) (progDesc "Publish a Nix closure after checking its runtime cohort"))+      <> command "show" (info (ShowPayload <$> strArgument (metavar "FILE") <**> helper) (progDesc "Show a payload descriptor after checking its bundle exists"))++publishPayloadParser :: Parser CellAction+publishPayloadParser =+  PublishPayload+    <$> ( PayloadPublishCli+            <$> strOption (long "cohort" <> metavar "released|head" <> help "Pinned runtime cohort")+            <*> strOption (long "variant" <> metavar "default|info-table|profiled" <> value "default" <> showDefault <> help "Executable build variant")+            <*> optional (strOption (long "control-bucket" <> metavar "BUCKET" <> help "Payload control bucket"))+            <*> strOption (long "out" <> metavar "FILE" <> help "Payload descriptor JSON output")+            <*> strOption (long "root" <> metavar "DIR" <> value "." <> showDefault <> help "Flake and cohort descriptor root")+            <*> switch (long "allow-dirty" <> help "Permit a payload from a dirty worktree")+        )++cellLocationParser :: Parser CellLocation+cellLocationParser =+  CellLocation+    <$> strOption (long "cell" <> metavar "NAME" <> help "Cell name")+    <*> optional (strOption (long "control-bucket" <> metavar "BUCKET" <> help "Cell control bucket"))++statusParser :: Parser CellAction+statusParser = Status <$> cellLocationParser <*> switch (long "json" <> help "Print JSON")++leaseParser :: Parser CellAction+leaseParser =+  LeaseCell+    <$> ( LeaseCli+            <$> cellLocationParser+            <*> strOption (long "purpose" <> metavar "TEXT" <> help "Purpose recorded in the lease")+            <*> optional (strOption (long "owner" <> metavar "TEXT" <> help "Lease owner; defaults to USER"))+            <*> option auto (long "ttl" <> metavar "SECONDS" <> value 120 <> showDefault <> help "Lease lifetime between renewals")+            <*> option auto (long "wait" <> metavar "SECONDS" <> value 0 <> showDefault <> help "Wait for a busy lease")+            <*> switch (long "start" <> help "Start stopped cell instances before acquiring a lease")+            <*> switch (long "hold" <> help "Renew in the foreground until interrupted")+            <*> switch (long "json" <> help "Print the lease document as JSON")+        )++releaseParser :: Parser CellAction+releaseParser = Release <$> cellLocationParser <*> strOption (long "lease-id" <> metavar "UUID" <> help "Expected lease identifier")++watchParser :: Parser CellAction+watchParser = Watch <$> cellLocationParser <*> strArgument (metavar "CELL_RUN_ID")++debugParser :: Parser CellAction+debugParser =+  Debug+    <$> cellLocationParser+    <*> hsubparser+      ( command "ssh" (info (DebugSsh <$> strArgument (metavar "ROLE") <*> many (strArgument (metavar "COMMAND")) <**> helper) (progDesc "Open a role shell or run a remote command"))+          <> command "journal" (info (pure DebugJournal <**> helper) (progDesc "Read the driver agent journal"))+          <> command "tunnel" (info (DebugTunnel <$> strArgument (metavar "ROLE") <*> argument auto (metavar "REMOTE_PORT") <*> argument auto (metavar "LOCAL_PORT") <**> helper) (progDesc "Forward a local port through IAP SSH until interrupted"))+      )++routeParser :: Parser CellAction+routeParser =+  Route+    <$> ( RouteCli+            <$> some (strOption (long "cell" <> metavar "NAME" <> help "Candidate cell in preference order; repeat to add cells"))+            <*> optional (strOption (long "control-bucket" <> metavar "BUCKET" <> help "Cell control bucket"))+            <*> strOption (long "plan" <> metavar "FILE" <> help "Run plan JSON, or - for stdin")+            <*> optional (strOption (long "payload" <> metavar "FILE" <> help "Payload descriptor for cohort preparation"))+            <*> optional (strOption (long "routing-rules" <> metavar "FILE" <> help "Replace the built-in routing policy with JSON, or - for stdin"))+            <*> strOption (long "out" <> metavar "DIR" <> help "Directory for routed plans and refusal report")+            <*> switch (long "coerce-durable" <> help "Use durable PostgreSQL when the scenario supports it")+            <*> switch (long "ephemeral-on-driver" <> help "Keep server-control correctness work on the cell driver")+        )++submitParser :: Parser CellAction+submitParser =+  Submit+    <$> ( SubmitCli+            <$> cellLocationParser+            <*> strOption (long "lease-id" <> metavar "UUID" <> help "Active lease identifier")+            <*> submitSettingsParser (switch (long "dry-run" <> help "Print a planned session without writing or submitting"))+            <*> pure False+        )++runParser :: Parser CellAction+runParser =+  Run+    <$> ( RunCli+            <$> cellLocationParser+            <*> submitSettingsParser (pure False)+            <*> option auto (long "ttl" <> metavar "SECONDS" <> value 120 <> showDefault <> help "Held lease lifetime between renewals")+            <*> option auto (long "wait" <> metavar "SECONDS" <> value 0 <> showDefault <> help "Wait for a busy cell")+            <*> switch (long "start" <> help "Start stopped cell instances before acquiring a lease")+            <*> switch (long "detach" <> help "Submit one slice and leave its budgeted lease to run after this process exits")+        )++probeParser :: Parser CellAction+probeParser =+  Probe+    <$> ( ProbeCli+            <$> cellLocationParser+            <*> strOption (long "payload" <> metavar "FILE" <> help "Published payload descriptor with the cell-environment scenario")+            <*> optional (strOption (long "out" <> metavar "FILE" <> help "Capability cache JSON; defaults to the configured cache directory"))+        )++pairParser :: Parser CellAction+pairParser =+  Pair+    <$> ( PairCli+            <$> cellLocationParser+            <*> strOption (long "candidate" <> metavar "FILE" <> help "Candidate payload descriptor")+            <*> strOption (long "baseline" <> metavar "FILE" <> help "Baseline payload descriptor")+            <*> strOption (long "plan" <> metavar "FILE" <> help "Benchmark run plan JSON")+            <*> optional (option auto (long "pairs" <> metavar "N" <> help "Requested valid pairs; defaults to the policy minimum"))+            <*> option (eitherReader parsePairOrdering) (long "ordering" <> metavar "abba|baab" <> value ABBA <> showDefaultWith (const "abba") <> help "Order baseline and candidate in adjacent pairs")+            <*> strOption (long "policy" <> metavar "FILE" <> value "policies/default.json" <> showDefault <> help "Comparison policy JSON")+            <*> many (strOption (long "pg-setting" <> metavar "KEY=VALUE" <> help "PostgreSQL reset setting; repeat as needed"))+            <*> option auto (long "max-replacements" <> metavar "N" <> value (-1) <> help "Additional pairs after invalid trials; defaults to requested pairs")+            <*> switch (long "start" <> help "Start stopped cell instances before acquiring a lease")+            <*> strOption (long "out" <> metavar "DIR" <> help "New directory for paired cell sessions and comparison")+        )++parsePairOrdering :: String -> Either String PairedOrdering+parsePairOrdering "abba" = Right ABBA+parsePairOrdering "baab" = Right BAAB+parsePairOrdering _ = Left "expected abba or baab"++overheadCellParser :: Parser CellAction+overheadCellParser =+  Overhead+    <$> ( OverheadCli+            <$> cellLocationParser+            <*> strOption (long "payload" <> metavar "FILE" <> help "Payload descriptor for every telemetry arm")+            <*> option (eitherReader parseOtlpSink) (long "otlp-sink" <> metavar "null|file" <> value NullSink <> showDefaultWith (const "null") <> help "Cell OTLP sink")+            <*> many (strOption (long "pg-setting" <> metavar "KEY=VALUE" <> help "PostgreSQL reset setting; repeat as needed"))+            <*> switch (long "start" <> help "Start stopped cell instances before acquiring a lease")+            <*> LocalOverhead.overheadParser+        )++submitSettingsParser :: Parser Bool -> Parser SubmitSettings+submitSettingsParser dryFlag =+  SubmitSettings+    <$> some (strOption (long "payload" <> metavar "[LABEL=]FILE" <> help "Payload descriptor; repeat for trial arms"))+    <*> strOption (long "plan" <> metavar "FILE" <> help "Run plan JSON, or - for stdin")+    <*> strOption (long "out" <> metavar "DIR" <> help "New session directory")+    <*> option (eitherReader parseGranularity) (long "granularity" <> metavar "auto|plan|run" <> value GranularityAuto <> showDefaultWith (const "auto") <> help "Submission slice boundary")+    <*> option (eitherReader parseCachePolicy) (long "cache-policy" <> metavar "cold|warm" <> value Cold <> showDefaultWith (const "cold") <> help "Cell cache reset policy")+    <*> many (strOption (long "pg-setting" <> metavar "KEY=VALUE" <> help "PostgreSQL reset setting; repeat as needed"))+    <*> switch (long "skip-incompatible" <> help "Record incompatible runs and submit compatible ones")+    <*> switch (long "coerce-durable" <> help "Use durable PostgreSQL when the scenario supports it")+    <*> switch (long "ephemeral-on-driver" <> help "Keep server-control correctness work on the cell driver")+    <*> optional (strOption (long "routing-rules" <> metavar "FILE" <> help "Replace the built-in routing policy JSON"))+    <*> option (eitherReader parseOtlpSink) (long "otlp-sink" <> metavar "null|file" <> value NullSink <> showDefaultWith (const "null") <> help "Cell OTLP sink")+    <*> optional (strOption (long "rts" <> metavar "OPTS" <> help "Runtime system options passed to the payload"))+    <*> dryFlag++parseGranularity :: String -> Either String Granularity+parseGranularity "auto" = Right GranularityAuto+parseGranularity "plan" = Right GranularityPlan+parseGranularity "run" = Right GranularityRun+parseGranularity _ = Left "expected auto, plan or run"++parseCachePolicy :: String -> Either String CachePolicy+parseCachePolicy "cold" = Right Cold+parseCachePolicy "warm" = Right Warm+parseCachePolicy _ = Left "expected cold or warm"++parseOtlpSink :: String -> Either String OtlpSink+parseOtlpSink "null" = Right NullSink+parseOtlpSink "file" = Right FileSink+parseOtlpSink _ = Left "expected null or file"++resumeParser :: Parser CellAction+resumeParser =+  Resume+    <$> strOption (long "session" <> metavar "DIR" <> help "Directory containing session.json")+    <*> optional (strOption (long "lease-id" <> metavar "UUID" <> help "Use this active lease instead of the journal lease"))+    <*> switch (long "start" <> help "Start stopped cell instances before resuming")++cellExecParser :: Parser CellAction+cellExecParser = Exec <$> strArgument (metavar "WORK_FILE") <*> strArgument (metavar "OUT_DIR")++fetchParser :: Parser CellAction+fetchParser =+  Fetch+    <$> strOption (long "results-bucket" <> metavar "BUCKET" <> help "Results bucket containing the sealed run")+    <*> strArgument (metavar "CELL_RUN_ID")+    <*> strOption (long "out" <> metavar "DIR" <> help "Directory for the fetched cell run")++verifyParser :: Parser CellAction+verifyParser = Verify <$> strArgument (metavar "DIR_OR_GS_URI")++parityParser :: Parser CellAction+parityParser =+  Parity+    <$> strOption (long "local" <> metavar "DIR" <> help "Local run directory")+    <*> strOption (long "cell" <> metavar "DIR" <> help "Fetched nested cell run directory")+    <*> optional (strOption (long "out" <> metavar "FILE" <> help "Write the parity report to FILE instead of stdout"))+    <*> many (strOption (long "volatile" <> metavar "PATH" <> help "Allow a named volatile verdict field; repeat as needed"))++runCell :: CellAction -> CliEnv -> IO ExitCode+runCell selected cli = case selected of+  PublishPayload (PayloadPublishCli cohort variant selectedBucket output root allowDirty) -> do+    configured <- lookupEnv "KENSHOU_CELL_CONTROL_BUCKET"+    let bucketName = fromMaybe (maybe "tan-nb-exp-cells-control" Text.pack configured) selectedBucket+    case validateBucket bucketName of+      Left problem -> usage problem+      Right bucket -> guardIO do+        store <- openStore bucket+        published <- publishPayload store (PublishOptions root cohort variant bucket output allowDirty)+        case published of+          Left (PublishInvalidSelection problem) -> usage problem+          Left (PublishCohortMismatch problem) -> failVerification problem+          Left PublishDirtyWorktree -> unavailable "worktree is dirty; pass --allow-dirty to publish it"+          Left problem -> unavailable (Text.pack (show problem))+          Right _ -> TextIO.putStrLn (Text.pack output) >> pure ExitSuccess+  ShowPayload path -> guardIO do+    decoded <- eitherDecodeFileStrict' path+    case decoded of+      Left problem -> failVerification (Text.pack problem)+      Right (descriptor :: PayloadDescriptor) -> case parsePayloadUri descriptor.cell.bundle.uri of+        Left problem -> failVerification problem+        Right (bucket, bundleObject) -> do+          store <- openStore bucket+          observed <- store.statObject bucket bundleObject+          case observed of+            Nothing -> failVerification "payload bundle does not exist"+            Just meta+              | meta.size /= descriptor.cell.bundle.bytes -> failVerification "payload bundle size differs from descriptor"+              | otherwise -> LazyByteString.putStrLn (encode descriptor) >> pure ExitSuccess+  Fetch bucket identifier outDir -> case (validateBucket bucket, parseRunId identifier) of+    (Left problem, _) -> usage problem+    (_, Left problem) -> usage problem+    (Right resultsBucket, Right cellRun) -> guardIO $ do+      store <- openStore resultsBucket+      fetched <- fetchCellRun store resultsBucket cellRun outDir+      case fetched of+        Left problem -> fetchFailure problem+        Right tree -> do+          indexed <- deriveCellRunIndex resultsBucket Nothing tree+          case indexed of+            Left problems -> failVerification (Text.pack (show (NonEmpty.toList problems)))+            Right index -> do+              path <- writeCellRunIndex tree index+              TextIO.putStrLn (Text.pack path)+              pure ExitSuccess+  Verify location -> case Text.stripPrefix "gs://" (Text.pack location) of+    Nothing -> guardIO $ do+      let nested = location </> "tree"+      isRunDirectory <- doesDirectoryExist nested+      exists <- doesDirectoryExist location+      if exists then verifyTree (if isRunDirectory then nested else location) else unavailable "verification directory does not exist"+    Just suffix -> case parseResultsUri suffix of+      Left problem -> usage problem+      Right (bucket, identifier) -> guardIO $+        withSystemTempDirectory "kenshou-cell-verify" \temporary -> do+          store <- openStore bucket+          fetched <- fetchCellRun store bucket identifier temporary+          case fetched of+            Left problem -> fetchFailure problem+            Right tree -> verifyTree tree+  Parity localDir cellDir output volatile+    | any (\path -> not ("result.summaries.verdicts." `Text.isPrefixOf` path || "verdicts." `Text.isPrefixOf` path)) volatile -> usage "--volatile must name a field under result.summaries.verdicts or verdicts"+    | otherwise -> guardIO do+        report <- compareForParity (ParityOptions volatile) localDir cellDir+        case output of+          Nothing -> LazyByteString.putStrLn (encode report)+          Just path -> do+            createDirectoryIfMissing True (takeDirectory path)+            LazyByteString.writeFile path (encode report)+            TextIO.putStrLn (Text.pack path)+        pure (if null report.unexpected then ExitSuccess else ExitFailure 1)+  Status location asJson -> withControl location \_ _ snapshot -> do+    if asJson+      then LazyByteString.putStrLn (encode (object ["descriptor" .= snapshot.descriptor, "lease" .= snapshot.lease, "quarantine" .= snapshot.quarantine]))+      else do+        let descriptor = snapshot.descriptor+        TextIO.putStrLn ("cell " <> descriptor.name <> " project=" <> descriptor.project <> " zone=" <> descriptor.zone <> " pg=" <> Text.pack (show descriptor.postgresMajor))+        TextIO.putStrLn ("lease " <> maybe "none" (renderRunId . (.leaseId)) snapshot.lease)+        TextIO.putStrLn ("quarantine " <> maybe "none" (.reason) snapshot.quarantine)+    pure ExitSuccess+  Debug location debugCommand -> withControl location \_ _ snapshot -> do+    result <- runCellDebug snapshot.descriptor debugCommand+    either unavailable pure result+  LeaseCell (LeaseCli location purpose owner ttl waitSeconds startInstances hold asJson)+    | ttl <= 0 || waitSeconds < 0 || Text.null purpose || maybe False Text.null owner -> usage "lease purpose and TTL must be positive; wait cannot be negative"+    | otherwise -> withControl location \store ref observed -> do+        if startInstances then startCellInstances observed.descriptor else pure ()+        defaultOwner <- Text.pack . fromMaybe "kenshou" <$> lookupEnv "USER"+        let request = LeaseRequest (fromMaybe defaultOwner owner) purpose ttl+        start <- getCurrentTime+        let acquire = do+              outcome <- acquireLease store ref request+              case outcome of+                Acquired handle -> do+                  record <- leaseSnapshot handle+                  if asJson+                    then LazyByteString.putStrLn (encode record)+                    else TextIO.putStrLn ("lease " <> renderRunId record.leaseId <> " ttl=" <> Text.pack (show record.ttlSeconds))+                  if hold+                    then withHeartbeat store ref handle (holdLease) `finally` (do _ <- releaseLease store ref handle; pure ())+                    else pure ExitSuccess+                Busy current -> do+                  now <- getCurrentTime+                  if diffUTCTime now start < fromIntegral waitSeconds+                    then threadDelay 2000000 >> acquire+                    else unavailable ("cell is busy under lease " <> renderRunId current.leaseId)+                Quarantined record -> unavailable ("cell is quarantined: " <> record.reason)+        acquire+  Release location leaseText -> case parseRunId leaseText of+    Left problem -> usage problem+    Right expected -> withControl location \store ref _ -> do+      reattached <- reattachLease store ref expected+      case reattached of+        Nothing -> unavailable "cell has no matching lease"+        Just handle -> do+          released <- releaseLease store ref handle+          if released then TextIO.putStrLn ("released " <> leaseText) >> pure ExitSuccess else unavailable "cell lease changed before release"+  Watch location runText -> case parseRunId runText of+    Left problem -> usage problem+    Right identifier -> withControl location \store ref _ -> do+      terminal <- watchCellRun store ref identifier emitWatchEvent+      case terminal of+        RunRejected rejected -> do+          TextIO.hPutStrLn stderr ("rejected: " <> rejected.reason)+          pure (if rejected.reason `elem` ["lease-mismatch", "payload-digest-mismatch"] then ExitFailure 4 else ExitFailure 2)+        RunSealed status -> do+          TextIO.hPutStrLn stderr ("sealed: " <> Text.pack (show status.outcome))+          pure (if status.outcome == Just Completed then ExitSuccess else ExitFailure 4)+  Route (RouteCli names bucket planFile payloadFile rulesFile outDir coerce ephemeral)+    | length (nub names) /= length names || not (all validCellName names) -> usage "cell names must be valid and unique"+    | planFile == "-" && rulesFile == Just "-" -> usage "only one JSON document may be read from stdin"+    | payloadFile == Just "-" -> usage "payload must name a file"+    | otherwise -> case NonEmpty.nonEmpty names of+        Nothing -> usage "at least one cell is required"+        Just (first :| rest) -> withControl (CellLocation first bucket) \store ref observed -> do+          allowed <- maybe ["tan-nb-exp"] (map Text.strip . Text.splitOn "," . Text.pack) <$> lookupEnv "KENSHOU_GCP_ALLOWED_PROJECTS"+          remaining <- traverse (\name -> readCellSnapshot store (CellRef name ref.controlBucket) allowed) rest+          case sequence remaining of+            Left problem -> unavailable problem+            Right snapshots -> do+              let descriptors = observed.descriptor : fmap (.descriptor) snapshots+              capabilityResult <- loadCapabilitiesFor descriptors+              planResult <- readJsonDocument planFile+              rulesResult <- maybe (pure (Right (registryRoutingRules cli.registry))) readJsonDocument rulesFile+              payloadResult <- traverse eitherDecodeFileStrict' payloadFile+              case (capabilityResult, planResult, rulesResult, sequence payloadResult) of+                (Left problem, _, _, _) -> usage problem+                (_, Left failure, _, _) -> usage (Text.pack failure)+                (_, _, Left failure, _) -> usage (Text.pack failure)+                (_, _, _, Left failure) -> usage (Text.pack failure)+                (Right caches, Right plan, Right rules, Right payload) ->+                  case NonEmpty.nonEmpty (zip descriptors caches) >>= \cells -> Just (routeWorkJson cli.registry cells rules (Map.singleton "default" <$> payload) (PrepareOptions coerce ephemeral [] Cold) plan) of+                    Nothing -> usage "at least one cell is required"+                    Just (Left problem) -> usage problem+                    Just (Right documents) -> guardIO do+                      exists <- doesPathExist outDir+                      if exists+                        then usage "route output path already exists; choose a new directory"+                        else do+                          createDirectoryIfMissing True outDir+                          traverse_ (\(name, planDocument) -> LazyByteString.writeFile (outDir </> "plan." <> Text.unpack name <> ".json") (encode planDocument)) (Map.toAscList documents.perCell)+                          traverse_ (\planDocument -> LazyByteString.writeFile (outDir </> "plan.local.json") (encode planDocument)) documents.local+                          LazyByteString.writeFile (outDir </> "unroutable.json") (encode documents.report)+                          TextIO.hPutStrLn stderr ("routed plans: " <> Text.pack (show (Map.size documents.perCell)) <> "; see " <> Text.pack (outDir </> "unroutable.json"))+                          pure (if documents.routeComplete then ExitSuccess else ExitFailure 2)+  Submit (SubmitCli location leaseText (SubmitSettings payloadFiles planFile outDir granularity cache settingTexts skip coerce ephemeral rulesFile sink rts dryRun) detached)+    | planFile == "-" && rulesFile == Just "-" -> usage "only one JSON document may be read from stdin"+    | otherwise -> case (parseRunId leaseText, traverse parsePgSetting settingTexts) of+        (Left problem, _) -> usage problem+        (_, Left problem) -> usage problem+        (Right leaseId, Right settings) -> withControl location \store ref observed -> do+          handle <- reattachLease store ref leaseId+          case handle of+            Nothing -> unavailable "cell has no matching lease"+            Just active -> do+              payloadResult <- loadPayloads payloadFiles+              rulesResult <- maybe (pure (Right (registryRoutingRules cli.registry))) readJsonDocument rulesFile+              case (payloadResult, rulesResult) of+                (Left problem, _) -> usage problem+                (_, Left failure) -> usage (Text.pack failure)+                (Right payloads, Right rules) -> do+                  capabilityResult <- loadCapabilitiesFor [observed.descriptor]+                  case capabilityResult of+                    Left problem -> usage problem+                    Right [cachedCapabilities] -> do+                      bytes <- if planFile == "-" then LazyByteString.getContents else LazyByteString.readFile planFile+                      selectedStore <- lookupEnv "KENSHOU_CELL_STORE"+                      let options = BuildOptions (PrepareOptions coerce ephemeral settings cache) granularity skip sink rts (24 * 1024 * 1024 * 1024) (20 * 1024 * 1024 * 1024) "0.1.0"+                          storeLabel = Text.pack (fromMaybe "gs://" selectedStore)+                      built <- buildSession cli.registry observed.descriptor cachedCapabilities rules payloads options storeLabel leaseId bytes+                      case built of+                        Left problem -> usage problem+                        Right session+                          | dryRun -> LazyByteString.putStrLn (encode session.journal) >> pure ExitSuccess+                          | detached && length session.journal.slices /= 1 -> usage "detached sessions require exactly one slice; use --granularity plan for a compatible plan"+                          | otherwise -> guardIO do+                              exists <- doesPathExist outDir+                              if exists+                                then usage "session output path already exists; choose a new directory"+                                else do+                                  if detached+                                    then do+                                      slice <- singleSlice session.journal+                                      let budget = toInteger slice.submission.limits.wallClockSeconds + 600+                                      if budget > toInteger (maxBound :: Int)+                                        then ioError (userError "detached lease budget exceeds supported TTL")+                                        else do+                                          lease <- leaseSnapshot active+                                          enlarged <- resizeLease store ref active (max lease.ttlSeconds (fromInteger budget))+                                          if enlarged then pure () else ioError (userError "cell lease changed before detached submission")+                                    else pure ()+                                  traverse_+                                    ( \(relative, content) -> do+                                        let destination = outDir </> relative+                                        createDirectoryIfMissing True (takeDirectory destination)+                                        LazyByteString.writeFile destination content+                                    )+                                    session.workFiles+                                  if detached+                                    then do+                                      slice <- singleSlice session.journal+                                      let journal = session.journal {leaseMode = Detached}+                                          journalPath = outDir </> "session.json"+                                      writeSessionJournal journalPath journal+                                      work <- LazyByteString.readFile (outDir </> slice.workPath)+                                      published <- publishSubmission store ref active slice.submission work+                                      case published of+                                        Submitted -> do+                                          now <- getCurrentTime+                                          checkpoint <- either (ioError . userError . Text.unpack) pure (applyTransition now slice.cellRun SubmissionPublished journal)+                                          writeSessionJournal journalPath checkpoint+                                          TextIO.hPutStrLn stderr ("detached session " <> renderRunId checkpoint.sessionId <> " submitted; collect with cell resume --session " <> Text.pack outDir)+                                          pure ExitSuccess+                                        other -> unavailable ("detached submission failed: " <> Text.pack (show other))+                                    else do+                                      result <- withHeartbeat store ref active \_ -> runPlannedSlices store ref (Bucket observed.descriptor.buckets.results) active (outDir </> "session.json") session.journal emitWatchEvent+                                      case result of+                                        Left failure -> unavailable (Text.pack (show failure))+                                        Right finished -> do+                                          TextIO.hPutStrLn stderr ("session " <> renderRunId finished.sessionId <> " verified; journal " <> Text.pack (outDir </> "session.json"))+                                          pure (sessionExitCode finished)+                    Right _ -> unavailable "cell capability cache count differs from descriptor count"+  Run (RunCli location settings ttl waitSeconds startInstances detached)+    | ttl <= 0 || waitSeconds < 0 -> usage "lease TTL must be positive and wait cannot be negative"+    | otherwise -> withControl location \store ref observed -> do+        if startInstances then startCellInstances observed.descriptor else pure ()+        defaultOwner <- Text.pack . fromMaybe "kenshou" <$> lookupEnv "USER"+        start <- getCurrentTime+        let request = LeaseRequest defaultOwner "kenshou cell run" ttl+            acquire = do+              outcome <- acquireLease store ref request+              case outcome of+                Acquired handle -> do+                  lease <- leaseSnapshot handle+                  let execute = runCell (Submit (SubmitCli location (renderRunId lease.leaseId) settings detached)) cli+                      releaseCurrent = do+                        current <- reattachLease store ref lease.leaseId+                        traverse_ (\active -> do _ <- releaseLease store ref active; pure ()) current+                      releaseUnlessPublished = do+                        let SubmitSettings _ _ outDir _ _ _ _ _ _ _ _ _ _ = settings+                            journalPath = outDir </> "session.json"+                        present <- doesPathExist journalPath+                        if not present+                          then releaseCurrent+                          else do+                            decoded <- readSessionJournal journalPath+                            case decoded of+                              Left _ -> pure ()+                              Right journal | journal.leaseId /= lease.leaseId || journal.cell /= ref.cellName -> releaseCurrent+                              Right journal -> do+                                markers <- traverse (\slice -> store.statObject ref.controlBucket (ObjectName ("cells/" <> ref.cellName <> "/submissions/" <> renderRunId slice.cellRun <> "/submission.json"))) journal.slices+                                if all isNothing markers then releaseCurrent else pure ()+                  execute `finally` (if detached then releaseUnlessPublished else releaseCurrent)+                Busy current -> do+                  now <- getCurrentTime+                  if diffUTCTime now start < fromIntegral waitSeconds+                    then threadDelay 2000000 >> acquire+                    else unavailable ("cell is busy under lease " <> renderRunId current.leaseId)+                Quarantined record -> unavailable ("cell is quarantined: " <> record.reason)+        acquire+  Probe request -> runProbe cli request+  Pair request -> runPair cli request+  Overhead request -> runCellOverhead cli request+  Resume sessionDir selectedLease startInstances -> guardIO do+    let journalPath = sessionDir </> "session.json"+    present <- doesPathExist journalPath+    if not present+      then unavailable "session.json is missing"+      else do+        decoded <- readSessionJournal journalPath+        case decoded of+          Left problem -> usage problem+          Right journal -> do+            selectedStore <- Text.pack . fromMaybe "gs://" <$> lookupEnv "KENSHOU_CELL_STORE"+            if selectedStore /= journal.store+              then usage "selected cell store differs from the session journal"+              else case traverse parseRunId selectedLease of+                Left problem -> usage problem+                Right requested -> withControl (CellLocation journal.cell (Just journal.controlBucket)) \store ref observed -> do+                  if observed.descriptor.buckets.results /= journal.resultsBucket+                    then usage "cell results bucket differs from the session journal"+                    else do+                      if startInstances then startCellInstances observed.descriptor else pure ()+                      let expected = fromMaybe journal.leaseId requested+                      existing <- reattachLease store ref expected+                      case existing of+                        Just _ | journal.leaseMode == Detached -> continueDetached store ref journalPath journal+                        Just handle -> continueSession store ref journalPath journal handle `finally` (do _ <- releaseLease store ref handle; pure ())+                        Nothing | requested /= Nothing -> unavailable "cell has no matching selected lease"+                        Nothing | journal.leaseMode == Detached -> continueDetached store ref journalPath journal+                        Nothing -> do+                          owner <- Text.pack . fromMaybe "kenshou" <$> lookupEnv "USER"+                          acquired <- acquireLease store ref (LeaseRequest owner ("resume " <> renderRunId journal.sessionId) 120)+                          case acquired of+                            Busy current -> unavailable ("cell is busy under lease " <> renderRunId current.leaseId)+                            Quarantined record -> unavailable ("cell is quarantined: " <> record.reason)+                            Acquired handle -> continueSession store ref journalPath journal handle `finally` (do _ <- releaseLease store ref handle; pure ())+  Exec workFile outDir -> cellExec cli.registry workFile outDir++continueSession :: ObjectStore -> CellRef -> FilePath -> SessionJournal -> LeaseHandle -> IO ExitCode+continueSession store ref journalPath journal handle = do+  resumed <- withHeartbeat store ref handle \_ -> resumeHeldSession store ref (Bucket journal.resultsBucket) handle journalPath emitWatchEvent+  case resumed of+    Left failure -> unavailable (Text.pack (show failure))+    Right finished -> do+      TextIO.hPutStrLn stderr ("session " <> renderRunId finished.sessionId <> " verified; journal " <> Text.pack journalPath)+      pure (sessionExitCode finished)++continueDetached :: ObjectStore -> CellRef -> FilePath -> SessionJournal -> IO ExitCode+continueDetached store ref journalPath journal = do+  observed <- resumeObservedSlices store ref (Bucket journal.resultsBucket) journalPath emitWatchEvent+  case observed of+    Left failure -> unavailable (Text.pack (show failure))+    Right finished ->+      if not (null finished.slices) && all (\slice -> slice.state `elem` [SliceRejected, SliceVerified]) finished.slices+        then do+          current <- reattachLease store ref journal.leaseId+          traverse_ (\active -> do _ <- releaseLease store ref active; pure ()) current+          TextIO.hPutStrLn stderr ("detached session " <> renderRunId finished.sessionId <> " collected; journal " <> Text.pack journalPath)+          pure (sessionExitCode finished)+        else unavailable "detached session has not reached a terminal cell status; resume later"++singleSlice :: SessionJournal -> IO SliceJournal+singleSlice journal = case journal.slices of+  [slice] -> pure slice+  _ -> ioError (userError "detached session must contain exactly one slice")++sessionExitCode :: SessionJournal -> ExitCode+sessionExitCode journal+  | any ((/= SliceVerified) . (.state)) journal.slices = ExitFailure 4+  | any ((/= Just Completed) . (.cellOutcome)) journal.slices = ExitFailure 4+  | any ((== Just 4) . (.entryExitCode)) journal.slices = ExitFailure 4+  | any ((== Just 3) . (.entryExitCode)) journal.slices = ExitFailure 3+  | any ((== Just 1) . (.entryExitCode)) journal.slices = ExitFailure 1+  | all ((== Just 0) . (.entryExitCode)) journal.slices = ExitSuccess+  | otherwise = ExitFailure 4++runProbe :: CliEnv -> ProbeCli -> IO ExitCode+runProbe cli (ProbeCli location payload output) = withControl location \_ _ observed -> do+  cacheDirectory <- fromMaybe ".dev/cells" <$> lookupEnv "KENSHOU_CELL_CAPABILITIES_DIR"+  identifier <- newRunId+  let sessionDirectory = cacheDirectory </> "probes" </> Text.unpack (renderRunId identifier)+      planFile = sessionDirectory <> ".plan.json"+      cacheFile = fromMaybe (cacheDirectory </> Text.unpack observed.descriptor.name <> ".capabilities.json") output+      settings = SubmitSettings [payload] planFile sessionDirectory GranularityRun Cold [] False False False Nothing NullSink Nothing False+      planArguments =+        [ "plan",+          "--all",+          "--select",+          "selftest/remote/correctness/cell-environment",+          "--placement",+          "cell",+          "--seed",+          "7",+          "--dim",+          "pg.durability=durable",+          "--dim",+          "pg.version=" <> show observed.descriptor.postgresMajor,+          "--out",+          planFile+        ]+  createDirectoryIfMissing True (takeDirectory planFile)+  executable <- getExecutablePath+  (planned, _, planErrors) <- readProcessWithExitCode executable planArguments ""+  if planned /= ExitSuccess+    then usage ("cell probe could not plan its environment scenario: " <> Text.pack planErrors)+    else do+      result <- runCell (Run (RunCli location settings 120 0 False False)) cli+      if result /= ExitSuccess+        then pure result+        else do+          saved <- readSessionJournal (sessionDirectory </> "session.json")+          case saved of+            Right journal -> case journal.slices of+              [slice] | slice.state == SliceVerified -> case slice.runIds of+                [nested] -> do+                  let runResult = takeDirectory sessionDirectory </> Text.unpack (renderRunId slice.cellRun) </> "tree" </> "output" </> Text.unpack (renderRunId nested) </> "run-result.json"+                  document <- eitherDecodeFileStrict' runResult+                  case document >>= extractProbeCapabilities slice.cellRun of+                    Left problem -> failVerification ("cell probe result has no valid capabilities: " <> Text.pack problem)+                    Right capabilities -> do+                      now <- getCurrentTime+                      let cache = CellCapabilities observed.descriptor.name (descriptorDigest observed.descriptor) slice.cellRun now capabilities+                          temporary = cacheFile <> "." <> Text.unpack (renderRunId identifier) <> ".tmp"+                      createDirectoryIfMissing True (takeDirectory cacheFile)+                      LazyByteString.writeFile temporary (encode cache)+                      renameFile temporary cacheFile+                      TextIO.putStrLn (Text.pack cacheFile)+                      pure ExitSuccess+                _ -> failVerification "cell probe verified a slice with an unexpected nested run count"+              _ -> failVerification "cell probe did not verify exactly one slice"+            Left problem -> failVerification ("cell probe session is invalid: " <> problem)++extractProbeCapabilities :: RunId -> Value -> Either String (Map.Map Text Value)+extractProbeCapabilities expectedCellRun (Object root) = do+  schema <- member "schema" root+  if schema == String "kenshou.run-result/v1" then pure () else Left "unexpected run-result schema"+  outcome <- member "outcome" root+  if outcome == String "passed" then pure () else Left "probe run did not pass"+  fingerprint <- member "fingerprint" root+  cell <- asObject "fingerprint.cell" =<< member "cell" =<< asObject "fingerprint" fingerprint+  cellRun <- member "cellRun" cell+  if cellRun == Aeson.toJSON expectedCellRun then pure () else Left "probe cell run identity differs from the verified slice"+  capabilities <- member "capabilities" cell+  case Aeson.fromJSON capabilities of+    Aeson.Error problem -> Left problem+    Aeson.Success values+      | all primitive (Map.elems values) && all (`Map.member` values) required -> Right values+      | otherwise -> Left "probe capabilities are missing or contain a non-primitive value"+  where+    primitive (Bool _) = True+    primitive (String _) = True+    primitive (Number _) = True+    primitive _ = False+    required = ["postgres.major", "postgres.pg_partman", "postgres.second-server", "postgres.control-hook", "broker", "otlp.null", "otlp.file", "fault-hook"]+    member :: Text -> KeyMap.KeyMap Value -> Either String Value+    member name fields = maybe (Left ("missing " <> Text.unpack name)) Right (KeyMap.lookup (Key.fromText name) fields)+    asObject :: String -> Value -> Either String (KeyMap.KeyMap Value)+    asObject _ (Object fields) = Right fields+    asObject label _ = Left (label <> " is not an object")+extractProbeCapabilities _ _ = Left "run result is not an object"++runPair :: CliEnv -> PairCli -> IO ExitCode+runPair cli (PairCli location candidateFile baselineFile planFile selectedPairs ordering policyFile settingTexts replacementOption startInstances outDir) = guardIO do+  settings <- case traverse parsePgSetting settingTexts of+    Left problem -> pure (Left problem)+    Right parsed -> pure (Right parsed)+  policyBytes <- ByteString.readFile policyFile+  sourceDocument <- eitherDecodeFileStrict' planFile :: IO (Either String Value)+  payloadResult <- loadPayloads ["baseline=" <> baselineFile, "candidate=" <> candidateFile]+  case (settings, decodePolicy policyBytes, sourceDocument >>= either (Left . Text.unpack) Right . decodeWorkPlan, payloadResult) of+    (Left problem, _, _, _) -> usage problem+    (_, Left problem, _, _) -> usage problem+    (_, _, Left problem, _) -> usage (Text.pack problem)+    (_, _, _, Left problem) -> usage problem+    (Right selectedPgSettings, Right policy, Right source, Right payloads) -> do+      let requested = fromMaybe policy.minimumPairs selectedPairs+          replacements = if replacementOption < 0 then requested else replacementOption+          request = PairRequest "candidate" "baseline" requested ordering (unSeed source.policy.seed) replacements+          pgSettings = selectedPgSettings <> [(name, assigned) | (name, assigned) <- [("checkpoint_timeout", "30min"), ("max_wal_size", "16GB")], name `notElem` fmap fst selectedPgSettings]+      case pairPlan request source of+        Left problem -> usage problem+        Right schedule -> do+          let groups = nub [trial.group | entry <- schedule.runs, Just trial <- [entry.trial]]+          if length groups /= 1+            then usage "cell pair currently requires one benchmark configuration per plan"+            else withControl location \store ref observed -> do+              exists <- doesPathExist outDir+              if exists+                then usage "pair output path already exists; choose a new directory"+                else do+                  if startInstances then startCellInstances observed.descriptor else pure ()+                  defaultOwner <- Text.pack . fromMaybe "kenshou" <$> lookupEnv "USER"+                  acquired <- acquireLease store ref (LeaseRequest defaultOwner "kenshou cell pair" 120)+                  case acquired of+                    Busy lease -> unavailable ("cell is busy under lease " <> renderRunId lease.leaseId)+                    Quarantined record -> unavailable ("cell is quarantined: " <> record.reason)+                    Acquired handle -> do+                      lease <- leaseSnapshot handle+                      let releaseCurrent = do+                            current <- reattachLease store ref lease.leaseId+                            traverse_ (\active -> do _ <- releaseLease store ref active; pure ()) current+                      pairUnderLease cli location lease.leaseId baselineFile candidateFile request schedule policy payloads pgSettings outDir `finally` releaseCurrent++runCellOverhead :: CliEnv -> OverheadCli -> IO ExitCode+runCellOverhead cli (OverheadCli location payloadFile sink settingTexts startInstances options) = guardIO do+  let requestedScenario =+        parseScenarioId options.scenario >>= \identifier ->+          maybe (Left ("unknown scenario " <> options.scenario)) Right (lookupScenario cli.registry identifier)+      input = LocalOverhead.parseInputs options+      settings = traverse parsePgSetting settingTexts+  selectedPolicy <- LocalOverhead.loadPolicy options.policy+  payloadResult <- loadPayloads [payloadFile]+  case (requestedScenario, input, settings, selectedPolicy, payloadResult) of+    (Left problem, _, _, _, _) -> usage problem+    (_, Left problem, _, _, _) -> usage problem+    (_, _, Left problem, _, _) -> usage problem+    (_, _, _, Left problem, _) -> usage problem+    (_, _, _, _, Left problem) -> usage problem+    (Right scenario, Right (factors, fixedKnobs, fixedDimensions), Right pgSettings, Right policy, Right payloads) ->+      case Map.lookup "default" payloads of+        Nothing -> usage "cell overhead requires one default payload"+        Just payload -> do+          resumed <- if options.resume || options.analyseOnly then LocalOverhead.findResumeDirectory options.out scenario.id else pure Nothing+          case (options.resume || options.analyseOnly, resumed) of+            (True, Nothing) -> usage "no matching overhead state was found under --out"+            (_, Just directory) -> do+              loaded <- loadOverheadState directory+              case loaded of+                Left problem -> usage ("invalid overhead state: " <> Text.pack problem)+                Right state -> continue directory state.plan (Just state) payload policy pgSettings+            (_, Nothing) -> do+              identifier <- renderRunId <$> newRunId+              let directory = options.out </> ("overhead-" <> Text.unpack identifier)+                  request = OverheadRequest identifier scenario.id factors options.mode options.control options.trials fixedKnobs fixedDimensions options.seed options.settleSeconds options.retries+              case planOverhead request scenario of+                Left (UsageError problem) -> usage problem+                Right plan -> continue directory plan Nothing payload policy pgSettings+  where+    continue directory plan existing payload policy pgSettings = do+      configuredBucket <- lookupEnv "KENSHOU_CELL_CONTROL_BUCKET"+      let CellLocation name selectedBucket = location+          bucketName = fromMaybe (maybe "tan-nb-exp-cells-control" Text.pack configuredBucket) selectedBucket+          configuration = object ["schema" .= ("kenshou.cell-overhead/v1" :: Text), "cell" .= name, "controlBucket" .= bucketName, "payload" .= payload, "otlpSink" .= show sink, "pgSettings" .= pgSettings]+          configPath = directory </> "cell-overhead.json"+      case existing of+        Nothing -> do+          present <- doesPathExist directory+          if present+            then usage "overhead output path already exists"+            else do+              createDirectoryIfMissing True directory+              LazyByteString.writeFile configPath (encode configuration)+              execute directory plan payload policy pgSettings+        Just state -> do+          saved <- eitherDecodeFileStrict' configPath :: IO (Either String Value)+          if saved /= Right configuration+            then usage "cell overhead resume arguments differ from the saved cell and payload configuration"+            else+              if options.analyseOnly+                then analyse directory plan state policy+                else execute directory plan payload policy pgSettings++    execute directory plan payload policy pgSettings = withControl location \store ref observed -> do+      if startInstances then startCellInstances observed.descriptor else pure ()+      owner <- Text.pack . fromMaybe "kenshou" <$> lookupEnv "USER"+      acquired <- acquireLease store ref (LeaseRequest owner "kenshou cell overhead" 120)+      case acquired of+        Busy current -> unavailable ("cell is busy under lease " <> renderRunId current.leaseId)+        Quarantined record -> unavailable ("cell is quarantined: " <> record.reason)+        Acquired handle -> do+          lease <- leaseSnapshot handle+          let releaseCurrent = do+                current <- reattachLease store ref lease.leaseId+                traverse_ (\active -> do _ <- releaseLease store ref active; pure ()) current+              hooks = overheadHooks plan (runCellOverheadChild cli location lease.leaseId payloadFile payload sink pgSettings directory)+          withHeartbeat+            store+            ref+            handle+            ( \_ -> do+                state <- executeOverhead hooks plan directory+                analyseOverheadResult options.json directory plan state policy hooks+            )+            `finally` releaseCurrent++    analyse directory plan state policy = analyseOverheadResult options.json directory plan state policy (overheadHooks plan (\_ _ -> pure (ExitFailure 4)))++overheadHooks :: OverheadPlan -> (RunSpec -> FilePath -> IO ExitCode) -> OverheadHooks+overheadHooks plan runChild =+  OverheadHooks runChild compareCellOverhead (Just leakCheck)+  where+    leakCheck directory = fmap (either (const Nothing) (Just . Aeson.toJSON)) (analyseRunDirectory directory defaultLeakSpec plan.seed)++-- On a cell, the resolver binds the chosen collector address only for an+-- sdk-otlp tracing arm. That address is a consequence of the declared tracing+-- change, so name its knob when the two arms actually have different values.+compareCellOverhead :: Policy -> NonEmpty VaryingAxis -> [FilePath] -> [FilePath] -> IO (Either CompareError Comparison)+compareCellOverhead policy axes baselines candidates = do+  derivedEndpoint <- case (baselines, candidates) of+    (baseline : _, candidate : _)+      | VaryDimension "telemetry.tracing" `elem` NonEmpty.toList axes -> do+          left <- readInputs baseline+          right <- readInputs candidate+          pure case (left, right) of+            (Just a, Just b) ->+              case (tracing a, tracing b, endpoint a, endpoint b) of+                (Just (String "sdk-otlp"), Just _, Just leftEndpoint, Just rightEndpoint) -> leftEndpoint /= rightEndpoint+                (Just _, Just (String "sdk-otlp"), Just leftEndpoint, Just rightEndpoint) -> leftEndpoint /= rightEndpoint+                _ -> False+            _ -> False+    _ -> pure False+  let selectedAxes = if derivedEndpoint then axes <> (VaryKnob "otel.endpoint" :| []) else axes+  compareRuns policy selectedAxes baselines candidates+  where+    tracing = varyingValue (VaryDimension "telemetry.tracing")+    endpoint = varyingValue (VaryKnob "otel.endpoint")+    readInputs directory = do+      decoded <- eitherDecodeFileStrict' (directory </> "run-result.json") :: IO (Either String Value)+      pure (either (const Nothing) compatibilityInputs decoded)++analyseOverheadResult :: Bool -> FilePath -> OverheadPlan -> OverheadState -> OverheadPolicy -> OverheadHooks -> IO ExitCode+analyseOverheadResult asJson directory plan state policy hooks = do+  report <- analyseOverhead hooks policy plan state directory+  if asJson+    then LazyByteString.putStrLn (encode report)+    else TextIO.putStrLn ("verdict " <> LocalOverhead.verdictText report.verdict <> "  " <> Text.pack directory)+  pure case overheadVerdictExitCode report.verdict of 0 -> ExitSuccess; code -> ExitFailure code++runCellOverheadChild :: CliEnv -> CellLocation -> RunId -> FilePath -> PayloadDescriptor -> OtlpSink -> [(Text, Text)] -> FilePath -> RunSpec -> FilePath -> IO ExitCode+runCellOverheadChild cli location leaseId payloadFile payload sink pgSettings directory rawSpec runsRoot = case rawSpec.runId of+  Nothing -> pure (ExitFailure 4)+  Just runId -> do+    planId <- newRunId+    now <- getCurrentTime+    selector <- either (ioError . userError . Text.unpack) pure (parseSelector "**")+    seed <- either (ioError . userError . Text.unpack) pure (mkSeed 1)+    let reason = Reason (Change (ComponentRef (ComponentId "cell-overhead") Nothing) Named "telemetry overhead slot") [] selector 0+        spec = rawSpec {environment = rawSpec.environment {Kenshou.Core.RunSpec.placement = RunOnCell}}+        context = PlanContext Nothing "cell-overhead" payload.cohort payload.cohortIdentity.identityPlanHash.unPlanHash (PlanInputs (object ["overhead" .= True])) [] []+        plan = RunPlan planId now context (defaultPlanPolicy seed) [PlannedRun 0 runId 10 (reason :| []) Nothing spec] [] 10+        planPath = directory </> "cell-sessions" </> Text.unpack (renderRunId runId) <> ".plan.json"+        sessionDir = directory </> "cell-sessions" </> Text.unpack (renderRunId runId)+        settings = SubmitSettings [payloadFile] planPath sessionDir GranularityRun Cold (fmap (\(name, assigned) -> name <> "=" <> assigned) pgSettings) False True False Nothing sink Nothing False+    createDirectoryIfMissing True (takeDirectory planPath)+    LazyByteString.writeFile planPath (encode plan)+    result <- runCell (Submit (SubmitCli location (renderRunId leaseId) settings False)) cli+    journal <- readSessionJournal (sessionDir </> "session.json")+    case journal of+      Right saved -> case saved.slices of+        [slice] | slice.state == SliceVerified && slice.runIds == [runId] -> do+          let target = takeDirectory sessionDir </> Text.unpack (renderRunId slice.cellRun) </> "tree" </> "output" </> Text.unpack (renderRunId runId)+              link = runsRoot </> Text.unpack (renderRunId runId)+          available <- doesDirectoryExist target+          if available+            then do+              createDirectoryIfMissing True runsRoot+              absolute <- makeAbsolute target+              createDirectoryLink absolute link+              pure result+            else pure (ExitFailure 4)+        _ -> pure (ExitFailure 4)+      Left _ -> pure (ExitFailure 4)++pairUnderLease :: CliEnv -> CellLocation -> RunId -> FilePath -> FilePath -> PairRequest -> RunPlan -> Policy -> Map.Map Text PayloadDescriptor -> [(Text, Text)] -> FilePath -> IO ExitCode+pairUnderLease cli location leaseId baselineFile candidateFile request schedule policy payloads pgSettings outDir = do+  createDirectoryIfMissing True outDir+  LazyByteString.writeFile (outDir </> "pair-plan.json") (encode schedule)+  let requested = request.pairs+      limit = requested + request.maxReplacements+      batchFor predicate = filter (\entry -> maybe False predicate entry.trial) schedule.runs+      initial = batchFor (\trial -> trial.index < requested)+      runRound :: Int -> [PlannedRun] -> IO (Either Text [PairObservation])+      runRound roundNumber entries = do+        let planPath = outDir </> ("round-" <> show roundNumber <> ".plan.json")+            sessionDir = outDir </> ("round-" <> show roundNumber)+            plan = schedule {runs = entries, estimateMinutes = sum (fmap (.estimateMinutes) entries)}+            settings = SubmitSettings ["baseline=" <> baselineFile, "candidate=" <> candidateFile] planPath sessionDir GranularityRun Cold (fmap (\(name, assigned) -> name <> "=" <> assigned) pgSettings) False False False Nothing NullSink Nothing False+        LazyByteString.writeFile planPath (encode plan)+        result <- runCell (Submit (SubmitCli location (renderRunId leaseId) settings False)) cli+        journal <- readSessionJournal (sessionDir </> "session.json")+        case journal of+          Left problem -> pure (Left ("pair submission did not produce a readable journal (" <> Text.pack (show result) <> "): " <> problem))+          Right verified -> case find ((== SliceRejected) . (.state)) verified.slices of+            Just rejected -> pure (Left ("pair submission rejected: " <> fromMaybe "unknown reason" rejected.rejectionReason))+            Nothing -> case find ((/= SliceVerified) . (.state)) verified.slices of+              Just incomplete -> pure (Left ("pair submission is incomplete at " <> renderRunId incomplete.cellRun <> "; resume " <> Text.pack sessionDir))+              Nothing -> pure (collectPairObservations outDir entries verified)+      continue roundNumber nextIndex observed = do+        let valid = validPairIndexes observed+        if length valid >= request.pairs || nextIndex >= limit+          then finishPair policy payloads outDir request observed valid+          else do+            let entries = batchFor (\trial -> trial.index == nextIndex)+            next <- runRound roundNumber entries+            case next of+              Left problem -> unavailable problem+              Right results -> continue (roundNumber + 1) (nextIndex + 1) (observed <> results)+  initialResult <- runRound 0 initial+  case initialResult of+    Left problem -> unavailable problem+    Right observed -> continue 1 requested observed++collectPairObservations :: FilePath -> [PlannedRun] -> SessionJournal -> Either Text [PairObservation]+collectPairObservations outDir entries journal = traverse collect entries+  where+    collect entry = case find (elem entry.runId . (.runIds)) journal.slices of+      Nothing -> Left ("pair journal omits planned run " <> renderRunId entry.runId)+      Just slice ->+        let path = outDir </> Text.unpack (renderRunId slice.cellRun) </> "tree" </> "output" </> Text.unpack (renderRunId entry.runId)+            passed = slice.state == SliceVerified && slice.cellOutcome == Just Completed && slice.entryExitCode == Just 0+         in Right (PairObservation entry path passed)++validPairIndexes :: [PairObservation] -> [Int]+validPairIndexes observations = sort [index | ((_, index), PairValid) <- judgePairs (fmap asSlice observations)]+  where+    asSlice (PairObservation entry _ passed) = case entry.trial of+      Just trial -> SliceResult trial.group trial.index (if trial.arm == "baseline" then Baseline else Candidate) passed+      Nothing -> error "paired plan lost trial membership"++finishPair :: Policy -> Map.Map Text PayloadDescriptor -> FilePath -> PairRequest -> [PairObservation] -> [Int] -> IO ExitCode+finishPair policy payloads outDir request observations valid =+  if null valid+    then unavailable "all paired trials failed; no comparison can be computed"+    else do+      let chosen = take request.pairs valid+          pathFor index arm = case find (matches index arm) observations of+            Just (PairObservation _ path _) -> path+            Nothing -> error "valid pair has no run path"+          matches index arm (PairObservation entry _ _) = case entry.trial of+            Just trial -> trial.index == index && trial.arm == arm+            Nothing -> False+          baselines = fmap (`pathFor` "baseline") chosen+          candidates = fmap (`pathFor` "candidate") chosen+          varying = case (Map.lookup "baseline" payloads, Map.lookup "candidate" payloads) of+            (Just baseline, Just candidate) | baseline.cohortIdentity.identityPlanHash /= candidate.cohortIdentity.identityPlanHash -> VaryCohort+            _ -> VaryControl+      compared <- compareRuns policy (varying :| []) baselines candidates+      case compared of+        Left (CompareError problem) -> usage ("cell pair comparison failed: " <> problem)+        Right comparison -> do+          LazyByteString.writeFile (outDir </> "comparison.json") (encode comparison <> "\n")+          TextIO.putStrLn ("comparison " <> Text.pack (show comparison.verdict) <> "; valid pairs " <> Text.pack (show (length chosen)) <> "/" <> Text.pack (show request.pairs))+          pure case verdictExitCode comparison.verdict of 0 -> ExitSuccess; code -> ExitFailure code++parsePgSetting :: Text -> Either Text (Text, Text)+parsePgSetting setting = case Text.breakOn "=" setting of+  (name, assignment) | not (Text.null name) && Text.length assignment > 1 -> Right (name, Text.drop 1 assignment)+  _ -> Left "--pg-setting must be KEY=VALUE"++loadPayloads :: [String] -> IO (Either Text (Map.Map Text PayloadDescriptor))+loadPayloads paths = do+  loaded <- traverse load paths+  pure do+    entries <- sequence loaded+    let labels = fmap fst entries+    if length (nub labels) == length labels+      then Right (Map.fromList entries)+      else Left "payload labels must be unique"+  where+    load input = case break (== '=') input of+      (label, '=' : path) | not (null label) && not (null path) && path /= "-" -> readPayload (Text.pack label) path+      (_, '=' : _) -> pure (Left "--payload must be [LABEL=]FILE with a nonempty file path")+      (_, _) | input == "-" -> pure (Left "payload must name a file")+      _ -> readPayload "default" input+    readPayload label path = do+      result <- eitherDecodeFileStrict' path+      pure (either (Left . Text.pack) (\descriptor -> Right (label, descriptor)) result)++loadCapabilitiesFor :: [CellDescriptor] -> IO (Either Text [Maybe CellCapabilities])+loadCapabilitiesFor descriptors = do+  directory <- fromMaybe ".dev/cells" <$> lookupEnv "KENSHOU_CELL_CAPABILITIES_DIR"+  sequence <$> traverse (loadCellCapabilities directory) descriptors++readJsonDocument :: (FromJSON value) => FilePath -> IO (Either String value)+readJsonDocument "-" = eitherDecode <$> LazyByteString.getContents+readJsonDocument path = eitherDecodeFileStrict' path++withControl :: CellLocation -> (ObjectStore -> CellRef -> CellSnapshot -> IO ExitCode) -> IO ExitCode+withControl (CellLocation name selectedBucket) operation = do+  configured <- lookupEnv "KENSHOU_CELL_CONTROL_BUCKET"+  let bucketName = fromMaybe (maybe "tan-nb-exp-cells-control" Text.pack configured) selectedBucket+  case validateBucket bucketName of+    Left problem -> usage problem+    Right bucket+      | not (validCellName name) -> usage "invalid cell name"+      | otherwise -> guardIO do+          store <- openStore bucket+          allowed <- maybe ["tan-nb-exp"] (map Text.strip . Text.splitOn "," . Text.pack) <$> lookupEnv "KENSHOU_GCP_ALLOWED_PROJECTS"+          let ref = CellRef name bucket+          snapshot <- readCellSnapshot store ref allowed+          case snapshot of+            Left problem -> unavailable problem+            Right observed -> operation store ref observed++startCellInstances :: CellDescriptor -> IO ()+startCellInstances descriptor = do+  let names = descriptor.instances.postgres : descriptor.instances.drivers <> [descriptor.instances.monitoring] <> maybe [] pure descriptor.instances.broker+      location = ["--project=" <> Text.unpack descriptor.project, "--zone=" <> Text.unpack descriptor.zone]+  states <-+    traverse+      ( \name -> do+          (code, output, failure) <- readProcessWithExitCode "gcloud" (["compute", "instances", "describe", Text.unpack name] <> location <> ["--format=value(status)"]) ""+          if code == ExitSuccess+            then pure (name, Text.strip (Text.pack output))+            else ioError (userError ("cannot inspect cell instance " <> Text.unpack name <> ": " <> failure))+      )+      names+  case [(name, state) | (name, state) <- states, state `notElem` ["RUNNING", "TERMINATED"]] of+    (name, state) : _ -> ioError (userError ("cell instance " <> Text.unpack name <> " is " <> Text.unpack state))+    [] -> do+      let stopped = [Text.unpack name | (name, "TERMINATED") <- states]+      if null stopped+        then pure ()+        else do+          (code, _, failure) <- readProcessWithExitCode "gcloud" (["compute", "instances", "start"] <> stopped <> location <> ["--quiet"]) ""+          if code == ExitSuccess+            then TextIO.hPutStrLn stderr ("started " <> Text.pack (show (length stopped)) <> " instances for cell " <> descriptor.name)+            else ioError (userError ("could not start cell " <> Text.unpack descriptor.name <> ": " <> failure))++holdLease :: IO Bool -> IO ExitCode+holdLease stillHeld = do+  held <- stillHeld+  if held then threadDelay 1000000 >> holdLease stillHeld else unavailable "lease was lost"++emitWatchEvent :: WatchEvent -> IO ()+emitWatchEvent event = case event of+  PhaseChanged phase -> TextIO.hPutStrLn stderr ("phase: " <> Text.pack (show phase))+  StdoutChunk bytes -> LazyByteString.hPut stdout bytes >> hFlush stdout+  StderrChunk bytes -> LazyByteString.hPut stderr bytes >> hFlush stderr++verifyTree :: FilePath -> IO ExitCode+verifyTree tree = do+  verified <- verifyCellRun tree+  case verified of+    Left problems -> failVerification (Text.pack (show (NonEmpty.toList problems)))+    Right manifest -> TextIO.putStrLn ("verified " <> renderRunId manifest.runId) >> pure ExitSuccess++openStore :: Bucket -> IO ObjectStore+openStore bucket = do+  selected <- lookupEnv "KENSHOU_CELL_STORE"+  case selected of+    Nothing -> newTokenProvider bucket >>= newGcsStore+    Just setting -> case Text.stripPrefix "file:" (Text.pack setting) of+      Just root | not (Text.null root) -> newFileStore (Text.unpack root)+      _ -> ioError (userError "KENSHOU_CELL_STORE must be file:<dir> or unset for GCS")++parseResultsUri :: Text -> Either Text (Bucket, RunId)+parseResultsUri suffix = case Text.splitOn "/" suffix of+  [bucket, "runs", identifier] -> do+    checked <- validateBucket bucket+    parsed <- parseRunId identifier+    pure (checked, parsed)+  _ -> Left "expected gs://BUCKET/runs/CELL_RUN_ID"++parsePayloadUri :: Text -> Either Text (Bucket, ObjectName)+parsePayloadUri uri = case Text.stripPrefix "gs://" uri of+  Nothing -> Left "payload bundle URI must use gs://"+  Just suffix -> case Text.breakOn "/" suffix of+    (bucketName, path) | not (Text.null path) -> do+      bucket <- validateBucket bucketName+      pure (bucket, ObjectName (Text.drop 1 path))+    _ -> Left "payload bundle URI must name an object"++validateBucket :: Text -> Either Text Bucket+validateBucket bucket+  | Text.null bucket || Text.any (\character -> not (character `elem` (['a' .. 'z'] <> ['A' .. 'Z'] <> ['0' .. '9'] <> "-_."))) bucket = Left "invalid results bucket"+  | otherwise = Right (Bucket bucket)++guardIO :: IO ExitCode -> IO ExitCode+guardIO operation = do+  result <- try operation :: IO (Either SomeException ExitCode)+  case result of+    Left failure | Just async <- (fromException failure :: Maybe SomeAsyncException) -> throwIO async+    Left failure -> unavailable (Text.pack (displayException failure))+    Right code -> pure code++fetchFailure :: FetchError -> IO ExitCode+fetchFailure problem = case problem of+  Unsealed -> unavailable "cell run is not sealed"+  ObjectChanged _ -> unavailable (Text.pack (show problem))+  _ -> failVerification (Text.pack (show problem))++unavailable :: Text -> IO ExitCode+unavailable problem = TextIO.hPutStrLn stderr ("kenshou cell: " <> problem) >> pure (ExitFailure 4)++failVerification :: Text -> IO ExitCode+failVerification problem = TextIO.hPutStrLn stderr ("kenshou cell: " <> problem) >> pure (ExitFailure 1)++usage :: Text -> IO ExitCode+usage problem = TextIO.hPutStrLn stderr ("kenshou cell: " <> problem) >> pure (ExitFailure 2)
+ src/Kenshou/Cli/Command/Cohort.hs view
@@ -0,0 +1,15 @@+module Kenshou.Cli.Command.Cohort (cohortCommand) where++import Kenshou.Cli.Cohort (runCohortCommand)+import Kenshou.Cli.Options (cohortCommandParser)+import Kenshou.Core.Cli (CliCommand (..), CliGroup (..))++cohortCommand :: CliCommand+cohortCommand =+  CliCommand+    { name = "cohort",+      description = "Inspect the pinned runtime cohort",+      group = Maintenance,+      hidden = False,+      parser = (\command _ -> runCohortCommand command) <$> cohortCommandParser+    }
+ src/Kenshou/Cli/Command/Execute.hs view
@@ -0,0 +1,64 @@+{-# LANGUAGE FieldSelectors #-}++module Kenshou.Cli.Command.Execute (executeCommand) where++import Control.Exception (IOException, displayException, try)+import Data.ByteString qualified as ByteString+import Data.Text qualified as Text+import Data.Text.IO qualified as Text.IO+import Kenshou.Core.Cli+import Kenshou.Plan.Execute+import Kenshou.Plan.Summary+import Options.Applicative+import System.Exit (ExitCode (..))+import System.FilePath ((</>))+import System.IO (stderr)+import System.IO.Temp (withSystemTempDirectory)++data ExecuteCliOptions = ExecuteCliOptions+  { planSource :: InputSource,+    outDir :: FilePath,+    resume :: Bool,+    failFast :: Bool,+    environmentSource :: Maybe InputSource,+    timeoutFactor :: Maybe Double+  }++executeCommand :: CliCommand+executeCommand = CliCommand "execute" "Execute or resume a run plan" Execution False (runExecute <$> executeParser)++executeParser :: Parser ExecuteCliOptions+executeParser =+  ExecuteCliOptions+    <$> option (parseInputSource <$> str) (long "plan" <> metavar "FILE" <> help "Read a kenshou.run-plan/v1 document; use - for stdin")+    <*> strOption (long "out" <> metavar "DIR" <> help "Execution output directory")+    <*> switch (long "resume" <> help "Resume the identical plan in an existing output directory")+    <*> switch (long "fail-fast" <> help "Stop after the first blocking failure")+    <*> optional (option (parseInputSource <$> str) (long "environment" <> metavar "FILE" <> help "Replace each run's environment object; use - for stdin"))+    <*> optional (option auto (long "timeout-factor" <> metavar "F" <> help "Limit runs to F times their estimate"))++runExecute :: ExecuteCliOptions -> CliEnv -> IO ExitCode+runExecute options _+  | stdinCount options > 1 = Text.IO.hPutStrLn stderr "kenshou: at most one document input may use standard input" >> pure (ExitFailure 2)+  | maybe False (<= 0) options.timeoutFactor = Text.IO.hPutStrLn stderr "kenshou: --timeout-factor must be positive" >> pure (ExitFailure 2)+  | otherwise = withSystemTempDirectory "kenshou-execute" \temporary -> do+      planFile <- materialize temporary "plan.json" options.planSource+      environment <- traverse (materialize temporary "environment.json") options.environmentSource+      let executeOptions = ExecuteOptions planFile options.outDir options.resume options.failFast environment options.timeoutFactor+      result <- try (executePlan executeOptions) :: IO (Either IOException PlanSummary)+      case result of+        Left err -> Text.IO.hPutStrLn stderr ("kenshou: " <> Text.pack (displayException err)) >> pure (ExitFailure 2)+        Right summary -> do+          Text.IO.putStrLn ("exit=" <> Text.pack (show summary.exitCode) <> " worst=" <> Text.pack (show summary.worst))+          pure (summaryExitCode summary)++materialize :: FilePath -> FilePath -> InputSource -> IO FilePath+materialize _ _ (InputFile path) = pure path+materialize directory name InputStdin = do+  bytes <- readInputSource InputStdin+  let path = directory </> name+  ByteString.writeFile path bytes+  pure path++stdinCount :: ExecuteCliOptions -> Int+stdinCount options = length [() | InputStdin <- options.planSource : maybe [] pure options.environmentSource]
+ src/Kenshou/Cli/Command/List.hs view
@@ -0,0 +1,87 @@+module Kenshou.Cli.Command.List (listCommand) where++import Data.Aeson qualified as Aeson+import Data.Bifunctor (first)+import Data.ByteString.Lazy.Char8 qualified as LazyByteString+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Text.IO qualified as Text.IO+import Kenshou.Core.Bundle+import Kenshou.Core.Cli (CliCommand (..), CliEnv (..), CliGroup (..))+import Kenshou.Core.Id+import Kenshou.Core.Role (WorkerRole (..), renderRoleName)+import Kenshou.Core.Scenario+import Kenshou.Core.Selector (parseSelector)+import Options.Applicative+import System.Exit (ExitCode (..))+import System.IO (stderr)++data ListOptions = ListOptions+  { selectors :: [Text],+    layers :: [LayerId],+    kinds :: [Kind],+    maxTier :: Maybe Tier,+    placements :: [Placement],+    roles :: Bool,+    json :: Bool+  }++listCommand :: CliCommand+listCommand =+  CliCommand+    { name = "list",+      description = "List registered verification scenarios",+      group = Discovery,+      hidden = False,+      parser = runList <$> parserOptionGroup "Selection" listParser+    }++listParser :: Parser ListOptions+listParser =+  ListOptions+    <$> many (Text.pack <$> strArgument (metavar "SELECTOR" <> help "Scenario selector"))+    <*> many (option (eitherReader readLayer) (long "layer" <> metavar "LAYER" <> help "Restrict to a layer"))+    <*> many (option (eitherReader readKind) (long "kind" <> metavar "KIND" <> help "Restrict to an evidence kind"))+    <*> optional (option (eitherReader readTier) (long "max-tier" <> metavar "TIER" <> help "Maximum cost tier"))+    <*> many (option (eitherReader readPlacement) (long "placement" <> metavar "PLACEMENT" <> help "Restrict placement"))+    <*> switch (long "roles" <> help "List worker roles instead of scenarios")+    <*> switch (long "json" <> help "Write kenshou.scenario-list/v1 JSON")++runList :: ListOptions -> CliEnv -> IO ExitCode+runList options environment = case traverse parseSelector options.selectors of+  Left err -> Text.IO.hPutStrLn stderr ("kenshou: " <> err) >> pure (ExitFailure 2)+  Right selectors -> do+    let filterValue = ListFilter selectors options.layers options.kinds options.maxTier options.placements+        selected = selectScenarios environment.registry filterValue+    if options.json+      then LazyByteString.putStrLn (Aeson.encode (scenarioListDocumentFor selected (if options.roles then allRoles environment.registry else [])))+      else+        if options.roles+          then mapM_ (Text.IO.putStrLn . (\role -> renderRoleName role.name <> "  " <> role.summary)) (allRoles environment.registry)+          else mapM_ (Text.IO.putStrLn . renderScenario) selected+    pure ExitSuccess++renderScenario :: Scenario -> Text+renderScenario scenario =+  Text.intercalate "  " [renderScenarioId scenario.id, renderTier scenario.tier, renderPlacement scenario.placement, scenario.summary <> known]+  where+    known = maybe "" (const " [known defect]") scenario.knownDefect++readLayer :: String -> Either String LayerId+readLayer = first Text.unpack . parseLayer . Text.pack++readKind :: String -> Either String Kind+readKind = first Text.unpack . parseKind . Text.pack++readTier :: String -> Either String Tier+readTier "smoke" = Right TierSmoke+readTier "standard" = Right TierStandard+readTier "extended" = Right TierExtended+readTier "soak" = Right TierSoak+readTier value = Left ("unknown tier " <> show value)++readPlacement :: String -> Either String Placement+readPlacement "local" = Right PlaceLocal+readPlacement "cell" = Right PlaceCell+readPlacement "either" = Right PlaceEither+readPlacement value = Left ("unknown placement " <> show value)
+ src/Kenshou/Cli/Command/Overhead.hs view
@@ -0,0 +1,204 @@+module Kenshou.Cli.Command.Overhead+  ( OverheadOptions (..),+    overheadCommand,+    overheadParser,+    parseInputs,+    loadPolicy,+    findResumeDirectory,+    verdictText,+  )+where++import Control.Exception (IOException, try)+import Data.Aeson qualified as Aeson+import Data.ByteString.Lazy.Char8 qualified as LazyByteString+import Data.List (isPrefixOf, sort)+import Data.Map.Strict (Map)+import Data.Map.Strict qualified as Map+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Text.IO qualified as Text.IO+import Data.Word (Word64)+import Kenshou.Core.Bundle (lookupScenario)+import Kenshou.Core.Cli+import Kenshou.Core.Id (ScenarioId, newRunId, parseScenarioId, renderRunId)+import Kenshou.Core.Knob (KnobError (..), KnobName, RawKnob, parseAssignment)+import Kenshou.Core.RunSpec (RunSpec (..))+import Kenshou.Diagnose.Leak (analyseRunDirectory, defaultLeakSpec)+import Kenshou.Measure.Compare (Verdict (..), compareRuns)+import Kenshou.Telemetry.Overhead+import Kenshou.Telemetry.Overhead.Policy+import Options.Applicative+import System.Directory (createDirectoryIfMissing, doesFileExist, listDirectory)+import System.Environment (getExecutablePath)+import System.Exit (ExitCode (..))+import System.FilePath (takeFileName, (</>))+import System.IO (IOMode (..), stderr, withFile)+import System.Process (StdStream (..), createProcess, proc, std_err, std_out, waitForProcess)++data OverheadOptions = OverheadOptions+  { scenario :: Text,+    arms :: [Text],+    mode :: OverheadMode,+    control :: Bool,+    trials :: Int,+    knobs :: [Text],+    dimensions :: [Text],+    policy :: Maybe InputSource,+    seed :: Word64,+    settleSeconds :: Int,+    retries :: Int,+    resume :: Bool,+    analyseOnly :: Bool,+    json :: Bool,+    out :: FilePath+  }++overheadCommand :: CliCommand+overheadCommand = CliCommand "overhead" "Run interleaved telemetry overhead arms" Execution False (overheadHandler <$> overheadParser)++overheadParser :: Parser OverheadOptions+overheadParser =+  OverheadOptions+    <$> (Text.pack <$> strArgument (metavar "SCENARIO" <> help "Registered benchmark scenario"))+    <*> some (Text.pack <$> strOption (long "arms" <> metavar "FACTOR=VALUE,..." <> help "Telemetry factor and ordered arm values"))+    <*> option (eitherReader parseMode) (long "mode" <> metavar "one-factor|full" <> value OneFactor <> showDefaultWith (const "one-factor") <> help "Arm expansion mode")+    <*> switch (long "control" <> help "Add an identical baseline A/A arm")+    <*> option auto (long "trials" <> metavar "N" <> value 3 <> showDefault <> help "Valid paired blocks")+    <*> many (Text.pack <$> strOption (long "set" <> metavar "NAME=VALUE" <> help "Set a fixed scenario knob"))+    <*> many (Text.pack <$> strOption (long "dim" <> metavar "NAME=VALUE" <> help "Set a fixed dimension"))+    <*> optional (option (parseInputSource <$> str) (long "policy" <> metavar "FILE" <> help "Read kenshou.overhead-policy/v1; use - for stdin"))+    <*> option auto (long "seed" <> metavar "N" <> value 7 <> showDefault <> help "Replayable ordering seed")+    <*> option auto (long "settle-seconds" <> metavar "S" <> value 2 <> showDefault <> help "Pause between child runs")+    <*> option auto (long "retries" <> metavar "R" <> value 1 <> showDefault <> help "Retries for infrastructure failures")+    <*> switch (long "resume" <> help "Resume the latest matching overhead directory")+    <*> switch (long "analyse-only" <> help "Rebuild a report without running child slots")+    <*> switch (long "json" <> help "Print only the overhead report JSON")+    <*> strOption (long "out" <> metavar "DIR" <> help "Output root or overhead invocation directory")++overheadHandler :: OverheadOptions -> CliEnv -> IO ExitCode+overheadHandler options environment = case parseScenarioId options.scenario of+  Left err -> usage err+  Right scenarioId -> case lookupScenario environment.registry scenarioId of+    Nothing -> usage ("unknown scenario " <> options.scenario)+    Just scenario -> do+      parsed <- pure (parseInputs options)+      case parsed of+        Left err -> usage err+        Right (factors, fixedKnobs, fixedDimensions) -> do+          selectedPolicy <- loadPolicy options.policy+          case selectedPolicy of+            Left err -> usage err+            Right policy -> do+              resumed <- if options.resume || options.analyseOnly then findResumeDirectory options.out scenarioId else pure Nothing+              case (options.resume || options.analyseOnly, resumed) of+                (True, Nothing) -> usage "no matching overhead state was found under --out"+                (_, Just directory) -> continue policy directory+                (_, Nothing) -> do+                  identifier <- renderRunId <$> newRunId+                  let directory = options.out </> ("overhead-" <> Text.unpack identifier)+                      request = OverheadRequest identifier scenarioId factors options.mode options.control options.trials fixedKnobs fixedDimensions options.seed options.settleSeconds options.retries+                  case planOverhead request scenario of+                    Left (UsageError err) -> usage err+                    Right plan -> do+                      createDirectoryIfMissing True directory+                      runPlan policy directory plan Nothing+  where+    continue policy directory = do+      loaded <- loadOverheadState directory+      case loaded of+        Left err -> usage ("invalid overhead state: " <> Text.pack err)+        Right state -> runPlan policy directory state.plan (Just state)++    runPlan policy directory plan existing = do+      executable <- getExecutablePath+      let leakCheck runDirectory = fmap (either (const Nothing) (Just . Aeson.toJSON)) (analyseRunDirectory runDirectory defaultLeakSpec plan.seed)+          hooks = OverheadHooks (runChildProcess executable directory) compareRuns (Just leakCheck)+      state <- if options.analyseOnly then maybe (fail "missing overhead state") pure existing else executeOverhead hooks plan directory+      report <- analyseOverhead hooks policy plan state directory+      if options.json+        then LazyByteString.putStrLn (Aeson.encode report)+        else do+          mapM_ printComparison report.comparisons+          Text.IO.putStrLn ("verdict " <> verdictText report.verdict <> "  " <> Text.pack directory)+      pure case overheadVerdictExitCode report.verdict of 0 -> ExitSuccess; code -> ExitFailure code++    printComparison comparison =+      Text.IO.putStrLn (comparison.candidate <> " vs " <> comparison.baseline <> "  " <> verdictText comparison.verdict)++usage :: Text -> IO ExitCode+usage message = Text.IO.hPutStrLn stderr ("kenshou: " <> message) >> pure (ExitFailure 2)++parseInputs :: OverheadOptions -> Either Text (Map Text [Text], [(KnobName, RawKnob)], Map Text Text)+parseInputs options = do+  factorPairs <- traverse parseArms options.arms+  let factorNames = fmap fst factorPairs+  if length factorNames /= Map.size (Map.fromList [(name, ()) | name <- factorNames])+    then Left "each --arms factor may be specified only once"+    else pure ()+  fixedKnobs <- traverse parseKnob options.knobs+  fixedDimensions <- Map.fromList <$> traverse parseAssignmentText options.dimensions+  pure (Map.fromList factorPairs, fixedKnobs, fixedDimensions)+  where+    parseKnob input = case parseAssignment input of Left (KnobError err) -> Left err; Right parsed -> Right parsed++parseArms :: Text -> Either Text (Text, [Text])+parseArms input = do+  (rawName, rawValues) <- parseAssignmentText input+  name <- case rawName of+    "tracing" -> Right "telemetry.tracing"+    "metrics" -> Right "telemetry.metrics"+    "telemetry.tracing" -> Right rawName+    "telemetry.metrics" -> Right rawName+    _ -> Left ("unknown overhead factor " <> rawName)+  let values = Text.splitOn "," rawValues+  if any Text.null values then Left (name <> " contains an empty arm value") else Right (name, values)++parseAssignmentText :: Text -> Either Text (Text, Text)+parseAssignmentText input = case Text.breakOn "=" input of+  (name, assignment) | not (Text.null name) && not (Text.null assignment) -> Right (name, Text.drop 1 assignment)+  _ -> Left ("expected NAME=VALUE, got " <> input)++parseMode :: String -> Either String OverheadMode+parseMode "one-factor" = Right OneFactor+parseMode "full" = Right FullFactorial+parseMode input = Left ("unknown mode " <> input)++loadPolicy :: Maybe InputSource -> IO (Either Text OverheadPolicy)+loadPolicy source = do+  bytes <- readInputSource (maybe (InputFile "policies/telemetry-overhead.json") id source)+  pure (either (Left . Text.pack) Right (decodeOverheadPolicy bytes))++findResumeDirectory :: FilePath -> ScenarioId -> IO (Maybe FilePath)+findResumeDirectory root scenarioId = do+  direct <- doesFileExist (root </> "state.json")+  if direct+    then matching root+    else do+      namesResult <- try (listDirectory root) :: IO (Either IOException [FilePath])+      let candidates = case namesResult of Left _ -> []; Right names -> reverse (sort [root </> name | name <- names, "overhead-" `isPrefixOf` takeFileName name])+      firstMatching candidates+  where+    matching directory = do+      loaded <- loadOverheadState directory+      pure case loaded of Right state | state.plan.scenarioId == scenarioId -> Just directory; _ -> Nothing+    firstMatching [] = pure Nothing+    firstMatching (directory : rest) = matching directory >>= maybe (firstMatching rest) (pure . Just)++runChildProcess :: FilePath -> FilePath -> RunSpec -> FilePath -> IO ExitCode+runChildProcess executable invocation spec runsRoot = do+  createDirectoryIfMissing True (invocation </> ".specs")+  createDirectoryIfMissing True (invocation </> "logs")+  let identifier = maybe "unassigned" (Text.unpack . renderRunId) spec.runId+      specPath = invocation </> ".specs" </> identifier <> ".json"+      logPath = invocation </> "logs" </> identifier <> ".log"+  Aeson.encodeFile specPath spec+  withFile logPath WriteMode \handle -> do+    (_, _, _, process) <- createProcess (proc executable ["run", "--spec", specPath, "--out", runsRoot]) {std_out = UseHandle handle, std_err = UseHandle handle}+    waitForProcess process++verdictText :: Verdict -> Text+verdictText VerdictPass = "pass"+verdictText VerdictRegression = "regression"+verdictText VerdictInconclusive = "inconclusive"+verdictText VerdictInfrastructureFailure = "infrastructure-failure"
+ src/Kenshou/Cli/Command/Perf.hs view
@@ -0,0 +1,484 @@+{-# LANGUAGE FieldSelectors #-}++module Kenshou.Cli.Command.Perf (perfCommand, validateProfilePolicy) where++import Control.Exception (Exception, IOException, catch, displayException, throwIO)+import Control.Monad (foldM, unless, when)+import Data.Aeson hiding (decode)+import Data.Aeson.Key qualified as Key+import Data.Aeson.Types (parseEither)+import Data.Aeson.Types qualified as Aeson+import Data.ByteString qualified as ByteString+import Data.ByteString.Lazy.Char8 qualified as LazyByteString+import Data.List (nub, sort)+import Data.Map.Strict qualified as Map+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Text.Encoding qualified as Text.Encoding+import Data.Text.IO qualified as Text.IO+import Data.Time.Clock (diffUTCTime, getCurrentTime)+import Kenshou.Cli.Command.Perf.Session (sessionCommands, verifyCalibrationAdmission)+import Kenshou.Core.Bundle (allScenarios)+import Kenshou.Core.Canonical (canonicalEncode, sha256Hex)+import Kenshou.Core.Cli+import Kenshou.Core.Cohort qualified as Cohort+import Kenshou.Measure.Compare.Policy qualified as Policy+import Kenshou.Measure.Perf (validateProfilePolicy)+import Kenshou.Perf.Artifacts (sealPerfDirectory, verifyPerfManifest)+import Kenshou.Perf.Calibration (toolchainDigest)+import Kenshou.Perf.Contracts (PerfContracts (..))+import Kenshou.Perf.Plan+import Kenshou.Perf.Prepared qualified as Prepared+import Kenshou.Perf.Request+import Kenshou.Perf.Source+import Kenshou.Plan.Catalog (fromScenario)+import Kenshou.Plan.Change.Git qualified as Git+import Kenshou.Plan.Components qualified as Graph+import Kenshou.Plan.Suite (Suite (..), readSuite)+import Kenshou.Remote.Payload qualified as Payload+import Kenshou.Remote.Payload.Publish (digestFile)+import Kenshou.Remote.Payload.Publisher qualified as Publisher+import Kenshou.Remote.Store (Bucket (..), ObjectName (..), ObjectStore (..))+import Kenshou.Remote.Store.File (newFileStore)+import Kenshou.Remote.Store.Gcs (newGcsStore, newTokenProvider)+import Options.Applicative hiding (value)+import Options.Applicative qualified as Options+import System.Directory (createDirectory, createDirectoryIfMissing, doesFileExist, doesPathExist, getCurrentDirectory, listDirectory, makeAbsolute, renameFile)+import System.Environment (lookupEnv)+import System.Exit (ExitCode (..))+import System.FilePath (takeDirectory, takeExtension, (</>))+import System.IO (stderr)+import System.IO.Temp (withSystemTempDirectory, withTempDirectory)+import System.Process (readProcessWithExitCode)++data PerfFailure = PerfFailure Int Text deriving stock (Show)++instance Exception PerfFailure++perfCommand :: CliCommand+perfCommand =+  CliCommand+    { name = "perf",+      description = "Plan and prepare scoped runtime performance comparisons",+      group = Execution,+      hidden = False,+      parser =+        hsubparser+          ( command "plan" (info (planParser <**> helper) (progDesc "Resolve exact source commits and explain performance scope without building or leasing"))+              <> command "prepare" (info (prepareParser <**> helper) (progDesc "Check independent build inputs and publish both exact-revision payloads without leasing"))+              <> sessionCommands+          )+    }++planParser :: Parser (CliEnv -> IO ExitCode)+planParser =+  runPlan+    <$> strOption (long "request" <> metavar "FILE" <> help "kenshou.perf-request/v1 document")+    <*> strOption (long "out" <> metavar "DIRECTORY" <> help "New directory for pinned planning inputs and manifest")++prepareParser :: Parser (CliEnv -> IO ExitCode)+prepareParser =+  runPrepare+    <$> strOption (long "request" <> metavar "FILE")+    <*> strOption (long "out" <> metavar "DIRECTORY")+    <*> strOption (long "cache" <> metavar "DIRECTORY" <> Options.value ".dev/perf-payload-cache" <> showDefault)+    <*> strOption (long "control-bucket" <> metavar "BUCKET" <> Options.value "tan-nb-exp-cells-control" <> showDefault)++data ArmInputs = ArmInputs+  { armDescriptor :: Cohort.CohortDescriptor,+    armLock :: Value,+    armInventory :: Prepared.BuildInventory,+    armBounds :: Prepared.BuildInventory,+    armIdentity :: Cohort.CohortIdentity,+    armFlake :: FilePath,+    armDescriptorFile :: FilePath,+    armBuildKey :: Text+  }++runPrepare :: FilePath -> FilePath -> FilePath -> Text -> CliEnv -> IO ExitCode+runPrepare requestPath destination cachePath bucketName environment = guarded do+  started <- getCurrentTime+  unless (not (Text.null bucketName) && Text.all (\c -> c `elem` ['a' .. 'z'] || c `elem` ['0' .. '9'] || c `elem` ['-', '.']) bucketName) (throwIO (PerfFailure 2 "invalid control bucket"))+  occupied <- doesPathExist destination+  when occupied (throwIO (PerfFailure 2 "preparation output already exists"))+  _ <- ByteString.readFile requestPath >>= (decode "request" :: ByteString.ByteString -> IO PerfRequest)+  dirty <- process "git" ["status", "--porcelain"]+  unless (Text.null dirty) (throwIO (PerfFailure 4 "preparation requires a clean committed harness"))+  revision <- Text.strip <$> process "git" ["rev-parse", "HEAD"]+  root <- getCurrentDirectory+  output <- makeAbsolute destination+  cache <- makeAbsolute cachePath+  let evidence = output </> "evidence"+      work = output </> "work"+      harness = work </> "harness"+      bucket = Bucket bucketName+  createDirectoryIfMissing True (takeDirectory output)+  createDirectory output+  createDirectory evidence+  createDirectory work+  plan <- createPlan requestPath (evidence </> "planning") environment (Just (revision, False))+  planned <- getCurrentTime+  inputs <- ByteString.readFile (evidence </> "planning" </> "inputs.json") >>= decode "pinned planning inputs"+  -- Clone the committed harness, not the mutable working directory. Generated+  -- projects, locks and Cabal caches are siblings outside that checkout.+  _ <- loggedProcess (work </> "clone.log") "git" ["clone", "--quiet", "--no-hardlinks", root, harness]+  _ <- loggedProcess (work </> "checkout.log") "git" ["-C", harness, "checkout", "--quiet", "--detach", Text.unpack revision]+  baseline <- resolveArm environment.perf output harness revision "baseline"+  candidate <- if plan.request.baselineRevision == plan.request.candidateRevision then pure baseline else resolveArm environment.perf output harness revision "candidate"+  targets <- requireInput (targetPackages plan.request baseline.armDescriptor)+  requireInput (Prepared.compareInventories targets baseline.armInventory candidate.armInventory)+  requireInput (Prepared.compareInventories targets baseline.armBounds candidate.armBounds)+  resolutionFinished <- getCurrentTime+  progress "both inventories and declared bounds checked; publishing payloads"+  store <- openPerfStore bucket+  createDirectoryIfMissing True cache+  (baselinePayload, baselineHit) <- publishArm store bucket cache harness revision baseline+  (candidatePayload, candidateHit) <- if baseline.armBuildKey == candidate.armBuildKey then pure (baselinePayload, True) else publishArm store bucket cache harness revision candidate+  published <- getCurrentTime+  let arm resolved payload = Prepared.PreparedArm resolved.armDescriptor resolved.armLock resolved.armBounds resolved.armInventory (toJSON payload) resolved.armBuildKey+      prepared = Prepared.PerfPrepared plan inputs revision (arm baseline baselinePayload) (arm candidate candidatePayload)+  requireInput (Prepared.validatePrepared environment.perf prepared)+  encodeFile (evidence </> "prepared.json") prepared+  encodeFile+    (evidence </> "preparation-summary.json")+    ( object+        [ "baselineCacheHit" .= baselineHit,+          "candidateCacheHit" .= candidateHit,+          "preparedSha256" .= Prepared.preparedDigest prepared,+          "startedAt" .= started,+          "payloadsReadyAt" .= published,+          "stageSeconds"+            .= object+              [ "planning" .= (realToFrac (diffUTCTime planned started) :: Double),+                "resolutionAndBounds" .= (realToFrac (diffUTCTime resolutionFinished planned) :: Double),+                "publication" .= (realToFrac (diffUTCTime published resolutionFinished) :: Double),+                "payloadPreparation" .= (realToFrac (diffUTCTime published started) :: Double)+              ]+        ]+    )+  sealPerfDirectory evidence >>= requireInput+  verifyPerfManifest evidence >>= requireInput+  LazyByteString.putStrLn (encode (object ["prepared" .= (evidence </> "prepared.json"), "preparedSha256" .= Prepared.preparedDigest prepared, "baselineCacheHit" .= baselineHit, "candidateCacheHit" .= candidateHit]))+  pure ExitSuccess++resolveArm :: PerfContracts -> FilePath -> FilePath -> Text -> Text -> IO ArmInputs+resolveArm contracts output harness revision arm = do+  let evidence = output </> "evidence" </> "planning"+      work = output </> "work" </> Text.unpack arm+      flake = work </> "build"+      descriptorPath = flake </> "cohort.json"+      lockPath = flake </> "source-lock.json"+  createDirectory work+  createDirectory flake+  descriptor <- ByteString.readFile (evidence </> Text.unpack arm <> "-cohort.json") >>= decode "generated cohort"+  encodeFile descriptorPath (descriptor :: Cohort.CohortDescriptor)+  _ <- loggedProcess (work </> "source-lock.log") "bash" [harness </> "scripts" </> "payload-lock.sh", "--descriptor", descriptorPath, "--out", lockPath]+  lock <- ByteString.readFile lockPath >>= decode "source lock"+  names <- listDirectory harness+  packages <- filterMIO (\name -> doesFileExist (harness </> name </> name <> ".cabal")) (filter (Text.isPrefixOf "kenshou-" . Text.pack) names)+  let selectedPackages = contracts.lanePackages descriptor packages+  boundsProject <- requireInput (Prepared.renderBoundsProject contracts [harness </> name </> name <> ".cabal" | name <- selectedPackages] descriptor)+  let projectPath = work </> "bounds.project"+      planPath = work </> "bounds-dist" </> "cache" </> "plan.json"+  Text.IO.writeFile projectPath boundsProject+  _ <- loggedProcess (work </> "resolution.log") "cabal" ["build", "all", "--dry-run", "--project-file=" <> projectPath, "--builddir=" <> work </> "bounds-dist"]+  initialPlan <- ByteString.readFile planPath >>= decode "initial bounds plan"+  _ <- requireInput (Prepared.cabalInventory harness revision initialPlan)+  cacheDocument <- loggedProcess (work </> "cabal-path.log") "cabal" ["path", "--ignore-project", "--remote-repo-cache", "--output-format=json"] >>= decodeText "Cabal cache location"+  repoCache <- either (throwIO . PerfFailure 4 . Text.pack) pure (parseEither (withObject "CabalPaths" (.: "remote-repo-cache")) cacheDocument :: Either String FilePath)+  _ <-+    loggedProcess+      (work </> "resolved-lock.log")+      "python3"+      [harness </> "scripts" </> "perf-resolved-lock.py", "--plan", planPath, "--index", repoCache </> "hackage.haskell.org" </> "01-index.tar", "--cohort-lock", lockPath, "--out", flake </> "resolved-lock.json"]+  wrapper <- requireInput (generatedFlake harness revision descriptor)+  Text.IO.writeFile (flake </> "flake.nix") wrapper+  _ <- loggedProcess (work </> "flake-lock.log") "nix" ["flake", "lock", "path:" <> flake]+  let attribute = "path:" <> flake <> "#packages.x86_64-linux.kenshou-" <> Text.unpack (Cohort.unCohortName descriptor.descriptorName)+  inventory <- loggedProcess (work </> "build-inventory.log") "nix" ["eval", "--json", attribute <> ".buildInventory"] >>= decodeText "Nix inventory"+  identity <- loggedProcess (work </> "cohort-identity.log") "nix" ["eval", "--json", attribute <> ".cohortIdentity"] >>= decodeText "Nix identity"+  descriptorBytes <- ByteString.readFile descriptorPath+  _ <- requireInput (Publisher.validateCohortIdentity descriptor (Text.drop 7 (sha256Hex descriptorBytes)) identity)+  let actualConstraints = Text.intercalate ", " [entry.name <> " ==" <> entry.version | entry <- (inventory :: Prepared.BuildInventory).packages, not ("kenshou-" `Text.isPrefixOf` entry.name)]+  Text.IO.writeFile (work </> "nix-bounds.project") (boundsProject <> "\nconstraints: " <> actualConstraints <> "\n")+  _ <- loggedProcess (work </> "bounds.log") "cabal" ["build", "all", "--dry-run", "--project-file=" <> work </> "nix-bounds.project", "--builddir=" <> work </> "nix-bounds-dist"]+  cabalPlan <- ByteString.readFile (work </> "nix-bounds-dist" </> "cache" </> "plan.json") >>= decode "bounds plan"+  bounds <- requireInput (Prepared.cabalInventory harness revision cabalPlan)+  boundsIdentity <- either (throwIO . PerfFailure 4 . Text.pack . show) pure (Cohort.identityFromPlan descriptor (Text.drop 7 (sha256Hex descriptorBytes)) cabalPlan)+  unless (null (Cohort.checkCohort descriptor boundsIdentity)) (throwIO (PerfFailure 4 "independent bounds plan differs from the generated cohort"))+  requireInput (Prepared.validateArmInventories descriptor inventory bounds)+  encodeFile (work </> "build-inventory.json") inventory+  encodeFile (work </> "bounds-inventory.json") bounds+  let key = Prepared.buildKeyFor revision descriptor lock inventory bounds+  pure (ArmInputs descriptor lock inventory bounds identity flake descriptorPath key)++generatedFlake :: FilePath -> Text -> Cohort.CohortDescriptor -> Either Text Text+generatedFlake harness revision descriptor = do+  unless (not (any (`elem` ['\n', '\r', ' ', '"', '\\']) harness) && not ("${" `Text.isInfixOf` Text.pack harness) && immutableRevision revision) (Left "unsafe pinned harness path")+  let quote = Text.Encoding.decodeUtf8 . canonicalEncode . String+      location = "git+file://" <> Text.pack harness <> "?rev=" <> revision+  pure+    ( Text.unlines+        [ "{",+          "  inputs.harness.url = " <> quote location <> ";",+          "  outputs = { self, harness }: let",+          "    inputs = harness.inputs // { self = harness; };",+          "    pkgs = import inputs.nixpkgs { system = \"x86_64-linux\"; };",+          "  in { packages.x86_64-linux." <> quote ("kenshou-" <> Cohort.unCohortName descriptor.descriptorName) <> " = import (harness + \"/nix/kenshou/packages.nix\") {",+          "    inherit inputs pkgs; cohort = " <> quote (Cohort.unCohortName descriptor.descriptorName) <> "; variant = \"default\";",+          "    descriptorFile = ./cohort.json; lockFile = ./source-lock.json; resolvedLockFile = ./resolved-lock.json; strictBounds = true;",+          "  }; };",+          "}"+        ]+    )++publishArm :: ObjectStore -> Bucket -> FilePath -> FilePath -> Text -> ArmInputs -> IO (Payload.PayloadDescriptor, Bool)+publishArm store bucket cache harness revision arm = do+  let cached = cache </> Text.unpack (Text.drop 7 arm.armBuildKey) <> ".json"+  exists <- doesFileExist cached+  if exists+    then do+      payload <- ByteString.readFile cached >>= decode "cached payload"+      unless ((payload :: Payload.PayloadDescriptor).cohortIdentity == arm.armIdentity && payload.harness == Payload.Harness revision False && payload.variant == "default") (throwIO (PerfFailure 4 "cached payload identity mismatch"))+      withSystemTempDirectory "kenshou-perf-cache-check-" \temporary -> do+        let bundleObject = ObjectName ("payloads/sha256/" <> payload.cell.bundle.sha256 <> ".nar.zst")+            file = temporary </> "bundle.nar.zst"+        unless (payload.cell.bundle.uri == "gs://" <> bucket.unBucket <> "/payloads/sha256/" <> payload.cell.bundle.sha256 <> ".nar.zst") (throwIO (PerfFailure 4 "cached payload belongs to another bucket"))+        found <- store.downloadTo bucket bundleObject file+        when (found == Nothing) (throwIO (PerfFailure 4 "cached payload bundle is missing"))+        (digest, bytes) <- digestFile file+        unless ((digest, bytes) == (payload.cell.bundle.sha256, payload.cell.bundle.bytes)) (throwIO (PerfFailure 4 "cached bundle digest/size mismatch"))+      progress "reused verified published payload"+      pure (payload, True)+    else do+      result <- Publisher.publishPreparedPayload store (Publisher.PreparedPublishOptions harness revision ("path:" <> arm.armFlake) arm.armDescriptorFile (Cohort.unCohortName arm.armDescriptor.descriptorName) "default" bucket (arm.armFlake </> "payload.json"))+      payload <- case result of+        Left problem -> do+          encodeFile (takeDirectory arm.armFlake </> "publication-failure.json") (object ["error" .= show problem])+          throwIO (PerfFailure 4 (Text.pack (show problem)))+        Right value -> pure value+      withTempDirectory cache ".pending-" \temporary -> do+        let staged = temporary </> "payload.json"+        encodeFile staged payload+        renameFile staged cached+      pure (payload, False)++loggedProcess :: FilePath -> String -> [String] -> IO Text+loggedProcess logPath executable arguments = do+  (status, output, errors) <- readProcessWithExitCode executable arguments ""+  Text.IO.writeFile logPath (Text.pack output <> Text.pack errors)+  case status of+    ExitSuccess -> pure (Text.pack output)+    _ -> throwIO (PerfFailure 4 (Text.pack executable <> " failed; retained log: " <> Text.pack logPath <> "\n" <> Text.take 2000 (Text.pack errors)))++decodeText :: (FromJSON a) => Text -> Text -> IO a+decodeText label = decode label . Text.Encoding.encodeUtf8++filterMIO :: (a -> IO Bool) -> [a] -> IO [a]+filterMIO predicate values = fmap (map fst . filter snd) (traverse (\item -> (item,) <$> predicate item) values)++openPerfStore :: Bucket -> IO ObjectStore+openPerfStore bucket =+  lookupEnv "KENSHOU_CELL_STORE" >>= \case+    Nothing -> newTokenProvider bucket >>= newGcsStore+    Just setting -> case Text.stripPrefix "file:" (Text.pack setting) of+      Just root | not (Text.null root) -> newFileStore (Text.unpack root)+      _ -> throwIO (PerfFailure 2 "KENSHOU_CELL_STORE must be file:<directory> or unset")++runPlan :: FilePath -> FilePath -> CliEnv -> IO ExitCode+runPlan requestPath output environment = guarded do+  plan <- createPlan requestPath output environment Nothing+  LazyByteString.putStrLn (encode plan)+  pure ExitSuccess++planningToolchain :: Cohort.CohortDescriptor -> IO Prepared.BuildInventory+planningToolchain descriptor = do+  bytes <- ByteString.readFile "flake.lock"+  lock <- decode "flake lock" bytes+  (nixpkgs, haskellNix) <- requireInput (either (Left . Text.pack) Right (parseEither pins lock))+  pure (Prepared.BuildInventory "nix" descriptor.descriptorCompiler "x86_64-linux" (object ["nixpkgsRevision" .= nixpkgs, "haskellNixRevision" .= haskellNix, "flakeLockSha256" .= sha256Hex bytes, "variant" .= ("default" :: Text), "strictBounds" .= True]) [])+  where+    pins = withObject "FlakeLock" \v -> do+      nodes <- v .: "nodes"+      root <- v .: "root"+      let node name = withObject "LockNodes" (.: Key.fromText name) nodes+          follow :: Int -> Text -> Text -> Aeson.Parser Text+          follow depth name key = do+            when (depth > 32) (fail "cyclic or excessive flake follows chain")+            value <- node name >>= withObject "LockNode" (.: "inputs") >>= withObject "LockInputs" (.: Key.fromText key)+            case value of+              String destination -> pure destination+              Array _ -> parseJSON @[Text] value >>= foldM (follow (depth + 1)) root+              _ -> fail "unsupported flake input reference"+          revision input = do+            name <- follow 0 root input+            value <- node name >>= withObject "LockNode" (.: "locked") >>= withObject "LockedInput" (.: "rev")+            unless (immutableRevision value) (fail "toolchain input needs an immutable commit")+            pure value+      (,) <$> revision "nixpkgs" <*> revision "haskell-nix"++harnessSemantics :: IO Text+harnessSemantics = do+  listed <- process "git" ["ls-files", "-z"]+  let paths = sort [Text.unpack path | path <- Text.splitOn "\0" listed, takeExtension (Text.unpack path) `elem` [".hs", ".cabal", ".nix", ".py"] || path `elem` ["flake.lock", "cabal.project"]]+  records <- traverse (\path -> do bytes <- ByteString.readFile path; pure (object ["path" .= path, "sha256" .= sha256Hex bytes])) paths+  pure (sha256Hex (canonicalEncode (toJSON records)))++createPlan :: FilePath -> FilePath -> CliEnv -> Maybe (Text, Bool) -> IO PerfPlan+createPlan requestPath output environment pinnedHarness = do+  occupied <- doesPathExist output+  when occupied (throwIO (PerfFailure 2 "output already exists; choose a new planning directory"))+  requestBytes <- ByteString.readFile requestPath+  request <- decode "request" requestBytes+  descriptorBytes <- ByteString.readFile request.cohortTemplate+  descriptor <- decode "cohort template" descriptorBytes+  expectedPackages <- requireInput (targetPackages request descriptor)+  graph <- either (throwIO . PerfFailure 2 . (.errorText)) pure (Graph.decodeGraph environment.componentGraph)+  let nodes = [component | component <- graph.components, any (`elem` expectedPackages) component.packages]+  unless (all (\package -> any (elem package . (.packages)) nodes) expectedPackages) (throwIO (PerfFailure 2 "target packages are not represented in the component graph"))+  profilesBytes <- ByteString.readFile "policies/performance/catalog.v1.json"+  profiles <- decode "performance catalog" profilesBytes+  unless (all (`elem` map (.name) profiles.profiles) request.profiles) (throwIO (PerfFailure 2 "unknown requested performance profile"))+  suiteBytes <- ByteString.readFile ("suites/" <> Text.unpack request.suite <> ".json")+  suite <- readSuite ("suites/" <> Text.unpack request.suite <> ".json") >>= requireInput+  unless (suite.name == request.suite) (throwIO (PerfFailure 2 "suite name differs from request"))+  policies <- traverse loadPolicy (nub (map (.policy) profiles.profiles))+  let policyMap = Map.fromList [(Text.pack path, sha256Hex (canonicalEncode value)) | (path, value, _) <- policies]+  mapM_ (checkPolicy policies) profiles.profiles+  let catalog = map fromScenario (allScenarios environment.registry)+  mapM_ (requireInput . validateProfile environment.perf catalog) profiles.profiles+  sourceProject <- resolveProject request.project+  unless (all ((== sourceProject.repository) . (.repository)) nodes) (throwIO (PerfFailure 2 "target graph nodes belong to another repository"))+  withSystemTempDirectory "kenshou-perf-source-" \checkout -> do+    progress "verifying requested commits in the registered HTTPS upstream"+    _ <- process "git" ["init", "--quiet", checkout]+    mapM_ (fetch checkout sourceProject.location) (nub [request.baselineRevision, request.candidateRevision])+    baseline <- inspect checkout request descriptor sourceProject.location request.baselineRevision+    candidate <- if request.baselineRevision == request.candidateRevision then pure baseline else inspect checkout request descriptor sourceProject.location request.candidateRevision+    baselineDescriptor <- requireInput (descriptorForSnapshot request descriptor "baseline" baseline)+    candidateDescriptor <- requireInput (descriptorForSnapshot request descriptor "candidate" candidate)+    changedText <- process "git" ["-C", checkout, "diff", "--no-renames", "--name-only", "-z", Text.unpack request.baselineRevision, Text.unpack request.candidateRevision]+    let paths = map Text.unpack (filter (not . Text.null) (Text.splitOn "\0" changedText))+    (mapped, _) <- Git.changesFromUpstream graph (Git.UpstreamDiff sourceProject.repository checkout request.baselineRevision request.candidateRevision) >>= requireInput+    let (changes, unknown, documentation) = conservativeChanges graph sourceProject.repository paths mapped+    provisional <- requireInput (planPerformance environment.perf request {purpose = "calibration"} graph catalog profiles changes unknown documentation policyMap Map.empty)+    toolchain <- planningToolchain descriptor+    semantics <- harnessSemantics+    admissionChecks <-+      traverse+        ( \entry -> do+            policy <- case [document | (path, document, _) <- policies, path == entry.profile.policy] of [value] -> pure value; _ -> throwIO (PerfFailure 2 "missing calibration policy")+            verified <- if request.purpose == "calibration" then pure (Left "calibration requests do not consume gating admissions") else case entry.profile.calibration of Nothing -> pure (Left "no calibration evidence"); Just _ -> verifyCalibrationAdmission environment.perf entry (toolchainDigest toolchain) semantics policy+            pure (entry, verified)+        )+        provisional.selected+    let admitted = Map.fromList [(entry.workloadSha256, digest) | (entry, Right digest) <- admissionChecks]+        calibrationProblems = Map.fromList [(entry.profile.name, problem) | (entry, Left problem) <- admissionChecks]+    plan <- requireInput (planPerformance environment.perf request graph catalog profiles changes unknown documentation policyMap admitted)+    (harnessRevision, harnessDirty) <- case pinnedHarness of+      Just snapshot -> pure snapshot+      Nothing -> do+        revision <- Text.strip <$> process "git" ["rev-parse", "HEAD"]+        status <- process "git" ["status", "--porcelain"]+        pure (revision, not (Text.null status))+    let inputs =+          object+            [ "request" .= request,+              "template" .= descriptor,+              "templateSha256" .= sha256Hex descriptorBytes,+              "baselineSource" .= baseline,+              "candidateSource" .= candidate,+              "harness" .= object ["revision" .= harnessRevision, "dirty" .= harnessDirty],+              "harnessSemanticsSha256" .= semantics,+              "buildToolchain" .= toolchain.toolchain,+              "calibrationAdmissions" .= admitted,+              "calibrationProblems" .= calibrationProblems,+              "graph" .= graph,+              "scenarioCatalog" .= catalog,+              "profileCatalog" .= profiles,+              "suite" .= either (const Null) id (eitherDecodeStrict' suiteBytes :: Either String Value),+              "policies" .= Map.fromList [(path, document) | (path, document, _) <- policies],+              "changedPaths" .= paths+            ]+    createDirectoryIfMissing True (takeDirectory output)+    -- Exclusive creation also rejects a concurrent writer racing the preflight.+    createDirectory output+    ByteString.writeFile (output </> "template.json") descriptorBytes+    encodeFile (output </> "request.json") request+    encodeFile (output </> "inputs.json") inputs+    encodeFile (output </> "plan.json") plan+    encodeFile (output </> "baseline-cohort.json") baselineDescriptor+    encodeFile (output </> "candidate-cohort.json") candidateDescriptor+    sealPerfDirectory output >>= requireInput+    progress ("planned " <> Text.pack (show (length plan.selected)) <> " configurations; scope " <> plan.coverage)+    pure plan+  where+    loadPolicy path = do+      bytes <- ByteString.readFile path+      parsed <- requireInput (Policy.decodePolicy bytes)+      document <- decode "comparison policy" bytes+      pure (path, document, parsed)+    checkPolicy policies profile = case [policy | (path, _, policy) <- policies, path == profile.policy] of+      [policy] -> requireInput (validateProfilePolicy profile policy)+      _ -> throwIO (PerfFailure 2 "profile policy is missing")++data SourceProject = SourceProject {repository :: Text, location :: Text}++resolveProject :: Text -> IO SourceProject+resolveProject uri = do+  let qualified = Text.unpack (Text.drop 7 uri)+  raw <- process "mori" ["registry", "show", qualified, "--full", "--json"]+  document <- requireInput (either (Left . Text.pack) Right (eitherDecodeStrict' (Text.Encoding.encodeUtf8 raw)))+  repositories <- requireInput (either (Left . Text.pack) Right (parseEither (withObject "Mori project" (.: "repositories")) document :: Either String [Value]))+  case repositories of+    [repository] -> requireInput (either (Left . Text.pack) Right (parseEither parseRepository repository))+    _ -> throwIO (PerfFailure 2 "source project must identify exactly one upstream repository")+  where+    parseRepository = withObject "Mori repository" \value -> do+      repository <- value .: "name"+      github <- value .:? "github"+      git <- value .:? "git"+      location <- case (github :: Maybe Text, git :: Maybe Text) of+        (Just hosted, _) | length (Text.splitOn "/" hosted) == 2 && Text.all (\c -> c `elem` ("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-_/" :: String)) hosted -> pure ("https://github.com/" <> hosted <> ".git")+        (Nothing, Just url) | "https://" `Text.isPrefixOf` url && not (Text.any (`elem` ['@', '#', '?', '\n', '\r']) url) -> pure url+        _ -> fail "registered upstream must use HTTPS Git"+      pure (SourceProject repository location)++fetch :: FilePath -> Text -> Text -> IO ()+fetch checkout location revision = do+  _ <- process "git" ["-c", "credential.interactive=false", "-C", checkout, "fetch", "--quiet", "--no-tags", "--depth=1", Text.unpack location, Text.unpack revision]+  actual <- Text.strip <$> process "git" ["-C", checkout, "rev-parse", Text.unpack revision <> "^{commit}"]+  unless (actual == revision) (throwIO (PerfFailure 2 "upstream revision does not identify the requested commit"))++inspect :: FilePath -> PerfRequest -> Cohort.CohortDescriptor -> Text -> Text -> IO SourceSnapshot+inspect checkout request descriptor location revision = do+  pathsText <- process "git" ["-C", checkout, "ls-tree", "-r", "--name-only", "-z", Text.unpack revision]+  let paths = filter (".cabal" `Text.isSuffixOf`) (Text.splitOn "\0" pathsText)+  manifests <- traverse readManifest paths+  requireInput (sourceSnapshot request descriptor location revision [manifest | Right manifest <- manifests])+  where+    readManifest path = do+      contents <- process "git" ["-C", checkout, "show", Text.unpack revision <> ":" <> Text.unpack path]+      pure (parsePackageManifest (Text.unpack path) contents)++process :: String -> [String] -> IO Text+process executable arguments = do+  (status, output, errors) <- readProcessWithExitCode executable arguments ""+  case status of+    ExitSuccess -> pure (Text.pack output)+    _ -> throwIO (PerfFailure 4 (Text.pack executable <> ": " <> Text.strip (Text.pack errors)))++decode :: (FromJSON a) => Text -> ByteString.ByteString -> IO a+decode label bytes = requireInput (either (Left . ((label <> ": ") <>) . Text.pack) Right (eitherDecodeStrict' bytes))++requireInput :: Either Text a -> IO a+requireInput = either (throwIO . PerfFailure 2) pure++progress :: Text -> IO ()+progress = Text.IO.hPutStrLn stderr . ("kenshou perf: " <>)++guarded :: IO ExitCode -> IO ExitCode+guarded operation = operation `catch` failure `catch` ioFailure+  where+    failure (PerfFailure code message) = progress message >> pure (ExitFailure code)+    ioFailure (problem :: IOException) = progress (Text.pack (displayException problem)) >> pure (ExitFailure 4)
+ src/Kenshou/Cli/Command/Perf/Session.hs view
@@ -0,0 +1,1014 @@+module Kenshou.Cli.Command.Perf.Session (sessionCommands, replayCalibrationReceipt, verifyCalibrationAdmission) where++import Control.Concurrent (threadDelay)+import Control.Exception (Exception, IOException, catch, displayException, throwIO, try)+import Control.Monad (unless, when)+import Data.Aeson hiding (decode)+import Data.Aeson.Types (parseEither)+import Data.ByteString qualified as Bytes+import Data.ByteString.Lazy.Char8 qualified as Lazy+import Data.List (find, isSuffixOf)+import Data.List qualified as List+import Data.List.NonEmpty (NonEmpty (..))+import Data.Map.Strict qualified as Map+import Data.Maybe (fromMaybe)+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Text.IO qualified as TextIO+import Data.Time (UTCTime, diffUTCTime, getCurrentTime)+import Data.Time.Clock.POSIX (posixSecondsToUTCTime)+import Kenshou.Cli.Command.Cell (startCellInstances)+import Kenshou.Core.Bundle (registryRoutingRules)+import Kenshou.Core.Canonical (canonicalEncode, sha256Hex)+import Kenshou.Core.Cli (CliEnv (..))+import Kenshou.Core.Id+import Kenshou.Core.Phase (zeroPhases)+import Kenshou.Measure.Compare.Compatibility (VaryingAxis (..))+import Kenshou.Measure.Compare.Ordering (PairedOrdering (ABBA))+import Kenshou.Measure.Compare.Policy (Direction (..), MetricRule (..), Policy, matchesRule)+import Kenshou.Measure.Compare.Policy qualified as Policy+import Kenshou.Perf.Artifacts+import Kenshou.Perf.Calibration qualified as Calibration+import Kenshou.Perf.Contracts (PerfContracts)+import Kenshou.Perf.Plan+import Kenshou.Perf.Plan qualified as PerfPlan+import Kenshou.Perf.Prepared qualified as Prepared+import Kenshou.Perf.Request (PerfRequest (..), strictObject)+import Kenshou.Perf.Result+import Kenshou.Plan.RunPlan (PlannedRun (..), RunPlan (..))+import Kenshou.Remote.Cell.Control+import Kenshou.Remote.Cell.Docs (CachePolicy (Cold), CellBuckets (..), CellDescriptor (..), Submission (..), WorkObject (..))+import Kenshou.Remote.Cell.Lease qualified as Lease+import Kenshou.Remote.Cell.Prepare+import Kenshou.Remote.Cell.RouteRules+import Kenshou.Remote.Cell.Session.Build (BuildOptions (..), buildSession)+import Kenshou.Remote.Cell.Session.Journal (SessionJournal (..), SliceJournal (..), SliceState (..), readSessionJournal)+import Kenshou.Remote.Cell.Session.Resume (resumeObservedSlices)+import Kenshou.Remote.Cell.WorkJson (decodeWorkPlan)+import Kenshou.Remote.Pair+import Kenshou.Remote.Payload (Bundle (..), CellPayload (..), PayloadDescriptor (..))+import Kenshou.Remote.Payload.Publish (digestFile)+import Kenshou.Remote.Perf.Attempt+import Kenshou.Remote.Perf.Cell+import Kenshou.Remote.Perf.Cleanup+import Kenshou.Remote.Perf.Conditions (validateHealthPolicy)+import Kenshou.Remote.Perf.Journal+import Kenshou.Remote.Perf.Lock+import Kenshou.Remote.Perf.Publication+import Kenshou.Remote.Perf.Runner+import Kenshou.Remote.Store+import Kenshou.Remote.Store.File (newFileStore)+import Kenshou.Remote.Store.Gcs (newGcsStore, newTokenProvider)+import Options.Applicative hiding (Success, action)+import System.Directory+import System.Environment (lookupEnv)+import System.Exit (ExitCode (..))+import System.FilePath (splitDirectories, takeDirectory, (</>))+import System.IO (stderr)+import System.IO.Temp (withSystemTempDirectory)+import Text.Printf (printf)++data SessionFailure = SessionFailure Int Text deriving stock (Show)++instance Exception SessionFailure++data PublicationReceipt = PublicationReceipt+  { snapshot :: Text,+    manifestUri :: Text,+    manifestSha256 :: Text,+    resultSha256 :: Text,+    payloadsReadyAt :: UTCTime,+    publicationStartedAt :: UTCTime,+    verifiedReportAt :: UTCTime,+    totalElapsedSeconds :: Double+  }+  deriving stock (Eq, Show)++instance ToJSON PublicationReceipt where+  toJSON receipt = object ["schema" .= ("kenshou.perf-publication/v1" :: Text), "snapshot" .= receipt.snapshot, "manifestUri" .= receipt.manifestUri, "manifestSha256" .= receipt.manifestSha256, "resultSha256" .= receipt.resultSha256, "payloadsReadyAt" .= receipt.payloadsReadyAt, "publicationStartedAt" .= receipt.publicationStartedAt, "verifiedReportAt" .= receipt.verifiedReportAt, "totalElapsedSeconds" .= receipt.totalElapsedSeconds]++instance FromJSON PublicationReceipt where+  parseJSON = withObject "PerformanceReceipt" \v -> do+    strictObject ["schema", "snapshot", "manifestUri", "manifestSha256", "resultSha256", "payloadsReadyAt", "publicationStartedAt", "verifiedReportAt", "totalElapsedSeconds"] v+    schema <- v .: "schema"+    unless (schema == ("kenshou.perf-publication/v1" :: Text)) (fail "unsupported publication receipt")+    receipt <- PublicationReceipt <$> v .: "snapshot" <*> v .: "manifestUri" <*> v .: "manifestSha256" <*> v .: "resultSha256" <*> v .: "payloadsReadyAt" <*> v .: "publicationStartedAt" <*> v .: "verifiedReportAt" <*> v .: "totalElapsedSeconds"+    unless (receipt.payloadsReadyAt <= receipt.publicationStartedAt && receipt.publicationStartedAt <= receipt.verifiedReportAt && abs (receipt.totalElapsedSeconds - realToFrac (diffUTCTime receipt.verifiedReportAt receipt.payloadsReadyAt)) < 0.000001) (fail "publication receipt timing mismatch")+    pure receipt++data ExecutionInputs = ExecutionInputs+  { controlBucket :: Text,+    storeLabel :: Text,+    allowedProjects :: [Text],+    start :: Bool,+    healthPolicies :: Map.Map Text Value,+    cells :: [(CellDescriptor, Maybe CellCapabilities)]+  }++instance ToJSON ExecutionInputs where+  toJSON inputs = object ["controlBucket" .= inputs.controlBucket, "storeLabel" .= inputs.storeLabel, "allowedProjects" .= inputs.allowedProjects, "start" .= inputs.start, "healthPolicies" .= inputs.healthPolicies, "cells" .= [object ["descriptor" .= descriptor, "capabilities" .= capabilities] | (descriptor, capabilities) <- inputs.cells]]++instance FromJSON ExecutionInputs where+  parseJSON = withObject "PerformanceExecutionInputs" \v -> do+    strictObject ["controlBucket", "storeLabel", "allowedProjects", "start", "healthPolicies", "cells"] v+    entries <-+      v .: "cells"+        >>= traverse+          ( withObject "PinnedCell" \c -> do+              strictObject ["descriptor", "capabilities"] c+              (,) <$> c .: "descriptor" <*> c .:? "capabilities"+          )+    ExecutionInputs <$> v .: "controlBucket" <*> v .: "storeLabel" <*> v .: "allowedProjects" <*> v .: "start" <*> (v .:? "healthPolicies" .!= Map.empty) <*> pure entries++sessionCommands :: Mod CommandFields (CliEnv -> IO ExitCode)+sessionCommands =+  command "run" (info (runParser <**> helper) (progDesc "Execute a pinned performance preparation with persistent budgets"))+    <> command "status" (info (statusParser <**> helper) (progDesc "Read persistent performance session state without cell actions"))+    <> command "resume" (info (resumeParser <**> helper) (progDesc "Verify inputs and reconcile a retained performance session"))+    <> command "verify" (info (verifyParser <**> helper) (progDesc "Replay retained comparisons and verify published evidence without executing trials"))+    <> command "calibration" (info (calibrationParser <**> helper) (progDesc "Freeze calibration batches and verify their retained acceptance evidence"))+  where+    session = strOption (long "session" <> metavar "DIRECTORY")+    runParser = runNew <$> strOption (long "prepared" <> metavar "FILE") <*> strOption (long "evidence-base-uri" <> metavar "GS_URI") <*> strOption (long "out" <> metavar "DIRECTORY") <*> strOption (long "control-bucket" <> value "tan-nb-exp-cells-control" <> showDefault) <*> switch (long "start" <> help "Start stopped instances after acquiring their cell lease")+    statusParser = runStatus <$> session+    resumeParser = runResume <$> session+    verifyParser = runVerify <$> session <*> switch (long "offline" <> help "Verify the retained sealed snapshot and receipt without contacting storage")+    calibrationParser =+      hsubparser+        ( command "declare" (info (runCalibrationPublish True <$> strOption (long "definition" <> metavar "FILE") <*> strOption (long "uri" <> metavar "GS_URI")) (progDesc "Create and retrieve a frozen definition before any acceptance trials"))+            <> command "fingerprint" (info (runCalibrationFingerprint <$> strOption (long "descriptor" <> metavar "FILE") <*> strOption (long "cell-policy" <> metavar "FILE")) (progDesc "Read the exact machine/image profile and health-policy digest before a batch"))+            <> command "publish" (info (runCalibrationPublish False <$> strOption (long "acceptance" <> metavar "FILE") <*> strOption (long "uri" <> metavar "GS_URI")) (progDesc "Create and retrieve the complete acceptance receipt inventory"))+            <> command "verify" (info (runCalibrationVerify <$> strOption (long "reference" <> metavar "FILE") <*> strOption (long "prepared" <> metavar "FILE")) (progDesc "Replay every calibration receipt against one prepared profile"))+        )++runCalibrationFingerprint :: FilePath -> FilePath -> CliEnv -> IO ExitCode+runCalibrationFingerprint descriptorPath policyPath _ = guarded do+  descriptor <- readJSON descriptorPath+  policy <- readJSON policyPath+  input (validateHealthPolicy policy)+  Lazy.putStrLn (encode (object ["machineProfile" .= cellMachineProfile descriptor, "healthPolicySha256" .= sha256Hex (canonicalEncode policy)]))+  pure ExitSuccess++runNew :: FilePath -> Text -> FilePath -> Text -> Bool -> CliEnv -> IO ExitCode+runNew preparedPath uri destination bucket start environment = guarded do+  prepared <- eitherDecodeFileStrict' preparedPath >>= either (throwIO . SessionFailure 2 . Text.pack) pure+  input (Prepared.validatePrepared environment.perf prepared)+  _ <- input (parseDestination uri)+  input (initialSessionPlaceholder prepared uri)+  verifyPerfManifest (takeDirectory preparedPath) >>= require+  exists <- doesPathExist destination+  when exists (throwIO (SessionFailure 2 "performance output already exists"))+  verifyGatingCalibration environment.perf prepared+  output <- makeAbsolute destination+  (store, label) <- openStore (Bucket bucket)+  projects <- maybe ["tan-nb-exp"] (map Text.strip . Text.splitOn "," . Text.pack) <$> lookupEnv "KENSHOU_GCP_ALLOWED_PROJECTS"+  capabilityDirectory <- fromMaybe ".dev/cells" <$> lookupEnv "KENSHOU_CELL_CAPABILITIES_DIR"+  pinned <- traverse (loadCell store projects capabilityDirectory bucket) prepared.plan.request.cells+  healthPolicies <- if null prepared.plan.selected then pure Map.empty else Map.fromList <$> traverse (\(descriptor, _) -> (descriptor.name,) <$> loadHealthPolicy store bucket descriptor.name) pinned+  let inputs = ExecutionInputs bucket label projects start healthPolicies pinned+  checkCalibratedPolicies prepared inputs+  payloads <- loadPayloads store prepared+  payloadsReady <- getCurrentTime+  identifier <- newRunId+  preflight environment store inputs payloads prepared identifier+  -- The create-only probe is retained as evidence of the actual destination+  -- permissions. It needs object creation and retrieval, never deletion.+  accessProbe store uri identifier prepared+  session <- require (initialSession identifier (Prepared.preparedDigest prepared) prepared.plan uri payloadsReady)+  createDirectoryIfMissing True (takeDirectory output)+  createDirectory output+  copyTree (takeDirectory preparedPath) (output </> "preparation")+  encodeFile (output </> "prepared.json") prepared+  encodeFile (output </> "request.json") prepared.plan.request+  encodeFile (output </> "plan.json") prepared.plan+  writePerfSession (output </> "session.json") session {executionInputs = Just (toJSON inputs)}+  withPerformanceLock output (execute environment store inputs payloads prepared output)++-- Validate the destination using the same immutable journal admission rules+-- before obtaining an ID, writing output or performing any remote operation.+initialSessionPlaceholder :: Prepared.PerfPrepared -> Text -> Either Text ()+initialSessionPlaceholder prepared uri = do+  identifier <- parseRunId "00000000-0000-7000-8000-000000000000"+  let epoch = posixSecondsToUTCTime 0+  _ <- initialSession identifier (Prepared.preparedDigest prepared) prepared.plan uri epoch+  pure ()++runResume :: FilePath -> CliEnv -> IO ExitCode+runResume destination environment = guarded do+  output <- makeAbsolute destination+  withPerformanceLock output do+    prepared <- loadPreparation environment.perf (output </> "preparation/prepared.json")+    session <- readPerfSession (output </> "session.json") >>= require+    require (validateSessionInputs environment.perf prepared session)+    checkRootInputs output prepared+    verifyGatingCalibration environment.perf prepared+    inputs <- require (parseInputs session)+    checkCalibratedPolicies prepared inputs+    state <- require (journalState session)+    receiptExists <- doesFileExist (output </> "receipt.json")+    if receiptExists && state.lease == Nothing && state.pendingClaim == Nothing && state.activeComparison == Nothing && Map.size state.completed == length session.plan.selected+      then do+        (result, _) <- verifySessionEvidence environment.perf output False+        receipt <- readJSON (output </> "receipt.json")+        retained <- readPerfSession (output </> Text.unpack (receipt :: PublicationReceipt).snapshot </> "session.json") >>= require+        unless (retained == session) (ioError (userError "published report does not describe the current session"))+        Lazy.putStrLn (encode result)+        pure (exit result.exitCode)+      else do+        (store, label) <- openStore (Bucket inputs.controlBucket)+        unless (inputs.storeLabel == label) (ioError (userError "object store differs from the immutable performance session"))+        mapM_ (checkCell store inputs) inputs.cells+        payloads <- loadPayloads store prepared+        preflight environment store inputs payloads prepared session.sessionId+        execute environment store inputs payloads prepared output++runStatus :: FilePath -> CliEnv -> IO ExitCode+runStatus output _ = guarded do+  session <- readPerfSession (output </> "session.json") >>= require+  state <- require (journalState session)+  now <- getCurrentTime+  Lazy.putStrLn (encode (object ["session" .= session.sessionId, "requestSha256" .= session.plan.requestSha256, "admissionSpentSeconds" .= state.admissionSpentSeconds, "executionDeadline" .= state.executionDeadline, "cleanupDeadline" .= state.cleanupDeadline, "lease" .= state.lease, "pendingClaim" .= state.pendingClaim, "activeComparison" .= state.activeComparison, "attempts" .= state.attempts, "replacements" .= state.replacements, "completed" .= Map.elems state.completed, "runtimeBlockers" .= state.runtimeBlockers, "totalElapsedSeconds" .= (realToFrac (diffUTCTime now session.createdAt) :: Double)]))+  pure ExitSuccess++loadPreparation :: PerfContracts -> FilePath -> IO Prepared.PerfPrepared+loadPreparation contracts path = do+  verifyPerfManifest (takeDirectory path) >>= require+  prepared <- readJSON path+  require (Prepared.validatePrepared contracts prepared)+  pure prepared++loadCell :: ObjectStore -> [Text] -> FilePath -> Text -> Text -> IO (CellDescriptor, Maybe CellCapabilities)+loadCell store projects directory bucket name = do+  snapshot <- readCellSnapshot store (Lease.CellRef name (Bucket bucket)) projects >>= require+  capabilities <- loadCellCapabilities directory snapshot.descriptor >>= require+  pure (snapshot.descriptor, capabilities)++loadHealthPolicy :: ObjectStore -> Text -> Text -> IO Value+loadHealthPolicy store bucket cell = do+  retained <- store.getObject (Bucket bucket) (ObjectName ("cells/" <> cell <> "/policy.json"))+  bytes <- maybe (ioError (userError "cell health policy is missing")) (pure . fst) retained+  policy <- either (ioError . userError) pure (eitherDecode bytes)+  require (validateHealthPolicy policy)+  pure policy++checkCell :: ObjectStore -> ExecutionInputs -> (CellDescriptor, Maybe CellCapabilities) -> IO ()+checkCell store inputs (descriptor, _) = do+  snapshot <- readCellSnapshot store (Lease.CellRef descriptor.name (Bucket inputs.controlBucket)) inputs.allowedProjects >>= require+  case Map.lookup descriptor.name inputs.healthPolicies of+    Nothing -> pure ()+    Just frozen -> do+      policy <- loadHealthPolicy store inputs.controlBucket descriptor.name+      unless (policy == frozen) (ioError (userError "cell health policy changed since performance admission"))+  unless (snapshot.descriptor == descriptor) (ioError (userError "cell descriptor changed since performance preparation"))++parseInputs :: PerfSession -> Either Text ExecutionInputs+parseInputs session = do+  inputs <- maybe (Left "performance session has no frozen execution inputs") decode session.executionInputs+  unless (map ((.name) . fst) inputs.cells == session.plan.request.cells && not (Text.null inputs.controlBucket) && not (null inputs.allowedProjects)) (Left "frozen cell pool differs from the request")+  unless (null session.plan.selected || Map.keys inputs.healthPolicies == List.sort (map ((.name) . fst) inputs.cells)) (Left "frozen cell health policies are missing or differ from the pool")+  mapM_ validateHealthPolicy (Map.elems inputs.healthPolicies)+  mapM_+    ( \(descriptor, capabilities) -> do+        unless (descriptor.buckets.control == inputs.controlBucket && descriptor.project `elem` inputs.allowedProjects) (Left "frozen cell bucket or project is invalid")+        case capabilities of+          Nothing -> pure ()+          Just record -> unless (record.cell == descriptor.name && record.descriptorSha256 == descriptorDigest descriptor) (Left "frozen capabilities differ from the cell descriptor")+    )+    inputs.cells+  pure inputs++checkRootInputs :: FilePath -> Prepared.PerfPrepared -> IO ()+checkRootInputs directory prepared = mapM_ check [("prepared.json", toJSON prepared), ("plan.json", toJSON prepared.plan), ("request.json", toJSON prepared.plan.request)]+  where+    check (name, expected) = do+      actual <- readJSON (directory </> name)+      unless (actual == expected) (ioError (userError ("retained " <> name <> " differs from immutable preparation")))++loadPayloads :: ObjectStore -> Prepared.PerfPrepared -> IO (Map.Map Text PayloadDescriptor)+loadPayloads store prepared = do+  entries <-+    traverse+      ( \(label, arm) -> do+          payload <- require (decode arm.payload)+          let prefix = "gs://"+              (bucket, suffix) = Text.breakOn "/" (Text.drop (Text.length prefix) payload.cell.bundle.uri)+          unless (prefix `Text.isPrefixOf` payload.cell.bundle.uri && not (Text.null bucket) && suffix == "/payloads/sha256/" <> payload.cell.bundle.sha256 <> ".nar.zst") (ioError (userError "invalid pinned payload location"))+          withSystemTempDirectory "kenshou-perf-payload-check" \temporary -> do+            let path = temporary </> "bundle.nar.zst"+            present <- store.downloadTo (Bucket bucket) (ObjectName (Text.drop 1 suffix)) path+            unless (present /= Nothing) (ioError (userError "pinned payload is missing"))+            (digest, bytes) <- digestFile path+            unless (digest == payload.cell.bundle.sha256 && bytes == payload.cell.bundle.bytes) (ioError (userError "pinned payload retrieval digest differs"))+          pure (label, payload)+      )+      [("baseline", prepared.baseline), ("candidate", prepared.candidate)]+  pure (Map.fromList entries)++options :: BuildOptions+options = BuildOptions (PrepareOptions False False [] Cold) GranularityRun False NullSink Nothing (24 * 1024 * 1024 * 1024) (20 * 1024 * 1024 * 1024) "0.1.0"++adapter :: CliEnv -> ObjectStore -> ExecutionInputs -> Map.Map Text PayloadDescriptor -> (CellDescriptor, Maybe CellCapabilities) -> CellAdapter+adapter environment store inputs payloads (descriptor, capabilities) = CellAdapter environment.registry store (Lease.CellRef descriptor.name (Bucket inputs.controlBucket)) descriptor (Map.findWithDefault Null descriptor.name inputs.healthPolicies) capabilities (registryRoutingRules environment.registry) payloads options inputs.storeLabel (\event -> TextIO.hPutStrLn stderr (Text.pack (show event)))++preflight :: CliEnv -> ObjectStore -> ExecutionInputs -> Map.Map Text PayloadDescriptor -> Prepared.PerfPrepared -> RunId -> IO ()+preflight environment store inputs payloads prepared identifier = mapM_ check prepared.plan.selected+  where+    check entry = do+      schedule <- makeSchedule prepared entry identifier+      compatible <-+        traverse+          ( \pinned -> do+              let a = adapter environment store inputs payloads pinned+              built <- buildSession a.registry a.descriptor a.capabilities a.rules a.payloads a.options a.storeLabel identifier (encode schedule)+              pure (either (const False) (const (prepared.plan.request.purpose == "calibration" || entry.profile.machineProfile == cellMachineProfile a.descriptor)) built)+          )+          inputs.cells+      unless (or compatible) (ioError (userError ("no pinned cell supports profile " <> Text.unpack entry.profile.name)))++makeSchedule :: Prepared.PerfPrepared -> PlannedProfile -> RunId -> IO RunPlan+makeSchedule prepared entry identifier = do+  let profile = entry.profile+      source = object ["schema" .= ("kenshou.run-plan/v1" :: Text), "planId" .= identifier, "inputs" .= object ["preparedSha256" .= Prepared.preparedDigest prepared, "workloadSha256" .= entry.workloadSha256], "runs" .= [object ["ordinal" .= (1 :: Int), "runId" .= identifier, "estimateMinutes" .= (1 :: Int), "spec" .= object ["schema" .= ("kenshou.run-spec/v1" :: Text), "runId" .= identifier, "scenario" .= profile.scenario, "scenarioRevision" .= profile.scenarioRevision, "knobs" .= profile.knobs, "dimensions" .= profile.dimensions, "seed" .= prepared.plan.request.seed, "phases" .= zeroPhases, "timeoutSeconds" .= profile.timeoutSeconds, "environment" .= object ["placement" .= ("cell" :: Text), "machineProfile" .= profile.machineProfile]]]]]+  projected <- require (decodeWorkPlan source)+  require (pairPlan (PairRequest "candidate" "baseline" profile.minimumPairs ABBA (fromIntegral prepared.plan.request.seed) profile.replacementPairs) projected)++policyFor :: Prepared.PerfPrepared -> PlannedProfile -> Either Text Policy+policyFor prepared entry = do+  policies <- either (Left . Text.pack) Right (parseEither (withObject "PinnedInputs" (.: "policies")) prepared.inputs)+  document <- maybe (Left "pinned comparison policy missing") Right (Map.lookup entry.profile.policy (policies :: Map.Map FilePath Value))+  decode document++axesFor :: Prepared.PerfPrepared -> NonEmpty VaryingAxis+axesFor prepared = (if prepared.plan.request.control == "aa" then VaryControl else VaryCohort) :| []++attemptDirectory :: FilePath -> PlannedProfile -> Int -> FilePath+attemptDirectory output entry index = output </> "configurations" </> Text.unpack entry.profile.name </> "attempt-" <> show index++execute :: CliEnv -> ObjectStore -> ExecutionInputs -> Map.Map Text PayloadDescriptor -> Prepared.PerfPrepared -> FilePath -> IO ExitCode+execute environment store inputs payloads prepared output = do+  let ref identity = Lease.CellRef identity.cell (Bucket inputs.controlBucket)+      pinned identity = maybe (ioError (userError "leased cell is outside frozen pool")) pure (find ((== identity.cell) . (.name) . fst) inputs.cells)+      handle identity = Lease.reattachActiveLease store (ref identity) identity.leaseId >>= maybe (ioError (userError "performance lease is no longer active")) pure+      supports entry p = do+        planned <- makeSchedule prepared entry =<< newRunId+        let a = adapter environment store inputs payloads p+        built <- buildSession a.registry a.descriptor a.capabilities a.rules a.payloads a.options a.storeLabel planned.planId (encode planned)+        pure (either (const False) (const (prepared.plan.request.purpose == "calibration" || entry.profile.machineProfile == cellMachineProfile a.descriptor)) built)+      admission entry seconds pending callbacks = case pending of+        Just (name, identity, _) -> do+          unless (name == entry.profile.name) (ioError (userError "proposed claim belongs to another profile"))+          active <- Lease.reattachActiveLease store (ref identity) identity.leaseId+          pure (maybe (AdmissionFailed "proposed cell claim is unresolved, expired or cancelled; no new claim issued") (const (Admitted identity)) active)+        Nothing -> waitFor entry seconds callbacks+      waitFor entry left callbacks = do+        granted <- findLease entry inputs.cells callbacks+        case granted of+          Occupied | left > 1 -> threadDelay 1000000 >> waitFor entry (left - 1) callbacks+          _ -> pure granted+      findLease _ [] _ = pure Occupied+      findLease entry (p@(descriptor, _) : rest) callbacks = do+        compatible <- supports entry p+        if not compatible+          then findLease entry rest callbacks+          else do+            checkCell store inputs p+            identifier <- newRunId+            let identity = LeaseIdentity descriptor.name identifier+            callbacks.proposeClaim identity+            granted <- Lease.acquireLeaseWithId store (Lease.CellRef descriptor.name (Bucket inputs.controlBucket)) (Lease.LeaseRequest "kenshou-perf" "pinned runtime performance validation" 180) identifier+            case granted of+              Lease.Acquired _ -> pure (Admitted identity)+              Lease.Busy _ -> callbacks.withdrawClaim >> findLease entry rest callbacks+              Lease.Quarantined _ -> pure (AdmissionFailed "cell was quarantined during the proposed claim")+      compareOne identity entry index _ callbacks = do+        p <- pinned identity+        acquired <- handle identity+        Lease.withHeartbeat store (ref identity) acquired \_ -> do+          when inputs.start (startCellInstances (fst p))+          let directory = attemptDirectory output entry index+              schedulePath = directory </> "schedule.json"+          exists <- doesFileExist schedulePath+          schedule <-+            if exists+              then readJSON schedulePath >>= require . decodeWorkPlan+              else do+                identifier <- newRunId+                planned <- makeSchedule prepared entry identifier+                createDirectoryIfMissing True directory+                encodeFile (schedulePath <> ".next") planned+                renameFile (schedulePath <> ".next") schedulePath+                pure planned+          policy <- require (policyFor prepared entry)+          computed <- executeAttempt identity entry schedule policy (axesFor prepared) directory callbacks (executeCellTrial (adapter environment store inputs payloads p) acquired schedule directory entry.profile callbacks.recordRuntimeFailure)+          retainAttemptResult directory computed+      verify result = verifyConfiguration inputs prepared output result+      hooks =+        RunnerHooks+          environment.perf+          getCurrentTime+          admission+          (\identity -> maybe False (const True) <$> Lease.reattachActiveLease store (ref identity) identity.leaseId)+          (\identity entry -> pinned identity >>= supports entry)+          compareOne+          verify+          ( \identity -> do+              _ <- Lease.requestCancel store (ref identity) (Just identity.leaseId) False+              pure ()+          )+          ( \identity -> do+              files <- treeFiles (output </> "configurations")+              journals <- traverse (\path -> readSessionJournal (output </> "configurations" </> path) >>= require) [path | path <- files, [_, _, "trials", _, "session.json"] <- [splitDirectories path]]+              -- Fence first, then inspect markers. A marker arriving after+              -- this fence cannot start work under the old lease identity.+              _ <- Lease.requestCancel store (ref identity) (Just identity.leaseId) False+              settled <- ownedWorkSettled store (ref identity) identity journals+              unless settled do+                _ <- Lease.requestCancel store (ref identity) (Just identity.leaseId) False+                waitForOwnedWork store (ref identity) identity journals+              owned <- Lease.reattachLease store (ref identity) identity.leaseId+              case owned of+                Just acquired -> Lease.releaseLease store (ref identity) acquired+                Nothing -> do+                  history <- readPerfSession (output </> "session.json") >>= require+                  -- An absent ambiguous proposal must stay unresolved. A known+                  -- lost lease with terminal owned work needs no deletion.+                  pure (any ((== LeaseAcquired identity) . (.action)) history.events)+          )+          (recoverSealedRuntime store inputs prepared output)+          (describeReport output)+          (publishSnapshot store inputs prepared output)+  createDirectoryIfMissing True (output </> "configurations")+  auditAllAttempts True inputs prepared output+  result <- runPerformance hooks prepared (output </> "session.json")+  encodeFile (output </> "result.json") result+  TextIO.writeFile (output </> "report.md") (renderReport result)+  Lazy.putStrLn (encode result)+  pure (exit result.exitCode)++verifyConfiguration :: ExecutionInputs -> Prepared.PerfPrepared -> FilePath -> ConfigurationResult -> IO Bool+verifyConfiguration inputs prepared output result = do+  entry <- maybe (ioError (userError "result names an unselected profile")) pure (find ((== result.profile) . (.name) . (.profile)) prepared.plan.selected)+  auditProfileAttempts False inputs prepared output entry (Just result)++recoverSealedRuntime :: ObjectStore -> ExecutionInputs -> Prepared.PerfPrepared -> FilePath -> LeaseIdentity -> IO [(Text, Text)]+recoverSealedRuntime store inputs prepared output identity = fmap (List.nub . concat) (traverse profile prepared.plan.selected)+  where+    profile entry = do+      let base = output </> "configurations" </> Text.unpack entry.profile.name+      exists <- doesDirectoryExist base+      attempts <- if exists then listDirectory base else pure []+      fmap concat (traverse (attempt entry base) attempts)+    attempt entry base name = do+      let directory = base </> name+          schedulePath = directory </> "schedule.json"+      exists <- doesFileExist schedulePath+      if not exists+        then pure []+        else do+          schedule <- readJSON schedulePath >>= require . decodeWorkPlan+          fmap concat (traverse (trial entry directory) schedule.runs)+    trial entry directory run = do+      let trialDirectory = directory </> "trials" </> Text.unpack (renderRunId run.runId)+          path = trialDirectory </> "session.json"+      exists <- doesFileExist path+      if not exists+        then pure []+        else do+          retained <- readSessionJournal path >>= require+          if retained.leaseId /= identity.leaseId || retained.cell /= identity.cell+            then pure []+            else do+              observed <- resumeObservedSlices store (Lease.CellRef identity.cell (Bucket inputs.controlBucket)) (Bucket retained.resultsBucket) path (\event -> TextIO.hPutStrLn stderr (Text.pack (show event)))+              journal <- either (ioError . userError . show) pure observed+              reasons <- retainedRuntimeProblems entry.profile run trialDirectory journal+              pure [(entry.profile.name, reason) | reason <- reasons]++auditAllAttempts :: Bool -> ExecutionInputs -> Prepared.PerfPrepared -> FilePath -> IO ()+auditAllAttempts recover inputs prepared output = do+  names <- listDirectory (output </> "configurations")+  unless (all (`elem` map (Text.unpack . (.name) . (.profile)) prepared.plan.selected) names) (ioError (userError "retained evidence contains an unselected profile"))+  mapM_ (\entry -> auditProfileAttempts recover inputs prepared output entry Nothing >>= \valid -> unless valid (ioError (userError "retained attempt audit failed"))) prepared.plan.selected++-- Every retained attempt is checked, including abandoned and partially written+-- attempts. Only a complete matching attempt supplies the final comparison.+auditProfileAttempts :: Bool -> ExecutionInputs -> Prepared.PerfPrepared -> FilePath -> PlannedProfile -> Maybe ConfigurationResult -> IO Bool+auditProfileAttempts recover inputs prepared output entry wanted = do+  session <- readPerfSession (output </> "session.json") >>= require+  let (_, identities) = foldl' remember (Nothing, Map.empty) session.events+      base = output </> "configurations" </> Text.unpack entry.profile.name+      remember (lease, known) event = case event.action of+        LeaseAcquired identity -> (Just identity, known)+        LeaseLost _ -> (Nothing, known)+        LeaseReleased -> (Nothing, known)+        ComparisonStarted profile index -> (lease, maybe known (\identity -> Map.insert (profile, "attempt-" <> show index) identity known) lease)+        _ -> (lease, known)+  exists <- doesDirectoryExist base+  names <- if exists then listDirectory base else pure []+  baseline <- require (decode prepared.baseline.payload)+  candidate <- require (decode prepared.candidate.payload)+  let payloads = Map.fromList [("baseline", baseline), ("candidate", candidate)]+  candidates <-+    traverse+      ( \name -> do+          identity <- maybe (ioError (userError "retained attempt has no journaled lease identity")) pure (Map.lookup (entry.profile.name, name) identities)+          let directory = base </> name+          files <- treeFiles directory+          scheduleExists <- doesFileExist (directory </> "schedule.json")+          if not scheduleExists+            then do+              unless (all (== "schedule.json.next") files) (ioError (userError "retained attempt has evidence before its paired schedule"))+              pure False+            else do+              schedule <- readJSON (directory </> "schedule.json") >>= require . decodeWorkPlan+              expected <- makeSchedule prepared entry schedule.planId+              unless (toJSON schedule == toJSON expected) (ioError (userError "retained paired schedule differs from its pinned request and seed"))+              metadataExists <- doesFileExist (directory </> "attempt.json")+              if not metadataExists+                then do+                  unless (all (`elem` ["schedule.json", "attempt.json.next"]) files) (ioError (userError "retained attempt has evidence but no immutable attempt header"))+                  pure False+                else do+                  verifyAttemptInputs identity entry schedule directory+                  pinned <- maybe (ioError (userError "attempt cell is outside frozen pool")) pure (find ((== identity.cell) . (.name) . fst) inputs.cells)+                  pathsExists <- doesFileExist (directory </> "trial-paths.json")+                  paths <- if pathsExists then readJSON (directory </> "trial-paths.json") else pure Map.empty+                  failureExists <- doesFileExist (directory </> "trial-failures.json")+                  failures <- if failureExists then readJSON (directory </> "trial-failures.json") else pure (Map.empty :: Map.Map Text Text)+                  let journalPaths = [path | path <- files, ["trials", _, "session.json"] <- [splitDirectories path]]+                      expectedPaths = ["trials" </> Text.unpack (renderRunId run.runId) </> "session.json" | run <- schedule.runs]+                  unless (all (`elem` expectedPaths) journalPaths) (ioError (userError "retained cell journal names an unplanned trial"))+                  journals <- traverse (\path -> readSessionJournal (directory </> path) >>= require) journalPaths+                  let linkedDirectories = map (Text.unpack . renderRunId . (.runId)) schedule.runs <> [Text.unpack (renderRunId slice.cellRun) | journal <- journals, slice <- journal.slices]+                      attemptFiles = ["schedule.json", "attempt.json", "trial-paths.json", "trial-failures.json", "comparison.json", "result.json"]+                      interpreted path = case splitDirectories path of+                        [name] -> name `elem` attemptFiles || name `elem` map (<> ".next") attemptFiles+                        "trials" : identifier : _ -> identifier `elem` linkedDirectories+                        _ -> False+                  unless (all interpreted files) (ioError (userError "retained attempt contains unlinked evidence outside its scheduled trials and cell journals"))+                  mapM_+                    ( \run -> do+                        let identifier = renderRunId run.runId+                            trial = directory </> "trials" </> Text.unpack identifier+                            journalPath = trial </> "session.json"+                            planBytes = encode schedule {runs = [run], estimateMinutes = run.estimateMinutes}+                        journalExists <- doesFileExist journalPath+                        if not journalExists+                          then unless (Map.notMember identifier (paths :: Map.Map Text FilePath)) (ioError (userError "retained accepted trial has no cell journal"))+                          else do+                            journal <- readSessionJournal journalPath >>= require+                            unless (journal.leaseId == identity.leaseId && journal.cell == identity.cell && journal.controlBucket == inputs.controlBucket && journal.resultsBucket == (fst pinned).buckets.results && journal.payloads == payloads && journal.planSha256 == Text.drop 7 (sha256Hex (Lazy.toStrict planBytes))) (ioError (userError "retained cell journal differs from pinned payloads, trial or lease"))+                            storedPlan <- Lazy.readFile (trial </> "plan.json")+                            unless (storedPlan == planBytes) (ioError (userError "retained trial plan bytes differ"))+                            mapM_+                              ( \slice -> do+                                  work <- Lazy.readFile (trial </> slice.workPath)+                                  unless (slice.submission.leaseId == identity.leaseId && slice.submission.runId == slice.cellRun && slice.runIds == [run.runId] && slice.submission.work.sha256 == Text.drop 7 (sha256Hex (Lazy.toStrict work)) && slice.submission.work.bytes == Lazy.length work) (ioError (userError "retained submission work or identity differs"))+                              )+                              journal.slices+                            when (not recover && any (\slice -> slice.state `notElem` [SlicePlanned, SliceRejected, SliceVerified]) journal.slices) (ioError (userError "retained report contains unresolved submitted cell work"))+                            verified <- verifyCellTrial (Map.findWithDefault Null identity.cell inputs.healthPolicies) (fst pinned) run trial journal+                            runtimeProblems <- retainedRuntimeProblems entry.profile run trial journal+                            mapM_+                              ( \reason -> do+                                  current <- readPerfSession (output </> "session.json") >>= require+                                  state <- require (journalState current)+                                  unless (reason `elem` Map.findWithDefault [] entry.profile.name state.runtimeBlockers) do+                                    unless recover (ioError (userError "sealed runtime blocker is missing from the session"))+                                    now <- getCurrentTime+                                    changed <- require (appendEvent (SessionEvent now (RuntimeBlocked entry.profile.name reason)) current)+                                    writePerfSession (output </> "session.json") changed+                              )+                              runtimeProblems+                            case Map.lookup identifier paths of+                              Just path -> unless (verified == Right (directory </> path)) (ioError (userError "retained accepted trial outer evidence differs"))+                              Nothing -> when (Map.member identifier failures && either (const False) (const True) verified) (ioError (userError "retained failed trial has healthy completed outer evidence"))+                    )+                    schedule.runs+                  present <- doesFileExist (directory </> "result.json")+                  if not present+                    then pure False+                    else do+                      record <- readJSON (directory </> "result.json")+                      policy <- require (policyFor prepared entry)+                      replayed <- verifyAttempt entry schedule policy (axesFor prepared) directory record+                      unless replayed (ioError (userError "retained attempt result replay differs"))+                      comparisonExists <- doesFileExist (directory </> "comparison.json")+                      comparison <- if comparisonExists then Just <$> readJSON (directory </> "comparison.json") else pure Nothing+                      unless (comparison == record.comparison) (ioError (userError "retained ordinary comparison differs from attempt result"))+                      pure (maybe False (\result -> record.comparison == result.comparison && record.validPairs == result.validPairs && record.attempts == result.attempts && all (`elem` result.runtimeFailures) record.runtimeFailures) wanted)+      )+      names+  pure (maybe True (\result -> or candidates || (result.comparison == Nothing && result.validPairs == 0 && result.infrastructureState == "pending" && result.performanceVerdict == "inconclusive")) wanted)++describeReport :: FilePath -> PerfSession -> PerfResult -> IO PerfResult+describeReport output session result = do+  identifier <- newRunId+  now <- getCurrentTime+  let summaryPath = output </> "preparation/preparation-summary.json"+  summaryExists <- doesFileExist summaryPath+  preparationTimes <- if summaryExists then readJSON summaryPath >>= either (ioError . userError) pure . parseEither (withObject "PreparationSummary" (.: "stageSeconds")) else pure Map.empty+  let base = session.evidenceBaseUri <> "/sessions/" <> renderRunId session.sessionId <> "/snapshots/" <> renderRunId identifier+      leaseTimes = [event.at | event <- session.events, LeaseAcquired _ <- [event.action]]+      elapsed from to = realToFrac (diffUTCTime to from) :: Double+      timings =+        Map.fromList+          ( [("totalAtReportAssembly", elapsed session.createdAt now)] <> case leaseTimes of+              [] -> [("initialAdmission", elapsed session.createdAt now), ("measurementAndCleanupToAssembly", 0)]+              first : _ -> [("initialAdmission", elapsed session.createdAt first), ("measurementAndCleanupToAssembly", elapsed first now)]+          )+  files <- treeFiles (output </> "configurations")+  let relativeFiles = ["prepared.json", "request.json", "plan.json", "session.json"] <> ["configurations/" <> path | path <- files, "comparison.json" `isSuffixOf` path || "manifest.json" `isSuffixOf` path]+  references <-+    traverse+      ( \path -> do+          (digest, _) <- digestFile (output </> path)+          pure (ArtifactReference (if "manifest.json" `isSuffixOf` path then "cell-or-runtime-manifest" else if "comparison.json" `isSuffixOf` path then "comparison" else Text.pack path) (base <> "/" <> Text.pack path) (Just ("sha256:" <> digest)))+      )+      relativeFiles+  pure result {stageSeconds = timings <> Map.mapKeys ("preparation." <>) preparationTimes, artifactReferences = ArtifactReference "aggregate-manifest" (base <> "/manifest.json") Nothing : ArtifactReference "publication-receipt" (base <> "/receipt.json") Nothing : references}++treeFiles :: FilePath -> IO [FilePath]+treeFiles root = do+  names <- listDirectory root+  fmap+    concat+    ( traverse+        ( \name -> do+            let path = root </> name+            linked <- pathIsSymbolicLink path+            when linked (ioError (userError "performance evidence contains a symbolic link"))+            directory <- doesDirectoryExist path+            if directory then map (name </>) <$> treeFiles path else pure [name]+        )+        names+    )++publishSnapshot :: ObjectStore -> ExecutionInputs -> Prepared.PerfPrepared -> FilePath -> PerfSession -> PerfResult -> IO Bool+publishSnapshot store inputs prepared output session result = do+  auditAllAttempts False inputs prepared output+  publicationStarted <- getCurrentTime+  manifestReference <- case filter ((== "aggregate-manifest") . (.kind)) result.artifactReferences of+    [reference] -> pure reference.uri+    _ -> ioError (userError "report has no unique aggregate manifest reference")+  relative <- maybe (ioError (userError "aggregate reference is outside its pinned session")) pure (Text.stripPrefix (session.evidenceBaseUri <> "/sessions/" <> renderRunId session.sessionId <> "/") manifestReference >>= Text.stripSuffix "/manifest.json")+  let directory = output </> Text.unpack relative+  createDirectoryIfMissing True directory+  mapM_ (\name -> copyTree (output </> name) (directory </> name)) ["preparation", "configurations"]+  mapM_ (\name -> copyFile (output </> name) (directory </> name)) ["prepared.json", "request.json", "plan.json", "session.json"]+  encodeFile (directory </> "result.json") result+  TextIO.writeFile (directory </> "report.md") (renderReport result)+  sealPerfDirectory directory >>= require+  (bucket, prefix) <- require (parseDestination session.evidenceBaseUri)+  published <- publishEvidence store bucket (prefix <> "/sessions/" <> renderRunId session.sessionId <> "/" <> relative) directory+  case published of+    Left problem -> TextIO.hPutStrLn stderr problem >> pure False+    Right uri -> do+      manifest <- Bytes.readFile (directory </> "manifest.json")+      verified <- getCurrentTime+      state <- require (journalState session)+      unless (maybe True (verified <=) state.executionDeadline) (ioError (userError "publication verification exceeded the original execution deadline"))+      let receipt = PublicationReceipt relative uri (sha256Hex manifest) (sha256Hex (canonicalEncode (toJSON result))) session.createdAt publicationStarted verified (realToFrac (diffUTCTime verified session.createdAt))+          bytes = encode receipt+          name = ObjectName (prefix <> "/sessions/" <> renderRunId session.sessionId <> "/" <> relative <> "/receipt.json")+      _ <- store.putObject bucket name "application/json" DoesNotExist bytes+      retained <- store.getObject bucket name+      unless (fmap fst retained == Just bytes) (ioError (userError "durable publication receipt retrieval differs"))+      receiptRetrievedAt <- getCurrentTime+      unless (maybe True (receiptRetrievedAt <=) state.executionDeadline) (ioError (userError "receipt retrieval exceeded the original execution deadline"))+      atomicBytes (output </> "receipt.json") bytes+      pure True++-- An interruption after comparison publication but before the root checkpoint+-- must preserve that comparison's original identity and bytes on reanalysis.+retainAttemptResult :: FilePath -> ConfigurationResult -> IO ConfigurationResult+retainAttemptResult directory computed = do+  let comparisonPath = directory </> "comparison.json"+      resultPath = directory </> "result.json"+  comparisonExists <- doesFileExist comparisonPath+  comparison <- case (comparisonExists, computed.comparison) of+    (False, Nothing) -> pure Nothing+    (True, Nothing) -> ioError (userError "retained comparison no longer matches completed trial evidence")+    (False, Just document) -> atomicBytes comparisonPath (encode document) >> pure (Just document)+    (True, Just document) -> do+      original <- readJSON comparisonPath+      retained <- require (semanticComparison original)+      replayed <- require (semanticComparison document)+      unless (retained == replayed) (ioError (userError "retained comparison differs from interrupted attempt replay"))+      pure (Just original)+  let result = computed {comparison = comparison}+  exists <- doesFileExist resultPath+  if exists+    then do+      original <- readJSON resultPath+      unless (original == result) (ioError (userError "retained attempt result differs from interrupted replay"))+      pure original+    else atomicBytes resultPath (encode result) >> pure result++atomicBytes :: FilePath -> Lazy.ByteString -> IO ()+atomicBytes path bytes = Lazy.writeFile (path <> ".next") bytes >> renameFile (path <> ".next") path++runCalibrationPublish :: Bool -> FilePath -> Text -> CliEnv -> IO ExitCode+runCalibrationPublish definition path uri _ = guarded do+  bytes <- Lazy.readFile path+  (bucket, name) <- input (parseDestination uri)+  now <- getCurrentTime+  if definition+    then do+      document <- either (throwIO . SessionFailure 2 . Text.pack) pure (eitherDecode bytes :: Either String Calibration.CalibrationDefinition)+      input (Calibration.validateCalibrationDefinition now now document)+    else do+      document <- either (throwIO . SessionFailure 2 . Text.pack) pure (eitherDecode bytes :: Either String Calibration.CalibrationAcceptance)+      unless (length document.aa == 20 && length document.slowdown == 20 && all (not . null . (.receipts)) (document.aa <> document.slowdown)) (throwIO (SessionFailure 2 "acceptance inventory needs both twenty-slot experiments"))+  (store, _) <- openStore bucket+  _ <- store.putObject bucket (ObjectName name) "application/json" DoesNotExist bytes+  retained <- store.getObject bucket (ObjectName name)+  (retrieved, metadata) <- maybe (ioError (userError "calibration publication retrieval failed")) pure retained+  unless (retrieved == bytes) (ioError (userError "create-only calibration publication differs"))+  Lazy.putStrLn (encode (object ["uri" .= uri, "sha256" .= sha256Hex (Lazy.toStrict bytes), "declaredAt" .= metadata.updated]))+  pure ExitSuccess++runCalibrationVerify :: FilePath -> FilePath -> CliEnv -> IO ExitCode+runCalibrationVerify referencePath preparedPath environment = guarded do+  prepared <- loadPreparation environment.perf preparedPath+  reference <- readJSON referencePath :: IO Calibration.CalibrationReference+  entry <- case prepared.plan.selected of [selected] -> pure selected; _ -> throwIO (SessionFailure 2 "calibration verification needs one prepared profile")+  (semantics, policies) <- require (either (Left . Text.pack) Right (parseEither (withObject "PinnedInputs" \v -> (,) <$> v .: "harnessSemanticsSha256" <*> v .: "policies") prepared.inputs))+  policy <- maybe (ioError (userError "missing pinned calibration policy")) pure (Map.lookup entry.profile.policy (policies :: Map.Map FilePath Value))+  digest <- verifyCalibrationAdmission environment.perf entry {PerfPlan.profile = entry.profile {calibration = Just (toJSON reference)}} (Calibration.toolchainDigest prepared.baseline.buildInventory) semantics policy >>= require+  Lazy.putStrLn (encode (object ["verified" .= True, "profile" .= entry.profile.name, "acceptanceSha256" .= digest, "gatingEligible" .= False]))+  pure ExitSuccess++verifyCalibrationAdmission :: PerfContracts -> PlannedProfile -> Text -> Text -> Value -> IO (Either Text Text)+verifyCalibrationAdmission contracts entry expectedToolchain semantics policyValue = do+  outcome <- try @IOException do+    reference <- require (either (Left . Text.pack) Right (parseEither (parseJSON @Calibration.CalibrationReference) (fromMaybe Null entry.profile.calibration)))+    (definitionBytes, declared) <- readPinned reference.definitionUri reference.definitionSha256+    (acceptanceBytes, _) <- readPinned reference.acceptanceUri reference.acceptanceSha256+    definition <- either (ioError . userError) pure (eitherDecode definitionBytes :: Either String Calibration.CalibrationDefinition)+    acceptance <- either (ioError . userError) pure (eitherDecode acceptanceBytes :: Either String Calibration.CalibrationAcceptance)+    unless ((definition.profile, definition.machineProfile, definition.workloadSha256, definition.toolchainSha256, definition.harnessSemanticsSha256) == (entry.profile.name, entry.profile.machineProfile, entry.workloadSha256, expectedToolchain, semantics)) (ioError (userError "calibration definition differs from the current workload, toolchain or harness"))+    unless (not ("pending" `Text.isInfixOf` entry.profile.machineProfile)) (ioError (userError "calibration requires an actual designated machine profile"))+    policy <- require (either (Left . Text.pack) Right (parseEither parseJSON policyValue))+    rule <- case filter (\rule -> matchesRule rule.match definition.targetMetric) (policy :: Policy).metricRules of+      [metricRule] -> pure metricRule+      _ -> ioError (userError "injected metric needs one unambiguous policy rule")+    unless (definition.targetMetric `elem` map (.name) entry.profile.requiredMetrics && rule.direction == LowerIsBetter && definition.targetRelativeEffect > rule.relativeLimit && definition.targetAbsoluteEffect > rule.absoluteFloor) (ioError (userError "injected latency effect must exceed both frozen policy thresholds"))+    aa <- traverse (traverse (replayCalibrationReceipt contracts) . (.receipts)) acceptance.aa+    slowdown <- traverse (traverse (replayCalibrationReceipt contracts) . (.receipts)) acceptance.slowdown+    now <- getCurrentTime+    _ <- require (Calibration.validateCalibration now declared definition acceptance aa slowdown)+    pure reference.acceptanceSha256+  pure (either (Left . Text.pack . displayException) Right outcome)+  where+    readPinned uri digest = do+      (bucket, name) <- require (parseDestination uri)+      (store, _) <- openStore bucket+      retained <- store.getObject bucket (ObjectName name)+      (bytes, metadata) <- maybe (ioError (userError "calibration record is missing")) pure retained+      unless (sha256Hex (Lazy.toStrict bytes) == digest) (ioError (userError "calibration record digest differs"))+      pure (bytes, metadata.updated)++-- A change in the owner's health thresholds invalidates the batch even when+-- hardware and image names stayed the same. Check only compatible pool cells.+checkCalibratedPolicies :: Prepared.PerfPrepared -> ExecutionInputs -> IO ()+checkCalibratedPolicies prepared inputs = when prepared.plan.gatingEligible do+  mapM_+    ( \entry -> do+        definition <- calibrationDefinition entry+        let compatible = [descriptor | (descriptor, _) <- inputs.cells, cellMachineProfile descriptor == entry.profile.machineProfile]+        unless (not (null compatible) && all (\descriptor -> ((sha256Hex . canonicalEncode) <$> Map.lookup descriptor.name inputs.healthPolicies) == Just definition.healthPolicySha256) compatible) (ioError (userError "cell health policy differs from the calibrated policy"))+    )+    prepared.plan.selected++calibrationDefinition :: PlannedProfile -> IO Calibration.CalibrationDefinition+calibrationDefinition entry = do+  reference <- require (either (Left . Text.pack) Right (parseEither (parseJSON @Calibration.CalibrationReference) (fromMaybe Null entry.profile.calibration)))+  (bucket, name) <- require (parseDestination reference.definitionUri)+  (store, _) <- openStore bucket+  retained <- store.getObject bucket (ObjectName name)+  bytes <- maybe (ioError (userError "calibration declaration is missing")) (pure . fst) retained+  unless (sha256Hex (Lazy.toStrict bytes) == reference.definitionSha256) (ioError (userError "calibration declaration digest differs"))+  either (ioError . userError) pure (eitherDecode bytes)++verifyGatingCalibration :: PerfContracts -> Prepared.PerfPrepared -> IO ()+verifyGatingCalibration contracts prepared = when prepared.plan.gatingEligible do+  (semantics, policies, admissions) <- require (either (Left . Text.pack) Right (parseEither (withObject "PinnedInputs" \v -> (,,) <$> v .: "harnessSemanticsSha256" <*> v .: "policies" <*> v .: "calibrationAdmissions") prepared.inputs))+  mapM_+    ( \entry -> do+        policy <- maybe (ioError (userError "pinned calibration policy is absent")) pure (Map.lookup entry.profile.policy (policies :: Map.Map FilePath Value))+        evidence <- verifyCalibrationAdmission contracts entry (Calibration.toolchainDigest prepared.baseline.buildInventory) semantics policy >>= require+        unless (Map.lookup entry.workloadSha256 (admissions :: Map.Map Text Text) == Just evidence) (ioError (userError "pinned calibration admission differs"))+    )+    prepared.plan.selected++replayCalibrationReceipt :: PerfContracts -> Calibration.CalibrationReceipt -> IO Calibration.VerifiedCalibrationTrial+replayCalibrationReceipt contracts reference = do+  (bucket, objectName) <- require (parseDestination reference.uri)+  prefix <- maybe (ioError (userError "calibration must reference a publication receipt")) pure (Text.stripSuffix "/receipt.json" objectName)+  (store, _) <- openStore bucket+  retained <- store.getObject bucket (ObjectName objectName)+  bytes <- maybe (ioError (userError "calibration receipt is missing")) (pure . fst) retained+  unless (sha256Hex (Lazy.toStrict bytes) == reference.sha256) (ioError (userError "calibration receipt digest differs"))+  receipt <- either (ioError . userError) pure (eitherDecode bytes :: Either String PublicationReceipt)+  unless (receipt.manifestUri == "gs://" <> bucket.unBucket <> "/" <> prefix <> "/manifest.json") (ioError (userError "calibration receipt destination differs"))+  unless ("snapshots/" `Text.isPrefixOf` receipt.snapshot && Text.count "/" receipt.snapshot == 1 && Text.all (\c -> c `elem` (['a' .. 'z'] <> ['0' .. '9'] <> ['-', '/'])) receipt.snapshot) (ioError (userError "unsafe calibration snapshot"))+  withSystemTempDirectory "kenshou-calibration-replay-" \root -> do+    let directory = root </> Text.unpack receipt.snapshot+    retrieveEvidence store bucket prefix receipt.manifestSha256 directory >>= require+    Lazy.writeFile (root </> "receipt.json") bytes+    (result, _) <- verifySessionEvidence contracts root True+    prepared <- loadPreparation contracts (directory </> "preparation/prepared.json")+    session <- readPerfSession (directory </> "session.json") >>= require+    unless (result.purpose == "calibration" && not result.gatingEligible && length result.configurations == 1 && length prepared.plan.selected == 1) (ioError (userError "calibration needs a single-profile non-gating report"))+    configuration <- case result.configurations of [retainedConfiguration] -> pure retainedConfiguration; _ -> ioError (userError "missing calibration configuration")+    semantics <- require (either (Left . Text.pack) Right (parseEither (withObject "PinnedInputs" (.: "harnessSemanticsSha256")) prepared.inputs))+    inputs <- require (parseInputs session)+    entry <- case prepared.plan.selected of [selected] -> pure selected; _ -> ioError (userError "missing calibration profile")+    let leasedCells = [identity.cell | event <- session.events, LeaseAcquired identity <- [event.action]]+    actualProfiles <-+      traverse+        (\cell -> maybe (ioError (userError "calibration lease is outside the frozen pool")) (pure . cellMachineProfile . fst) (find ((== cell) . (.name) . fst) inputs.cells))+        leasedCells+    unless (all (== entry.profile.machineProfile) actualProfiles && (result.infrastructureState /= "healthy" || not (null actualProfiles))) (ioError (userError "calibration ran on a different machine or image profile"))+    policies <- traverse (\cell -> maybe (ioError (userError "calibration health policy is absent")) pure (Map.lookup cell inputs.healthPolicies)) leasedCells+    policy <- case policies of first : rest | all (== first) rest -> pure first; [] | result.infrastructureState /= "healthy" -> maybe (ioError (userError "calibration health policy is absent")) pure (Map.lookup (case prepared.plan.request.cells of cell : _ -> cell; [] -> "") inputs.healthPolicies); _ -> ioError (userError "calibration health policy changed across leases")+    pure (Calibration.VerifiedCalibrationTrial session.sessionId prepared.plan.request.seed prepared.plan.request.control prepared.plan.request.project prepared.plan.request.baselineRevision prepared.plan.request.candidateRevision configuration.profile entry.profile.machineProfile (sha256Hex (canonicalEncode policy)) configuration.workloadSha256 (Calibration.toolchainDigest prepared.baseline.buildInventory) semantics configuration.performanceVerdict result.infrastructureState result.blockingRuntimeFailures session.createdAt receipt.totalElapsedSeconds)++runVerify :: FilePath -> Bool -> CliEnv -> IO ExitCode+runVerify output offline environment = guarded do+  (result, uri) <- verifySessionEvidence environment.perf output offline+  Lazy.putStrLn (encode (object ["verified" .= True, "offline" .= offline, "manifestUri" .= uri, "result" .= result]))+  pure (exit result.exitCode)++verifySessionEvidence :: PerfContracts -> FilePath -> Bool -> IO (PerfResult, Text)+verifySessionEvidence contracts output offline = do+  receipt <- readJSON (output </> "receipt.json")+  let relative = (receipt :: PublicationReceipt).snapshot+      uri = receipt.manifestUri+      manifestDigest = receipt.manifestSha256+      resultDigest = receipt.resultSha256+  unless ("snapshots/" `Text.isPrefixOf` relative && Text.all (\c -> c `elem` (['a' .. 'z'] <> ['0' .. '9'] <> ['-', '/'])) relative && Text.count "/" relative == 1) (ioError (userError "unsafe retained snapshot"))+  let directory = output </> Text.unpack relative+  verifyPerfManifest directory >>= require+  manifest <- Bytes.readFile (directory </> "manifest.json")+  unless (sha256Hex manifest == manifestDigest) (ioError (userError "retained manifest differs from durable receipt"))+  prepared <- loadPreparation contracts (directory </> "preparation/prepared.json")+  checkRootInputs directory prepared+  session <- readPerfSession (directory </> "session.json") >>= require+  require (validateSessionInputs contracts prepared session)+  unless offline (verifyGatingCalibration contracts prepared)+  unless (receipt.payloadsReadyAt == session.createdAt) (ioError (userError "receipt timing origin differs from its pinned session"))+  unless (uri == session.evidenceBaseUri <> "/sessions/" <> renderRunId session.sessionId <> "/" <> relative <> "/manifest.json") (ioError (userError "receipt destination differs from its pinned session"))+  inputs <- require (parseInputs session)+  unless offline (checkCalibratedPolicies prepared inputs)+  result <- readJSON (directory </> "result.json")+  unless (sha256Hex (canonicalEncode (toJSON (result :: PerfResult))) == resultDigest) (ioError (userError "retained aggregate differs from durable receipt"))+  unless (maybe False (<= receipt.totalElapsedSeconds) (Map.lookup "totalAtReportAssembly" result.stageSeconds)) (ioError (userError "report timing exceeds its verified publication time"))+  let base = Text.dropEnd (Text.length ("/manifest.json" :: Text)) uri+  unless ([reference.uri | reference <- result.artifactReferences, reference.kind == "aggregate-manifest"] == [uri] && [reference.uri | reference <- result.artifactReferences, reference.kind == "publication-receipt"] == [base <> "/receipt.json"]) (ioError (userError "report does not reference its own publication and receipt"))+  mapM_+    ( \reference -> case reference.sha256 of+        Nothing -> unless (reference.kind `elem` ["aggregate-manifest", "publication-receipt"]) (ioError (userError "artifact reference has no digest"))+        Just expected -> do+          path <- maybe (ioError (userError "artifact reference is outside the sealed snapshot")) pure (Text.stripPrefix (base <> "/") reference.uri)+          unless (all (\part -> not (Text.null part) && part `notElem` [".", ".."]) (Text.splitOn "/" path) && not (Text.any (`elem` ['\\', '\n', '\r']) path)) (ioError (userError "unsafe artifact reference"))+          (actual, _) <- digestFile (directory </> Text.unpack path)+          unless (expected == "sha256:" <> actual) (ioError (userError "artifact reference digest differs"))+    )+    result.artifactReferences+  state <- require (journalState session)+  auditAllAttempts False inputs prepared directory+  unless (maybe True (receipt.verifiedReportAt <=) state.executionDeadline) (ioError (userError "publication receipt exceeds the original execution deadline"))+  let completed = Map.elems state.completed+  valid <- traverse (verifyConfiguration inputs prepared directory) completed+  unless (and valid) (ioError (userError "retained comparison replay differs"))+  recomputed <- require (aggregateResult session.plan session.preparedSha256 result.evidenceVerified result.configurations)+  unless (recomputed == result {stageSeconds = Map.empty, artifactReferences = []} && all (\entry -> any (\retained -> retained.profile == entry.profile && retained.comparison == entry.comparison && retained.runtimeFailures == entry.runtimeFailures) result.configurations) completed) (ioError (userError "aggregate differs from the pinned session results"))+  unless offline do+    destination <- maybe (ioError (userError "receipt has no manifest URI")) pure (Text.stripSuffix "/manifest.json" uri)+    (bucket, prefix) <- require (parseDestination destination)+    (store, _) <- openStore bucket+    verifyPublication store bucket prefix directory >>= require+    stored <- store.getObject bucket (ObjectName (prefix <> "/receipt.json"))+    bytes <- Lazy.readFile (output </> "receipt.json")+    unless (fmap fst stored == Just bytes) (ioError (userError "retained publication receipt differs from durable storage"))+  pure (result, uri)++accessProbe :: ObjectStore -> Text -> RunId -> Prepared.PerfPrepared -> IO ()+accessProbe store uri identifier prepared = do+  (bucket, prefix) <- require (parseDestination uri)+  let name = ObjectName (prefix <> "/access-checks/" <> renderRunId identifier <> ".json")+      bytes = encode (object ["session" .= identifier, "preparedSha256" .= Prepared.preparedDigest prepared])+  outcome <- store.putObject bucket name "application/json" DoesNotExist bytes+  unless (case outcome of Written _ -> True; _ -> False) (ioError (userError "performance evidence destination failed create-only access check"))+  found <- store.getObject bucket name+  unless (fmap fst found == Just bytes) (ioError (userError "performance evidence destination failed retrieval access check"))++parseDestination :: Text -> Either Text (Bucket, Text)+parseDestination uri = do+  suffix <- maybe (Left "evidence destination must be gs://BUCKET/PREFIX") Right (Text.stripPrefix "gs://" uri)+  let (bucket, path) = Text.breakOn "/" suffix+      prefix = Text.drop 1 path+  unless (not (Text.null bucket) && Text.all (\c -> c `elem` ['a' .. 'z'] <> ['0' .. '9'] <> ['-', '.']) bucket && not (Text.null prefix) && all (\part -> not (Text.null part) && part `notElem` [".", ".."]) (Text.splitOn "/" prefix) && not (Text.any (`elem` ['\\', '\n', '\r', ' ', '?', '#']) prefix)) (Left "unsafe evidence destination")+  pure (Bucket bucket, prefix)++copyTree :: FilePath -> FilePath -> IO ()+copyTree source destination = do+  link <- pathIsSymbolicLink source+  when link (ioError (userError "performance artifacts may not contain symbolic links"))+  directory <- doesDirectoryExist source+  if directory+    then do+      createDirectoryIfMissing True destination+      names <- listDirectory source+      mapM_ (\name -> copyTree (source </> name) (destination </> name)) names+    else copyFile source destination++renderReport :: PerfResult -> Text+renderReport result =+  Text.unlines+    ( ["# Runtime performance validation", "", "Performance: " <> result.performanceVerdict, "Coverage: " <> result.coverage <> " (whole change: " <> result.changeCoverage <> ")", "Infrastructure: " <> result.infrastructureState, "Purpose: " <> result.purpose, "Gating eligible: " <> Text.pack (show result.gatingEligible), "Exit: " <> Text.pack (show result.exitCode), "", "| Profile | Verdict | Valid pairs | Infrastructure |", "| --- | --- | --- | --- |"]+        <> ["| " <> item.profile <> " | " <> item.performanceVerdict <> " | " <> Text.pack (show item.validPairs) <> "/" <> Text.pack (show item.requiredPairs) <> " | " <> item.infrastructureState <> " |" | item <- result.configurations]+        <> concatMap effects result.configurations+        <> ["", "Runtime blockers: " <> if null result.blockingRuntimeFailures then "none" else Text.intercalate "; " result.blockingRuntimeFailures]+        <> ["", "## Scope", ""]+        <> ["- Excluded " <> scenario <> ": " <> reason | (scenario, reason) <- result.exclusions]+        <> ["- Uncovered: " <> scenario | scenario <- result.uncoveredScenarios]+        <> ["", "## Timings", "", "| Stage | Seconds |", "| --- | ---: |"]+        <> ["| " <> stage <> " | " <> number seconds <> " |" | (stage, seconds) <- Map.toAscList result.stageSeconds]+        <> ["", "Full report retrieval time is recorded in the linked publication receipt.", "", "## Evidence", ""]+        <> ["- [" <> reference.kind <> "](" <> reference.uri <> ")" <> maybe "" (\digest -> " (`" <> digest <> "`)") reference.sha256 | reference <- result.artifactReferences]+    )+  where+    number amount = Text.pack (printf "%.4g" (amount :: Double))+    effects configuration = case configuration.comparison of+      Nothing -> []+      Just document -> case parseEither metrics document of+        Left problem -> ["", "Retained metric report could not be rendered: " <> Text.pack problem]+        Right (rows, reasons) -> ["", "## " <> configuration.profile, "", "Intervals and thresholds below are retained comparison results.", "", "| Metric | Unit | Ratio estimate [low, high] | Delta estimate [low, high] | Relative limit | Absolute floor | Status | Explanation |", "| --- | --- | --- | --- | ---: | ---: | --- | --- |"] <> rows <> ["", "Comparison reasons: " <> if null reasons then "none" else Text.intercalate "; " reasons]+    metrics = withObject "Comparison" \document -> do+      values <- document .: "metrics"+      reasons <- document .:? "reasons" .!= []+      rows <- traverse metric (Map.toAscList (values :: Map.Map Text Value))+      pure (rows, reasons)+    metric (name, document) =+      withObject+        "MetricComparison"+        ( \fields -> do+            unit <- fields .: "unit"+            ratio <- fields .: "ratio" >>= interval+            delta <- fields .: "delta" >>= interval+            limit <- fields .: "relativeLimit"+            floorValue <- fields .: "absoluteFloor"+            status <- fields .: "status"+            message <- fields .: "message"+            pure ("| " <> name <> " | " <> unit <> " | " <> ratio <> " | " <> delta <> " | " <> number limit <> " | " <> number floorValue <> " | " <> status <> " | " <> Text.replace "\n" " " (Text.replace "|" "\\|" message) <> " |")+        )+        document+    interval = withObject "Interval" \fields -> do+      low <- fields .: "low"+      estimate <- fields .: "estimate"+      high <- fields .: "high"+      pure (number estimate <> " [" <> number low <> ", " <> number high <> "]")++openStore :: Bucket -> IO (ObjectStore, Text)+openStore bucket = do+  selected <- lookupEnv "KENSHOU_CELL_STORE"+  case selected of+    Just setting -> do+      path <- maybe (ioError (userError "KENSHOU_CELL_STORE must be file:<dir> or unset for GCS")) (pure . Text.unpack) (Text.stripPrefix "file:" (Text.pack setting))+      when (null path) (ioError (userError "file-backed cell store directory is empty"))+      absolute <- makeAbsolute path+      store <- newFileStore absolute+      pure (store, "file:" <> Text.pack absolute)+    Nothing -> do+      tokens <- newTokenProvider bucket+      store <- newGcsStore tokens+      pure (store, "gcs")++readJSON :: (FromJSON a) => FilePath -> IO a+readJSON path = eitherDecodeFileStrict' path >>= either (ioError . userError . ((path <> ": ") <>)) pure++decode :: (FromJSON a) => Value -> Either Text a+decode document = case fromJSON document of Error problem -> Left (Text.pack problem); Success result -> Right result++require :: Either Text a -> IO a+require = either (ioError . userError . Text.unpack) pure++input :: Either Text a -> IO a+input = either (throwIO . SessionFailure 2) pure++exit :: Int -> ExitCode+exit 0 = ExitSuccess+exit code = ExitFailure code++guarded :: IO ExitCode -> IO ExitCode+guarded action = catch executeFailure (\(SessionFailure code problem) -> TextIO.hPutStrLn stderr problem >> pure (exit code))+  where+    executeFailure = do+      result <- try @IOException action+      case result of+        Right code -> pure code+        Left problem -> do+          TextIO.hPutStrLn stderr (Text.pack (displayException problem))+          pure (ExitFailure 4)
+ src/Kenshou/Cli/Command/Plan.hs view
@@ -0,0 +1,345 @@+{-# LANGUAGE FieldSelectors #-}++module Kenshou.Cli.Command.Plan (planCommand) where++import Data.Aeson qualified as Aeson+import Data.Bifunctor (first)+import Data.ByteString.Lazy.Char8 qualified as LazyByteString+import Data.List (nubBy, partition)+import Data.List.NonEmpty qualified as NonEmpty+import Data.Set qualified as Set+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Text.IO qualified as Text.IO+import Kenshou.Cli.Cohort (resolveDefaultCohortIdentity)+import Kenshou.Core.Bundle (allScenarios)+import Kenshou.Core.Cli+import Kenshou.Core.Cohort qualified as Cohort+import Kenshou.Core.Id qualified as Id+import Kenshou.Core.Knob (KnobSpec (..), renderKnobName)+import Kenshou.Core.RunSpec (SpecPlacement (..))+import Kenshou.Core.Scenario (Tier (..))+import Kenshou.Plan.Catalog (ScenarioInfo (..), decodeCatalog, fromScenario)+import Kenshou.Plan.Change+import Kenshou.Plan.Change.Cohort+import Kenshou.Plan.Change.Git+import Kenshou.Plan.Components+import Kenshou.Plan.Components qualified as Components+import Kenshou.Plan.Components.Check+import Kenshou.Plan.Policy+import Kenshou.Plan.RunPlan+import Kenshou.Plan.Selector (Selector, parseSelector, renderSelector)+import Kenshou.Plan.Suite+import Options.Applicative hiding (value)+import System.Exit (ExitCode (..))+import System.IO (stderr)+import System.Random (randomIO)++data PlanOptions = PlanOptions+  { graphShow :: Bool,+    graphCheck :: Bool,+    json :: Bool,+    graphSource :: Maybe InputSource,+    catalogSource :: Maybe InputSource,+    suiteName :: Maybe Text,+    suiteSource :: Maybe InputSource,+    changed :: [Text],+    cohortFrom :: Maybe Text,+    cohortTo :: Maybe Text,+    since :: Maybe Text,+    upstreamDiffs :: [Text],+    allScenariosFlag :: Bool,+    includes :: [Text],+    excludes :: [Text],+    explain :: Bool,+    outputPath :: Maybe FilePath,+    maxTierText :: Maybe Text,+    kindTexts :: [Text],+    placementText :: Maybe Text,+    dimensionPolicyText :: Maybe Text,+    knobPolicyText :: Maybe Text,+    trialCount :: Maybe Int,+    budget :: Maybe Int,+    seedValue :: Maybe Integer,+    knobPins :: [Text],+    dimensionPins :: [Text]+  }++planCommand :: CliCommand+planCommand =+  CliCommand+    { name = "plan",+      description = "Select and plan verification runs",+      group = Discovery,+      hidden = False,+      parser = runPlan <$> planParser+    }++planParser :: Parser PlanOptions+planParser =+  PlanOptions+    <$> switch (long "graph-show" <> help "Print the checked-in component graph")+    <*> switch (long "graph-check" <> help "Check graph validity, build edges, and selectors")+    <*> switch (long "json" <> help "Write machine-readable JSON")+    <*> optional (option (parseInputSource <$> str) (long "graph" <> metavar "FILE" <> help "Override the component graph; use - for stdin"))+    <*> optional (option (parseInputSource <$> str) (long "catalog" <> metavar "FILE" <> help "Plan against a scenario catalog; use - for stdin"))+    <*> optional (Text.pack <$> strOption (long "suite" <> metavar "NAME" <> help "Apply suites/NAME.json"))+    <*> optional (option (parseInputSource <$> str) (long "suite-file" <> metavar "FILE" <> help "Apply a suite document; use - for stdin"))+    <*> many (Text.pack <$> strOption (long "changed" <> metavar "COMPONENT[,COMPONENT]" <> help "Name changed components or sub-components"))+    <*> optional (Text.pack <$> strOption (long "cohort-from" <> metavar "COHORT" <> help "Baseline cohort name, file, or git object"))+    <*> optional (Text.pack <$> strOption (long "cohort-to" <> metavar "COHORT" <> help "Candidate cohort name, file, or git object"))+    <*> optional (Text.pack <$> strOption (long "since" <> metavar "REV" <> help "Map repository changes since a revision"))+    <*> many (Text.pack <$> strOption (long "upstream-diff" <> metavar "REPO=PATH@REVA..REVB" <> help "Map paths changed in an upstream checkout"))+    <*> switch (long "all" <> help "Select every catalog scenario")+    <*> many (Text.pack <$> strOption (long "select" <> metavar "SELECTOR" <> help "Keep matching scenarios"))+    <*> many (Text.pack <$> strOption (long "exclude" <> metavar "SELECTOR" <> help "Remove matching scenarios"))+    <*> switch (long "explain" <> help "Explain changes, dependency paths, and selections")+    <*> optional (strOption (long "out" <> metavar "FILE" <> help "Write the run plan to a file"))+    <*> optional (Text.pack <$> strOption (long "max-tier" <> metavar "TIER" <> help "Maximum cost tier"))+    <*> many (Text.pack <$> strOption (long "kind" <> metavar "KIND" <> help "Include a scenario kind; repeatable"))+    <*> optional (Text.pack <$> strOption (long "placement" <> metavar "local|cell" <> help "Target placement"))+    <*> optional (Text.pack <$> strOption (long "dimension-policy" <> metavar "POLICY" <> help "default-only, telemetry-corners, pairwise, or full"))+    <*> optional (Text.pack <$> strOption (long "knob-policy" <> metavar "POLICY" <> help "defaults or declared-variants"))+    <*> optional (option auto (long "trials" <> metavar "N" <> help "Benchmark trials (minimum 3)"))+    <*> optional (option auto (long "budget-minutes" <> metavar "N" <> help "Maximum estimated run time"))+    <*> optional (option auto (long "seed" <> metavar "N" <> help "Deterministic plan seed"))+    <*> many (Text.pack <$> strOption (long "set" <> metavar "NAME=VALUE" <> help "Pin a knob value"))+    <*> many (Text.pack <$> strOption (long "dim" <> metavar "NAME=VALUE" <> help "Pin a dimension value"))++runPlan :: PlanOptions -> CliEnv -> IO ExitCode+runPlan options environment+  | stdinCount options > 1 = usage "at most one document input may use standard input"+  | options.graphShow && options.graphCheck = usage "choose only one of --graph-show or --graph-check"+  | options.suiteName /= Nothing && options.suiteSource /= Nothing = usage "choose only one of --suite or --suite-file"+  | otherwise = do+      graphResult <- loadGraph environment options.graphSource+      catalogResult <- loadCatalog environment options.catalogSource+      suiteResult <- loadSuite options+      case (graphResult, catalogResult, suiteResult) of+        (Left err, _, _) -> usage err+        (_, Left err, _) -> usage err+        (_, _, Left err) -> usage err+        (Right graph, Right catalog, Right suite)+          | options.graphShow -> showGraph options.json graph+          | options.graphCheck -> checkGraph graph catalog+          | otherwise -> planSelection options suite graph catalog++planSelection :: PlanOptions -> Maybe Suite -> ComponentGraph -> [ScenarioInfo] -> IO ExitCode+planSelection options suite graph catalog = case (traverse (parseSelector) options.includes, traverse parseSelector options.excludes) of+  (Left err, _) -> usage err+  (_, Left err) -> usage err+  (Right includes, Right excludes) -> do+    gathered <- gatherChanges options graph+    case gathered of+      Left err -> usage err+      Right (changes, directSelectors, warnings)+        | null changes && null directSelectors && not options.allScenariosFlag && not (maybe False ((== SuiteAll) . (.mode) . (.selection)) suite) -> usage "say what changed with --changed, --cohort-from/--cohort-to, --since, --upstream-diff, --all, or an all-mode suite"+        | otherwise -> do+            let componentSelected = if options.allScenariosFlag then selectAll catalog else selectScenarios graph catalog changes+                pathSelected = selectBySelectors Since "repository path" directSelectors catalog+                changedSelected = deduplicate (componentSelected <> pathSelected)+                suiteSelected = maybe changedSelected (\suiteValue -> applySuite suiteValue catalog changedSelected) suite+                selected = applySelectors includes excludes (deduplicate suiteSelected)+                unmatchedPins = [name | raw <- options.knobPins, let (name, _) = splitAssignment raw, all (\selection -> not (declares name selection.scenario)) selected]+            if not (null unmatchedPins)+              then usage ("no selected scenario declares knob: " <> Text.intercalate ", " unmatchedPins)+              else+                if options.explain+                  then renderSelection True changes warnings selected >> pure ExitSuccess+                  else writeRunPlan options suite graph changes warnings selected+  where+    declares name scenario = any ((== name) . renderKnobName . (.name)) scenario.knobs++writeRunPlan :: PlanOptions -> Maybe Suite -> ComponentGraph -> [Change] -> [Warning] -> [Selected] -> IO ExitCode+writeRunPlan options suite graph changes warnings selected = do+  seedResult <- makeSeed options.seedValue+  cohortResult <- first (("unable to resolve cohort identity: " <>) . Text.pack . show) <$> resolveDefaultCohortIdentity+  case (seedResult, cohortResult, (\seed -> makePolicy options suite seed) =<< seedResult) of+    (Left err, _, _) -> usage err+    (_, Left err, _) -> usage err+    (_, _, Left err) -> usage err+    (Right _, Right cohort, Right policy) -> do+      let context =+            PlanContext+              { suite = fmap (.name) suite,+                graphDigest = Components.graphDigest graph,+                cohortName = Cohort.unCohortName cohort.identityCohort,+                cohortPlanHash = Cohort.unPlanHash cohort.identityPlanHash,+                inputs = PlanInputs (inputValue options),+                changes,+                warnings = fmap (.message) warnings+              }+      plan <- stampPlan (buildPlan context policy selected)+      let bytes = Aeson.encode plan+      case options.outputPath of+        Nothing -> LazyByteString.putStrLn bytes+        Just path -> LazyByteString.writeFile path bytes+      Text.IO.hPutStrLn stderr ("planned " <> Text.pack (show (length plan.runs)) <> " runs, estimate " <> Text.pack (show plan.estimateMinutes) <> " min, " <> Text.pack (show (length plan.skipped)) <> " skipped")+      pure ExitSuccess++makeSeed :: Maybe Integer -> IO (Either Text Id.Seed)+makeSeed (Just value)+  | value < 0 = pure (Left "seed must be non-negative")+  | otherwise = pure (Id.mkSeed (fromInteger value))+makeSeed Nothing = do+  value <- randomIO+  pure (Id.mkSeed (value `mod` 9007199254740992))++makePolicy :: PlanOptions -> Maybe Suite -> Id.Seed -> Either Text PlanPolicy+makePolicy options suite seed = do+  base <- maybe (Right (defaultPlanPolicy seed)) (suitePolicy seed) suite+  maxTier <- maybe (Right base.maxTier) parseTierValue options.maxTierText+  let kindInputs = concatMap (filter (not . Text.null) . fmap Text.strip . Text.splitOn ",") options.kindTexts+  kinds <- if null kindInputs then Right base.kinds else Set.fromList <$> traverse Id.parseKind kindInputs+  placement <- maybe (Right base.placement) parsePlacementValue options.placementText+  dimensionPolicy <- maybe (Right base.dimensionPolicy) (maybe (Left "unknown dimension policy") Right . parseDimensionPolicy) options.dimensionPolicyText+  knobPolicy <- maybe (Right base.knobPolicy) (maybe (Left "unknown knob policy") Right . parseKnobPolicy) options.knobPolicyText+  let trials = maybe base.trials (\value -> value) options.trialCount+  if trials < 3 then Left "--trials must be at least 3" else pure ()+  case options.budget of Just value | value < 0 -> Left "--budget-minutes must be non-negative"; _ -> pure ()+  pinnedKnobs <- traverse parseAssignmentText options.knobPins+  pinnedDimensions <- traverse parseAssignmentText options.dimensionPins+  pure+    base+      { maxTier,+        kinds,+        placement,+        dimensionPolicy,+        knobPolicy,+        trials,+        budgetMinutes = options.budget <|> base.budgetMinutes,+        pinnedKnobs,+        pinnedDimensions+      }++parseTierValue :: Text -> Either Text Tier+parseTierValue "smoke" = Right TierSmoke+parseTierValue "standard" = Right TierStandard+parseTierValue "extended" = Right TierExtended+parseTierValue "soak" = Right TierSoak+parseTierValue value = Left ("unknown tier " <> value)++parsePlacementValue :: Text -> Either Text SpecPlacement+parsePlacementValue "local" = Right RunLocal+parsePlacementValue "cell" = Right RunOnCell+parsePlacementValue value = Left ("unknown placement " <> value)++parseAssignmentText :: Text -> Either Text (Text, Text)+parseAssignmentText raw = case splitAssignment raw of+  (name, value) | Text.null name || Text.null value -> Left "expected NAME=VALUE"+  pair -> Right pair++splitAssignment :: Text -> (Text, Text)+splitAssignment raw = let (name, rest) = Text.breakOn "=" raw in (name, Text.drop 1 rest)++inputValue :: PlanOptions -> Aeson.Value+inputValue options =+  Aeson.object+    [ "changed" Aeson..= options.changed,+      "cohortFrom" Aeson..= options.cohortFrom,+      "cohortTo" Aeson..= options.cohortTo,+      "since" Aeson..= options.since,+      "upstreamDiffs" Aeson..= options.upstreamDiffs,+      "all" Aeson..= options.allScenariosFlag,+      "select" Aeson..= options.includes,+      "exclude" Aeson..= options.excludes+    ]++gatherChanges :: PlanOptions -> ComponentGraph -> IO (Either Text ([Change], [Selector], [Warning]))+gatherChanges options graph = do+  cohortResult <- gatherCohort+  sinceResult <- maybe (pure (Right ([], [], []))) (changesSince graph ".") options.since+  upstreamResults <- traverse gatherUpstream options.upstreamDiffs+  pure do+    named <- traverse (parseRef graph) namedInputs+    (cohortChanges, cohortWarnings) <- cohortResult+    (sinceChanges, sinceSelectors, sinceWarnings) <- sinceResult+    upstream <- sequence upstreamResults+    let upstreamChanges = concatMap fst upstream+        upstreamWarnings = [Warning "ignored-upstream-paths" (Text.pack (show ignored) <> " upstream paths were outside component source roots") | (_, ignored) <- upstream, ignored > 0]+        namedChanges = [Change reference Named ("--changed " <> renderRef reference) | reference <- named]+    Right (namedChanges <> cohortChanges <> sinceChanges <> upstreamChanges, sinceSelectors, cohortWarnings <> sinceWarnings <> upstreamWarnings)+  where+    namedInputs = concatMap (filter (not . Text.null) . fmap Text.strip . Text.splitOn ",") options.changed+    gatherCohort = case (options.cohortFrom, options.cohortTo) of+      (Nothing, Nothing) -> pure (Right ([], []))+      (Just from, Just to) -> do+        old <- readCohortPackages "." (parseCohortInput from)+        new <- readCohortPackages "." (parseCohortInput to)+        pure ((uncurry (diffCohorts graph)) <$> ((,) <$> old <*> new))+      _ -> pure (Left "--cohort-from and --cohort-to must be supplied together")+    gatherUpstream raw = case parseUpstreamDiff raw of+      Left err -> pure (Left err)+      Right input -> changesFromUpstream graph input++deduplicate :: [Selected] -> [Selected]+deduplicate = nubBy (\left right -> left.scenario.id == right.scenario.id)++renderSelection :: Bool -> [Change] -> [Warning] -> [Selected] -> IO ()+renderSelection explain changes warnings selected = do+  mapM_ (Text.IO.hPutStrLn stderr . ("warning: " <>) . (.message)) warnings+  if explain+    then do+      mapM_ renderChange changes+      mapM_ renderSelected selected+      Text.IO.putStrLn ("selected " <> Text.pack (show (length selected)) <> " scenarios")+    else mapM_ (Text.IO.putStrLn . Id.renderScenarioId . (.id) . (.scenario)) selected+  where+    renderChange changeValue = Text.IO.putStrLn ("changed    " <> renderRef changeValue.ref <> "  " <> changeValue.detail)+    renderSelected selectedValue = do+      let reason = NonEmpty.head selectedValue.reasons+      Text.IO.putStrLn ("selected   " <> Id.renderScenarioId selectedValue.scenario.id <> "  via " <> Text.intercalate " <- " (fmap renderRef reason.via) <> "  selector " <> renderSelector reason.selector)++loadGraph :: CliEnv -> Maybe InputSource -> IO (Either Text ComponentGraph)+loadGraph environment Nothing = pure (firstGraph (decodeGraph environment.componentGraph))+loadGraph _ (Just source) = firstGraph . decodeGraph <$> readInputSource source++firstGraph :: Either GraphError ComponentGraph -> Either Text ComponentGraph+firstGraph = either (Left . (.errorText)) Right++loadCatalog :: CliEnv -> Maybe InputSource -> IO (Either Text [ScenarioInfo])+loadCatalog environment Nothing = pure (Right (fmap fromScenario (allScenarios environment.registry)))+loadCatalog _ (Just source) = decodeCatalog <$> readInputSource source++loadSuite :: PlanOptions -> IO (Either Text (Maybe Suite))+loadSuite options = case (options.suiteName, options.suiteSource) of+  (Nothing, Nothing) -> pure (Right Nothing)+  (Just name, Nothing) -> fmap Just <$> readSuite ("suites/" <> Text.unpack name <> ".json")+  (Nothing, Just source) -> fmap Just . decodeSuite <$> readInputSource source+  (Just _, Just _) -> pure (Left "choose only one of --suite or --suite-file")++showGraph :: Bool -> ComponentGraph -> IO ExitCode+showGraph asJson graph = do+  if asJson+    then LazyByteString.putStrLn (Aeson.encode graph)+    else do+      let subCount = sum (fmap (length . (.subcomponents)) graph.components)+          edges = concatMap (.dependsOn) graph.components+          buildCount = length (filter ((== Build) . (.kind)) edges)+      Text.IO.putStrLn (Text.pack (show (length graph.components)) <> " components, " <> Text.pack (show subCount) <> " sub-components, " <> Text.pack (show (length edges)) <> " edges (" <> Text.pack (show buildCount) <> " build, " <> Text.pack (show (length edges - buildCount)) <> " runtime)")+      mapM_ renderComponent graph.components+  pure ExitSuccess+  where+    renderComponent componentValue = Text.IO.putStrLn (Components.unComponentId componentValue.id <> "  " <> Text.pack (show componentValue.kind) <> "  " <> Text.intercalate "," componentValue.packages)++checkGraph :: ComponentGraph -> [ScenarioInfo] -> IO ExitCode+checkGraph graph catalog = do+  driftResult <- checkAgainstPlanJson graph "dist-newstyle/cache/plan.json"+  case driftResult of+    Left err -> usage ("cannot read Cabal plan: " <> err)+    Right drift -> do+      let lint = lintAgainstCatalog graph catalog+          (driftErrors, driftInfo) = partition isDriftError drift+          (lintErrors, lintWarnings) = partition isLintError lint+      Text.IO.putStrLn ("component graph: " <> Text.pack (show (length graph.components)) <> " components; digest " <> Components.graphDigest graph)+      mapM_ (Text.IO.putStrLn . ("error: " <>) . Text.pack . show) driftErrors+      mapM_ (Text.IO.putStrLn . ("error: " <>) . Text.pack . show) lintErrors+      mapM_ (Text.IO.putStrLn . ("info: " <>) . Text.pack . show) driftInfo+      mapM_ (Text.IO.putStrLn . ("warning: " <>) . Text.pack . show) lintWarnings+      pure if null driftErrors && null lintErrors then ExitSuccess else ExitFailure 1++stdinCount :: PlanOptions -> Int+stdinCount options = length [() | Just InputStdin <- [options.graphSource, options.catalogSource, options.suiteSource]]++usage :: Text -> IO ExitCode+usage message = Text.IO.hPutStrLn stderr ("kenshou: " <> message) >> pure (ExitFailure 2)
+ src/Kenshou/Cli/Command/Run.hs view
@@ -0,0 +1,207 @@+module Kenshou.Cli.Command.Run (runCommand) where++import Data.Aeson qualified as Aeson+import Data.Bifunctor (first)+import Data.ByteString.Lazy.Char8 qualified as LazyByteString+import Data.List.NonEmpty qualified as NonEmpty+import Data.Map.Strict qualified as Map+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Text.IO qualified as Text.IO+import Kenshou.Cli.Cohort (resolveDefaultCohortIdentity)+import Kenshou.Cli.Config+import Kenshou.Core.Cli+import Kenshou.Core.Cli.Config+import Kenshou.Core.Id+import Kenshou.Core.Knob (parseAssignment)+import Kenshou.Core.Outcome (renderOutcome)+import Kenshou.Core.Phase (PhasePlan (..), zeroPhases)+import Kenshou.Core.Run+import Kenshou.Core.RunResult (RunResult (..))+import Kenshou.Core.RunSpec+import Kenshou.Core.RunSpec.Resolve+import Options.Applicative+import Settei hiding (optional)+import Settei.Env (envSnapshot)+import Settei.Optparse+import System.Environment (getArgs, getEnvironment)+import System.Exit (ExitCode (..))+import System.IO (stderr)++data RunOptions = RunOptions+  { specFile :: Maybe InputSource,+    scenario :: Maybe Text,+    config :: ConfigInputs,+    knobs :: [Text],+    dimensions :: [Text],+    seed :: Maybe Seed,+    runId :: Maybe RunId,+    timeout :: Maybe Int,+    phases :: [Text],+    pgUrl :: Maybe Text,+    pgUrlEnv :: Maybe Text,+    placement :: Maybe SpecPlacement,+    machineProfile :: Maybe Text,+    cohortIdentity :: Maybe InputSource,+    cellFingerprint :: Maybe InputSource,+    keepEnvironment :: Bool,+    printSpec :: Bool,+    json :: Bool,+    strictKnownDefects :: Bool+  }++runCommand :: CliCommand+runCommand = CliCommand "run" "Resolve or execute one scenario" Execution False (runHandler <$> runParser)++runParser :: Parser RunOptions+runParser =+  RunOptions+    <$> optional (option (parseInputSource <$> str) (long "spec" <> metavar "FILE" <> help "Read a kenshou.run-spec/v1 document; use - for stdin"))+    <*> optional (Text.pack <$> strArgument (metavar "SCENARIO" <> help "Registered scenario identifier"))+    <*> configInputsParser ((: []) <$> namedOption "--out" outputRootKey (long "out" <> metavar "DIR" <> help "Run output directory"))+    <*> many (Text.pack <$> strOption (long "set" <> metavar "NAME=VALUE" <> help "Set a scenario knob"))+    <*> many (Text.pack <$> strOption (long "dim" <> metavar "NAME=VALUE" <> help "Set a cross-cutting dimension"))+    <*> optional (option (eitherReader readSeed) (long "seed" <> metavar "N" <> help "Replayable random seed"))+    <*> optional (option (eitherReader readRunId) (long "run-id" <> metavar "UUID" <> help "Pre-assign a UUIDv7 run id"))+    <*> optional (option auto (long "timeout" <> metavar "SECONDS" <> help "Run timeout"))+    <*> many (Text.pack <$> strOption (long "phase" <> metavar "NAME=SECONDS" <> help "Override warm-up, steady, or drain duration"))+    <*> optional (Text.pack <$> strOption (long "pg-url" <> metavar "CONNECTION" <> help "Use an external PostgreSQL connection string"))+    <*> optional (Text.pack <$> strOption (long "pg-url-env" <> metavar "VAR" <> help "Read the external PostgreSQL connection string from VAR"))+    <*> optional (option (eitherReader readPlacement) (long "placement" <> metavar "local|cell" <> help "Execution placement"))+    <*> optional (Text.pack <$> strOption (long "machine-profile" <> metavar "NAME" <> help "Stable machine profile name"))+    <*> optional (option (parseInputSource <$> str) (long "cohort-identity" <> metavar "FILE" <> help "Use a resolved cohort identity; use - for stdin"))+    <*> optional (option (parseInputSource <$> str) (long "cell-fingerprint" <> metavar "FILE" <> help "Attach a cell fingerprint; use - for stdin"))+    <*> switch (long "keep-env" <> help "Leave disposable environments running for inspection")+    <*> switch (long "print-spec" <> help "Print the effective run specification without executing")+    <*> switch (long "json" <> help "Print the run result as JSON")+    <*> switch (long "strict-known-defects" <> help "Treat reproduced known defects as blocking")++runHandler :: RunOptions -> CliEnv -> IO ExitCode+runHandler options environment = case schemaDiagnostic options.config.diagnostic (describe runDefaultsConfig) of+  Just output -> Text.IO.putStr output >> pure ExitSuccess+  Nothing -> do+    processEnvironment <- fmap (fmap (\(name, value) -> (Text.pack name, Text.pack value))) getEnvironment+    defaultsResult <- resolveRunDefaults (envSnapshot processEnvironment) options.config+    case defaultsResult of+      Left err -> Text.IO.hPutStrLn stderr err >> pure (ExitFailure 2)+      Right resolved -> case resolved.answer of+        Left problems -> Text.IO.hPutStr stderr (renderErrorsText problems) >> pure (ExitFailure 2)+        Right defaults -> case resolutionDiagnostic options.config.diagnostic resolved of+          Just output -> Text.IO.putStr output >> pure ExitSuccess+          Nothing -> resolveAndRun defaults+  where+    resolveAndRun defaults = do+      if stdinInputCount options > 1+        then Text.IO.hPutStrLn stderr "kenshou: at most one document input may use standard input" >> pure (ExitFailure 2)+        else do+          input <- loadInput options+          case input >>= applyOverrides options of+            Left err -> Text.IO.hPutStrLn stderr ("kenshou: " <> err) >> pure (ExitFailure 2)+            Right spec -> do+              resolved <- resolveRunSpec environment.registry spec+              case resolved of+                Left problems -> Text.IO.hPutStrLn stderr (Text.intercalate "\n" [message | SpecError message <- NonEmpty.toList problems]) >> pure (ExitFailure 2)+                Right (_, effective)+                  | options.printSpec -> LazyByteString.putStrLn (Aeson.encode effective) >> pure ExitSuccess+                  | otherwise -> execute defaults spec++    execute defaults spec = do+      cohortResult <- loadCohort options.cohortIdentity+      cellResult <- loadJsonDocument options.cellFingerprint+      case (cohortResult, cellResult) of+        (Left err, _) -> Text.IO.hPutStrLn stderr ("kenshou: " <> err) >> pure (ExitFailure 2)+        (_, Left err) -> Text.IO.hPutStrLn stderr ("kenshou: " <> err) >> pure (ExitFailure 2)+        (Right cohort, Right cellFingerprint) -> do+          argv <- getArgs+          result <- executeRun (RunnerConfig environment.registry (Text.unpack defaults.outputRoot) cohort cellFingerprint options.keepEnvironment options.strictKnownDefects argv) spec+          case result of+            Left problems -> Text.IO.hPutStrLn stderr (Text.intercalate "\n" [message | SpecError message <- NonEmpty.toList problems]) >> pure (ExitFailure 2)+            Right output -> do+              if options.json+                then LazyByteString.putStrLn (Aeson.encode output.result)+                else Text.IO.putStrLn (renderOutcome output.result.outcome <> "  " <> renderScenarioId output.result.scenario <> "  " <> Text.pack output.directory)+              pure (if output.result.exitCode == 0 then ExitSuccess else ExitFailure output.result.exitCode)++    loadCohort Nothing = first (("unable to resolve cohort identity: " <>) . Text.pack . show) <$> resolveDefaultCohortIdentity+    loadCohort (Just source) = loadDocument "cohort identity" source++    loadJsonDocument Nothing = pure (Right Nothing)+    loadJsonDocument (Just source) = fmap Just <$> loadDocument "cell fingerprint" source++    loadDocument :: (Aeson.FromJSON value) => Text -> InputSource -> IO (Either Text value)+    loadDocument label source = first (("invalid " <> label <> ": ") <>) . first Text.pack . Aeson.eitherDecodeStrict' <$> readInputSource source++loadInput :: RunOptions -> IO (Either Text RunSpec)+loadInput options = case (options.specFile, options.scenario) of+  (Just _, Just _) -> pure (Left "choose either --spec FILE or SCENARIO")+  (Nothing, Nothing) -> pure (Left "provide --spec FILE or SCENARIO")+  (Just source, Nothing) -> do+    bytes <- readInputSource source+    pure (first Text.pack (Aeson.eitherDecodeStrict' bytes))+  (Nothing, Just scenarioText) -> pure (minimalRunSpec <$> parseScenarioId scenarioText)++applyOverrides :: RunOptions -> RunSpec -> Either Text RunSpec+applyOverrides options spec = do+  knobs <- first (Text.pack . show) (traverse parseAssignment options.knobs)+  dimensions <- traverse parseDimension options.dimensions+  phases <- applyPhaseOverrides spec.phases options.phases+  postgres <- case (options.pgUrl, options.pgUrlEnv) of+    (Just _, Just _) -> Left "choose either --pg-url or --pg-url-env"+    (Just connection, Nothing) -> Right (Just (PostgresExternal (ConnLiteral connection)))+    (Nothing, Just variable) -> Right (Just (PostgresExternal (ConnFromEnv variable)))+    (Nothing, Nothing) -> Right spec.environment.postgres+  let environment =+        spec.environment+          { placement = maybe spec.environment.placement id options.placement,+            machineProfile = options.machineProfile <> spec.environment.machineProfile,+            postgres+          }+  pure+    RunSpec+      { runId = options.runId <|> spec.runId,+        scenario = spec.scenario,+        scenarioRevision = spec.scenarioRevision,+        knobs = spec.knobs <> knobs,+        dimensions = spec.dimensions <> dimensions,+        seed = options.seed <|> spec.seed,+        phases,+        timeoutSeconds = options.timeout <|> spec.timeoutSeconds,+        environment,+        cohortExpectation = spec.cohortExpectation,+        comparison = spec.comparison,+        labels = spec.labels+      }+  where+    parseDimension input = case Text.breakOn "=" input of+      (name, assignment) | not (Text.null name) && not (Text.null assignment) -> Right (name, Text.drop 1 assignment)+      _ -> Left ("invalid dimension assignment \"" <> input <> "\"")++applyPhaseOverrides :: Maybe PhasePlan -> [Text] -> Either Text (Maybe PhasePlan)+applyPhaseOverrides existing [] = Right existing+applyPhaseOverrides existing assignments = Just <$> foldl step (Right (maybe zeroPhases id existing)) assignments+  where+    step result assignment = do+      plan <- result+      let (name, rawValue) = Text.breakOn "=" assignment+      seconds <- case reads (Text.unpack (Text.drop 1 rawValue)) of+        [(value, "")] | value >= (0 :: Double) -> Right value+        _ -> Left ("invalid phase assignment \"" <> assignment <> "\"")+      case name of+        "warm-up" -> Right plan {warmUpSeconds = seconds}+        "steady" -> Right plan {steadySeconds = seconds}+        "drain" -> Right plan {drainSeconds = seconds}+        _ -> Left ("unknown phase \"" <> name <> "\"")++stdinInputCount :: RunOptions -> Int+stdinInputCount options = length [() | Just InputStdin <- [options.specFile, options.cohortIdentity, options.cellFingerprint]]++readSeed :: String -> Either String Seed+readSeed input = case reads input of [(value, "")] -> first Text.unpack (mkSeed value); _ -> Left "seed must be an integer"++readRunId :: String -> Either String RunId+readRunId = first Text.unpack . parseRunId . Text.pack++readPlacement :: String -> Either String SpecPlacement+readPlacement "local" = Right RunLocal+readPlacement "cell" = Right RunOnCell+readPlacement value = Left ("unknown placement " <> show value)
+ src/Kenshou/Cli/Command/Worker.hs view
@@ -0,0 +1,20 @@+module Kenshou.Cli.Command.Worker (workerCommand) where++import Data.Bifunctor (first)+import Data.Foldable (traverse_)+import Data.Text qualified as Text+import Kenshou.Core.Cli (CliCommand (..), CliEnv (..), CliGroup (..))+import Kenshou.Core.Role (mkRoleName, renderRoleName)+import Kenshou.Core.Role.Dispatch (runWorker)+import Kenshou.Diagnose.Profile (reexecWithWorkerRts)+import Kenshou.Diagnose.Threads (installThreadDumpSignal)+import Options.Applicative++workerCommand :: CliCommand+workerCommand = CliCommand "worker" "Run an internal worker role" Execution True (uncurry handler <$> parser)+  where+    parser = (,) <$> option (eitherReader (first Text.unpack . mkRoleName . Text.pack)) (long "role" <> metavar "ROLE") <*> optional (strOption (long "diagnosis-root" <> metavar "DIR" <> internal))+    handler role diagnosisRoot environment = do+      reexecWithWorkerRts (renderRoleName role)+      traverse_ (\root -> installThreadDumpSignal root (renderRoleName role)) diagnosisRoot+      runWorker environment.registry role
+ src/Kenshou/Cli/Completions.hs view
@@ -0,0 +1,32 @@+module Kenshou.Cli.Completions (completionsCommand) where++import Data.Text (Text)+import Data.Text.IO qualified as Text.IO+import Kenshou.Core.Cli (CliCommand (..), CliGroup (..))+import Options.Applicative+import System.Exit (ExitCode (..))++data Shell = Bash | Zsh | Fish++completionsCommand :: CliCommand+completionsCommand =+  CliCommand+    { name = "completions",+      description = "Generate parser-derived shell completions",+      group = Maintenance,+      hidden = False,+      parser = (\shell _ -> Text.IO.putStr (script shell) >> pure ExitSuccess) <$> shellParser+    }++shellParser :: Parser Shell+shellParser = argument (eitherReader parseShell) (metavar "bash|zsh|fish" <> help "Target shell")+  where+    parseShell "bash" = Right Bash+    parseShell "zsh" = Right Zsh+    parseShell "fish" = Right Fish+    parseShell value = Left ("unsupported shell " <> show value)++script :: Shell -> Text+script Bash = "_kenshou_completions() {\n  local CMDLINE=(--bash-completion-index $COMP_CWORD)\n  for arg in ${COMP_WORDS[@]}; do CMDLINE+=(--bash-completion-word \"$arg\"); done\n  COMPREPLY=( $(kenshou \"${CMDLINE[@]}\" 2>/dev/null) )\n}\ncomplete -o filenames -F _kenshou_completions kenshou\n"+script Zsh = "#compdef kenshou\n_kenshou() {\n  local -a completions\n  local CMDLINE=(--bash-completion-enriched --bash-completion-index $((CURRENT - 1)))\n  for arg in ${words[@]}; do CMDLINE+=(--bash-completion-word \"$arg\"); done\n  completions=(\"${(@f)$(kenshou $CMDLINE 2>/dev/null)}\")\n  _describe kenshou completions\n}\n_kenshou\n"+script Fish = "complete -c kenshou -f\nfunction __kenshou_complete\n  set -l tokens (commandline -cop)\n  set -l args --bash-completion-enriched --bash-completion-index (count $tokens)\n  for token in $tokens; set args $args --bash-completion-word $token; end\n  kenshou $args 2>/dev/null\nend\ncomplete -c kenshou -a '(__kenshou_complete)'\n"
+ src/Kenshou/Cli/Config.hs view
@@ -0,0 +1,54 @@+module Kenshou.Cli.Config+  ( RunDefaults (..),+    runDefaultsConfig,+    runEnvironmentBindings,+    resolveRunDefaults,+    outputRootKey,+  )+where++import Data.Map.Strict qualified as Map+import Data.Text (Text)+import Data.Text qualified as Text+import Kenshou.Core.Cli.Config (ConfigInputs (..), loadConfigSources)+import Settei+import Settei.Env++data RunDefaults = RunDefaults+  { outputRoot :: Text,+    pg17Bin :: Maybe Text,+    pg18Bin :: Maybe Text+  }+  deriving stock (Eq, Show)++runDefaultsConfig :: Config RunDefaults+runDefaultsConfig =+  RunDefaults+    <$> required (publicSetting outputRootKey "Default run output directory" textDecoder)+    <*> optional (publicSetting pg17BinKey "PostgreSQL 17 binary directory" textDecoder)+    <*> optional (publicSetting pg18BinKey "PostgreSQL 18 binary directory" textDecoder)++runEnvironmentBindings :: Bindings+runEnvironmentBindings =+  either+    (error . Text.unpack . renderEnvErrorsText)+    id+    (bindings [binding (EnvName "KENSHOU_PG17_BIN") pg17BinKey, binding (EnvName "KENSHOU_PG18_BIN") pg18BinKey])++resolveRunDefaults :: EnvSnapshot -> ConfigInputs -> IO (Either Text (ResolveResult RunDefaults))+resolveRunDefaults snapshot inputs = do+  loaded <- loadConfigSources inputs+  pure $ do+    fileAndNamed <- loaded+    pure (resolve defaultResolveOptions ([builtIns] <> fileAndNamed <> [environmentSource runEnvironmentBindings snapshot]) runDefaultsConfig)++builtIns :: Source+builtIns = source "kenshou built-ins" BuiltInSource (RawObject (Map.singleton "run" (RawObject (Map.singleton "output-root" (RawText "runs")))))++outputRootKey, pg17BinKey, pg18BinKey :: Key+outputRootKey = validKey "run.output-root"+pg17BinKey = validKey "postgres.pg17-bin"+pg18BinKey = validKey "postgres.pg18-bin"++validKey :: Text -> Key+validKey value = either (error . show) id (parseKey value)
+ src/Kenshou/Cli/Diagnose.hs view
@@ -0,0 +1,254 @@+module Kenshou.Cli.Diagnose+  ( diagnoseCommand,+  )+where++import Control.Exception (SomeException, bracket, displayException, try)+import Data.Aeson+import Data.ByteString.Lazy qualified as LazyByteString+import Data.Foldable (traverse_)+import Data.List (isPrefixOf, isSuffixOf, sort)+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Text.IO qualified as Text.IO+import Data.Time (getCurrentTime)+import Data.Word (Word64)+import Hasql.Connection qualified as Connection+import Hasql.Connection.Settings qualified as Settings+import Kenshou.Core.Cli+import Kenshou.Diagnose.Document+import Kenshou.Diagnose.Leak+import Kenshou.Diagnose.LockGraph (buildGraph, renderDot)+import Kenshou.Diagnose.Postgres+import Kenshou.Diagnose.Profile+import Kenshou.Diagnose.Render+import Kenshou.Diagnose.Stall.Classify (classify)+import Kenshou.Diagnose.Stall.Types+import Kenshou.Diagnose.Threads (dumpThreads)+import Options.Applicative hiding (Success)+import System.Directory (doesDirectoryExist, doesFileExist, listDirectory)+import System.Exit (ExitCode (..))+import System.FilePath (takeFileName, (</>))+import System.IO (stderr)+import System.Process (callProcess)++data DiagnoseOptions = DiagnoseLeak LeakOptions | DiagnoseStall StallOptions | DiagnoseProfile ProfileOptions++data LeakOptions = LeakOptions+  { runDirectory :: !FilePath,+    policy :: !(Maybe InputSource),+    seed :: !Word64,+    output :: !(Maybe FilePath),+    json :: !Bool+  }++data StallOptions = StallOptions+  { runDirectory :: !(Maybe FilePath),+    live :: !Bool,+    connection :: !(Maybe Text),+    reclassify :: !Bool,+    dotOutput :: !(Maybe FilePath),+    output :: !(Maybe FilePath),+    json :: !Bool+  }++data ProfileOptions = ProfileOptions+  { scenario :: !Text,+    modeName :: !Text,+    classes :: !String,+    breakdown :: !Char,+    intervalSeconds :: !Double,+    eventlogMaxBytes :: !Word64,+    worker :: !(Maybe Text),+    outputRoot :: !FilePath,+    knobs :: ![String],+    dimensions :: ![String]+  }++diagnoseCommand :: CliCommand+diagnoseCommand = CliCommand "diagnose" "Diagnose leaks, stalls, and profiles" Analysis False (runDiagnose <$> diagnoseParser)++diagnoseParser :: Parser DiagnoseOptions+diagnoseParser = hsubparser (command "leak" (info (DiagnoseLeak <$> leakParser) (progDesc "Re-judge sampled series for resource leaks")) <> command "stall" (info (DiagnoseStall <$> stallParser) (progDesc "Render or capture a concurrency stall")) <> command "profile" (info (DiagnoseProfile <$> profileParser) (progDesc "Run a scenario under heap or event-log profiling")))++leakParser :: Parser LeakOptions+leakParser =+  LeakOptions+    <$> strArgument (metavar "RUN_DIR")+    <*> optional (parseInputSource <$> strOption (long "policy" <> metavar "FILE" <> help "Leak policy file, or - for stdin"))+    <*> option auto (long "seed" <> metavar "N" <> value 0 <> showDefault)+    <*> optional (strOption (long "out" <> metavar "FILE"))+    <*> switch (long "json")++stallParser :: Parser StallOptions+stallParser =+  StallOptions+    <$> optional (strArgument (metavar "RUN_DIR"))+    <*> switch (long "live" <> help "Capture PostgreSQL and local threads now")+    <*> optional (Text.pack <$> strOption (long "connection" <> metavar "CONNECTION"))+    <*> switch (long "reclassify")+    <*> optional (strOption (long "dot" <> metavar "FILE"))+    <*> optional (strOption (long "out" <> metavar "FILE"))+    <*> switch (long "json")++profileParser :: Parser ProfileOptions+profileParser =+  ProfileOptions+    <$> (Text.pack <$> strArgument (metavar "SCENARIO"))+    <*> (Text.pack <$> strOption (long "mode" <> metavar "closure-type|info-table|eventlog|profiled"))+    <*> strOption (long "classes" <> metavar "CLASSES" <> value "gu" <> showDefault)+    <*> option (eitherReader parseBreakdown) (long "breakdown" <> metavar "c|r|d|y" <> value 'c' <> showDefault)+    <*> option auto (long "interval-s" <> metavar "SECONDS" <> value 10 <> showDefault)+    <*> option auto (long "eventlog-max-bytes" <> metavar "BYTES" <> value 536870912 <> showDefault)+    <*> optional (Text.pack <$> strOption (long "worker" <> metavar "ROLE"))+    <*> strOption (long "out" <> metavar "DIR" <> value ".dev/profiles" <> showDefault)+    <*> many (strOption (long "set" <> metavar "NAME=VALUE"))+    <*> many (strOption (long "dim" <> metavar "NAME=VALUE"))++runDiagnose :: DiagnoseOptions -> CliEnv -> IO ExitCode+runDiagnose (DiagnoseLeak options) _ = runLeak options+runDiagnose (DiagnoseStall options) _ = runStall options+runDiagnose (DiagnoseProfile options) _ = runProfile options++runLeak :: LeakOptions -> IO ExitCode+runLeak options = do+  attempted <- try @SomeException do+    spec <- loadPolicy options.policy+    analyseRunDirectory options.runDirectory spec options.seed+  case attempted of+    Left err -> diagnostic "leak" (Text.pack (displayException err)) >> pure (ExitFailure 4)+    Right (Left err) -> diagnostic "leak" (Text.pack (show err)) >> pure (ExitFailure 4)+    Right (Right report) -> do+      now <- getCurrentTime+      let document = Diagnosis LeakDiagnosis (Text.pack (takeFileName options.runDirectory)) "offline" now (Generator "kenshou-diagnose" "0.1.0.0" "theil-sen+moving-block-bootstrap/1") (toJSON report)+      emit options.json options.output document (renderLeakReport report)+      pure case report.verdict of+        Stable -> ExitSuccess+        LeakSuspected -> ExitFailure 1+        InsufficientData -> ExitFailure 3++loadPolicy :: Maybe InputSource -> IO LeakSpec+loadPolicy Nothing = pure defaultLeakSpec+loadPolicy (Just source) = do+  bytes <- readInputSource source+  either (ioError . userError) pure (eitherDecodeStrict' bytes)++runStall :: StallOptions -> IO ExitCode+runStall options+  | options.live = runLiveStall options+  | otherwise = case options.runDirectory of+      Nothing -> diagnostic "stall" "provide RUN_DIR, or use --live --connection CONNECTION" >> pure (ExitFailure 2)+      Just runDirectory -> do+        attempted <- try @SomeException (loadStallReports runDirectory)+        case attempted of+          Left err -> diagnostic "stall" (Text.pack (displayException err)) >> pure (ExitFailure 4)+          Right reports -> do+            let rendered = fmap (if options.reclassify then reclassified else id) reports+            traverse_ (writeDot options.dotOutput) (take 1 rendered)+            if null rendered+              then Text.IO.putStrLn "no stall diagnoses" >> pure ExitSuccess+              else do+                now <- getCurrentTime+                let body = toJSON rendered+                    document = Diagnosis StallDiagnosis (Text.pack (takeFileName runDirectory)) "offline" now (Generator "kenshou-diagnose" "0.1.0.0" "stall-render-v1") body+                emit options.json options.output document (Text.intercalate "\n" (fmap renderStallReport rendered))+                pure (ExitFailure 1)++runLiveStall :: StallOptions -> IO ExitCode+runLiveStall options = case options.connection of+  Nothing -> diagnostic "stall" "--live requires --connection CONNECTION" >> pure (ExitFailure 2)+  Just connectionString -> do+    attempted <- try @SomeException $ bracket (acquire connectionString) Connection.release \connection -> do+      postgres <- capturePostgres connection 2+      threads <- dumpThreads False+      now <- getCurrentTime+      let graph = buildGraph postgres.activity postgres.locks+          snapshot = StallSnapshot 0 [] threads (Just postgres) graph [] Null (SpinEvidence 0 (maybe 0 (.callsPerSecond) postgres.statementRate))+          (primary, secondary, reasons) = classify snapshot+      pure (StallReport now 1 primary secondary reasons snapshot)+    case attempted of+      Left err -> diagnostic "stall" (Text.pack (displayException err)) >> pure (ExitFailure 4)+      Right report -> do+        writeDot options.dotOutput report+        let document = Diagnosis StallDiagnosis "live" "live" report.detectedAt (Generator "kenshou-diagnose" "0.1.0.0" "live-stall-v1") (toJSON report)+        emit options.json options.output document (renderStallReport report)+        pure if report.classification == Unknown then ExitSuccess else ExitFailure 1++acquire :: Text -> IO Connection.Connection+acquire connectionString = Connection.acquire (Settings.connectionString connectionString <> Settings.applicationName "kenshou-diagnose-live") >>= either (ioError . userError . show) pure++loadStallReports :: FilePath -> IO [StallReport]+loadStallReports runDirectory = do+  exists <- doesDirectoryExist runDirectory+  if not exists then ioError (userError ("run directory does not exist: " <> runDirectory)) else pure ()+  let diagnosisDirectory = runDirectory </> "diagnosis"+  diagnosisExists <- doesDirectoryExist diagnosisDirectory+  if not diagnosisExists+    then pure []+    else do+      names <- sort <$> listDirectory diagnosisDirectory+      traverse (loadOne . (diagnosisDirectory </>)) [name | name <- names, "stall-" `isPrefixOf` name, ".json" `isSuffixOf` name]+  where+    loadOne path = do+      decoded <- eitherDecodeFileStrict' path :: IO (Either String Diagnosis)+      document <- either (ioError . userError) pure decoded+      case fromJSON document.body of+        Error err -> ioError (userError (path <> ": " <> err))+        Success report -> pure report++reclassified :: StallReport -> StallReport+reclassified report =+  let (primary, secondary, reasons) = classify report.snapshot+   in StallReport report.detectedAt report.captureNumber primary secondary reasons report.snapshot++writeDot :: Maybe FilePath -> StallReport -> IO ()+writeDot Nothing _ = pure ()+writeDot (Just path) report = Text.IO.writeFile path (renderDot report.snapshot.graph)++runProfile :: ProfileOptions -> IO ExitCode+runProfile options = case parseMode options of+  Left message -> diagnostic "profile" message >> pure (ExitFailure 2)+  Right mode -> do+    let arguments = concatMap (\assignment -> ["--set", assignment]) options.knobs <> concatMap (\assignment -> ["--dim", assignment]) options.dimensions+        request = ProfileRequest mode options.scenario arguments options.outputRoot options.intervalSeconds options.eventlogMaxBytes options.worker+    attempted <- try @SomeException (runProfileSession request >>= postprocessProfile)+    case attempted of+      Left err -> diagnostic "profile" (Text.pack (displayException err)) >> pure (ExitFailure 4)+      Right report -> Text.IO.putStr (renderProfileReport report) >> pure (toExitCode report.exitCode)++postprocessProfile :: ProfileReport -> IO ProfileReport+postprocessProfile report = do+  let tool = ".dev/bin/eventlog2html"+      eventlog = report.sessionDirectory </> "kenshou.eventlog"+  available <- doesFileExist tool+  eventlogExists <- doesFileExist eventlog+  if available && eventlogExists then callProcess tool [eventlog] else pure ()+  pure report++parseMode :: ProfileOptions -> Either Text ProfileMode+parseMode options = case options.modeName of+  "closure-type" -> Right ClosureType+  "info-table" -> Right InfoTable+  "eventlog" -> Right (Eventlog options.classes)+  "profiled" -> Right (Profiled options.breakdown)+  other -> Left ("unknown profile mode " <> other)++parseBreakdown :: String -> Either String Char+parseBreakdown [breakdown] | breakdown `elem` ("crdy" :: String) = Right breakdown+parseBreakdown _ = Left "breakdown must be c, r, d, or y"++emit :: (ToJSON value) => Bool -> Maybe FilePath -> value -> Text -> IO ()+emit json output document rendered = do+  let bytes = encode document <> "\n"+  traverse_ (\path -> LazyByteString.writeFile path bytes) output+  case (json, output) of+    (True, Nothing) -> LazyByteString.putStr bytes+    (True, Just _) -> pure ()+    (False, _) -> Text.IO.putStr rendered++diagnostic :: Text -> Text -> IO ()+diagnostic commandName message = Text.IO.hPutStrLn stderr ("kenshou diagnose " <> commandName <> ": " <> message)++toExitCode :: Int -> ExitCode+toExitCode 0 = ExitSuccess+toExitCode code = ExitFailure code
+ src/Kenshou/Cli/Help.hs view
@@ -0,0 +1,55 @@+{-# LANGUAGE TemplateHaskell #-}++module Kenshou.Cli.Help (helpCommand, topics) where++import Data.FileEmbed (embedStringFile)+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Text.IO qualified as Text.IO+import Kenshou.Cli.Help.Width (renderForWidth, resolveWidth)+import Kenshou.Core.Cli (CliCommand (..), CliGroup (..), HelpTopic (..))+import Options.Applicative+import System.Exit (ExitCode (..))+import System.IO (stderr)++topics :: [HelpTopic]+topics =+  [ HelpTopic "scenarios" "Scenario identifiers and metadata" (Text.pack $(embedStringFile "help/scenarios.md")),+    HelpTopic "selectors" "Selecting groups of scenarios" (Text.pack $(embedStringFile "help/selectors.md")),+    HelpTopic "run-specs" "Versioned run specification documents" (Text.pack $(embedStringFile "help/run-specs.md")),+    HelpTopic "planning" "Change-aware plans, suites, and safe resume" (Text.pack $(embedStringFile "help/planning.md")),+    HelpTopic "outcomes" "Run outcomes and known defects" (Text.pack $(embedStringFile "help/outcomes.md")),+    HelpTopic "exit-codes" "Stable process exit codes" (Text.pack $(embedStringFile "help/exit-codes.md")),+    HelpTopic "comparisons" "Paired benchmark comparisons" (Text.pack $(embedStringFile "help/comparisons.md")),+    HelpTopic "diagnostics" "Leak, stall, and profiling diagnosis" (Text.pack $(embedStringFile "help/diagnostics.md")),+    HelpTopic "evidence" "Durable run records and data links" (Text.pack $(embedStringFile "help/evidence.md")),+    HelpTopic "cells" "Leased cell execution and result verification" (Text.pack $(embedStringFile "help/cells.md"))+  ]++data HelpOptions = HelpOptions {topic :: Maybe Text, width :: Maybe Int}++helpCommand :: CliCommand+helpCommand =+  CliCommand+    { name = "help",+      description = "Read a long-form help topic",+      group = Discovery,+      hidden = False,+      parser = runHelp <$> helpParser+    }++helpParser :: Parser HelpOptions+helpParser =+  HelpOptions+    <$> optional (Text.pack <$> strArgument (metavar "TOPIC" <> help "Topic name"))+    <*> optional (option auto (long "width" <> metavar "COLUMNS" <> help "Wrap prose to this width"))++runHelp :: HelpOptions -> environment -> IO ExitCode+runHelp options _ = case options.topic of+  Nothing -> Text.IO.putStr (Text.unlines [topic.name <> "  " <> topic.description | topic <- topics]) >> pure ExitSuccess+  Just wanted -> case filter ((== Text.toCaseFold wanted) . Text.toCaseFold . (.name)) topics of+    [] -> Text.IO.hPutStrLn stderr ("kenshou: unknown help topic \"" <> wanted <> "\"") >> pure (ExitFailure 2)+    topic : _ -> do+      width <- resolveWidth options.width+      Text.IO.putStr (renderForWidth width topic.content)+      pure ExitSuccess
+ src/Kenshou/Cli/Help/Width.hs view
@@ -0,0 +1,40 @@+module Kenshou.Cli.Help.Width (renderForWidth, resolveWidth) where++import Data.Text (Text)+import Data.Text qualified as Text+import System.Console.Terminal.Size qualified as Terminal+import System.IO (hIsTerminalDevice, stdout)++resolveWidth :: Maybe Int -> IO (Maybe Int)+resolveWidth (Just width) = pure (Just (max 1 width))+resolveWidth Nothing = do+  terminal <- hIsTerminalDevice stdout+  if not terminal+    then pure Nothing+    else do+      window <- Terminal.size+      pure (min 140 . Terminal.width <$> window)++renderForWidth :: Maybe Int -> Text -> Text+renderForWidth Nothing body = body+renderForWidth (Just width) body = Text.intercalate "\n\n" (fmap renderParagraph (splitParagraphs body))+  where+    renderParagraph paragraph+      | all (Text.isPrefixOf "  ") (filter (not . Text.null . Text.strip) (Text.lines paragraph)) = paragraph+      | otherwise = Text.intercalate "\n" (packWords (Text.words paragraph))+    packWords [] = []+    packWords (firstWord : rest) = go firstWord rest+    go current [] = [current]+    go current (word : rest)+      | Text.length current + 1 + Text.length word <= width = go (current <> " " <> word) rest+      | otherwise = current : go word rest++splitParagraphs :: Text -> [Text]+splitParagraphs body = go [] [] (Text.lines body)+  where+    go result current [] = reverse (flush result current)+    go result current (line : rest)+      | Text.null (Text.strip line) = go (flush result current) [] rest+      | otherwise = go result (line : current) rest+    flush result [] = result+    flush result current = Text.intercalate "\n" (reverse current) : result
+ src/Kenshou/Cli/Main.hs view
@@ -0,0 +1,17 @@+module Kenshou.Cli.Main (mainWith, runWithArgsWith) where++import Data.Text.IO qualified as Text.IO+import Kenshou.Cli.Registry (CliAssembly)+import Kenshou.Cli.Version (appVersionWithGit)+import Kenshou.Core.Cli (runCli)+import Kenshou.Diagnose.Profile.EventlogGuard (withEventlogGuard)+import Kenshou.Diagnose.Profile.GhcDebug (withGhcDebugIfRequested)+import System.Environment (getArgs)+import System.Exit (ExitCode (..), exitWith)++mainWith :: CliAssembly -> IO ()+mainWith assembly = withGhcDebugIfRequested (withEventlogGuard (getArgs >>= runWithArgsWith assembly >>= exitWith))++runWithArgsWith :: CliAssembly -> [String] -> IO ExitCode+runWithArgsWith _ ["--version"] = Text.IO.putStrLn appVersionWithGit >> pure ExitSuccess+runWithArgsWith assembly arguments = runCli assembly arguments
+ src/Kenshou/Cli/Options.hs view
@@ -0,0 +1,76 @@+module Kenshou.Cli.Options+  ( Command (..),+    CohortCommand (..),+    commandParserInfo,+    cohortCommandParser,+  )+where++import Data.Text qualified as Text+import Kenshou.Cli.Version (appVersionWithGit)+import Options.Applicative+  ( Parser,+    ParserInfo,+    command,+    fullDesc,+    header,+    help,+    helper,+    info,+    infoOption,+    long,+    metavar,+    optional,+    progDesc,+    strOption,+    subparser,+    switch,+    value,+    (<**>),+  )++data Command = CohortCommand CohortCommand+  deriving stock (Eq, Show)++data CohortCommand+  = CohortShow Bool FilePath (Maybe FilePath) (Maybe FilePath)+  | CohortCheck FilePath (Maybe FilePath) (Maybe FilePath) (Maybe FilePath)+  deriving stock (Eq, Show)++commandParserInfo :: ParserInfo Command+commandParserInfo =+  info+    (commandParser <**> helper <**> versionOption)+    (fullDesc <> header "kenshou - run verification scenarios")+  where+    versionOption = infoOption (Text.unpack appVersionWithGit) (long "version" <> help "Show version")++commandParser :: Parser Command+commandParser = CohortCommand <$> subparser (command "cohort" (info cohortCommandParser (progDesc "Inspect the pinned runtime cohort")))++cohortCommandParser :: Parser CohortCommand+cohortCommandParser =+  subparser+    ( command "show" (info showParser (progDesc "Print the cohort resolved by cabal"))+        <> command "check" (info checkParser (progDesc "Check the resolved cohort against its descriptor"))+    )+  where+    showParser =+      CohortShow+        <$> switch (long "json" <> help "Write kenshou.cohort-identity/v1 JSON")+        <*> projectDirOption+        <*> optional planJsonOption+        <*> optional identityOption+    checkParser = CohortCheck <$> projectDirOption <*> optional planJsonOption <*> optional descriptorOption <*> optional identityOption++projectDirOption :: Parser FilePath+projectDirOption = strOption (long "project-dir" <> metavar "DIR" <> value "." <> help "Cabal project directory")++planJsonOption :: Parser FilePath+planJsonOption = strOption (long "plan-json" <> metavar "FILE" <> help "Override dist-newstyle/cache/plan.json")++identityOption :: Parser FilePath+identityOption = strOption (long "identity" <> metavar "FILE" <> help "Read a captured cohort identity instead of a cabal plan")++descriptorOption :: Parser FilePath+descriptorOption = strOption (long "descriptor" <> metavar "FILE" <> help "Override the active cohort descriptor")
+ src/Kenshou/Cli/Registry.hs view
@@ -0,0 +1,62 @@+module Kenshou.Cli.Registry+  ( CliAssembly,+    Program (..),+    selfTestBundles,+    standardCommands,+    standardTopics,+    selfTestAssembly,+  )+where++import Kenshou.Check.Selftest qualified as CheckSelftest+import Kenshou.Cli.Attest (measurementRecomputer, pairedComparisonRecomputer, runOutcomeRecomputer)+import Kenshou.Cli.Command.Cell (cellCommand)+import Kenshou.Cli.Command.Cohort (cohortCommand)+import Kenshou.Cli.Command.Execute (executeCommand)+import Kenshou.Cli.Command.List (listCommand)+import Kenshou.Cli.Command.Overhead (overheadCommand)+import Kenshou.Cli.Command.Perf (perfCommand)+import Kenshou.Cli.Command.Plan (planCommand)+import Kenshou.Cli.Command.Run (runCommand)+import Kenshou.Cli.Command.Worker (workerCommand)+import Kenshou.Cli.Completions (completionsCommand)+import Kenshou.Cli.Diagnose qualified as Diagnose+import Kenshou.Cli.Help qualified as Help+import Kenshou.Core.Bundle (LayerBundle)+import Kenshou.Core.Cli (CliCommand, HelpTopic, Program (..), defaultProgram)+import Kenshou.Core.Selftest qualified as Selftest+import Kenshou.Diagnose.SelfTest qualified as DiagnoseSelftest+import Kenshou.Evidence.Attest (Recomputer)+import Kenshou.Evidence.Cli (attestCommandWith, evidenceCommand, recordCommand)+import Kenshou.Evidence.HistoryCli (historyCommand)+import Kenshou.Measure.Cli qualified as Measure+import Kenshou.Measure.Selftest qualified as MeasureSelftest+import Kenshou.Remote.Selftest qualified as RemoteSelftest+import Kenshou.Telemetry.SelfTest qualified as TelemetrySelftest++-- | Everything an executable registers: its name and header, the layer bundles+-- whose scenarios and roles it runs, the operator commands it exposes, the help+-- topics, its component graph, and its performance contracts.+-- Extension contract: a suite adds its bundles to 'bundles', and its own+-- recomputers through 'standardCommands', without changing this library.+type CliAssembly = Program++-- | The self-test bundles the kernel and toolkits define for themselves.+selfTestBundles :: [LayerBundle]+selfTestBundles = [Selftest.bundle, MeasureSelftest.bundle, CheckSelftest.bundle, DiagnoseSelftest.bundle, TelemetrySelftest.selfTestBundle, RemoteSelftest.bundle]++-- | The generic operator commands, with @attest@ using the given recomputers.+standardCommands :: [Recomputer] -> [CliCommand]+standardCommands recomputers = [listCommand, planCommand, Help.helpCommand, runCommand, executeCommand, cellCommand, perfCommand, overheadCommand, Measure.summarizeCommand, Measure.compareCommand, Diagnose.diagnoseCommand, historyCommand, recordCommand, attestCommandWith recomputers, evidenceCommand, workerCommand, cohortCommand, completionsCommand]++standardTopics :: [HelpTopic]+standardTopics = Help.topics++-- | The self-test bundles with the standard commands and the generic recomputers.+selfTestAssembly :: CliAssembly+selfTestAssembly =+  defaultProgram+    { bundles = selfTestBundles,+      commands = standardCommands [runOutcomeRecomputer, measurementRecomputer, pairedComparisonRecomputer],+      topics = standardTopics+    }
+ src/Kenshou/Cli/Version.hs view
@@ -0,0 +1,36 @@+{-# LANGUAGE CPP #-}+{-# LANGUAGE TemplateHaskell #-}++module Kenshou.Cli.Version+  ( appVersion,+    appVersionWithGit,+    gitCommitShort,+  )+where++import Data.Text (Text)+import Data.Text qualified as Text+import Data.Version (showVersion)+import GitHash (GitInfo, giHash, tGitInfoCwdTry)+import Paths_kenshou_cli (version)++appVersion :: Text+appVersion = Text.pack (showVersion version)++gitInfo :: Either String GitInfo+gitInfo = $$tGitInfoCwdTry++nixGitHash :: Maybe Text+#ifdef GIT_HASH+nixGitHash = Just GIT_HASH+#else+nixGitHash = Nothing+#endif++gitCommitShort :: Maybe Text+gitCommitShort = case gitInfo of+  Right info -> Just (Text.pack (take 7 (giHash info)))+  Left _ -> nixGitHash++appVersionWithGit :: Text+appVersionWithGit = "kenshou v" <> appVersion <> maybe "" (\commit -> " (" <> commit <> ")") gitCommitShort
+ test/Main.hs view
@@ -0,0 +1,801 @@+module Main (main) where++import Control.Concurrent (forkIO, killThread, threadDelay)+import Control.Exception (bracket, finally)+import Control.Monad (forM_)+import Data.Aeson (Value, object, (.=))+import Data.Aeson qualified as Aeson+import Data.Aeson.Key qualified as Key+import Data.Aeson.KeyMap qualified as KeyMap+import Data.ByteString qualified as ByteString+import Data.ByteString.Lazy qualified as LazyByteString+import Data.Foldable (toList)+import Data.List (find)+import Data.List qualified as List+import Data.Set qualified as Set+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Text.IO qualified as Text+import Data.Time (addUTCTime, getCurrentTime)+import Kenshou.Cli (Program (..), runWithArgsWith, selfTestAssembly)+import Kenshou.Cli.Cohort (resolveDefaultCohortIdentity)+import Kenshou.Cli.Version (appVersionWithGit)+import Kenshou.Core.Canonical (sha256Hex)+import Kenshou.Core.Cohort (CohortIdentity (..), CohortName (..))+import Kenshou.Core.Id (newRunId, renderRunId)+import Kenshou.Core.Manifest (Manifest (..), ManifestFile (..))+import Kenshou.Core.Outcome qualified as Outcome+import Kenshou.Perf.Artifacts (sealPerfDirectory)+import Kenshou.Perf.Plan qualified as Perf+import Kenshou.Perf.Prepared qualified as Prepared+import Kenshou.Remote.Cell.Docs (Artifact (..), CellBuckets (..), CellDescriptor (..), CellManifest (..), CellOutcome (..), CellPhase (..), CellRunResult (..), CellStatus (..), Limits (..), LogChunks (..), ManifestPayload (..), Rejected (..), Submission (..), WorkObject (..))+import Kenshou.Remote.Cell.Lease (Lease (..))+import Kenshou.Remote.Cell.RouteRules (descriptorDigest)+import Kenshou.Remote.Cell.Session.Journal (LeaseMode (..), SessionJournal (..), SliceJournal (..), SliceState (..), readSessionJournal, writeSessionJournal)+import Kenshou.Remote.Cell.Submit (workObjectFor)+import Kenshou.Remote.Payload (Bundle (..), CellPayload (..), PayloadDescriptor (..))+import Kenshou.Remote.Payload qualified as Payload+import Kenshou.Remote.Store (Bucket (..), ObjectName (..), ObjectStore (..), Precondition (..))+import Kenshou.Remote.Store.File (newFileStore)+import Kenshou.Telemetry.Overhead (OverheadState (..), SlotRun (..), loadOverheadState)+import PerfSessionSpec qualified+import System.Directory (Permissions (..), copyFile, createDirectoryIfMissing, doesFileExist, doesPathExist, getPermissions, listDirectory, makeAbsolute, setPermissions, withCurrentDirectory)+import System.Environment (lookupEnv, setEnv, unsetEnv)+import System.Exit (ExitCode (..))+import System.FilePath ((</>))+import System.IO.Temp (withSystemTempDirectory)+import System.Timeout (timeout)+import Test.Hspec (describe, expectationFailure, hspec, it, shouldBe, shouldReturn, shouldSatisfy)++main :: IO ()+main = hspec do+  PerfSessionSpec.spec+  describe "performance planning CLI" do+    it "exposes calibration publication and rejects malformed batches before storage effects" $ withSystemTempDirectory "kenshou-calibration-malformed" \directory -> do+      let input = directory </> "definition.json"+      Aeson.encodeFile input Aeson.Null+      runWithArgs ["perf", "calibration", "--help"] `shouldReturnCode` ExitSuccess+      runWithArgs ["perf", "calibration", "declare", "--definition", input, "--uri", "gs://fixture-evidence/calibration/definition.json"] `shouldReturnCode` ExitFailure 2+    it "publishes a calibration definition create-only and retrieves its identical bytes" $ withSystemTempDirectory "kenshou-calibration-publication" \directory -> do+      raw <- withPerfRoot (Aeson.eitherDecodeFileStrict' "kenshou-core/test/golden/perf-calibration-definition.synthetic.json" >>= either fail pure)+      now <- getCurrentTime+      let input = directory </> "definition.json"+          storeRoot = directory </> "store"+          document = replacePerfField "expiresAt" (Aeson.toJSON (addUTCTime 3600 now)) raw+          uri = "gs://fixture-evidence/calibration/definition.json"+      Aeson.encodeFile input document+      store <- newFileStore storeRoot+      withCellStore storeRoot do+        runWithArgs ["perf", "calibration", "declare", "--definition", input, "--uri", uri] `shouldReturnCode` ExitSuccess+        bytes <- LazyByteString.readFile input+        retained <- store.getObject (Bucket "fixture-evidence") (ObjectName "calibration/definition.json")+        fmap fst retained `shouldBe` Just bytes+        runWithArgs ["perf", "calibration", "declare", "--definition", input, "--uri", uri] `shouldReturnCode` ExitSuccess+        Aeson.encodeFile input (replacePerfField "injectionMechanism" (Aeson.String "changed after declaration") document)+        runWithArgs ["perf", "calibration", "declare", "--definition", input, "--uri", uri] `shouldReturnCode` ExitFailure 4+        store.getObject (Bucket "fixture-evidence") (ObjectName "calibration/definition.json") >>= \current -> fmap fst current `shouldBe` Just bytes+    it "exposes persistent execution, status, resume and evidence replay" do+      forM_ ["run", "status", "resume", "verify"] \name -> do+        runWithArgs ["perf", name, "--help"] `shouldReturnCode` ExitSuccess+        runWithArgs ["perf", name] `shouldReturnCode` ExitFailure 2+    it "rejects malformed prepared input before creating a session or contacting a cell" $ withSystemTempDirectory "kenshou-perf-run-malformed" \directory -> do+      let input = directory </> "prepared.json"+          output = directory </> "session"+      Aeson.encodeFile input Aeson.Null+      runWithArgs ["perf", "run", "--prepared", input, "--out", output, "--evidence-base-uri", "gs://fixture-evidence/performance"] `shouldReturnCode` ExitFailure 2+      doesPathExist output `shouldReturn` False+    it "reads retained status without cloud authentication or execution" $ withSystemTempDirectory "kenshou-perf-status" \directory -> do+      withPerfRoot (copyFile "fixtures/consumer-baseline/kenshou-remote/test/golden/perf-session.fixture.json" (directory </> "session.json"))+      runWithArgs ["perf", "status", "--session", directory] `shouldReturnCode` ExitSuccess+    it "publishes and replays incomplete scope without leasing, and detects storage tampering" $ withSystemTempDirectory "kenshou-perf-incomplete" \directory -> do+      let storeRoot = directory </> "store"+          preparation = directory </> "preparation"+          output = directory </> "session"+          bytes = "synthetic payload; this control executes no runtime work"+          digest = Text.drop 7 (sha256Hex (LazyByteString.toStrict bytes))+          bundle = Bundle ("gs://control/payloads/sha256/" <> digest <> ".nar.zst") digest (LazyByteString.length bytes)+          changePayload arm = case Aeson.fromJSON arm.payload of+            Aeson.Error problem -> fail problem+            Aeson.Success payload -> pure arm {Prepared.payload = Aeson.toJSON (payload {Payload.cell = payload.cell {Payload.bundle = bundle}} :: PayloadDescriptor)}+      original <- withPerfRoot (Aeson.eitherDecodeFileStrict' "fixtures/consumer-baseline/kenshou-core/test/golden/perf-prepared.fixture.json" >>= either fail pure)+      baseline <- changePayload (original :: Prepared.PerfPrepared).baseline+      candidate <- changePayload original.candidate+      let prepared = original {Prepared.plan = original.plan {Perf.selected = [], Perf.coverage = "incomplete", Perf.estimateSeconds = 0}, Prepared.baseline = baseline, Prepared.candidate = candidate}+      createDirectoryIfMissing True preparation+      Aeson.encodeFile (preparation </> "prepared.json") prepared+      sealPerfDirectory preparation >>= either (fail . Text.unpack) pure+      descriptor <- withPerfRoot (LazyByteString.readFile "kenshou-remote/test/golden/cell/cell.descriptor.v1.json")+      store <- newFileStore storeRoot+      _ <- store.putObject (Bucket "control") (ObjectName ("payloads/sha256/" <> digest <> ".nar.zst")) "application/octet-stream" DoesNotExist bytes+      _ <- store.putObject (Bucket "tan-nb-exp-cells-control") (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist descriptor+      withCellStore storeRoot $ withCapabilityDir (directory </> "capabilities") do+        runWithArgs ["perf", "run", "--prepared", preparation </> "prepared.json", "--out", output, "--evidence-base-uri", "gs://fixture-evidence/performance"] `shouldReturnCode` ExitFailure 3+        store.getObject (Bucket "tan-nb-exp-cells-control") (ObjectName "cells/alpha/lease.json") `shouldReturn` Nothing+        runWithArgs ["perf", "verify", "--session", output] `shouldReturnCode` ExitFailure 3+        runWithArgs ["perf", "verify", "--session", output, "--offline"] `shouldReturnCode` ExitFailure 3+        originalResult <- LazyByteString.readFile (output </> "result.json")+        runWithArgs ["perf", "resume", "--session", output] `shouldReturnCode` ExitFailure 3+        LazyByteString.readFile (output </> "result.json") `shouldReturn` originalResult+        published <- store.listObjects (Bucket "fixture-evidence") "performance/sessions/"+        let reports = [name | (name, _) <- published, "/report.md" `Text.isSuffixOf` name.unObjectName]+        length reports `shouldBe` 1+        forM_ reports \name -> do+          _ <- store.putObject (Bucket "fixture-evidence") name "text/markdown" NoPrecondition "tampered"+          pure ()+        runWithArgs ["perf", "verify", "--session", output] `shouldReturnCode` ExitFailure 4+    it "exposes the planning command and requires request/output arguments" do+      runWithArgs ["perf", "plan", "--help"] `shouldReturnCode` ExitSuccess+      runWithArgs ["perf", "plan"] `shouldReturnCode` ExitFailure 2+    it "exposes exact-revision preparation and rejects malformed requests without building" do+      runWithArgs ["perf", "prepare", "--help"] `shouldReturnCode` ExitSuccess+      withSystemTempDirectory "kenshou-perf-prepare-malformed" \directory -> do+        let input = directory </> "request.json"+            output = directory </> "preparation"+        Aeson.encodeFile input Aeson.Null+        runWithArgs ["perf", "prepare", "--request", input, "--out", output] `shouldReturnCode` ExitFailure 2+        doesPathExist output `shouldReturn` False+    it "rejects malformed requests before creating an output directory" do+      withSystemTempDirectory "kenshou-perf-malformed" \directory -> do+        let input = directory </> "request.json"+            output = directory </> "plan"+        Aeson.encodeFile input Aeson.Null+        runWithArgs ["perf", "plan", "--request", input, "--out", output] `shouldReturnCode` ExitFailure 2+        doesPathExist output `shouldReturn` False+    it "rejects target components from a second repository before fetching" do+      withPerfRoot $ withSystemTempDirectory "kenshou-perf-targets" \directory -> do+        raw <- Aeson.eitherDecodeFileStrict' "fixtures/consumer-baseline/kenshou-core/test/golden/perf-request.json" >>= either fail pure+        let input = directory </> "request.json"+            output = directory </> "plan"+        targets <- foreignTargets raw+        Aeson.encodeFile input (replacePerfField "targetComponents" (Aeson.toJSON targets) raw)+        withConsumerBaseline ["perf", "plan", "--request", input, "--out", output] `shouldReturnCode` ExitFailure 2+        doesPathExist output `shouldReturn` False+    it "rejects unknown profiles before fetching immutable sources" do+      withPerfRoot $ withSystemTempDirectory "kenshou-perf-profiles" \directory -> do+        raw <- Aeson.eitherDecodeFileStrict' "fixtures/consumer-baseline/kenshou-core/test/golden/perf-request.json" >>= either fail pure+        let input = directory </> "request.json"+            output = directory </> "plan"+        Aeson.encodeFile input (replacePerfField "profiles" (Aeson.toJSON (["missing"] :: [Text])) raw)+        withConsumerBaseline ["perf", "plan", "--request", input, "--out", output] `shouldReturnCode` ExitFailure 2+        doesPathExist output `shouldReturn` False+    it "refuses to overwrite an existing planning directory" do+      withSystemTempDirectory "kenshou-perf-existing" \directory -> do+        runWithArgs ["perf", "plan", "--request", "missing", "--out", directory] `shouldReturnCode` ExitFailure 2+  describe "CLI exit contract" do+    it "resolves the wrapped cohort identity outside a source checkout" $+      withSystemTempDirectory "kenshou-wrapped-cohort" \root -> do+        fixture <- makeAbsolute "../kenshou-core/test/fixtures/cohort-identity.golden.json"+        withEnvVariable "KENSHOU_COHORT_IDENTITY" fixture $+          withCurrentDirectory root do+            resolved <- resolveDefaultCohortIdentity+            case resolved of+              Left problem -> expectationFailure (show problem)+              Right identity -> identity.identityCohort `shouldBe` CohortName "released"++    it "returns success for help" do+      runWithArgs ["--help"] `shouldReturnCode` ExitSuccess++    it "rejects a pinned knob absent from every selected scenario" $+      withSystemTempDirectory "kenshou-plan-pin" \root -> do+        let output = root </> "run-plan.json"+        runWithArgs+          [ "plan",+            "--all",+            "--select",+            "selftest/kernel/correctness/always-pass",+            "--set",+            "diagnose.major-gc-interval-ms=5000",+            "--out",+            output+          ]+          `shouldReturnCode` ExitFailure 2+        doesFileExist output `shouldReturn` False++    it "returns success for command-specific help" do+      runWithArgs ["record", "--help"] `shouldReturnCode` ExitSuccess+      runWithArgs ["attest", "--help"] `shouldReturnCode` ExitSuccess+      runWithArgs ["history", "--help"] `shouldReturnCode` ExitSuccess+      runWithArgs ["run", "--help"] `shouldReturnCode` ExitSuccess+      runWithArgs ["cell", "--help"] `shouldReturnCode` ExitSuccess+      runWithArgs ["cell", "fetch", "--help"] `shouldReturnCode` ExitSuccess+      runWithArgs ["cell", "verify", "--help"] `shouldReturnCode` ExitSuccess+      runWithArgs ["cell", "status", "--help"] `shouldReturnCode` ExitSuccess+      runWithArgs ["cell", "lease", "--help"] `shouldReturnCode` ExitSuccess+      runWithArgs ["cell", "release", "--help"] `shouldReturnCode` ExitSuccess+      runWithArgs ["cell", "watch", "--help"] `shouldReturnCode` ExitSuccess+      runWithArgs ["cell", "route", "--help"] `shouldReturnCode` ExitSuccess+      runWithArgs ["cell", "submit", "--help"] `shouldReturnCode` ExitSuccess+      runWithArgs ["cell", "run", "--help"] `shouldReturnCode` ExitSuccess+      runWithArgs ["cell", "probe", "--help"] `shouldReturnCode` ExitSuccess+      runWithArgs ["cell", "pair", "--help"] `shouldReturnCode` ExitSuccess+      runWithArgs ["cell", "overhead", "--help"] `shouldReturnCode` ExitSuccess+      runWithArgs ["cell", "resume", "--help"] `shouldReturnCode` ExitSuccess+      runWithArgs ["cell", "payload", "publish", "--help"] `shouldReturnCode` ExitSuccess+      runWithArgs ["cell", "payload", "show", "--help"] `shouldReturnCode` ExitSuccess+      runWithArgs ["help", "cells"] `shouldReturnCode` ExitSuccess++    it "shows a payload only when its bundle exists at the recorded size" $+      withSystemTempDirectory "kenshou-cell-payload" \root -> do+        store <- newFileStore root+        let descriptor = "../kenshou-remote/test/golden/payload.json"+            bundle = ObjectName "payloads/sha256/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.nar.zst"+            showPayload = ["cell", "payload", "show", descriptor]+        withCellStore root do+          runWithArgs showPayload `shouldReturnCode` ExitFailure 1+          _ <- store.putObject (Bucket "control") bundle "application/octet-stream" DoesNotExist "xx"+          runWithArgs showPayload `shouldReturnCode` ExitFailure 1+          _ <- store.putObject (Bucket "control") bundle "application/octet-stream" NoPrecondition "x"+          runWithArgs showPayload `shouldReturnCode` ExitSuccess++    it "rejects malformed cell result identifiers and URIs" do+      runWithArgs ["cell", "fetch", "--results-bucket", "test-results", "not-a-run-id", "--out", "test-output"] `shouldReturnCode` ExitFailure 2+      runWithArgs ["cell", "verify", "gs://test-results/runs/not-a-run-id"] `shouldReturnCode` ExitFailure 2++    it "distinguishes an unavailable cell tree from invalid evidence" do+      runWithArgs ["cell", "verify", "test/fixtures/does-not-exist"] `shouldReturnCode` ExitFailure 4+      runWithArgs ["cell", "verify", leakingRun] `shouldReturnCode` ExitFailure 1++    it "acquires, reports and releases a lease through the file-backed cell CLI" $+      withSystemTempDirectory "kenshou-cell-cli" \root -> do+        store <- newFileStore root+        descriptor <- LazyByteString.readFile "../kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+        let control = Bucket "tan-nb-exp-cells-control"+            leaseObject = ObjectName "cells/alpha/lease.json"+            common = ["--cell", "alpha", "--control-bucket", "tan-nb-exp-cells-control"]+        _ <- store.putObject control (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist descriptor+        withCellStore root do+          runWithArgs (["cell", "status"] <> common <> ["--json"]) `shouldReturnCode` ExitSuccess+          runWithArgs (["cell", "lease"] <> common <> ["--purpose", "cli-test"]) `shouldReturnCode` ExitSuccess+          stored <- store.getObject control leaseObject+          record <- case stored of+            Just (bytes, _) -> case Aeson.eitherDecode bytes of+              Right lease -> pure (lease :: Lease)+              Left problem -> expectationFailure problem >> error "unreachable"+            Nothing -> expectationFailure "lease was not published" >> error "unreachable"+          runWithArgs (["cell", "status"] <> common) `shouldReturnCode` ExitSuccess+          runWithArgs (["cell", "release"] <> common <> ["--lease-id", Text.unpack (renderRunId record.leaseId)]) `shouldReturnCode` ExitSuccess+          store.statObject control leaseObject `shouldReturn` Nothing++    it "routes a public plan through file-backed cell descriptors" $+      withSystemTempDirectory "kenshou-cell-route" \root -> do+        store <- newFileStore root+        descriptor <- LazyByteString.readFile "../kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+        observed <- either fail pure (Aeson.eitherDecode descriptor :: Either String CellDescriptor)+        let control = Bucket "tan-nb-exp-cells-control"+            outDir = root </> "routed"+            common = ["cell", "route", "--cell", "beta", "--cell", "alpha", "--control-bucket", "tan-nb-exp-cells-control", "--plan", "../kenshou-core/test/golden/run-plan.minimal.json"]+        _ <- store.putObject control (ObjectName "cells/beta/descriptor.json") "application/json" DoesNotExist (Aeson.encode (observed {name = "beta", postgresMajor = 17}))+        _ <- store.putObject control (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist descriptor+        withCellStore root do+          runWithArgs (common <> ["--out", outDir, "--coerce-durable"]) `shouldReturnCode` ExitSuccess+          doesFileExist (outDir </> "plan.alpha.json") `shouldReturn` True+          doesFileExist (outDir </> "plan.beta.json") `shouldReturn` False+          report <- Aeson.eitherDecodeFileStrict' (outDir </> "unroutable.json") >>= either fail pure+          case report of+            Aeson.Object fields -> KeyMap.lookup "schema" fields `shouldBe` Just (Aeson.String "kenshou.cell-route/v1")+            _ -> expectationFailure "expected route report"+          runWithArgs (common <> ["--out", outDir, "--coerce-durable"]) `shouldReturnCode` ExitFailure 2+          runWithArgs (common <> ["--out", root </> "refused"]) `shouldReturnCode` ExitFailure 2+          doesFileExist (root </> "refused" </> "unroutable.json") `shouldReturn` True++    it "uses a matching capability cache and ignores one from an old descriptor" $+      withSystemTempDirectory "kenshou-cell-capability-route" \root -> do+        store <- newFileStore root+        descriptorBytes <- LazyByteString.readFile "../kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+        descriptor <- either fail pure (Aeson.eitherDecode descriptorBytes :: Either String CellDescriptor)+        probeId <- newRunId+        now <- getCurrentTime+        let control = Bucket "tan-nb-exp-cells-control"+            directory = root </> "capabilities"+            rulesPath = root </> "rules.json"+            cachePath = directory </> "alpha.capabilities.json"+            command outDir = ["cell", "route", "--cell", "alpha", "--control-bucket", "tan-nb-exp-cells-control", "--plan", "../kenshou-core/test/golden/run-plan.minimal.json", "--routing-rules", rulesPath, "--out", outDir, "--coerce-durable"]+            cache digest = object ["schema" .= ("kenshou.cell-capabilities/v1" :: Text), "cell" .= ("alpha" :: Text), "descriptorSha256" .= digest, "probedBy" .= renderRunId probeId, "probedAt" .= now, "capabilities" .= object ["postgres.pg_partman" .= False]]+            rule = object ["when" .= object ["scenario" .= ("selftest/kernel/**" :: Text)], "requires" .= ("postgres.pg_partman" :: Text), "why" .= ("requires extension" :: Text)]+        createDirectoryIfMissing True directory+        LazyByteString.writeFile rulesPath (Aeson.encode [rule])+        _ <- store.putObject control (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist descriptorBytes+        withCellStore root $ withCapabilityDir directory do+          LazyByteString.writeFile cachePath (Aeson.encode (cache (descriptorDigest descriptor)))+          runWithArgs (command (root </> "denied")) `shouldReturnCode` ExitFailure 2+          doesFileExist (root </> "denied" </> "plan.alpha.json") `shouldReturn` False+          LazyByteString.writeFile cachePath (Aeson.encode (cache (Text.replicate 64 "0")))+          runWithArgs (command (root </> "unprobed")) `shouldReturnCode` ExitSuccess+          doesFileExist (root </> "unprobed" </> "plan.alpha.json") `shouldReturn` True++    it "prepares a dry-run submission under an existing file-backed lease" $+      withSystemTempDirectory "kenshou-cell-submit" \root -> do+        store <- newFileStore root+        descriptorBytes <- LazyByteString.readFile "../kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+        observed <- either fail pure (Aeson.eitherDecode descriptorBytes :: Either String CellDescriptor)+        let control = Bucket "control"+            descriptor = observed {buckets = observed.buckets {control = "control"}}+            location = ["--cell", "alpha", "--control-bucket", "control"]+            outDir = root </> "session"+        _ <- store.putObject control (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist (Aeson.encode descriptor)+        withCellStore root do+          runWithArgs (["cell", "lease"] <> location <> ["--purpose", "submit-test"]) `shouldReturnCode` ExitSuccess+          stored <- store.getObject control (ObjectName "cells/alpha/lease.json")+          lease <- case stored of+            Just (bytes, _) -> either fail pure (Aeson.eitherDecode bytes :: Either String Lease)+            Nothing -> expectationFailure "lease was not published" >> error "unreachable"+          let common = ["cell", "submit"] <> location <> ["--lease-id", Text.unpack (renderRunId lease.leaseId), "--payload", "../kenshou-remote/test/golden/payload.json", "--plan", "../kenshou-core/test/golden/run-plan.minimal.json", "--out", outDir, "--dry-run"]+          runWithArgs (common <> ["--coerce-durable"]) `shouldReturnCode` ExitSuccess+          doesPathExist outDir `shouldReturn` False+          runWithArgs common `shouldReturnCode` ExitFailure 2+          template <- Aeson.eitherDecodeFileStrict' "../kenshou-remote/test/golden/cell-session.json" >>= either fail pure+          let complete = (template :: SessionJournal) {store = Text.pack ("file:" <> root), controlBucket = "control", resultsBucket = descriptor.buckets.results, leaseId = lease.leaseId, slices = []}+          createDirectoryIfMissing True outDir+          writeSessionJournal (outDir </> "session.json") complete+          runWithArgs ["cell", "resume", "--session", outDir] `shouldReturnCode` ExitSuccess++    it "releases a one-command lease when plan preparation fails" $+      withSystemTempDirectory "kenshou-cell-run" \root -> do+        store <- newFileStore root+        descriptorBytes <- LazyByteString.readFile "../kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+        observed <- either fail pure (Aeson.eitherDecode descriptorBytes :: Either String CellDescriptor)+        let control = Bucket "control"+            descriptor = observed {buckets = observed.buckets {control = "control"}}+            outDir = root </> "run-session"+            command = ["cell", "run", "--cell", "alpha", "--control-bucket", "control", "--payload", "../kenshou-remote/test/golden/payload.json", "--plan", "../kenshou-core/test/golden/run-plan.minimal.json", "--out", outDir]+        _ <- store.putObject control (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist (Aeson.encode descriptor)+        withCellStore root do+          runWithArgs command `shouldReturnCode` ExitFailure 2+          store.statObject control (ObjectName "cells/alpha/lease.json") `shouldReturn` Nothing+          doesPathExist outDir `shouldReturn` False++    it "checkpoints a rejected one-command submission and releases its lease" $+      withSystemTempDirectory "kenshou-cell-run-rejected" \root -> do+        store <- newFileStore root+        descriptorBytes <- LazyByteString.readFile "../kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+        observed <- either fail pure (Aeson.eitherDecode descriptorBytes :: Either String CellDescriptor)+        let control = Bucket "control"+            descriptor = observed {buckets = observed.buckets {control = "control"}}+            outDir = root </> "run-session"+            command = ["cell", "run", "--cell", "alpha", "--control-bucket", "control", "--payload", "../kenshou-remote/test/golden/payload.json", "--plan", "../kenshou-core/test/golden/run-plan.minimal.json", "--out", outDir, "--coerce-durable"]+            awaitMarker = do+              objects <- store.listObjects control "cells/alpha/submissions/"+              case find (Text.isSuffixOf "/submission.json" . (.unObjectName) . fst) objects of+                Nothing -> threadDelay 10000 >> awaitMarker+                Just (name, _) -> do+                  stored <- store.getObject control name+                  case stored of+                    Nothing -> expectationFailure "submission marker disappeared"+                    Just (bytes, _) -> do+                      submission <- either fail pure (Aeson.eitherDecode bytes :: Either String Submission)+                      now <- getCurrentTime+                      let prefix = "cells/alpha/submissions/" <> renderRunId submission.runId <> "/rejected.json"+                      _ <- store.putObject control (ObjectName prefix) "application/json" DoesNotExist (Aeson.encode (Rejected submission.runId "fixture-rejected" now))+                      pure ()+        _ <- store.putObject control (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist (Aeson.encode descriptor)+        worker <- forkIO awaitMarker+        ( withCellStore root do+            result <- timeout 15000000 (runWithArgs command)+            result `shouldBe` Just (ExitFailure 4)+            store.statObject control (ObjectName "cells/alpha/lease.json") `shouldReturn` Nothing+            journal <- readSessionJournal (outDir </> "session.json") >>= either (fail . Text.unpack) pure+            fmap (.state) journal.slices `shouldBe` [SliceRejected]+          )+          `finally` killThread worker++    it "fetches and verifies a sealed one-command submission" $+      withSystemTempDirectory "kenshou-cell-run-sealed" \root -> do+        store <- newFileStore root+        descriptorBytes <- LazyByteString.readFile "../kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+        observed <- either fail pure (Aeson.eitherDecode descriptorBytes :: Either String CellDescriptor)+        let control = Bucket "control"+            results = Bucket observed.buckets.results+            descriptor = observed {buckets = observed.buckets {control = "control"}}+            outDir = root </> "run-session"+            command = ["cell", "run", "--cell", "alpha", "--control-bucket", "control", "--payload", "../kenshou-remote/test/golden/payload.json", "--plan", "../kenshou-core/test/golden/run-plan.minimal.json", "--out", outDir, "--coerce-durable"]+            awaitMarker = do+              objects <- store.listObjects control "cells/alpha/submissions/"+              case find (Text.isSuffixOf "/submission.json" . (.unObjectName) . fst) objects of+                Nothing -> threadDelay 10000 >> awaitMarker+                Just (name, _) -> do+                  marker <- store.getObject control name+                  submission <- case marker of+                    Just (bytes, _) -> either fail pure (Aeson.eitherDecode bytes :: Either String Submission)+                    Nothing -> fail "submission marker disappeared"+                  journal <- readSessionJournal (outDir </> "session.json") >>= either (fail . Text.unpack) pure+                  nestedId <- case journal.slices of+                    [slice] -> case slice.runIds of+                      [identifier] -> pure identifier+                      _ -> fail "expected one nested run"+                    _ -> fail "expected one cell slice"+                  let prefix = "cells/alpha/submissions/" <> renderRunId submission.runId <> "/"+                  work <- store.getObject control (ObjectName (prefix <> "work"))+                  workBytes <- case work of+                    Just (bytes, _) -> pure bytes+                    Nothing -> fail "submission work disappeared"+                  now <- getCurrentTime+                  let payloadDigest = submission.payload.bundle.sha256+                      nestedBytes = Aeson.encode (object ["schema" .= ("kenshou.run-result/v1" :: Text), "runId" .= nestedId, "scenario" .= ("selftest/kernel/correctness/postgres-roundtrip" :: Text), "outcome" .= Outcome.Passed, "fingerprint" .= object ["cell" .= object ["payload" .= object ["bundleSha256" .= payloadDigest]]]])+                      nestedInfo = workObjectFor "application/json" nestedBytes+                      nestedManifest = Manifest nestedId now [ManifestFile "run-result.json" ("sha256:" <> nestedInfo.sha256) (fromIntegral nestedInfo.bytes) "application/json"]+                      files =+                        [ ("submission/work", workBytes),+                          ("submission/submission.json", Aeson.encode submission),+                          ("cell/result.json", Aeson.encode (CellRunResult submission.runId "alpha" submission.leaseId 1 Completed (Just 0) Nothing [])),+                          ("output/" <> renderRunId nestedId <> "/run-result.json", nestedBytes),+                          ("output/" <> renderRunId nestedId <> "/manifest.json", Aeson.encode nestedManifest)+                        ]+                      artifactFor (path, bytes) = let info = workObjectFor "application/json" bytes in Artifact path info.sha256 info.bytes "application/json"+                      manifest = CellManifest submission.runId "alpha" submission.leaseId 1 now "0.1.0" (ManifestPayload payloadDigest submission.payload.storePath) Completed 3600 (map artifactFor files)+                      manifestBytes = Aeson.encode manifest+                      manifestDigest = (workObjectFor "application/json" manifestBytes).sha256+                  mapM_+                    ( \(path, bytes) -> do+                        _ <- store.putObject results (ObjectName ("runs/" <> renderRunId submission.runId <> "/" <> path)) "application/json" DoesNotExist bytes+                        pure ()+                    )+                    files+                  _ <- store.putObject results (ObjectName ("runs/" <> renderRunId submission.runId <> "/manifest.json")) "application/json" DoesNotExist manifestBytes+                  _ <- store.putObject control (ObjectName (prefix <> "status.json")) "application/json" DoesNotExist (Aeson.encode (CellStatus submission.runId Sealed (Just 1) now Nothing (LogChunks 0 0) (Just Completed) (Just manifestDigest) (Just [])))+                  pure ()+        _ <- store.putObject control (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist (Aeson.encode descriptor)+        worker <- forkIO awaitMarker+        ( withCellStore root do+            result <- timeout 15000000 (runWithArgs command)+            result `shouldBe` Just ExitSuccess+            store.statObject control (ObjectName "cells/alpha/lease.json") `shouldReturn` Nothing+            journal <- readSessionJournal (outDir </> "session.json") >>= either (fail . Text.unpack) pure+            fmap (.state) journal.slices `shouldBe` [SliceVerified]+            case journal.slices of+              [slice] -> doesFileExist (root </> Text.unpack (renderRunId slice.cellRun) </> "cell-run.json") `shouldReturn` True+              _ -> expectationFailure "expected one verified cell slice"+          )+          `finally` killThread worker++    it "runs overhead slots behind one cell lease and links their verified run directories" $+      withSystemTempDirectory "kenshou-cell-overhead" \root -> do+        store <- newFileStore root+        descriptorBytes <- LazyByteString.readFile "../kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+        observed <- either fail pure (Aeson.eitherDecode descriptorBytes :: Either String CellDescriptor)+        let control = Bucket "control"+            results = Bucket observed.buckets.results+            descriptor = observed {buckets = observed.buckets {control = "control"}}+            outRoot = root </> "overhead-output"+            command =+              [ "cell",+                "overhead",+                "selftest/telemetry/benchmark/arms-on-synthetic-service",+                "--cell",+                "alpha",+                "--control-bucket",+                "control",+                "--payload",+                "../kenshou-remote/test/golden/payload.json",+                "--arms",+                "tracing=off,noop",+                "--trials",+                "3",+                "--settle-seconds",+                "0",+                "--retries",+                "0",+                "--policy",+                "../policies/telemetry-overhead.json",+                "--out",+                outRoot+              ]+            awaitMarkers seen = do+              objects <- store.listObjects control "cells/alpha/submissions/"+              case find (\(name, _) -> Text.isSuffixOf "/submission.json" name.unObjectName && name `Set.notMember` seen) objects of+                Nothing -> threadDelay 10000 >> awaitMarkers seen+                Just (name, _) -> do+                  marker <- store.getObject control name+                  submission <- case marker of+                    Just (bytes, _) -> either fail pure (Aeson.eitherDecode bytes :: Either String Submission)+                    Nothing -> fail "overhead submission marker disappeared"+                  let prefix = "cells/alpha/submissions/" <> renderRunId submission.runId <> "/"+                  work <- store.getObject control (ObjectName (prefix <> "work"))+                  workBytes <- case work of+                    Just (bytes, _) -> pure bytes+                    Nothing -> fail "overhead work disappeared"+                  workValue <- either fail pure (Aeson.eitherDecode workBytes :: Either String Value)+                  nestedId <- case workValue of+                    Aeson.Object fields -> case KeyMap.lookup "runs" fields of+                      Just (Aeson.Array entries) -> case toList entries of+                        [Aeson.Object entry] -> case KeyMap.lookup "runId" entry of+                          Just value -> case Aeson.fromJSON value of Aeson.Success identifier -> pure identifier; Aeson.Error problem -> fail problem+                          Nothing -> fail "overhead work has no nested run ID"+                        _ -> fail "overhead work must contain one valid run entry"+                      _ -> fail "overhead work has no runs"+                    _ -> fail "overhead work is invalid"+                  now <- getCurrentTime+                  let sequenceNumber = Set.size seen + 1+                      payloadDigest = submission.payload.bundle.sha256+                      nestedBytes = Aeson.encode (object ["schema" .= ("kenshou.run-result/v1" :: Text), "runId" .= nestedId, "scenario" .= ("selftest/telemetry/benchmark/arms-on-synthetic-service" :: Text), "outcome" .= Outcome.Passed, "fingerprint" .= object ["cell" .= object ["payload" .= object ["bundleSha256" .= payloadDigest]]]])+                      nestedInfo = workObjectFor "application/json" nestedBytes+                      nestedManifest = Manifest nestedId now [ManifestFile "run-result.json" ("sha256:" <> nestedInfo.sha256) (fromIntegral nestedInfo.bytes) "application/json"]+                      files =+                        [ ("submission/work", workBytes),+                          ("submission/submission.json", Aeson.encode submission),+                          ("cell/result.json", Aeson.encode (CellRunResult submission.runId "alpha" submission.leaseId sequenceNumber Completed (Just 0) Nothing [])),+                          ("output/" <> renderRunId nestedId <> "/run-result.json", nestedBytes),+                          ("output/" <> renderRunId nestedId <> "/manifest.json", Aeson.encode nestedManifest)+                        ]+                      artifactFor (path, bytes) = let info = workObjectFor "application/json" bytes in Artifact path info.sha256 info.bytes "application/json"+                      manifest = CellManifest submission.runId "alpha" submission.leaseId sequenceNumber now "0.1.0" (ManifestPayload payloadDigest submission.payload.storePath) Completed 3600 (map artifactFor files)+                      manifestBytes = Aeson.encode manifest+                      manifestDigest = (workObjectFor "application/json" manifestBytes).sha256+                  mapM_ (\(path, bytes) -> do _ <- store.putObject results (ObjectName ("runs/" <> renderRunId submission.runId <> "/" <> path)) "application/json" DoesNotExist bytes; pure ()) files+                  _ <- store.putObject results (ObjectName ("runs/" <> renderRunId submission.runId <> "/manifest.json")) "application/json" DoesNotExist manifestBytes+                  _ <- store.putObject control (ObjectName (prefix <> "status.json")) "application/json" DoesNotExist (Aeson.encode (CellStatus submission.runId Sealed (Just sequenceNumber) now Nothing (LogChunks 0 0) (Just Completed) (Just manifestDigest) (Just [])))+                  if sequenceNumber < 6 then awaitMarkers (Set.insert name seen) else pure ()+        _ <- store.putObject control (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist (Aeson.encode descriptor)+        worker <- forkIO (awaitMarkers Set.empty)+        ( withCellStore root do+            result <- timeout 30000000 (runWithArgs command)+            result `shouldBe` Just (ExitFailure 4)+            store.statObject control (ObjectName "cells/alpha/lease.json") `shouldReturn` Nothing+            [invocation] <- listDirectory outRoot+            state <- loadOverheadState (outRoot </> invocation) >>= either fail pure+            length state.slots `shouldBe` 6+            all (.complete) state.slots `shouldBe` True+            observations <-+              traverse+                ( \slot -> do+                    runId <- case reverse slot.runIds of identifier : _ -> pure identifier; [] -> fail "overhead slot has no run"+                    doesFileExist (outRoot </> invocation </> "runs" </> Text.unpack (renderRunId runId) </> "manifest.json") `shouldReturn` True+                    journal <- readSessionJournal (outRoot </> invocation </> "cell-sessions" </> Text.unpack (renderRunId runId) </> "session.json") >>= either (fail . Text.unpack) pure+                    slice <- case journal.slices of [one] -> pure one; _ -> fail "overhead child must have one slice"+                    sealed <- store.getObject results (ObjectName ("runs/" <> renderRunId slice.cellRun <> "/manifest.json"))+                    manifest <- case sealed of+                      Just (bytes, _) -> either fail pure (Aeson.eitherDecode bytes :: Either String CellManifest)+                      Nothing -> fail "overhead child has no cell manifest"+                    pure (journal.leaseId, manifest.leaseSequence)+                )+                state.slots+            length (Set.fromList (fmap fst observations)) `shouldBe` 1+            List.sort (fmap snd observations) `shouldBe` [1 .. 6]+            resumed <- timeout 15000000 (runWithArgs (command <> ["--resume"]))+            resumed `shouldBe` Just (ExitFailure 4)+            markers <- store.listObjects control "cells/alpha/submissions/"+            length (filter (Text.isSuffixOf "/submission.json" . (.unObjectName) . fst) markers) `shouldBe` 6+          )+          `finally` killThread worker++    it "leaves a detached single-slice lease alive and collects its rejection later" $+      withSystemTempDirectory "kenshou-cell-run-detached" \root -> do+        store <- newFileStore root+        descriptorBytes <- LazyByteString.readFile "../kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+        observed <- either fail pure (Aeson.eitherDecode descriptorBytes :: Either String CellDescriptor)+        let control = Bucket "control"+            descriptor = observed {buckets = observed.buckets {control = "control"}}+            outDir = root </> "run-session"+            leaseObject = ObjectName "cells/alpha/lease.json"+            command = ["cell", "run", "--cell", "alpha", "--control-bucket", "control", "--payload", "../kenshou-remote/test/golden/payload.json", "--plan", "../kenshou-core/test/golden/run-plan.minimal.json", "--out", outDir, "--coerce-durable", "--detach"]+        _ <- store.putObject control (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist (Aeson.encode descriptor)+        withCellStore root do+          runWithArgs command `shouldReturnCode` ExitSuccess+          journal <- readSessionJournal (outDir </> "session.json") >>= either (fail . Text.unpack) pure+          journal.leaseMode `shouldBe` Detached+          fmap (.state) journal.slices `shouldBe` [SliceSubmitted]+          leaseStored <- store.getObject control leaseObject+          lease <- case leaseStored of+            Just (bytes, _) -> either fail pure (Aeson.eitherDecode bytes :: Either String Lease)+            Nothing -> expectationFailure "detached lease was released" >> error "unreachable"+          slice <- case journal.slices of+            [entry] -> pure entry+            _ -> expectationFailure "expected one detached slice" >> error "unreachable"+          lease.ttlSeconds `shouldSatisfy` (>= fromIntegral slice.submission.limits.wallClockSeconds + 600)+          runWithArgs ["cell", "resume", "--session", outDir] `shouldReturnCode` ExitFailure 4+          store.statObject control leaseObject >>= (`shouldSatisfy` (maybe False (const True)))+          now <- getCurrentTime+          let rejectedObject = ObjectName ("cells/alpha/submissions/" <> renderRunId slice.cellRun <> "/rejected.json")+          _ <- store.putObject control rejectedObject "application/json" DoesNotExist (Aeson.encode (Rejected slice.cellRun "fixture-rejected" now))+          runWithArgs ["cell", "resume", "--session", outDir] `shouldReturnCode` ExitFailure 4+          store.statObject control leaseObject `shouldReturn` Nothing+          finished <- readSessionJournal (outDir </> "session.json") >>= either (fail . Text.unpack) pure+          fmap (.state) finished.slices `shouldBe` [SliceRejected]+          runWithArgs command `shouldReturnCode` ExitFailure 2+          store.statObject control leaseObject `shouldReturn` Nothing++    it "starts stopped instances and releases a reattached held lease after a terminal resume" $+      withSystemTempDirectory "kenshou-cell-resume-release" \root -> do+        store <- newFileStore root+        descriptorBytes <- LazyByteString.readFile "../kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+        observed <- either fail pure (Aeson.eitherDecode descriptorBytes :: Either String CellDescriptor)+        template <- Aeson.eitherDecodeFileStrict' "../kenshou-remote/test/golden/cell-session.json" >>= either fail pure+        let control = Bucket "control"+            descriptor = observed {buckets = observed.buckets {control = "control"}}+            outDir = root </> "session"+            leaseObject = ObjectName "cells/alpha/lease.json"+        _ <- store.putObject control (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist (Aeson.encode descriptor)+        withCellStore root do+          runWithArgs ["cell", "lease", "--cell", "alpha", "--control-bucket", "control", "--purpose", "resume-test"] `shouldReturnCode` ExitSuccess+          stored <- store.getObject control leaseObject+          lease <- case stored of+            Just (bytes, _) -> either fail pure (Aeson.eitherDecode bytes :: Either String Lease)+            Nothing -> fail "fixture lease missing"+          let journal = (template :: SessionJournal) {store = Text.pack ("file:" <> root), controlBucket = "control", resultsBucket = descriptor.buckets.results, leaseId = lease.leaseId, leaseMode = Held, slices = map (\slice -> slice {state = SliceRejected, rejectionReason = Just "fixture-rejected", cellOutcome = Nothing, entryExitCode = Nothing, manifestSha256 = Nothing, fetchedPath = Nothing}) template.slices}+          createDirectoryIfMissing True outDir+          writeSessionJournal (outDir </> "session.json") journal+          let binDir = root </> "bin"+              fakeGcloud = binDir </> "gcloud"+              started = root </> "instances-started"+          createDirectoryIfMissing True binDir+          writeFile fakeGcloud "#!/bin/sh\nif [ \"$3\" = describe ]; then echo TERMINATED; else printf started > \"$KENSHOU_TEST_STARTED\"; fi\n"+          permissions <- getPermissions fakeGcloud+          setPermissions fakeGcloud permissions {executable = True}+          oldPath <- lookupEnv "PATH"+          withEnvVariable "PATH" (binDir <> ":" <> maybe "" id oldPath) $+            withEnvVariable "KENSHOU_TEST_STARTED" started $+              runWithArgs ["cell", "resume", "--start", "--session", outDir] `shouldReturnCode` ExitFailure 4+          doesFileExist started `shouldReturn` True+          store.statObject control leaseObject `shouldReturn` Nothing+          runWithArgs ["cell", "resume", "--start", "--help"] `shouldReturnCode` ExitSuccess++    it "rebinds a submitted marker that a prior held lease never accepted" $+      withSystemTempDirectory "kenshou-cell-resume-unaccepted" \root -> do+        store <- newFileStore root+        descriptorBytes <- LazyByteString.readFile "../kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+        observed <- either fail pure (Aeson.eitherDecode descriptorBytes :: Either String CellDescriptor)+        template <- Aeson.eitherDecodeFileStrict' "../kenshou-remote/test/golden/cell-session.json" >>= either fail pure+        let control = Bucket "control"+            descriptor = observed {buckets = observed.buckets {control = "control"}}+            outDir = root </> "session"+            leaseObject = ObjectName "cells/alpha/lease.json"+        _ <- store.putObject control (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist (Aeson.encode descriptor)+        withCellStore root do+          runWithArgs ["cell", "lease", "--cell", "alpha", "--control-bucket", "control", "--purpose", "old-session"] `shouldReturnCode` ExitSuccess+          oldStored <- store.getObject control leaseObject+          oldLease <- case oldStored of+            Just (bytes, _) -> either fail pure (Aeson.eitherDecode bytes :: Either String Lease)+            Nothing -> expectationFailure "old lease missing" >> error "unreachable"+          fixtureSlice <- case (template :: SessionJournal).slices of+            [entry] -> pure entry+            _ -> expectationFailure "expected one fixture slice" >> error "unreachable"+          let fixtureSubmission = fixtureSlice.submission+              fixturePayload = fixtureSubmission.payload+              fixtureBundle = fixturePayload.bundle+              payload = fixturePayload {bundle = fixtureBundle {uri = "gs://control/payloads/sha256/" <> fixtureBundle.sha256 <> ".nar.zst"}}+              oldSubmission = fixtureSubmission {leaseId = oldLease.leaseId, payload = payload, work = workObjectFor "application/json" "{}"}+              oldSlice = fixtureSlice {submission = oldSubmission, state = SliceSubmitted}+              journal = template {store = Text.pack ("file:" <> root), controlBucket = "control", resultsBucket = descriptor.buckets.results, leaseId = oldLease.leaseId, slices = [oldSlice]}+              oldMarker = ObjectName ("cells/alpha/submissions/" <> renderRunId oldSlice.cellRun <> "/submission.json")+              awaitRetry = do+                objects <- store.listObjects control "cells/alpha/submissions/"+                case find (\(name, _) -> name /= oldMarker && Text.isSuffixOf "/submission.json" name.unObjectName) objects of+                  Nothing -> threadDelay 10000 >> awaitRetry+                  Just (name, _) -> do+                    stored <- store.getObject control name+                    submission <- case stored of+                      Just (bytes, _) -> either fail pure (Aeson.eitherDecode bytes :: Either String Submission)+                      Nothing -> fail "retry marker disappeared"+                    now <- getCurrentTime+                    let rejected = ObjectName ("cells/alpha/submissions/" <> renderRunId submission.runId <> "/rejected.json")+                    _ <- store.putObject control rejected "application/json" DoesNotExist (Aeson.encode (Rejected submission.runId "fixture-rejected" now))+                    pure ()+          createDirectoryIfMissing True (outDir </> "slice-0")+          LazyByteString.writeFile (outDir </> "slice-0/work.json") "{}"+          writeSessionJournal (outDir </> "session.json") journal+          _ <- store.putObject control oldMarker "application/json" DoesNotExist (Aeson.encode oldSubmission)+          runWithArgs ["cell", "release", "--cell", "alpha", "--control-bucket", "control", "--lease-id", Text.unpack (renderRunId oldLease.leaseId)] `shouldReturnCode` ExitSuccess+          worker <- forkIO awaitRetry+          ( do+              result <- timeout 15000000 (runWithArgs ["cell", "resume", "--session", outDir])+              result `shouldBe` Just (ExitFailure 4)+              finished <- readSessionJournal (outDir </> "session.json") >>= either (fail . Text.unpack) pure+              fmap (.state) finished.slices `shouldBe` [SliceRejected]+              fmap (.cellRun) finished.slices `shouldSatisfy` (/= [oldSlice.cellRun])+              store.statObject control leaseObject `shouldReturn` Nothing+            )+            `finally` killThread worker++    it "reads scenario history from a bundle" do+      runWithArgs ["history", "--bundle", "../fixtures/consumer-baseline/docs/verification", "--bundle-uri", "mori://shinzui/kenshou/okf/verification", "--scenario", "selftest/kernel/correctness/always-pass", "--json"] `shouldReturnCode` ExitSuccess+      runWithArgs ["history", "--bundle", "../fixtures/consumer-baseline/docs/verification", "--scenario", "selftest/kernel/correctness/always-pass", "--json"] `shouldReturnCode` ExitFailure 2++    it "requires one record source" do+      runWithArgs ["record", "--purpose", "release"] `shouldReturnCode` ExitFailure 2+      runWithArgs ["record", "a-run", "--comparison", "comparison.json", "--purpose", "release"] `shouldReturnCode` ExitFailure 2++    it "rejects a human anomaly exception without an interactive terminal" do+      runWithArgs ["attest", "invalid", "--project", "fixture", "--accept-anomaly", "--authority", "human:fixture", "--reason", "fixture"] `shouldReturnCode` ExitFailure 2++    it "returns 2 for an unknown subcommand" do+      runWithArgs ["cohort", "bogus"] `shouldReturnCode` ExitFailure 2++    it "returns 1 when offline leak diagnosis finds growth" do+      runWithArgs ["diagnose", "leak", leakingRun] `shouldReturnCode` ExitFailure 1++    it "returns 1 when an offline stall diagnosis exists" do+      runWithArgs ["diagnose", "stall", stalledRun] `shouldReturnCode` ExitFailure 1++    it "returns 0 when no offline stall diagnosis exists" do+      runWithArgs ["diagnose", "stall", leakingRun] `shouldReturnCode` ExitSuccess++    it "returns 3 when leak evidence is insufficient" do+      runWithArgs ["diagnose", "leak", stalledRun] `shouldReturnCode` ExitFailure 3++    it "returns 4 when diagnosis input is unavailable" do+      runWithArgs ["diagnose", "leak", "test/fixtures/does-not-exist"] `shouldReturnCode` ExitFailure 4++    it "returns 2 for invalid diagnose syntax" do+      runWithArgs ["diagnose", "leak", "--bogus"] `shouldReturnCode` ExitFailure 2++    it "does not mutate a sealed run manifest" do+      before <- Text.readFile (leakingRun <> "/manifest.json")+      _ <- runWithArgs ["diagnose", "leak", leakingRun]+      after <- Text.readFile (leakingRun <> "/manifest.json")+      after `shouldBe` before++  describe "version" do+    it "includes the cabal package version and a short revision" do+      appVersionWithGit `shouldSatisfy` Text.isPrefixOf "kenshou v0.1.0.0 ("+      Text.dropAround (`elem` ['(', ')']) (Text.takeWhileEnd (/= ' ') appVersionWithGit)+        `shouldSatisfy` (\revision -> revision == "dirty" || Text.length revision == 7)++runWithArgs :: [String] -> IO ExitCode+runWithArgs = runWithArgsWith selfTestAssembly++-- Runs from the consumer-baseline bridge with the consumer's component graph, for the+-- perf planning checks whose request names the consumer's packages. Called from the+-- repository root.+withConsumerBaseline :: [String] -> IO ExitCode+withConsumerBaseline arguments = do+  graph <- ByteString.readFile "fixtures/consumer-baseline/kenshou-core/data/components.json"+  withCurrentDirectory "fixtures/consumer-baseline" (runWithArgsWith selfTestAssembly {componentGraph = graph} arguments)++shouldReturnCode :: IO ExitCode -> ExitCode -> IO ()+shouldReturnCode action expected = action >>= (`shouldBe` expected)++leakingRun :: FilePath+leakingRun = "../kenshou-diagnose/test/fixtures/run-leaking"++stalledRun :: FilePath+stalledRun = "../kenshou-diagnose/test/fixtures/run-stalled"++withCellStore :: FilePath -> IO value -> IO value+withCellStore root operation = bracket (lookupEnv "KENSHOU_CELL_STORE") (restore "KENSHOU_CELL_STORE") \_ ->+  bracket (lookupEnv "KENSHOU_GCP_ALLOWED_PROJECTS") (restore "KENSHOU_GCP_ALLOWED_PROJECTS") \_ -> do+    setEnv "KENSHOU_CELL_STORE" ("file:" <> root)+    setEnv "KENSHOU_GCP_ALLOWED_PROJECTS" "tan-nb-exp"+    operation+  where+    restore name Nothing = unsetEnv name+    restore name (Just prior) = setEnv name prior++withCapabilityDir :: FilePath -> IO value -> IO value+withCapabilityDir directory operation = bracket (lookupEnv "KENSHOU_CELL_CAPABILITIES_DIR") restore \_ -> do+  setEnv "KENSHOU_CELL_CAPABILITIES_DIR" directory+  operation+  where+    restore Nothing = unsetEnv "KENSHOU_CELL_CAPABILITIES_DIR"+    restore (Just prior) = setEnv "KENSHOU_CELL_CAPABILITIES_DIR" prior++withEnvVariable :: String -> String -> IO value -> IO value+withEnvVariable name selected operation = bracket (lookupEnv name) restore \_ -> do+  setEnv name selected+  operation+  where+    restore Nothing = unsetEnv name+    restore (Just prior) = setEnv name prior++withPerfRoot :: IO a -> IO a+withPerfRoot operation = do+  exists <- doesFileExist "cabal.project"+  withCurrentDirectory (if exists then "." else "..") operation++replacePerfField :: Key.Key -> Value -> Value -> Value+replacePerfField key value (Aeson.Object fields) = Aeson.Object (KeyMap.insert key value fields)+replacePerfField _ _ value = value++-- The request's own target components plus one component of its cohort template+-- that belongs to a different project, read from the consumer-baseline bridge.+foreignTargets :: Value -> IO [Text]+foreignTargets request = do+  let field name (Aeson.Object fields) = KeyMap.lookup name fields+      field _ _ = Nothing+      texts value = case value of Just (Aeson.Array items) -> [text | Aeson.String text <- toList items]; _ -> []+      owned = texts (field "targetComponents" request)+  template <- case field "cohortTemplate" request of+    Just (Aeson.String path) -> Aeson.eitherDecodeFileStrict' ("fixtures/consumer-baseline/" <> Text.unpack path) >>= either fail pure+    _ -> fail "the request names no cohort template"+  let components = case field "components" template of Just (Aeson.Array items) -> toList items; _ -> []+  case [name | component <- components, Just (Aeson.String name) <- [field "id" component], field "moriUri" component /= field "project" request] of+    other : _ -> pure (owned <> [other])+    [] -> fail "the cohort template has no component from another project"
+ test/PerfSessionSpec.hs view
@@ -0,0 +1,264 @@+module PerfSessionSpec (PerfSessionSpec.spec) where++import Control.Concurrent (forkIO, killThread, threadDelay)+import Control.Exception (bracket, finally)+import Control.Monad (forM, forM_)+import Data.Aeson+import Data.Aeson.KeyMap qualified as KeyMap+import Data.ByteString qualified as Bytes+import Data.ByteString.Lazy qualified as Lazy+import Data.Map.Strict qualified as Map+import Data.Set qualified as Set+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Time (UTCTime, addUTCTime, getCurrentTime)+import Kenshou.Cli (runWithArgsWith, selfTestAssembly)+import Kenshou.Core.Canonical (sha256Hex)+import Kenshou.Core.Id (renderRunId)+import Kenshou.Core.Manifest (writeManifest)+import Kenshou.Core.RunSpec (RunSpec (..))+import Kenshou.Measure.Clock (Origin (..))+import Kenshou.Measure.Histogram qualified as Histogram+import Kenshou.Measure.Histogram.Codec (encodeHistogram)+import Kenshou.Measure.Samples+import Kenshou.Measure.Summary (summarizeRunDir)+import Kenshou.Perf.Artifacts (sealPerfDirectory)+import Kenshou.Perf.Plan (PerfPlan (..), PerfProfile (..), PlannedProfile (..))+import Kenshou.Perf.Prepared qualified as Prepared+import Kenshou.Perf.Result qualified as Result+import Kenshou.Plan.RunPlan (PlannedRun (..), RunPlan (..))+import Kenshou.Remote.Cell.Docs+import Kenshou.Remote.Cell.Lease qualified as Lease+import Kenshou.Remote.Cell.Prepare (cellMachineProfile)+import Kenshou.Remote.Cell.Submit (workObjectFor)+import Kenshou.Remote.Cell.WorkJson (decodeWorkPlan)+import Kenshou.Remote.Payload (Bundle (..), PayloadDescriptor (..))+import Kenshou.Remote.Payload qualified as Payload+import Kenshou.Remote.Perf.Journal qualified as Journal+import Kenshou.Remote.Store+import Kenshou.Remote.Store.File (newFileStore)+import System.Directory (createDirectoryIfMissing, doesFileExist, listDirectory, removeFile, withCurrentDirectory)+import System.Environment (lookupEnv, setEnv, unsetEnv)+import System.Exit (ExitCode (..))+import System.FilePath ((</>))+import System.IO.Temp (withSystemTempDirectory)+import System.Timeout (timeout)+import Test.Hspec++runWithArgs :: [String] -> IO ExitCode+runWithArgs = runWithArgsWith selfTestAssembly++spec :: Spec+-- The prepared fixture pins scenarios of a consumer layer that this repository does not+-- register, so dispatch cannot resolve them. The kernel plan that regenerates the perf+-- fixtures from synthetic scenarios removes this guard:+-- docs/plans/2-open-the-kernel-layer-identity-and-schema-composition-to-registered-suites.md (M4).+spec = describe "file-backed performance CLI integration" $ before_ (pendingWith "perf-prepared fixture names unregistered consumer scenarios; regenerated in docs/plans/2-… M4") do+  forM_ [(False, "executes two profiles under one lease and replays the full sealed graph without redispatch"), (True, "retains an exit-zero known defect through complete report replay and resume")] \(knownFault, title) -> it title $+    withSystemTempDirectory "perf-cli-healthy" \root -> projectRoot do+      let preparation = root </> "preparation"+          output = root </> "session"+          storeRoot = root </> "store"+          payloadBytes = "synthetic protocol payload; no runtime calibration is claimed"+          digest = Text.drop 7 (sha256Hex (Lazy.toStrict payloadBytes))+          bundle = Bundle ("gs://tan-nb-exp-cells-control/payloads/sha256/" <> digest <> ".nar.zst") digest (Lazy.length payloadBytes)+          replacePayload arm = case fromJSON arm.payload of+            Error problem -> fail problem+            Success payload -> pure arm {Prepared.payload = toJSON (payload {Payload.cell = payload.cell {Payload.bundle = bundle}} :: PayloadDescriptor)}+      original <- readJSON "fixtures/consumer-baseline/kenshou-core/test/golden/perf-prepared.fixture.json"+      baseline <- replacePayload (original :: Prepared.PerfPrepared).baseline+      candidate <- replacePayload original.candidate+      let prepared = original {Prepared.baseline = baseline, Prepared.candidate = candidate}+          -- The captured preparation's profiles, in selection order.+          profileNames = [entry.profile.name | entry <- prepared.plan.selected]+          (firstProfile, lastProfile) = case profileNames of+            first : rest -> (Text.unpack first, last (first : rest))+            [] -> error "the captured preparation selects no profile"+      createDirectoryIfMissing True preparation+      encodeFile (preparation </> "prepared.json") prepared+      sealPerfDirectory preparation >>= either (fail . Text.unpack) pure+      descriptor <- readJSON "kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+      policy <- Lazy.readFile "kenshou-remote/test/golden/cell/cell.policy.v1.json"+      store <- newFileStore storeRoot+      let control = Bucket descriptor.buckets.control+          put bucket name bytes = store.putObject bucket (ObjectName name) "application/json" DoesNotExist bytes >>= requireWritten+      put control ("payloads/sha256/" <> digest <> ".nar.zst") payloadBytes+      put control "cells/alpha/descriptor.json" (encode descriptor)+      put control "cells/alpha/policy.json" policy+      agent <- forkIO (sealMarkers knownFault store descriptor root Set.empty)+      ( withSetting "KENSHOU_CELL_STORE" ("file:" <> storeRoot) $+          withSetting "KENSHOU_CELL_CAPABILITIES_DIR" (root </> "capabilities") do+            execution <- timeout 90000000 (runWithArgs ["perf", "run", "--prepared", preparation </> "prepared.json", "--out", output, "--evidence-base-uri", "gs://fixture-evidence/performance"])+            let expectedExit = if knownFault then ExitFailure 1 else ExitSuccess+            execution `shouldBe` Just expectedExit+            result <- readJSON (output </> "result.json")+            (result :: Result.PerfResult).performanceVerdict `shouldBe` "pass"+            result.gatingEligible `shouldBe` False+            length result.configurations `shouldBe` 2+            map (.validPairs) result.configurations `shouldBe` [5, 5]+            null result.blockingRuntimeFailures `shouldBe` not knownFault+            store.getObject control (ObjectName "cells/alpha/lease.json") `shouldReturn` Nothing+            initialMarkers <- store.listObjects control "cells/alpha/submissions/"+            let markers = filter (Text.isSuffixOf "/submission.json" . (.unObjectName) . fst) initialMarkers+            length markers `shouldBe` 20+            submissions <- forM markers \(name, _) -> do+              retained <- store.getObject control name+              maybe (fail "marker vanished") (either fail pure . eitherDecode . fst) retained+            Set.size (Set.fromList [submission.leaseId | submission :: Submission <- submissions]) `shouldBe` 1+            retainedResult <- Lazy.readFile (output </> "result.json")+            forM_ [[], ["--offline"]] \flags -> runWithArgs (["perf", "verify", "--session", output] <> flags) `shouldReturn` expectedExit+            runWithArgs ["perf", "resume", "--session", output] `shouldReturn` expectedExit+            Lazy.readFile (output </> "result.json") `shouldReturn` retainedResult+            store.listObjects control "cells/alpha/submissions/" `shouldReturn` initialMarkers+            -- A second snapshot is forbidden when completed evidence is reused.+            snapshots <- listDirectory (output </> "snapshots")+            length snapshots `shouldBe` 1+            report <- Bytes.readFile (output </> "report.md")+            report `shouldSatisfy` Bytes.isInfixOf "within policy limits"+            report `shouldSatisfy` Bytes.isInfixOf "Comparison reasons: none"+            -- Reconstruct an interruption after the last comparison file was+            -- saved, before its attempt result and root completion checkpoint.+            -- Every sealed cell object and paired schedule stays unchanged.+            saved <- Journal.readPerfSession (output </> "session.json") >>= either (fail . Text.unpack) pure+            let beforeCompletion = takeWhile (\event -> case event.action of Journal.ComparisonCompleted item -> item.profile /= lastProfile; _ -> True) saved.events+                attempt = output </> "configurations" </> Text.unpack lastProfile </> "attempt-0"+            identity <- case [one | event <- beforeCompletion, Journal.LeaseAcquired one <- [event.action]] of+              [one] -> pure one+              _ -> fail "expected exactly one original lease"+            beforeComparison <- Lazy.readFile (attempt </> "comparison.json")+            -- Fixture construction only: the production writer correctly+            -- refuses to rewind an already committed append-only history.+            encodeFile (output </> "session.json") saved {Journal.events = beforeCompletion}+            removeFile (output </> "receipt.json")+            removeFile (attempt </> "result.json")+            Lazy.writeFile (attempt </> "result.json.next") "interrupted JSON write"+            now <- getCurrentTime+            let activeLease = Lease.Lease identity.leaseId identity.cell "kenshou-perf" "pinned runtime performance validation" 180 now now False 20+            put control "cells/alpha/lease.json" (encode activeLease)+            runWithArgs ["perf", "resume", "--session", output] `shouldReturn` expectedExit+            Lazy.readFile (attempt </> "comparison.json") `shouldReturn` beforeComparison+            store.listObjects control "cells/alpha/submissions/" `shouldReturn` initialMarkers+            forM_ [[], ["--offline"]] \flags -> runWithArgs (["perf", "verify", "--session", output] <> flags) `shouldReturn` expectedExit+            -- Even a locally resealed inventory and matching receipt cannot+            -- make an unlinked cell tree part of the interpreted evidence.+            receipt <- readJSON (output </> "receipt.json")+            fields <- case receipt of Object value -> pure value; _ -> fail "invalid receipt"+            relative <- case KeyMap.lookup "snapshot" fields of Just (String value) -> pure value; _ -> fail "receipt omits snapshot"+            let snapshot = output </> Text.unpack relative+                orphan = snapshot </> "configurations" </> firstProfile </> "attempt-0/trials/unlinked-cell/tree"+            createDirectoryIfMissing True orphan+            encodeFile (orphan </> "run-result.json") (object ["outcome" .= ("passed" :: Text)])+            sealPerfDirectory snapshot >>= either (fail . Text.unpack) pure+            changedManifest <- Bytes.readFile (snapshot </> "manifest.json")+            encodeFile (output </> "receipt.json") (Object (KeyMap.insert "manifestSha256" (String (sha256Hex changedManifest)) fields))+            runWithArgs ["perf", "verify", "--session", output, "--offline"] `shouldReturn` ExitFailure 4+        )+        `finally` killThread agent++-- Implements only the immutable protocol in a file store. Its measurements and+-- machine samples are synthetic controls, never acceptance calibration evidence.+sealMarkers :: Bool -> ObjectStore -> CellDescriptor -> FilePath -> Set.Set ObjectName -> IO ()+sealMarkers knownFault store descriptor root seen = do+  let control = Bucket descriptor.buckets.control+  objects <- store.listObjects control "cells/alpha/submissions/"+  let fresh = [name | (name, _) <- objects, "/submission.json" `Text.isSuffixOf` name.unObjectName, name `Set.notMember` seen]+  forM_ fresh \name -> do+    (markerBytes, _) <- store.getObject control name >>= maybe (fail "missing marker") pure+    submission <- either fail pure (eitherDecode markerBytes)+    let prefix = Text.dropEnd (Text.length "submission.json") name.unObjectName+    (workBytes, _) <- store.getObject control (ObjectName (prefix <> "work")) >>= maybe (fail "missing work") pure+    document <- either fail pure (eitherDecode workBytes)+    plan <- either (fail . Text.unpack) pure (decodeWorkPlan document)+    run <- case plan.runs of [one] -> pure one; _ -> fail "expected exactly one paired trial"+    let local = root </> "agent-output" </> Text.unpack (renderRunId submission.runId)+        nested = local </> "output" </> Text.unpack (renderRunId run.runId)+    writeRawRun (knownFault && Set.null seen) descriptor submission run nested+    nestedFiles <- readTree (local </> "output")+    now <- getCurrentTime+    let sequenceNumber = Set.size seen + length (takeWhile (/= name) fresh) + 1+        files =+          [("submission/work", workBytes), ("submission/submission.json", markerBytes), ("cell/result.json", encode (CellRunResult submission.runId "alpha" submission.leaseId sequenceNumber Completed (Just 0) Nothing []))]+            <> healthFiles descriptor submission+            <> [("output/" <> path, bytes) | (path, bytes) <- nestedFiles]+        artifactFor (path, bytes) = let info = workObjectFor "application/json" bytes in Artifact path info.sha256 info.bytes "application/json"+        manifest = CellManifest submission.runId "alpha" submission.leaseId sequenceNumber now "0.1.0" (ManifestPayload submission.payload.bundle.sha256 submission.payload.storePath) Completed 3600 (map artifactFor files)+        manifestBytes = encode manifest+        put bucket path bytes = store.putObject bucket (ObjectName path) "application/json" DoesNotExist bytes >>= requireWritten+        resultPrefix = "runs/" <> renderRunId submission.runId <> "/"+    forM_ files \(path, bytes) -> put (Bucket descriptor.buckets.results) (resultPrefix <> path) bytes+    put (Bucket descriptor.buckets.results) (resultPrefix <> "manifest.json") manifestBytes+    put control (prefix <> "status.json") (encode (CellStatus submission.runId Sealed (Just sequenceNumber) now Nothing (LogChunks 0 0) (Just Completed) (Just (workObjectFor "application/json" manifestBytes).sha256) (Just [])))+  threadDelay 10000+  sealMarkers knownFault store descriptor root (Set.union seen (Set.fromList fresh))++writeRawRun :: Bool -> CellDescriptor -> Submission -> PlannedRun -> FilePath -> IO ()+writeRawRun knownFault descriptor submission run directory = do+  createDirectoryIfMissing True (directory </> "samples")+  createDirectoryIfMissing True (directory </> "series")+  Bytes.writeFile (directory </> "series/load.csv") "mono_ns,wall,phase\n1000,0,steady\n5000001000,0,drain\n"+  encodeFile (directory </> "run-spec.json") run.spec+  mutable <- Histogram.newHistogram Histogram.defaultHistogramConfig+  forM_ [1 :: Int .. 1000] (const (Histogram.recordValue mutable 1000000))+  histogram <- Histogram.freeze mutable+  let operation = "publish-to-finalize"+      base = directory </> "samples" </> Text.unpack operation+  Bytes.writeFile (base <> ".hist") (encodeHistogram histogram)+  Bytes.writeFile (base <> ".service.hist") (encodeHistogram histogram)+  writer <- openSampleWriter (base <> ".raw") (SampleHeader (Origin 0 0) MonotonicClock operation "synthetic")+  writeSampleBlock writer 0 [SampleRecord start start (start + 1000000) 0 1 | index <- [0 .. 999], let start = 1000 + index * 5000000]+  closeSampleWriter writer+  encodeFile (base <> ".meta.json") (object ["operation" .= operation, "latencyBasis" .= ("intended-start" :: Text), "rawSamples" .= ("full" :: Text), "phaseCounts" .= object ["steady" .= object ["successes" .= (1000 :: Int), "failures" .= (0 :: Int), "units" .= (1000 :: Int)]]])+  measurements <- summarizeRunDir directory >>= either (fail . show) pure+  encodeFile (directory </> "run-result.json") (object ["runId" .= run.runId, "scenario" .= run.spec.scenario, "outcome" .= (if knownFault then "failed" else "passed" :: Text), "exitCode" .= (0 :: Int), "blocking" .= False, "compatibility" .= object ["inputs" .= object ["cohortPlanHash" .= ("synthetic-control" :: Text)]], "fingerprint" .= object ["machineProfile" .= cellMachineProfile descriptor, "cell" .= object ["payload" .= object ["bundleSha256" .= submission.payload.bundle.sha256]]], "timings" .= object ["startedAt" .= addUTCTime (fromIntegral run.ordinal) (at 0)], "summaries" .= object ["measurements" .= object ["measurements" .= measurements]]])+  writeManifest directory run.runId Map.empty >>= encodeFile (directory </> "manifest.json")++healthFiles :: CellDescriptor -> Submission -> [(Text, Lazy.ByteString)]+healthFiles descriptor submission =+  [ ("cell/execution-window.json", encode (object ["schema" .= ("cell.execution-window/v1" :: Text), "runId" .= submission.runId, "resetStarted" .= at 0, "runStarted" .= at 10, "runFinished" .= at 60])),+    ("cell/health.json", encode (object ["schema" .= ("cell.health/v1" :: Text), "runId" .= submission.runId, "passed" .= True, "windowStart" .= at 0, "windowEnd" .= at 60, "gates" .= concatMap gates roles])),+    ("cell/reset-evidence.json", encode (object ["schema" .= ("cell.reset-evidence/v1" :: Text), "runId" .= submission.runId, "cell" .= ("alpha" :: Text), "cachePolicy" .= ("cold" :: Text), "verified" .= True, "startedAt" .= at 0, "finishedAt" .= at 5, "steps" .= [object ["name" .= ("stop-driver-processes" :: Text), "status" .= ("ok" :: Text), "startedAt" .= at 0, "finishedAt" .= at 5]]]))+  ]+    <> [("cell/observations/" <> role <> ".json", encode (object ["schema" .= ("cell.observations/v1" :: Text), "machine" .= machine, "image" .= image, "agentVersion" .= ("0.1.0" :: Text), "systemClosure" .= ("/nix/store/synthetic-system-identity" :: Text), "from" .= at 0, "to" .= at 60, "samples" .= [sample i | i <- [0 .. 60]], "maintenanceEvents" .= ([] :: [Value]), "samplingErrors" .= ([] :: [Value])])) | (role, machine, image) <- roles]+  where+    roles = [("driver", machine, descriptor.images.driver) | machine <- take 1 descriptor.instances.drivers] <> [("postgres", descriptor.instances.postgres, descriptor.images.postgres), ("monitoring", descriptor.instances.monitoring, descriptor.images.monitoring)]+    sample i = object ["at" .= at i, "cpuTotalTicks" .= (1000 + i * 100), "cpuIdleTicks" .= (990 + i * 99), "cpuStealTicks" .= (0 :: Int), "diskFreePercent" .= (if i <= 10 then 80 else 75 :: Int)]+    gates (role, machine, _) =+      [ gate "host-maintenance" (String "NONE") (String "NONE"),+        gate "background-load" (object ["settleCpuPercent" .= (if role == "monitoring" then 0 else 1 :: Int), "runStealCpuPercent" .= (0 :: Int)]) (object ["settleCpuMaxPercent" .= (5 :: Int), "runStealCpuMaxPercent" .= (2 :: Int)]),+        gate "storage-pressure" (object ["startFreePercent" .= (80 :: Int), "runFreePercent" .= (75 :: Int), "outputExceeded" .= False]) (object ["startFreeMinPercent" .= (20 :: Int), "runFreeMinPercent" .= (5 :: Int), "outputExceeded" .= False]),+        gate "incomplete-reset" (Bool True) (Bool True),+        gate "version-mismatch" (Bool True) (Bool True)+      ]+      where+        gate name observed threshold = object ["name" .= (name :: Text), "machine" .= machine, "status" .= ("ok" :: Text), "at" .= at 60, "observed" .= observed, "threshold" .= threshold]++at :: Int -> UTCTime+at seconds = addUTCTime (fromIntegral seconds) (read "2026-10-11 00:00:00 UTC")++readTree :: FilePath -> IO [(Text, Lazy.ByteString)]+readTree directory = do+  names <- listDirectory directory+  concat <$> forM names \name -> do+    let path = directory </> name+    regular <- doesFileExist path+    if regular+      then do bytes <- Lazy.readFile path; pure [(Text.pack name, bytes)]+      else map (\(relative, bytes) -> (Text.pack name <> "/" <> relative, bytes)) <$> readTree path++readJSON :: (FromJSON a) => FilePath -> IO a+readJSON path = eitherDecodeFileStrict' path >>= either fail pure++requireWritten :: PutOutcome -> IO ()+requireWritten (Written _) = pure ()+requireWritten PreconditionFailed = fail "synthetic protocol object already exists"++projectRoot :: IO a -> IO a+projectRoot action = do+  root <- doesFileExist "cabal.project"+  withCurrentDirectory (if root then "." else "..") action++withSetting :: String -> String -> IO a -> IO a+withSetting name value action = bracket (lookupEnv name) restore \_ -> setEnv name value >> action+  where+    restore Nothing = unsetEnv name+    restore (Just previous) = setEnv name previous
+ test/fixtures/perf-publication.synthetic.json view
@@ -0,0 +1,11 @@+{+  "schema": "kenshou.perf-publication/v1",+  "snapshot": "snapshots/00000000-0000-7000-8000-000000000000",+  "manifestUri": "gs://synthetic-evidence/performance/sessions/00000000-0000-7000-8000-000000000000/snapshots/00000000-0000-7000-8000-000000000000/manifest.json",+  "manifestSha256": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",+  "resultSha256": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",+  "payloadsReadyAt": "2026-10-11T00:00:00Z",+  "publicationStartedAt": "2026-10-11T00:00:05Z",+  "verifiedReportAt": "2026-10-11T00:00:10Z",+  "totalElapsedSeconds": 10+}
+ test/golden/help-evidence-80.txt view
@@ -0,0 +1,75 @@+# Verification evidence++`kenshou record RUN-DIR --data-base-uri gs://BUCKET/PREFIX --purpose release+--project PROJECT` publishes a finished run's files and adds one record to the+OKF verification bundle. The record names the run, cohort, outcome, and+SHA-256-pinned object links. It contains no measurements; those remain in the+linked run files.++`kenshou record --comparison FILE --data-base-uri gs://BUCKET/PREFIX --purpose+release --project PROJECT` records a `kenshou.comparison/v1` document. Every+baseline and candidate run named by the comparison must already have a run+record in the bundle. The comparison record links the source document by digest+and references those run concepts. Supply either `RUN-DIR` or `--comparison+FILE`.++The default bundle is `docs/verification`. `--bundle DIR` selects another+bundle. A real GCS upload requires `--project PROJECT`, and the command checks+that the active gcloud project matches it. `--verify-only` checks that all+objects already exist; `--deep-verify` downloads them and checks their bytes.+Failed or inconclusive runs link logs automatically; `--link-logs` also links+logs for a passing run.++`--config PATH` reads ordered YAML configuration. The supported keys are+`evidence.bundle-root`, `gcp.project`, and `evidence.data-base-uri`; the+corresponding environment variables are `KENSHOU_EVIDENCE_BUNDLE`,+`KENSHOU_GCP_PROJECT`, and `KENSHOU_EVIDENCE_DATA_BASE_URI`. Named flags take+precedence over the environment and files. `kenshou evidence check` uses the+same bundle default and configuration sources. Unknown keys are rejected.++A dirty harness requires `--allow-dirty` and is recorded with `purpose:+investigation` regardless of the requested purpose. Use `--subject MORI-URI`+with `--subject-kind project|package` when the default layer project is too+broad. Repeat `--produced MORI-URI` for reports caused by this run.++For a local rehearsal, copy the verification bundle outside the repository and+pass `--bundle COPY --store-root DIR`. The record still contains `gs://` URIs,+while the bytes are stored under the scratch directory. The command refuses+`--store-root` with a bundle inside this repository.++`--json` prints one `kenshou.record-result/v1` document and sends errors to+standard error. A repeat with the same record content succeeds and reports+`already recorded`; different content at the same record path is a conflict.++`kenshou evidence check` checks record paths, IDs, digest and revision shapes,+textual fields, event-only keys, timestamps, required data links, comparison arm+references and verdicts, and changes to committed records. `--base REF` limits+the Git history span. `--network --project PROJECT` checks object size and+SHA-256 metadata, and `--deep` downloads bytes to verify each digest. The+command exits 0 when clean, 1 for findings, and 4 when it cannot check. `--json`+prints one `kenshou.evidence-check/v1` document.++`kenshou history --scenario LAYER/COMPONENT/KIND/NAME --json` reads matching+records in time order and emits `kenshou.evidence-history/v1`. Use+`--cohort-component PACKAGE[=VERSION|REVISION]`, repeatable `--outcome`,+`--since YYYY-MM-DD`, or `--confirmed-only` to narrow the series. Each entry+includes linked attestations, a derived trust tier, and the latest earlier+confirmed baseline with the same scenario and compatibility key. The history+view does not modify the bundle.++`kenshou attest RECORD-OR-ID --project PROJECT` fetches the recorded run or+comparison data and writes an immutable attestation with six named checks. For a+comparison, it verifies both recorded arms and replays the paired algorithm from+their samples under the stored policy. `--offline` skips revision resolution+that needs network access; `--linked-only` checks only directly linked objects.+It reports `refuted` when fetched data contradicts the record and `incomplete`+when a check cannot establish the claim. A scenario oracle must be registered+before `verdict-recomputed` can pass for an ordinary run, so generic runs+currently remain incomplete. Exit codes are 0 for confirmed, 1 for refuted, 3+for incomplete, 2 for invalid input, and 4 for an operational failure. `--json`+emits one `kenshou.attest-result/v1` result document.++A person may annotate an incomplete or refuted attestation with+`--accept-anomaly --authority human:ID --reason TEXT` from an interactive+terminal. The computed verdict does not change. The command rejects this action+in CI or without a terminal.