kenshou-cli (empty) → 0.1.0.0
raw patch · 39 files changed
+6034/−0 lines, 39 filesdep +aesondep +basedep +bytestring
Dependencies added: aeson, base, bytestring, containers, directory, file-embed, filepath, githash, hasql, hspec, kenshou-check, kenshou-cli, kenshou-core, kenshou-diagnose, kenshou-evidence, kenshou-measure, kenshou-remote, kenshou-telemetry, optparse-applicative, process, random, settei, settei-env, settei-optparse-applicative, temporary, terminal-size, text, time, uuid
Files
- CHANGELOG.md +12/−0
- LICENSE +28/−0
- help/cells.md +132/−0
- help/comparisons.md +20/−0
- help/diagnostics.md +24/−0
- help/evidence.md +23/−0
- help/exit-codes.md +3/−0
- help/outcomes.md +3/−0
- help/planning.md +20/−0
- help/run-specs.md +3/−0
- help/scenarios.md +5/−0
- help/selectors.md +3/−0
- kenshou-cli.cabal +125/−0
- src/Kenshou/Cli.hs +14/−0
- src/Kenshou/Cli/Attest.hs +101/−0
- src/Kenshou/Cli/Cohort.hs +93/−0
- src/Kenshou/Cli/Command/Cell.hs +1208/−0
- src/Kenshou/Cli/Command/Cohort.hs +15/−0
- src/Kenshou/Cli/Command/Execute.hs +64/−0
- src/Kenshou/Cli/Command/List.hs +87/−0
- src/Kenshou/Cli/Command/Overhead.hs +204/−0
- src/Kenshou/Cli/Command/Perf.hs +484/−0
- src/Kenshou/Cli/Command/Perf/Session.hs +1014/−0
- src/Kenshou/Cli/Command/Plan.hs +345/−0
- src/Kenshou/Cli/Command/Run.hs +207/−0
- src/Kenshou/Cli/Command/Worker.hs +20/−0
- src/Kenshou/Cli/Completions.hs +32/−0
- src/Kenshou/Cli/Config.hs +54/−0
- src/Kenshou/Cli/Diagnose.hs +254/−0
- src/Kenshou/Cli/Help.hs +55/−0
- src/Kenshou/Cli/Help/Width.hs +40/−0
- src/Kenshou/Cli/Main.hs +17/−0
- src/Kenshou/Cli/Options.hs +76/−0
- src/Kenshou/Cli/Registry.hs +62/−0
- src/Kenshou/Cli/Version.hs +36/−0
- test/Main.hs +801/−0
- test/PerfSessionSpec.hs +264/−0
- test/fixtures/perf-publication.synthetic.json +11/−0
- test/golden/help-evidence-80.txt +75/−0
+ CHANGELOG.md view
@@ -0,0 +1,12 @@+# Changelog++## 0.1.0.0 - 2026-10-28++### New Features++- First public release of the operator command library: `list`, `plan`, `run`, `execute`, `summarize`, `compare`, `diagnose`, `overhead`, `cell`, `record`, `attest`, `evidence`, `history`, `cohort`, and `perf`.+- The library takes its assembly from the executable: `mainWith`, `runWithArgsWith`, `standardCommands`, `standardTopics`, and `selfTestAssembly`.++### Other Changes++- Extracted from `mori://shinzui/keiro-runtime-kenshou` at commit `a1648293443ad88ff1695ed800ff44c99993574e`, with its package and module names unchanged.
+ LICENSE view
@@ -0,0 +1,28 @@+BSD 3-Clause License++Copyright (c) 2026, Nadeem Bitar++Redistribution and use in source and binary forms, with or without+modification, are permitted provided that the following conditions are met:++1. Redistributions of source code must retain the above copyright notice, this+ list of conditions and the following disclaimer.++2. Redistributions in binary form must reproduce the above copyright notice,+ this list of conditions and the following disclaimer in the documentation+ and/or other materials provided with the distribution.++3. Neither the name of the copyright holder nor the names of its contributors+ may be used to endorse or promote products derived from this software without+ specific prior written permission.++THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ help/cells.md view
@@ -0,0 +1,132 @@+# Leased verification cells++A cell runs a prepared verification plan on an exclusively leased machine set.+The payload identifies the Nix-built executable and the runtime cohort it+contains. Cell results are sealed in the results bucket and can be fetched and+verified again later.++Build and publish a payload descriptor:++```bash+kenshou cell payload publish --cohort released --out payloads/released.json+kenshou cell payload show payloads/released.json+```++`payload show` validates the descriptor and checks that the bundle object+exists with the recorded byte size before printing the descriptor as JSON.++Probe a cell after publishing a payload that contains the environment+scenario:++```bash+kenshou cell probe --cell alpha --payload payloads/head.json+```++The probe runs under a lease, verifies the sealed result, and writes+`.dev/cells/alpha.capabilities.json`. Route and submit use this cache to check+optional capabilities such as `pg_partman`. The cache is tied to the cell+descriptor and is refreshed by running the probe again.++Inspect a cell and run a plan in one process:++```bash+kenshou cell status --cell alpha+kenshou cell run --cell alpha --payload payloads/released.json \+ --plan plan.json --out cell-runs/alpha-1 --coerce-durable+```++The run command acquires a lease, prepares compatible runs for the cell,+submits them in reset-aware slices, follows the cell status, fetches and+verifies sealed output, then releases the lease. `--out` must name a new+directory. Keep that directory to resume an interrupted session:++```bash+kenshou cell resume --session cell-runs/alpha-1+```++Use `cell lease`, `cell submit`, `cell watch`, `cell fetch`, `cell verify`, and+`cell release` when the steps need separate processes. `cell fetch` needs the+results bucket and cell run ID; it writes a verified tree and an index beside+it. `cell verify` also accepts `gs://BUCKET/runs/CELL_RUN_ID` and checks a+sealed run without an existing session directory.++Compare a local run directory with a fetched nested cell run directory:++```bash+kenshou cell parity --local runs/LOCAL_RUN_ID \+ --cell cell-runs/CELL_RUN_ID/tree/output/NESTED_RUN_ID \+ --out parity.json+```++The report lists equal fields, expected placement differences, and unexpected+differences. The command exits 1 if a scenario input or verdict differs. Use+`--volatile result.summaries.verdicts.FIELD` only for a known path inside a+verdict summary.++Measure a benchmark with interleaved baseline and candidate trials on one+leased cell:++```bash+kenshou cell pair --cell alpha --start --baseline payloads/released.json \+ --candidate payloads/head.json --plan benchmark.plan.json \+ --policy policies/default.json --out cell-pairs/run-1+```++Each trial is a separate cell submission with a fresh reset. The command+keeps one lease across the schedule, replaces invalid pairs within its budget,+and writes `comparison.json`. Use the same payload for both arms as an A/A+control before comparing different cohorts.+`--start` starts stopped instances named by the verified cell descriptor; it is+also available on `cell lease` and `cell run`.++Measure telemetry overhead under one lease:++```bash+kenshou cell overhead selftest/telemetry/benchmark/arms-on-synthetic-service \+ --cell alpha --payload payloads/released.json \+ --arms tracing=off,sdk-otlp --trials 3 --otlp-sink file \+ --out cell-overhead/tracing+```++Each arm slot is a separate cold-reset submission. The command links verified+nested run directories into its `runs/` tree, retains the cell sessions,+reuses the overhead planner's retry and replacement-block rules, and writes+`overhead-report.json`. `--resume` continues the saved invocation under a+new lease; `--analyse-only` recomputes the report without submitting work.++Route a plan across cells with different PostgreSQL majors:++```bash+kenshou cell route --plan plan.json --cell beta --cell alpha \+ --payload payloads/released.json --out routed --coerce-durable+```++The output contains `plan.<cell>.json` for admitted runs, `plan.local.json`+for work requiring local server control, and `unroutable.json` for remaining+refusals. Cell order is preference order. Routing and submission read a+capability cache from `.dev/cells/<cell>.capabilities.json` when present;+`KENSHOU_CELL_CAPABILITIES_DIR` selects another directory. A cache for an old+descriptor is ignored. An unknown capability produces a warning, and an+explicitly unavailable capability prevents admission.++The control bucket defaults to `tan-nb-exp-cells-control`; pass+`--control-bucket` to select another. The client accepts only cells in the+projects named by `KENSHOU_GCP_ALLOWED_PROJECTS` (default `tan-nb-exp`).++For a running cell, use the owner IAP script through the verified descriptor:++```bash+kenshou cell debug --cell alpha ssh monitoring -- systemctl is-active opentelemetry-collector.service+kenshou cell debug --cell alpha journal+kenshou cell debug --cell alpha tunnel monitoring 8888 18888+```++`journal` reads the last 100 driver agent lines. `tunnel` holds an SSH port+forward until interrupted; while it runs, the collector metrics endpoint is+at `http://127.0.0.1:18888/metrics`. Keep a lease held during a longer debug+session so the cell does not idle off. The command locates the owner checkout+with Mori, or uses `KENSHOU_LTI_DIR` when set.++For profiling, publish a `profiled` payload and pass `--rts=-p` to `cell run`.+The cell includes the generated `.prof` files under `tree/output/profiles/` in+the sealed result, subject to its 128 MiB profile collection limit.
+ help/comparisons.md view
@@ -0,0 +1,20 @@+# Comparing benchmark runs++`kenshou compare` evaluates paired baseline and candidate runs. Run both arms in+the same environment and interleave them in ABBA or BAAB order so drift affects+each arm equally. Supply one `--baseline` and one `--candidate` directory per+pair.++The declared `--vary` axes are the only compatibility fields allowed to differ.+Use `cohort`, `dim:NAME`, or `knob:NAME`. A scenario, knob, dimension, schema, or+cohort difference outside those axes is a usage error. A machine-profile change+is an infrastructure failure.++Policies combine a relative limit with an absolute floor. A metric regresses+only when the confidence interval clears both limits. Wide or crossing intervals+are inconclusive. Hard health observations produce infrastructure failure and+soft observations produce an inconclusive result. Historical series can inform+future work, but they never determine a comparison verdict.++`kenshou summarize RUN_DIR --verify` recomputes measurements from retained+samples and series and checks the section stored in `run-result.json`.
+ help/diagnostics.md view
@@ -0,0 +1,24 @@+# Diagnostics++Re-evaluate the sampled series from a completed run without changing its sealed+artifacts:++ kenshou diagnose leak RUN_DIR++Render saved watchdog captures, rebuild their classification, or export the+PostgreSQL wait graph as Graphviz DOT:++ kenshou diagnose stall RUN_DIR+ kenshou diagnose stall RUN_DIR --reclassify --dot wait-graph.dot++Rerun a scenario under a bounded heap profile or GHC event log:++ kenshou diagnose profile SCENARIO --mode closure-type+ kenshou diagnose profile SCENARIO --mode info-table+ kenshou diagnose profile SCENARIO --mode eventlog --classes gu++Diagnosis uses the ordinary process exit contract: 0 means no finding, 1 means+a finding was confirmed, 2 is invalid usage, 3 means insufficient evidence, and+4 is an input, execution, or infrastructure error. See+`docs/guides/diagnosing-leaks-and-stalls.md` in the source distribution for the+full investigation workflow.
+ help/evidence.md view
@@ -0,0 +1,23 @@+# Verification evidence++`kenshou record RUN-DIR --data-base-uri gs://BUCKET/PREFIX --purpose release --project PROJECT` publishes a finished run's files and adds one record to the OKF verification bundle. The record names the run, cohort, outcome, and SHA-256-pinned object links. It contains no measurements; those remain in the linked run files.++`kenshou record --comparison FILE --data-base-uri gs://BUCKET/PREFIX --purpose release --project PROJECT` records a `kenshou.comparison/v1` document. Every baseline and candidate run named by the comparison must already have a run record in the bundle. The comparison record links the source document by digest and references those run concepts. Supply either `RUN-DIR` or `--comparison FILE`.++The default bundle is `docs/verification`. `--bundle DIR` selects another bundle. A real GCS upload requires `--project PROJECT`, and the command checks that the active gcloud project matches it. `--verify-only` checks that all objects already exist; `--deep-verify` downloads them and checks their bytes. Failed or inconclusive runs link logs automatically; `--link-logs` also links logs for a passing run.++`--config PATH` reads ordered YAML configuration. The supported keys are `evidence.bundle-root`, `gcp.project`, `evidence.data-base-uri`, `evidence.default-subject`, and `evidence.bundle-uri`; the corresponding environment variables are `KENSHOU_EVIDENCE_BUNDLE`, `KENSHOU_GCP_PROJECT`, `KENSHOU_EVIDENCE_DATA_BASE_URI`, `KENSHOU_EVIDENCE_SUBJECT`, and `KENSHOU_EVIDENCE_BUNDLE_URI`. Named flags take precedence over the environment and files. `kenshou evidence check` uses the same bundle default and configuration sources. Unknown keys are rejected.++A dirty harness requires `--allow-dirty` and is recorded with `purpose: investigation` regardless of the requested purpose. Use `--subject MORI-URI` with `--subject-kind project|package` when the default layer project is too broad. Repeat `--produced MORI-URI` for reports caused by this run.++For a local rehearsal, copy the verification bundle outside the repository and pass `--bundle COPY --store-root DIR`. The record still contains `gs://` URIs, while the bytes are stored under the scratch directory. The command refuses `--store-root` with a bundle inside this repository.++`--json` prints one `kenshou.record-result/v1` document and sends errors to standard error. A repeat with the same record content succeeds and reports `already recorded`; different content at the same record path is a conflict.++`kenshou evidence check` checks record paths, IDs, digest and revision shapes, textual fields, event-only keys, timestamps, required data links, comparison arm references and verdicts, and changes to committed records. `--base REF` limits the Git history span. `--network --project PROJECT` checks object size and SHA-256 metadata, and `--deep` downloads bytes to verify each digest. The command exits 0 when clean, 1 for findings, and 4 when it cannot check. `--json` prints one `kenshou.evidence-check/v1` document.++`kenshou history --scenario LAYER/COMPONENT/KIND/NAME --json` reads matching records in time order and emits `kenshou.evidence-history/v1`. Use `--cohort-component PACKAGE[=VERSION|REVISION]`, repeatable `--outcome`, `--since YYYY-MM-DD`, or `--confirmed-only` to narrow the series. Each entry includes linked attestations, a derived trust tier, and the latest earlier confirmed baseline with the same scenario and compatibility key. The history view does not modify the bundle.++`kenshou attest RECORD-OR-ID --project PROJECT` fetches the recorded run or comparison data and writes an immutable attestation with six named checks. For a comparison, it verifies both recorded arms and replays the paired algorithm from their samples under the stored policy. `--offline` skips revision resolution that needs network access; `--linked-only` checks only directly linked objects. It reports `refuted` when fetched data contradicts the record and `incomplete` when a check cannot establish the claim. A scenario oracle must be registered before `verdict-recomputed` can pass for an ordinary run, so generic runs currently remain incomplete. Exit codes are 0 for confirmed, 1 for refuted, 3 for incomplete, 2 for invalid input, and 4 for an operational failure. `--json` emits one `kenshou.attest-result/v1` result document.++A person may annotate an incomplete or refuted attestation with `--accept-anomaly --authority human:ID --reason TEXT` from an interactive terminal. The computed verdict does not change. The command rejects this action in CI or without a terminal.
+ help/exit-codes.md view
@@ -0,0 +1,3 @@+# Exit codes++Exit code 0 means passed, 1 failed, 2 usage error, 3 inconclusive, and 4 errored or infrastructure failure. A reproduced known defect exits 0 unless strict known-defect handling is requested.
+ help/outcomes.md view
@@ -0,0 +1,3 @@+# Outcomes++Runs finish as `passed`, `failed`, `errored`, `inconclusive`, or `infrastructure-failure`. A known defect never changes the true outcome; it can only make an explained failure non-blocking for automation.
+ help/planning.md view
@@ -0,0 +1,20 @@+# Planning and execution++`kenshou plan` maps named components, cohort differences, local changes, or+upstream revision ranges through the checked-in component graph. The transitive+dependent closure selects scenarios, and `--explain` shows why each one was+included. Unmapped inputs select everything rather than risk missing evidence.++Dimension policies are `default-only`, `telemetry-corners`, `pairwise`, and+`full`. Telemetry changes raise their minimum policy automatically. Benchmarks+always use durable PostgreSQL and are budgeted as complete interleaved trial+groups.++Use `--suite smoke`, `change`, `nightly`, `weekly-soak`, or `release` for the+checked-in policies. The generated run plan binds the graph digest, cohort,+seed, reasons, skips, and preassigned run IDs.++`kenshou execute --plan FILE --out DIR` runs entries sequentially and writes an+atomic plan summary. `--resume` requires the byte-identical plan, skips completed+entries, and gives interrupted entries fresh run IDs. See `docs/planning.md` for+the complete guide.
+ help/run-specs.md view
@@ -0,0 +1,3 @@+# Run specifications++A run specification is a versioned JSON request to execute one scenario. The effective document records every default explicitly, including dimensions, knobs, seed, phases, environment, and expected runtime cohort.
+ help/scenarios.md view
@@ -0,0 +1,5 @@+# Scenarios++A scenario is one named verification procedure. Its stable identifier has four lowercase segments: `layer/component/kind/name`. The layer identifies the runtime boundary under test, and the kind is `correctness`, `concurrency`, `soak`, or `benchmark`.++Use `kenshou list` to inspect registered scenarios. Scenario identifiers are versioned by a separate revision number so a changed workload is never silently compared with an older one.
+ help/selectors.md view
@@ -0,0 +1,3 @@+# Selectors++Selectors match scenario identifiers one segment at a time. `*` matches one segment and a final `**` matches the remainder. For example, `selftest/**` selects every self-test and `*/*/benchmark/*` selects all benchmark scenarios.
+ kenshou-cli.cabal view
@@ -0,0 +1,125 @@+cabal-version: 3.0+name: kenshou-cli+version: 0.1.0.0+synopsis: Operator command library for the Kenshou verification harness+description: Operator and automation interface for the Kenshou verification harness.+license: BSD-3-Clause+license-file: LICENSE+author: Nadeem Bitar+maintainer: nadeem@gmail.com+homepage: https://github.com/shinzui/kenshou+bug-reports: https://github.com/shinzui/kenshou/issues+category: Testing+build-type: Simple+extra-doc-files: CHANGELOG.md+tested-with: ghc >=9.12 && <9.13+extra-source-files:+ help/*.md+ test/fixtures/*.json+ test/golden/*.txt++common warnings+ ghc-options:+ -Wall+ -Wcompat+ -Widentities+ -Wincomplete-record-updates+ -Wincomplete-uni-patterns+ -Wpartial-fields+ -Wredundant-constraints++ default-language: GHC2024+ default-extensions:+ BlockArguments+ DuplicateRecordFields+ ImportQualifiedPost+ NoFieldSelectors+ OverloadedLabels+ OverloadedRecordDot+ OverloadedStrings++library+ import: warnings+ hs-source-dirs: src+ exposed-modules:+ Kenshou.Cli+ Kenshou.Cli.Attest+ Kenshou.Cli.Cohort+ Kenshou.Cli.Command.Cell+ Kenshou.Cli.Command.Cohort+ Kenshou.Cli.Command.Execute+ Kenshou.Cli.Command.List+ Kenshou.Cli.Command.Overhead+ Kenshou.Cli.Command.Perf+ Kenshou.Cli.Command.Perf.Session+ Kenshou.Cli.Command.Plan+ Kenshou.Cli.Command.Run+ Kenshou.Cli.Command.Worker+ Kenshou.Cli.Completions+ Kenshou.Cli.Config+ Kenshou.Cli.Diagnose+ Kenshou.Cli.Help+ Kenshou.Cli.Help.Width+ Kenshou.Cli.Main+ Kenshou.Cli.Options+ Kenshou.Cli.Registry+ Kenshou.Cli.Version++ other-modules: Paths_kenshou_cli+ autogen-modules: Paths_kenshou_cli+ build-depends:+ aeson >=2.2 && <2.3,+ base >=4.21 && <4.22,+ bytestring >=0.12 && <0.13,+ containers >=0.7 && <0.8,+ directory >=1.3 && <1.4,+ file-embed >=0.0.16 && <0.1,+ filepath >=1.5 && <1.6,+ githash >=0.1.7 && <0.2,+ hasql >=1.10 && <1.11,+ kenshou-check ^>=0.1.0.0,+ kenshou-core ^>=0.1.0.0,+ kenshou-diagnose ^>=0.1.0.0,+ kenshou-evidence ^>=0.1.0.0,+ kenshou-measure ^>=0.1.0.0,+ kenshou-remote ^>=0.1.0.0,+ kenshou-telemetry ^>=0.1.0.0,+ optparse-applicative >=0.19 && <0.20,+ process >=1.6 && <1.7,+ random >=1.2.1 && <1.4,+ settei ^>=0.2.0.0,+ settei-env ^>=0.2.0.0,+ settei-optparse-applicative ^>=0.2.0.0,+ temporary >=1.3 && <1.4,+ terminal-size >=0.3.4 && <0.4,+ text >=2.1 && <2.2,+ time >=1.14 && <1.15,+ uuid >=1.3 && <1.4,++test-suite kenshou-cli-test+ import: warnings+ type: exitcode-stdio-1.0+ hs-source-dirs: test+ main-is: Main.hs+ other-modules: PerfSessionSpec+ build-depends:+ aeson >=2.2 && <2.3,+ base >=4.21 && <4.22,+ bytestring,+ containers >=0.7 && <0.8,+ directory >=1.3 && <1.4,+ filepath,+ hspec >=2.11 && <2.12,+ kenshou-cli,+ kenshou-core ^>=0.1.0.0,+ kenshou-measure ^>=0.1.0.0,+ kenshou-remote ^>=0.1.0.0,+ kenshou-telemetry ^>=0.1.0.0,+ temporary >=1.3 && <1.4,+ text >=2.1 && <2.2,+ time >=1.14 && <1.15,+ uuid >=1.3 && <1.4,++source-repository head+ type: git+ location: https://github.com/shinzui/kenshou.git
+ src/Kenshou/Cli.hs view
@@ -0,0 +1,14 @@+module Kenshou.Cli+ ( mainWith,+ runWithArgsWith,+ CliAssembly,+ Program (..),+ selfTestBundles,+ standardCommands,+ standardTopics,+ selfTestAssembly,+ )+where++import Kenshou.Cli.Main (mainWith, runWithArgsWith)+import Kenshou.Cli.Registry (CliAssembly, Program (..), selfTestAssembly, selfTestBundles, standardCommands, standardTopics)
+ src/Kenshou/Cli/Attest.hs view
@@ -0,0 +1,101 @@+module Kenshou.Cli.Attest (runOutcomeRecomputer, measurementRecomputer, pairedComparisonRecomputer) where++import Control.Monad (forM)+import Data.Aeson (Result (..), Value (..), eitherDecodeFileStrict', fromJSON, toJSON)+import Data.Aeson.KeyMap qualified as KeyMap+import Data.List.NonEmpty qualified as NonEmpty+import Data.Text (Text)+import Data.Text qualified as Text+import Kenshou.Evidence.Attest (Recomputation (..), Recomputer (..), coreRecomputers)+import Kenshou.Measure.Compare (CompareError (..), compareRuns)+import Kenshou.Measure.Compare.Compatibility (VaryingAxis, parseVaryingAxis)+import Kenshou.Measure.Compare.Policy (Policy)+import Kenshou.Measure.Summary (SummaryError (..), summarizeRunDir)+import System.FilePath ((</>))++runOutcomeRecomputer :: Recomputer+runOutcomeRecomputer = Recomputer "run-outcome" 1 $ \root -> case coreRecomputers of+ (core : _) -> core.recompute root+ [] -> pure (Left "the core outcome oracle is unavailable")++measurementRecomputer :: Recomputer+measurementRecomputer = Recomputer "kenshou-summary" 1 $ \root -> do+ measured <- summarizeRunDir root+ case measured of+ Left (SummaryError message) -> pure (Left message)+ Right summary -> do+ stored <- eitherDecodeFileStrict' (root </> "run-result.json") :: IO (Either String Value)+ pure case stored of+ Left message -> Left (Text.pack message)+ Right document ->+ Right+ Recomputation+ { agreesWithDocuments = storedMeasurements document == Just (toJSON summary),+ outcome = Nothing,+ comparisonVerdict = Nothing,+ detail = "recomputed kenshou.measurements/v1 from the sealed samples and series"+ }++storedMeasurements :: Value -> Maybe Value+storedMeasurements (Object result) = do+ Object summaries <- KeyMap.lookup "summaries" result+ Object measurements <- KeyMap.lookup "measurements" summaries+ KeyMap.lookup "measurements" measurements+storedMeasurements _ = Nothing++pairedComparisonRecomputer :: Recomputer+pairedComparisonRecomputer = Recomputer "paired-bootstrap-t-envelope" 1 $ \root -> do+ stored <- eitherDecodeFileStrict' (root </> "comparison.json") :: IO (Either String Value)+ case stored of+ Left message -> pure (Left (Text.pack message))+ Right document -> case comparisonInputs document of+ Left message -> pure (Left message)+ Right (policy, axes, pairs) -> do+ let baselines = [root </> "baseline" </> show index | index <- [0 .. pairs - 1]]+ candidates = [root </> "candidate" </> show index | index <- [0 .. pairs - 1]]+ verified <- forM (baselines <> candidates) $ \path -> do+ measured <- summarizeRunDir path+ result <- eitherDecodeFileStrict' (path </> "run-result.json") :: IO (Either String Value)+ pure case (measured, result) of+ (Right summary, Right value) | storedMeasurements value == Just (toJSON summary) -> Right ()+ (Left (SummaryError message), _) -> Left message+ (_, Left message) -> Left (Text.pack message)+ _ -> Left "arm measurements differ from the sealed samples and series"+ case sequence verified of+ Left message -> pure (Left message)+ Right _ -> do+ replayed <- compareRuns policy axes baselines candidates+ pure case replayed of+ Left (CompareError message) -> Left message+ Right comparison ->+ let actual = toJSON comparison+ stableKeys = ["design", "baselineRuns", "candidateRuns", "algorithm", "policy", "variedFactors", "pairCount", "metrics", "reasons", "verdict", "exitCode"]+ agrees = all (\key -> field key actual == field key document) stableKeys+ in Right+ Recomputation+ { agreesWithDocuments = agrees,+ outcome = Nothing,+ comparisonVerdict = case field "verdict" actual of Just (String verdict) -> Just verdict; _ -> Nothing,+ detail = "recomputed arm summaries and the paired comparison from sealed data"+ }+ where+ field name (Object value) = KeyMap.lookup name value+ field _ _ = Nothing++comparisonInputs :: Value -> Either Text (Policy, NonEmpty.NonEmpty VaryingAxis, Int)+comparisonInputs (Object document) = do+ rawPolicy <- maybe (Left "comparison has no policy") Right (KeyMap.lookup "policy" document)+ policy <- case fromJSON rawPolicy of+ Error message -> Left (Text.pack message)+ Success value -> Right value+ rawAxes <- maybe (Left "comparison has no varied factors") Right (KeyMap.lookup "variedFactors" document)+ axisNames <- case fromJSON rawAxes of+ Error message -> Left (Text.pack message)+ Success value -> Right value+ axes <- traverse parseVaryingAxis axisNames >>= maybe (Left "comparison has no varied factors") Right . NonEmpty.nonEmpty+ rawPairs <- maybe (Left "comparison has no pair count") Right (KeyMap.lookup "pairCount" document)+ pairs <- case fromJSON rawPairs of+ Error message -> Left (Text.pack message)+ Success value -> Right value+ if pairs > 0 then Right (policy, axes, pairs) else Left "comparison pair count must be positive"+comparisonInputs _ = Left "comparison document is not an object"
+ src/Kenshou/Cli/Cohort.hs view
@@ -0,0 +1,93 @@+{-# LANGUAGE FieldSelectors #-}++module Kenshou.Cli.Cohort+ ( runCohortCommand,+ resolveDefaultCohortIdentity,+ )+where++import Control.Applicative ((<|>))+import Data.Aeson qualified as Aeson+import Data.ByteString.Lazy.Char8 qualified as LazyByteString+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Text.IO qualified as Text.IO+import Kenshou.Cli.Options (CohortCommand (..))+import Kenshou.Core.Cohort+import System.Environment (lookupEnv)+import System.Exit (ExitCode (..))+import System.FilePath ((</>))+import System.IO (stderr)++resolveDefaultCohortIdentity :: IO (Either CohortError CohortIdentity)+resolveDefaultCohortIdentity = do+ selected <- lookupEnv "KENSHOU_COHORT_IDENTITY"+ resolveCohortIdentity (maybe (FromProject "." Nothing Nothing) FromIdentityFile selected)++runCohortCommand :: CohortCommand -> IO ExitCode+runCohortCommand (CohortShow json projectDir planJson identityOption) = do+ identityEnvironment <- lookupEnv "KENSHOU_COHORT_IDENTITY"+ let source =+ maybe+ (FromProject projectDir planJson Nothing)+ FromIdentityFile+ (identityOption <|> identityEnvironment)+ result <- resolveCohortIdentity source+ case result of+ Left err -> reportError err+ Right identity -> do+ if json+ then LazyByteString.putStrLn (Aeson.encode identity)+ else Text.IO.putStr (renderCohortIdentity identity)+ pure ExitSuccess+runCohortCommand (CohortCheck projectDir planJson descriptorOverride identityOption) = do+ identityEnvironment <- lookupEnv "KENSHOU_COHORT_IDENTITY"+ descriptorResult <- loadActiveDescriptor projectDir descriptorOverride+ identityResult <-+ resolveCohortIdentity $+ maybe+ (FromProject projectDir planJson descriptorOverride)+ FromIdentityFile+ (identityOption <|> identityEnvironment)+ case (descriptorResult, identityResult) of+ (Left err, _) -> reportError err+ (_, Left err) -> reportError err+ (Right descriptor, Right identity) ->+ case checkCohort descriptor identity of+ [] -> pure ExitSuccess+ mismatches -> do+ mapM_ (Text.IO.hPutStrLn stderr . renderMismatch) mismatches+ pure (ExitFailure 1)++loadActiveDescriptor :: FilePath -> Maybe FilePath -> IO (Either CohortError CohortDescriptor)+loadActiveDescriptor _ (Just path) = loadCohortDescriptor path+loadActiveDescriptor projectDir Nothing = do+ nameResult <- activeCohortName projectDir+ case nameResult of+ Left err -> pure (Left err)+ Right (CohortName name) -> loadCohortDescriptor (projectDir </> "cohort" </> Text.unpack name <> ".json")++reportError :: CohortError -> IO ExitCode+reportError err = do+ Text.IO.hPutStrLn stderr (renderError err)+ pure (ExitFailure 4)++renderError :: CohortError -> Text+renderError (CohortIoError message)+ | "dist-newstyle/cache/plan.json" `Text.isInfixOf` message = "no dist-newstyle/cache/plan.json; run cabal build all first"+ | otherwise = message+renderError (CohortDecodeError message) = message+renderError (CohortPlanError message) = message+renderError (CohortInvalidActive message) = message++renderMismatch :: CohortMismatch -> Text+renderMismatch (CohortNameMismatch (CohortName expected) (CohortName actual)) =+ "cohort: expected " <> expected <> ", resolved " <> actual+renderMismatch (CompilerMismatch expected actual) =+ "compiler: expected " <> expected <> ", resolved " <> actual+renderMismatch (MissingPackage packageName) = "missing package: " <> packageName+renderMismatch (VersionMismatch packageName expected actual) =+ packageName <> ": expected version " <> expected <> ", resolved " <> actual+renderMismatch (SourceMismatch packageName expected actual) =+ packageName <> ": expected source " <> Text.pack (show expected) <> ", resolved " <> Text.pack (show actual)+renderMismatch (LocalPathSource packageName path) = packageName <> ": resolved from forbidden local path " <> Text.pack path
+ src/Kenshou/Cli/Command/Cell.hs view
@@ -0,0 +1,1208 @@+module Kenshou.Cli.Command.Cell (cellCommand, startCellInstances) where++import Control.Concurrent (threadDelay)+import Control.Exception (SomeAsyncException, SomeException, displayException, finally, fromException, throwIO, try)+import Data.Aeson (FromJSON, Value (..), eitherDecode, eitherDecodeFileStrict', encode, object, (.=))+import Data.Aeson qualified as Aeson+import Data.Aeson.Key qualified as Key+import Data.Aeson.KeyMap qualified as KeyMap+import Data.ByteString qualified as ByteString+import Data.ByteString.Lazy.Char8 qualified as LazyByteString+import Data.Foldable (traverse_)+import Data.List (find, nub, sort)+import Data.List.NonEmpty (NonEmpty (..))+import Data.List.NonEmpty qualified as NonEmpty+import Data.Map.Strict qualified as Map+import Data.Maybe (fromMaybe, isNothing)+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Text.IO qualified as TextIO+import Data.Time (diffUTCTime, getCurrentTime)+import Kenshou.Cli.Command.Overhead qualified as LocalOverhead+import Kenshou.Core.Bundle (lookupScenario, registryRoutingRules)+import Kenshou.Core.Cli (CliCommand (..), CliEnv (..), CliGroup (..))+import Kenshou.Core.Cohort (CohortIdentity (..), PlanHash (..))+import Kenshou.Core.Id (RunId, mkSeed, newRunId, parseRunId, parseScenarioId, renderRunId, unSeed)+import Kenshou.Core.RunSpec (EnvironmentSpec (..), RunSpec (..), SpecPlacement (..))+import Kenshou.Core.Scenario (Scenario (..))+import Kenshou.Diagnose.Leak (analyseRunDirectory, defaultLeakSpec)+import Kenshou.Measure.Compare (CompareError (..), Comparison (..), compareRuns, verdictExitCode)+import Kenshou.Measure.Compare.Compatibility (VaryingAxis (..), compatibilityInputs, varyingValue)+import Kenshou.Measure.Compare.Ordering (Arm (..), PairedOrdering (..))+import Kenshou.Measure.Compare.Policy (Policy (..), decodePolicy)+import Kenshou.Plan.Change (Change (..), ChangeSource (..), Reason (..))+import Kenshou.Plan.Components (ComponentId (..), ComponentRef (..))+import Kenshou.Plan.Policy (PlanPolicy (..), defaultPlanPolicy)+import Kenshou.Plan.RunPlan (PlanContext (..), PlanInputs (PlanInputs), PlannedRun (..), RunPlan (..), TrialInfo (..))+import Kenshou.Plan.Selector (parseSelector)+import Kenshou.Remote.Cell.Control (CellSnapshot (..), readCellSnapshot)+import Kenshou.Remote.Cell.Debug (DebugCommand (..), runCellDebug)+import Kenshou.Remote.Cell.Docs (CachePolicy (..), CellBuckets (..), CellDescriptor (..), CellManifest (..), CellNodes (..), CellOutcome (..), CellStatus (..), Limits (..), Rejected (..), Submission (..))+import Kenshou.Remote.Cell.Exec (cellExec)+import Kenshou.Remote.Cell.Fetch (FetchError (..), fetchCellRun, verifyCellRun)+import Kenshou.Remote.Cell.Index (deriveCellRunIndex, writeCellRunIndex)+import Kenshou.Remote.Cell.Lease (AcquireOutcome (..), CellRef (..), Lease (..), LeaseHandle, LeaseRequest (..), Quarantine (..), acquireLease, leaseSnapshot, reattachLease, releaseLease, resizeLease, validCellName, withHeartbeat)+import Kenshou.Remote.Cell.Parity (ParityOptions (..), ParityReport (..), compareForParity)+import Kenshou.Remote.Cell.Prepare (Granularity (..), OtlpSink (..), PrepareOptions (..))+import Kenshou.Remote.Cell.RouteJson (RouteDocuments (..), routeWorkJson)+import Kenshou.Remote.Cell.RouteRules (CellCapabilities (..), descriptorDigest, loadCellCapabilities)+import Kenshou.Remote.Cell.Session (SessionTransition (..))+import Kenshou.Remote.Cell.Session.Build (BuildOptions (..), BuiltSession (..), buildSession)+import Kenshou.Remote.Cell.Session.Journal (LeaseMode (..), SessionJournal (..), SliceJournal (..), SliceState (..), applyTransition, readSessionJournal, writeSessionJournal)+import Kenshou.Remote.Cell.Session.Resume (resumeHeldSession, resumeObservedSlices)+import Kenshou.Remote.Cell.Session.Runner (runPlannedSlices)+import Kenshou.Remote.Cell.Submit (PublishOutcome (Submitted), publishSubmission)+import Kenshou.Remote.Cell.Watch (WatchEvent (..), WatchTerminal (..), watchCellRun)+import Kenshou.Remote.Cell.WorkJson (decodeWorkPlan)+import Kenshou.Remote.Pair (PairRequest (..), PairState (..), SliceResult (..), judgePairs, pairPlan)+import Kenshou.Remote.Payload (Bundle (..), CellPayload (..), PayloadDescriptor (..))+import Kenshou.Remote.Payload.Publisher (PublishError (..), PublishOptions (..), publishPayload)+import Kenshou.Remote.Store (Bucket (..), ObjectMeta (..), ObjectName (..), ObjectStore (..))+import Kenshou.Remote.Store.File (newFileStore)+import Kenshou.Remote.Store.Gcs (newGcsStore, newTokenProvider)+import Kenshou.Telemetry.Overhead (OverheadHooks (..), OverheadPlan (..), OverheadReport (..), OverheadRequest (..), OverheadState (..), UsageError (..), analyseOverhead, executeOverhead, loadOverheadState, overheadVerdictExitCode, planOverhead)+import Kenshou.Telemetry.Overhead.Policy (OverheadPolicy)+import Options.Applicative+import System.Directory (createDirectoryIfMissing, createDirectoryLink, doesDirectoryExist, doesPathExist, makeAbsolute, renameFile)+import System.Environment (getExecutablePath, lookupEnv)+import System.Exit (ExitCode (..))+import System.FilePath (takeDirectory, (</>))+import System.IO (hFlush, stderr, stdout)+import System.IO.Temp (withSystemTempDirectory)+import System.Process (readProcessWithExitCode)++data CellLocation = CellLocation !Text !(Maybe Text)++data LeaseCli = LeaseCli !CellLocation !Text !(Maybe Text) !Int !Int !Bool !Bool !Bool++data RouteCli = RouteCli ![Text] !(Maybe Text) !FilePath !(Maybe FilePath) !(Maybe FilePath) !FilePath !Bool !Bool++data SubmitSettings = SubmitSettings ![String] !FilePath !FilePath !Granularity !CachePolicy ![Text] !Bool !Bool !Bool !(Maybe FilePath) !OtlpSink !(Maybe Text) !Bool++data SubmitCli = SubmitCli !CellLocation !Text !SubmitSettings !Bool++data RunCli = RunCli !CellLocation !SubmitSettings !Int !Int !Bool !Bool++data ProbeCli = ProbeCli !CellLocation !FilePath !(Maybe FilePath)++data PairCli = PairCli !CellLocation !FilePath !FilePath !FilePath !(Maybe Int) !PairedOrdering !FilePath ![Text] !Int !Bool !FilePath++data PairObservation = PairObservation !PlannedRun !FilePath !Bool++data OverheadCli = OverheadCli !CellLocation !FilePath !OtlpSink ![Text] !Bool !LocalOverhead.OverheadOptions++data PayloadPublishCli = PayloadPublishCli !Text !Text !(Maybe Text) !FilePath !FilePath !Bool++data CellAction+ = Fetch !Text !Text !FilePath+ | Verify !FilePath+ | Parity !FilePath !FilePath !(Maybe FilePath) ![Text]+ | Status !CellLocation !Bool+ | LeaseCell !LeaseCli+ | Release !CellLocation !Text+ | Watch !CellLocation !Text+ | Debug !CellLocation !DebugCommand+ | Route !RouteCli+ | Submit !SubmitCli+ | Run !RunCli+ | Probe !ProbeCli+ | Pair !PairCli+ | Overhead !OverheadCli+ | Resume !FilePath !(Maybe Text) !Bool+ | PublishPayload !PayloadPublishCli+ | ShowPayload !FilePath+ | Exec !FilePath !FilePath++cellCommand :: CliCommand+cellCommand = CliCommand "cell" "Run and inspect leased verification cells" Execution False (runCell <$> cellParser)++cellParser :: Parser CellAction+cellParser =+ hsubparser $+ command "fetch" (info (fetchParser <**> helper) (progDesc "Fetch and verify one sealed cell run"))+ <> command "verify" (info (verifyParser <**> helper) (progDesc "Verify a fetched tree or a sealed GCS run"))+ <> command "parity" (info (parityParser <**> helper) (progDesc "Compare local and cell verdicts with named placement differences"))+ <> command "status" (info (statusParser <**> helper) (progDesc "Inspect cell descriptor and lease state"))+ <> command "lease" (info (leaseParser <**> helper) (progDesc "Acquire a cell lease"))+ <> command "release" (info (releaseParser <**> helper) (progDesc "Release an owned cell lease"))+ <> command "watch" (info (watchParser <**> helper) (progDesc "Follow a cell run to its terminal status"))+ <> command "debug" (info (debugParser <**> helper) (progDesc "Inspect a cell role through the owner IAP script"))+ <> command "route" (info (routeParser <**> helper) (progDesc "Split a plan across compatible cells"))+ <> command "submit" (info (submitParser <**> helper) (progDesc "Submit a prepared plan under an existing lease"))+ <> command "run" (info (runParser <**> helper) (progDesc "Lease, submit, verify and release a cell session"))+ <> command "probe" (info (probeParser <**> helper) (progDesc "Run a cell environment probe and cache capabilities"))+ <> command "pair" (info (pairParser <**> helper) (progDesc "Measure interleaved payload arms under one cell lease"))+ <> command "overhead" (info (overheadCellParser <**> helper) (progDesc "Measure telemetry overhead under one cell lease"))+ <> command "resume" (info (resumeParser <**> helper) (progDesc "Continue a saved cell session"))+ <> command "payload" (info (payloadParser <**> helper) (progDesc "Build and publish a checked cell payload"))+ <> (command "exec" (info (cellExecParser <**> helper) (progDesc "Execute prepared work on a cell driver")) <> internal)++payloadParser :: Parser CellAction+payloadParser =+ hsubparser $+ command "publish" (info (publishPayloadParser <**> helper) (progDesc "Publish a Nix closure after checking its runtime cohort"))+ <> command "show" (info (ShowPayload <$> strArgument (metavar "FILE") <**> helper) (progDesc "Show a payload descriptor after checking its bundle exists"))++publishPayloadParser :: Parser CellAction+publishPayloadParser =+ PublishPayload+ <$> ( PayloadPublishCli+ <$> strOption (long "cohort" <> metavar "released|head" <> help "Pinned runtime cohort")+ <*> strOption (long "variant" <> metavar "default|info-table|profiled" <> value "default" <> showDefault <> help "Executable build variant")+ <*> optional (strOption (long "control-bucket" <> metavar "BUCKET" <> help "Payload control bucket"))+ <*> strOption (long "out" <> metavar "FILE" <> help "Payload descriptor JSON output")+ <*> strOption (long "root" <> metavar "DIR" <> value "." <> showDefault <> help "Flake and cohort descriptor root")+ <*> switch (long "allow-dirty" <> help "Permit a payload from a dirty worktree")+ )++cellLocationParser :: Parser CellLocation+cellLocationParser =+ CellLocation+ <$> strOption (long "cell" <> metavar "NAME" <> help "Cell name")+ <*> optional (strOption (long "control-bucket" <> metavar "BUCKET" <> help "Cell control bucket"))++statusParser :: Parser CellAction+statusParser = Status <$> cellLocationParser <*> switch (long "json" <> help "Print JSON")++leaseParser :: Parser CellAction+leaseParser =+ LeaseCell+ <$> ( LeaseCli+ <$> cellLocationParser+ <*> strOption (long "purpose" <> metavar "TEXT" <> help "Purpose recorded in the lease")+ <*> optional (strOption (long "owner" <> metavar "TEXT" <> help "Lease owner; defaults to USER"))+ <*> option auto (long "ttl" <> metavar "SECONDS" <> value 120 <> showDefault <> help "Lease lifetime between renewals")+ <*> option auto (long "wait" <> metavar "SECONDS" <> value 0 <> showDefault <> help "Wait for a busy lease")+ <*> switch (long "start" <> help "Start stopped cell instances before acquiring a lease")+ <*> switch (long "hold" <> help "Renew in the foreground until interrupted")+ <*> switch (long "json" <> help "Print the lease document as JSON")+ )++releaseParser :: Parser CellAction+releaseParser = Release <$> cellLocationParser <*> strOption (long "lease-id" <> metavar "UUID" <> help "Expected lease identifier")++watchParser :: Parser CellAction+watchParser = Watch <$> cellLocationParser <*> strArgument (metavar "CELL_RUN_ID")++debugParser :: Parser CellAction+debugParser =+ Debug+ <$> cellLocationParser+ <*> hsubparser+ ( command "ssh" (info (DebugSsh <$> strArgument (metavar "ROLE") <*> many (strArgument (metavar "COMMAND")) <**> helper) (progDesc "Open a role shell or run a remote command"))+ <> command "journal" (info (pure DebugJournal <**> helper) (progDesc "Read the driver agent journal"))+ <> command "tunnel" (info (DebugTunnel <$> strArgument (metavar "ROLE") <*> argument auto (metavar "REMOTE_PORT") <*> argument auto (metavar "LOCAL_PORT") <**> helper) (progDesc "Forward a local port through IAP SSH until interrupted"))+ )++routeParser :: Parser CellAction+routeParser =+ Route+ <$> ( RouteCli+ <$> some (strOption (long "cell" <> metavar "NAME" <> help "Candidate cell in preference order; repeat to add cells"))+ <*> optional (strOption (long "control-bucket" <> metavar "BUCKET" <> help "Cell control bucket"))+ <*> strOption (long "plan" <> metavar "FILE" <> help "Run plan JSON, or - for stdin")+ <*> optional (strOption (long "payload" <> metavar "FILE" <> help "Payload descriptor for cohort preparation"))+ <*> optional (strOption (long "routing-rules" <> metavar "FILE" <> help "Replace the built-in routing policy with JSON, or - for stdin"))+ <*> strOption (long "out" <> metavar "DIR" <> help "Directory for routed plans and refusal report")+ <*> switch (long "coerce-durable" <> help "Use durable PostgreSQL when the scenario supports it")+ <*> switch (long "ephemeral-on-driver" <> help "Keep server-control correctness work on the cell driver")+ )++submitParser :: Parser CellAction+submitParser =+ Submit+ <$> ( SubmitCli+ <$> cellLocationParser+ <*> strOption (long "lease-id" <> metavar "UUID" <> help "Active lease identifier")+ <*> submitSettingsParser (switch (long "dry-run" <> help "Print a planned session without writing or submitting"))+ <*> pure False+ )++runParser :: Parser CellAction+runParser =+ Run+ <$> ( RunCli+ <$> cellLocationParser+ <*> submitSettingsParser (pure False)+ <*> option auto (long "ttl" <> metavar "SECONDS" <> value 120 <> showDefault <> help "Held lease lifetime between renewals")+ <*> option auto (long "wait" <> metavar "SECONDS" <> value 0 <> showDefault <> help "Wait for a busy cell")+ <*> switch (long "start" <> help "Start stopped cell instances before acquiring a lease")+ <*> switch (long "detach" <> help "Submit one slice and leave its budgeted lease to run after this process exits")+ )++probeParser :: Parser CellAction+probeParser =+ Probe+ <$> ( ProbeCli+ <$> cellLocationParser+ <*> strOption (long "payload" <> metavar "FILE" <> help "Published payload descriptor with the cell-environment scenario")+ <*> optional (strOption (long "out" <> metavar "FILE" <> help "Capability cache JSON; defaults to the configured cache directory"))+ )++pairParser :: Parser CellAction+pairParser =+ Pair+ <$> ( PairCli+ <$> cellLocationParser+ <*> strOption (long "candidate" <> metavar "FILE" <> help "Candidate payload descriptor")+ <*> strOption (long "baseline" <> metavar "FILE" <> help "Baseline payload descriptor")+ <*> strOption (long "plan" <> metavar "FILE" <> help "Benchmark run plan JSON")+ <*> optional (option auto (long "pairs" <> metavar "N" <> help "Requested valid pairs; defaults to the policy minimum"))+ <*> option (eitherReader parsePairOrdering) (long "ordering" <> metavar "abba|baab" <> value ABBA <> showDefaultWith (const "abba") <> help "Order baseline and candidate in adjacent pairs")+ <*> strOption (long "policy" <> metavar "FILE" <> value "policies/default.json" <> showDefault <> help "Comparison policy JSON")+ <*> many (strOption (long "pg-setting" <> metavar "KEY=VALUE" <> help "PostgreSQL reset setting; repeat as needed"))+ <*> option auto (long "max-replacements" <> metavar "N" <> value (-1) <> help "Additional pairs after invalid trials; defaults to requested pairs")+ <*> switch (long "start" <> help "Start stopped cell instances before acquiring a lease")+ <*> strOption (long "out" <> metavar "DIR" <> help "New directory for paired cell sessions and comparison")+ )++parsePairOrdering :: String -> Either String PairedOrdering+parsePairOrdering "abba" = Right ABBA+parsePairOrdering "baab" = Right BAAB+parsePairOrdering _ = Left "expected abba or baab"++overheadCellParser :: Parser CellAction+overheadCellParser =+ Overhead+ <$> ( OverheadCli+ <$> cellLocationParser+ <*> strOption (long "payload" <> metavar "FILE" <> help "Payload descriptor for every telemetry arm")+ <*> option (eitherReader parseOtlpSink) (long "otlp-sink" <> metavar "null|file" <> value NullSink <> showDefaultWith (const "null") <> help "Cell OTLP sink")+ <*> many (strOption (long "pg-setting" <> metavar "KEY=VALUE" <> help "PostgreSQL reset setting; repeat as needed"))+ <*> switch (long "start" <> help "Start stopped cell instances before acquiring a lease")+ <*> LocalOverhead.overheadParser+ )++submitSettingsParser :: Parser Bool -> Parser SubmitSettings+submitSettingsParser dryFlag =+ SubmitSettings+ <$> some (strOption (long "payload" <> metavar "[LABEL=]FILE" <> help "Payload descriptor; repeat for trial arms"))+ <*> strOption (long "plan" <> metavar "FILE" <> help "Run plan JSON, or - for stdin")+ <*> strOption (long "out" <> metavar "DIR" <> help "New session directory")+ <*> option (eitherReader parseGranularity) (long "granularity" <> metavar "auto|plan|run" <> value GranularityAuto <> showDefaultWith (const "auto") <> help "Submission slice boundary")+ <*> option (eitherReader parseCachePolicy) (long "cache-policy" <> metavar "cold|warm" <> value Cold <> showDefaultWith (const "cold") <> help "Cell cache reset policy")+ <*> many (strOption (long "pg-setting" <> metavar "KEY=VALUE" <> help "PostgreSQL reset setting; repeat as needed"))+ <*> switch (long "skip-incompatible" <> help "Record incompatible runs and submit compatible ones")+ <*> switch (long "coerce-durable" <> help "Use durable PostgreSQL when the scenario supports it")+ <*> switch (long "ephemeral-on-driver" <> help "Keep server-control correctness work on the cell driver")+ <*> optional (strOption (long "routing-rules" <> metavar "FILE" <> help "Replace the built-in routing policy JSON"))+ <*> option (eitherReader parseOtlpSink) (long "otlp-sink" <> metavar "null|file" <> value NullSink <> showDefaultWith (const "null") <> help "Cell OTLP sink")+ <*> optional (strOption (long "rts" <> metavar "OPTS" <> help "Runtime system options passed to the payload"))+ <*> dryFlag++parseGranularity :: String -> Either String Granularity+parseGranularity "auto" = Right GranularityAuto+parseGranularity "plan" = Right GranularityPlan+parseGranularity "run" = Right GranularityRun+parseGranularity _ = Left "expected auto, plan or run"++parseCachePolicy :: String -> Either String CachePolicy+parseCachePolicy "cold" = Right Cold+parseCachePolicy "warm" = Right Warm+parseCachePolicy _ = Left "expected cold or warm"++parseOtlpSink :: String -> Either String OtlpSink+parseOtlpSink "null" = Right NullSink+parseOtlpSink "file" = Right FileSink+parseOtlpSink _ = Left "expected null or file"++resumeParser :: Parser CellAction+resumeParser =+ Resume+ <$> strOption (long "session" <> metavar "DIR" <> help "Directory containing session.json")+ <*> optional (strOption (long "lease-id" <> metavar "UUID" <> help "Use this active lease instead of the journal lease"))+ <*> switch (long "start" <> help "Start stopped cell instances before resuming")++cellExecParser :: Parser CellAction+cellExecParser = Exec <$> strArgument (metavar "WORK_FILE") <*> strArgument (metavar "OUT_DIR")++fetchParser :: Parser CellAction+fetchParser =+ Fetch+ <$> strOption (long "results-bucket" <> metavar "BUCKET" <> help "Results bucket containing the sealed run")+ <*> strArgument (metavar "CELL_RUN_ID")+ <*> strOption (long "out" <> metavar "DIR" <> help "Directory for the fetched cell run")++verifyParser :: Parser CellAction+verifyParser = Verify <$> strArgument (metavar "DIR_OR_GS_URI")++parityParser :: Parser CellAction+parityParser =+ Parity+ <$> strOption (long "local" <> metavar "DIR" <> help "Local run directory")+ <*> strOption (long "cell" <> metavar "DIR" <> help "Fetched nested cell run directory")+ <*> optional (strOption (long "out" <> metavar "FILE" <> help "Write the parity report to FILE instead of stdout"))+ <*> many (strOption (long "volatile" <> metavar "PATH" <> help "Allow a named volatile verdict field; repeat as needed"))++runCell :: CellAction -> CliEnv -> IO ExitCode+runCell selected cli = case selected of+ PublishPayload (PayloadPublishCli cohort variant selectedBucket output root allowDirty) -> do+ configured <- lookupEnv "KENSHOU_CELL_CONTROL_BUCKET"+ let bucketName = fromMaybe (maybe "tan-nb-exp-cells-control" Text.pack configured) selectedBucket+ case validateBucket bucketName of+ Left problem -> usage problem+ Right bucket -> guardIO do+ store <- openStore bucket+ published <- publishPayload store (PublishOptions root cohort variant bucket output allowDirty)+ case published of+ Left (PublishInvalidSelection problem) -> usage problem+ Left (PublishCohortMismatch problem) -> failVerification problem+ Left PublishDirtyWorktree -> unavailable "worktree is dirty; pass --allow-dirty to publish it"+ Left problem -> unavailable (Text.pack (show problem))+ Right _ -> TextIO.putStrLn (Text.pack output) >> pure ExitSuccess+ ShowPayload path -> guardIO do+ decoded <- eitherDecodeFileStrict' path+ case decoded of+ Left problem -> failVerification (Text.pack problem)+ Right (descriptor :: PayloadDescriptor) -> case parsePayloadUri descriptor.cell.bundle.uri of+ Left problem -> failVerification problem+ Right (bucket, bundleObject) -> do+ store <- openStore bucket+ observed <- store.statObject bucket bundleObject+ case observed of+ Nothing -> failVerification "payload bundle does not exist"+ Just meta+ | meta.size /= descriptor.cell.bundle.bytes -> failVerification "payload bundle size differs from descriptor"+ | otherwise -> LazyByteString.putStrLn (encode descriptor) >> pure ExitSuccess+ Fetch bucket identifier outDir -> case (validateBucket bucket, parseRunId identifier) of+ (Left problem, _) -> usage problem+ (_, Left problem) -> usage problem+ (Right resultsBucket, Right cellRun) -> guardIO $ do+ store <- openStore resultsBucket+ fetched <- fetchCellRun store resultsBucket cellRun outDir+ case fetched of+ Left problem -> fetchFailure problem+ Right tree -> do+ indexed <- deriveCellRunIndex resultsBucket Nothing tree+ case indexed of+ Left problems -> failVerification (Text.pack (show (NonEmpty.toList problems)))+ Right index -> do+ path <- writeCellRunIndex tree index+ TextIO.putStrLn (Text.pack path)+ pure ExitSuccess+ Verify location -> case Text.stripPrefix "gs://" (Text.pack location) of+ Nothing -> guardIO $ do+ let nested = location </> "tree"+ isRunDirectory <- doesDirectoryExist nested+ exists <- doesDirectoryExist location+ if exists then verifyTree (if isRunDirectory then nested else location) else unavailable "verification directory does not exist"+ Just suffix -> case parseResultsUri suffix of+ Left problem -> usage problem+ Right (bucket, identifier) -> guardIO $+ withSystemTempDirectory "kenshou-cell-verify" \temporary -> do+ store <- openStore bucket+ fetched <- fetchCellRun store bucket identifier temporary+ case fetched of+ Left problem -> fetchFailure problem+ Right tree -> verifyTree tree+ Parity localDir cellDir output volatile+ | any (\path -> not ("result.summaries.verdicts." `Text.isPrefixOf` path || "verdicts." `Text.isPrefixOf` path)) volatile -> usage "--volatile must name a field under result.summaries.verdicts or verdicts"+ | otherwise -> guardIO do+ report <- compareForParity (ParityOptions volatile) localDir cellDir+ case output of+ Nothing -> LazyByteString.putStrLn (encode report)+ Just path -> do+ createDirectoryIfMissing True (takeDirectory path)+ LazyByteString.writeFile path (encode report)+ TextIO.putStrLn (Text.pack path)+ pure (if null report.unexpected then ExitSuccess else ExitFailure 1)+ Status location asJson -> withControl location \_ _ snapshot -> do+ if asJson+ then LazyByteString.putStrLn (encode (object ["descriptor" .= snapshot.descriptor, "lease" .= snapshot.lease, "quarantine" .= snapshot.quarantine]))+ else do+ let descriptor = snapshot.descriptor+ TextIO.putStrLn ("cell " <> descriptor.name <> " project=" <> descriptor.project <> " zone=" <> descriptor.zone <> " pg=" <> Text.pack (show descriptor.postgresMajor))+ TextIO.putStrLn ("lease " <> maybe "none" (renderRunId . (.leaseId)) snapshot.lease)+ TextIO.putStrLn ("quarantine " <> maybe "none" (.reason) snapshot.quarantine)+ pure ExitSuccess+ Debug location debugCommand -> withControl location \_ _ snapshot -> do+ result <- runCellDebug snapshot.descriptor debugCommand+ either unavailable pure result+ LeaseCell (LeaseCli location purpose owner ttl waitSeconds startInstances hold asJson)+ | ttl <= 0 || waitSeconds < 0 || Text.null purpose || maybe False Text.null owner -> usage "lease purpose and TTL must be positive; wait cannot be negative"+ | otherwise -> withControl location \store ref observed -> do+ if startInstances then startCellInstances observed.descriptor else pure ()+ defaultOwner <- Text.pack . fromMaybe "kenshou" <$> lookupEnv "USER"+ let request = LeaseRequest (fromMaybe defaultOwner owner) purpose ttl+ start <- getCurrentTime+ let acquire = do+ outcome <- acquireLease store ref request+ case outcome of+ Acquired handle -> do+ record <- leaseSnapshot handle+ if asJson+ then LazyByteString.putStrLn (encode record)+ else TextIO.putStrLn ("lease " <> renderRunId record.leaseId <> " ttl=" <> Text.pack (show record.ttlSeconds))+ if hold+ then withHeartbeat store ref handle (holdLease) `finally` (do _ <- releaseLease store ref handle; pure ())+ else pure ExitSuccess+ Busy current -> do+ now <- getCurrentTime+ if diffUTCTime now start < fromIntegral waitSeconds+ then threadDelay 2000000 >> acquire+ else unavailable ("cell is busy under lease " <> renderRunId current.leaseId)+ Quarantined record -> unavailable ("cell is quarantined: " <> record.reason)+ acquire+ Release location leaseText -> case parseRunId leaseText of+ Left problem -> usage problem+ Right expected -> withControl location \store ref _ -> do+ reattached <- reattachLease store ref expected+ case reattached of+ Nothing -> unavailable "cell has no matching lease"+ Just handle -> do+ released <- releaseLease store ref handle+ if released then TextIO.putStrLn ("released " <> leaseText) >> pure ExitSuccess else unavailable "cell lease changed before release"+ Watch location runText -> case parseRunId runText of+ Left problem -> usage problem+ Right identifier -> withControl location \store ref _ -> do+ terminal <- watchCellRun store ref identifier emitWatchEvent+ case terminal of+ RunRejected rejected -> do+ TextIO.hPutStrLn stderr ("rejected: " <> rejected.reason)+ pure (if rejected.reason `elem` ["lease-mismatch", "payload-digest-mismatch"] then ExitFailure 4 else ExitFailure 2)+ RunSealed status -> do+ TextIO.hPutStrLn stderr ("sealed: " <> Text.pack (show status.outcome))+ pure (if status.outcome == Just Completed then ExitSuccess else ExitFailure 4)+ Route (RouteCli names bucket planFile payloadFile rulesFile outDir coerce ephemeral)+ | length (nub names) /= length names || not (all validCellName names) -> usage "cell names must be valid and unique"+ | planFile == "-" && rulesFile == Just "-" -> usage "only one JSON document may be read from stdin"+ | payloadFile == Just "-" -> usage "payload must name a file"+ | otherwise -> case NonEmpty.nonEmpty names of+ Nothing -> usage "at least one cell is required"+ Just (first :| rest) -> withControl (CellLocation first bucket) \store ref observed -> do+ allowed <- maybe ["tan-nb-exp"] (map Text.strip . Text.splitOn "," . Text.pack) <$> lookupEnv "KENSHOU_GCP_ALLOWED_PROJECTS"+ remaining <- traverse (\name -> readCellSnapshot store (CellRef name ref.controlBucket) allowed) rest+ case sequence remaining of+ Left problem -> unavailable problem+ Right snapshots -> do+ let descriptors = observed.descriptor : fmap (.descriptor) snapshots+ capabilityResult <- loadCapabilitiesFor descriptors+ planResult <- readJsonDocument planFile+ rulesResult <- maybe (pure (Right (registryRoutingRules cli.registry))) readJsonDocument rulesFile+ payloadResult <- traverse eitherDecodeFileStrict' payloadFile+ case (capabilityResult, planResult, rulesResult, sequence payloadResult) of+ (Left problem, _, _, _) -> usage problem+ (_, Left failure, _, _) -> usage (Text.pack failure)+ (_, _, Left failure, _) -> usage (Text.pack failure)+ (_, _, _, Left failure) -> usage (Text.pack failure)+ (Right caches, Right plan, Right rules, Right payload) ->+ case NonEmpty.nonEmpty (zip descriptors caches) >>= \cells -> Just (routeWorkJson cli.registry cells rules (Map.singleton "default" <$> payload) (PrepareOptions coerce ephemeral [] Cold) plan) of+ Nothing -> usage "at least one cell is required"+ Just (Left problem) -> usage problem+ Just (Right documents) -> guardIO do+ exists <- doesPathExist outDir+ if exists+ then usage "route output path already exists; choose a new directory"+ else do+ createDirectoryIfMissing True outDir+ traverse_ (\(name, planDocument) -> LazyByteString.writeFile (outDir </> "plan." <> Text.unpack name <> ".json") (encode planDocument)) (Map.toAscList documents.perCell)+ traverse_ (\planDocument -> LazyByteString.writeFile (outDir </> "plan.local.json") (encode planDocument)) documents.local+ LazyByteString.writeFile (outDir </> "unroutable.json") (encode documents.report)+ TextIO.hPutStrLn stderr ("routed plans: " <> Text.pack (show (Map.size documents.perCell)) <> "; see " <> Text.pack (outDir </> "unroutable.json"))+ pure (if documents.routeComplete then ExitSuccess else ExitFailure 2)+ Submit (SubmitCli location leaseText (SubmitSettings payloadFiles planFile outDir granularity cache settingTexts skip coerce ephemeral rulesFile sink rts dryRun) detached)+ | planFile == "-" && rulesFile == Just "-" -> usage "only one JSON document may be read from stdin"+ | otherwise -> case (parseRunId leaseText, traverse parsePgSetting settingTexts) of+ (Left problem, _) -> usage problem+ (_, Left problem) -> usage problem+ (Right leaseId, Right settings) -> withControl location \store ref observed -> do+ handle <- reattachLease store ref leaseId+ case handle of+ Nothing -> unavailable "cell has no matching lease"+ Just active -> do+ payloadResult <- loadPayloads payloadFiles+ rulesResult <- maybe (pure (Right (registryRoutingRules cli.registry))) readJsonDocument rulesFile+ case (payloadResult, rulesResult) of+ (Left problem, _) -> usage problem+ (_, Left failure) -> usage (Text.pack failure)+ (Right payloads, Right rules) -> do+ capabilityResult <- loadCapabilitiesFor [observed.descriptor]+ case capabilityResult of+ Left problem -> usage problem+ Right [cachedCapabilities] -> do+ bytes <- if planFile == "-" then LazyByteString.getContents else LazyByteString.readFile planFile+ selectedStore <- lookupEnv "KENSHOU_CELL_STORE"+ let options = BuildOptions (PrepareOptions coerce ephemeral settings cache) granularity skip sink rts (24 * 1024 * 1024 * 1024) (20 * 1024 * 1024 * 1024) "0.1.0"+ storeLabel = Text.pack (fromMaybe "gs://" selectedStore)+ built <- buildSession cli.registry observed.descriptor cachedCapabilities rules payloads options storeLabel leaseId bytes+ case built of+ Left problem -> usage problem+ Right session+ | dryRun -> LazyByteString.putStrLn (encode session.journal) >> pure ExitSuccess+ | detached && length session.journal.slices /= 1 -> usage "detached sessions require exactly one slice; use --granularity plan for a compatible plan"+ | otherwise -> guardIO do+ exists <- doesPathExist outDir+ if exists+ then usage "session output path already exists; choose a new directory"+ else do+ if detached+ then do+ slice <- singleSlice session.journal+ let budget = toInteger slice.submission.limits.wallClockSeconds + 600+ if budget > toInteger (maxBound :: Int)+ then ioError (userError "detached lease budget exceeds supported TTL")+ else do+ lease <- leaseSnapshot active+ enlarged <- resizeLease store ref active (max lease.ttlSeconds (fromInteger budget))+ if enlarged then pure () else ioError (userError "cell lease changed before detached submission")+ else pure ()+ traverse_+ ( \(relative, content) -> do+ let destination = outDir </> relative+ createDirectoryIfMissing True (takeDirectory destination)+ LazyByteString.writeFile destination content+ )+ session.workFiles+ if detached+ then do+ slice <- singleSlice session.journal+ let journal = session.journal {leaseMode = Detached}+ journalPath = outDir </> "session.json"+ writeSessionJournal journalPath journal+ work <- LazyByteString.readFile (outDir </> slice.workPath)+ published <- publishSubmission store ref active slice.submission work+ case published of+ Submitted -> do+ now <- getCurrentTime+ checkpoint <- either (ioError . userError . Text.unpack) pure (applyTransition now slice.cellRun SubmissionPublished journal)+ writeSessionJournal journalPath checkpoint+ TextIO.hPutStrLn stderr ("detached session " <> renderRunId checkpoint.sessionId <> " submitted; collect with cell resume --session " <> Text.pack outDir)+ pure ExitSuccess+ other -> unavailable ("detached submission failed: " <> Text.pack (show other))+ else do+ result <- withHeartbeat store ref active \_ -> runPlannedSlices store ref (Bucket observed.descriptor.buckets.results) active (outDir </> "session.json") session.journal emitWatchEvent+ case result of+ Left failure -> unavailable (Text.pack (show failure))+ Right finished -> do+ TextIO.hPutStrLn stderr ("session " <> renderRunId finished.sessionId <> " verified; journal " <> Text.pack (outDir </> "session.json"))+ pure (sessionExitCode finished)+ Right _ -> unavailable "cell capability cache count differs from descriptor count"+ Run (RunCli location settings ttl waitSeconds startInstances detached)+ | ttl <= 0 || waitSeconds < 0 -> usage "lease TTL must be positive and wait cannot be negative"+ | otherwise -> withControl location \store ref observed -> do+ if startInstances then startCellInstances observed.descriptor else pure ()+ defaultOwner <- Text.pack . fromMaybe "kenshou" <$> lookupEnv "USER"+ start <- getCurrentTime+ let request = LeaseRequest defaultOwner "kenshou cell run" ttl+ acquire = do+ outcome <- acquireLease store ref request+ case outcome of+ Acquired handle -> do+ lease <- leaseSnapshot handle+ let execute = runCell (Submit (SubmitCli location (renderRunId lease.leaseId) settings detached)) cli+ releaseCurrent = do+ current <- reattachLease store ref lease.leaseId+ traverse_ (\active -> do _ <- releaseLease store ref active; pure ()) current+ releaseUnlessPublished = do+ let SubmitSettings _ _ outDir _ _ _ _ _ _ _ _ _ _ = settings+ journalPath = outDir </> "session.json"+ present <- doesPathExist journalPath+ if not present+ then releaseCurrent+ else do+ decoded <- readSessionJournal journalPath+ case decoded of+ Left _ -> pure ()+ Right journal | journal.leaseId /= lease.leaseId || journal.cell /= ref.cellName -> releaseCurrent+ Right journal -> do+ markers <- traverse (\slice -> store.statObject ref.controlBucket (ObjectName ("cells/" <> ref.cellName <> "/submissions/" <> renderRunId slice.cellRun <> "/submission.json"))) journal.slices+ if all isNothing markers then releaseCurrent else pure ()+ execute `finally` (if detached then releaseUnlessPublished else releaseCurrent)+ Busy current -> do+ now <- getCurrentTime+ if diffUTCTime now start < fromIntegral waitSeconds+ then threadDelay 2000000 >> acquire+ else unavailable ("cell is busy under lease " <> renderRunId current.leaseId)+ Quarantined record -> unavailable ("cell is quarantined: " <> record.reason)+ acquire+ Probe request -> runProbe cli request+ Pair request -> runPair cli request+ Overhead request -> runCellOverhead cli request+ Resume sessionDir selectedLease startInstances -> guardIO do+ let journalPath = sessionDir </> "session.json"+ present <- doesPathExist journalPath+ if not present+ then unavailable "session.json is missing"+ else do+ decoded <- readSessionJournal journalPath+ case decoded of+ Left problem -> usage problem+ Right journal -> do+ selectedStore <- Text.pack . fromMaybe "gs://" <$> lookupEnv "KENSHOU_CELL_STORE"+ if selectedStore /= journal.store+ then usage "selected cell store differs from the session journal"+ else case traverse parseRunId selectedLease of+ Left problem -> usage problem+ Right requested -> withControl (CellLocation journal.cell (Just journal.controlBucket)) \store ref observed -> do+ if observed.descriptor.buckets.results /= journal.resultsBucket+ then usage "cell results bucket differs from the session journal"+ else do+ if startInstances then startCellInstances observed.descriptor else pure ()+ let expected = fromMaybe journal.leaseId requested+ existing <- reattachLease store ref expected+ case existing of+ Just _ | journal.leaseMode == Detached -> continueDetached store ref journalPath journal+ Just handle -> continueSession store ref journalPath journal handle `finally` (do _ <- releaseLease store ref handle; pure ())+ Nothing | requested /= Nothing -> unavailable "cell has no matching selected lease"+ Nothing | journal.leaseMode == Detached -> continueDetached store ref journalPath journal+ Nothing -> do+ owner <- Text.pack . fromMaybe "kenshou" <$> lookupEnv "USER"+ acquired <- acquireLease store ref (LeaseRequest owner ("resume " <> renderRunId journal.sessionId) 120)+ case acquired of+ Busy current -> unavailable ("cell is busy under lease " <> renderRunId current.leaseId)+ Quarantined record -> unavailable ("cell is quarantined: " <> record.reason)+ Acquired handle -> continueSession store ref journalPath journal handle `finally` (do _ <- releaseLease store ref handle; pure ())+ Exec workFile outDir -> cellExec cli.registry workFile outDir++continueSession :: ObjectStore -> CellRef -> FilePath -> SessionJournal -> LeaseHandle -> IO ExitCode+continueSession store ref journalPath journal handle = do+ resumed <- withHeartbeat store ref handle \_ -> resumeHeldSession store ref (Bucket journal.resultsBucket) handle journalPath emitWatchEvent+ case resumed of+ Left failure -> unavailable (Text.pack (show failure))+ Right finished -> do+ TextIO.hPutStrLn stderr ("session " <> renderRunId finished.sessionId <> " verified; journal " <> Text.pack journalPath)+ pure (sessionExitCode finished)++continueDetached :: ObjectStore -> CellRef -> FilePath -> SessionJournal -> IO ExitCode+continueDetached store ref journalPath journal = do+ observed <- resumeObservedSlices store ref (Bucket journal.resultsBucket) journalPath emitWatchEvent+ case observed of+ Left failure -> unavailable (Text.pack (show failure))+ Right finished ->+ if not (null finished.slices) && all (\slice -> slice.state `elem` [SliceRejected, SliceVerified]) finished.slices+ then do+ current <- reattachLease store ref journal.leaseId+ traverse_ (\active -> do _ <- releaseLease store ref active; pure ()) current+ TextIO.hPutStrLn stderr ("detached session " <> renderRunId finished.sessionId <> " collected; journal " <> Text.pack journalPath)+ pure (sessionExitCode finished)+ else unavailable "detached session has not reached a terminal cell status; resume later"++singleSlice :: SessionJournal -> IO SliceJournal+singleSlice journal = case journal.slices of+ [slice] -> pure slice+ _ -> ioError (userError "detached session must contain exactly one slice")++sessionExitCode :: SessionJournal -> ExitCode+sessionExitCode journal+ | any ((/= SliceVerified) . (.state)) journal.slices = ExitFailure 4+ | any ((/= Just Completed) . (.cellOutcome)) journal.slices = ExitFailure 4+ | any ((== Just 4) . (.entryExitCode)) journal.slices = ExitFailure 4+ | any ((== Just 3) . (.entryExitCode)) journal.slices = ExitFailure 3+ | any ((== Just 1) . (.entryExitCode)) journal.slices = ExitFailure 1+ | all ((== Just 0) . (.entryExitCode)) journal.slices = ExitSuccess+ | otherwise = ExitFailure 4++runProbe :: CliEnv -> ProbeCli -> IO ExitCode+runProbe cli (ProbeCli location payload output) = withControl location \_ _ observed -> do+ cacheDirectory <- fromMaybe ".dev/cells" <$> lookupEnv "KENSHOU_CELL_CAPABILITIES_DIR"+ identifier <- newRunId+ let sessionDirectory = cacheDirectory </> "probes" </> Text.unpack (renderRunId identifier)+ planFile = sessionDirectory <> ".plan.json"+ cacheFile = fromMaybe (cacheDirectory </> Text.unpack observed.descriptor.name <> ".capabilities.json") output+ settings = SubmitSettings [payload] planFile sessionDirectory GranularityRun Cold [] False False False Nothing NullSink Nothing False+ planArguments =+ [ "plan",+ "--all",+ "--select",+ "selftest/remote/correctness/cell-environment",+ "--placement",+ "cell",+ "--seed",+ "7",+ "--dim",+ "pg.durability=durable",+ "--dim",+ "pg.version=" <> show observed.descriptor.postgresMajor,+ "--out",+ planFile+ ]+ createDirectoryIfMissing True (takeDirectory planFile)+ executable <- getExecutablePath+ (planned, _, planErrors) <- readProcessWithExitCode executable planArguments ""+ if planned /= ExitSuccess+ then usage ("cell probe could not plan its environment scenario: " <> Text.pack planErrors)+ else do+ result <- runCell (Run (RunCli location settings 120 0 False False)) cli+ if result /= ExitSuccess+ then pure result+ else do+ saved <- readSessionJournal (sessionDirectory </> "session.json")+ case saved of+ Right journal -> case journal.slices of+ [slice] | slice.state == SliceVerified -> case slice.runIds of+ [nested] -> do+ let runResult = takeDirectory sessionDirectory </> Text.unpack (renderRunId slice.cellRun) </> "tree" </> "output" </> Text.unpack (renderRunId nested) </> "run-result.json"+ document <- eitherDecodeFileStrict' runResult+ case document >>= extractProbeCapabilities slice.cellRun of+ Left problem -> failVerification ("cell probe result has no valid capabilities: " <> Text.pack problem)+ Right capabilities -> do+ now <- getCurrentTime+ let cache = CellCapabilities observed.descriptor.name (descriptorDigest observed.descriptor) slice.cellRun now capabilities+ temporary = cacheFile <> "." <> Text.unpack (renderRunId identifier) <> ".tmp"+ createDirectoryIfMissing True (takeDirectory cacheFile)+ LazyByteString.writeFile temporary (encode cache)+ renameFile temporary cacheFile+ TextIO.putStrLn (Text.pack cacheFile)+ pure ExitSuccess+ _ -> failVerification "cell probe verified a slice with an unexpected nested run count"+ _ -> failVerification "cell probe did not verify exactly one slice"+ Left problem -> failVerification ("cell probe session is invalid: " <> problem)++extractProbeCapabilities :: RunId -> Value -> Either String (Map.Map Text Value)+extractProbeCapabilities expectedCellRun (Object root) = do+ schema <- member "schema" root+ if schema == String "kenshou.run-result/v1" then pure () else Left "unexpected run-result schema"+ outcome <- member "outcome" root+ if outcome == String "passed" then pure () else Left "probe run did not pass"+ fingerprint <- member "fingerprint" root+ cell <- asObject "fingerprint.cell" =<< member "cell" =<< asObject "fingerprint" fingerprint+ cellRun <- member "cellRun" cell+ if cellRun == Aeson.toJSON expectedCellRun then pure () else Left "probe cell run identity differs from the verified slice"+ capabilities <- member "capabilities" cell+ case Aeson.fromJSON capabilities of+ Aeson.Error problem -> Left problem+ Aeson.Success values+ | all primitive (Map.elems values) && all (`Map.member` values) required -> Right values+ | otherwise -> Left "probe capabilities are missing or contain a non-primitive value"+ where+ primitive (Bool _) = True+ primitive (String _) = True+ primitive (Number _) = True+ primitive _ = False+ required = ["postgres.major", "postgres.pg_partman", "postgres.second-server", "postgres.control-hook", "broker", "otlp.null", "otlp.file", "fault-hook"]+ member :: Text -> KeyMap.KeyMap Value -> Either String Value+ member name fields = maybe (Left ("missing " <> Text.unpack name)) Right (KeyMap.lookup (Key.fromText name) fields)+ asObject :: String -> Value -> Either String (KeyMap.KeyMap Value)+ asObject _ (Object fields) = Right fields+ asObject label _ = Left (label <> " is not an object")+extractProbeCapabilities _ _ = Left "run result is not an object"++runPair :: CliEnv -> PairCli -> IO ExitCode+runPair cli (PairCli location candidateFile baselineFile planFile selectedPairs ordering policyFile settingTexts replacementOption startInstances outDir) = guardIO do+ settings <- case traverse parsePgSetting settingTexts of+ Left problem -> pure (Left problem)+ Right parsed -> pure (Right parsed)+ policyBytes <- ByteString.readFile policyFile+ sourceDocument <- eitherDecodeFileStrict' planFile :: IO (Either String Value)+ payloadResult <- loadPayloads ["baseline=" <> baselineFile, "candidate=" <> candidateFile]+ case (settings, decodePolicy policyBytes, sourceDocument >>= either (Left . Text.unpack) Right . decodeWorkPlan, payloadResult) of+ (Left problem, _, _, _) -> usage problem+ (_, Left problem, _, _) -> usage problem+ (_, _, Left problem, _) -> usage (Text.pack problem)+ (_, _, _, Left problem) -> usage problem+ (Right selectedPgSettings, Right policy, Right source, Right payloads) -> do+ let requested = fromMaybe policy.minimumPairs selectedPairs+ replacements = if replacementOption < 0 then requested else replacementOption+ request = PairRequest "candidate" "baseline" requested ordering (unSeed source.policy.seed) replacements+ pgSettings = selectedPgSettings <> [(name, assigned) | (name, assigned) <- [("checkpoint_timeout", "30min"), ("max_wal_size", "16GB")], name `notElem` fmap fst selectedPgSettings]+ case pairPlan request source of+ Left problem -> usage problem+ Right schedule -> do+ let groups = nub [trial.group | entry <- schedule.runs, Just trial <- [entry.trial]]+ if length groups /= 1+ then usage "cell pair currently requires one benchmark configuration per plan"+ else withControl location \store ref observed -> do+ exists <- doesPathExist outDir+ if exists+ then usage "pair output path already exists; choose a new directory"+ else do+ if startInstances then startCellInstances observed.descriptor else pure ()+ defaultOwner <- Text.pack . fromMaybe "kenshou" <$> lookupEnv "USER"+ acquired <- acquireLease store ref (LeaseRequest defaultOwner "kenshou cell pair" 120)+ case acquired of+ Busy lease -> unavailable ("cell is busy under lease " <> renderRunId lease.leaseId)+ Quarantined record -> unavailable ("cell is quarantined: " <> record.reason)+ Acquired handle -> do+ lease <- leaseSnapshot handle+ let releaseCurrent = do+ current <- reattachLease store ref lease.leaseId+ traverse_ (\active -> do _ <- releaseLease store ref active; pure ()) current+ pairUnderLease cli location lease.leaseId baselineFile candidateFile request schedule policy payloads pgSettings outDir `finally` releaseCurrent++runCellOverhead :: CliEnv -> OverheadCli -> IO ExitCode+runCellOverhead cli (OverheadCli location payloadFile sink settingTexts startInstances options) = guardIO do+ let requestedScenario =+ parseScenarioId options.scenario >>= \identifier ->+ maybe (Left ("unknown scenario " <> options.scenario)) Right (lookupScenario cli.registry identifier)+ input = LocalOverhead.parseInputs options+ settings = traverse parsePgSetting settingTexts+ selectedPolicy <- LocalOverhead.loadPolicy options.policy+ payloadResult <- loadPayloads [payloadFile]+ case (requestedScenario, input, settings, selectedPolicy, payloadResult) of+ (Left problem, _, _, _, _) -> usage problem+ (_, Left problem, _, _, _) -> usage problem+ (_, _, Left problem, _, _) -> usage problem+ (_, _, _, Left problem, _) -> usage problem+ (_, _, _, _, Left problem) -> usage problem+ (Right scenario, Right (factors, fixedKnobs, fixedDimensions), Right pgSettings, Right policy, Right payloads) ->+ case Map.lookup "default" payloads of+ Nothing -> usage "cell overhead requires one default payload"+ Just payload -> do+ resumed <- if options.resume || options.analyseOnly then LocalOverhead.findResumeDirectory options.out scenario.id else pure Nothing+ case (options.resume || options.analyseOnly, resumed) of+ (True, Nothing) -> usage "no matching overhead state was found under --out"+ (_, Just directory) -> do+ loaded <- loadOverheadState directory+ case loaded of+ Left problem -> usage ("invalid overhead state: " <> Text.pack problem)+ Right state -> continue directory state.plan (Just state) payload policy pgSettings+ (_, Nothing) -> do+ identifier <- renderRunId <$> newRunId+ let directory = options.out </> ("overhead-" <> Text.unpack identifier)+ request = OverheadRequest identifier scenario.id factors options.mode options.control options.trials fixedKnobs fixedDimensions options.seed options.settleSeconds options.retries+ case planOverhead request scenario of+ Left (UsageError problem) -> usage problem+ Right plan -> continue directory plan Nothing payload policy pgSettings+ where+ continue directory plan existing payload policy pgSettings = do+ configuredBucket <- lookupEnv "KENSHOU_CELL_CONTROL_BUCKET"+ let CellLocation name selectedBucket = location+ bucketName = fromMaybe (maybe "tan-nb-exp-cells-control" Text.pack configuredBucket) selectedBucket+ configuration = object ["schema" .= ("kenshou.cell-overhead/v1" :: Text), "cell" .= name, "controlBucket" .= bucketName, "payload" .= payload, "otlpSink" .= show sink, "pgSettings" .= pgSettings]+ configPath = directory </> "cell-overhead.json"+ case existing of+ Nothing -> do+ present <- doesPathExist directory+ if present+ then usage "overhead output path already exists"+ else do+ createDirectoryIfMissing True directory+ LazyByteString.writeFile configPath (encode configuration)+ execute directory plan payload policy pgSettings+ Just state -> do+ saved <- eitherDecodeFileStrict' configPath :: IO (Either String Value)+ if saved /= Right configuration+ then usage "cell overhead resume arguments differ from the saved cell and payload configuration"+ else+ if options.analyseOnly+ then analyse directory plan state policy+ else execute directory plan payload policy pgSettings++ execute directory plan payload policy pgSettings = withControl location \store ref observed -> do+ if startInstances then startCellInstances observed.descriptor else pure ()+ owner <- Text.pack . fromMaybe "kenshou" <$> lookupEnv "USER"+ acquired <- acquireLease store ref (LeaseRequest owner "kenshou cell overhead" 120)+ case acquired of+ Busy current -> unavailable ("cell is busy under lease " <> renderRunId current.leaseId)+ Quarantined record -> unavailable ("cell is quarantined: " <> record.reason)+ Acquired handle -> do+ lease <- leaseSnapshot handle+ let releaseCurrent = do+ current <- reattachLease store ref lease.leaseId+ traverse_ (\active -> do _ <- releaseLease store ref active; pure ()) current+ hooks = overheadHooks plan (runCellOverheadChild cli location lease.leaseId payloadFile payload sink pgSettings directory)+ withHeartbeat+ store+ ref+ handle+ ( \_ -> do+ state <- executeOverhead hooks plan directory+ analyseOverheadResult options.json directory plan state policy hooks+ )+ `finally` releaseCurrent++ analyse directory plan state policy = analyseOverheadResult options.json directory plan state policy (overheadHooks plan (\_ _ -> pure (ExitFailure 4)))++overheadHooks :: OverheadPlan -> (RunSpec -> FilePath -> IO ExitCode) -> OverheadHooks+overheadHooks plan runChild =+ OverheadHooks runChild compareCellOverhead (Just leakCheck)+ where+ leakCheck directory = fmap (either (const Nothing) (Just . Aeson.toJSON)) (analyseRunDirectory directory defaultLeakSpec plan.seed)++-- On a cell, the resolver binds the chosen collector address only for an+-- sdk-otlp tracing arm. That address is a consequence of the declared tracing+-- change, so name its knob when the two arms actually have different values.+compareCellOverhead :: Policy -> NonEmpty VaryingAxis -> [FilePath] -> [FilePath] -> IO (Either CompareError Comparison)+compareCellOverhead policy axes baselines candidates = do+ derivedEndpoint <- case (baselines, candidates) of+ (baseline : _, candidate : _)+ | VaryDimension "telemetry.tracing" `elem` NonEmpty.toList axes -> do+ left <- readInputs baseline+ right <- readInputs candidate+ pure case (left, right) of+ (Just a, Just b) ->+ case (tracing a, tracing b, endpoint a, endpoint b) of+ (Just (String "sdk-otlp"), Just _, Just leftEndpoint, Just rightEndpoint) -> leftEndpoint /= rightEndpoint+ (Just _, Just (String "sdk-otlp"), Just leftEndpoint, Just rightEndpoint) -> leftEndpoint /= rightEndpoint+ _ -> False+ _ -> False+ _ -> pure False+ let selectedAxes = if derivedEndpoint then axes <> (VaryKnob "otel.endpoint" :| []) else axes+ compareRuns policy selectedAxes baselines candidates+ where+ tracing = varyingValue (VaryDimension "telemetry.tracing")+ endpoint = varyingValue (VaryKnob "otel.endpoint")+ readInputs directory = do+ decoded <- eitherDecodeFileStrict' (directory </> "run-result.json") :: IO (Either String Value)+ pure (either (const Nothing) compatibilityInputs decoded)++analyseOverheadResult :: Bool -> FilePath -> OverheadPlan -> OverheadState -> OverheadPolicy -> OverheadHooks -> IO ExitCode+analyseOverheadResult asJson directory plan state policy hooks = do+ report <- analyseOverhead hooks policy plan state directory+ if asJson+ then LazyByteString.putStrLn (encode report)+ else TextIO.putStrLn ("verdict " <> LocalOverhead.verdictText report.verdict <> " " <> Text.pack directory)+ pure case overheadVerdictExitCode report.verdict of 0 -> ExitSuccess; code -> ExitFailure code++runCellOverheadChild :: CliEnv -> CellLocation -> RunId -> FilePath -> PayloadDescriptor -> OtlpSink -> [(Text, Text)] -> FilePath -> RunSpec -> FilePath -> IO ExitCode+runCellOverheadChild cli location leaseId payloadFile payload sink pgSettings directory rawSpec runsRoot = case rawSpec.runId of+ Nothing -> pure (ExitFailure 4)+ Just runId -> do+ planId <- newRunId+ now <- getCurrentTime+ selector <- either (ioError . userError . Text.unpack) pure (parseSelector "**")+ seed <- either (ioError . userError . Text.unpack) pure (mkSeed 1)+ let reason = Reason (Change (ComponentRef (ComponentId "cell-overhead") Nothing) Named "telemetry overhead slot") [] selector 0+ spec = rawSpec {environment = rawSpec.environment {Kenshou.Core.RunSpec.placement = RunOnCell}}+ context = PlanContext Nothing "cell-overhead" payload.cohort payload.cohortIdentity.identityPlanHash.unPlanHash (PlanInputs (object ["overhead" .= True])) [] []+ plan = RunPlan planId now context (defaultPlanPolicy seed) [PlannedRun 0 runId 10 (reason :| []) Nothing spec] [] 10+ planPath = directory </> "cell-sessions" </> Text.unpack (renderRunId runId) <> ".plan.json"+ sessionDir = directory </> "cell-sessions" </> Text.unpack (renderRunId runId)+ settings = SubmitSettings [payloadFile] planPath sessionDir GranularityRun Cold (fmap (\(name, assigned) -> name <> "=" <> assigned) pgSettings) False True False Nothing sink Nothing False+ createDirectoryIfMissing True (takeDirectory planPath)+ LazyByteString.writeFile planPath (encode plan)+ result <- runCell (Submit (SubmitCli location (renderRunId leaseId) settings False)) cli+ journal <- readSessionJournal (sessionDir </> "session.json")+ case journal of+ Right saved -> case saved.slices of+ [slice] | slice.state == SliceVerified && slice.runIds == [runId] -> do+ let target = takeDirectory sessionDir </> Text.unpack (renderRunId slice.cellRun) </> "tree" </> "output" </> Text.unpack (renderRunId runId)+ link = runsRoot </> Text.unpack (renderRunId runId)+ available <- doesDirectoryExist target+ if available+ then do+ createDirectoryIfMissing True runsRoot+ absolute <- makeAbsolute target+ createDirectoryLink absolute link+ pure result+ else pure (ExitFailure 4)+ _ -> pure (ExitFailure 4)+ Left _ -> pure (ExitFailure 4)++pairUnderLease :: CliEnv -> CellLocation -> RunId -> FilePath -> FilePath -> PairRequest -> RunPlan -> Policy -> Map.Map Text PayloadDescriptor -> [(Text, Text)] -> FilePath -> IO ExitCode+pairUnderLease cli location leaseId baselineFile candidateFile request schedule policy payloads pgSettings outDir = do+ createDirectoryIfMissing True outDir+ LazyByteString.writeFile (outDir </> "pair-plan.json") (encode schedule)+ let requested = request.pairs+ limit = requested + request.maxReplacements+ batchFor predicate = filter (\entry -> maybe False predicate entry.trial) schedule.runs+ initial = batchFor (\trial -> trial.index < requested)+ runRound :: Int -> [PlannedRun] -> IO (Either Text [PairObservation])+ runRound roundNumber entries = do+ let planPath = outDir </> ("round-" <> show roundNumber <> ".plan.json")+ sessionDir = outDir </> ("round-" <> show roundNumber)+ plan = schedule {runs = entries, estimateMinutes = sum (fmap (.estimateMinutes) entries)}+ settings = SubmitSettings ["baseline=" <> baselineFile, "candidate=" <> candidateFile] planPath sessionDir GranularityRun Cold (fmap (\(name, assigned) -> name <> "=" <> assigned) pgSettings) False False False Nothing NullSink Nothing False+ LazyByteString.writeFile planPath (encode plan)+ result <- runCell (Submit (SubmitCli location (renderRunId leaseId) settings False)) cli+ journal <- readSessionJournal (sessionDir </> "session.json")+ case journal of+ Left problem -> pure (Left ("pair submission did not produce a readable journal (" <> Text.pack (show result) <> "): " <> problem))+ Right verified -> case find ((== SliceRejected) . (.state)) verified.slices of+ Just rejected -> pure (Left ("pair submission rejected: " <> fromMaybe "unknown reason" rejected.rejectionReason))+ Nothing -> case find ((/= SliceVerified) . (.state)) verified.slices of+ Just incomplete -> pure (Left ("pair submission is incomplete at " <> renderRunId incomplete.cellRun <> "; resume " <> Text.pack sessionDir))+ Nothing -> pure (collectPairObservations outDir entries verified)+ continue roundNumber nextIndex observed = do+ let valid = validPairIndexes observed+ if length valid >= request.pairs || nextIndex >= limit+ then finishPair policy payloads outDir request observed valid+ else do+ let entries = batchFor (\trial -> trial.index == nextIndex)+ next <- runRound roundNumber entries+ case next of+ Left problem -> unavailable problem+ Right results -> continue (roundNumber + 1) (nextIndex + 1) (observed <> results)+ initialResult <- runRound 0 initial+ case initialResult of+ Left problem -> unavailable problem+ Right observed -> continue 1 requested observed++collectPairObservations :: FilePath -> [PlannedRun] -> SessionJournal -> Either Text [PairObservation]+collectPairObservations outDir entries journal = traverse collect entries+ where+ collect entry = case find (elem entry.runId . (.runIds)) journal.slices of+ Nothing -> Left ("pair journal omits planned run " <> renderRunId entry.runId)+ Just slice ->+ let path = outDir </> Text.unpack (renderRunId slice.cellRun) </> "tree" </> "output" </> Text.unpack (renderRunId entry.runId)+ passed = slice.state == SliceVerified && slice.cellOutcome == Just Completed && slice.entryExitCode == Just 0+ in Right (PairObservation entry path passed)++validPairIndexes :: [PairObservation] -> [Int]+validPairIndexes observations = sort [index | ((_, index), PairValid) <- judgePairs (fmap asSlice observations)]+ where+ asSlice (PairObservation entry _ passed) = case entry.trial of+ Just trial -> SliceResult trial.group trial.index (if trial.arm == "baseline" then Baseline else Candidate) passed+ Nothing -> error "paired plan lost trial membership"++finishPair :: Policy -> Map.Map Text PayloadDescriptor -> FilePath -> PairRequest -> [PairObservation] -> [Int] -> IO ExitCode+finishPair policy payloads outDir request observations valid =+ if null valid+ then unavailable "all paired trials failed; no comparison can be computed"+ else do+ let chosen = take request.pairs valid+ pathFor index arm = case find (matches index arm) observations of+ Just (PairObservation _ path _) -> path+ Nothing -> error "valid pair has no run path"+ matches index arm (PairObservation entry _ _) = case entry.trial of+ Just trial -> trial.index == index && trial.arm == arm+ Nothing -> False+ baselines = fmap (`pathFor` "baseline") chosen+ candidates = fmap (`pathFor` "candidate") chosen+ varying = case (Map.lookup "baseline" payloads, Map.lookup "candidate" payloads) of+ (Just baseline, Just candidate) | baseline.cohortIdentity.identityPlanHash /= candidate.cohortIdentity.identityPlanHash -> VaryCohort+ _ -> VaryControl+ compared <- compareRuns policy (varying :| []) baselines candidates+ case compared of+ Left (CompareError problem) -> usage ("cell pair comparison failed: " <> problem)+ Right comparison -> do+ LazyByteString.writeFile (outDir </> "comparison.json") (encode comparison <> "\n")+ TextIO.putStrLn ("comparison " <> Text.pack (show comparison.verdict) <> "; valid pairs " <> Text.pack (show (length chosen)) <> "/" <> Text.pack (show request.pairs))+ pure case verdictExitCode comparison.verdict of 0 -> ExitSuccess; code -> ExitFailure code++parsePgSetting :: Text -> Either Text (Text, Text)+parsePgSetting setting = case Text.breakOn "=" setting of+ (name, assignment) | not (Text.null name) && Text.length assignment > 1 -> Right (name, Text.drop 1 assignment)+ _ -> Left "--pg-setting must be KEY=VALUE"++loadPayloads :: [String] -> IO (Either Text (Map.Map Text PayloadDescriptor))+loadPayloads paths = do+ loaded <- traverse load paths+ pure do+ entries <- sequence loaded+ let labels = fmap fst entries+ if length (nub labels) == length labels+ then Right (Map.fromList entries)+ else Left "payload labels must be unique"+ where+ load input = case break (== '=') input of+ (label, '=' : path) | not (null label) && not (null path) && path /= "-" -> readPayload (Text.pack label) path+ (_, '=' : _) -> pure (Left "--payload must be [LABEL=]FILE with a nonempty file path")+ (_, _) | input == "-" -> pure (Left "payload must name a file")+ _ -> readPayload "default" input+ readPayload label path = do+ result <- eitherDecodeFileStrict' path+ pure (either (Left . Text.pack) (\descriptor -> Right (label, descriptor)) result)++loadCapabilitiesFor :: [CellDescriptor] -> IO (Either Text [Maybe CellCapabilities])+loadCapabilitiesFor descriptors = do+ directory <- fromMaybe ".dev/cells" <$> lookupEnv "KENSHOU_CELL_CAPABILITIES_DIR"+ sequence <$> traverse (loadCellCapabilities directory) descriptors++readJsonDocument :: (FromJSON value) => FilePath -> IO (Either String value)+readJsonDocument "-" = eitherDecode <$> LazyByteString.getContents+readJsonDocument path = eitherDecodeFileStrict' path++withControl :: CellLocation -> (ObjectStore -> CellRef -> CellSnapshot -> IO ExitCode) -> IO ExitCode+withControl (CellLocation name selectedBucket) operation = do+ configured <- lookupEnv "KENSHOU_CELL_CONTROL_BUCKET"+ let bucketName = fromMaybe (maybe "tan-nb-exp-cells-control" Text.pack configured) selectedBucket+ case validateBucket bucketName of+ Left problem -> usage problem+ Right bucket+ | not (validCellName name) -> usage "invalid cell name"+ | otherwise -> guardIO do+ store <- openStore bucket+ allowed <- maybe ["tan-nb-exp"] (map Text.strip . Text.splitOn "," . Text.pack) <$> lookupEnv "KENSHOU_GCP_ALLOWED_PROJECTS"+ let ref = CellRef name bucket+ snapshot <- readCellSnapshot store ref allowed+ case snapshot of+ Left problem -> unavailable problem+ Right observed -> operation store ref observed++startCellInstances :: CellDescriptor -> IO ()+startCellInstances descriptor = do+ let names = descriptor.instances.postgres : descriptor.instances.drivers <> [descriptor.instances.monitoring] <> maybe [] pure descriptor.instances.broker+ location = ["--project=" <> Text.unpack descriptor.project, "--zone=" <> Text.unpack descriptor.zone]+ states <-+ traverse+ ( \name -> do+ (code, output, failure) <- readProcessWithExitCode "gcloud" (["compute", "instances", "describe", Text.unpack name] <> location <> ["--format=value(status)"]) ""+ if code == ExitSuccess+ then pure (name, Text.strip (Text.pack output))+ else ioError (userError ("cannot inspect cell instance " <> Text.unpack name <> ": " <> failure))+ )+ names+ case [(name, state) | (name, state) <- states, state `notElem` ["RUNNING", "TERMINATED"]] of+ (name, state) : _ -> ioError (userError ("cell instance " <> Text.unpack name <> " is " <> Text.unpack state))+ [] -> do+ let stopped = [Text.unpack name | (name, "TERMINATED") <- states]+ if null stopped+ then pure ()+ else do+ (code, _, failure) <- readProcessWithExitCode "gcloud" (["compute", "instances", "start"] <> stopped <> location <> ["--quiet"]) ""+ if code == ExitSuccess+ then TextIO.hPutStrLn stderr ("started " <> Text.pack (show (length stopped)) <> " instances for cell " <> descriptor.name)+ else ioError (userError ("could not start cell " <> Text.unpack descriptor.name <> ": " <> failure))++holdLease :: IO Bool -> IO ExitCode+holdLease stillHeld = do+ held <- stillHeld+ if held then threadDelay 1000000 >> holdLease stillHeld else unavailable "lease was lost"++emitWatchEvent :: WatchEvent -> IO ()+emitWatchEvent event = case event of+ PhaseChanged phase -> TextIO.hPutStrLn stderr ("phase: " <> Text.pack (show phase))+ StdoutChunk bytes -> LazyByteString.hPut stdout bytes >> hFlush stdout+ StderrChunk bytes -> LazyByteString.hPut stderr bytes >> hFlush stderr++verifyTree :: FilePath -> IO ExitCode+verifyTree tree = do+ verified <- verifyCellRun tree+ case verified of+ Left problems -> failVerification (Text.pack (show (NonEmpty.toList problems)))+ Right manifest -> TextIO.putStrLn ("verified " <> renderRunId manifest.runId) >> pure ExitSuccess++openStore :: Bucket -> IO ObjectStore+openStore bucket = do+ selected <- lookupEnv "KENSHOU_CELL_STORE"+ case selected of+ Nothing -> newTokenProvider bucket >>= newGcsStore+ Just setting -> case Text.stripPrefix "file:" (Text.pack setting) of+ Just root | not (Text.null root) -> newFileStore (Text.unpack root)+ _ -> ioError (userError "KENSHOU_CELL_STORE must be file:<dir> or unset for GCS")++parseResultsUri :: Text -> Either Text (Bucket, RunId)+parseResultsUri suffix = case Text.splitOn "/" suffix of+ [bucket, "runs", identifier] -> do+ checked <- validateBucket bucket+ parsed <- parseRunId identifier+ pure (checked, parsed)+ _ -> Left "expected gs://BUCKET/runs/CELL_RUN_ID"++parsePayloadUri :: Text -> Either Text (Bucket, ObjectName)+parsePayloadUri uri = case Text.stripPrefix "gs://" uri of+ Nothing -> Left "payload bundle URI must use gs://"+ Just suffix -> case Text.breakOn "/" suffix of+ (bucketName, path) | not (Text.null path) -> do+ bucket <- validateBucket bucketName+ pure (bucket, ObjectName (Text.drop 1 path))+ _ -> Left "payload bundle URI must name an object"++validateBucket :: Text -> Either Text Bucket+validateBucket bucket+ | Text.null bucket || Text.any (\character -> not (character `elem` (['a' .. 'z'] <> ['A' .. 'Z'] <> ['0' .. '9'] <> "-_."))) bucket = Left "invalid results bucket"+ | otherwise = Right (Bucket bucket)++guardIO :: IO ExitCode -> IO ExitCode+guardIO operation = do+ result <- try operation :: IO (Either SomeException ExitCode)+ case result of+ Left failure | Just async <- (fromException failure :: Maybe SomeAsyncException) -> throwIO async+ Left failure -> unavailable (Text.pack (displayException failure))+ Right code -> pure code++fetchFailure :: FetchError -> IO ExitCode+fetchFailure problem = case problem of+ Unsealed -> unavailable "cell run is not sealed"+ ObjectChanged _ -> unavailable (Text.pack (show problem))+ _ -> failVerification (Text.pack (show problem))++unavailable :: Text -> IO ExitCode+unavailable problem = TextIO.hPutStrLn stderr ("kenshou cell: " <> problem) >> pure (ExitFailure 4)++failVerification :: Text -> IO ExitCode+failVerification problem = TextIO.hPutStrLn stderr ("kenshou cell: " <> problem) >> pure (ExitFailure 1)++usage :: Text -> IO ExitCode+usage problem = TextIO.hPutStrLn stderr ("kenshou cell: " <> problem) >> pure (ExitFailure 2)
+ src/Kenshou/Cli/Command/Cohort.hs view
@@ -0,0 +1,15 @@+module Kenshou.Cli.Command.Cohort (cohortCommand) where++import Kenshou.Cli.Cohort (runCohortCommand)+import Kenshou.Cli.Options (cohortCommandParser)+import Kenshou.Core.Cli (CliCommand (..), CliGroup (..))++cohortCommand :: CliCommand+cohortCommand =+ CliCommand+ { name = "cohort",+ description = "Inspect the pinned runtime cohort",+ group = Maintenance,+ hidden = False,+ parser = (\command _ -> runCohortCommand command) <$> cohortCommandParser+ }
+ src/Kenshou/Cli/Command/Execute.hs view
@@ -0,0 +1,64 @@+{-# LANGUAGE FieldSelectors #-}++module Kenshou.Cli.Command.Execute (executeCommand) where++import Control.Exception (IOException, displayException, try)+import Data.ByteString qualified as ByteString+import Data.Text qualified as Text+import Data.Text.IO qualified as Text.IO+import Kenshou.Core.Cli+import Kenshou.Plan.Execute+import Kenshou.Plan.Summary+import Options.Applicative+import System.Exit (ExitCode (..))+import System.FilePath ((</>))+import System.IO (stderr)+import System.IO.Temp (withSystemTempDirectory)++data ExecuteCliOptions = ExecuteCliOptions+ { planSource :: InputSource,+ outDir :: FilePath,+ resume :: Bool,+ failFast :: Bool,+ environmentSource :: Maybe InputSource,+ timeoutFactor :: Maybe Double+ }++executeCommand :: CliCommand+executeCommand = CliCommand "execute" "Execute or resume a run plan" Execution False (runExecute <$> executeParser)++executeParser :: Parser ExecuteCliOptions+executeParser =+ ExecuteCliOptions+ <$> option (parseInputSource <$> str) (long "plan" <> metavar "FILE" <> help "Read a kenshou.run-plan/v1 document; use - for stdin")+ <*> strOption (long "out" <> metavar "DIR" <> help "Execution output directory")+ <*> switch (long "resume" <> help "Resume the identical plan in an existing output directory")+ <*> switch (long "fail-fast" <> help "Stop after the first blocking failure")+ <*> optional (option (parseInputSource <$> str) (long "environment" <> metavar "FILE" <> help "Replace each run's environment object; use - for stdin"))+ <*> optional (option auto (long "timeout-factor" <> metavar "F" <> help "Limit runs to F times their estimate"))++runExecute :: ExecuteCliOptions -> CliEnv -> IO ExitCode+runExecute options _+ | stdinCount options > 1 = Text.IO.hPutStrLn stderr "kenshou: at most one document input may use standard input" >> pure (ExitFailure 2)+ | maybe False (<= 0) options.timeoutFactor = Text.IO.hPutStrLn stderr "kenshou: --timeout-factor must be positive" >> pure (ExitFailure 2)+ | otherwise = withSystemTempDirectory "kenshou-execute" \temporary -> do+ planFile <- materialize temporary "plan.json" options.planSource+ environment <- traverse (materialize temporary "environment.json") options.environmentSource+ let executeOptions = ExecuteOptions planFile options.outDir options.resume options.failFast environment options.timeoutFactor+ result <- try (executePlan executeOptions) :: IO (Either IOException PlanSummary)+ case result of+ Left err -> Text.IO.hPutStrLn stderr ("kenshou: " <> Text.pack (displayException err)) >> pure (ExitFailure 2)+ Right summary -> do+ Text.IO.putStrLn ("exit=" <> Text.pack (show summary.exitCode) <> " worst=" <> Text.pack (show summary.worst))+ pure (summaryExitCode summary)++materialize :: FilePath -> FilePath -> InputSource -> IO FilePath+materialize _ _ (InputFile path) = pure path+materialize directory name InputStdin = do+ bytes <- readInputSource InputStdin+ let path = directory </> name+ ByteString.writeFile path bytes+ pure path++stdinCount :: ExecuteCliOptions -> Int+stdinCount options = length [() | InputStdin <- options.planSource : maybe [] pure options.environmentSource]
+ src/Kenshou/Cli/Command/List.hs view
@@ -0,0 +1,87 @@+module Kenshou.Cli.Command.List (listCommand) where++import Data.Aeson qualified as Aeson+import Data.Bifunctor (first)+import Data.ByteString.Lazy.Char8 qualified as LazyByteString+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Text.IO qualified as Text.IO+import Kenshou.Core.Bundle+import Kenshou.Core.Cli (CliCommand (..), CliEnv (..), CliGroup (..))+import Kenshou.Core.Id+import Kenshou.Core.Role (WorkerRole (..), renderRoleName)+import Kenshou.Core.Scenario+import Kenshou.Core.Selector (parseSelector)+import Options.Applicative+import System.Exit (ExitCode (..))+import System.IO (stderr)++data ListOptions = ListOptions+ { selectors :: [Text],+ layers :: [LayerId],+ kinds :: [Kind],+ maxTier :: Maybe Tier,+ placements :: [Placement],+ roles :: Bool,+ json :: Bool+ }++listCommand :: CliCommand+listCommand =+ CliCommand+ { name = "list",+ description = "List registered verification scenarios",+ group = Discovery,+ hidden = False,+ parser = runList <$> parserOptionGroup "Selection" listParser+ }++listParser :: Parser ListOptions+listParser =+ ListOptions+ <$> many (Text.pack <$> strArgument (metavar "SELECTOR" <> help "Scenario selector"))+ <*> many (option (eitherReader readLayer) (long "layer" <> metavar "LAYER" <> help "Restrict to a layer"))+ <*> many (option (eitherReader readKind) (long "kind" <> metavar "KIND" <> help "Restrict to an evidence kind"))+ <*> optional (option (eitherReader readTier) (long "max-tier" <> metavar "TIER" <> help "Maximum cost tier"))+ <*> many (option (eitherReader readPlacement) (long "placement" <> metavar "PLACEMENT" <> help "Restrict placement"))+ <*> switch (long "roles" <> help "List worker roles instead of scenarios")+ <*> switch (long "json" <> help "Write kenshou.scenario-list/v1 JSON")++runList :: ListOptions -> CliEnv -> IO ExitCode+runList options environment = case traverse parseSelector options.selectors of+ Left err -> Text.IO.hPutStrLn stderr ("kenshou: " <> err) >> pure (ExitFailure 2)+ Right selectors -> do+ let filterValue = ListFilter selectors options.layers options.kinds options.maxTier options.placements+ selected = selectScenarios environment.registry filterValue+ if options.json+ then LazyByteString.putStrLn (Aeson.encode (scenarioListDocumentFor selected (if options.roles then allRoles environment.registry else [])))+ else+ if options.roles+ then mapM_ (Text.IO.putStrLn . (\role -> renderRoleName role.name <> " " <> role.summary)) (allRoles environment.registry)+ else mapM_ (Text.IO.putStrLn . renderScenario) selected+ pure ExitSuccess++renderScenario :: Scenario -> Text+renderScenario scenario =+ Text.intercalate " " [renderScenarioId scenario.id, renderTier scenario.tier, renderPlacement scenario.placement, scenario.summary <> known]+ where+ known = maybe "" (const " [known defect]") scenario.knownDefect++readLayer :: String -> Either String LayerId+readLayer = first Text.unpack . parseLayer . Text.pack++readKind :: String -> Either String Kind+readKind = first Text.unpack . parseKind . Text.pack++readTier :: String -> Either String Tier+readTier "smoke" = Right TierSmoke+readTier "standard" = Right TierStandard+readTier "extended" = Right TierExtended+readTier "soak" = Right TierSoak+readTier value = Left ("unknown tier " <> show value)++readPlacement :: String -> Either String Placement+readPlacement "local" = Right PlaceLocal+readPlacement "cell" = Right PlaceCell+readPlacement "either" = Right PlaceEither+readPlacement value = Left ("unknown placement " <> show value)
+ src/Kenshou/Cli/Command/Overhead.hs view
@@ -0,0 +1,204 @@+module Kenshou.Cli.Command.Overhead+ ( OverheadOptions (..),+ overheadCommand,+ overheadParser,+ parseInputs,+ loadPolicy,+ findResumeDirectory,+ verdictText,+ )+where++import Control.Exception (IOException, try)+import Data.Aeson qualified as Aeson+import Data.ByteString.Lazy.Char8 qualified as LazyByteString+import Data.List (isPrefixOf, sort)+import Data.Map.Strict (Map)+import Data.Map.Strict qualified as Map+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Text.IO qualified as Text.IO+import Data.Word (Word64)+import Kenshou.Core.Bundle (lookupScenario)+import Kenshou.Core.Cli+import Kenshou.Core.Id (ScenarioId, newRunId, parseScenarioId, renderRunId)+import Kenshou.Core.Knob (KnobError (..), KnobName, RawKnob, parseAssignment)+import Kenshou.Core.RunSpec (RunSpec (..))+import Kenshou.Diagnose.Leak (analyseRunDirectory, defaultLeakSpec)+import Kenshou.Measure.Compare (Verdict (..), compareRuns)+import Kenshou.Telemetry.Overhead+import Kenshou.Telemetry.Overhead.Policy+import Options.Applicative+import System.Directory (createDirectoryIfMissing, doesFileExist, listDirectory)+import System.Environment (getExecutablePath)+import System.Exit (ExitCode (..))+import System.FilePath (takeFileName, (</>))+import System.IO (IOMode (..), stderr, withFile)+import System.Process (StdStream (..), createProcess, proc, std_err, std_out, waitForProcess)++data OverheadOptions = OverheadOptions+ { scenario :: Text,+ arms :: [Text],+ mode :: OverheadMode,+ control :: Bool,+ trials :: Int,+ knobs :: [Text],+ dimensions :: [Text],+ policy :: Maybe InputSource,+ seed :: Word64,+ settleSeconds :: Int,+ retries :: Int,+ resume :: Bool,+ analyseOnly :: Bool,+ json :: Bool,+ out :: FilePath+ }++overheadCommand :: CliCommand+overheadCommand = CliCommand "overhead" "Run interleaved telemetry overhead arms" Execution False (overheadHandler <$> overheadParser)++overheadParser :: Parser OverheadOptions+overheadParser =+ OverheadOptions+ <$> (Text.pack <$> strArgument (metavar "SCENARIO" <> help "Registered benchmark scenario"))+ <*> some (Text.pack <$> strOption (long "arms" <> metavar "FACTOR=VALUE,..." <> help "Telemetry factor and ordered arm values"))+ <*> option (eitherReader parseMode) (long "mode" <> metavar "one-factor|full" <> value OneFactor <> showDefaultWith (const "one-factor") <> help "Arm expansion mode")+ <*> switch (long "control" <> help "Add an identical baseline A/A arm")+ <*> option auto (long "trials" <> metavar "N" <> value 3 <> showDefault <> help "Valid paired blocks")+ <*> many (Text.pack <$> strOption (long "set" <> metavar "NAME=VALUE" <> help "Set a fixed scenario knob"))+ <*> many (Text.pack <$> strOption (long "dim" <> metavar "NAME=VALUE" <> help "Set a fixed dimension"))+ <*> optional (option (parseInputSource <$> str) (long "policy" <> metavar "FILE" <> help "Read kenshou.overhead-policy/v1; use - for stdin"))+ <*> option auto (long "seed" <> metavar "N" <> value 7 <> showDefault <> help "Replayable ordering seed")+ <*> option auto (long "settle-seconds" <> metavar "S" <> value 2 <> showDefault <> help "Pause between child runs")+ <*> option auto (long "retries" <> metavar "R" <> value 1 <> showDefault <> help "Retries for infrastructure failures")+ <*> switch (long "resume" <> help "Resume the latest matching overhead directory")+ <*> switch (long "analyse-only" <> help "Rebuild a report without running child slots")+ <*> switch (long "json" <> help "Print only the overhead report JSON")+ <*> strOption (long "out" <> metavar "DIR" <> help "Output root or overhead invocation directory")++overheadHandler :: OverheadOptions -> CliEnv -> IO ExitCode+overheadHandler options environment = case parseScenarioId options.scenario of+ Left err -> usage err+ Right scenarioId -> case lookupScenario environment.registry scenarioId of+ Nothing -> usage ("unknown scenario " <> options.scenario)+ Just scenario -> do+ parsed <- pure (parseInputs options)+ case parsed of+ Left err -> usage err+ Right (factors, fixedKnobs, fixedDimensions) -> do+ selectedPolicy <- loadPolicy options.policy+ case selectedPolicy of+ Left err -> usage err+ Right policy -> do+ resumed <- if options.resume || options.analyseOnly then findResumeDirectory options.out scenarioId else pure Nothing+ case (options.resume || options.analyseOnly, resumed) of+ (True, Nothing) -> usage "no matching overhead state was found under --out"+ (_, Just directory) -> continue policy directory+ (_, Nothing) -> do+ identifier <- renderRunId <$> newRunId+ let directory = options.out </> ("overhead-" <> Text.unpack identifier)+ request = OverheadRequest identifier scenarioId factors options.mode options.control options.trials fixedKnobs fixedDimensions options.seed options.settleSeconds options.retries+ case planOverhead request scenario of+ Left (UsageError err) -> usage err+ Right plan -> do+ createDirectoryIfMissing True directory+ runPlan policy directory plan Nothing+ where+ continue policy directory = do+ loaded <- loadOverheadState directory+ case loaded of+ Left err -> usage ("invalid overhead state: " <> Text.pack err)+ Right state -> runPlan policy directory state.plan (Just state)++ runPlan policy directory plan existing = do+ executable <- getExecutablePath+ let leakCheck runDirectory = fmap (either (const Nothing) (Just . Aeson.toJSON)) (analyseRunDirectory runDirectory defaultLeakSpec plan.seed)+ hooks = OverheadHooks (runChildProcess executable directory) compareRuns (Just leakCheck)+ state <- if options.analyseOnly then maybe (fail "missing overhead state") pure existing else executeOverhead hooks plan directory+ report <- analyseOverhead hooks policy plan state directory+ if options.json+ then LazyByteString.putStrLn (Aeson.encode report)+ else do+ mapM_ printComparison report.comparisons+ Text.IO.putStrLn ("verdict " <> verdictText report.verdict <> " " <> Text.pack directory)+ pure case overheadVerdictExitCode report.verdict of 0 -> ExitSuccess; code -> ExitFailure code++ printComparison comparison =+ Text.IO.putStrLn (comparison.candidate <> " vs " <> comparison.baseline <> " " <> verdictText comparison.verdict)++usage :: Text -> IO ExitCode+usage message = Text.IO.hPutStrLn stderr ("kenshou: " <> message) >> pure (ExitFailure 2)++parseInputs :: OverheadOptions -> Either Text (Map Text [Text], [(KnobName, RawKnob)], Map Text Text)+parseInputs options = do+ factorPairs <- traverse parseArms options.arms+ let factorNames = fmap fst factorPairs+ if length factorNames /= Map.size (Map.fromList [(name, ()) | name <- factorNames])+ then Left "each --arms factor may be specified only once"+ else pure ()+ fixedKnobs <- traverse parseKnob options.knobs+ fixedDimensions <- Map.fromList <$> traverse parseAssignmentText options.dimensions+ pure (Map.fromList factorPairs, fixedKnobs, fixedDimensions)+ where+ parseKnob input = case parseAssignment input of Left (KnobError err) -> Left err; Right parsed -> Right parsed++parseArms :: Text -> Either Text (Text, [Text])+parseArms input = do+ (rawName, rawValues) <- parseAssignmentText input+ name <- case rawName of+ "tracing" -> Right "telemetry.tracing"+ "metrics" -> Right "telemetry.metrics"+ "telemetry.tracing" -> Right rawName+ "telemetry.metrics" -> Right rawName+ _ -> Left ("unknown overhead factor " <> rawName)+ let values = Text.splitOn "," rawValues+ if any Text.null values then Left (name <> " contains an empty arm value") else Right (name, values)++parseAssignmentText :: Text -> Either Text (Text, Text)+parseAssignmentText input = case Text.breakOn "=" input of+ (name, assignment) | not (Text.null name) && not (Text.null assignment) -> Right (name, Text.drop 1 assignment)+ _ -> Left ("expected NAME=VALUE, got " <> input)++parseMode :: String -> Either String OverheadMode+parseMode "one-factor" = Right OneFactor+parseMode "full" = Right FullFactorial+parseMode input = Left ("unknown mode " <> input)++loadPolicy :: Maybe InputSource -> IO (Either Text OverheadPolicy)+loadPolicy source = do+ bytes <- readInputSource (maybe (InputFile "policies/telemetry-overhead.json") id source)+ pure (either (Left . Text.pack) Right (decodeOverheadPolicy bytes))++findResumeDirectory :: FilePath -> ScenarioId -> IO (Maybe FilePath)+findResumeDirectory root scenarioId = do+ direct <- doesFileExist (root </> "state.json")+ if direct+ then matching root+ else do+ namesResult <- try (listDirectory root) :: IO (Either IOException [FilePath])+ let candidates = case namesResult of Left _ -> []; Right names -> reverse (sort [root </> name | name <- names, "overhead-" `isPrefixOf` takeFileName name])+ firstMatching candidates+ where+ matching directory = do+ loaded <- loadOverheadState directory+ pure case loaded of Right state | state.plan.scenarioId == scenarioId -> Just directory; _ -> Nothing+ firstMatching [] = pure Nothing+ firstMatching (directory : rest) = matching directory >>= maybe (firstMatching rest) (pure . Just)++runChildProcess :: FilePath -> FilePath -> RunSpec -> FilePath -> IO ExitCode+runChildProcess executable invocation spec runsRoot = do+ createDirectoryIfMissing True (invocation </> ".specs")+ createDirectoryIfMissing True (invocation </> "logs")+ let identifier = maybe "unassigned" (Text.unpack . renderRunId) spec.runId+ specPath = invocation </> ".specs" </> identifier <> ".json"+ logPath = invocation </> "logs" </> identifier <> ".log"+ Aeson.encodeFile specPath spec+ withFile logPath WriteMode \handle -> do+ (_, _, _, process) <- createProcess (proc executable ["run", "--spec", specPath, "--out", runsRoot]) {std_out = UseHandle handle, std_err = UseHandle handle}+ waitForProcess process++verdictText :: Verdict -> Text+verdictText VerdictPass = "pass"+verdictText VerdictRegression = "regression"+verdictText VerdictInconclusive = "inconclusive"+verdictText VerdictInfrastructureFailure = "infrastructure-failure"
+ src/Kenshou/Cli/Command/Perf.hs view
@@ -0,0 +1,484 @@+{-# LANGUAGE FieldSelectors #-}++module Kenshou.Cli.Command.Perf (perfCommand, validateProfilePolicy) where++import Control.Exception (Exception, IOException, catch, displayException, throwIO)+import Control.Monad (foldM, unless, when)+import Data.Aeson hiding (decode)+import Data.Aeson.Key qualified as Key+import Data.Aeson.Types (parseEither)+import Data.Aeson.Types qualified as Aeson+import Data.ByteString qualified as ByteString+import Data.ByteString.Lazy.Char8 qualified as LazyByteString+import Data.List (nub, sort)+import Data.Map.Strict qualified as Map+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Text.Encoding qualified as Text.Encoding+import Data.Text.IO qualified as Text.IO+import Data.Time.Clock (diffUTCTime, getCurrentTime)+import Kenshou.Cli.Command.Perf.Session (sessionCommands, verifyCalibrationAdmission)+import Kenshou.Core.Bundle (allScenarios)+import Kenshou.Core.Canonical (canonicalEncode, sha256Hex)+import Kenshou.Core.Cli+import Kenshou.Core.Cohort qualified as Cohort+import Kenshou.Measure.Compare.Policy qualified as Policy+import Kenshou.Measure.Perf (validateProfilePolicy)+import Kenshou.Perf.Artifacts (sealPerfDirectory, verifyPerfManifest)+import Kenshou.Perf.Calibration (toolchainDigest)+import Kenshou.Perf.Contracts (PerfContracts (..))+import Kenshou.Perf.Plan+import Kenshou.Perf.Prepared qualified as Prepared+import Kenshou.Perf.Request+import Kenshou.Perf.Source+import Kenshou.Plan.Catalog (fromScenario)+import Kenshou.Plan.Change.Git qualified as Git+import Kenshou.Plan.Components qualified as Graph+import Kenshou.Plan.Suite (Suite (..), readSuite)+import Kenshou.Remote.Payload qualified as Payload+import Kenshou.Remote.Payload.Publish (digestFile)+import Kenshou.Remote.Payload.Publisher qualified as Publisher+import Kenshou.Remote.Store (Bucket (..), ObjectName (..), ObjectStore (..))+import Kenshou.Remote.Store.File (newFileStore)+import Kenshou.Remote.Store.Gcs (newGcsStore, newTokenProvider)+import Options.Applicative hiding (value)+import Options.Applicative qualified as Options+import System.Directory (createDirectory, createDirectoryIfMissing, doesFileExist, doesPathExist, getCurrentDirectory, listDirectory, makeAbsolute, renameFile)+import System.Environment (lookupEnv)+import System.Exit (ExitCode (..))+import System.FilePath (takeDirectory, takeExtension, (</>))+import System.IO (stderr)+import System.IO.Temp (withSystemTempDirectory, withTempDirectory)+import System.Process (readProcessWithExitCode)++data PerfFailure = PerfFailure Int Text deriving stock (Show)++instance Exception PerfFailure++perfCommand :: CliCommand+perfCommand =+ CliCommand+ { name = "perf",+ description = "Plan and prepare scoped runtime performance comparisons",+ group = Execution,+ hidden = False,+ parser =+ hsubparser+ ( command "plan" (info (planParser <**> helper) (progDesc "Resolve exact source commits and explain performance scope without building or leasing"))+ <> command "prepare" (info (prepareParser <**> helper) (progDesc "Check independent build inputs and publish both exact-revision payloads without leasing"))+ <> sessionCommands+ )+ }++planParser :: Parser (CliEnv -> IO ExitCode)+planParser =+ runPlan+ <$> strOption (long "request" <> metavar "FILE" <> help "kenshou.perf-request/v1 document")+ <*> strOption (long "out" <> metavar "DIRECTORY" <> help "New directory for pinned planning inputs and manifest")++prepareParser :: Parser (CliEnv -> IO ExitCode)+prepareParser =+ runPrepare+ <$> strOption (long "request" <> metavar "FILE")+ <*> strOption (long "out" <> metavar "DIRECTORY")+ <*> strOption (long "cache" <> metavar "DIRECTORY" <> Options.value ".dev/perf-payload-cache" <> showDefault)+ <*> strOption (long "control-bucket" <> metavar "BUCKET" <> Options.value "tan-nb-exp-cells-control" <> showDefault)++data ArmInputs = ArmInputs+ { armDescriptor :: Cohort.CohortDescriptor,+ armLock :: Value,+ armInventory :: Prepared.BuildInventory,+ armBounds :: Prepared.BuildInventory,+ armIdentity :: Cohort.CohortIdentity,+ armFlake :: FilePath,+ armDescriptorFile :: FilePath,+ armBuildKey :: Text+ }++runPrepare :: FilePath -> FilePath -> FilePath -> Text -> CliEnv -> IO ExitCode+runPrepare requestPath destination cachePath bucketName environment = guarded do+ started <- getCurrentTime+ unless (not (Text.null bucketName) && Text.all (\c -> c `elem` ['a' .. 'z'] || c `elem` ['0' .. '9'] || c `elem` ['-', '.']) bucketName) (throwIO (PerfFailure 2 "invalid control bucket"))+ occupied <- doesPathExist destination+ when occupied (throwIO (PerfFailure 2 "preparation output already exists"))+ _ <- ByteString.readFile requestPath >>= (decode "request" :: ByteString.ByteString -> IO PerfRequest)+ dirty <- process "git" ["status", "--porcelain"]+ unless (Text.null dirty) (throwIO (PerfFailure 4 "preparation requires a clean committed harness"))+ revision <- Text.strip <$> process "git" ["rev-parse", "HEAD"]+ root <- getCurrentDirectory+ output <- makeAbsolute destination+ cache <- makeAbsolute cachePath+ let evidence = output </> "evidence"+ work = output </> "work"+ harness = work </> "harness"+ bucket = Bucket bucketName+ createDirectoryIfMissing True (takeDirectory output)+ createDirectory output+ createDirectory evidence+ createDirectory work+ plan <- createPlan requestPath (evidence </> "planning") environment (Just (revision, False))+ planned <- getCurrentTime+ inputs <- ByteString.readFile (evidence </> "planning" </> "inputs.json") >>= decode "pinned planning inputs"+ -- Clone the committed harness, not the mutable working directory. Generated+ -- projects, locks and Cabal caches are siblings outside that checkout.+ _ <- loggedProcess (work </> "clone.log") "git" ["clone", "--quiet", "--no-hardlinks", root, harness]+ _ <- loggedProcess (work </> "checkout.log") "git" ["-C", harness, "checkout", "--quiet", "--detach", Text.unpack revision]+ baseline <- resolveArm environment.perf output harness revision "baseline"+ candidate <- if plan.request.baselineRevision == plan.request.candidateRevision then pure baseline else resolveArm environment.perf output harness revision "candidate"+ targets <- requireInput (targetPackages plan.request baseline.armDescriptor)+ requireInput (Prepared.compareInventories targets baseline.armInventory candidate.armInventory)+ requireInput (Prepared.compareInventories targets baseline.armBounds candidate.armBounds)+ resolutionFinished <- getCurrentTime+ progress "both inventories and declared bounds checked; publishing payloads"+ store <- openPerfStore bucket+ createDirectoryIfMissing True cache+ (baselinePayload, baselineHit) <- publishArm store bucket cache harness revision baseline+ (candidatePayload, candidateHit) <- if baseline.armBuildKey == candidate.armBuildKey then pure (baselinePayload, True) else publishArm store bucket cache harness revision candidate+ published <- getCurrentTime+ let arm resolved payload = Prepared.PreparedArm resolved.armDescriptor resolved.armLock resolved.armBounds resolved.armInventory (toJSON payload) resolved.armBuildKey+ prepared = Prepared.PerfPrepared plan inputs revision (arm baseline baselinePayload) (arm candidate candidatePayload)+ requireInput (Prepared.validatePrepared environment.perf prepared)+ encodeFile (evidence </> "prepared.json") prepared+ encodeFile+ (evidence </> "preparation-summary.json")+ ( object+ [ "baselineCacheHit" .= baselineHit,+ "candidateCacheHit" .= candidateHit,+ "preparedSha256" .= Prepared.preparedDigest prepared,+ "startedAt" .= started,+ "payloadsReadyAt" .= published,+ "stageSeconds"+ .= object+ [ "planning" .= (realToFrac (diffUTCTime planned started) :: Double),+ "resolutionAndBounds" .= (realToFrac (diffUTCTime resolutionFinished planned) :: Double),+ "publication" .= (realToFrac (diffUTCTime published resolutionFinished) :: Double),+ "payloadPreparation" .= (realToFrac (diffUTCTime published started) :: Double)+ ]+ ]+ )+ sealPerfDirectory evidence >>= requireInput+ verifyPerfManifest evidence >>= requireInput+ LazyByteString.putStrLn (encode (object ["prepared" .= (evidence </> "prepared.json"), "preparedSha256" .= Prepared.preparedDigest prepared, "baselineCacheHit" .= baselineHit, "candidateCacheHit" .= candidateHit]))+ pure ExitSuccess++resolveArm :: PerfContracts -> FilePath -> FilePath -> Text -> Text -> IO ArmInputs+resolveArm contracts output harness revision arm = do+ let evidence = output </> "evidence" </> "planning"+ work = output </> "work" </> Text.unpack arm+ flake = work </> "build"+ descriptorPath = flake </> "cohort.json"+ lockPath = flake </> "source-lock.json"+ createDirectory work+ createDirectory flake+ descriptor <- ByteString.readFile (evidence </> Text.unpack arm <> "-cohort.json") >>= decode "generated cohort"+ encodeFile descriptorPath (descriptor :: Cohort.CohortDescriptor)+ _ <- loggedProcess (work </> "source-lock.log") "bash" [harness </> "scripts" </> "payload-lock.sh", "--descriptor", descriptorPath, "--out", lockPath]+ lock <- ByteString.readFile lockPath >>= decode "source lock"+ names <- listDirectory harness+ packages <- filterMIO (\name -> doesFileExist (harness </> name </> name <> ".cabal")) (filter (Text.isPrefixOf "kenshou-" . Text.pack) names)+ let selectedPackages = contracts.lanePackages descriptor packages+ boundsProject <- requireInput (Prepared.renderBoundsProject contracts [harness </> name </> name <> ".cabal" | name <- selectedPackages] descriptor)+ let projectPath = work </> "bounds.project"+ planPath = work </> "bounds-dist" </> "cache" </> "plan.json"+ Text.IO.writeFile projectPath boundsProject+ _ <- loggedProcess (work </> "resolution.log") "cabal" ["build", "all", "--dry-run", "--project-file=" <> projectPath, "--builddir=" <> work </> "bounds-dist"]+ initialPlan <- ByteString.readFile planPath >>= decode "initial bounds plan"+ _ <- requireInput (Prepared.cabalInventory harness revision initialPlan)+ cacheDocument <- loggedProcess (work </> "cabal-path.log") "cabal" ["path", "--ignore-project", "--remote-repo-cache", "--output-format=json"] >>= decodeText "Cabal cache location"+ repoCache <- either (throwIO . PerfFailure 4 . Text.pack) pure (parseEither (withObject "CabalPaths" (.: "remote-repo-cache")) cacheDocument :: Either String FilePath)+ _ <-+ loggedProcess+ (work </> "resolved-lock.log")+ "python3"+ [harness </> "scripts" </> "perf-resolved-lock.py", "--plan", planPath, "--index", repoCache </> "hackage.haskell.org" </> "01-index.tar", "--cohort-lock", lockPath, "--out", flake </> "resolved-lock.json"]+ wrapper <- requireInput (generatedFlake harness revision descriptor)+ Text.IO.writeFile (flake </> "flake.nix") wrapper+ _ <- loggedProcess (work </> "flake-lock.log") "nix" ["flake", "lock", "path:" <> flake]+ let attribute = "path:" <> flake <> "#packages.x86_64-linux.kenshou-" <> Text.unpack (Cohort.unCohortName descriptor.descriptorName)+ inventory <- loggedProcess (work </> "build-inventory.log") "nix" ["eval", "--json", attribute <> ".buildInventory"] >>= decodeText "Nix inventory"+ identity <- loggedProcess (work </> "cohort-identity.log") "nix" ["eval", "--json", attribute <> ".cohortIdentity"] >>= decodeText "Nix identity"+ descriptorBytes <- ByteString.readFile descriptorPath+ _ <- requireInput (Publisher.validateCohortIdentity descriptor (Text.drop 7 (sha256Hex descriptorBytes)) identity)+ let actualConstraints = Text.intercalate ", " [entry.name <> " ==" <> entry.version | entry <- (inventory :: Prepared.BuildInventory).packages, not ("kenshou-" `Text.isPrefixOf` entry.name)]+ Text.IO.writeFile (work </> "nix-bounds.project") (boundsProject <> "\nconstraints: " <> actualConstraints <> "\n")+ _ <- loggedProcess (work </> "bounds.log") "cabal" ["build", "all", "--dry-run", "--project-file=" <> work </> "nix-bounds.project", "--builddir=" <> work </> "nix-bounds-dist"]+ cabalPlan <- ByteString.readFile (work </> "nix-bounds-dist" </> "cache" </> "plan.json") >>= decode "bounds plan"+ bounds <- requireInput (Prepared.cabalInventory harness revision cabalPlan)+ boundsIdentity <- either (throwIO . PerfFailure 4 . Text.pack . show) pure (Cohort.identityFromPlan descriptor (Text.drop 7 (sha256Hex descriptorBytes)) cabalPlan)+ unless (null (Cohort.checkCohort descriptor boundsIdentity)) (throwIO (PerfFailure 4 "independent bounds plan differs from the generated cohort"))+ requireInput (Prepared.validateArmInventories descriptor inventory bounds)+ encodeFile (work </> "build-inventory.json") inventory+ encodeFile (work </> "bounds-inventory.json") bounds+ let key = Prepared.buildKeyFor revision descriptor lock inventory bounds+ pure (ArmInputs descriptor lock inventory bounds identity flake descriptorPath key)++generatedFlake :: FilePath -> Text -> Cohort.CohortDescriptor -> Either Text Text+generatedFlake harness revision descriptor = do+ unless (not (any (`elem` ['\n', '\r', ' ', '"', '\\']) harness) && not ("${" `Text.isInfixOf` Text.pack harness) && immutableRevision revision) (Left "unsafe pinned harness path")+ let quote = Text.Encoding.decodeUtf8 . canonicalEncode . String+ location = "git+file://" <> Text.pack harness <> "?rev=" <> revision+ pure+ ( Text.unlines+ [ "{",+ " inputs.harness.url = " <> quote location <> ";",+ " outputs = { self, harness }: let",+ " inputs = harness.inputs // { self = harness; };",+ " pkgs = import inputs.nixpkgs { system = \"x86_64-linux\"; };",+ " in { packages.x86_64-linux." <> quote ("kenshou-" <> Cohort.unCohortName descriptor.descriptorName) <> " = import (harness + \"/nix/kenshou/packages.nix\") {",+ " inherit inputs pkgs; cohort = " <> quote (Cohort.unCohortName descriptor.descriptorName) <> "; variant = \"default\";",+ " descriptorFile = ./cohort.json; lockFile = ./source-lock.json; resolvedLockFile = ./resolved-lock.json; strictBounds = true;",+ " }; };",+ "}"+ ]+ )++publishArm :: ObjectStore -> Bucket -> FilePath -> FilePath -> Text -> ArmInputs -> IO (Payload.PayloadDescriptor, Bool)+publishArm store bucket cache harness revision arm = do+ let cached = cache </> Text.unpack (Text.drop 7 arm.armBuildKey) <> ".json"+ exists <- doesFileExist cached+ if exists+ then do+ payload <- ByteString.readFile cached >>= decode "cached payload"+ unless ((payload :: Payload.PayloadDescriptor).cohortIdentity == arm.armIdentity && payload.harness == Payload.Harness revision False && payload.variant == "default") (throwIO (PerfFailure 4 "cached payload identity mismatch"))+ withSystemTempDirectory "kenshou-perf-cache-check-" \temporary -> do+ let bundleObject = ObjectName ("payloads/sha256/" <> payload.cell.bundle.sha256 <> ".nar.zst")+ file = temporary </> "bundle.nar.zst"+ unless (payload.cell.bundle.uri == "gs://" <> bucket.unBucket <> "/payloads/sha256/" <> payload.cell.bundle.sha256 <> ".nar.zst") (throwIO (PerfFailure 4 "cached payload belongs to another bucket"))+ found <- store.downloadTo bucket bundleObject file+ when (found == Nothing) (throwIO (PerfFailure 4 "cached payload bundle is missing"))+ (digest, bytes) <- digestFile file+ unless ((digest, bytes) == (payload.cell.bundle.sha256, payload.cell.bundle.bytes)) (throwIO (PerfFailure 4 "cached bundle digest/size mismatch"))+ progress "reused verified published payload"+ pure (payload, True)+ else do+ result <- Publisher.publishPreparedPayload store (Publisher.PreparedPublishOptions harness revision ("path:" <> arm.armFlake) arm.armDescriptorFile (Cohort.unCohortName arm.armDescriptor.descriptorName) "default" bucket (arm.armFlake </> "payload.json"))+ payload <- case result of+ Left problem -> do+ encodeFile (takeDirectory arm.armFlake </> "publication-failure.json") (object ["error" .= show problem])+ throwIO (PerfFailure 4 (Text.pack (show problem)))+ Right value -> pure value+ withTempDirectory cache ".pending-" \temporary -> do+ let staged = temporary </> "payload.json"+ encodeFile staged payload+ renameFile staged cached+ pure (payload, False)++loggedProcess :: FilePath -> String -> [String] -> IO Text+loggedProcess logPath executable arguments = do+ (status, output, errors) <- readProcessWithExitCode executable arguments ""+ Text.IO.writeFile logPath (Text.pack output <> Text.pack errors)+ case status of+ ExitSuccess -> pure (Text.pack output)+ _ -> throwIO (PerfFailure 4 (Text.pack executable <> " failed; retained log: " <> Text.pack logPath <> "\n" <> Text.take 2000 (Text.pack errors)))++decodeText :: (FromJSON a) => Text -> Text -> IO a+decodeText label = decode label . Text.Encoding.encodeUtf8++filterMIO :: (a -> IO Bool) -> [a] -> IO [a]+filterMIO predicate values = fmap (map fst . filter snd) (traverse (\item -> (item,) <$> predicate item) values)++openPerfStore :: Bucket -> IO ObjectStore+openPerfStore bucket =+ lookupEnv "KENSHOU_CELL_STORE" >>= \case+ Nothing -> newTokenProvider bucket >>= newGcsStore+ Just setting -> case Text.stripPrefix "file:" (Text.pack setting) of+ Just root | not (Text.null root) -> newFileStore (Text.unpack root)+ _ -> throwIO (PerfFailure 2 "KENSHOU_CELL_STORE must be file:<directory> or unset")++runPlan :: FilePath -> FilePath -> CliEnv -> IO ExitCode+runPlan requestPath output environment = guarded do+ plan <- createPlan requestPath output environment Nothing+ LazyByteString.putStrLn (encode plan)+ pure ExitSuccess++planningToolchain :: Cohort.CohortDescriptor -> IO Prepared.BuildInventory+planningToolchain descriptor = do+ bytes <- ByteString.readFile "flake.lock"+ lock <- decode "flake lock" bytes+ (nixpkgs, haskellNix) <- requireInput (either (Left . Text.pack) Right (parseEither pins lock))+ pure (Prepared.BuildInventory "nix" descriptor.descriptorCompiler "x86_64-linux" (object ["nixpkgsRevision" .= nixpkgs, "haskellNixRevision" .= haskellNix, "flakeLockSha256" .= sha256Hex bytes, "variant" .= ("default" :: Text), "strictBounds" .= True]) [])+ where+ pins = withObject "FlakeLock" \v -> do+ nodes <- v .: "nodes"+ root <- v .: "root"+ let node name = withObject "LockNodes" (.: Key.fromText name) nodes+ follow :: Int -> Text -> Text -> Aeson.Parser Text+ follow depth name key = do+ when (depth > 32) (fail "cyclic or excessive flake follows chain")+ value <- node name >>= withObject "LockNode" (.: "inputs") >>= withObject "LockInputs" (.: Key.fromText key)+ case value of+ String destination -> pure destination+ Array _ -> parseJSON @[Text] value >>= foldM (follow (depth + 1)) root+ _ -> fail "unsupported flake input reference"+ revision input = do+ name <- follow 0 root input+ value <- node name >>= withObject "LockNode" (.: "locked") >>= withObject "LockedInput" (.: "rev")+ unless (immutableRevision value) (fail "toolchain input needs an immutable commit")+ pure value+ (,) <$> revision "nixpkgs" <*> revision "haskell-nix"++harnessSemantics :: IO Text+harnessSemantics = do+ listed <- process "git" ["ls-files", "-z"]+ let paths = sort [Text.unpack path | path <- Text.splitOn "\0" listed, takeExtension (Text.unpack path) `elem` [".hs", ".cabal", ".nix", ".py"] || path `elem` ["flake.lock", "cabal.project"]]+ records <- traverse (\path -> do bytes <- ByteString.readFile path; pure (object ["path" .= path, "sha256" .= sha256Hex bytes])) paths+ pure (sha256Hex (canonicalEncode (toJSON records)))++createPlan :: FilePath -> FilePath -> CliEnv -> Maybe (Text, Bool) -> IO PerfPlan+createPlan requestPath output environment pinnedHarness = do+ occupied <- doesPathExist output+ when occupied (throwIO (PerfFailure 2 "output already exists; choose a new planning directory"))+ requestBytes <- ByteString.readFile requestPath+ request <- decode "request" requestBytes+ descriptorBytes <- ByteString.readFile request.cohortTemplate+ descriptor <- decode "cohort template" descriptorBytes+ expectedPackages <- requireInput (targetPackages request descriptor)+ graph <- either (throwIO . PerfFailure 2 . (.errorText)) pure (Graph.decodeGraph environment.componentGraph)+ let nodes = [component | component <- graph.components, any (`elem` expectedPackages) component.packages]+ unless (all (\package -> any (elem package . (.packages)) nodes) expectedPackages) (throwIO (PerfFailure 2 "target packages are not represented in the component graph"))+ profilesBytes <- ByteString.readFile "policies/performance/catalog.v1.json"+ profiles <- decode "performance catalog" profilesBytes+ unless (all (`elem` map (.name) profiles.profiles) request.profiles) (throwIO (PerfFailure 2 "unknown requested performance profile"))+ suiteBytes <- ByteString.readFile ("suites/" <> Text.unpack request.suite <> ".json")+ suite <- readSuite ("suites/" <> Text.unpack request.suite <> ".json") >>= requireInput+ unless (suite.name == request.suite) (throwIO (PerfFailure 2 "suite name differs from request"))+ policies <- traverse loadPolicy (nub (map (.policy) profiles.profiles))+ let policyMap = Map.fromList [(Text.pack path, sha256Hex (canonicalEncode value)) | (path, value, _) <- policies]+ mapM_ (checkPolicy policies) profiles.profiles+ let catalog = map fromScenario (allScenarios environment.registry)+ mapM_ (requireInput . validateProfile environment.perf catalog) profiles.profiles+ sourceProject <- resolveProject request.project+ unless (all ((== sourceProject.repository) . (.repository)) nodes) (throwIO (PerfFailure 2 "target graph nodes belong to another repository"))+ withSystemTempDirectory "kenshou-perf-source-" \checkout -> do+ progress "verifying requested commits in the registered HTTPS upstream"+ _ <- process "git" ["init", "--quiet", checkout]+ mapM_ (fetch checkout sourceProject.location) (nub [request.baselineRevision, request.candidateRevision])+ baseline <- inspect checkout request descriptor sourceProject.location request.baselineRevision+ candidate <- if request.baselineRevision == request.candidateRevision then pure baseline else inspect checkout request descriptor sourceProject.location request.candidateRevision+ baselineDescriptor <- requireInput (descriptorForSnapshot request descriptor "baseline" baseline)+ candidateDescriptor <- requireInput (descriptorForSnapshot request descriptor "candidate" candidate)+ changedText <- process "git" ["-C", checkout, "diff", "--no-renames", "--name-only", "-z", Text.unpack request.baselineRevision, Text.unpack request.candidateRevision]+ let paths = map Text.unpack (filter (not . Text.null) (Text.splitOn "\0" changedText))+ (mapped, _) <- Git.changesFromUpstream graph (Git.UpstreamDiff sourceProject.repository checkout request.baselineRevision request.candidateRevision) >>= requireInput+ let (changes, unknown, documentation) = conservativeChanges graph sourceProject.repository paths mapped+ provisional <- requireInput (planPerformance environment.perf request {purpose = "calibration"} graph catalog profiles changes unknown documentation policyMap Map.empty)+ toolchain <- planningToolchain descriptor+ semantics <- harnessSemantics+ admissionChecks <-+ traverse+ ( \entry -> do+ policy <- case [document | (path, document, _) <- policies, path == entry.profile.policy] of [value] -> pure value; _ -> throwIO (PerfFailure 2 "missing calibration policy")+ verified <- if request.purpose == "calibration" then pure (Left "calibration requests do not consume gating admissions") else case entry.profile.calibration of Nothing -> pure (Left "no calibration evidence"); Just _ -> verifyCalibrationAdmission environment.perf entry (toolchainDigest toolchain) semantics policy+ pure (entry, verified)+ )+ provisional.selected+ let admitted = Map.fromList [(entry.workloadSha256, digest) | (entry, Right digest) <- admissionChecks]+ calibrationProblems = Map.fromList [(entry.profile.name, problem) | (entry, Left problem) <- admissionChecks]+ plan <- requireInput (planPerformance environment.perf request graph catalog profiles changes unknown documentation policyMap admitted)+ (harnessRevision, harnessDirty) <- case pinnedHarness of+ Just snapshot -> pure snapshot+ Nothing -> do+ revision <- Text.strip <$> process "git" ["rev-parse", "HEAD"]+ status <- process "git" ["status", "--porcelain"]+ pure (revision, not (Text.null status))+ let inputs =+ object+ [ "request" .= request,+ "template" .= descriptor,+ "templateSha256" .= sha256Hex descriptorBytes,+ "baselineSource" .= baseline,+ "candidateSource" .= candidate,+ "harness" .= object ["revision" .= harnessRevision, "dirty" .= harnessDirty],+ "harnessSemanticsSha256" .= semantics,+ "buildToolchain" .= toolchain.toolchain,+ "calibrationAdmissions" .= admitted,+ "calibrationProblems" .= calibrationProblems,+ "graph" .= graph,+ "scenarioCatalog" .= catalog,+ "profileCatalog" .= profiles,+ "suite" .= either (const Null) id (eitherDecodeStrict' suiteBytes :: Either String Value),+ "policies" .= Map.fromList [(path, document) | (path, document, _) <- policies],+ "changedPaths" .= paths+ ]+ createDirectoryIfMissing True (takeDirectory output)+ -- Exclusive creation also rejects a concurrent writer racing the preflight.+ createDirectory output+ ByteString.writeFile (output </> "template.json") descriptorBytes+ encodeFile (output </> "request.json") request+ encodeFile (output </> "inputs.json") inputs+ encodeFile (output </> "plan.json") plan+ encodeFile (output </> "baseline-cohort.json") baselineDescriptor+ encodeFile (output </> "candidate-cohort.json") candidateDescriptor+ sealPerfDirectory output >>= requireInput+ progress ("planned " <> Text.pack (show (length plan.selected)) <> " configurations; scope " <> plan.coverage)+ pure plan+ where+ loadPolicy path = do+ bytes <- ByteString.readFile path+ parsed <- requireInput (Policy.decodePolicy bytes)+ document <- decode "comparison policy" bytes+ pure (path, document, parsed)+ checkPolicy policies profile = case [policy | (path, _, policy) <- policies, path == profile.policy] of+ [policy] -> requireInput (validateProfilePolicy profile policy)+ _ -> throwIO (PerfFailure 2 "profile policy is missing")++data SourceProject = SourceProject {repository :: Text, location :: Text}++resolveProject :: Text -> IO SourceProject+resolveProject uri = do+ let qualified = Text.unpack (Text.drop 7 uri)+ raw <- process "mori" ["registry", "show", qualified, "--full", "--json"]+ document <- requireInput (either (Left . Text.pack) Right (eitherDecodeStrict' (Text.Encoding.encodeUtf8 raw)))+ repositories <- requireInput (either (Left . Text.pack) Right (parseEither (withObject "Mori project" (.: "repositories")) document :: Either String [Value]))+ case repositories of+ [repository] -> requireInput (either (Left . Text.pack) Right (parseEither parseRepository repository))+ _ -> throwIO (PerfFailure 2 "source project must identify exactly one upstream repository")+ where+ parseRepository = withObject "Mori repository" \value -> do+ repository <- value .: "name"+ github <- value .:? "github"+ git <- value .:? "git"+ location <- case (github :: Maybe Text, git :: Maybe Text) of+ (Just hosted, _) | length (Text.splitOn "/" hosted) == 2 && Text.all (\c -> c `elem` ("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-_/" :: String)) hosted -> pure ("https://github.com/" <> hosted <> ".git")+ (Nothing, Just url) | "https://" `Text.isPrefixOf` url && not (Text.any (`elem` ['@', '#', '?', '\n', '\r']) url) -> pure url+ _ -> fail "registered upstream must use HTTPS Git"+ pure (SourceProject repository location)++fetch :: FilePath -> Text -> Text -> IO ()+fetch checkout location revision = do+ _ <- process "git" ["-c", "credential.interactive=false", "-C", checkout, "fetch", "--quiet", "--no-tags", "--depth=1", Text.unpack location, Text.unpack revision]+ actual <- Text.strip <$> process "git" ["-C", checkout, "rev-parse", Text.unpack revision <> "^{commit}"]+ unless (actual == revision) (throwIO (PerfFailure 2 "upstream revision does not identify the requested commit"))++inspect :: FilePath -> PerfRequest -> Cohort.CohortDescriptor -> Text -> Text -> IO SourceSnapshot+inspect checkout request descriptor location revision = do+ pathsText <- process "git" ["-C", checkout, "ls-tree", "-r", "--name-only", "-z", Text.unpack revision]+ let paths = filter (".cabal" `Text.isSuffixOf`) (Text.splitOn "\0" pathsText)+ manifests <- traverse readManifest paths+ requireInput (sourceSnapshot request descriptor location revision [manifest | Right manifest <- manifests])+ where+ readManifest path = do+ contents <- process "git" ["-C", checkout, "show", Text.unpack revision <> ":" <> Text.unpack path]+ pure (parsePackageManifest (Text.unpack path) contents)++process :: String -> [String] -> IO Text+process executable arguments = do+ (status, output, errors) <- readProcessWithExitCode executable arguments ""+ case status of+ ExitSuccess -> pure (Text.pack output)+ _ -> throwIO (PerfFailure 4 (Text.pack executable <> ": " <> Text.strip (Text.pack errors)))++decode :: (FromJSON a) => Text -> ByteString.ByteString -> IO a+decode label bytes = requireInput (either (Left . ((label <> ": ") <>) . Text.pack) Right (eitherDecodeStrict' bytes))++requireInput :: Either Text a -> IO a+requireInput = either (throwIO . PerfFailure 2) pure++progress :: Text -> IO ()+progress = Text.IO.hPutStrLn stderr . ("kenshou perf: " <>)++guarded :: IO ExitCode -> IO ExitCode+guarded operation = operation `catch` failure `catch` ioFailure+ where+ failure (PerfFailure code message) = progress message >> pure (ExitFailure code)+ ioFailure (problem :: IOException) = progress (Text.pack (displayException problem)) >> pure (ExitFailure 4)
+ src/Kenshou/Cli/Command/Perf/Session.hs view
@@ -0,0 +1,1014 @@+module Kenshou.Cli.Command.Perf.Session (sessionCommands, replayCalibrationReceipt, verifyCalibrationAdmission) where++import Control.Concurrent (threadDelay)+import Control.Exception (Exception, IOException, catch, displayException, throwIO, try)+import Control.Monad (unless, when)+import Data.Aeson hiding (decode)+import Data.Aeson.Types (parseEither)+import Data.ByteString qualified as Bytes+import Data.ByteString.Lazy.Char8 qualified as Lazy+import Data.List (find, isSuffixOf)+import Data.List qualified as List+import Data.List.NonEmpty (NonEmpty (..))+import Data.Map.Strict qualified as Map+import Data.Maybe (fromMaybe)+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Text.IO qualified as TextIO+import Data.Time (UTCTime, diffUTCTime, getCurrentTime)+import Data.Time.Clock.POSIX (posixSecondsToUTCTime)+import Kenshou.Cli.Command.Cell (startCellInstances)+import Kenshou.Core.Bundle (registryRoutingRules)+import Kenshou.Core.Canonical (canonicalEncode, sha256Hex)+import Kenshou.Core.Cli (CliEnv (..))+import Kenshou.Core.Id+import Kenshou.Core.Phase (zeroPhases)+import Kenshou.Measure.Compare.Compatibility (VaryingAxis (..))+import Kenshou.Measure.Compare.Ordering (PairedOrdering (ABBA))+import Kenshou.Measure.Compare.Policy (Direction (..), MetricRule (..), Policy, matchesRule)+import Kenshou.Measure.Compare.Policy qualified as Policy+import Kenshou.Perf.Artifacts+import Kenshou.Perf.Calibration qualified as Calibration+import Kenshou.Perf.Contracts (PerfContracts)+import Kenshou.Perf.Plan+import Kenshou.Perf.Plan qualified as PerfPlan+import Kenshou.Perf.Prepared qualified as Prepared+import Kenshou.Perf.Request (PerfRequest (..), strictObject)+import Kenshou.Perf.Result+import Kenshou.Plan.RunPlan (PlannedRun (..), RunPlan (..))+import Kenshou.Remote.Cell.Control+import Kenshou.Remote.Cell.Docs (CachePolicy (Cold), CellBuckets (..), CellDescriptor (..), Submission (..), WorkObject (..))+import Kenshou.Remote.Cell.Lease qualified as Lease+import Kenshou.Remote.Cell.Prepare+import Kenshou.Remote.Cell.RouteRules+import Kenshou.Remote.Cell.Session.Build (BuildOptions (..), buildSession)+import Kenshou.Remote.Cell.Session.Journal (SessionJournal (..), SliceJournal (..), SliceState (..), readSessionJournal)+import Kenshou.Remote.Cell.Session.Resume (resumeObservedSlices)+import Kenshou.Remote.Cell.WorkJson (decodeWorkPlan)+import Kenshou.Remote.Pair+import Kenshou.Remote.Payload (Bundle (..), CellPayload (..), PayloadDescriptor (..))+import Kenshou.Remote.Payload.Publish (digestFile)+import Kenshou.Remote.Perf.Attempt+import Kenshou.Remote.Perf.Cell+import Kenshou.Remote.Perf.Cleanup+import Kenshou.Remote.Perf.Conditions (validateHealthPolicy)+import Kenshou.Remote.Perf.Journal+import Kenshou.Remote.Perf.Lock+import Kenshou.Remote.Perf.Publication+import Kenshou.Remote.Perf.Runner+import Kenshou.Remote.Store+import Kenshou.Remote.Store.File (newFileStore)+import Kenshou.Remote.Store.Gcs (newGcsStore, newTokenProvider)+import Options.Applicative hiding (Success, action)+import System.Directory+import System.Environment (lookupEnv)+import System.Exit (ExitCode (..))+import System.FilePath (splitDirectories, takeDirectory, (</>))+import System.IO (stderr)+import System.IO.Temp (withSystemTempDirectory)+import Text.Printf (printf)++data SessionFailure = SessionFailure Int Text deriving stock (Show)++instance Exception SessionFailure++data PublicationReceipt = PublicationReceipt+ { snapshot :: Text,+ manifestUri :: Text,+ manifestSha256 :: Text,+ resultSha256 :: Text,+ payloadsReadyAt :: UTCTime,+ publicationStartedAt :: UTCTime,+ verifiedReportAt :: UTCTime,+ totalElapsedSeconds :: Double+ }+ deriving stock (Eq, Show)++instance ToJSON PublicationReceipt where+ toJSON receipt = object ["schema" .= ("kenshou.perf-publication/v1" :: Text), "snapshot" .= receipt.snapshot, "manifestUri" .= receipt.manifestUri, "manifestSha256" .= receipt.manifestSha256, "resultSha256" .= receipt.resultSha256, "payloadsReadyAt" .= receipt.payloadsReadyAt, "publicationStartedAt" .= receipt.publicationStartedAt, "verifiedReportAt" .= receipt.verifiedReportAt, "totalElapsedSeconds" .= receipt.totalElapsedSeconds]++instance FromJSON PublicationReceipt where+ parseJSON = withObject "PerformanceReceipt" \v -> do+ strictObject ["schema", "snapshot", "manifestUri", "manifestSha256", "resultSha256", "payloadsReadyAt", "publicationStartedAt", "verifiedReportAt", "totalElapsedSeconds"] v+ schema <- v .: "schema"+ unless (schema == ("kenshou.perf-publication/v1" :: Text)) (fail "unsupported publication receipt")+ receipt <- PublicationReceipt <$> v .: "snapshot" <*> v .: "manifestUri" <*> v .: "manifestSha256" <*> v .: "resultSha256" <*> v .: "payloadsReadyAt" <*> v .: "publicationStartedAt" <*> v .: "verifiedReportAt" <*> v .: "totalElapsedSeconds"+ unless (receipt.payloadsReadyAt <= receipt.publicationStartedAt && receipt.publicationStartedAt <= receipt.verifiedReportAt && abs (receipt.totalElapsedSeconds - realToFrac (diffUTCTime receipt.verifiedReportAt receipt.payloadsReadyAt)) < 0.000001) (fail "publication receipt timing mismatch")+ pure receipt++data ExecutionInputs = ExecutionInputs+ { controlBucket :: Text,+ storeLabel :: Text,+ allowedProjects :: [Text],+ start :: Bool,+ healthPolicies :: Map.Map Text Value,+ cells :: [(CellDescriptor, Maybe CellCapabilities)]+ }++instance ToJSON ExecutionInputs where+ toJSON inputs = object ["controlBucket" .= inputs.controlBucket, "storeLabel" .= inputs.storeLabel, "allowedProjects" .= inputs.allowedProjects, "start" .= inputs.start, "healthPolicies" .= inputs.healthPolicies, "cells" .= [object ["descriptor" .= descriptor, "capabilities" .= capabilities] | (descriptor, capabilities) <- inputs.cells]]++instance FromJSON ExecutionInputs where+ parseJSON = withObject "PerformanceExecutionInputs" \v -> do+ strictObject ["controlBucket", "storeLabel", "allowedProjects", "start", "healthPolicies", "cells"] v+ entries <-+ v .: "cells"+ >>= traverse+ ( withObject "PinnedCell" \c -> do+ strictObject ["descriptor", "capabilities"] c+ (,) <$> c .: "descriptor" <*> c .:? "capabilities"+ )+ ExecutionInputs <$> v .: "controlBucket" <*> v .: "storeLabel" <*> v .: "allowedProjects" <*> v .: "start" <*> (v .:? "healthPolicies" .!= Map.empty) <*> pure entries++sessionCommands :: Mod CommandFields (CliEnv -> IO ExitCode)+sessionCommands =+ command "run" (info (runParser <**> helper) (progDesc "Execute a pinned performance preparation with persistent budgets"))+ <> command "status" (info (statusParser <**> helper) (progDesc "Read persistent performance session state without cell actions"))+ <> command "resume" (info (resumeParser <**> helper) (progDesc "Verify inputs and reconcile a retained performance session"))+ <> command "verify" (info (verifyParser <**> helper) (progDesc "Replay retained comparisons and verify published evidence without executing trials"))+ <> command "calibration" (info (calibrationParser <**> helper) (progDesc "Freeze calibration batches and verify their retained acceptance evidence"))+ where+ session = strOption (long "session" <> metavar "DIRECTORY")+ runParser = runNew <$> strOption (long "prepared" <> metavar "FILE") <*> strOption (long "evidence-base-uri" <> metavar "GS_URI") <*> strOption (long "out" <> metavar "DIRECTORY") <*> strOption (long "control-bucket" <> value "tan-nb-exp-cells-control" <> showDefault) <*> switch (long "start" <> help "Start stopped instances after acquiring their cell lease")+ statusParser = runStatus <$> session+ resumeParser = runResume <$> session+ verifyParser = runVerify <$> session <*> switch (long "offline" <> help "Verify the retained sealed snapshot and receipt without contacting storage")+ calibrationParser =+ hsubparser+ ( command "declare" (info (runCalibrationPublish True <$> strOption (long "definition" <> metavar "FILE") <*> strOption (long "uri" <> metavar "GS_URI")) (progDesc "Create and retrieve a frozen definition before any acceptance trials"))+ <> command "fingerprint" (info (runCalibrationFingerprint <$> strOption (long "descriptor" <> metavar "FILE") <*> strOption (long "cell-policy" <> metavar "FILE")) (progDesc "Read the exact machine/image profile and health-policy digest before a batch"))+ <> command "publish" (info (runCalibrationPublish False <$> strOption (long "acceptance" <> metavar "FILE") <*> strOption (long "uri" <> metavar "GS_URI")) (progDesc "Create and retrieve the complete acceptance receipt inventory"))+ <> command "verify" (info (runCalibrationVerify <$> strOption (long "reference" <> metavar "FILE") <*> strOption (long "prepared" <> metavar "FILE")) (progDesc "Replay every calibration receipt against one prepared profile"))+ )++runCalibrationFingerprint :: FilePath -> FilePath -> CliEnv -> IO ExitCode+runCalibrationFingerprint descriptorPath policyPath _ = guarded do+ descriptor <- readJSON descriptorPath+ policy <- readJSON policyPath+ input (validateHealthPolicy policy)+ Lazy.putStrLn (encode (object ["machineProfile" .= cellMachineProfile descriptor, "healthPolicySha256" .= sha256Hex (canonicalEncode policy)]))+ pure ExitSuccess++runNew :: FilePath -> Text -> FilePath -> Text -> Bool -> CliEnv -> IO ExitCode+runNew preparedPath uri destination bucket start environment = guarded do+ prepared <- eitherDecodeFileStrict' preparedPath >>= either (throwIO . SessionFailure 2 . Text.pack) pure+ input (Prepared.validatePrepared environment.perf prepared)+ _ <- input (parseDestination uri)+ input (initialSessionPlaceholder prepared uri)+ verifyPerfManifest (takeDirectory preparedPath) >>= require+ exists <- doesPathExist destination+ when exists (throwIO (SessionFailure 2 "performance output already exists"))+ verifyGatingCalibration environment.perf prepared+ output <- makeAbsolute destination+ (store, label) <- openStore (Bucket bucket)+ projects <- maybe ["tan-nb-exp"] (map Text.strip . Text.splitOn "," . Text.pack) <$> lookupEnv "KENSHOU_GCP_ALLOWED_PROJECTS"+ capabilityDirectory <- fromMaybe ".dev/cells" <$> lookupEnv "KENSHOU_CELL_CAPABILITIES_DIR"+ pinned <- traverse (loadCell store projects capabilityDirectory bucket) prepared.plan.request.cells+ healthPolicies <- if null prepared.plan.selected then pure Map.empty else Map.fromList <$> traverse (\(descriptor, _) -> (descriptor.name,) <$> loadHealthPolicy store bucket descriptor.name) pinned+ let inputs = ExecutionInputs bucket label projects start healthPolicies pinned+ checkCalibratedPolicies prepared inputs+ payloads <- loadPayloads store prepared+ payloadsReady <- getCurrentTime+ identifier <- newRunId+ preflight environment store inputs payloads prepared identifier+ -- The create-only probe is retained as evidence of the actual destination+ -- permissions. It needs object creation and retrieval, never deletion.+ accessProbe store uri identifier prepared+ session <- require (initialSession identifier (Prepared.preparedDigest prepared) prepared.plan uri payloadsReady)+ createDirectoryIfMissing True (takeDirectory output)+ createDirectory output+ copyTree (takeDirectory preparedPath) (output </> "preparation")+ encodeFile (output </> "prepared.json") prepared+ encodeFile (output </> "request.json") prepared.plan.request+ encodeFile (output </> "plan.json") prepared.plan+ writePerfSession (output </> "session.json") session {executionInputs = Just (toJSON inputs)}+ withPerformanceLock output (execute environment store inputs payloads prepared output)++-- Validate the destination using the same immutable journal admission rules+-- before obtaining an ID, writing output or performing any remote operation.+initialSessionPlaceholder :: Prepared.PerfPrepared -> Text -> Either Text ()+initialSessionPlaceholder prepared uri = do+ identifier <- parseRunId "00000000-0000-7000-8000-000000000000"+ let epoch = posixSecondsToUTCTime 0+ _ <- initialSession identifier (Prepared.preparedDigest prepared) prepared.plan uri epoch+ pure ()++runResume :: FilePath -> CliEnv -> IO ExitCode+runResume destination environment = guarded do+ output <- makeAbsolute destination+ withPerformanceLock output do+ prepared <- loadPreparation environment.perf (output </> "preparation/prepared.json")+ session <- readPerfSession (output </> "session.json") >>= require+ require (validateSessionInputs environment.perf prepared session)+ checkRootInputs output prepared+ verifyGatingCalibration environment.perf prepared+ inputs <- require (parseInputs session)+ checkCalibratedPolicies prepared inputs+ state <- require (journalState session)+ receiptExists <- doesFileExist (output </> "receipt.json")+ if receiptExists && state.lease == Nothing && state.pendingClaim == Nothing && state.activeComparison == Nothing && Map.size state.completed == length session.plan.selected+ then do+ (result, _) <- verifySessionEvidence environment.perf output False+ receipt <- readJSON (output </> "receipt.json")+ retained <- readPerfSession (output </> Text.unpack (receipt :: PublicationReceipt).snapshot </> "session.json") >>= require+ unless (retained == session) (ioError (userError "published report does not describe the current session"))+ Lazy.putStrLn (encode result)+ pure (exit result.exitCode)+ else do+ (store, label) <- openStore (Bucket inputs.controlBucket)+ unless (inputs.storeLabel == label) (ioError (userError "object store differs from the immutable performance session"))+ mapM_ (checkCell store inputs) inputs.cells+ payloads <- loadPayloads store prepared+ preflight environment store inputs payloads prepared session.sessionId+ execute environment store inputs payloads prepared output++runStatus :: FilePath -> CliEnv -> IO ExitCode+runStatus output _ = guarded do+ session <- readPerfSession (output </> "session.json") >>= require+ state <- require (journalState session)+ now <- getCurrentTime+ Lazy.putStrLn (encode (object ["session" .= session.sessionId, "requestSha256" .= session.plan.requestSha256, "admissionSpentSeconds" .= state.admissionSpentSeconds, "executionDeadline" .= state.executionDeadline, "cleanupDeadline" .= state.cleanupDeadline, "lease" .= state.lease, "pendingClaim" .= state.pendingClaim, "activeComparison" .= state.activeComparison, "attempts" .= state.attempts, "replacements" .= state.replacements, "completed" .= Map.elems state.completed, "runtimeBlockers" .= state.runtimeBlockers, "totalElapsedSeconds" .= (realToFrac (diffUTCTime now session.createdAt) :: Double)]))+ pure ExitSuccess++loadPreparation :: PerfContracts -> FilePath -> IO Prepared.PerfPrepared+loadPreparation contracts path = do+ verifyPerfManifest (takeDirectory path) >>= require+ prepared <- readJSON path+ require (Prepared.validatePrepared contracts prepared)+ pure prepared++loadCell :: ObjectStore -> [Text] -> FilePath -> Text -> Text -> IO (CellDescriptor, Maybe CellCapabilities)+loadCell store projects directory bucket name = do+ snapshot <- readCellSnapshot store (Lease.CellRef name (Bucket bucket)) projects >>= require+ capabilities <- loadCellCapabilities directory snapshot.descriptor >>= require+ pure (snapshot.descriptor, capabilities)++loadHealthPolicy :: ObjectStore -> Text -> Text -> IO Value+loadHealthPolicy store bucket cell = do+ retained <- store.getObject (Bucket bucket) (ObjectName ("cells/" <> cell <> "/policy.json"))+ bytes <- maybe (ioError (userError "cell health policy is missing")) (pure . fst) retained+ policy <- either (ioError . userError) pure (eitherDecode bytes)+ require (validateHealthPolicy policy)+ pure policy++checkCell :: ObjectStore -> ExecutionInputs -> (CellDescriptor, Maybe CellCapabilities) -> IO ()+checkCell store inputs (descriptor, _) = do+ snapshot <- readCellSnapshot store (Lease.CellRef descriptor.name (Bucket inputs.controlBucket)) inputs.allowedProjects >>= require+ case Map.lookup descriptor.name inputs.healthPolicies of+ Nothing -> pure ()+ Just frozen -> do+ policy <- loadHealthPolicy store inputs.controlBucket descriptor.name+ unless (policy == frozen) (ioError (userError "cell health policy changed since performance admission"))+ unless (snapshot.descriptor == descriptor) (ioError (userError "cell descriptor changed since performance preparation"))++parseInputs :: PerfSession -> Either Text ExecutionInputs+parseInputs session = do+ inputs <- maybe (Left "performance session has no frozen execution inputs") decode session.executionInputs+ unless (map ((.name) . fst) inputs.cells == session.plan.request.cells && not (Text.null inputs.controlBucket) && not (null inputs.allowedProjects)) (Left "frozen cell pool differs from the request")+ unless (null session.plan.selected || Map.keys inputs.healthPolicies == List.sort (map ((.name) . fst) inputs.cells)) (Left "frozen cell health policies are missing or differ from the pool")+ mapM_ validateHealthPolicy (Map.elems inputs.healthPolicies)+ mapM_+ ( \(descriptor, capabilities) -> do+ unless (descriptor.buckets.control == inputs.controlBucket && descriptor.project `elem` inputs.allowedProjects) (Left "frozen cell bucket or project is invalid")+ case capabilities of+ Nothing -> pure ()+ Just record -> unless (record.cell == descriptor.name && record.descriptorSha256 == descriptorDigest descriptor) (Left "frozen capabilities differ from the cell descriptor")+ )+ inputs.cells+ pure inputs++checkRootInputs :: FilePath -> Prepared.PerfPrepared -> IO ()+checkRootInputs directory prepared = mapM_ check [("prepared.json", toJSON prepared), ("plan.json", toJSON prepared.plan), ("request.json", toJSON prepared.plan.request)]+ where+ check (name, expected) = do+ actual <- readJSON (directory </> name)+ unless (actual == expected) (ioError (userError ("retained " <> name <> " differs from immutable preparation")))++loadPayloads :: ObjectStore -> Prepared.PerfPrepared -> IO (Map.Map Text PayloadDescriptor)+loadPayloads store prepared = do+ entries <-+ traverse+ ( \(label, arm) -> do+ payload <- require (decode arm.payload)+ let prefix = "gs://"+ (bucket, suffix) = Text.breakOn "/" (Text.drop (Text.length prefix) payload.cell.bundle.uri)+ unless (prefix `Text.isPrefixOf` payload.cell.bundle.uri && not (Text.null bucket) && suffix == "/payloads/sha256/" <> payload.cell.bundle.sha256 <> ".nar.zst") (ioError (userError "invalid pinned payload location"))+ withSystemTempDirectory "kenshou-perf-payload-check" \temporary -> do+ let path = temporary </> "bundle.nar.zst"+ present <- store.downloadTo (Bucket bucket) (ObjectName (Text.drop 1 suffix)) path+ unless (present /= Nothing) (ioError (userError "pinned payload is missing"))+ (digest, bytes) <- digestFile path+ unless (digest == payload.cell.bundle.sha256 && bytes == payload.cell.bundle.bytes) (ioError (userError "pinned payload retrieval digest differs"))+ pure (label, payload)+ )+ [("baseline", prepared.baseline), ("candidate", prepared.candidate)]+ pure (Map.fromList entries)++options :: BuildOptions+options = BuildOptions (PrepareOptions False False [] Cold) GranularityRun False NullSink Nothing (24 * 1024 * 1024 * 1024) (20 * 1024 * 1024 * 1024) "0.1.0"++adapter :: CliEnv -> ObjectStore -> ExecutionInputs -> Map.Map Text PayloadDescriptor -> (CellDescriptor, Maybe CellCapabilities) -> CellAdapter+adapter environment store inputs payloads (descriptor, capabilities) = CellAdapter environment.registry store (Lease.CellRef descriptor.name (Bucket inputs.controlBucket)) descriptor (Map.findWithDefault Null descriptor.name inputs.healthPolicies) capabilities (registryRoutingRules environment.registry) payloads options inputs.storeLabel (\event -> TextIO.hPutStrLn stderr (Text.pack (show event)))++preflight :: CliEnv -> ObjectStore -> ExecutionInputs -> Map.Map Text PayloadDescriptor -> Prepared.PerfPrepared -> RunId -> IO ()+preflight environment store inputs payloads prepared identifier = mapM_ check prepared.plan.selected+ where+ check entry = do+ schedule <- makeSchedule prepared entry identifier+ compatible <-+ traverse+ ( \pinned -> do+ let a = adapter environment store inputs payloads pinned+ built <- buildSession a.registry a.descriptor a.capabilities a.rules a.payloads a.options a.storeLabel identifier (encode schedule)+ pure (either (const False) (const (prepared.plan.request.purpose == "calibration" || entry.profile.machineProfile == cellMachineProfile a.descriptor)) built)+ )+ inputs.cells+ unless (or compatible) (ioError (userError ("no pinned cell supports profile " <> Text.unpack entry.profile.name)))++makeSchedule :: Prepared.PerfPrepared -> PlannedProfile -> RunId -> IO RunPlan+makeSchedule prepared entry identifier = do+ let profile = entry.profile+ source = object ["schema" .= ("kenshou.run-plan/v1" :: Text), "planId" .= identifier, "inputs" .= object ["preparedSha256" .= Prepared.preparedDigest prepared, "workloadSha256" .= entry.workloadSha256], "runs" .= [object ["ordinal" .= (1 :: Int), "runId" .= identifier, "estimateMinutes" .= (1 :: Int), "spec" .= object ["schema" .= ("kenshou.run-spec/v1" :: Text), "runId" .= identifier, "scenario" .= profile.scenario, "scenarioRevision" .= profile.scenarioRevision, "knobs" .= profile.knobs, "dimensions" .= profile.dimensions, "seed" .= prepared.plan.request.seed, "phases" .= zeroPhases, "timeoutSeconds" .= profile.timeoutSeconds, "environment" .= object ["placement" .= ("cell" :: Text), "machineProfile" .= profile.machineProfile]]]]]+ projected <- require (decodeWorkPlan source)+ require (pairPlan (PairRequest "candidate" "baseline" profile.minimumPairs ABBA (fromIntegral prepared.plan.request.seed) profile.replacementPairs) projected)++policyFor :: Prepared.PerfPrepared -> PlannedProfile -> Either Text Policy+policyFor prepared entry = do+ policies <- either (Left . Text.pack) Right (parseEither (withObject "PinnedInputs" (.: "policies")) prepared.inputs)+ document <- maybe (Left "pinned comparison policy missing") Right (Map.lookup entry.profile.policy (policies :: Map.Map FilePath Value))+ decode document++axesFor :: Prepared.PerfPrepared -> NonEmpty VaryingAxis+axesFor prepared = (if prepared.plan.request.control == "aa" then VaryControl else VaryCohort) :| []++attemptDirectory :: FilePath -> PlannedProfile -> Int -> FilePath+attemptDirectory output entry index = output </> "configurations" </> Text.unpack entry.profile.name </> "attempt-" <> show index++execute :: CliEnv -> ObjectStore -> ExecutionInputs -> Map.Map Text PayloadDescriptor -> Prepared.PerfPrepared -> FilePath -> IO ExitCode+execute environment store inputs payloads prepared output = do+ let ref identity = Lease.CellRef identity.cell (Bucket inputs.controlBucket)+ pinned identity = maybe (ioError (userError "leased cell is outside frozen pool")) pure (find ((== identity.cell) . (.name) . fst) inputs.cells)+ handle identity = Lease.reattachActiveLease store (ref identity) identity.leaseId >>= maybe (ioError (userError "performance lease is no longer active")) pure+ supports entry p = do+ planned <- makeSchedule prepared entry =<< newRunId+ let a = adapter environment store inputs payloads p+ built <- buildSession a.registry a.descriptor a.capabilities a.rules a.payloads a.options a.storeLabel planned.planId (encode planned)+ pure (either (const False) (const (prepared.plan.request.purpose == "calibration" || entry.profile.machineProfile == cellMachineProfile a.descriptor)) built)+ admission entry seconds pending callbacks = case pending of+ Just (name, identity, _) -> do+ unless (name == entry.profile.name) (ioError (userError "proposed claim belongs to another profile"))+ active <- Lease.reattachActiveLease store (ref identity) identity.leaseId+ pure (maybe (AdmissionFailed "proposed cell claim is unresolved, expired or cancelled; no new claim issued") (const (Admitted identity)) active)+ Nothing -> waitFor entry seconds callbacks+ waitFor entry left callbacks = do+ granted <- findLease entry inputs.cells callbacks+ case granted of+ Occupied | left > 1 -> threadDelay 1000000 >> waitFor entry (left - 1) callbacks+ _ -> pure granted+ findLease _ [] _ = pure Occupied+ findLease entry (p@(descriptor, _) : rest) callbacks = do+ compatible <- supports entry p+ if not compatible+ then findLease entry rest callbacks+ else do+ checkCell store inputs p+ identifier <- newRunId+ let identity = LeaseIdentity descriptor.name identifier+ callbacks.proposeClaim identity+ granted <- Lease.acquireLeaseWithId store (Lease.CellRef descriptor.name (Bucket inputs.controlBucket)) (Lease.LeaseRequest "kenshou-perf" "pinned runtime performance validation" 180) identifier+ case granted of+ Lease.Acquired _ -> pure (Admitted identity)+ Lease.Busy _ -> callbacks.withdrawClaim >> findLease entry rest callbacks+ Lease.Quarantined _ -> pure (AdmissionFailed "cell was quarantined during the proposed claim")+ compareOne identity entry index _ callbacks = do+ p <- pinned identity+ acquired <- handle identity+ Lease.withHeartbeat store (ref identity) acquired \_ -> do+ when inputs.start (startCellInstances (fst p))+ let directory = attemptDirectory output entry index+ schedulePath = directory </> "schedule.json"+ exists <- doesFileExist schedulePath+ schedule <-+ if exists+ then readJSON schedulePath >>= require . decodeWorkPlan+ else do+ identifier <- newRunId+ planned <- makeSchedule prepared entry identifier+ createDirectoryIfMissing True directory+ encodeFile (schedulePath <> ".next") planned+ renameFile (schedulePath <> ".next") schedulePath+ pure planned+ policy <- require (policyFor prepared entry)+ computed <- executeAttempt identity entry schedule policy (axesFor prepared) directory callbacks (executeCellTrial (adapter environment store inputs payloads p) acquired schedule directory entry.profile callbacks.recordRuntimeFailure)+ retainAttemptResult directory computed+ verify result = verifyConfiguration inputs prepared output result+ hooks =+ RunnerHooks+ environment.perf+ getCurrentTime+ admission+ (\identity -> maybe False (const True) <$> Lease.reattachActiveLease store (ref identity) identity.leaseId)+ (\identity entry -> pinned identity >>= supports entry)+ compareOne+ verify+ ( \identity -> do+ _ <- Lease.requestCancel store (ref identity) (Just identity.leaseId) False+ pure ()+ )+ ( \identity -> do+ files <- treeFiles (output </> "configurations")+ journals <- traverse (\path -> readSessionJournal (output </> "configurations" </> path) >>= require) [path | path <- files, [_, _, "trials", _, "session.json"] <- [splitDirectories path]]+ -- Fence first, then inspect markers. A marker arriving after+ -- this fence cannot start work under the old lease identity.+ _ <- Lease.requestCancel store (ref identity) (Just identity.leaseId) False+ settled <- ownedWorkSettled store (ref identity) identity journals+ unless settled do+ _ <- Lease.requestCancel store (ref identity) (Just identity.leaseId) False+ waitForOwnedWork store (ref identity) identity journals+ owned <- Lease.reattachLease store (ref identity) identity.leaseId+ case owned of+ Just acquired -> Lease.releaseLease store (ref identity) acquired+ Nothing -> do+ history <- readPerfSession (output </> "session.json") >>= require+ -- An absent ambiguous proposal must stay unresolved. A known+ -- lost lease with terminal owned work needs no deletion.+ pure (any ((== LeaseAcquired identity) . (.action)) history.events)+ )+ (recoverSealedRuntime store inputs prepared output)+ (describeReport output)+ (publishSnapshot store inputs prepared output)+ createDirectoryIfMissing True (output </> "configurations")+ auditAllAttempts True inputs prepared output+ result <- runPerformance hooks prepared (output </> "session.json")+ encodeFile (output </> "result.json") result+ TextIO.writeFile (output </> "report.md") (renderReport result)+ Lazy.putStrLn (encode result)+ pure (exit result.exitCode)++verifyConfiguration :: ExecutionInputs -> Prepared.PerfPrepared -> FilePath -> ConfigurationResult -> IO Bool+verifyConfiguration inputs prepared output result = do+ entry <- maybe (ioError (userError "result names an unselected profile")) pure (find ((== result.profile) . (.name) . (.profile)) prepared.plan.selected)+ auditProfileAttempts False inputs prepared output entry (Just result)++recoverSealedRuntime :: ObjectStore -> ExecutionInputs -> Prepared.PerfPrepared -> FilePath -> LeaseIdentity -> IO [(Text, Text)]+recoverSealedRuntime store inputs prepared output identity = fmap (List.nub . concat) (traverse profile prepared.plan.selected)+ where+ profile entry = do+ let base = output </> "configurations" </> Text.unpack entry.profile.name+ exists <- doesDirectoryExist base+ attempts <- if exists then listDirectory base else pure []+ fmap concat (traverse (attempt entry base) attempts)+ attempt entry base name = do+ let directory = base </> name+ schedulePath = directory </> "schedule.json"+ exists <- doesFileExist schedulePath+ if not exists+ then pure []+ else do+ schedule <- readJSON schedulePath >>= require . decodeWorkPlan+ fmap concat (traverse (trial entry directory) schedule.runs)+ trial entry directory run = do+ let trialDirectory = directory </> "trials" </> Text.unpack (renderRunId run.runId)+ path = trialDirectory </> "session.json"+ exists <- doesFileExist path+ if not exists+ then pure []+ else do+ retained <- readSessionJournal path >>= require+ if retained.leaseId /= identity.leaseId || retained.cell /= identity.cell+ then pure []+ else do+ observed <- resumeObservedSlices store (Lease.CellRef identity.cell (Bucket inputs.controlBucket)) (Bucket retained.resultsBucket) path (\event -> TextIO.hPutStrLn stderr (Text.pack (show event)))+ journal <- either (ioError . userError . show) pure observed+ reasons <- retainedRuntimeProblems entry.profile run trialDirectory journal+ pure [(entry.profile.name, reason) | reason <- reasons]++auditAllAttempts :: Bool -> ExecutionInputs -> Prepared.PerfPrepared -> FilePath -> IO ()+auditAllAttempts recover inputs prepared output = do+ names <- listDirectory (output </> "configurations")+ unless (all (`elem` map (Text.unpack . (.name) . (.profile)) prepared.plan.selected) names) (ioError (userError "retained evidence contains an unselected profile"))+ mapM_ (\entry -> auditProfileAttempts recover inputs prepared output entry Nothing >>= \valid -> unless valid (ioError (userError "retained attempt audit failed"))) prepared.plan.selected++-- Every retained attempt is checked, including abandoned and partially written+-- attempts. Only a complete matching attempt supplies the final comparison.+auditProfileAttempts :: Bool -> ExecutionInputs -> Prepared.PerfPrepared -> FilePath -> PlannedProfile -> Maybe ConfigurationResult -> IO Bool+auditProfileAttempts recover inputs prepared output entry wanted = do+ session <- readPerfSession (output </> "session.json") >>= require+ let (_, identities) = foldl' remember (Nothing, Map.empty) session.events+ base = output </> "configurations" </> Text.unpack entry.profile.name+ remember (lease, known) event = case event.action of+ LeaseAcquired identity -> (Just identity, known)+ LeaseLost _ -> (Nothing, known)+ LeaseReleased -> (Nothing, known)+ ComparisonStarted profile index -> (lease, maybe known (\identity -> Map.insert (profile, "attempt-" <> show index) identity known) lease)+ _ -> (lease, known)+ exists <- doesDirectoryExist base+ names <- if exists then listDirectory base else pure []+ baseline <- require (decode prepared.baseline.payload)+ candidate <- require (decode prepared.candidate.payload)+ let payloads = Map.fromList [("baseline", baseline), ("candidate", candidate)]+ candidates <-+ traverse+ ( \name -> do+ identity <- maybe (ioError (userError "retained attempt has no journaled lease identity")) pure (Map.lookup (entry.profile.name, name) identities)+ let directory = base </> name+ files <- treeFiles directory+ scheduleExists <- doesFileExist (directory </> "schedule.json")+ if not scheduleExists+ then do+ unless (all (== "schedule.json.next") files) (ioError (userError "retained attempt has evidence before its paired schedule"))+ pure False+ else do+ schedule <- readJSON (directory </> "schedule.json") >>= require . decodeWorkPlan+ expected <- makeSchedule prepared entry schedule.planId+ unless (toJSON schedule == toJSON expected) (ioError (userError "retained paired schedule differs from its pinned request and seed"))+ metadataExists <- doesFileExist (directory </> "attempt.json")+ if not metadataExists+ then do+ unless (all (`elem` ["schedule.json", "attempt.json.next"]) files) (ioError (userError "retained attempt has evidence but no immutable attempt header"))+ pure False+ else do+ verifyAttemptInputs identity entry schedule directory+ pinned <- maybe (ioError (userError "attempt cell is outside frozen pool")) pure (find ((== identity.cell) . (.name) . fst) inputs.cells)+ pathsExists <- doesFileExist (directory </> "trial-paths.json")+ paths <- if pathsExists then readJSON (directory </> "trial-paths.json") else pure Map.empty+ failureExists <- doesFileExist (directory </> "trial-failures.json")+ failures <- if failureExists then readJSON (directory </> "trial-failures.json") else pure (Map.empty :: Map.Map Text Text)+ let journalPaths = [path | path <- files, ["trials", _, "session.json"] <- [splitDirectories path]]+ expectedPaths = ["trials" </> Text.unpack (renderRunId run.runId) </> "session.json" | run <- schedule.runs]+ unless (all (`elem` expectedPaths) journalPaths) (ioError (userError "retained cell journal names an unplanned trial"))+ journals <- traverse (\path -> readSessionJournal (directory </> path) >>= require) journalPaths+ let linkedDirectories = map (Text.unpack . renderRunId . (.runId)) schedule.runs <> [Text.unpack (renderRunId slice.cellRun) | journal <- journals, slice <- journal.slices]+ attemptFiles = ["schedule.json", "attempt.json", "trial-paths.json", "trial-failures.json", "comparison.json", "result.json"]+ interpreted path = case splitDirectories path of+ [name] -> name `elem` attemptFiles || name `elem` map (<> ".next") attemptFiles+ "trials" : identifier : _ -> identifier `elem` linkedDirectories+ _ -> False+ unless (all interpreted files) (ioError (userError "retained attempt contains unlinked evidence outside its scheduled trials and cell journals"))+ mapM_+ ( \run -> do+ let identifier = renderRunId run.runId+ trial = directory </> "trials" </> Text.unpack identifier+ journalPath = trial </> "session.json"+ planBytes = encode schedule {runs = [run], estimateMinutes = run.estimateMinutes}+ journalExists <- doesFileExist journalPath+ if not journalExists+ then unless (Map.notMember identifier (paths :: Map.Map Text FilePath)) (ioError (userError "retained accepted trial has no cell journal"))+ else do+ journal <- readSessionJournal journalPath >>= require+ unless (journal.leaseId == identity.leaseId && journal.cell == identity.cell && journal.controlBucket == inputs.controlBucket && journal.resultsBucket == (fst pinned).buckets.results && journal.payloads == payloads && journal.planSha256 == Text.drop 7 (sha256Hex (Lazy.toStrict planBytes))) (ioError (userError "retained cell journal differs from pinned payloads, trial or lease"))+ storedPlan <- Lazy.readFile (trial </> "plan.json")+ unless (storedPlan == planBytes) (ioError (userError "retained trial plan bytes differ"))+ mapM_+ ( \slice -> do+ work <- Lazy.readFile (trial </> slice.workPath)+ unless (slice.submission.leaseId == identity.leaseId && slice.submission.runId == slice.cellRun && slice.runIds == [run.runId] && slice.submission.work.sha256 == Text.drop 7 (sha256Hex (Lazy.toStrict work)) && slice.submission.work.bytes == Lazy.length work) (ioError (userError "retained submission work or identity differs"))+ )+ journal.slices+ when (not recover && any (\slice -> slice.state `notElem` [SlicePlanned, SliceRejected, SliceVerified]) journal.slices) (ioError (userError "retained report contains unresolved submitted cell work"))+ verified <- verifyCellTrial (Map.findWithDefault Null identity.cell inputs.healthPolicies) (fst pinned) run trial journal+ runtimeProblems <- retainedRuntimeProblems entry.profile run trial journal+ mapM_+ ( \reason -> do+ current <- readPerfSession (output </> "session.json") >>= require+ state <- require (journalState current)+ unless (reason `elem` Map.findWithDefault [] entry.profile.name state.runtimeBlockers) do+ unless recover (ioError (userError "sealed runtime blocker is missing from the session"))+ now <- getCurrentTime+ changed <- require (appendEvent (SessionEvent now (RuntimeBlocked entry.profile.name reason)) current)+ writePerfSession (output </> "session.json") changed+ )+ runtimeProblems+ case Map.lookup identifier paths of+ Just path -> unless (verified == Right (directory </> path)) (ioError (userError "retained accepted trial outer evidence differs"))+ Nothing -> when (Map.member identifier failures && either (const False) (const True) verified) (ioError (userError "retained failed trial has healthy completed outer evidence"))+ )+ schedule.runs+ present <- doesFileExist (directory </> "result.json")+ if not present+ then pure False+ else do+ record <- readJSON (directory </> "result.json")+ policy <- require (policyFor prepared entry)+ replayed <- verifyAttempt entry schedule policy (axesFor prepared) directory record+ unless replayed (ioError (userError "retained attempt result replay differs"))+ comparisonExists <- doesFileExist (directory </> "comparison.json")+ comparison <- if comparisonExists then Just <$> readJSON (directory </> "comparison.json") else pure Nothing+ unless (comparison == record.comparison) (ioError (userError "retained ordinary comparison differs from attempt result"))+ pure (maybe False (\result -> record.comparison == result.comparison && record.validPairs == result.validPairs && record.attempts == result.attempts && all (`elem` result.runtimeFailures) record.runtimeFailures) wanted)+ )+ names+ pure (maybe True (\result -> or candidates || (result.comparison == Nothing && result.validPairs == 0 && result.infrastructureState == "pending" && result.performanceVerdict == "inconclusive")) wanted)++describeReport :: FilePath -> PerfSession -> PerfResult -> IO PerfResult+describeReport output session result = do+ identifier <- newRunId+ now <- getCurrentTime+ let summaryPath = output </> "preparation/preparation-summary.json"+ summaryExists <- doesFileExist summaryPath+ preparationTimes <- if summaryExists then readJSON summaryPath >>= either (ioError . userError) pure . parseEither (withObject "PreparationSummary" (.: "stageSeconds")) else pure Map.empty+ let base = session.evidenceBaseUri <> "/sessions/" <> renderRunId session.sessionId <> "/snapshots/" <> renderRunId identifier+ leaseTimes = [event.at | event <- session.events, LeaseAcquired _ <- [event.action]]+ elapsed from to = realToFrac (diffUTCTime to from) :: Double+ timings =+ Map.fromList+ ( [("totalAtReportAssembly", elapsed session.createdAt now)] <> case leaseTimes of+ [] -> [("initialAdmission", elapsed session.createdAt now), ("measurementAndCleanupToAssembly", 0)]+ first : _ -> [("initialAdmission", elapsed session.createdAt first), ("measurementAndCleanupToAssembly", elapsed first now)]+ )+ files <- treeFiles (output </> "configurations")+ let relativeFiles = ["prepared.json", "request.json", "plan.json", "session.json"] <> ["configurations/" <> path | path <- files, "comparison.json" `isSuffixOf` path || "manifest.json" `isSuffixOf` path]+ references <-+ traverse+ ( \path -> do+ (digest, _) <- digestFile (output </> path)+ pure (ArtifactReference (if "manifest.json" `isSuffixOf` path then "cell-or-runtime-manifest" else if "comparison.json" `isSuffixOf` path then "comparison" else Text.pack path) (base <> "/" <> Text.pack path) (Just ("sha256:" <> digest)))+ )+ relativeFiles+ pure result {stageSeconds = timings <> Map.mapKeys ("preparation." <>) preparationTimes, artifactReferences = ArtifactReference "aggregate-manifest" (base <> "/manifest.json") Nothing : ArtifactReference "publication-receipt" (base <> "/receipt.json") Nothing : references}++treeFiles :: FilePath -> IO [FilePath]+treeFiles root = do+ names <- listDirectory root+ fmap+ concat+ ( traverse+ ( \name -> do+ let path = root </> name+ linked <- pathIsSymbolicLink path+ when linked (ioError (userError "performance evidence contains a symbolic link"))+ directory <- doesDirectoryExist path+ if directory then map (name </>) <$> treeFiles path else pure [name]+ )+ names+ )++publishSnapshot :: ObjectStore -> ExecutionInputs -> Prepared.PerfPrepared -> FilePath -> PerfSession -> PerfResult -> IO Bool+publishSnapshot store inputs prepared output session result = do+ auditAllAttempts False inputs prepared output+ publicationStarted <- getCurrentTime+ manifestReference <- case filter ((== "aggregate-manifest") . (.kind)) result.artifactReferences of+ [reference] -> pure reference.uri+ _ -> ioError (userError "report has no unique aggregate manifest reference")+ relative <- maybe (ioError (userError "aggregate reference is outside its pinned session")) pure (Text.stripPrefix (session.evidenceBaseUri <> "/sessions/" <> renderRunId session.sessionId <> "/") manifestReference >>= Text.stripSuffix "/manifest.json")+ let directory = output </> Text.unpack relative+ createDirectoryIfMissing True directory+ mapM_ (\name -> copyTree (output </> name) (directory </> name)) ["preparation", "configurations"]+ mapM_ (\name -> copyFile (output </> name) (directory </> name)) ["prepared.json", "request.json", "plan.json", "session.json"]+ encodeFile (directory </> "result.json") result+ TextIO.writeFile (directory </> "report.md") (renderReport result)+ sealPerfDirectory directory >>= require+ (bucket, prefix) <- require (parseDestination session.evidenceBaseUri)+ published <- publishEvidence store bucket (prefix <> "/sessions/" <> renderRunId session.sessionId <> "/" <> relative) directory+ case published of+ Left problem -> TextIO.hPutStrLn stderr problem >> pure False+ Right uri -> do+ manifest <- Bytes.readFile (directory </> "manifest.json")+ verified <- getCurrentTime+ state <- require (journalState session)+ unless (maybe True (verified <=) state.executionDeadline) (ioError (userError "publication verification exceeded the original execution deadline"))+ let receipt = PublicationReceipt relative uri (sha256Hex manifest) (sha256Hex (canonicalEncode (toJSON result))) session.createdAt publicationStarted verified (realToFrac (diffUTCTime verified session.createdAt))+ bytes = encode receipt+ name = ObjectName (prefix <> "/sessions/" <> renderRunId session.sessionId <> "/" <> relative <> "/receipt.json")+ _ <- store.putObject bucket name "application/json" DoesNotExist bytes+ retained <- store.getObject bucket name+ unless (fmap fst retained == Just bytes) (ioError (userError "durable publication receipt retrieval differs"))+ receiptRetrievedAt <- getCurrentTime+ unless (maybe True (receiptRetrievedAt <=) state.executionDeadline) (ioError (userError "receipt retrieval exceeded the original execution deadline"))+ atomicBytes (output </> "receipt.json") bytes+ pure True++-- An interruption after comparison publication but before the root checkpoint+-- must preserve that comparison's original identity and bytes on reanalysis.+retainAttemptResult :: FilePath -> ConfigurationResult -> IO ConfigurationResult+retainAttemptResult directory computed = do+ let comparisonPath = directory </> "comparison.json"+ resultPath = directory </> "result.json"+ comparisonExists <- doesFileExist comparisonPath+ comparison <- case (comparisonExists, computed.comparison) of+ (False, Nothing) -> pure Nothing+ (True, Nothing) -> ioError (userError "retained comparison no longer matches completed trial evidence")+ (False, Just document) -> atomicBytes comparisonPath (encode document) >> pure (Just document)+ (True, Just document) -> do+ original <- readJSON comparisonPath+ retained <- require (semanticComparison original)+ replayed <- require (semanticComparison document)+ unless (retained == replayed) (ioError (userError "retained comparison differs from interrupted attempt replay"))+ pure (Just original)+ let result = computed {comparison = comparison}+ exists <- doesFileExist resultPath+ if exists+ then do+ original <- readJSON resultPath+ unless (original == result) (ioError (userError "retained attempt result differs from interrupted replay"))+ pure original+ else atomicBytes resultPath (encode result) >> pure result++atomicBytes :: FilePath -> Lazy.ByteString -> IO ()+atomicBytes path bytes = Lazy.writeFile (path <> ".next") bytes >> renameFile (path <> ".next") path++runCalibrationPublish :: Bool -> FilePath -> Text -> CliEnv -> IO ExitCode+runCalibrationPublish definition path uri _ = guarded do+ bytes <- Lazy.readFile path+ (bucket, name) <- input (parseDestination uri)+ now <- getCurrentTime+ if definition+ then do+ document <- either (throwIO . SessionFailure 2 . Text.pack) pure (eitherDecode bytes :: Either String Calibration.CalibrationDefinition)+ input (Calibration.validateCalibrationDefinition now now document)+ else do+ document <- either (throwIO . SessionFailure 2 . Text.pack) pure (eitherDecode bytes :: Either String Calibration.CalibrationAcceptance)+ unless (length document.aa == 20 && length document.slowdown == 20 && all (not . null . (.receipts)) (document.aa <> document.slowdown)) (throwIO (SessionFailure 2 "acceptance inventory needs both twenty-slot experiments"))+ (store, _) <- openStore bucket+ _ <- store.putObject bucket (ObjectName name) "application/json" DoesNotExist bytes+ retained <- store.getObject bucket (ObjectName name)+ (retrieved, metadata) <- maybe (ioError (userError "calibration publication retrieval failed")) pure retained+ unless (retrieved == bytes) (ioError (userError "create-only calibration publication differs"))+ Lazy.putStrLn (encode (object ["uri" .= uri, "sha256" .= sha256Hex (Lazy.toStrict bytes), "declaredAt" .= metadata.updated]))+ pure ExitSuccess++runCalibrationVerify :: FilePath -> FilePath -> CliEnv -> IO ExitCode+runCalibrationVerify referencePath preparedPath environment = guarded do+ prepared <- loadPreparation environment.perf preparedPath+ reference <- readJSON referencePath :: IO Calibration.CalibrationReference+ entry <- case prepared.plan.selected of [selected] -> pure selected; _ -> throwIO (SessionFailure 2 "calibration verification needs one prepared profile")+ (semantics, policies) <- require (either (Left . Text.pack) Right (parseEither (withObject "PinnedInputs" \v -> (,) <$> v .: "harnessSemanticsSha256" <*> v .: "policies") prepared.inputs))+ policy <- maybe (ioError (userError "missing pinned calibration policy")) pure (Map.lookup entry.profile.policy (policies :: Map.Map FilePath Value))+ digest <- verifyCalibrationAdmission environment.perf entry {PerfPlan.profile = entry.profile {calibration = Just (toJSON reference)}} (Calibration.toolchainDigest prepared.baseline.buildInventory) semantics policy >>= require+ Lazy.putStrLn (encode (object ["verified" .= True, "profile" .= entry.profile.name, "acceptanceSha256" .= digest, "gatingEligible" .= False]))+ pure ExitSuccess++verifyCalibrationAdmission :: PerfContracts -> PlannedProfile -> Text -> Text -> Value -> IO (Either Text Text)+verifyCalibrationAdmission contracts entry expectedToolchain semantics policyValue = do+ outcome <- try @IOException do+ reference <- require (either (Left . Text.pack) Right (parseEither (parseJSON @Calibration.CalibrationReference) (fromMaybe Null entry.profile.calibration)))+ (definitionBytes, declared) <- readPinned reference.definitionUri reference.definitionSha256+ (acceptanceBytes, _) <- readPinned reference.acceptanceUri reference.acceptanceSha256+ definition <- either (ioError . userError) pure (eitherDecode definitionBytes :: Either String Calibration.CalibrationDefinition)+ acceptance <- either (ioError . userError) pure (eitherDecode acceptanceBytes :: Either String Calibration.CalibrationAcceptance)+ unless ((definition.profile, definition.machineProfile, definition.workloadSha256, definition.toolchainSha256, definition.harnessSemanticsSha256) == (entry.profile.name, entry.profile.machineProfile, entry.workloadSha256, expectedToolchain, semantics)) (ioError (userError "calibration definition differs from the current workload, toolchain or harness"))+ unless (not ("pending" `Text.isInfixOf` entry.profile.machineProfile)) (ioError (userError "calibration requires an actual designated machine profile"))+ policy <- require (either (Left . Text.pack) Right (parseEither parseJSON policyValue))+ rule <- case filter (\rule -> matchesRule rule.match definition.targetMetric) (policy :: Policy).metricRules of+ [metricRule] -> pure metricRule+ _ -> ioError (userError "injected metric needs one unambiguous policy rule")+ unless (definition.targetMetric `elem` map (.name) entry.profile.requiredMetrics && rule.direction == LowerIsBetter && definition.targetRelativeEffect > rule.relativeLimit && definition.targetAbsoluteEffect > rule.absoluteFloor) (ioError (userError "injected latency effect must exceed both frozen policy thresholds"))+ aa <- traverse (traverse (replayCalibrationReceipt contracts) . (.receipts)) acceptance.aa+ slowdown <- traverse (traverse (replayCalibrationReceipt contracts) . (.receipts)) acceptance.slowdown+ now <- getCurrentTime+ _ <- require (Calibration.validateCalibration now declared definition acceptance aa slowdown)+ pure reference.acceptanceSha256+ pure (either (Left . Text.pack . displayException) Right outcome)+ where+ readPinned uri digest = do+ (bucket, name) <- require (parseDestination uri)+ (store, _) <- openStore bucket+ retained <- store.getObject bucket (ObjectName name)+ (bytes, metadata) <- maybe (ioError (userError "calibration record is missing")) pure retained+ unless (sha256Hex (Lazy.toStrict bytes) == digest) (ioError (userError "calibration record digest differs"))+ pure (bytes, metadata.updated)++-- A change in the owner's health thresholds invalidates the batch even when+-- hardware and image names stayed the same. Check only compatible pool cells.+checkCalibratedPolicies :: Prepared.PerfPrepared -> ExecutionInputs -> IO ()+checkCalibratedPolicies prepared inputs = when prepared.plan.gatingEligible do+ mapM_+ ( \entry -> do+ definition <- calibrationDefinition entry+ let compatible = [descriptor | (descriptor, _) <- inputs.cells, cellMachineProfile descriptor == entry.profile.machineProfile]+ unless (not (null compatible) && all (\descriptor -> ((sha256Hex . canonicalEncode) <$> Map.lookup descriptor.name inputs.healthPolicies) == Just definition.healthPolicySha256) compatible) (ioError (userError "cell health policy differs from the calibrated policy"))+ )+ prepared.plan.selected++calibrationDefinition :: PlannedProfile -> IO Calibration.CalibrationDefinition+calibrationDefinition entry = do+ reference <- require (either (Left . Text.pack) Right (parseEither (parseJSON @Calibration.CalibrationReference) (fromMaybe Null entry.profile.calibration)))+ (bucket, name) <- require (parseDestination reference.definitionUri)+ (store, _) <- openStore bucket+ retained <- store.getObject bucket (ObjectName name)+ bytes <- maybe (ioError (userError "calibration declaration is missing")) (pure . fst) retained+ unless (sha256Hex (Lazy.toStrict bytes) == reference.definitionSha256) (ioError (userError "calibration declaration digest differs"))+ either (ioError . userError) pure (eitherDecode bytes)++verifyGatingCalibration :: PerfContracts -> Prepared.PerfPrepared -> IO ()+verifyGatingCalibration contracts prepared = when prepared.plan.gatingEligible do+ (semantics, policies, admissions) <- require (either (Left . Text.pack) Right (parseEither (withObject "PinnedInputs" \v -> (,,) <$> v .: "harnessSemanticsSha256" <*> v .: "policies" <*> v .: "calibrationAdmissions") prepared.inputs))+ mapM_+ ( \entry -> do+ policy <- maybe (ioError (userError "pinned calibration policy is absent")) pure (Map.lookup entry.profile.policy (policies :: Map.Map FilePath Value))+ evidence <- verifyCalibrationAdmission contracts entry (Calibration.toolchainDigest prepared.baseline.buildInventory) semantics policy >>= require+ unless (Map.lookup entry.workloadSha256 (admissions :: Map.Map Text Text) == Just evidence) (ioError (userError "pinned calibration admission differs"))+ )+ prepared.plan.selected++replayCalibrationReceipt :: PerfContracts -> Calibration.CalibrationReceipt -> IO Calibration.VerifiedCalibrationTrial+replayCalibrationReceipt contracts reference = do+ (bucket, objectName) <- require (parseDestination reference.uri)+ prefix <- maybe (ioError (userError "calibration must reference a publication receipt")) pure (Text.stripSuffix "/receipt.json" objectName)+ (store, _) <- openStore bucket+ retained <- store.getObject bucket (ObjectName objectName)+ bytes <- maybe (ioError (userError "calibration receipt is missing")) (pure . fst) retained+ unless (sha256Hex (Lazy.toStrict bytes) == reference.sha256) (ioError (userError "calibration receipt digest differs"))+ receipt <- either (ioError . userError) pure (eitherDecode bytes :: Either String PublicationReceipt)+ unless (receipt.manifestUri == "gs://" <> bucket.unBucket <> "/" <> prefix <> "/manifest.json") (ioError (userError "calibration receipt destination differs"))+ unless ("snapshots/" `Text.isPrefixOf` receipt.snapshot && Text.count "/" receipt.snapshot == 1 && Text.all (\c -> c `elem` (['a' .. 'z'] <> ['0' .. '9'] <> ['-', '/'])) receipt.snapshot) (ioError (userError "unsafe calibration snapshot"))+ withSystemTempDirectory "kenshou-calibration-replay-" \root -> do+ let directory = root </> Text.unpack receipt.snapshot+ retrieveEvidence store bucket prefix receipt.manifestSha256 directory >>= require+ Lazy.writeFile (root </> "receipt.json") bytes+ (result, _) <- verifySessionEvidence contracts root True+ prepared <- loadPreparation contracts (directory </> "preparation/prepared.json")+ session <- readPerfSession (directory </> "session.json") >>= require+ unless (result.purpose == "calibration" && not result.gatingEligible && length result.configurations == 1 && length prepared.plan.selected == 1) (ioError (userError "calibration needs a single-profile non-gating report"))+ configuration <- case result.configurations of [retainedConfiguration] -> pure retainedConfiguration; _ -> ioError (userError "missing calibration configuration")+ semantics <- require (either (Left . Text.pack) Right (parseEither (withObject "PinnedInputs" (.: "harnessSemanticsSha256")) prepared.inputs))+ inputs <- require (parseInputs session)+ entry <- case prepared.plan.selected of [selected] -> pure selected; _ -> ioError (userError "missing calibration profile")+ let leasedCells = [identity.cell | event <- session.events, LeaseAcquired identity <- [event.action]]+ actualProfiles <-+ traverse+ (\cell -> maybe (ioError (userError "calibration lease is outside the frozen pool")) (pure . cellMachineProfile . fst) (find ((== cell) . (.name) . fst) inputs.cells))+ leasedCells+ unless (all (== entry.profile.machineProfile) actualProfiles && (result.infrastructureState /= "healthy" || not (null actualProfiles))) (ioError (userError "calibration ran on a different machine or image profile"))+ policies <- traverse (\cell -> maybe (ioError (userError "calibration health policy is absent")) pure (Map.lookup cell inputs.healthPolicies)) leasedCells+ policy <- case policies of first : rest | all (== first) rest -> pure first; [] | result.infrastructureState /= "healthy" -> maybe (ioError (userError "calibration health policy is absent")) pure (Map.lookup (case prepared.plan.request.cells of cell : _ -> cell; [] -> "") inputs.healthPolicies); _ -> ioError (userError "calibration health policy changed across leases")+ pure (Calibration.VerifiedCalibrationTrial session.sessionId prepared.plan.request.seed prepared.plan.request.control prepared.plan.request.project prepared.plan.request.baselineRevision prepared.plan.request.candidateRevision configuration.profile entry.profile.machineProfile (sha256Hex (canonicalEncode policy)) configuration.workloadSha256 (Calibration.toolchainDigest prepared.baseline.buildInventory) semantics configuration.performanceVerdict result.infrastructureState result.blockingRuntimeFailures session.createdAt receipt.totalElapsedSeconds)++runVerify :: FilePath -> Bool -> CliEnv -> IO ExitCode+runVerify output offline environment = guarded do+ (result, uri) <- verifySessionEvidence environment.perf output offline+ Lazy.putStrLn (encode (object ["verified" .= True, "offline" .= offline, "manifestUri" .= uri, "result" .= result]))+ pure (exit result.exitCode)++verifySessionEvidence :: PerfContracts -> FilePath -> Bool -> IO (PerfResult, Text)+verifySessionEvidence contracts output offline = do+ receipt <- readJSON (output </> "receipt.json")+ let relative = (receipt :: PublicationReceipt).snapshot+ uri = receipt.manifestUri+ manifestDigest = receipt.manifestSha256+ resultDigest = receipt.resultSha256+ unless ("snapshots/" `Text.isPrefixOf` relative && Text.all (\c -> c `elem` (['a' .. 'z'] <> ['0' .. '9'] <> ['-', '/'])) relative && Text.count "/" relative == 1) (ioError (userError "unsafe retained snapshot"))+ let directory = output </> Text.unpack relative+ verifyPerfManifest directory >>= require+ manifest <- Bytes.readFile (directory </> "manifest.json")+ unless (sha256Hex manifest == manifestDigest) (ioError (userError "retained manifest differs from durable receipt"))+ prepared <- loadPreparation contracts (directory </> "preparation/prepared.json")+ checkRootInputs directory prepared+ session <- readPerfSession (directory </> "session.json") >>= require+ require (validateSessionInputs contracts prepared session)+ unless offline (verifyGatingCalibration contracts prepared)+ unless (receipt.payloadsReadyAt == session.createdAt) (ioError (userError "receipt timing origin differs from its pinned session"))+ unless (uri == session.evidenceBaseUri <> "/sessions/" <> renderRunId session.sessionId <> "/" <> relative <> "/manifest.json") (ioError (userError "receipt destination differs from its pinned session"))+ inputs <- require (parseInputs session)+ unless offline (checkCalibratedPolicies prepared inputs)+ result <- readJSON (directory </> "result.json")+ unless (sha256Hex (canonicalEncode (toJSON (result :: PerfResult))) == resultDigest) (ioError (userError "retained aggregate differs from durable receipt"))+ unless (maybe False (<= receipt.totalElapsedSeconds) (Map.lookup "totalAtReportAssembly" result.stageSeconds)) (ioError (userError "report timing exceeds its verified publication time"))+ let base = Text.dropEnd (Text.length ("/manifest.json" :: Text)) uri+ unless ([reference.uri | reference <- result.artifactReferences, reference.kind == "aggregate-manifest"] == [uri] && [reference.uri | reference <- result.artifactReferences, reference.kind == "publication-receipt"] == [base <> "/receipt.json"]) (ioError (userError "report does not reference its own publication and receipt"))+ mapM_+ ( \reference -> case reference.sha256 of+ Nothing -> unless (reference.kind `elem` ["aggregate-manifest", "publication-receipt"]) (ioError (userError "artifact reference has no digest"))+ Just expected -> do+ path <- maybe (ioError (userError "artifact reference is outside the sealed snapshot")) pure (Text.stripPrefix (base <> "/") reference.uri)+ unless (all (\part -> not (Text.null part) && part `notElem` [".", ".."]) (Text.splitOn "/" path) && not (Text.any (`elem` ['\\', '\n', '\r']) path)) (ioError (userError "unsafe artifact reference"))+ (actual, _) <- digestFile (directory </> Text.unpack path)+ unless (expected == "sha256:" <> actual) (ioError (userError "artifact reference digest differs"))+ )+ result.artifactReferences+ state <- require (journalState session)+ auditAllAttempts False inputs prepared directory+ unless (maybe True (receipt.verifiedReportAt <=) state.executionDeadline) (ioError (userError "publication receipt exceeds the original execution deadline"))+ let completed = Map.elems state.completed+ valid <- traverse (verifyConfiguration inputs prepared directory) completed+ unless (and valid) (ioError (userError "retained comparison replay differs"))+ recomputed <- require (aggregateResult session.plan session.preparedSha256 result.evidenceVerified result.configurations)+ unless (recomputed == result {stageSeconds = Map.empty, artifactReferences = []} && all (\entry -> any (\retained -> retained.profile == entry.profile && retained.comparison == entry.comparison && retained.runtimeFailures == entry.runtimeFailures) result.configurations) completed) (ioError (userError "aggregate differs from the pinned session results"))+ unless offline do+ destination <- maybe (ioError (userError "receipt has no manifest URI")) pure (Text.stripSuffix "/manifest.json" uri)+ (bucket, prefix) <- require (parseDestination destination)+ (store, _) <- openStore bucket+ verifyPublication store bucket prefix directory >>= require+ stored <- store.getObject bucket (ObjectName (prefix <> "/receipt.json"))+ bytes <- Lazy.readFile (output </> "receipt.json")+ unless (fmap fst stored == Just bytes) (ioError (userError "retained publication receipt differs from durable storage"))+ pure (result, uri)++accessProbe :: ObjectStore -> Text -> RunId -> Prepared.PerfPrepared -> IO ()+accessProbe store uri identifier prepared = do+ (bucket, prefix) <- require (parseDestination uri)+ let name = ObjectName (prefix <> "/access-checks/" <> renderRunId identifier <> ".json")+ bytes = encode (object ["session" .= identifier, "preparedSha256" .= Prepared.preparedDigest prepared])+ outcome <- store.putObject bucket name "application/json" DoesNotExist bytes+ unless (case outcome of Written _ -> True; _ -> False) (ioError (userError "performance evidence destination failed create-only access check"))+ found <- store.getObject bucket name+ unless (fmap fst found == Just bytes) (ioError (userError "performance evidence destination failed retrieval access check"))++parseDestination :: Text -> Either Text (Bucket, Text)+parseDestination uri = do+ suffix <- maybe (Left "evidence destination must be gs://BUCKET/PREFIX") Right (Text.stripPrefix "gs://" uri)+ let (bucket, path) = Text.breakOn "/" suffix+ prefix = Text.drop 1 path+ unless (not (Text.null bucket) && Text.all (\c -> c `elem` ['a' .. 'z'] <> ['0' .. '9'] <> ['-', '.']) bucket && not (Text.null prefix) && all (\part -> not (Text.null part) && part `notElem` [".", ".."]) (Text.splitOn "/" prefix) && not (Text.any (`elem` ['\\', '\n', '\r', ' ', '?', '#']) prefix)) (Left "unsafe evidence destination")+ pure (Bucket bucket, prefix)++copyTree :: FilePath -> FilePath -> IO ()+copyTree source destination = do+ link <- pathIsSymbolicLink source+ when link (ioError (userError "performance artifacts may not contain symbolic links"))+ directory <- doesDirectoryExist source+ if directory+ then do+ createDirectoryIfMissing True destination+ names <- listDirectory source+ mapM_ (\name -> copyTree (source </> name) (destination </> name)) names+ else copyFile source destination++renderReport :: PerfResult -> Text+renderReport result =+ Text.unlines+ ( ["# Runtime performance validation", "", "Performance: " <> result.performanceVerdict, "Coverage: " <> result.coverage <> " (whole change: " <> result.changeCoverage <> ")", "Infrastructure: " <> result.infrastructureState, "Purpose: " <> result.purpose, "Gating eligible: " <> Text.pack (show result.gatingEligible), "Exit: " <> Text.pack (show result.exitCode), "", "| Profile | Verdict | Valid pairs | Infrastructure |", "| --- | --- | --- | --- |"]+ <> ["| " <> item.profile <> " | " <> item.performanceVerdict <> " | " <> Text.pack (show item.validPairs) <> "/" <> Text.pack (show item.requiredPairs) <> " | " <> item.infrastructureState <> " |" | item <- result.configurations]+ <> concatMap effects result.configurations+ <> ["", "Runtime blockers: " <> if null result.blockingRuntimeFailures then "none" else Text.intercalate "; " result.blockingRuntimeFailures]+ <> ["", "## Scope", ""]+ <> ["- Excluded " <> scenario <> ": " <> reason | (scenario, reason) <- result.exclusions]+ <> ["- Uncovered: " <> scenario | scenario <- result.uncoveredScenarios]+ <> ["", "## Timings", "", "| Stage | Seconds |", "| --- | ---: |"]+ <> ["| " <> stage <> " | " <> number seconds <> " |" | (stage, seconds) <- Map.toAscList result.stageSeconds]+ <> ["", "Full report retrieval time is recorded in the linked publication receipt.", "", "## Evidence", ""]+ <> ["- [" <> reference.kind <> "](" <> reference.uri <> ")" <> maybe "" (\digest -> " (`" <> digest <> "`)") reference.sha256 | reference <- result.artifactReferences]+ )+ where+ number amount = Text.pack (printf "%.4g" (amount :: Double))+ effects configuration = case configuration.comparison of+ Nothing -> []+ Just document -> case parseEither metrics document of+ Left problem -> ["", "Retained metric report could not be rendered: " <> Text.pack problem]+ Right (rows, reasons) -> ["", "## " <> configuration.profile, "", "Intervals and thresholds below are retained comparison results.", "", "| Metric | Unit | Ratio estimate [low, high] | Delta estimate [low, high] | Relative limit | Absolute floor | Status | Explanation |", "| --- | --- | --- | --- | ---: | ---: | --- | --- |"] <> rows <> ["", "Comparison reasons: " <> if null reasons then "none" else Text.intercalate "; " reasons]+ metrics = withObject "Comparison" \document -> do+ values <- document .: "metrics"+ reasons <- document .:? "reasons" .!= []+ rows <- traverse metric (Map.toAscList (values :: Map.Map Text Value))+ pure (rows, reasons)+ metric (name, document) =+ withObject+ "MetricComparison"+ ( \fields -> do+ unit <- fields .: "unit"+ ratio <- fields .: "ratio" >>= interval+ delta <- fields .: "delta" >>= interval+ limit <- fields .: "relativeLimit"+ floorValue <- fields .: "absoluteFloor"+ status <- fields .: "status"+ message <- fields .: "message"+ pure ("| " <> name <> " | " <> unit <> " | " <> ratio <> " | " <> delta <> " | " <> number limit <> " | " <> number floorValue <> " | " <> status <> " | " <> Text.replace "\n" " " (Text.replace "|" "\\|" message) <> " |")+ )+ document+ interval = withObject "Interval" \fields -> do+ low <- fields .: "low"+ estimate <- fields .: "estimate"+ high <- fields .: "high"+ pure (number estimate <> " [" <> number low <> ", " <> number high <> "]")++openStore :: Bucket -> IO (ObjectStore, Text)+openStore bucket = do+ selected <- lookupEnv "KENSHOU_CELL_STORE"+ case selected of+ Just setting -> do+ path <- maybe (ioError (userError "KENSHOU_CELL_STORE must be file:<dir> or unset for GCS")) (pure . Text.unpack) (Text.stripPrefix "file:" (Text.pack setting))+ when (null path) (ioError (userError "file-backed cell store directory is empty"))+ absolute <- makeAbsolute path+ store <- newFileStore absolute+ pure (store, "file:" <> Text.pack absolute)+ Nothing -> do+ tokens <- newTokenProvider bucket+ store <- newGcsStore tokens+ pure (store, "gcs")++readJSON :: (FromJSON a) => FilePath -> IO a+readJSON path = eitherDecodeFileStrict' path >>= either (ioError . userError . ((path <> ": ") <>)) pure++decode :: (FromJSON a) => Value -> Either Text a+decode document = case fromJSON document of Error problem -> Left (Text.pack problem); Success result -> Right result++require :: Either Text a -> IO a+require = either (ioError . userError . Text.unpack) pure++input :: Either Text a -> IO a+input = either (throwIO . SessionFailure 2) pure++exit :: Int -> ExitCode+exit 0 = ExitSuccess+exit code = ExitFailure code++guarded :: IO ExitCode -> IO ExitCode+guarded action = catch executeFailure (\(SessionFailure code problem) -> TextIO.hPutStrLn stderr problem >> pure (exit code))+ where+ executeFailure = do+ result <- try @IOException action+ case result of+ Right code -> pure code+ Left problem -> do+ TextIO.hPutStrLn stderr (Text.pack (displayException problem))+ pure (ExitFailure 4)
+ src/Kenshou/Cli/Command/Plan.hs view
@@ -0,0 +1,345 @@+{-# LANGUAGE FieldSelectors #-}++module Kenshou.Cli.Command.Plan (planCommand) where++import Data.Aeson qualified as Aeson+import Data.Bifunctor (first)+import Data.ByteString.Lazy.Char8 qualified as LazyByteString+import Data.List (nubBy, partition)+import Data.List.NonEmpty qualified as NonEmpty+import Data.Set qualified as Set+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Text.IO qualified as Text.IO+import Kenshou.Cli.Cohort (resolveDefaultCohortIdentity)+import Kenshou.Core.Bundle (allScenarios)+import Kenshou.Core.Cli+import Kenshou.Core.Cohort qualified as Cohort+import Kenshou.Core.Id qualified as Id+import Kenshou.Core.Knob (KnobSpec (..), renderKnobName)+import Kenshou.Core.RunSpec (SpecPlacement (..))+import Kenshou.Core.Scenario (Tier (..))+import Kenshou.Plan.Catalog (ScenarioInfo (..), decodeCatalog, fromScenario)+import Kenshou.Plan.Change+import Kenshou.Plan.Change.Cohort+import Kenshou.Plan.Change.Git+import Kenshou.Plan.Components+import Kenshou.Plan.Components qualified as Components+import Kenshou.Plan.Components.Check+import Kenshou.Plan.Policy+import Kenshou.Plan.RunPlan+import Kenshou.Plan.Selector (Selector, parseSelector, renderSelector)+import Kenshou.Plan.Suite+import Options.Applicative hiding (value)+import System.Exit (ExitCode (..))+import System.IO (stderr)+import System.Random (randomIO)++data PlanOptions = PlanOptions+ { graphShow :: Bool,+ graphCheck :: Bool,+ json :: Bool,+ graphSource :: Maybe InputSource,+ catalogSource :: Maybe InputSource,+ suiteName :: Maybe Text,+ suiteSource :: Maybe InputSource,+ changed :: [Text],+ cohortFrom :: Maybe Text,+ cohortTo :: Maybe Text,+ since :: Maybe Text,+ upstreamDiffs :: [Text],+ allScenariosFlag :: Bool,+ includes :: [Text],+ excludes :: [Text],+ explain :: Bool,+ outputPath :: Maybe FilePath,+ maxTierText :: Maybe Text,+ kindTexts :: [Text],+ placementText :: Maybe Text,+ dimensionPolicyText :: Maybe Text,+ knobPolicyText :: Maybe Text,+ trialCount :: Maybe Int,+ budget :: Maybe Int,+ seedValue :: Maybe Integer,+ knobPins :: [Text],+ dimensionPins :: [Text]+ }++planCommand :: CliCommand+planCommand =+ CliCommand+ { name = "plan",+ description = "Select and plan verification runs",+ group = Discovery,+ hidden = False,+ parser = runPlan <$> planParser+ }++planParser :: Parser PlanOptions+planParser =+ PlanOptions+ <$> switch (long "graph-show" <> help "Print the checked-in component graph")+ <*> switch (long "graph-check" <> help "Check graph validity, build edges, and selectors")+ <*> switch (long "json" <> help "Write machine-readable JSON")+ <*> optional (option (parseInputSource <$> str) (long "graph" <> metavar "FILE" <> help "Override the component graph; use - for stdin"))+ <*> optional (option (parseInputSource <$> str) (long "catalog" <> metavar "FILE" <> help "Plan against a scenario catalog; use - for stdin"))+ <*> optional (Text.pack <$> strOption (long "suite" <> metavar "NAME" <> help "Apply suites/NAME.json"))+ <*> optional (option (parseInputSource <$> str) (long "suite-file" <> metavar "FILE" <> help "Apply a suite document; use - for stdin"))+ <*> many (Text.pack <$> strOption (long "changed" <> metavar "COMPONENT[,COMPONENT]" <> help "Name changed components or sub-components"))+ <*> optional (Text.pack <$> strOption (long "cohort-from" <> metavar "COHORT" <> help "Baseline cohort name, file, or git object"))+ <*> optional (Text.pack <$> strOption (long "cohort-to" <> metavar "COHORT" <> help "Candidate cohort name, file, or git object"))+ <*> optional (Text.pack <$> strOption (long "since" <> metavar "REV" <> help "Map repository changes since a revision"))+ <*> many (Text.pack <$> strOption (long "upstream-diff" <> metavar "REPO=PATH@REVA..REVB" <> help "Map paths changed in an upstream checkout"))+ <*> switch (long "all" <> help "Select every catalog scenario")+ <*> many (Text.pack <$> strOption (long "select" <> metavar "SELECTOR" <> help "Keep matching scenarios"))+ <*> many (Text.pack <$> strOption (long "exclude" <> metavar "SELECTOR" <> help "Remove matching scenarios"))+ <*> switch (long "explain" <> help "Explain changes, dependency paths, and selections")+ <*> optional (strOption (long "out" <> metavar "FILE" <> help "Write the run plan to a file"))+ <*> optional (Text.pack <$> strOption (long "max-tier" <> metavar "TIER" <> help "Maximum cost tier"))+ <*> many (Text.pack <$> strOption (long "kind" <> metavar "KIND" <> help "Include a scenario kind; repeatable"))+ <*> optional (Text.pack <$> strOption (long "placement" <> metavar "local|cell" <> help "Target placement"))+ <*> optional (Text.pack <$> strOption (long "dimension-policy" <> metavar "POLICY" <> help "default-only, telemetry-corners, pairwise, or full"))+ <*> optional (Text.pack <$> strOption (long "knob-policy" <> metavar "POLICY" <> help "defaults or declared-variants"))+ <*> optional (option auto (long "trials" <> metavar "N" <> help "Benchmark trials (minimum 3)"))+ <*> optional (option auto (long "budget-minutes" <> metavar "N" <> help "Maximum estimated run time"))+ <*> optional (option auto (long "seed" <> metavar "N" <> help "Deterministic plan seed"))+ <*> many (Text.pack <$> strOption (long "set" <> metavar "NAME=VALUE" <> help "Pin a knob value"))+ <*> many (Text.pack <$> strOption (long "dim" <> metavar "NAME=VALUE" <> help "Pin a dimension value"))++runPlan :: PlanOptions -> CliEnv -> IO ExitCode+runPlan options environment+ | stdinCount options > 1 = usage "at most one document input may use standard input"+ | options.graphShow && options.graphCheck = usage "choose only one of --graph-show or --graph-check"+ | options.suiteName /= Nothing && options.suiteSource /= Nothing = usage "choose only one of --suite or --suite-file"+ | otherwise = do+ graphResult <- loadGraph environment options.graphSource+ catalogResult <- loadCatalog environment options.catalogSource+ suiteResult <- loadSuite options+ case (graphResult, catalogResult, suiteResult) of+ (Left err, _, _) -> usage err+ (_, Left err, _) -> usage err+ (_, _, Left err) -> usage err+ (Right graph, Right catalog, Right suite)+ | options.graphShow -> showGraph options.json graph+ | options.graphCheck -> checkGraph graph catalog+ | otherwise -> planSelection options suite graph catalog++planSelection :: PlanOptions -> Maybe Suite -> ComponentGraph -> [ScenarioInfo] -> IO ExitCode+planSelection options suite graph catalog = case (traverse (parseSelector) options.includes, traverse parseSelector options.excludes) of+ (Left err, _) -> usage err+ (_, Left err) -> usage err+ (Right includes, Right excludes) -> do+ gathered <- gatherChanges options graph+ case gathered of+ Left err -> usage err+ Right (changes, directSelectors, warnings)+ | null changes && null directSelectors && not options.allScenariosFlag && not (maybe False ((== SuiteAll) . (.mode) . (.selection)) suite) -> usage "say what changed with --changed, --cohort-from/--cohort-to, --since, --upstream-diff, --all, or an all-mode suite"+ | otherwise -> do+ let componentSelected = if options.allScenariosFlag then selectAll catalog else selectScenarios graph catalog changes+ pathSelected = selectBySelectors Since "repository path" directSelectors catalog+ changedSelected = deduplicate (componentSelected <> pathSelected)+ suiteSelected = maybe changedSelected (\suiteValue -> applySuite suiteValue catalog changedSelected) suite+ selected = applySelectors includes excludes (deduplicate suiteSelected)+ unmatchedPins = [name | raw <- options.knobPins, let (name, _) = splitAssignment raw, all (\selection -> not (declares name selection.scenario)) selected]+ if not (null unmatchedPins)+ then usage ("no selected scenario declares knob: " <> Text.intercalate ", " unmatchedPins)+ else+ if options.explain+ then renderSelection True changes warnings selected >> pure ExitSuccess+ else writeRunPlan options suite graph changes warnings selected+ where+ declares name scenario = any ((== name) . renderKnobName . (.name)) scenario.knobs++writeRunPlan :: PlanOptions -> Maybe Suite -> ComponentGraph -> [Change] -> [Warning] -> [Selected] -> IO ExitCode+writeRunPlan options suite graph changes warnings selected = do+ seedResult <- makeSeed options.seedValue+ cohortResult <- first (("unable to resolve cohort identity: " <>) . Text.pack . show) <$> resolveDefaultCohortIdentity+ case (seedResult, cohortResult, (\seed -> makePolicy options suite seed) =<< seedResult) of+ (Left err, _, _) -> usage err+ (_, Left err, _) -> usage err+ (_, _, Left err) -> usage err+ (Right _, Right cohort, Right policy) -> do+ let context =+ PlanContext+ { suite = fmap (.name) suite,+ graphDigest = Components.graphDigest graph,+ cohortName = Cohort.unCohortName cohort.identityCohort,+ cohortPlanHash = Cohort.unPlanHash cohort.identityPlanHash,+ inputs = PlanInputs (inputValue options),+ changes,+ warnings = fmap (.message) warnings+ }+ plan <- stampPlan (buildPlan context policy selected)+ let bytes = Aeson.encode plan+ case options.outputPath of+ Nothing -> LazyByteString.putStrLn bytes+ Just path -> LazyByteString.writeFile path bytes+ Text.IO.hPutStrLn stderr ("planned " <> Text.pack (show (length plan.runs)) <> " runs, estimate " <> Text.pack (show plan.estimateMinutes) <> " min, " <> Text.pack (show (length plan.skipped)) <> " skipped")+ pure ExitSuccess++makeSeed :: Maybe Integer -> IO (Either Text Id.Seed)+makeSeed (Just value)+ | value < 0 = pure (Left "seed must be non-negative")+ | otherwise = pure (Id.mkSeed (fromInteger value))+makeSeed Nothing = do+ value <- randomIO+ pure (Id.mkSeed (value `mod` 9007199254740992))++makePolicy :: PlanOptions -> Maybe Suite -> Id.Seed -> Either Text PlanPolicy+makePolicy options suite seed = do+ base <- maybe (Right (defaultPlanPolicy seed)) (suitePolicy seed) suite+ maxTier <- maybe (Right base.maxTier) parseTierValue options.maxTierText+ let kindInputs = concatMap (filter (not . Text.null) . fmap Text.strip . Text.splitOn ",") options.kindTexts+ kinds <- if null kindInputs then Right base.kinds else Set.fromList <$> traverse Id.parseKind kindInputs+ placement <- maybe (Right base.placement) parsePlacementValue options.placementText+ dimensionPolicy <- maybe (Right base.dimensionPolicy) (maybe (Left "unknown dimension policy") Right . parseDimensionPolicy) options.dimensionPolicyText+ knobPolicy <- maybe (Right base.knobPolicy) (maybe (Left "unknown knob policy") Right . parseKnobPolicy) options.knobPolicyText+ let trials = maybe base.trials (\value -> value) options.trialCount+ if trials < 3 then Left "--trials must be at least 3" else pure ()+ case options.budget of Just value | value < 0 -> Left "--budget-minutes must be non-negative"; _ -> pure ()+ pinnedKnobs <- traverse parseAssignmentText options.knobPins+ pinnedDimensions <- traverse parseAssignmentText options.dimensionPins+ pure+ base+ { maxTier,+ kinds,+ placement,+ dimensionPolicy,+ knobPolicy,+ trials,+ budgetMinutes = options.budget <|> base.budgetMinutes,+ pinnedKnobs,+ pinnedDimensions+ }++parseTierValue :: Text -> Either Text Tier+parseTierValue "smoke" = Right TierSmoke+parseTierValue "standard" = Right TierStandard+parseTierValue "extended" = Right TierExtended+parseTierValue "soak" = Right TierSoak+parseTierValue value = Left ("unknown tier " <> value)++parsePlacementValue :: Text -> Either Text SpecPlacement+parsePlacementValue "local" = Right RunLocal+parsePlacementValue "cell" = Right RunOnCell+parsePlacementValue value = Left ("unknown placement " <> value)++parseAssignmentText :: Text -> Either Text (Text, Text)+parseAssignmentText raw = case splitAssignment raw of+ (name, value) | Text.null name || Text.null value -> Left "expected NAME=VALUE"+ pair -> Right pair++splitAssignment :: Text -> (Text, Text)+splitAssignment raw = let (name, rest) = Text.breakOn "=" raw in (name, Text.drop 1 rest)++inputValue :: PlanOptions -> Aeson.Value+inputValue options =+ Aeson.object+ [ "changed" Aeson..= options.changed,+ "cohortFrom" Aeson..= options.cohortFrom,+ "cohortTo" Aeson..= options.cohortTo,+ "since" Aeson..= options.since,+ "upstreamDiffs" Aeson..= options.upstreamDiffs,+ "all" Aeson..= options.allScenariosFlag,+ "select" Aeson..= options.includes,+ "exclude" Aeson..= options.excludes+ ]++gatherChanges :: PlanOptions -> ComponentGraph -> IO (Either Text ([Change], [Selector], [Warning]))+gatherChanges options graph = do+ cohortResult <- gatherCohort+ sinceResult <- maybe (pure (Right ([], [], []))) (changesSince graph ".") options.since+ upstreamResults <- traverse gatherUpstream options.upstreamDiffs+ pure do+ named <- traverse (parseRef graph) namedInputs+ (cohortChanges, cohortWarnings) <- cohortResult+ (sinceChanges, sinceSelectors, sinceWarnings) <- sinceResult+ upstream <- sequence upstreamResults+ let upstreamChanges = concatMap fst upstream+ upstreamWarnings = [Warning "ignored-upstream-paths" (Text.pack (show ignored) <> " upstream paths were outside component source roots") | (_, ignored) <- upstream, ignored > 0]+ namedChanges = [Change reference Named ("--changed " <> renderRef reference) | reference <- named]+ Right (namedChanges <> cohortChanges <> sinceChanges <> upstreamChanges, sinceSelectors, cohortWarnings <> sinceWarnings <> upstreamWarnings)+ where+ namedInputs = concatMap (filter (not . Text.null) . fmap Text.strip . Text.splitOn ",") options.changed+ gatherCohort = case (options.cohortFrom, options.cohortTo) of+ (Nothing, Nothing) -> pure (Right ([], []))+ (Just from, Just to) -> do+ old <- readCohortPackages "." (parseCohortInput from)+ new <- readCohortPackages "." (parseCohortInput to)+ pure ((uncurry (diffCohorts graph)) <$> ((,) <$> old <*> new))+ _ -> pure (Left "--cohort-from and --cohort-to must be supplied together")+ gatherUpstream raw = case parseUpstreamDiff raw of+ Left err -> pure (Left err)+ Right input -> changesFromUpstream graph input++deduplicate :: [Selected] -> [Selected]+deduplicate = nubBy (\left right -> left.scenario.id == right.scenario.id)++renderSelection :: Bool -> [Change] -> [Warning] -> [Selected] -> IO ()+renderSelection explain changes warnings selected = do+ mapM_ (Text.IO.hPutStrLn stderr . ("warning: " <>) . (.message)) warnings+ if explain+ then do+ mapM_ renderChange changes+ mapM_ renderSelected selected+ Text.IO.putStrLn ("selected " <> Text.pack (show (length selected)) <> " scenarios")+ else mapM_ (Text.IO.putStrLn . Id.renderScenarioId . (.id) . (.scenario)) selected+ where+ renderChange changeValue = Text.IO.putStrLn ("changed " <> renderRef changeValue.ref <> " " <> changeValue.detail)+ renderSelected selectedValue = do+ let reason = NonEmpty.head selectedValue.reasons+ Text.IO.putStrLn ("selected " <> Id.renderScenarioId selectedValue.scenario.id <> " via " <> Text.intercalate " <- " (fmap renderRef reason.via) <> " selector " <> renderSelector reason.selector)++loadGraph :: CliEnv -> Maybe InputSource -> IO (Either Text ComponentGraph)+loadGraph environment Nothing = pure (firstGraph (decodeGraph environment.componentGraph))+loadGraph _ (Just source) = firstGraph . decodeGraph <$> readInputSource source++firstGraph :: Either GraphError ComponentGraph -> Either Text ComponentGraph+firstGraph = either (Left . (.errorText)) Right++loadCatalog :: CliEnv -> Maybe InputSource -> IO (Either Text [ScenarioInfo])+loadCatalog environment Nothing = pure (Right (fmap fromScenario (allScenarios environment.registry)))+loadCatalog _ (Just source) = decodeCatalog <$> readInputSource source++loadSuite :: PlanOptions -> IO (Either Text (Maybe Suite))+loadSuite options = case (options.suiteName, options.suiteSource) of+ (Nothing, Nothing) -> pure (Right Nothing)+ (Just name, Nothing) -> fmap Just <$> readSuite ("suites/" <> Text.unpack name <> ".json")+ (Nothing, Just source) -> fmap Just . decodeSuite <$> readInputSource source+ (Just _, Just _) -> pure (Left "choose only one of --suite or --suite-file")++showGraph :: Bool -> ComponentGraph -> IO ExitCode+showGraph asJson graph = do+ if asJson+ then LazyByteString.putStrLn (Aeson.encode graph)+ else do+ let subCount = sum (fmap (length . (.subcomponents)) graph.components)+ edges = concatMap (.dependsOn) graph.components+ buildCount = length (filter ((== Build) . (.kind)) edges)+ Text.IO.putStrLn (Text.pack (show (length graph.components)) <> " components, " <> Text.pack (show subCount) <> " sub-components, " <> Text.pack (show (length edges)) <> " edges (" <> Text.pack (show buildCount) <> " build, " <> Text.pack (show (length edges - buildCount)) <> " runtime)")+ mapM_ renderComponent graph.components+ pure ExitSuccess+ where+ renderComponent componentValue = Text.IO.putStrLn (Components.unComponentId componentValue.id <> " " <> Text.pack (show componentValue.kind) <> " " <> Text.intercalate "," componentValue.packages)++checkGraph :: ComponentGraph -> [ScenarioInfo] -> IO ExitCode+checkGraph graph catalog = do+ driftResult <- checkAgainstPlanJson graph "dist-newstyle/cache/plan.json"+ case driftResult of+ Left err -> usage ("cannot read Cabal plan: " <> err)+ Right drift -> do+ let lint = lintAgainstCatalog graph catalog+ (driftErrors, driftInfo) = partition isDriftError drift+ (lintErrors, lintWarnings) = partition isLintError lint+ Text.IO.putStrLn ("component graph: " <> Text.pack (show (length graph.components)) <> " components; digest " <> Components.graphDigest graph)+ mapM_ (Text.IO.putStrLn . ("error: " <>) . Text.pack . show) driftErrors+ mapM_ (Text.IO.putStrLn . ("error: " <>) . Text.pack . show) lintErrors+ mapM_ (Text.IO.putStrLn . ("info: " <>) . Text.pack . show) driftInfo+ mapM_ (Text.IO.putStrLn . ("warning: " <>) . Text.pack . show) lintWarnings+ pure if null driftErrors && null lintErrors then ExitSuccess else ExitFailure 1++stdinCount :: PlanOptions -> Int+stdinCount options = length [() | Just InputStdin <- [options.graphSource, options.catalogSource, options.suiteSource]]++usage :: Text -> IO ExitCode+usage message = Text.IO.hPutStrLn stderr ("kenshou: " <> message) >> pure (ExitFailure 2)
+ src/Kenshou/Cli/Command/Run.hs view
@@ -0,0 +1,207 @@+module Kenshou.Cli.Command.Run (runCommand) where++import Data.Aeson qualified as Aeson+import Data.Bifunctor (first)+import Data.ByteString.Lazy.Char8 qualified as LazyByteString+import Data.List.NonEmpty qualified as NonEmpty+import Data.Map.Strict qualified as Map+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Text.IO qualified as Text.IO+import Kenshou.Cli.Cohort (resolveDefaultCohortIdentity)+import Kenshou.Cli.Config+import Kenshou.Core.Cli+import Kenshou.Core.Cli.Config+import Kenshou.Core.Id+import Kenshou.Core.Knob (parseAssignment)+import Kenshou.Core.Outcome (renderOutcome)+import Kenshou.Core.Phase (PhasePlan (..), zeroPhases)+import Kenshou.Core.Run+import Kenshou.Core.RunResult (RunResult (..))+import Kenshou.Core.RunSpec+import Kenshou.Core.RunSpec.Resolve+import Options.Applicative+import Settei hiding (optional)+import Settei.Env (envSnapshot)+import Settei.Optparse+import System.Environment (getArgs, getEnvironment)+import System.Exit (ExitCode (..))+import System.IO (stderr)++data RunOptions = RunOptions+ { specFile :: Maybe InputSource,+ scenario :: Maybe Text,+ config :: ConfigInputs,+ knobs :: [Text],+ dimensions :: [Text],+ seed :: Maybe Seed,+ runId :: Maybe RunId,+ timeout :: Maybe Int,+ phases :: [Text],+ pgUrl :: Maybe Text,+ pgUrlEnv :: Maybe Text,+ placement :: Maybe SpecPlacement,+ machineProfile :: Maybe Text,+ cohortIdentity :: Maybe InputSource,+ cellFingerprint :: Maybe InputSource,+ keepEnvironment :: Bool,+ printSpec :: Bool,+ json :: Bool,+ strictKnownDefects :: Bool+ }++runCommand :: CliCommand+runCommand = CliCommand "run" "Resolve or execute one scenario" Execution False (runHandler <$> runParser)++runParser :: Parser RunOptions+runParser =+ RunOptions+ <$> optional (option (parseInputSource <$> str) (long "spec" <> metavar "FILE" <> help "Read a kenshou.run-spec/v1 document; use - for stdin"))+ <*> optional (Text.pack <$> strArgument (metavar "SCENARIO" <> help "Registered scenario identifier"))+ <*> configInputsParser ((: []) <$> namedOption "--out" outputRootKey (long "out" <> metavar "DIR" <> help "Run output directory"))+ <*> many (Text.pack <$> strOption (long "set" <> metavar "NAME=VALUE" <> help "Set a scenario knob"))+ <*> many (Text.pack <$> strOption (long "dim" <> metavar "NAME=VALUE" <> help "Set a cross-cutting dimension"))+ <*> optional (option (eitherReader readSeed) (long "seed" <> metavar "N" <> help "Replayable random seed"))+ <*> optional (option (eitherReader readRunId) (long "run-id" <> metavar "UUID" <> help "Pre-assign a UUIDv7 run id"))+ <*> optional (option auto (long "timeout" <> metavar "SECONDS" <> help "Run timeout"))+ <*> many (Text.pack <$> strOption (long "phase" <> metavar "NAME=SECONDS" <> help "Override warm-up, steady, or drain duration"))+ <*> optional (Text.pack <$> strOption (long "pg-url" <> metavar "CONNECTION" <> help "Use an external PostgreSQL connection string"))+ <*> optional (Text.pack <$> strOption (long "pg-url-env" <> metavar "VAR" <> help "Read the external PostgreSQL connection string from VAR"))+ <*> optional (option (eitherReader readPlacement) (long "placement" <> metavar "local|cell" <> help "Execution placement"))+ <*> optional (Text.pack <$> strOption (long "machine-profile" <> metavar "NAME" <> help "Stable machine profile name"))+ <*> optional (option (parseInputSource <$> str) (long "cohort-identity" <> metavar "FILE" <> help "Use a resolved cohort identity; use - for stdin"))+ <*> optional (option (parseInputSource <$> str) (long "cell-fingerprint" <> metavar "FILE" <> help "Attach a cell fingerprint; use - for stdin"))+ <*> switch (long "keep-env" <> help "Leave disposable environments running for inspection")+ <*> switch (long "print-spec" <> help "Print the effective run specification without executing")+ <*> switch (long "json" <> help "Print the run result as JSON")+ <*> switch (long "strict-known-defects" <> help "Treat reproduced known defects as blocking")++runHandler :: RunOptions -> CliEnv -> IO ExitCode+runHandler options environment = case schemaDiagnostic options.config.diagnostic (describe runDefaultsConfig) of+ Just output -> Text.IO.putStr output >> pure ExitSuccess+ Nothing -> do+ processEnvironment <- fmap (fmap (\(name, value) -> (Text.pack name, Text.pack value))) getEnvironment+ defaultsResult <- resolveRunDefaults (envSnapshot processEnvironment) options.config+ case defaultsResult of+ Left err -> Text.IO.hPutStrLn stderr err >> pure (ExitFailure 2)+ Right resolved -> case resolved.answer of+ Left problems -> Text.IO.hPutStr stderr (renderErrorsText problems) >> pure (ExitFailure 2)+ Right defaults -> case resolutionDiagnostic options.config.diagnostic resolved of+ Just output -> Text.IO.putStr output >> pure ExitSuccess+ Nothing -> resolveAndRun defaults+ where+ resolveAndRun defaults = do+ if stdinInputCount options > 1+ then Text.IO.hPutStrLn stderr "kenshou: at most one document input may use standard input" >> pure (ExitFailure 2)+ else do+ input <- loadInput options+ case input >>= applyOverrides options of+ Left err -> Text.IO.hPutStrLn stderr ("kenshou: " <> err) >> pure (ExitFailure 2)+ Right spec -> do+ resolved <- resolveRunSpec environment.registry spec+ case resolved of+ Left problems -> Text.IO.hPutStrLn stderr (Text.intercalate "\n" [message | SpecError message <- NonEmpty.toList problems]) >> pure (ExitFailure 2)+ Right (_, effective)+ | options.printSpec -> LazyByteString.putStrLn (Aeson.encode effective) >> pure ExitSuccess+ | otherwise -> execute defaults spec++ execute defaults spec = do+ cohortResult <- loadCohort options.cohortIdentity+ cellResult <- loadJsonDocument options.cellFingerprint+ case (cohortResult, cellResult) of+ (Left err, _) -> Text.IO.hPutStrLn stderr ("kenshou: " <> err) >> pure (ExitFailure 2)+ (_, Left err) -> Text.IO.hPutStrLn stderr ("kenshou: " <> err) >> pure (ExitFailure 2)+ (Right cohort, Right cellFingerprint) -> do+ argv <- getArgs+ result <- executeRun (RunnerConfig environment.registry (Text.unpack defaults.outputRoot) cohort cellFingerprint options.keepEnvironment options.strictKnownDefects argv) spec+ case result of+ Left problems -> Text.IO.hPutStrLn stderr (Text.intercalate "\n" [message | SpecError message <- NonEmpty.toList problems]) >> pure (ExitFailure 2)+ Right output -> do+ if options.json+ then LazyByteString.putStrLn (Aeson.encode output.result)+ else Text.IO.putStrLn (renderOutcome output.result.outcome <> " " <> renderScenarioId output.result.scenario <> " " <> Text.pack output.directory)+ pure (if output.result.exitCode == 0 then ExitSuccess else ExitFailure output.result.exitCode)++ loadCohort Nothing = first (("unable to resolve cohort identity: " <>) . Text.pack . show) <$> resolveDefaultCohortIdentity+ loadCohort (Just source) = loadDocument "cohort identity" source++ loadJsonDocument Nothing = pure (Right Nothing)+ loadJsonDocument (Just source) = fmap Just <$> loadDocument "cell fingerprint" source++ loadDocument :: (Aeson.FromJSON value) => Text -> InputSource -> IO (Either Text value)+ loadDocument label source = first (("invalid " <> label <> ": ") <>) . first Text.pack . Aeson.eitherDecodeStrict' <$> readInputSource source++loadInput :: RunOptions -> IO (Either Text RunSpec)+loadInput options = case (options.specFile, options.scenario) of+ (Just _, Just _) -> pure (Left "choose either --spec FILE or SCENARIO")+ (Nothing, Nothing) -> pure (Left "provide --spec FILE or SCENARIO")+ (Just source, Nothing) -> do+ bytes <- readInputSource source+ pure (first Text.pack (Aeson.eitherDecodeStrict' bytes))+ (Nothing, Just scenarioText) -> pure (minimalRunSpec <$> parseScenarioId scenarioText)++applyOverrides :: RunOptions -> RunSpec -> Either Text RunSpec+applyOverrides options spec = do+ knobs <- first (Text.pack . show) (traverse parseAssignment options.knobs)+ dimensions <- traverse parseDimension options.dimensions+ phases <- applyPhaseOverrides spec.phases options.phases+ postgres <- case (options.pgUrl, options.pgUrlEnv) of+ (Just _, Just _) -> Left "choose either --pg-url or --pg-url-env"+ (Just connection, Nothing) -> Right (Just (PostgresExternal (ConnLiteral connection)))+ (Nothing, Just variable) -> Right (Just (PostgresExternal (ConnFromEnv variable)))+ (Nothing, Nothing) -> Right spec.environment.postgres+ let environment =+ spec.environment+ { placement = maybe spec.environment.placement id options.placement,+ machineProfile = options.machineProfile <> spec.environment.machineProfile,+ postgres+ }+ pure+ RunSpec+ { runId = options.runId <|> spec.runId,+ scenario = spec.scenario,+ scenarioRevision = spec.scenarioRevision,+ knobs = spec.knobs <> knobs,+ dimensions = spec.dimensions <> dimensions,+ seed = options.seed <|> spec.seed,+ phases,+ timeoutSeconds = options.timeout <|> spec.timeoutSeconds,+ environment,+ cohortExpectation = spec.cohortExpectation,+ comparison = spec.comparison,+ labels = spec.labels+ }+ where+ parseDimension input = case Text.breakOn "=" input of+ (name, assignment) | not (Text.null name) && not (Text.null assignment) -> Right (name, Text.drop 1 assignment)+ _ -> Left ("invalid dimension assignment \"" <> input <> "\"")++applyPhaseOverrides :: Maybe PhasePlan -> [Text] -> Either Text (Maybe PhasePlan)+applyPhaseOverrides existing [] = Right existing+applyPhaseOverrides existing assignments = Just <$> foldl step (Right (maybe zeroPhases id existing)) assignments+ where+ step result assignment = do+ plan <- result+ let (name, rawValue) = Text.breakOn "=" assignment+ seconds <- case reads (Text.unpack (Text.drop 1 rawValue)) of+ [(value, "")] | value >= (0 :: Double) -> Right value+ _ -> Left ("invalid phase assignment \"" <> assignment <> "\"")+ case name of+ "warm-up" -> Right plan {warmUpSeconds = seconds}+ "steady" -> Right plan {steadySeconds = seconds}+ "drain" -> Right plan {drainSeconds = seconds}+ _ -> Left ("unknown phase \"" <> name <> "\"")++stdinInputCount :: RunOptions -> Int+stdinInputCount options = length [() | Just InputStdin <- [options.specFile, options.cohortIdentity, options.cellFingerprint]]++readSeed :: String -> Either String Seed+readSeed input = case reads input of [(value, "")] -> first Text.unpack (mkSeed value); _ -> Left "seed must be an integer"++readRunId :: String -> Either String RunId+readRunId = first Text.unpack . parseRunId . Text.pack++readPlacement :: String -> Either String SpecPlacement+readPlacement "local" = Right RunLocal+readPlacement "cell" = Right RunOnCell+readPlacement value = Left ("unknown placement " <> show value)
+ src/Kenshou/Cli/Command/Worker.hs view
@@ -0,0 +1,20 @@+module Kenshou.Cli.Command.Worker (workerCommand) where++import Data.Bifunctor (first)+import Data.Foldable (traverse_)+import Data.Text qualified as Text+import Kenshou.Core.Cli (CliCommand (..), CliEnv (..), CliGroup (..))+import Kenshou.Core.Role (mkRoleName, renderRoleName)+import Kenshou.Core.Role.Dispatch (runWorker)+import Kenshou.Diagnose.Profile (reexecWithWorkerRts)+import Kenshou.Diagnose.Threads (installThreadDumpSignal)+import Options.Applicative++workerCommand :: CliCommand+workerCommand = CliCommand "worker" "Run an internal worker role" Execution True (uncurry handler <$> parser)+ where+ parser = (,) <$> option (eitherReader (first Text.unpack . mkRoleName . Text.pack)) (long "role" <> metavar "ROLE") <*> optional (strOption (long "diagnosis-root" <> metavar "DIR" <> internal))+ handler role diagnosisRoot environment = do+ reexecWithWorkerRts (renderRoleName role)+ traverse_ (\root -> installThreadDumpSignal root (renderRoleName role)) diagnosisRoot+ runWorker environment.registry role
+ src/Kenshou/Cli/Completions.hs view
@@ -0,0 +1,32 @@+module Kenshou.Cli.Completions (completionsCommand) where++import Data.Text (Text)+import Data.Text.IO qualified as Text.IO+import Kenshou.Core.Cli (CliCommand (..), CliGroup (..))+import Options.Applicative+import System.Exit (ExitCode (..))++data Shell = Bash | Zsh | Fish++completionsCommand :: CliCommand+completionsCommand =+ CliCommand+ { name = "completions",+ description = "Generate parser-derived shell completions",+ group = Maintenance,+ hidden = False,+ parser = (\shell _ -> Text.IO.putStr (script shell) >> pure ExitSuccess) <$> shellParser+ }++shellParser :: Parser Shell+shellParser = argument (eitherReader parseShell) (metavar "bash|zsh|fish" <> help "Target shell")+ where+ parseShell "bash" = Right Bash+ parseShell "zsh" = Right Zsh+ parseShell "fish" = Right Fish+ parseShell value = Left ("unsupported shell " <> show value)++script :: Shell -> Text+script Bash = "_kenshou_completions() {\n local CMDLINE=(--bash-completion-index $COMP_CWORD)\n for arg in ${COMP_WORDS[@]}; do CMDLINE+=(--bash-completion-word \"$arg\"); done\n COMPREPLY=( $(kenshou \"${CMDLINE[@]}\" 2>/dev/null) )\n}\ncomplete -o filenames -F _kenshou_completions kenshou\n"+script Zsh = "#compdef kenshou\n_kenshou() {\n local -a completions\n local CMDLINE=(--bash-completion-enriched --bash-completion-index $((CURRENT - 1)))\n for arg in ${words[@]}; do CMDLINE+=(--bash-completion-word \"$arg\"); done\n completions=(\"${(@f)$(kenshou $CMDLINE 2>/dev/null)}\")\n _describe kenshou completions\n}\n_kenshou\n"+script Fish = "complete -c kenshou -f\nfunction __kenshou_complete\n set -l tokens (commandline -cop)\n set -l args --bash-completion-enriched --bash-completion-index (count $tokens)\n for token in $tokens; set args $args --bash-completion-word $token; end\n kenshou $args 2>/dev/null\nend\ncomplete -c kenshou -a '(__kenshou_complete)'\n"
+ src/Kenshou/Cli/Config.hs view
@@ -0,0 +1,54 @@+module Kenshou.Cli.Config+ ( RunDefaults (..),+ runDefaultsConfig,+ runEnvironmentBindings,+ resolveRunDefaults,+ outputRootKey,+ )+where++import Data.Map.Strict qualified as Map+import Data.Text (Text)+import Data.Text qualified as Text+import Kenshou.Core.Cli.Config (ConfigInputs (..), loadConfigSources)+import Settei+import Settei.Env++data RunDefaults = RunDefaults+ { outputRoot :: Text,+ pg17Bin :: Maybe Text,+ pg18Bin :: Maybe Text+ }+ deriving stock (Eq, Show)++runDefaultsConfig :: Config RunDefaults+runDefaultsConfig =+ RunDefaults+ <$> required (publicSetting outputRootKey "Default run output directory" textDecoder)+ <*> optional (publicSetting pg17BinKey "PostgreSQL 17 binary directory" textDecoder)+ <*> optional (publicSetting pg18BinKey "PostgreSQL 18 binary directory" textDecoder)++runEnvironmentBindings :: Bindings+runEnvironmentBindings =+ either+ (error . Text.unpack . renderEnvErrorsText)+ id+ (bindings [binding (EnvName "KENSHOU_PG17_BIN") pg17BinKey, binding (EnvName "KENSHOU_PG18_BIN") pg18BinKey])++resolveRunDefaults :: EnvSnapshot -> ConfigInputs -> IO (Either Text (ResolveResult RunDefaults))+resolveRunDefaults snapshot inputs = do+ loaded <- loadConfigSources inputs+ pure $ do+ fileAndNamed <- loaded+ pure (resolve defaultResolveOptions ([builtIns] <> fileAndNamed <> [environmentSource runEnvironmentBindings snapshot]) runDefaultsConfig)++builtIns :: Source+builtIns = source "kenshou built-ins" BuiltInSource (RawObject (Map.singleton "run" (RawObject (Map.singleton "output-root" (RawText "runs")))))++outputRootKey, pg17BinKey, pg18BinKey :: Key+outputRootKey = validKey "run.output-root"+pg17BinKey = validKey "postgres.pg17-bin"+pg18BinKey = validKey "postgres.pg18-bin"++validKey :: Text -> Key+validKey value = either (error . show) id (parseKey value)
+ src/Kenshou/Cli/Diagnose.hs view
@@ -0,0 +1,254 @@+module Kenshou.Cli.Diagnose+ ( diagnoseCommand,+ )+where++import Control.Exception (SomeException, bracket, displayException, try)+import Data.Aeson+import Data.ByteString.Lazy qualified as LazyByteString+import Data.Foldable (traverse_)+import Data.List (isPrefixOf, isSuffixOf, sort)+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Text.IO qualified as Text.IO+import Data.Time (getCurrentTime)+import Data.Word (Word64)+import Hasql.Connection qualified as Connection+import Hasql.Connection.Settings qualified as Settings+import Kenshou.Core.Cli+import Kenshou.Diagnose.Document+import Kenshou.Diagnose.Leak+import Kenshou.Diagnose.LockGraph (buildGraph, renderDot)+import Kenshou.Diagnose.Postgres+import Kenshou.Diagnose.Profile+import Kenshou.Diagnose.Render+import Kenshou.Diagnose.Stall.Classify (classify)+import Kenshou.Diagnose.Stall.Types+import Kenshou.Diagnose.Threads (dumpThreads)+import Options.Applicative hiding (Success)+import System.Directory (doesDirectoryExist, doesFileExist, listDirectory)+import System.Exit (ExitCode (..))+import System.FilePath (takeFileName, (</>))+import System.IO (stderr)+import System.Process (callProcess)++data DiagnoseOptions = DiagnoseLeak LeakOptions | DiagnoseStall StallOptions | DiagnoseProfile ProfileOptions++data LeakOptions = LeakOptions+ { runDirectory :: !FilePath,+ policy :: !(Maybe InputSource),+ seed :: !Word64,+ output :: !(Maybe FilePath),+ json :: !Bool+ }++data StallOptions = StallOptions+ { runDirectory :: !(Maybe FilePath),+ live :: !Bool,+ connection :: !(Maybe Text),+ reclassify :: !Bool,+ dotOutput :: !(Maybe FilePath),+ output :: !(Maybe FilePath),+ json :: !Bool+ }++data ProfileOptions = ProfileOptions+ { scenario :: !Text,+ modeName :: !Text,+ classes :: !String,+ breakdown :: !Char,+ intervalSeconds :: !Double,+ eventlogMaxBytes :: !Word64,+ worker :: !(Maybe Text),+ outputRoot :: !FilePath,+ knobs :: ![String],+ dimensions :: ![String]+ }++diagnoseCommand :: CliCommand+diagnoseCommand = CliCommand "diagnose" "Diagnose leaks, stalls, and profiles" Analysis False (runDiagnose <$> diagnoseParser)++diagnoseParser :: Parser DiagnoseOptions+diagnoseParser = hsubparser (command "leak" (info (DiagnoseLeak <$> leakParser) (progDesc "Re-judge sampled series for resource leaks")) <> command "stall" (info (DiagnoseStall <$> stallParser) (progDesc "Render or capture a concurrency stall")) <> command "profile" (info (DiagnoseProfile <$> profileParser) (progDesc "Run a scenario under heap or event-log profiling")))++leakParser :: Parser LeakOptions+leakParser =+ LeakOptions+ <$> strArgument (metavar "RUN_DIR")+ <*> optional (parseInputSource <$> strOption (long "policy" <> metavar "FILE" <> help "Leak policy file, or - for stdin"))+ <*> option auto (long "seed" <> metavar "N" <> value 0 <> showDefault)+ <*> optional (strOption (long "out" <> metavar "FILE"))+ <*> switch (long "json")++stallParser :: Parser StallOptions+stallParser =+ StallOptions+ <$> optional (strArgument (metavar "RUN_DIR"))+ <*> switch (long "live" <> help "Capture PostgreSQL and local threads now")+ <*> optional (Text.pack <$> strOption (long "connection" <> metavar "CONNECTION"))+ <*> switch (long "reclassify")+ <*> optional (strOption (long "dot" <> metavar "FILE"))+ <*> optional (strOption (long "out" <> metavar "FILE"))+ <*> switch (long "json")++profileParser :: Parser ProfileOptions+profileParser =+ ProfileOptions+ <$> (Text.pack <$> strArgument (metavar "SCENARIO"))+ <*> (Text.pack <$> strOption (long "mode" <> metavar "closure-type|info-table|eventlog|profiled"))+ <*> strOption (long "classes" <> metavar "CLASSES" <> value "gu" <> showDefault)+ <*> option (eitherReader parseBreakdown) (long "breakdown" <> metavar "c|r|d|y" <> value 'c' <> showDefault)+ <*> option auto (long "interval-s" <> metavar "SECONDS" <> value 10 <> showDefault)+ <*> option auto (long "eventlog-max-bytes" <> metavar "BYTES" <> value 536870912 <> showDefault)+ <*> optional (Text.pack <$> strOption (long "worker" <> metavar "ROLE"))+ <*> strOption (long "out" <> metavar "DIR" <> value ".dev/profiles" <> showDefault)+ <*> many (strOption (long "set" <> metavar "NAME=VALUE"))+ <*> many (strOption (long "dim" <> metavar "NAME=VALUE"))++runDiagnose :: DiagnoseOptions -> CliEnv -> IO ExitCode+runDiagnose (DiagnoseLeak options) _ = runLeak options+runDiagnose (DiagnoseStall options) _ = runStall options+runDiagnose (DiagnoseProfile options) _ = runProfile options++runLeak :: LeakOptions -> IO ExitCode+runLeak options = do+ attempted <- try @SomeException do+ spec <- loadPolicy options.policy+ analyseRunDirectory options.runDirectory spec options.seed+ case attempted of+ Left err -> diagnostic "leak" (Text.pack (displayException err)) >> pure (ExitFailure 4)+ Right (Left err) -> diagnostic "leak" (Text.pack (show err)) >> pure (ExitFailure 4)+ Right (Right report) -> do+ now <- getCurrentTime+ let document = Diagnosis LeakDiagnosis (Text.pack (takeFileName options.runDirectory)) "offline" now (Generator "kenshou-diagnose" "0.1.0.0" "theil-sen+moving-block-bootstrap/1") (toJSON report)+ emit options.json options.output document (renderLeakReport report)+ pure case report.verdict of+ Stable -> ExitSuccess+ LeakSuspected -> ExitFailure 1+ InsufficientData -> ExitFailure 3++loadPolicy :: Maybe InputSource -> IO LeakSpec+loadPolicy Nothing = pure defaultLeakSpec+loadPolicy (Just source) = do+ bytes <- readInputSource source+ either (ioError . userError) pure (eitherDecodeStrict' bytes)++runStall :: StallOptions -> IO ExitCode+runStall options+ | options.live = runLiveStall options+ | otherwise = case options.runDirectory of+ Nothing -> diagnostic "stall" "provide RUN_DIR, or use --live --connection CONNECTION" >> pure (ExitFailure 2)+ Just runDirectory -> do+ attempted <- try @SomeException (loadStallReports runDirectory)+ case attempted of+ Left err -> diagnostic "stall" (Text.pack (displayException err)) >> pure (ExitFailure 4)+ Right reports -> do+ let rendered = fmap (if options.reclassify then reclassified else id) reports+ traverse_ (writeDot options.dotOutput) (take 1 rendered)+ if null rendered+ then Text.IO.putStrLn "no stall diagnoses" >> pure ExitSuccess+ else do+ now <- getCurrentTime+ let body = toJSON rendered+ document = Diagnosis StallDiagnosis (Text.pack (takeFileName runDirectory)) "offline" now (Generator "kenshou-diagnose" "0.1.0.0" "stall-render-v1") body+ emit options.json options.output document (Text.intercalate "\n" (fmap renderStallReport rendered))+ pure (ExitFailure 1)++runLiveStall :: StallOptions -> IO ExitCode+runLiveStall options = case options.connection of+ Nothing -> diagnostic "stall" "--live requires --connection CONNECTION" >> pure (ExitFailure 2)+ Just connectionString -> do+ attempted <- try @SomeException $ bracket (acquire connectionString) Connection.release \connection -> do+ postgres <- capturePostgres connection 2+ threads <- dumpThreads False+ now <- getCurrentTime+ let graph = buildGraph postgres.activity postgres.locks+ snapshot = StallSnapshot 0 [] threads (Just postgres) graph [] Null (SpinEvidence 0 (maybe 0 (.callsPerSecond) postgres.statementRate))+ (primary, secondary, reasons) = classify snapshot+ pure (StallReport now 1 primary secondary reasons snapshot)+ case attempted of+ Left err -> diagnostic "stall" (Text.pack (displayException err)) >> pure (ExitFailure 4)+ Right report -> do+ writeDot options.dotOutput report+ let document = Diagnosis StallDiagnosis "live" "live" report.detectedAt (Generator "kenshou-diagnose" "0.1.0.0" "live-stall-v1") (toJSON report)+ emit options.json options.output document (renderStallReport report)+ pure if report.classification == Unknown then ExitSuccess else ExitFailure 1++acquire :: Text -> IO Connection.Connection+acquire connectionString = Connection.acquire (Settings.connectionString connectionString <> Settings.applicationName "kenshou-diagnose-live") >>= either (ioError . userError . show) pure++loadStallReports :: FilePath -> IO [StallReport]+loadStallReports runDirectory = do+ exists <- doesDirectoryExist runDirectory+ if not exists then ioError (userError ("run directory does not exist: " <> runDirectory)) else pure ()+ let diagnosisDirectory = runDirectory </> "diagnosis"+ diagnosisExists <- doesDirectoryExist diagnosisDirectory+ if not diagnosisExists+ then pure []+ else do+ names <- sort <$> listDirectory diagnosisDirectory+ traverse (loadOne . (diagnosisDirectory </>)) [name | name <- names, "stall-" `isPrefixOf` name, ".json" `isSuffixOf` name]+ where+ loadOne path = do+ decoded <- eitherDecodeFileStrict' path :: IO (Either String Diagnosis)+ document <- either (ioError . userError) pure decoded+ case fromJSON document.body of+ Error err -> ioError (userError (path <> ": " <> err))+ Success report -> pure report++reclassified :: StallReport -> StallReport+reclassified report =+ let (primary, secondary, reasons) = classify report.snapshot+ in StallReport report.detectedAt report.captureNumber primary secondary reasons report.snapshot++writeDot :: Maybe FilePath -> StallReport -> IO ()+writeDot Nothing _ = pure ()+writeDot (Just path) report = Text.IO.writeFile path (renderDot report.snapshot.graph)++runProfile :: ProfileOptions -> IO ExitCode+runProfile options = case parseMode options of+ Left message -> diagnostic "profile" message >> pure (ExitFailure 2)+ Right mode -> do+ let arguments = concatMap (\assignment -> ["--set", assignment]) options.knobs <> concatMap (\assignment -> ["--dim", assignment]) options.dimensions+ request = ProfileRequest mode options.scenario arguments options.outputRoot options.intervalSeconds options.eventlogMaxBytes options.worker+ attempted <- try @SomeException (runProfileSession request >>= postprocessProfile)+ case attempted of+ Left err -> diagnostic "profile" (Text.pack (displayException err)) >> pure (ExitFailure 4)+ Right report -> Text.IO.putStr (renderProfileReport report) >> pure (toExitCode report.exitCode)++postprocessProfile :: ProfileReport -> IO ProfileReport+postprocessProfile report = do+ let tool = ".dev/bin/eventlog2html"+ eventlog = report.sessionDirectory </> "kenshou.eventlog"+ available <- doesFileExist tool+ eventlogExists <- doesFileExist eventlog+ if available && eventlogExists then callProcess tool [eventlog] else pure ()+ pure report++parseMode :: ProfileOptions -> Either Text ProfileMode+parseMode options = case options.modeName of+ "closure-type" -> Right ClosureType+ "info-table" -> Right InfoTable+ "eventlog" -> Right (Eventlog options.classes)+ "profiled" -> Right (Profiled options.breakdown)+ other -> Left ("unknown profile mode " <> other)++parseBreakdown :: String -> Either String Char+parseBreakdown [breakdown] | breakdown `elem` ("crdy" :: String) = Right breakdown+parseBreakdown _ = Left "breakdown must be c, r, d, or y"++emit :: (ToJSON value) => Bool -> Maybe FilePath -> value -> Text -> IO ()+emit json output document rendered = do+ let bytes = encode document <> "\n"+ traverse_ (\path -> LazyByteString.writeFile path bytes) output+ case (json, output) of+ (True, Nothing) -> LazyByteString.putStr bytes+ (True, Just _) -> pure ()+ (False, _) -> Text.IO.putStr rendered++diagnostic :: Text -> Text -> IO ()+diagnostic commandName message = Text.IO.hPutStrLn stderr ("kenshou diagnose " <> commandName <> ": " <> message)++toExitCode :: Int -> ExitCode+toExitCode 0 = ExitSuccess+toExitCode code = ExitFailure code
+ src/Kenshou/Cli/Help.hs view
@@ -0,0 +1,55 @@+{-# LANGUAGE TemplateHaskell #-}++module Kenshou.Cli.Help (helpCommand, topics) where++import Data.FileEmbed (embedStringFile)+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Text.IO qualified as Text.IO+import Kenshou.Cli.Help.Width (renderForWidth, resolveWidth)+import Kenshou.Core.Cli (CliCommand (..), CliGroup (..), HelpTopic (..))+import Options.Applicative+import System.Exit (ExitCode (..))+import System.IO (stderr)++topics :: [HelpTopic]+topics =+ [ HelpTopic "scenarios" "Scenario identifiers and metadata" (Text.pack $(embedStringFile "help/scenarios.md")),+ HelpTopic "selectors" "Selecting groups of scenarios" (Text.pack $(embedStringFile "help/selectors.md")),+ HelpTopic "run-specs" "Versioned run specification documents" (Text.pack $(embedStringFile "help/run-specs.md")),+ HelpTopic "planning" "Change-aware plans, suites, and safe resume" (Text.pack $(embedStringFile "help/planning.md")),+ HelpTopic "outcomes" "Run outcomes and known defects" (Text.pack $(embedStringFile "help/outcomes.md")),+ HelpTopic "exit-codes" "Stable process exit codes" (Text.pack $(embedStringFile "help/exit-codes.md")),+ HelpTopic "comparisons" "Paired benchmark comparisons" (Text.pack $(embedStringFile "help/comparisons.md")),+ HelpTopic "diagnostics" "Leak, stall, and profiling diagnosis" (Text.pack $(embedStringFile "help/diagnostics.md")),+ HelpTopic "evidence" "Durable run records and data links" (Text.pack $(embedStringFile "help/evidence.md")),+ HelpTopic "cells" "Leased cell execution and result verification" (Text.pack $(embedStringFile "help/cells.md"))+ ]++data HelpOptions = HelpOptions {topic :: Maybe Text, width :: Maybe Int}++helpCommand :: CliCommand+helpCommand =+ CliCommand+ { name = "help",+ description = "Read a long-form help topic",+ group = Discovery,+ hidden = False,+ parser = runHelp <$> helpParser+ }++helpParser :: Parser HelpOptions+helpParser =+ HelpOptions+ <$> optional (Text.pack <$> strArgument (metavar "TOPIC" <> help "Topic name"))+ <*> optional (option auto (long "width" <> metavar "COLUMNS" <> help "Wrap prose to this width"))++runHelp :: HelpOptions -> environment -> IO ExitCode+runHelp options _ = case options.topic of+ Nothing -> Text.IO.putStr (Text.unlines [topic.name <> " " <> topic.description | topic <- topics]) >> pure ExitSuccess+ Just wanted -> case filter ((== Text.toCaseFold wanted) . Text.toCaseFold . (.name)) topics of+ [] -> Text.IO.hPutStrLn stderr ("kenshou: unknown help topic \"" <> wanted <> "\"") >> pure (ExitFailure 2)+ topic : _ -> do+ width <- resolveWidth options.width+ Text.IO.putStr (renderForWidth width topic.content)+ pure ExitSuccess
+ src/Kenshou/Cli/Help/Width.hs view
@@ -0,0 +1,40 @@+module Kenshou.Cli.Help.Width (renderForWidth, resolveWidth) where++import Data.Text (Text)+import Data.Text qualified as Text+import System.Console.Terminal.Size qualified as Terminal+import System.IO (hIsTerminalDevice, stdout)++resolveWidth :: Maybe Int -> IO (Maybe Int)+resolveWidth (Just width) = pure (Just (max 1 width))+resolveWidth Nothing = do+ terminal <- hIsTerminalDevice stdout+ if not terminal+ then pure Nothing+ else do+ window <- Terminal.size+ pure (min 140 . Terminal.width <$> window)++renderForWidth :: Maybe Int -> Text -> Text+renderForWidth Nothing body = body+renderForWidth (Just width) body = Text.intercalate "\n\n" (fmap renderParagraph (splitParagraphs body))+ where+ renderParagraph paragraph+ | all (Text.isPrefixOf " ") (filter (not . Text.null . Text.strip) (Text.lines paragraph)) = paragraph+ | otherwise = Text.intercalate "\n" (packWords (Text.words paragraph))+ packWords [] = []+ packWords (firstWord : rest) = go firstWord rest+ go current [] = [current]+ go current (word : rest)+ | Text.length current + 1 + Text.length word <= width = go (current <> " " <> word) rest+ | otherwise = current : go word rest++splitParagraphs :: Text -> [Text]+splitParagraphs body = go [] [] (Text.lines body)+ where+ go result current [] = reverse (flush result current)+ go result current (line : rest)+ | Text.null (Text.strip line) = go (flush result current) [] rest+ | otherwise = go result (line : current) rest+ flush result [] = result+ flush result current = Text.intercalate "\n" (reverse current) : result
+ src/Kenshou/Cli/Main.hs view
@@ -0,0 +1,17 @@+module Kenshou.Cli.Main (mainWith, runWithArgsWith) where++import Data.Text.IO qualified as Text.IO+import Kenshou.Cli.Registry (CliAssembly)+import Kenshou.Cli.Version (appVersionWithGit)+import Kenshou.Core.Cli (runCli)+import Kenshou.Diagnose.Profile.EventlogGuard (withEventlogGuard)+import Kenshou.Diagnose.Profile.GhcDebug (withGhcDebugIfRequested)+import System.Environment (getArgs)+import System.Exit (ExitCode (..), exitWith)++mainWith :: CliAssembly -> IO ()+mainWith assembly = withGhcDebugIfRequested (withEventlogGuard (getArgs >>= runWithArgsWith assembly >>= exitWith))++runWithArgsWith :: CliAssembly -> [String] -> IO ExitCode+runWithArgsWith _ ["--version"] = Text.IO.putStrLn appVersionWithGit >> pure ExitSuccess+runWithArgsWith assembly arguments = runCli assembly arguments
+ src/Kenshou/Cli/Options.hs view
@@ -0,0 +1,76 @@+module Kenshou.Cli.Options+ ( Command (..),+ CohortCommand (..),+ commandParserInfo,+ cohortCommandParser,+ )+where++import Data.Text qualified as Text+import Kenshou.Cli.Version (appVersionWithGit)+import Options.Applicative+ ( Parser,+ ParserInfo,+ command,+ fullDesc,+ header,+ help,+ helper,+ info,+ infoOption,+ long,+ metavar,+ optional,+ progDesc,+ strOption,+ subparser,+ switch,+ value,+ (<**>),+ )++data Command = CohortCommand CohortCommand+ deriving stock (Eq, Show)++data CohortCommand+ = CohortShow Bool FilePath (Maybe FilePath) (Maybe FilePath)+ | CohortCheck FilePath (Maybe FilePath) (Maybe FilePath) (Maybe FilePath)+ deriving stock (Eq, Show)++commandParserInfo :: ParserInfo Command+commandParserInfo =+ info+ (commandParser <**> helper <**> versionOption)+ (fullDesc <> header "kenshou - run verification scenarios")+ where+ versionOption = infoOption (Text.unpack appVersionWithGit) (long "version" <> help "Show version")++commandParser :: Parser Command+commandParser = CohortCommand <$> subparser (command "cohort" (info cohortCommandParser (progDesc "Inspect the pinned runtime cohort")))++cohortCommandParser :: Parser CohortCommand+cohortCommandParser =+ subparser+ ( command "show" (info showParser (progDesc "Print the cohort resolved by cabal"))+ <> command "check" (info checkParser (progDesc "Check the resolved cohort against its descriptor"))+ )+ where+ showParser =+ CohortShow+ <$> switch (long "json" <> help "Write kenshou.cohort-identity/v1 JSON")+ <*> projectDirOption+ <*> optional planJsonOption+ <*> optional identityOption+ checkParser = CohortCheck <$> projectDirOption <*> optional planJsonOption <*> optional descriptorOption <*> optional identityOption++projectDirOption :: Parser FilePath+projectDirOption = strOption (long "project-dir" <> metavar "DIR" <> value "." <> help "Cabal project directory")++planJsonOption :: Parser FilePath+planJsonOption = strOption (long "plan-json" <> metavar "FILE" <> help "Override dist-newstyle/cache/plan.json")++identityOption :: Parser FilePath+identityOption = strOption (long "identity" <> metavar "FILE" <> help "Read a captured cohort identity instead of a cabal plan")++descriptorOption :: Parser FilePath+descriptorOption = strOption (long "descriptor" <> metavar "FILE" <> help "Override the active cohort descriptor")
+ src/Kenshou/Cli/Registry.hs view
@@ -0,0 +1,62 @@+module Kenshou.Cli.Registry+ ( CliAssembly,+ Program (..),+ selfTestBundles,+ standardCommands,+ standardTopics,+ selfTestAssembly,+ )+where++import Kenshou.Check.Selftest qualified as CheckSelftest+import Kenshou.Cli.Attest (measurementRecomputer, pairedComparisonRecomputer, runOutcomeRecomputer)+import Kenshou.Cli.Command.Cell (cellCommand)+import Kenshou.Cli.Command.Cohort (cohortCommand)+import Kenshou.Cli.Command.Execute (executeCommand)+import Kenshou.Cli.Command.List (listCommand)+import Kenshou.Cli.Command.Overhead (overheadCommand)+import Kenshou.Cli.Command.Perf (perfCommand)+import Kenshou.Cli.Command.Plan (planCommand)+import Kenshou.Cli.Command.Run (runCommand)+import Kenshou.Cli.Command.Worker (workerCommand)+import Kenshou.Cli.Completions (completionsCommand)+import Kenshou.Cli.Diagnose qualified as Diagnose+import Kenshou.Cli.Help qualified as Help+import Kenshou.Core.Bundle (LayerBundle)+import Kenshou.Core.Cli (CliCommand, HelpTopic, Program (..), defaultProgram)+import Kenshou.Core.Selftest qualified as Selftest+import Kenshou.Diagnose.SelfTest qualified as DiagnoseSelftest+import Kenshou.Evidence.Attest (Recomputer)+import Kenshou.Evidence.Cli (attestCommandWith, evidenceCommand, recordCommand)+import Kenshou.Evidence.HistoryCli (historyCommand)+import Kenshou.Measure.Cli qualified as Measure+import Kenshou.Measure.Selftest qualified as MeasureSelftest+import Kenshou.Remote.Selftest qualified as RemoteSelftest+import Kenshou.Telemetry.SelfTest qualified as TelemetrySelftest++-- | Everything an executable registers: its name and header, the layer bundles+-- whose scenarios and roles it runs, the operator commands it exposes, the help+-- topics, its component graph, and its performance contracts.+-- Extension contract: a suite adds its bundles to 'bundles', and its own+-- recomputers through 'standardCommands', without changing this library.+type CliAssembly = Program++-- | The self-test bundles the kernel and toolkits define for themselves.+selfTestBundles :: [LayerBundle]+selfTestBundles = [Selftest.bundle, MeasureSelftest.bundle, CheckSelftest.bundle, DiagnoseSelftest.bundle, TelemetrySelftest.selfTestBundle, RemoteSelftest.bundle]++-- | The generic operator commands, with @attest@ using the given recomputers.+standardCommands :: [Recomputer] -> [CliCommand]+standardCommands recomputers = [listCommand, planCommand, Help.helpCommand, runCommand, executeCommand, cellCommand, perfCommand, overheadCommand, Measure.summarizeCommand, Measure.compareCommand, Diagnose.diagnoseCommand, historyCommand, recordCommand, attestCommandWith recomputers, evidenceCommand, workerCommand, cohortCommand, completionsCommand]++standardTopics :: [HelpTopic]+standardTopics = Help.topics++-- | The self-test bundles with the standard commands and the generic recomputers.+selfTestAssembly :: CliAssembly+selfTestAssembly =+ defaultProgram+ { bundles = selfTestBundles,+ commands = standardCommands [runOutcomeRecomputer, measurementRecomputer, pairedComparisonRecomputer],+ topics = standardTopics+ }
+ src/Kenshou/Cli/Version.hs view
@@ -0,0 +1,36 @@+{-# LANGUAGE CPP #-}+{-# LANGUAGE TemplateHaskell #-}++module Kenshou.Cli.Version+ ( appVersion,+ appVersionWithGit,+ gitCommitShort,+ )+where++import Data.Text (Text)+import Data.Text qualified as Text+import Data.Version (showVersion)+import GitHash (GitInfo, giHash, tGitInfoCwdTry)+import Paths_kenshou_cli (version)++appVersion :: Text+appVersion = Text.pack (showVersion version)++gitInfo :: Either String GitInfo+gitInfo = $$tGitInfoCwdTry++nixGitHash :: Maybe Text+#ifdef GIT_HASH+nixGitHash = Just GIT_HASH+#else+nixGitHash = Nothing+#endif++gitCommitShort :: Maybe Text+gitCommitShort = case gitInfo of+ Right info -> Just (Text.pack (take 7 (giHash info)))+ Left _ -> nixGitHash++appVersionWithGit :: Text+appVersionWithGit = "kenshou v" <> appVersion <> maybe "" (\commit -> " (" <> commit <> ")") gitCommitShort
+ test/Main.hs view
@@ -0,0 +1,801 @@+module Main (main) where++import Control.Concurrent (forkIO, killThread, threadDelay)+import Control.Exception (bracket, finally)+import Control.Monad (forM_)+import Data.Aeson (Value, object, (.=))+import Data.Aeson qualified as Aeson+import Data.Aeson.Key qualified as Key+import Data.Aeson.KeyMap qualified as KeyMap+import Data.ByteString qualified as ByteString+import Data.ByteString.Lazy qualified as LazyByteString+import Data.Foldable (toList)+import Data.List (find)+import Data.List qualified as List+import Data.Set qualified as Set+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Text.IO qualified as Text+import Data.Time (addUTCTime, getCurrentTime)+import Kenshou.Cli (Program (..), runWithArgsWith, selfTestAssembly)+import Kenshou.Cli.Cohort (resolveDefaultCohortIdentity)+import Kenshou.Cli.Version (appVersionWithGit)+import Kenshou.Core.Canonical (sha256Hex)+import Kenshou.Core.Cohort (CohortIdentity (..), CohortName (..))+import Kenshou.Core.Id (newRunId, renderRunId)+import Kenshou.Core.Manifest (Manifest (..), ManifestFile (..))+import Kenshou.Core.Outcome qualified as Outcome+import Kenshou.Perf.Artifacts (sealPerfDirectory)+import Kenshou.Perf.Plan qualified as Perf+import Kenshou.Perf.Prepared qualified as Prepared+import Kenshou.Remote.Cell.Docs (Artifact (..), CellBuckets (..), CellDescriptor (..), CellManifest (..), CellOutcome (..), CellPhase (..), CellRunResult (..), CellStatus (..), Limits (..), LogChunks (..), ManifestPayload (..), Rejected (..), Submission (..), WorkObject (..))+import Kenshou.Remote.Cell.Lease (Lease (..))+import Kenshou.Remote.Cell.RouteRules (descriptorDigest)+import Kenshou.Remote.Cell.Session.Journal (LeaseMode (..), SessionJournal (..), SliceJournal (..), SliceState (..), readSessionJournal, writeSessionJournal)+import Kenshou.Remote.Cell.Submit (workObjectFor)+import Kenshou.Remote.Payload (Bundle (..), CellPayload (..), PayloadDescriptor (..))+import Kenshou.Remote.Payload qualified as Payload+import Kenshou.Remote.Store (Bucket (..), ObjectName (..), ObjectStore (..), Precondition (..))+import Kenshou.Remote.Store.File (newFileStore)+import Kenshou.Telemetry.Overhead (OverheadState (..), SlotRun (..), loadOverheadState)+import PerfSessionSpec qualified+import System.Directory (Permissions (..), copyFile, createDirectoryIfMissing, doesFileExist, doesPathExist, getPermissions, listDirectory, makeAbsolute, setPermissions, withCurrentDirectory)+import System.Environment (lookupEnv, setEnv, unsetEnv)+import System.Exit (ExitCode (..))+import System.FilePath ((</>))+import System.IO.Temp (withSystemTempDirectory)+import System.Timeout (timeout)+import Test.Hspec (describe, expectationFailure, hspec, it, shouldBe, shouldReturn, shouldSatisfy)++main :: IO ()+main = hspec do+ PerfSessionSpec.spec+ describe "performance planning CLI" do+ it "exposes calibration publication and rejects malformed batches before storage effects" $ withSystemTempDirectory "kenshou-calibration-malformed" \directory -> do+ let input = directory </> "definition.json"+ Aeson.encodeFile input Aeson.Null+ runWithArgs ["perf", "calibration", "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["perf", "calibration", "declare", "--definition", input, "--uri", "gs://fixture-evidence/calibration/definition.json"] `shouldReturnCode` ExitFailure 2+ it "publishes a calibration definition create-only and retrieves its identical bytes" $ withSystemTempDirectory "kenshou-calibration-publication" \directory -> do+ raw <- withPerfRoot (Aeson.eitherDecodeFileStrict' "kenshou-core/test/golden/perf-calibration-definition.synthetic.json" >>= either fail pure)+ now <- getCurrentTime+ let input = directory </> "definition.json"+ storeRoot = directory </> "store"+ document = replacePerfField "expiresAt" (Aeson.toJSON (addUTCTime 3600 now)) raw+ uri = "gs://fixture-evidence/calibration/definition.json"+ Aeson.encodeFile input document+ store <- newFileStore storeRoot+ withCellStore storeRoot do+ runWithArgs ["perf", "calibration", "declare", "--definition", input, "--uri", uri] `shouldReturnCode` ExitSuccess+ bytes <- LazyByteString.readFile input+ retained <- store.getObject (Bucket "fixture-evidence") (ObjectName "calibration/definition.json")+ fmap fst retained `shouldBe` Just bytes+ runWithArgs ["perf", "calibration", "declare", "--definition", input, "--uri", uri] `shouldReturnCode` ExitSuccess+ Aeson.encodeFile input (replacePerfField "injectionMechanism" (Aeson.String "changed after declaration") document)+ runWithArgs ["perf", "calibration", "declare", "--definition", input, "--uri", uri] `shouldReturnCode` ExitFailure 4+ store.getObject (Bucket "fixture-evidence") (ObjectName "calibration/definition.json") >>= \current -> fmap fst current `shouldBe` Just bytes+ it "exposes persistent execution, status, resume and evidence replay" do+ forM_ ["run", "status", "resume", "verify"] \name -> do+ runWithArgs ["perf", name, "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["perf", name] `shouldReturnCode` ExitFailure 2+ it "rejects malformed prepared input before creating a session or contacting a cell" $ withSystemTempDirectory "kenshou-perf-run-malformed" \directory -> do+ let input = directory </> "prepared.json"+ output = directory </> "session"+ Aeson.encodeFile input Aeson.Null+ runWithArgs ["perf", "run", "--prepared", input, "--out", output, "--evidence-base-uri", "gs://fixture-evidence/performance"] `shouldReturnCode` ExitFailure 2+ doesPathExist output `shouldReturn` False+ it "reads retained status without cloud authentication or execution" $ withSystemTempDirectory "kenshou-perf-status" \directory -> do+ withPerfRoot (copyFile "fixtures/consumer-baseline/kenshou-remote/test/golden/perf-session.fixture.json" (directory </> "session.json"))+ runWithArgs ["perf", "status", "--session", directory] `shouldReturnCode` ExitSuccess+ it "publishes and replays incomplete scope without leasing, and detects storage tampering" $ withSystemTempDirectory "kenshou-perf-incomplete" \directory -> do+ let storeRoot = directory </> "store"+ preparation = directory </> "preparation"+ output = directory </> "session"+ bytes = "synthetic payload; this control executes no runtime work"+ digest = Text.drop 7 (sha256Hex (LazyByteString.toStrict bytes))+ bundle = Bundle ("gs://control/payloads/sha256/" <> digest <> ".nar.zst") digest (LazyByteString.length bytes)+ changePayload arm = case Aeson.fromJSON arm.payload of+ Aeson.Error problem -> fail problem+ Aeson.Success payload -> pure arm {Prepared.payload = Aeson.toJSON (payload {Payload.cell = payload.cell {Payload.bundle = bundle}} :: PayloadDescriptor)}+ original <- withPerfRoot (Aeson.eitherDecodeFileStrict' "fixtures/consumer-baseline/kenshou-core/test/golden/perf-prepared.fixture.json" >>= either fail pure)+ baseline <- changePayload (original :: Prepared.PerfPrepared).baseline+ candidate <- changePayload original.candidate+ let prepared = original {Prepared.plan = original.plan {Perf.selected = [], Perf.coverage = "incomplete", Perf.estimateSeconds = 0}, Prepared.baseline = baseline, Prepared.candidate = candidate}+ createDirectoryIfMissing True preparation+ Aeson.encodeFile (preparation </> "prepared.json") prepared+ sealPerfDirectory preparation >>= either (fail . Text.unpack) pure+ descriptor <- withPerfRoot (LazyByteString.readFile "kenshou-remote/test/golden/cell/cell.descriptor.v1.json")+ store <- newFileStore storeRoot+ _ <- store.putObject (Bucket "control") (ObjectName ("payloads/sha256/" <> digest <> ".nar.zst")) "application/octet-stream" DoesNotExist bytes+ _ <- store.putObject (Bucket "tan-nb-exp-cells-control") (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist descriptor+ withCellStore storeRoot $ withCapabilityDir (directory </> "capabilities") do+ runWithArgs ["perf", "run", "--prepared", preparation </> "prepared.json", "--out", output, "--evidence-base-uri", "gs://fixture-evidence/performance"] `shouldReturnCode` ExitFailure 3+ store.getObject (Bucket "tan-nb-exp-cells-control") (ObjectName "cells/alpha/lease.json") `shouldReturn` Nothing+ runWithArgs ["perf", "verify", "--session", output] `shouldReturnCode` ExitFailure 3+ runWithArgs ["perf", "verify", "--session", output, "--offline"] `shouldReturnCode` ExitFailure 3+ originalResult <- LazyByteString.readFile (output </> "result.json")+ runWithArgs ["perf", "resume", "--session", output] `shouldReturnCode` ExitFailure 3+ LazyByteString.readFile (output </> "result.json") `shouldReturn` originalResult+ published <- store.listObjects (Bucket "fixture-evidence") "performance/sessions/"+ let reports = [name | (name, _) <- published, "/report.md" `Text.isSuffixOf` name.unObjectName]+ length reports `shouldBe` 1+ forM_ reports \name -> do+ _ <- store.putObject (Bucket "fixture-evidence") name "text/markdown" NoPrecondition "tampered"+ pure ()+ runWithArgs ["perf", "verify", "--session", output] `shouldReturnCode` ExitFailure 4+ it "exposes the planning command and requires request/output arguments" do+ runWithArgs ["perf", "plan", "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["perf", "plan"] `shouldReturnCode` ExitFailure 2+ it "exposes exact-revision preparation and rejects malformed requests without building" do+ runWithArgs ["perf", "prepare", "--help"] `shouldReturnCode` ExitSuccess+ withSystemTempDirectory "kenshou-perf-prepare-malformed" \directory -> do+ let input = directory </> "request.json"+ output = directory </> "preparation"+ Aeson.encodeFile input Aeson.Null+ runWithArgs ["perf", "prepare", "--request", input, "--out", output] `shouldReturnCode` ExitFailure 2+ doesPathExist output `shouldReturn` False+ it "rejects malformed requests before creating an output directory" do+ withSystemTempDirectory "kenshou-perf-malformed" \directory -> do+ let input = directory </> "request.json"+ output = directory </> "plan"+ Aeson.encodeFile input Aeson.Null+ runWithArgs ["perf", "plan", "--request", input, "--out", output] `shouldReturnCode` ExitFailure 2+ doesPathExist output `shouldReturn` False+ it "rejects target components from a second repository before fetching" do+ withPerfRoot $ withSystemTempDirectory "kenshou-perf-targets" \directory -> do+ raw <- Aeson.eitherDecodeFileStrict' "fixtures/consumer-baseline/kenshou-core/test/golden/perf-request.json" >>= either fail pure+ let input = directory </> "request.json"+ output = directory </> "plan"+ targets <- foreignTargets raw+ Aeson.encodeFile input (replacePerfField "targetComponents" (Aeson.toJSON targets) raw)+ withConsumerBaseline ["perf", "plan", "--request", input, "--out", output] `shouldReturnCode` ExitFailure 2+ doesPathExist output `shouldReturn` False+ it "rejects unknown profiles before fetching immutable sources" do+ withPerfRoot $ withSystemTempDirectory "kenshou-perf-profiles" \directory -> do+ raw <- Aeson.eitherDecodeFileStrict' "fixtures/consumer-baseline/kenshou-core/test/golden/perf-request.json" >>= either fail pure+ let input = directory </> "request.json"+ output = directory </> "plan"+ Aeson.encodeFile input (replacePerfField "profiles" (Aeson.toJSON (["missing"] :: [Text])) raw)+ withConsumerBaseline ["perf", "plan", "--request", input, "--out", output] `shouldReturnCode` ExitFailure 2+ doesPathExist output `shouldReturn` False+ it "refuses to overwrite an existing planning directory" do+ withSystemTempDirectory "kenshou-perf-existing" \directory -> do+ runWithArgs ["perf", "plan", "--request", "missing", "--out", directory] `shouldReturnCode` ExitFailure 2+ describe "CLI exit contract" do+ it "resolves the wrapped cohort identity outside a source checkout" $+ withSystemTempDirectory "kenshou-wrapped-cohort" \root -> do+ fixture <- makeAbsolute "../kenshou-core/test/fixtures/cohort-identity.golden.json"+ withEnvVariable "KENSHOU_COHORT_IDENTITY" fixture $+ withCurrentDirectory root do+ resolved <- resolveDefaultCohortIdentity+ case resolved of+ Left problem -> expectationFailure (show problem)+ Right identity -> identity.identityCohort `shouldBe` CohortName "released"++ it "returns success for help" do+ runWithArgs ["--help"] `shouldReturnCode` ExitSuccess++ it "rejects a pinned knob absent from every selected scenario" $+ withSystemTempDirectory "kenshou-plan-pin" \root -> do+ let output = root </> "run-plan.json"+ runWithArgs+ [ "plan",+ "--all",+ "--select",+ "selftest/kernel/correctness/always-pass",+ "--set",+ "diagnose.major-gc-interval-ms=5000",+ "--out",+ output+ ]+ `shouldReturnCode` ExitFailure 2+ doesFileExist output `shouldReturn` False++ it "returns success for command-specific help" do+ runWithArgs ["record", "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["attest", "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["history", "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["run", "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["cell", "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["cell", "fetch", "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["cell", "verify", "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["cell", "status", "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["cell", "lease", "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["cell", "release", "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["cell", "watch", "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["cell", "route", "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["cell", "submit", "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["cell", "run", "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["cell", "probe", "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["cell", "pair", "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["cell", "overhead", "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["cell", "resume", "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["cell", "payload", "publish", "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["cell", "payload", "show", "--help"] `shouldReturnCode` ExitSuccess+ runWithArgs ["help", "cells"] `shouldReturnCode` ExitSuccess++ it "shows a payload only when its bundle exists at the recorded size" $+ withSystemTempDirectory "kenshou-cell-payload" \root -> do+ store <- newFileStore root+ let descriptor = "../kenshou-remote/test/golden/payload.json"+ bundle = ObjectName "payloads/sha256/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.nar.zst"+ showPayload = ["cell", "payload", "show", descriptor]+ withCellStore root do+ runWithArgs showPayload `shouldReturnCode` ExitFailure 1+ _ <- store.putObject (Bucket "control") bundle "application/octet-stream" DoesNotExist "xx"+ runWithArgs showPayload `shouldReturnCode` ExitFailure 1+ _ <- store.putObject (Bucket "control") bundle "application/octet-stream" NoPrecondition "x"+ runWithArgs showPayload `shouldReturnCode` ExitSuccess++ it "rejects malformed cell result identifiers and URIs" do+ runWithArgs ["cell", "fetch", "--results-bucket", "test-results", "not-a-run-id", "--out", "test-output"] `shouldReturnCode` ExitFailure 2+ runWithArgs ["cell", "verify", "gs://test-results/runs/not-a-run-id"] `shouldReturnCode` ExitFailure 2++ it "distinguishes an unavailable cell tree from invalid evidence" do+ runWithArgs ["cell", "verify", "test/fixtures/does-not-exist"] `shouldReturnCode` ExitFailure 4+ runWithArgs ["cell", "verify", leakingRun] `shouldReturnCode` ExitFailure 1++ it "acquires, reports and releases a lease through the file-backed cell CLI" $+ withSystemTempDirectory "kenshou-cell-cli" \root -> do+ store <- newFileStore root+ descriptor <- LazyByteString.readFile "../kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+ let control = Bucket "tan-nb-exp-cells-control"+ leaseObject = ObjectName "cells/alpha/lease.json"+ common = ["--cell", "alpha", "--control-bucket", "tan-nb-exp-cells-control"]+ _ <- store.putObject control (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist descriptor+ withCellStore root do+ runWithArgs (["cell", "status"] <> common <> ["--json"]) `shouldReturnCode` ExitSuccess+ runWithArgs (["cell", "lease"] <> common <> ["--purpose", "cli-test"]) `shouldReturnCode` ExitSuccess+ stored <- store.getObject control leaseObject+ record <- case stored of+ Just (bytes, _) -> case Aeson.eitherDecode bytes of+ Right lease -> pure (lease :: Lease)+ Left problem -> expectationFailure problem >> error "unreachable"+ Nothing -> expectationFailure "lease was not published" >> error "unreachable"+ runWithArgs (["cell", "status"] <> common) `shouldReturnCode` ExitSuccess+ runWithArgs (["cell", "release"] <> common <> ["--lease-id", Text.unpack (renderRunId record.leaseId)]) `shouldReturnCode` ExitSuccess+ store.statObject control leaseObject `shouldReturn` Nothing++ it "routes a public plan through file-backed cell descriptors" $+ withSystemTempDirectory "kenshou-cell-route" \root -> do+ store <- newFileStore root+ descriptor <- LazyByteString.readFile "../kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+ observed <- either fail pure (Aeson.eitherDecode descriptor :: Either String CellDescriptor)+ let control = Bucket "tan-nb-exp-cells-control"+ outDir = root </> "routed"+ common = ["cell", "route", "--cell", "beta", "--cell", "alpha", "--control-bucket", "tan-nb-exp-cells-control", "--plan", "../kenshou-core/test/golden/run-plan.minimal.json"]+ _ <- store.putObject control (ObjectName "cells/beta/descriptor.json") "application/json" DoesNotExist (Aeson.encode (observed {name = "beta", postgresMajor = 17}))+ _ <- store.putObject control (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist descriptor+ withCellStore root do+ runWithArgs (common <> ["--out", outDir, "--coerce-durable"]) `shouldReturnCode` ExitSuccess+ doesFileExist (outDir </> "plan.alpha.json") `shouldReturn` True+ doesFileExist (outDir </> "plan.beta.json") `shouldReturn` False+ report <- Aeson.eitherDecodeFileStrict' (outDir </> "unroutable.json") >>= either fail pure+ case report of+ Aeson.Object fields -> KeyMap.lookup "schema" fields `shouldBe` Just (Aeson.String "kenshou.cell-route/v1")+ _ -> expectationFailure "expected route report"+ runWithArgs (common <> ["--out", outDir, "--coerce-durable"]) `shouldReturnCode` ExitFailure 2+ runWithArgs (common <> ["--out", root </> "refused"]) `shouldReturnCode` ExitFailure 2+ doesFileExist (root </> "refused" </> "unroutable.json") `shouldReturn` True++ it "uses a matching capability cache and ignores one from an old descriptor" $+ withSystemTempDirectory "kenshou-cell-capability-route" \root -> do+ store <- newFileStore root+ descriptorBytes <- LazyByteString.readFile "../kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+ descriptor <- either fail pure (Aeson.eitherDecode descriptorBytes :: Either String CellDescriptor)+ probeId <- newRunId+ now <- getCurrentTime+ let control = Bucket "tan-nb-exp-cells-control"+ directory = root </> "capabilities"+ rulesPath = root </> "rules.json"+ cachePath = directory </> "alpha.capabilities.json"+ command outDir = ["cell", "route", "--cell", "alpha", "--control-bucket", "tan-nb-exp-cells-control", "--plan", "../kenshou-core/test/golden/run-plan.minimal.json", "--routing-rules", rulesPath, "--out", outDir, "--coerce-durable"]+ cache digest = object ["schema" .= ("kenshou.cell-capabilities/v1" :: Text), "cell" .= ("alpha" :: Text), "descriptorSha256" .= digest, "probedBy" .= renderRunId probeId, "probedAt" .= now, "capabilities" .= object ["postgres.pg_partman" .= False]]+ rule = object ["when" .= object ["scenario" .= ("selftest/kernel/**" :: Text)], "requires" .= ("postgres.pg_partman" :: Text), "why" .= ("requires extension" :: Text)]+ createDirectoryIfMissing True directory+ LazyByteString.writeFile rulesPath (Aeson.encode [rule])+ _ <- store.putObject control (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist descriptorBytes+ withCellStore root $ withCapabilityDir directory do+ LazyByteString.writeFile cachePath (Aeson.encode (cache (descriptorDigest descriptor)))+ runWithArgs (command (root </> "denied")) `shouldReturnCode` ExitFailure 2+ doesFileExist (root </> "denied" </> "plan.alpha.json") `shouldReturn` False+ LazyByteString.writeFile cachePath (Aeson.encode (cache (Text.replicate 64 "0")))+ runWithArgs (command (root </> "unprobed")) `shouldReturnCode` ExitSuccess+ doesFileExist (root </> "unprobed" </> "plan.alpha.json") `shouldReturn` True++ it "prepares a dry-run submission under an existing file-backed lease" $+ withSystemTempDirectory "kenshou-cell-submit" \root -> do+ store <- newFileStore root+ descriptorBytes <- LazyByteString.readFile "../kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+ observed <- either fail pure (Aeson.eitherDecode descriptorBytes :: Either String CellDescriptor)+ let control = Bucket "control"+ descriptor = observed {buckets = observed.buckets {control = "control"}}+ location = ["--cell", "alpha", "--control-bucket", "control"]+ outDir = root </> "session"+ _ <- store.putObject control (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist (Aeson.encode descriptor)+ withCellStore root do+ runWithArgs (["cell", "lease"] <> location <> ["--purpose", "submit-test"]) `shouldReturnCode` ExitSuccess+ stored <- store.getObject control (ObjectName "cells/alpha/lease.json")+ lease <- case stored of+ Just (bytes, _) -> either fail pure (Aeson.eitherDecode bytes :: Either String Lease)+ Nothing -> expectationFailure "lease was not published" >> error "unreachable"+ let common = ["cell", "submit"] <> location <> ["--lease-id", Text.unpack (renderRunId lease.leaseId), "--payload", "../kenshou-remote/test/golden/payload.json", "--plan", "../kenshou-core/test/golden/run-plan.minimal.json", "--out", outDir, "--dry-run"]+ runWithArgs (common <> ["--coerce-durable"]) `shouldReturnCode` ExitSuccess+ doesPathExist outDir `shouldReturn` False+ runWithArgs common `shouldReturnCode` ExitFailure 2+ template <- Aeson.eitherDecodeFileStrict' "../kenshou-remote/test/golden/cell-session.json" >>= either fail pure+ let complete = (template :: SessionJournal) {store = Text.pack ("file:" <> root), controlBucket = "control", resultsBucket = descriptor.buckets.results, leaseId = lease.leaseId, slices = []}+ createDirectoryIfMissing True outDir+ writeSessionJournal (outDir </> "session.json") complete+ runWithArgs ["cell", "resume", "--session", outDir] `shouldReturnCode` ExitSuccess++ it "releases a one-command lease when plan preparation fails" $+ withSystemTempDirectory "kenshou-cell-run" \root -> do+ store <- newFileStore root+ descriptorBytes <- LazyByteString.readFile "../kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+ observed <- either fail pure (Aeson.eitherDecode descriptorBytes :: Either String CellDescriptor)+ let control = Bucket "control"+ descriptor = observed {buckets = observed.buckets {control = "control"}}+ outDir = root </> "run-session"+ command = ["cell", "run", "--cell", "alpha", "--control-bucket", "control", "--payload", "../kenshou-remote/test/golden/payload.json", "--plan", "../kenshou-core/test/golden/run-plan.minimal.json", "--out", outDir]+ _ <- store.putObject control (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist (Aeson.encode descriptor)+ withCellStore root do+ runWithArgs command `shouldReturnCode` ExitFailure 2+ store.statObject control (ObjectName "cells/alpha/lease.json") `shouldReturn` Nothing+ doesPathExist outDir `shouldReturn` False++ it "checkpoints a rejected one-command submission and releases its lease" $+ withSystemTempDirectory "kenshou-cell-run-rejected" \root -> do+ store <- newFileStore root+ descriptorBytes <- LazyByteString.readFile "../kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+ observed <- either fail pure (Aeson.eitherDecode descriptorBytes :: Either String CellDescriptor)+ let control = Bucket "control"+ descriptor = observed {buckets = observed.buckets {control = "control"}}+ outDir = root </> "run-session"+ command = ["cell", "run", "--cell", "alpha", "--control-bucket", "control", "--payload", "../kenshou-remote/test/golden/payload.json", "--plan", "../kenshou-core/test/golden/run-plan.minimal.json", "--out", outDir, "--coerce-durable"]+ awaitMarker = do+ objects <- store.listObjects control "cells/alpha/submissions/"+ case find (Text.isSuffixOf "/submission.json" . (.unObjectName) . fst) objects of+ Nothing -> threadDelay 10000 >> awaitMarker+ Just (name, _) -> do+ stored <- store.getObject control name+ case stored of+ Nothing -> expectationFailure "submission marker disappeared"+ Just (bytes, _) -> do+ submission <- either fail pure (Aeson.eitherDecode bytes :: Either String Submission)+ now <- getCurrentTime+ let prefix = "cells/alpha/submissions/" <> renderRunId submission.runId <> "/rejected.json"+ _ <- store.putObject control (ObjectName prefix) "application/json" DoesNotExist (Aeson.encode (Rejected submission.runId "fixture-rejected" now))+ pure ()+ _ <- store.putObject control (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist (Aeson.encode descriptor)+ worker <- forkIO awaitMarker+ ( withCellStore root do+ result <- timeout 15000000 (runWithArgs command)+ result `shouldBe` Just (ExitFailure 4)+ store.statObject control (ObjectName "cells/alpha/lease.json") `shouldReturn` Nothing+ journal <- readSessionJournal (outDir </> "session.json") >>= either (fail . Text.unpack) pure+ fmap (.state) journal.slices `shouldBe` [SliceRejected]+ )+ `finally` killThread worker++ it "fetches and verifies a sealed one-command submission" $+ withSystemTempDirectory "kenshou-cell-run-sealed" \root -> do+ store <- newFileStore root+ descriptorBytes <- LazyByteString.readFile "../kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+ observed <- either fail pure (Aeson.eitherDecode descriptorBytes :: Either String CellDescriptor)+ let control = Bucket "control"+ results = Bucket observed.buckets.results+ descriptor = observed {buckets = observed.buckets {control = "control"}}+ outDir = root </> "run-session"+ command = ["cell", "run", "--cell", "alpha", "--control-bucket", "control", "--payload", "../kenshou-remote/test/golden/payload.json", "--plan", "../kenshou-core/test/golden/run-plan.minimal.json", "--out", outDir, "--coerce-durable"]+ awaitMarker = do+ objects <- store.listObjects control "cells/alpha/submissions/"+ case find (Text.isSuffixOf "/submission.json" . (.unObjectName) . fst) objects of+ Nothing -> threadDelay 10000 >> awaitMarker+ Just (name, _) -> do+ marker <- store.getObject control name+ submission <- case marker of+ Just (bytes, _) -> either fail pure (Aeson.eitherDecode bytes :: Either String Submission)+ Nothing -> fail "submission marker disappeared"+ journal <- readSessionJournal (outDir </> "session.json") >>= either (fail . Text.unpack) pure+ nestedId <- case journal.slices of+ [slice] -> case slice.runIds of+ [identifier] -> pure identifier+ _ -> fail "expected one nested run"+ _ -> fail "expected one cell slice"+ let prefix = "cells/alpha/submissions/" <> renderRunId submission.runId <> "/"+ work <- store.getObject control (ObjectName (prefix <> "work"))+ workBytes <- case work of+ Just (bytes, _) -> pure bytes+ Nothing -> fail "submission work disappeared"+ now <- getCurrentTime+ let payloadDigest = submission.payload.bundle.sha256+ nestedBytes = Aeson.encode (object ["schema" .= ("kenshou.run-result/v1" :: Text), "runId" .= nestedId, "scenario" .= ("selftest/kernel/correctness/postgres-roundtrip" :: Text), "outcome" .= Outcome.Passed, "fingerprint" .= object ["cell" .= object ["payload" .= object ["bundleSha256" .= payloadDigest]]]])+ nestedInfo = workObjectFor "application/json" nestedBytes+ nestedManifest = Manifest nestedId now [ManifestFile "run-result.json" ("sha256:" <> nestedInfo.sha256) (fromIntegral nestedInfo.bytes) "application/json"]+ files =+ [ ("submission/work", workBytes),+ ("submission/submission.json", Aeson.encode submission),+ ("cell/result.json", Aeson.encode (CellRunResult submission.runId "alpha" submission.leaseId 1 Completed (Just 0) Nothing [])),+ ("output/" <> renderRunId nestedId <> "/run-result.json", nestedBytes),+ ("output/" <> renderRunId nestedId <> "/manifest.json", Aeson.encode nestedManifest)+ ]+ artifactFor (path, bytes) = let info = workObjectFor "application/json" bytes in Artifact path info.sha256 info.bytes "application/json"+ manifest = CellManifest submission.runId "alpha" submission.leaseId 1 now "0.1.0" (ManifestPayload payloadDigest submission.payload.storePath) Completed 3600 (map artifactFor files)+ manifestBytes = Aeson.encode manifest+ manifestDigest = (workObjectFor "application/json" manifestBytes).sha256+ mapM_+ ( \(path, bytes) -> do+ _ <- store.putObject results (ObjectName ("runs/" <> renderRunId submission.runId <> "/" <> path)) "application/json" DoesNotExist bytes+ pure ()+ )+ files+ _ <- store.putObject results (ObjectName ("runs/" <> renderRunId submission.runId <> "/manifest.json")) "application/json" DoesNotExist manifestBytes+ _ <- store.putObject control (ObjectName (prefix <> "status.json")) "application/json" DoesNotExist (Aeson.encode (CellStatus submission.runId Sealed (Just 1) now Nothing (LogChunks 0 0) (Just Completed) (Just manifestDigest) (Just [])))+ pure ()+ _ <- store.putObject control (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist (Aeson.encode descriptor)+ worker <- forkIO awaitMarker+ ( withCellStore root do+ result <- timeout 15000000 (runWithArgs command)+ result `shouldBe` Just ExitSuccess+ store.statObject control (ObjectName "cells/alpha/lease.json") `shouldReturn` Nothing+ journal <- readSessionJournal (outDir </> "session.json") >>= either (fail . Text.unpack) pure+ fmap (.state) journal.slices `shouldBe` [SliceVerified]+ case journal.slices of+ [slice] -> doesFileExist (root </> Text.unpack (renderRunId slice.cellRun) </> "cell-run.json") `shouldReturn` True+ _ -> expectationFailure "expected one verified cell slice"+ )+ `finally` killThread worker++ it "runs overhead slots behind one cell lease and links their verified run directories" $+ withSystemTempDirectory "kenshou-cell-overhead" \root -> do+ store <- newFileStore root+ descriptorBytes <- LazyByteString.readFile "../kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+ observed <- either fail pure (Aeson.eitherDecode descriptorBytes :: Either String CellDescriptor)+ let control = Bucket "control"+ results = Bucket observed.buckets.results+ descriptor = observed {buckets = observed.buckets {control = "control"}}+ outRoot = root </> "overhead-output"+ command =+ [ "cell",+ "overhead",+ "selftest/telemetry/benchmark/arms-on-synthetic-service",+ "--cell",+ "alpha",+ "--control-bucket",+ "control",+ "--payload",+ "../kenshou-remote/test/golden/payload.json",+ "--arms",+ "tracing=off,noop",+ "--trials",+ "3",+ "--settle-seconds",+ "0",+ "--retries",+ "0",+ "--policy",+ "../policies/telemetry-overhead.json",+ "--out",+ outRoot+ ]+ awaitMarkers seen = do+ objects <- store.listObjects control "cells/alpha/submissions/"+ case find (\(name, _) -> Text.isSuffixOf "/submission.json" name.unObjectName && name `Set.notMember` seen) objects of+ Nothing -> threadDelay 10000 >> awaitMarkers seen+ Just (name, _) -> do+ marker <- store.getObject control name+ submission <- case marker of+ Just (bytes, _) -> either fail pure (Aeson.eitherDecode bytes :: Either String Submission)+ Nothing -> fail "overhead submission marker disappeared"+ let prefix = "cells/alpha/submissions/" <> renderRunId submission.runId <> "/"+ work <- store.getObject control (ObjectName (prefix <> "work"))+ workBytes <- case work of+ Just (bytes, _) -> pure bytes+ Nothing -> fail "overhead work disappeared"+ workValue <- either fail pure (Aeson.eitherDecode workBytes :: Either String Value)+ nestedId <- case workValue of+ Aeson.Object fields -> case KeyMap.lookup "runs" fields of+ Just (Aeson.Array entries) -> case toList entries of+ [Aeson.Object entry] -> case KeyMap.lookup "runId" entry of+ Just value -> case Aeson.fromJSON value of Aeson.Success identifier -> pure identifier; Aeson.Error problem -> fail problem+ Nothing -> fail "overhead work has no nested run ID"+ _ -> fail "overhead work must contain one valid run entry"+ _ -> fail "overhead work has no runs"+ _ -> fail "overhead work is invalid"+ now <- getCurrentTime+ let sequenceNumber = Set.size seen + 1+ payloadDigest = submission.payload.bundle.sha256+ nestedBytes = Aeson.encode (object ["schema" .= ("kenshou.run-result/v1" :: Text), "runId" .= nestedId, "scenario" .= ("selftest/telemetry/benchmark/arms-on-synthetic-service" :: Text), "outcome" .= Outcome.Passed, "fingerprint" .= object ["cell" .= object ["payload" .= object ["bundleSha256" .= payloadDigest]]]])+ nestedInfo = workObjectFor "application/json" nestedBytes+ nestedManifest = Manifest nestedId now [ManifestFile "run-result.json" ("sha256:" <> nestedInfo.sha256) (fromIntegral nestedInfo.bytes) "application/json"]+ files =+ [ ("submission/work", workBytes),+ ("submission/submission.json", Aeson.encode submission),+ ("cell/result.json", Aeson.encode (CellRunResult submission.runId "alpha" submission.leaseId sequenceNumber Completed (Just 0) Nothing [])),+ ("output/" <> renderRunId nestedId <> "/run-result.json", nestedBytes),+ ("output/" <> renderRunId nestedId <> "/manifest.json", Aeson.encode nestedManifest)+ ]+ artifactFor (path, bytes) = let info = workObjectFor "application/json" bytes in Artifact path info.sha256 info.bytes "application/json"+ manifest = CellManifest submission.runId "alpha" submission.leaseId sequenceNumber now "0.1.0" (ManifestPayload payloadDigest submission.payload.storePath) Completed 3600 (map artifactFor files)+ manifestBytes = Aeson.encode manifest+ manifestDigest = (workObjectFor "application/json" manifestBytes).sha256+ mapM_ (\(path, bytes) -> do _ <- store.putObject results (ObjectName ("runs/" <> renderRunId submission.runId <> "/" <> path)) "application/json" DoesNotExist bytes; pure ()) files+ _ <- store.putObject results (ObjectName ("runs/" <> renderRunId submission.runId <> "/manifest.json")) "application/json" DoesNotExist manifestBytes+ _ <- store.putObject control (ObjectName (prefix <> "status.json")) "application/json" DoesNotExist (Aeson.encode (CellStatus submission.runId Sealed (Just sequenceNumber) now Nothing (LogChunks 0 0) (Just Completed) (Just manifestDigest) (Just [])))+ if sequenceNumber < 6 then awaitMarkers (Set.insert name seen) else pure ()+ _ <- store.putObject control (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist (Aeson.encode descriptor)+ worker <- forkIO (awaitMarkers Set.empty)+ ( withCellStore root do+ result <- timeout 30000000 (runWithArgs command)+ result `shouldBe` Just (ExitFailure 4)+ store.statObject control (ObjectName "cells/alpha/lease.json") `shouldReturn` Nothing+ [invocation] <- listDirectory outRoot+ state <- loadOverheadState (outRoot </> invocation) >>= either fail pure+ length state.slots `shouldBe` 6+ all (.complete) state.slots `shouldBe` True+ observations <-+ traverse+ ( \slot -> do+ runId <- case reverse slot.runIds of identifier : _ -> pure identifier; [] -> fail "overhead slot has no run"+ doesFileExist (outRoot </> invocation </> "runs" </> Text.unpack (renderRunId runId) </> "manifest.json") `shouldReturn` True+ journal <- readSessionJournal (outRoot </> invocation </> "cell-sessions" </> Text.unpack (renderRunId runId) </> "session.json") >>= either (fail . Text.unpack) pure+ slice <- case journal.slices of [one] -> pure one; _ -> fail "overhead child must have one slice"+ sealed <- store.getObject results (ObjectName ("runs/" <> renderRunId slice.cellRun <> "/manifest.json"))+ manifest <- case sealed of+ Just (bytes, _) -> either fail pure (Aeson.eitherDecode bytes :: Either String CellManifest)+ Nothing -> fail "overhead child has no cell manifest"+ pure (journal.leaseId, manifest.leaseSequence)+ )+ state.slots+ length (Set.fromList (fmap fst observations)) `shouldBe` 1+ List.sort (fmap snd observations) `shouldBe` [1 .. 6]+ resumed <- timeout 15000000 (runWithArgs (command <> ["--resume"]))+ resumed `shouldBe` Just (ExitFailure 4)+ markers <- store.listObjects control "cells/alpha/submissions/"+ length (filter (Text.isSuffixOf "/submission.json" . (.unObjectName) . fst) markers) `shouldBe` 6+ )+ `finally` killThread worker++ it "leaves a detached single-slice lease alive and collects its rejection later" $+ withSystemTempDirectory "kenshou-cell-run-detached" \root -> do+ store <- newFileStore root+ descriptorBytes <- LazyByteString.readFile "../kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+ observed <- either fail pure (Aeson.eitherDecode descriptorBytes :: Either String CellDescriptor)+ let control = Bucket "control"+ descriptor = observed {buckets = observed.buckets {control = "control"}}+ outDir = root </> "run-session"+ leaseObject = ObjectName "cells/alpha/lease.json"+ command = ["cell", "run", "--cell", "alpha", "--control-bucket", "control", "--payload", "../kenshou-remote/test/golden/payload.json", "--plan", "../kenshou-core/test/golden/run-plan.minimal.json", "--out", outDir, "--coerce-durable", "--detach"]+ _ <- store.putObject control (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist (Aeson.encode descriptor)+ withCellStore root do+ runWithArgs command `shouldReturnCode` ExitSuccess+ journal <- readSessionJournal (outDir </> "session.json") >>= either (fail . Text.unpack) pure+ journal.leaseMode `shouldBe` Detached+ fmap (.state) journal.slices `shouldBe` [SliceSubmitted]+ leaseStored <- store.getObject control leaseObject+ lease <- case leaseStored of+ Just (bytes, _) -> either fail pure (Aeson.eitherDecode bytes :: Either String Lease)+ Nothing -> expectationFailure "detached lease was released" >> error "unreachable"+ slice <- case journal.slices of+ [entry] -> pure entry+ _ -> expectationFailure "expected one detached slice" >> error "unreachable"+ lease.ttlSeconds `shouldSatisfy` (>= fromIntegral slice.submission.limits.wallClockSeconds + 600)+ runWithArgs ["cell", "resume", "--session", outDir] `shouldReturnCode` ExitFailure 4+ store.statObject control leaseObject >>= (`shouldSatisfy` (maybe False (const True)))+ now <- getCurrentTime+ let rejectedObject = ObjectName ("cells/alpha/submissions/" <> renderRunId slice.cellRun <> "/rejected.json")+ _ <- store.putObject control rejectedObject "application/json" DoesNotExist (Aeson.encode (Rejected slice.cellRun "fixture-rejected" now))+ runWithArgs ["cell", "resume", "--session", outDir] `shouldReturnCode` ExitFailure 4+ store.statObject control leaseObject `shouldReturn` Nothing+ finished <- readSessionJournal (outDir </> "session.json") >>= either (fail . Text.unpack) pure+ fmap (.state) finished.slices `shouldBe` [SliceRejected]+ runWithArgs command `shouldReturnCode` ExitFailure 2+ store.statObject control leaseObject `shouldReturn` Nothing++ it "starts stopped instances and releases a reattached held lease after a terminal resume" $+ withSystemTempDirectory "kenshou-cell-resume-release" \root -> do+ store <- newFileStore root+ descriptorBytes <- LazyByteString.readFile "../kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+ observed <- either fail pure (Aeson.eitherDecode descriptorBytes :: Either String CellDescriptor)+ template <- Aeson.eitherDecodeFileStrict' "../kenshou-remote/test/golden/cell-session.json" >>= either fail pure+ let control = Bucket "control"+ descriptor = observed {buckets = observed.buckets {control = "control"}}+ outDir = root </> "session"+ leaseObject = ObjectName "cells/alpha/lease.json"+ _ <- store.putObject control (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist (Aeson.encode descriptor)+ withCellStore root do+ runWithArgs ["cell", "lease", "--cell", "alpha", "--control-bucket", "control", "--purpose", "resume-test"] `shouldReturnCode` ExitSuccess+ stored <- store.getObject control leaseObject+ lease <- case stored of+ Just (bytes, _) -> either fail pure (Aeson.eitherDecode bytes :: Either String Lease)+ Nothing -> fail "fixture lease missing"+ let journal = (template :: SessionJournal) {store = Text.pack ("file:" <> root), controlBucket = "control", resultsBucket = descriptor.buckets.results, leaseId = lease.leaseId, leaseMode = Held, slices = map (\slice -> slice {state = SliceRejected, rejectionReason = Just "fixture-rejected", cellOutcome = Nothing, entryExitCode = Nothing, manifestSha256 = Nothing, fetchedPath = Nothing}) template.slices}+ createDirectoryIfMissing True outDir+ writeSessionJournal (outDir </> "session.json") journal+ let binDir = root </> "bin"+ fakeGcloud = binDir </> "gcloud"+ started = root </> "instances-started"+ createDirectoryIfMissing True binDir+ writeFile fakeGcloud "#!/bin/sh\nif [ \"$3\" = describe ]; then echo TERMINATED; else printf started > \"$KENSHOU_TEST_STARTED\"; fi\n"+ permissions <- getPermissions fakeGcloud+ setPermissions fakeGcloud permissions {executable = True}+ oldPath <- lookupEnv "PATH"+ withEnvVariable "PATH" (binDir <> ":" <> maybe "" id oldPath) $+ withEnvVariable "KENSHOU_TEST_STARTED" started $+ runWithArgs ["cell", "resume", "--start", "--session", outDir] `shouldReturnCode` ExitFailure 4+ doesFileExist started `shouldReturn` True+ store.statObject control leaseObject `shouldReturn` Nothing+ runWithArgs ["cell", "resume", "--start", "--help"] `shouldReturnCode` ExitSuccess++ it "rebinds a submitted marker that a prior held lease never accepted" $+ withSystemTempDirectory "kenshou-cell-resume-unaccepted" \root -> do+ store <- newFileStore root+ descriptorBytes <- LazyByteString.readFile "../kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+ observed <- either fail pure (Aeson.eitherDecode descriptorBytes :: Either String CellDescriptor)+ template <- Aeson.eitherDecodeFileStrict' "../kenshou-remote/test/golden/cell-session.json" >>= either fail pure+ let control = Bucket "control"+ descriptor = observed {buckets = observed.buckets {control = "control"}}+ outDir = root </> "session"+ leaseObject = ObjectName "cells/alpha/lease.json"+ _ <- store.putObject control (ObjectName "cells/alpha/descriptor.json") "application/json" DoesNotExist (Aeson.encode descriptor)+ withCellStore root do+ runWithArgs ["cell", "lease", "--cell", "alpha", "--control-bucket", "control", "--purpose", "old-session"] `shouldReturnCode` ExitSuccess+ oldStored <- store.getObject control leaseObject+ oldLease <- case oldStored of+ Just (bytes, _) -> either fail pure (Aeson.eitherDecode bytes :: Either String Lease)+ Nothing -> expectationFailure "old lease missing" >> error "unreachable"+ fixtureSlice <- case (template :: SessionJournal).slices of+ [entry] -> pure entry+ _ -> expectationFailure "expected one fixture slice" >> error "unreachable"+ let fixtureSubmission = fixtureSlice.submission+ fixturePayload = fixtureSubmission.payload+ fixtureBundle = fixturePayload.bundle+ payload = fixturePayload {bundle = fixtureBundle {uri = "gs://control/payloads/sha256/" <> fixtureBundle.sha256 <> ".nar.zst"}}+ oldSubmission = fixtureSubmission {leaseId = oldLease.leaseId, payload = payload, work = workObjectFor "application/json" "{}"}+ oldSlice = fixtureSlice {submission = oldSubmission, state = SliceSubmitted}+ journal = template {store = Text.pack ("file:" <> root), controlBucket = "control", resultsBucket = descriptor.buckets.results, leaseId = oldLease.leaseId, slices = [oldSlice]}+ oldMarker = ObjectName ("cells/alpha/submissions/" <> renderRunId oldSlice.cellRun <> "/submission.json")+ awaitRetry = do+ objects <- store.listObjects control "cells/alpha/submissions/"+ case find (\(name, _) -> name /= oldMarker && Text.isSuffixOf "/submission.json" name.unObjectName) objects of+ Nothing -> threadDelay 10000 >> awaitRetry+ Just (name, _) -> do+ stored <- store.getObject control name+ submission <- case stored of+ Just (bytes, _) -> either fail pure (Aeson.eitherDecode bytes :: Either String Submission)+ Nothing -> fail "retry marker disappeared"+ now <- getCurrentTime+ let rejected = ObjectName ("cells/alpha/submissions/" <> renderRunId submission.runId <> "/rejected.json")+ _ <- store.putObject control rejected "application/json" DoesNotExist (Aeson.encode (Rejected submission.runId "fixture-rejected" now))+ pure ()+ createDirectoryIfMissing True (outDir </> "slice-0")+ LazyByteString.writeFile (outDir </> "slice-0/work.json") "{}"+ writeSessionJournal (outDir </> "session.json") journal+ _ <- store.putObject control oldMarker "application/json" DoesNotExist (Aeson.encode oldSubmission)+ runWithArgs ["cell", "release", "--cell", "alpha", "--control-bucket", "control", "--lease-id", Text.unpack (renderRunId oldLease.leaseId)] `shouldReturnCode` ExitSuccess+ worker <- forkIO awaitRetry+ ( do+ result <- timeout 15000000 (runWithArgs ["cell", "resume", "--session", outDir])+ result `shouldBe` Just (ExitFailure 4)+ finished <- readSessionJournal (outDir </> "session.json") >>= either (fail . Text.unpack) pure+ fmap (.state) finished.slices `shouldBe` [SliceRejected]+ fmap (.cellRun) finished.slices `shouldSatisfy` (/= [oldSlice.cellRun])+ store.statObject control leaseObject `shouldReturn` Nothing+ )+ `finally` killThread worker++ it "reads scenario history from a bundle" do+ runWithArgs ["history", "--bundle", "../fixtures/consumer-baseline/docs/verification", "--bundle-uri", "mori://shinzui/kenshou/okf/verification", "--scenario", "selftest/kernel/correctness/always-pass", "--json"] `shouldReturnCode` ExitSuccess+ runWithArgs ["history", "--bundle", "../fixtures/consumer-baseline/docs/verification", "--scenario", "selftest/kernel/correctness/always-pass", "--json"] `shouldReturnCode` ExitFailure 2++ it "requires one record source" do+ runWithArgs ["record", "--purpose", "release"] `shouldReturnCode` ExitFailure 2+ runWithArgs ["record", "a-run", "--comparison", "comparison.json", "--purpose", "release"] `shouldReturnCode` ExitFailure 2++ it "rejects a human anomaly exception without an interactive terminal" do+ runWithArgs ["attest", "invalid", "--project", "fixture", "--accept-anomaly", "--authority", "human:fixture", "--reason", "fixture"] `shouldReturnCode` ExitFailure 2++ it "returns 2 for an unknown subcommand" do+ runWithArgs ["cohort", "bogus"] `shouldReturnCode` ExitFailure 2++ it "returns 1 when offline leak diagnosis finds growth" do+ runWithArgs ["diagnose", "leak", leakingRun] `shouldReturnCode` ExitFailure 1++ it "returns 1 when an offline stall diagnosis exists" do+ runWithArgs ["diagnose", "stall", stalledRun] `shouldReturnCode` ExitFailure 1++ it "returns 0 when no offline stall diagnosis exists" do+ runWithArgs ["diagnose", "stall", leakingRun] `shouldReturnCode` ExitSuccess++ it "returns 3 when leak evidence is insufficient" do+ runWithArgs ["diagnose", "leak", stalledRun] `shouldReturnCode` ExitFailure 3++ it "returns 4 when diagnosis input is unavailable" do+ runWithArgs ["diagnose", "leak", "test/fixtures/does-not-exist"] `shouldReturnCode` ExitFailure 4++ it "returns 2 for invalid diagnose syntax" do+ runWithArgs ["diagnose", "leak", "--bogus"] `shouldReturnCode` ExitFailure 2++ it "does not mutate a sealed run manifest" do+ before <- Text.readFile (leakingRun <> "/manifest.json")+ _ <- runWithArgs ["diagnose", "leak", leakingRun]+ after <- Text.readFile (leakingRun <> "/manifest.json")+ after `shouldBe` before++ describe "version" do+ it "includes the cabal package version and a short revision" do+ appVersionWithGit `shouldSatisfy` Text.isPrefixOf "kenshou v0.1.0.0 ("+ Text.dropAround (`elem` ['(', ')']) (Text.takeWhileEnd (/= ' ') appVersionWithGit)+ `shouldSatisfy` (\revision -> revision == "dirty" || Text.length revision == 7)++runWithArgs :: [String] -> IO ExitCode+runWithArgs = runWithArgsWith selfTestAssembly++-- Runs from the consumer-baseline bridge with the consumer's component graph, for the+-- perf planning checks whose request names the consumer's packages. Called from the+-- repository root.+withConsumerBaseline :: [String] -> IO ExitCode+withConsumerBaseline arguments = do+ graph <- ByteString.readFile "fixtures/consumer-baseline/kenshou-core/data/components.json"+ withCurrentDirectory "fixtures/consumer-baseline" (runWithArgsWith selfTestAssembly {componentGraph = graph} arguments)++shouldReturnCode :: IO ExitCode -> ExitCode -> IO ()+shouldReturnCode action expected = action >>= (`shouldBe` expected)++leakingRun :: FilePath+leakingRun = "../kenshou-diagnose/test/fixtures/run-leaking"++stalledRun :: FilePath+stalledRun = "../kenshou-diagnose/test/fixtures/run-stalled"++withCellStore :: FilePath -> IO value -> IO value+withCellStore root operation = bracket (lookupEnv "KENSHOU_CELL_STORE") (restore "KENSHOU_CELL_STORE") \_ ->+ bracket (lookupEnv "KENSHOU_GCP_ALLOWED_PROJECTS") (restore "KENSHOU_GCP_ALLOWED_PROJECTS") \_ -> do+ setEnv "KENSHOU_CELL_STORE" ("file:" <> root)+ setEnv "KENSHOU_GCP_ALLOWED_PROJECTS" "tan-nb-exp"+ operation+ where+ restore name Nothing = unsetEnv name+ restore name (Just prior) = setEnv name prior++withCapabilityDir :: FilePath -> IO value -> IO value+withCapabilityDir directory operation = bracket (lookupEnv "KENSHOU_CELL_CAPABILITIES_DIR") restore \_ -> do+ setEnv "KENSHOU_CELL_CAPABILITIES_DIR" directory+ operation+ where+ restore Nothing = unsetEnv "KENSHOU_CELL_CAPABILITIES_DIR"+ restore (Just prior) = setEnv "KENSHOU_CELL_CAPABILITIES_DIR" prior++withEnvVariable :: String -> String -> IO value -> IO value+withEnvVariable name selected operation = bracket (lookupEnv name) restore \_ -> do+ setEnv name selected+ operation+ where+ restore Nothing = unsetEnv name+ restore (Just prior) = setEnv name prior++withPerfRoot :: IO a -> IO a+withPerfRoot operation = do+ exists <- doesFileExist "cabal.project"+ withCurrentDirectory (if exists then "." else "..") operation++replacePerfField :: Key.Key -> Value -> Value -> Value+replacePerfField key value (Aeson.Object fields) = Aeson.Object (KeyMap.insert key value fields)+replacePerfField _ _ value = value++-- The request's own target components plus one component of its cohort template+-- that belongs to a different project, read from the consumer-baseline bridge.+foreignTargets :: Value -> IO [Text]+foreignTargets request = do+ let field name (Aeson.Object fields) = KeyMap.lookup name fields+ field _ _ = Nothing+ texts value = case value of Just (Aeson.Array items) -> [text | Aeson.String text <- toList items]; _ -> []+ owned = texts (field "targetComponents" request)+ template <- case field "cohortTemplate" request of+ Just (Aeson.String path) -> Aeson.eitherDecodeFileStrict' ("fixtures/consumer-baseline/" <> Text.unpack path) >>= either fail pure+ _ -> fail "the request names no cohort template"+ let components = case field "components" template of Just (Aeson.Array items) -> toList items; _ -> []+ case [name | component <- components, Just (Aeson.String name) <- [field "id" component], field "moriUri" component /= field "project" request] of+ other : _ -> pure (owned <> [other])+ [] -> fail "the cohort template has no component from another project"
+ test/PerfSessionSpec.hs view
@@ -0,0 +1,264 @@+module PerfSessionSpec (PerfSessionSpec.spec) where++import Control.Concurrent (forkIO, killThread, threadDelay)+import Control.Exception (bracket, finally)+import Control.Monad (forM, forM_)+import Data.Aeson+import Data.Aeson.KeyMap qualified as KeyMap+import Data.ByteString qualified as Bytes+import Data.ByteString.Lazy qualified as Lazy+import Data.Map.Strict qualified as Map+import Data.Set qualified as Set+import Data.Text (Text)+import Data.Text qualified as Text+import Data.Time (UTCTime, addUTCTime, getCurrentTime)+import Kenshou.Cli (runWithArgsWith, selfTestAssembly)+import Kenshou.Core.Canonical (sha256Hex)+import Kenshou.Core.Id (renderRunId)+import Kenshou.Core.Manifest (writeManifest)+import Kenshou.Core.RunSpec (RunSpec (..))+import Kenshou.Measure.Clock (Origin (..))+import Kenshou.Measure.Histogram qualified as Histogram+import Kenshou.Measure.Histogram.Codec (encodeHistogram)+import Kenshou.Measure.Samples+import Kenshou.Measure.Summary (summarizeRunDir)+import Kenshou.Perf.Artifacts (sealPerfDirectory)+import Kenshou.Perf.Plan (PerfPlan (..), PerfProfile (..), PlannedProfile (..))+import Kenshou.Perf.Prepared qualified as Prepared+import Kenshou.Perf.Result qualified as Result+import Kenshou.Plan.RunPlan (PlannedRun (..), RunPlan (..))+import Kenshou.Remote.Cell.Docs+import Kenshou.Remote.Cell.Lease qualified as Lease+import Kenshou.Remote.Cell.Prepare (cellMachineProfile)+import Kenshou.Remote.Cell.Submit (workObjectFor)+import Kenshou.Remote.Cell.WorkJson (decodeWorkPlan)+import Kenshou.Remote.Payload (Bundle (..), PayloadDescriptor (..))+import Kenshou.Remote.Payload qualified as Payload+import Kenshou.Remote.Perf.Journal qualified as Journal+import Kenshou.Remote.Store+import Kenshou.Remote.Store.File (newFileStore)+import System.Directory (createDirectoryIfMissing, doesFileExist, listDirectory, removeFile, withCurrentDirectory)+import System.Environment (lookupEnv, setEnv, unsetEnv)+import System.Exit (ExitCode (..))+import System.FilePath ((</>))+import System.IO.Temp (withSystemTempDirectory)+import System.Timeout (timeout)+import Test.Hspec++runWithArgs :: [String] -> IO ExitCode+runWithArgs = runWithArgsWith selfTestAssembly++spec :: Spec+-- The prepared fixture pins scenarios of a consumer layer that this repository does not+-- register, so dispatch cannot resolve them. The kernel plan that regenerates the perf+-- fixtures from synthetic scenarios removes this guard:+-- docs/plans/2-open-the-kernel-layer-identity-and-schema-composition-to-registered-suites.md (M4).+spec = describe "file-backed performance CLI integration" $ before_ (pendingWith "perf-prepared fixture names unregistered consumer scenarios; regenerated in docs/plans/2-… M4") do+ forM_ [(False, "executes two profiles under one lease and replays the full sealed graph without redispatch"), (True, "retains an exit-zero known defect through complete report replay and resume")] \(knownFault, title) -> it title $+ withSystemTempDirectory "perf-cli-healthy" \root -> projectRoot do+ let preparation = root </> "preparation"+ output = root </> "session"+ storeRoot = root </> "store"+ payloadBytes = "synthetic protocol payload; no runtime calibration is claimed"+ digest = Text.drop 7 (sha256Hex (Lazy.toStrict payloadBytes))+ bundle = Bundle ("gs://tan-nb-exp-cells-control/payloads/sha256/" <> digest <> ".nar.zst") digest (Lazy.length payloadBytes)+ replacePayload arm = case fromJSON arm.payload of+ Error problem -> fail problem+ Success payload -> pure arm {Prepared.payload = toJSON (payload {Payload.cell = payload.cell {Payload.bundle = bundle}} :: PayloadDescriptor)}+ original <- readJSON "fixtures/consumer-baseline/kenshou-core/test/golden/perf-prepared.fixture.json"+ baseline <- replacePayload (original :: Prepared.PerfPrepared).baseline+ candidate <- replacePayload original.candidate+ let prepared = original {Prepared.baseline = baseline, Prepared.candidate = candidate}+ -- The captured preparation's profiles, in selection order.+ profileNames = [entry.profile.name | entry <- prepared.plan.selected]+ (firstProfile, lastProfile) = case profileNames of+ first : rest -> (Text.unpack first, last (first : rest))+ [] -> error "the captured preparation selects no profile"+ createDirectoryIfMissing True preparation+ encodeFile (preparation </> "prepared.json") prepared+ sealPerfDirectory preparation >>= either (fail . Text.unpack) pure+ descriptor <- readJSON "kenshou-remote/test/golden/cell/cell.descriptor.v1.json"+ policy <- Lazy.readFile "kenshou-remote/test/golden/cell/cell.policy.v1.json"+ store <- newFileStore storeRoot+ let control = Bucket descriptor.buckets.control+ put bucket name bytes = store.putObject bucket (ObjectName name) "application/json" DoesNotExist bytes >>= requireWritten+ put control ("payloads/sha256/" <> digest <> ".nar.zst") payloadBytes+ put control "cells/alpha/descriptor.json" (encode descriptor)+ put control "cells/alpha/policy.json" policy+ agent <- forkIO (sealMarkers knownFault store descriptor root Set.empty)+ ( withSetting "KENSHOU_CELL_STORE" ("file:" <> storeRoot) $+ withSetting "KENSHOU_CELL_CAPABILITIES_DIR" (root </> "capabilities") do+ execution <- timeout 90000000 (runWithArgs ["perf", "run", "--prepared", preparation </> "prepared.json", "--out", output, "--evidence-base-uri", "gs://fixture-evidence/performance"])+ let expectedExit = if knownFault then ExitFailure 1 else ExitSuccess+ execution `shouldBe` Just expectedExit+ result <- readJSON (output </> "result.json")+ (result :: Result.PerfResult).performanceVerdict `shouldBe` "pass"+ result.gatingEligible `shouldBe` False+ length result.configurations `shouldBe` 2+ map (.validPairs) result.configurations `shouldBe` [5, 5]+ null result.blockingRuntimeFailures `shouldBe` not knownFault+ store.getObject control (ObjectName "cells/alpha/lease.json") `shouldReturn` Nothing+ initialMarkers <- store.listObjects control "cells/alpha/submissions/"+ let markers = filter (Text.isSuffixOf "/submission.json" . (.unObjectName) . fst) initialMarkers+ length markers `shouldBe` 20+ submissions <- forM markers \(name, _) -> do+ retained <- store.getObject control name+ maybe (fail "marker vanished") (either fail pure . eitherDecode . fst) retained+ Set.size (Set.fromList [submission.leaseId | submission :: Submission <- submissions]) `shouldBe` 1+ retainedResult <- Lazy.readFile (output </> "result.json")+ forM_ [[], ["--offline"]] \flags -> runWithArgs (["perf", "verify", "--session", output] <> flags) `shouldReturn` expectedExit+ runWithArgs ["perf", "resume", "--session", output] `shouldReturn` expectedExit+ Lazy.readFile (output </> "result.json") `shouldReturn` retainedResult+ store.listObjects control "cells/alpha/submissions/" `shouldReturn` initialMarkers+ -- A second snapshot is forbidden when completed evidence is reused.+ snapshots <- listDirectory (output </> "snapshots")+ length snapshots `shouldBe` 1+ report <- Bytes.readFile (output </> "report.md")+ report `shouldSatisfy` Bytes.isInfixOf "within policy limits"+ report `shouldSatisfy` Bytes.isInfixOf "Comparison reasons: none"+ -- Reconstruct an interruption after the last comparison file was+ -- saved, before its attempt result and root completion checkpoint.+ -- Every sealed cell object and paired schedule stays unchanged.+ saved <- Journal.readPerfSession (output </> "session.json") >>= either (fail . Text.unpack) pure+ let beforeCompletion = takeWhile (\event -> case event.action of Journal.ComparisonCompleted item -> item.profile /= lastProfile; _ -> True) saved.events+ attempt = output </> "configurations" </> Text.unpack lastProfile </> "attempt-0"+ identity <- case [one | event <- beforeCompletion, Journal.LeaseAcquired one <- [event.action]] of+ [one] -> pure one+ _ -> fail "expected exactly one original lease"+ beforeComparison <- Lazy.readFile (attempt </> "comparison.json")+ -- Fixture construction only: the production writer correctly+ -- refuses to rewind an already committed append-only history.+ encodeFile (output </> "session.json") saved {Journal.events = beforeCompletion}+ removeFile (output </> "receipt.json")+ removeFile (attempt </> "result.json")+ Lazy.writeFile (attempt </> "result.json.next") "interrupted JSON write"+ now <- getCurrentTime+ let activeLease = Lease.Lease identity.leaseId identity.cell "kenshou-perf" "pinned runtime performance validation" 180 now now False 20+ put control "cells/alpha/lease.json" (encode activeLease)+ runWithArgs ["perf", "resume", "--session", output] `shouldReturn` expectedExit+ Lazy.readFile (attempt </> "comparison.json") `shouldReturn` beforeComparison+ store.listObjects control "cells/alpha/submissions/" `shouldReturn` initialMarkers+ forM_ [[], ["--offline"]] \flags -> runWithArgs (["perf", "verify", "--session", output] <> flags) `shouldReturn` expectedExit+ -- Even a locally resealed inventory and matching receipt cannot+ -- make an unlinked cell tree part of the interpreted evidence.+ receipt <- readJSON (output </> "receipt.json")+ fields <- case receipt of Object value -> pure value; _ -> fail "invalid receipt"+ relative <- case KeyMap.lookup "snapshot" fields of Just (String value) -> pure value; _ -> fail "receipt omits snapshot"+ let snapshot = output </> Text.unpack relative+ orphan = snapshot </> "configurations" </> firstProfile </> "attempt-0/trials/unlinked-cell/tree"+ createDirectoryIfMissing True orphan+ encodeFile (orphan </> "run-result.json") (object ["outcome" .= ("passed" :: Text)])+ sealPerfDirectory snapshot >>= either (fail . Text.unpack) pure+ changedManifest <- Bytes.readFile (snapshot </> "manifest.json")+ encodeFile (output </> "receipt.json") (Object (KeyMap.insert "manifestSha256" (String (sha256Hex changedManifest)) fields))+ runWithArgs ["perf", "verify", "--session", output, "--offline"] `shouldReturn` ExitFailure 4+ )+ `finally` killThread agent++-- Implements only the immutable protocol in a file store. Its measurements and+-- machine samples are synthetic controls, never acceptance calibration evidence.+sealMarkers :: Bool -> ObjectStore -> CellDescriptor -> FilePath -> Set.Set ObjectName -> IO ()+sealMarkers knownFault store descriptor root seen = do+ let control = Bucket descriptor.buckets.control+ objects <- store.listObjects control "cells/alpha/submissions/"+ let fresh = [name | (name, _) <- objects, "/submission.json" `Text.isSuffixOf` name.unObjectName, name `Set.notMember` seen]+ forM_ fresh \name -> do+ (markerBytes, _) <- store.getObject control name >>= maybe (fail "missing marker") pure+ submission <- either fail pure (eitherDecode markerBytes)+ let prefix = Text.dropEnd (Text.length "submission.json") name.unObjectName+ (workBytes, _) <- store.getObject control (ObjectName (prefix <> "work")) >>= maybe (fail "missing work") pure+ document <- either fail pure (eitherDecode workBytes)+ plan <- either (fail . Text.unpack) pure (decodeWorkPlan document)+ run <- case plan.runs of [one] -> pure one; _ -> fail "expected exactly one paired trial"+ let local = root </> "agent-output" </> Text.unpack (renderRunId submission.runId)+ nested = local </> "output" </> Text.unpack (renderRunId run.runId)+ writeRawRun (knownFault && Set.null seen) descriptor submission run nested+ nestedFiles <- readTree (local </> "output")+ now <- getCurrentTime+ let sequenceNumber = Set.size seen + length (takeWhile (/= name) fresh) + 1+ files =+ [("submission/work", workBytes), ("submission/submission.json", markerBytes), ("cell/result.json", encode (CellRunResult submission.runId "alpha" submission.leaseId sequenceNumber Completed (Just 0) Nothing []))]+ <> healthFiles descriptor submission+ <> [("output/" <> path, bytes) | (path, bytes) <- nestedFiles]+ artifactFor (path, bytes) = let info = workObjectFor "application/json" bytes in Artifact path info.sha256 info.bytes "application/json"+ manifest = CellManifest submission.runId "alpha" submission.leaseId sequenceNumber now "0.1.0" (ManifestPayload submission.payload.bundle.sha256 submission.payload.storePath) Completed 3600 (map artifactFor files)+ manifestBytes = encode manifest+ put bucket path bytes = store.putObject bucket (ObjectName path) "application/json" DoesNotExist bytes >>= requireWritten+ resultPrefix = "runs/" <> renderRunId submission.runId <> "/"+ forM_ files \(path, bytes) -> put (Bucket descriptor.buckets.results) (resultPrefix <> path) bytes+ put (Bucket descriptor.buckets.results) (resultPrefix <> "manifest.json") manifestBytes+ put control (prefix <> "status.json") (encode (CellStatus submission.runId Sealed (Just sequenceNumber) now Nothing (LogChunks 0 0) (Just Completed) (Just (workObjectFor "application/json" manifestBytes).sha256) (Just [])))+ threadDelay 10000+ sealMarkers knownFault store descriptor root (Set.union seen (Set.fromList fresh))++writeRawRun :: Bool -> CellDescriptor -> Submission -> PlannedRun -> FilePath -> IO ()+writeRawRun knownFault descriptor submission run directory = do+ createDirectoryIfMissing True (directory </> "samples")+ createDirectoryIfMissing True (directory </> "series")+ Bytes.writeFile (directory </> "series/load.csv") "mono_ns,wall,phase\n1000,0,steady\n5000001000,0,drain\n"+ encodeFile (directory </> "run-spec.json") run.spec+ mutable <- Histogram.newHistogram Histogram.defaultHistogramConfig+ forM_ [1 :: Int .. 1000] (const (Histogram.recordValue mutable 1000000))+ histogram <- Histogram.freeze mutable+ let operation = "publish-to-finalize"+ base = directory </> "samples" </> Text.unpack operation+ Bytes.writeFile (base <> ".hist") (encodeHistogram histogram)+ Bytes.writeFile (base <> ".service.hist") (encodeHistogram histogram)+ writer <- openSampleWriter (base <> ".raw") (SampleHeader (Origin 0 0) MonotonicClock operation "synthetic")+ writeSampleBlock writer 0 [SampleRecord start start (start + 1000000) 0 1 | index <- [0 .. 999], let start = 1000 + index * 5000000]+ closeSampleWriter writer+ encodeFile (base <> ".meta.json") (object ["operation" .= operation, "latencyBasis" .= ("intended-start" :: Text), "rawSamples" .= ("full" :: Text), "phaseCounts" .= object ["steady" .= object ["successes" .= (1000 :: Int), "failures" .= (0 :: Int), "units" .= (1000 :: Int)]]])+ measurements <- summarizeRunDir directory >>= either (fail . show) pure+ encodeFile (directory </> "run-result.json") (object ["runId" .= run.runId, "scenario" .= run.spec.scenario, "outcome" .= (if knownFault then "failed" else "passed" :: Text), "exitCode" .= (0 :: Int), "blocking" .= False, "compatibility" .= object ["inputs" .= object ["cohortPlanHash" .= ("synthetic-control" :: Text)]], "fingerprint" .= object ["machineProfile" .= cellMachineProfile descriptor, "cell" .= object ["payload" .= object ["bundleSha256" .= submission.payload.bundle.sha256]]], "timings" .= object ["startedAt" .= addUTCTime (fromIntegral run.ordinal) (at 0)], "summaries" .= object ["measurements" .= object ["measurements" .= measurements]]])+ writeManifest directory run.runId Map.empty >>= encodeFile (directory </> "manifest.json")++healthFiles :: CellDescriptor -> Submission -> [(Text, Lazy.ByteString)]+healthFiles descriptor submission =+ [ ("cell/execution-window.json", encode (object ["schema" .= ("cell.execution-window/v1" :: Text), "runId" .= submission.runId, "resetStarted" .= at 0, "runStarted" .= at 10, "runFinished" .= at 60])),+ ("cell/health.json", encode (object ["schema" .= ("cell.health/v1" :: Text), "runId" .= submission.runId, "passed" .= True, "windowStart" .= at 0, "windowEnd" .= at 60, "gates" .= concatMap gates roles])),+ ("cell/reset-evidence.json", encode (object ["schema" .= ("cell.reset-evidence/v1" :: Text), "runId" .= submission.runId, "cell" .= ("alpha" :: Text), "cachePolicy" .= ("cold" :: Text), "verified" .= True, "startedAt" .= at 0, "finishedAt" .= at 5, "steps" .= [object ["name" .= ("stop-driver-processes" :: Text), "status" .= ("ok" :: Text), "startedAt" .= at 0, "finishedAt" .= at 5]]]))+ ]+ <> [("cell/observations/" <> role <> ".json", encode (object ["schema" .= ("cell.observations/v1" :: Text), "machine" .= machine, "image" .= image, "agentVersion" .= ("0.1.0" :: Text), "systemClosure" .= ("/nix/store/synthetic-system-identity" :: Text), "from" .= at 0, "to" .= at 60, "samples" .= [sample i | i <- [0 .. 60]], "maintenanceEvents" .= ([] :: [Value]), "samplingErrors" .= ([] :: [Value])])) | (role, machine, image) <- roles]+ where+ roles = [("driver", machine, descriptor.images.driver) | machine <- take 1 descriptor.instances.drivers] <> [("postgres", descriptor.instances.postgres, descriptor.images.postgres), ("monitoring", descriptor.instances.monitoring, descriptor.images.monitoring)]+ sample i = object ["at" .= at i, "cpuTotalTicks" .= (1000 + i * 100), "cpuIdleTicks" .= (990 + i * 99), "cpuStealTicks" .= (0 :: Int), "diskFreePercent" .= (if i <= 10 then 80 else 75 :: Int)]+ gates (role, machine, _) =+ [ gate "host-maintenance" (String "NONE") (String "NONE"),+ gate "background-load" (object ["settleCpuPercent" .= (if role == "monitoring" then 0 else 1 :: Int), "runStealCpuPercent" .= (0 :: Int)]) (object ["settleCpuMaxPercent" .= (5 :: Int), "runStealCpuMaxPercent" .= (2 :: Int)]),+ gate "storage-pressure" (object ["startFreePercent" .= (80 :: Int), "runFreePercent" .= (75 :: Int), "outputExceeded" .= False]) (object ["startFreeMinPercent" .= (20 :: Int), "runFreeMinPercent" .= (5 :: Int), "outputExceeded" .= False]),+ gate "incomplete-reset" (Bool True) (Bool True),+ gate "version-mismatch" (Bool True) (Bool True)+ ]+ where+ gate name observed threshold = object ["name" .= (name :: Text), "machine" .= machine, "status" .= ("ok" :: Text), "at" .= at 60, "observed" .= observed, "threshold" .= threshold]++at :: Int -> UTCTime+at seconds = addUTCTime (fromIntegral seconds) (read "2026-10-11 00:00:00 UTC")++readTree :: FilePath -> IO [(Text, Lazy.ByteString)]+readTree directory = do+ names <- listDirectory directory+ concat <$> forM names \name -> do+ let path = directory </> name+ regular <- doesFileExist path+ if regular+ then do bytes <- Lazy.readFile path; pure [(Text.pack name, bytes)]+ else map (\(relative, bytes) -> (Text.pack name <> "/" <> relative, bytes)) <$> readTree path++readJSON :: (FromJSON a) => FilePath -> IO a+readJSON path = eitherDecodeFileStrict' path >>= either fail pure++requireWritten :: PutOutcome -> IO ()+requireWritten (Written _) = pure ()+requireWritten PreconditionFailed = fail "synthetic protocol object already exists"++projectRoot :: IO a -> IO a+projectRoot action = do+ root <- doesFileExist "cabal.project"+ withCurrentDirectory (if root then "." else "..") action++withSetting :: String -> String -> IO a -> IO a+withSetting name value action = bracket (lookupEnv name) restore \_ -> setEnv name value >> action+ where+ restore Nothing = unsetEnv name+ restore (Just previous) = setEnv name previous
+ test/fixtures/perf-publication.synthetic.json view
@@ -0,0 +1,11 @@+{+ "schema": "kenshou.perf-publication/v1",+ "snapshot": "snapshots/00000000-0000-7000-8000-000000000000",+ "manifestUri": "gs://synthetic-evidence/performance/sessions/00000000-0000-7000-8000-000000000000/snapshots/00000000-0000-7000-8000-000000000000/manifest.json",+ "manifestSha256": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",+ "resultSha256": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",+ "payloadsReadyAt": "2026-10-11T00:00:00Z",+ "publicationStartedAt": "2026-10-11T00:00:05Z",+ "verifiedReportAt": "2026-10-11T00:00:10Z",+ "totalElapsedSeconds": 10+}
+ test/golden/help-evidence-80.txt view
@@ -0,0 +1,75 @@+# Verification evidence++`kenshou record RUN-DIR --data-base-uri gs://BUCKET/PREFIX --purpose release+--project PROJECT` publishes a finished run's files and adds one record to the+OKF verification bundle. The record names the run, cohort, outcome, and+SHA-256-pinned object links. It contains no measurements; those remain in the+linked run files.++`kenshou record --comparison FILE --data-base-uri gs://BUCKET/PREFIX --purpose+release --project PROJECT` records a `kenshou.comparison/v1` document. Every+baseline and candidate run named by the comparison must already have a run+record in the bundle. The comparison record links the source document by digest+and references those run concepts. Supply either `RUN-DIR` or `--comparison+FILE`.++The default bundle is `docs/verification`. `--bundle DIR` selects another+bundle. A real GCS upload requires `--project PROJECT`, and the command checks+that the active gcloud project matches it. `--verify-only` checks that all+objects already exist; `--deep-verify` downloads them and checks their bytes.+Failed or inconclusive runs link logs automatically; `--link-logs` also links+logs for a passing run.++`--config PATH` reads ordered YAML configuration. The supported keys are+`evidence.bundle-root`, `gcp.project`, and `evidence.data-base-uri`; the+corresponding environment variables are `KENSHOU_EVIDENCE_BUNDLE`,+`KENSHOU_GCP_PROJECT`, and `KENSHOU_EVIDENCE_DATA_BASE_URI`. Named flags take+precedence over the environment and files. `kenshou evidence check` uses the+same bundle default and configuration sources. Unknown keys are rejected.++A dirty harness requires `--allow-dirty` and is recorded with `purpose:+investigation` regardless of the requested purpose. Use `--subject MORI-URI`+with `--subject-kind project|package` when the default layer project is too+broad. Repeat `--produced MORI-URI` for reports caused by this run.++For a local rehearsal, copy the verification bundle outside the repository and+pass `--bundle COPY --store-root DIR`. The record still contains `gs://` URIs,+while the bytes are stored under the scratch directory. The command refuses+`--store-root` with a bundle inside this repository.++`--json` prints one `kenshou.record-result/v1` document and sends errors to+standard error. A repeat with the same record content succeeds and reports+`already recorded`; different content at the same record path is a conflict.++`kenshou evidence check` checks record paths, IDs, digest and revision shapes,+textual fields, event-only keys, timestamps, required data links, comparison arm+references and verdicts, and changes to committed records. `--base REF` limits+the Git history span. `--network --project PROJECT` checks object size and+SHA-256 metadata, and `--deep` downloads bytes to verify each digest. The+command exits 0 when clean, 1 for findings, and 4 when it cannot check. `--json`+prints one `kenshou.evidence-check/v1` document.++`kenshou history --scenario LAYER/COMPONENT/KIND/NAME --json` reads matching+records in time order and emits `kenshou.evidence-history/v1`. Use+`--cohort-component PACKAGE[=VERSION|REVISION]`, repeatable `--outcome`,+`--since YYYY-MM-DD`, or `--confirmed-only` to narrow the series. Each entry+includes linked attestations, a derived trust tier, and the latest earlier+confirmed baseline with the same scenario and compatibility key. The history+view does not modify the bundle.++`kenshou attest RECORD-OR-ID --project PROJECT` fetches the recorded run or+comparison data and writes an immutable attestation with six named checks. For a+comparison, it verifies both recorded arms and replays the paired algorithm from+their samples under the stored policy. `--offline` skips revision resolution+that needs network access; `--linked-only` checks only directly linked objects.+It reports `refuted` when fetched data contradicts the record and `incomplete`+when a check cannot establish the claim. A scenario oracle must be registered+before `verdict-recomputed` can pass for an ordinary run, so generic runs+currently remain incomplete. Exit codes are 0 for confirmed, 1 for refuted, 3+for incomplete, 2 for invalid input, and 4 for an operational failure. `--json`+emits one `kenshou.attest-result/v1` result document.++A person may annotate an incomplete or refuted attestation with+`--accept-anomaly --authority human:ID --reason TEXT` from an interactive+terminal. The computed verdict does not change. The command rejects this action+in CI or without a terminal.