diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,148 @@ ## Unreleased +## 0.2.0.0 — 2026-08-07 + +### Added + +- **Generated bundles target OKF v0.2.** The bundle root's `index.md` declares + `okf_version: "0.2"` and every concept records its producer in the v0.2 + `generated` family (`generated.by: process:shikumi-okf`), which is what + `okf validate --strict` asks for since okf-core 0.5. `okf validate --strict + --profile profile/shikumi.dhall --profile-enforce` passes on the shipped + example bundle. + +- `GenerateOptions`, `defaultGenerateOptions`, `defaultGeneratedBy`, and + `okfVersion02` in `Shikumi.Okf.Generate`. Build options by overriding + `defaultGenerateOptions` rather than as a record literal, so a field added in a + later release leaves the call site compiling. `generated = Nothing` writes no + provenance at all — this generator will not claim authorship on a caller's + behalf — and `okfVersion = Nothing` preserves whatever declaration the + destination already carries rather than walking a hand-migrated bundle back. + +- `profile/shikumi.dhall` is now shipped as a Cabal `data-files` entry, so + installing this package also installs the profile its bundles are meant to + satisfy. + +### Changed + +- **BREAKING** Upgraded `okf-core` from `^>=0.3.0.0` to `^>=0.5.0.0`, the release + that implements OKF v0.2. + +- **BREAKING** `generateBundle`, `writeProgramBundle`, `programConcept`, and + `appConcept` take a `GenerateOptions` where they took a `Maybe Text` + timestamp. Pass `defaultGenerateOptions` for the previous call shape; move a + timestamp you were passing into + `defaultGenerateOptions {generated = Just (Generated actor (Just t))}`. + + The parameter could not simply be reinterpreted. It wrote the v0.1 `timestamp` + key, which OKF v0.2 supersedes with `generated`, and a bundle that declares + v0.2 while still carrying `timestamp` is reported by okf-core 0.5 as retaining + a superseded key. Generation still reads no clock: `generated.at` remains + caller-supplied and absent by default, so regenerating an unchanged manifest is + still byte-identical and the regenerate-and-diff CI check still works. + +- **BREAKING** `profile/shikumi.dhall` targets OKF v0.2. It sets + `okfVersion = "0.2"` and `requireBundleVersion = Some "0.2"`, so a bundle that + declares no version — or an older one — is now a profile deviation. It replaces + the `timestamp` rule with a `generated` object rule constraining `by` (required, + `actor` format) and `at` (optional, RFC 3339 UTC). Because `allowUnknownFields` + is `False`, a stray `timestamp` is now reported as an undeclared key. + + The descriptor needs `okf` `>=0.5` to load: `requireBundleVersion` and + `objectFields` do not exist in the 0.3 schema. + +- **`profile/shikumi.dhall` imports the okf schema from a pinned URL** — + `https://raw.githubusercontent.com/shinzui/okf/v0.5.0.0/okf-core/dhall/package.dhall` + with a `sha256:` integrity hash — instead of the relative path + `../../../okf/okf-core/dhall/package.dhall` into a sibling checkout. + + The relative path resolved against whatever happened to be in a developer's + working tree. That is how this descriptor silently stopped type-checking when + okf's schema moved ahead of the pinned `okf-core` release, and it could never + have worked for anyone installing this package from Hackage, where no sibling + checkout exists. The hash covers the fully resolved, normalized expression, so + it pins every transitive `defaults/*` and `mk/*` file too, and a moved tag + fails the load rather than changing the schema underneath the profile. Dhall + serves a hashed import from `~/.cache/dhall` once resolved, so only the first + load on a machine touches the network. + +- The committed `example/out` bundle was regenerated: the root index gained its + version declaration and each concept gained `generated`. + +### Fixed + +- The test suite parses one Markdown document on the main thread before tasty + forks. `cmark-gfm`'s `commonmarkToNode` calls + `cmark_gfm_core_extensions_ensure_registered` inside `unsafePerformIO` on every + call, whatever extension list it is given, and that C function is not + thread-safe: two threads reaching it at once make `cmark_register_node_flag` + print `flag initialization error` and `abort()` the process. okf-core 0.5 made + this reachable by enabling footnotes and routing all three of its parse sites + through one options list, so validating a bundle now parses far more Markdown + than it used to, and this suite runs with `-N`. The result was a reproducible + SIGABRT partway through the run — not a test failure, so it reported as an + empty log rather than as a diagnosis. + + This is a workaround for a defect in the `cmark-gfm` bindings + (`mori://kivikakk/cmark-gfm-hs`), not in okf or shikumi, and it protects only + this test suite. A consumer that reads OKF bundles from several threads can hit + the same abort; the durable fix is to make the registration thread-safe + upstream. + +- The test suite now checks the real `profile/shikumi.dhall` instead of a Haskell + paraphrase of it. A new `Profile` group loads the descriptor (located through + `Paths_shikumi_okf.getDataFileName`, so it does not depend on the directory + `cabal` was invoked from), compiles it, and validates the generated bundle + against it under `PermissiveConformance` — the same mode + `okf validate --profile --profile-enforce` uses. A descriptor that stops + type-checking, stops compiling, or stops describing what the generator emits is + now a red test; previously it could rot unnoticed across releases, which is + exactly what had happened. The hermetic `Conformance` group it replaces asserted + the same conventions against an in-process copy and is gone. +- **Breaking for profile consumers.** The profile now checks what frontmatter + values *contain*, not only that keys are present. `type`, `title`, + `description`, and `resource` are declared `Cardinality.Scalar`, so a + list-valued `title` is a violation. `timestamp` is declared as a recommended + scalar with the `Rfc3339Utc` format, so a value like `yesterday` is a violation; + cardinality and format are checked in every validation mode, so declaring it + *recommended* buys the format check without demanding the key be present. +- **Breaking for profile consumers.** `allowUnknownFields = False` closes the + top-level key set. This cannot reject a generated document — a closed set still + always admits the six core OKF keys, which are exactly what the generator emits + — but a hand-added stray key in a generated bundle is now reported. +- **Breaking for profile consumers.** `resource` moved from the profile-wide + `recommended` list into each type's own `required` list and carries + `UriWithScheme "shikumi"`, so it is now required on both document kinds and its + value must parse as an absolute `shikumi://` URI. `tags` is recommended as a + `List` on `Shikumi Program` only. +- **Consequence for consumers.** `okf validate --strict` now reports a + missing-recommended `timestamp` on bundles generated without one. That is + correct for a strict authoring run: `timestamp` is optional by design, because + `generateBundle` takes it as an explicit argument and omitting it is what makes + regeneration byte-identical. Ordinary (non-`--strict`) validation is unaffected, + and the committed `example/out` bundle still validates and regenerates + byte-identically. +- Moved the `okf-core` bound from `^>=0.1.0.0` to `^>=0.3.0.0`, the newest release + on Hackage. No source changes: the producer API this package uses (`Okf.Bundle`, + `Okf.ConceptId`, `Okf.Document`, `Okf.Index`) is unchanged across both major + bumps, and the breaking changes in `0.2`/`0.3` are confined to the profile + compilation and validation API this package does not call. +- **Breaking for profile consumers.** Migrated `profile/shikumi.dhall` to the + `0.3.0.0` profile schema; reading it now requires `okf` `>=0.3`. The schema made + `frontmatter.required`/`recommended` `List FieldRule` instead of `List Text`, + added `description`, `allowUnknownFields`, and `idField` to `Profile`, and added + `description`, `frontmatter`, and `idPrefix` to `TypeRule`. The descriptor now + imports okf's published `dhall/package.dhall` entry point and uses + record-completion defaults (`::`) with the `mk.FieldRule` constructors, so later + additive, defaulted schema fields will not break it again. +- The migrated profile keeps its previous meaning exactly — closed types, the same + two required and two recommended keys, open frontmatter, no `idField`, and + default `Any` cardinality with no `format` — and now also documents each key and + concept type in prose, which the old schema had no field for. Checked with `okf` + `0.3.0.0` against the committed `example/out` bundle: `validate` and + `--profile-enforce` both exit 0 and regeneration stays byte-identical. + ## 0.1.0.1 - 2026-07-05 ### Changed diff --git a/example/Main.hs b/example/Main.hs --- a/example/Main.hs +++ b/example/Main.hs @@ -11,16 +11,17 @@ -- > cabal run shikumi-okf-example -- -- then validate the result with the standalone @okf@ CLI (see the package docs and --- EP-31). Because no timestamp is passed, regenerating an unchanged manifest yields --- byte-identical output, so the committed @example/out@ tree can be regenerated and --- diffed in CI. +-- EP-31). It uses 'defaultGenerateOptions', which declares OKF v0.2 and records +-- @process:shikumi-okf@ as the producer with no generation time — so regenerating +-- an unchanged manifest yields byte-identical output and the committed +-- @example/out@ tree can be regenerated and diffed in CI. module Main (main) where import Data.Text (Text) import GHC.Generics (Generic) import Shikumi.Adapter (ToPrompt) import Shikumi.Module (predict) -import Shikumi.Okf.Generate (writeProgramBundle) +import Shikumi.Okf.Generate (defaultGenerateOptions, writeProgramBundle) import Shikumi.Okf.Types ( AppInfo (..), ProgramDoc (..), @@ -100,7 +101,7 @@ let root = case args of (dir : _) -> dir [] -> "out" - result <- writeProgramBundle root exampleApp Nothing manifest + result <- writeProgramBundle root exampleApp defaultGenerateOptions manifest case result of Left err -> error ("bundle generation failed: " <> show err) Right () -> putStrLn ("Wrote OKF bundle to " <> root) diff --git a/profile/shikumi.dhall b/profile/shikumi.dhall new file mode 100644 --- /dev/null +++ b/profile/shikumi.dhall @@ -0,0 +1,185 @@ +-- OKF profile for shikumi program documentation bundles. +-- +-- A profile is a small Dhall descriptor that layers house conventions on top of +-- OKF: which `type` strings are allowed, which frontmatter keys are required, the +-- `resource:` URI scheme, and the file layout. `okf validate --profile` (and +-- mori, at register/observe time) check a generated bundle against it, so every +-- application's program documentation stays consistent. Profiles are NOT part of +-- the OKF standard; a bundle that deviates remains fully OKF-conformant. +-- +-- This bundle layout is produced by the `shikumi-okf` package: one `Shikumi App` +-- concept at `apps/` linking to one `Shikumi Program` concept per program at +-- `programs/`, with `shikumi://` resource URIs. +-- +-- The schema is imported through okf's published entry point (`package.dhall`) at +-- a pinned tag, protected by a Dhall integrity hash. It is deliberately NOT a +-- relative path into a sibling `okf` checkout: that spelling resolves against +-- whatever is in someone's working tree, so this descriptor silently stopped +-- type-checking when okf's schema moved ahead of the pinned `okf-core` release, +-- and it cannot work at all for anyone who installs this package from Hackage. +-- +-- The hash covers the fully resolved, normalized expression, so it pins every +-- transitive `./Profile.dhall`, `./defaults/*`, and `./mk/*` the entry point +-- pulls in — not just the one file. Moving the tag therefore fails the load +-- loudly instead of changing the schema underneath this profile. +-- +-- Dhall serves a hashed import from `~/.cache/dhall` when it is present, so only +-- the first resolution on a machine needs the network. To move to a new okf +-- release, bump the tag and refresh the hash with: +-- +-- echo 'https://raw.githubusercontent.com/shinzui/okf//okf-core/dhall/package.dhall' | dhall hash +-- +-- Every record below is written with okf's record-completion defaults (`::`) and +-- `mk` constructors, so a future additive, defaulted schema field leaves this +-- descriptor working unchanged. +-- +-- Types are closed (`allowUnknownTypes = False`): a shikumi bundle contains only +-- the two concept types `shikumi-okf` generates. Top-level frontmatter is closed +-- too (`allowUnknownFields = False`). That is safe because a closed key set still +-- always admits the six core OKF keys — `type`, `title`, `description`, +-- `timestamp`, `resource`, `tags` — and those six are exactly what the generator +-- emits: its frontmatter comes from three helpers (`okfCommon`, `setResource`, +-- `setTags`) and there is no mechanism for an author to inject other keys. So +-- closing the set cannot reject a generated document; what it catches is a +-- hand-added stray key in a bundle that is supposed to be generated. No `idField` +-- is declared, so document-handle checks stay off: shikumi concepts are addressed +-- by concept id and `shikumi://` resource, not by stable handles. +-- +-- Value shapes are checked, not just key presence: every single-valued key is +-- declared `Cardinality.Scalar`, so `title: [a, b]` is a violation rather than a +-- silent pass. +-- +-- This profile targets OKF v0.2 and requires its bundles to say so +-- (`requireBundleVersion = Some "0.2"`). The specification makes the root +-- `okf_version` declaration a MAY, so okf itself never asks for it — and an +-- undeclared bundle quietly opts out of every v0.2-only check, including the +-- report of concepts still carrying the superseded `timestamp`. `shikumi-okf` +-- writes the declaration on every bundle it generates, so requiring it here +-- costs a conformant producer nothing and catches a bundle that drifted back. +-- +-- Provenance is the v0.2 `generated` family rather than the v0.1 `timestamp` +-- key, which this profile no longer declares at all: `allowUnknownFields` is +-- `False`, so a stray `timestamp` is now reported as an undeclared key on top of +-- whatever the core v0.2 checks say about it. `generated` sits under +-- `recommended` rather than `required` because a caller may pass +-- `generated = Nothing` to disclaim provenance rather than have the generator +-- invent it; the consequence is that `okf validate --strict` advises on such a +-- bundle while an ordinary run does not. +-- +-- Rules that are true of one document kind only live in that kind's own +-- `frontmatter` record, which okf merges with the profile-wide one. `resource` is +-- declared there rather than profile-wide: the generator always emits it for both +-- kinds, so per-type `required` states the truth precisely, and declaring the same +-- key in both scopes would yield two overlapping presence clauses for no gain. Its +-- `UriWithScheme "shikumi"` format parses the value, so `resource: not-a-uri` is a +-- violation; that overlaps harmlessly with the `resourceScheme` check each type +-- rule already performs. `tags` is recommended on `Shikumi Program` only, as a +-- `List`, because only program documents carry it and only when the author +-- declared any. +let okf = + https://raw.githubusercontent.com/shinzui/okf/v0.5.0.0/okf-core/dhall/package.dhall + sha256:02a821061043976b0ec0d60745a792f5f536e5f5d0db43bc990890ab0f5af0e3 + +let field = okf.mk.FieldRule + +let nested = okf.mk.NestedFieldRule + +let scalarOf = + \(name : Text) -> + \(prose : Text) -> + \(fmt : Optional okf.FieldFormat) -> + okf.defaults.FieldRule::{ + , field = name + , description = Some prose + , cardinality = okf.Cardinality.Scalar + , format = fmt + } + +-- OKF v0.2 §5.2: `by` is REQUIRED within the `generated` mapping, `at` is +-- optional. `at` is declared `optional` rather than `recommended` because +-- shikumi-okf omits it unless the caller supplies a time — that omission is what +-- keeps regenerating an unchanged manifest byte-identical — so demanding it +-- under `--strict` would advise against the generator's own determinism. +let generatedMembers = + okf.defaults.NestedRules::{ + , required = + [ nested.actor "by" + // { description = Some + "§7. The actor that produced this document: `/`, `human:`, or `process:`." + } + ] + , optional = + [ nested.rfc3339Utc "at" + // { description = Some + "When the document was generated, if the caller supplied a time." + } + ] + } + +let appType = + okf.defaults.TypeRule::{ + , type = "Shikumi App" + , description = Some + "One application that ships shikumi programs; links to every program it exposes." + , frontmatter = okf.defaults.FrontmatterRules::{ + , required = + [ scalarOf + "resource" + "The `shikumi:///` URI this concept documents." + (Some (okf.FieldFormat.UriWithScheme "shikumi")) + ] + } + , pathPattern = Some "apps/*" + , resourceScheme = Some "shikumi" + } + +let programType = + okf.defaults.TypeRule::{ + , type = "Shikumi Program" + , description = Some + "One shikumi program: its signature, adapter, and declared metadata." + , frontmatter = okf.defaults.FrontmatterRules::{ + , required = + [ scalarOf + "resource" + "The `shikumi:////programs/` URI this concept documents." + (Some (okf.FieldFormat.UriWithScheme "shikumi")) + ] + , recommended = [ field.list "tags" ] + } + , pathPattern = Some "programs/*" + , resourceScheme = Some "shikumi" + } + +in okf.defaults.Profile::{ + , name = "shinzui-shikumi" + , description = Some + "House conventions for OKF bundles generated from shikumi programs by the shikumi-okf package." + , okfVersion = "0.2" + , requireBundleVersion = Some "0.2" + , frontmatter = okf.defaults.FrontmatterRules::{ + , required = + [ scalarOf + "type" + "The OKF concept type: `Shikumi App` or `Shikumi Program`." + (None okf.FieldFormat) + , scalarOf + "title" + "Human-readable name of the app or program." + (None okf.FieldFormat) + ] + , recommended = + [ scalarOf + "description" + "What the app or program is for, in one or two sentences." + (None okf.FieldFormat) + , field.record "generated" generatedMembers + // { description = Some + "OKF v0.2 §5.2. How this document was produced. `shikumi-okf` writes `by: process:shikumi-okf` on every concept it generates. Supersedes the v0.1 `timestamp` key, which this profile no longer declares: a bundle declaring `okf_version: \"0.2\"` that still carried `timestamp` would be reported as retaining a superseded key." + } + ] + } + , allowUnknownTypes = False + , allowUnknownFields = False + , types = [ appType, programType ] + } diff --git a/shikumi-okf.cabal b/shikumi-okf.cabal --- a/shikumi-okf.cabal +++ b/shikumi-okf.cabal @@ -1,6 +1,6 @@ cabal-version: 3.4 name: shikumi-okf -version: 0.1.0.1 +version: 0.2.0.0 synopsis: Generate OKF documentation bundles from shikumi programs (EP-31) @@ -24,6 +24,11 @@ build-type: Simple extra-doc-files: CHANGELOG.md +-- Shipped so consumers get the profile their bundles are checked against, and so +-- the test suite can locate it via Paths_shikumi_okf.getDataFileName regardless +-- of the directory cabal was invoked from. +data-files: profile/shikumi.dhall + common common-options ghc-options: -Wall -Wcompat -Widentities -Wincomplete-uni-patterns @@ -48,7 +53,7 @@ build-depends: , base >=4.20 && <5 - , okf-core ^>=0.1.0.0 + , okf-core ^>=0.5.0.0 , shikumi ^>=0.3.0.0 , text ^>=2.1 @@ -59,23 +64,25 @@ build-depends: , base , shikumi ^>=0.3.0.0 - , shikumi-okf ^>=0.1.0.1 + , shikumi-okf ^>=0.2.0.0 , text test-suite shikumi-okf-test - import: common-options - type: exitcode-stdio-1.0 - hs-source-dirs: test - main-is: Main.hs - ghc-options: -threaded -with-rtsopts=-N + import: common-options + type: exitcode-stdio-1.0 + hs-source-dirs: test + main-is: Main.hs + other-modules: Paths_shikumi_okf + autogen-modules: Paths_shikumi_okf + ghc-options: -threaded -with-rtsopts=-N build-depends: , base , directory , effectful , filepath - , okf-core ^>=0.1.0.0 + , okf-core ^>=0.5.0.0 , shikumi ^>=0.3.0.0 - , shikumi-okf ^>=0.1.0.1 + , shikumi-okf ^>=0.2.0.0 , tasty , tasty-hunit , text diff --git a/src/Shikumi/Okf/Generate.hs b/src/Shikumi/Okf/Generate.hs --- a/src/Shikumi/Okf/Generate.hs +++ b/src/Shikumi/Okf/Generate.hs @@ -7,11 +7,23 @@ -- only on an invalid concept name (an author error in the manifest), surfaced as -- 'GenerateError'; writing performs IO through the @okf-core@ producer API. -- --- Generation is deterministic: the timestamp is an explicit argument, never read --- from the wall clock, so regenerating an unchanged manifest yields byte-identical --- output and a "regenerate and diff" check stays meaningful. +-- The bundles target __OKF v0.2__: the root @index.md@ declares +-- @okf_version: "0.2"@ and every concept records its producer in the v0.2 +-- @generated@ family, which is what strict validation now asks for. Both come +-- from 'GenerateOptions'; start from 'defaultGenerateOptions' and override, +-- rather than building the record literally, so a later field with a default +-- leaves your call site working. +-- +-- Generation is deterministic: it never reads the wall clock. @generated.at@ is +-- caller-supplied and absent by default, so regenerating an unchanged manifest +-- yields byte-identical output and a "regenerate and diff" check stays +-- meaningful. module Shikumi.Okf.Generate ( GenerateError (..), + GenerateOptions (..), + defaultGenerateOptions, + defaultGeneratedBy, + okfVersion02, programConceptId, appConceptId, programConcept, @@ -25,17 +37,20 @@ import Data.Maybe (fromMaybe) import Data.Text (Text) import Data.Text qualified as T +import Okf.Actor (Actor (ProcessActor)) import Okf.Bundle (BundleError, Concept, conceptFromDocument, writeBundle) import Okf.ConceptId (ConceptId, ConceptIdError, parseConceptId, renderConceptLink) import Okf.Document ( Frontmatter, + Generated (..), OKFDocument (..), OkfCommon (..), okfCommon, + setGenerated, setResource, setTags, ) -import Okf.Index (writeBundleIndexes) +import Okf.Index (OkfVersion (..), writeBundleIndexesWith) import Shikumi.Okf.Render (renderProgramBody) import Shikumi.Okf.Types (AppInfo (..), ProgramDoc (..), ProgramManifest (..)) @@ -48,6 +63,54 @@ IndexWriteError BundleError deriving stock (Eq, Show) +-- | What the generator records about the bundle itself, as opposed to about the +-- programs in it. +-- +-- Build one by overriding 'defaultGenerateOptions' (@defaultGenerateOptions +-- {generatedAt = Just "2026-08-07T00:00:00Z"}@) rather than as a record literal: +-- a field added here in a later release keeps an overriding call site compiling. +data GenerateOptions = GenerateOptions + { -- | The OKF v0.2 @generated@ family written on every concept: who or what + -- produced it, and optionally when. 'Nothing' omits the key entirely, + -- which leaves the bundle without the provenance @okf validate --strict@ + -- asks for. + generated :: !(Maybe Generated), + -- | The version declared in the bundle root's @index.md@. 'Nothing' + -- preserves whatever declaration the destination already carries, so a + -- bundle hand-migrated to a later version is not silently walked back. + okfVersion :: !(Maybe OkfVersion) + } + +-- | Declare OKF v0.2 and record @process:shikumi-okf@ as the producer, with no +-- generation time. +-- +-- Time is absent by default on purpose: it is the only part of this record that +-- would differ between two runs over the same manifest, and leaving it out is +-- what keeps @regenerate && git diff --exit-code@ usable as a drift check. +defaultGenerateOptions :: GenerateOptions +defaultGenerateOptions = + GenerateOptions + { generated = Just Generated {generatedBy = defaultGeneratedBy, generatedAt = Nothing}, + okfVersion = Just okfVersion02 + } + +-- | The producer 'defaultGenerateOptions' names: @process:shikumi-okf@. +-- +-- Deliberately version-free. A version here would change the bytes of every +-- generated document on every shikumi-okf release, which would make the +-- regenerate-and-diff check report drift that is not drift. +defaultGeneratedBy :: Actor +defaultGeneratedBy = ProcessActor "shikumi-okf" + +-- | OKF v0.2, the dialect these bundles are written in. +-- +-- Pinned rather than taken from @okf-core@'s 'Okf.Index.supportedOkfVersion': +-- declaring a version is a claim about what this generator emits, so it must +-- move when the generator learns a new dialect's fields, not when the library +-- it links against learns to read one. +okfVersion02 :: OkfVersion +okfVersion02 = OkfVersion {okfVersionMajor = 0, okfVersionMinor = 2} + -- | The concept id of a program: @programs/@. programConceptId :: ProgramDoc -> Either GenerateError ConceptId programConceptId doc = mkConceptId ("programs/" <> name doc) @@ -60,26 +123,26 @@ mkConceptId raw = first (InvalidConceptName raw) (parseConceptId raw) -- | Build the @Shikumi Program@ concept for one documented program. -programConcept :: AppInfo -> Maybe Text -> ProgramDoc -> Either GenerateError Concept -programConcept app timestamp doc = do +programConcept :: AppInfo -> GenerateOptions -> ProgramDoc -> Either GenerateError Concept +programConcept app opts doc = do cid <- programConceptId doc let resource = "shikumi://" <> appNamespace app <> "/" <> appName app <> "/programs/" <> name doc frontmatter = - applyTags (tags doc) . setResource resource $ + applyTags (tags doc) . applyGenerated opts . setResource resource $ okfCommon OkfCommon { commonType = "Shikumi Program", commonTitle = Just (fromMaybe (name doc) (title doc)), commonDescription = description doc, - commonTimestamp = timestamp + commonTimestamp = Nothing } okfDoc = OKFDocument {frontmatter, body = renderProgramBody doc} pure (conceptFromDocument cid okfDoc) -- | Build the @Shikumi App@ concept that links to every program. -appConcept :: AppInfo -> Maybe Text -> ProgramManifest -> Either GenerateError Concept -appConcept app timestamp (ProgramManifest docs) = do +appConcept :: AppInfo -> GenerateOptions -> ProgramManifest -> Either GenerateError Concept +appConcept app opts (ProgramManifest docs) = do cid <- appConceptId app links <- traverse programLink docs let appLabel = fromMaybe (appName app) (appTitle app) @@ -90,13 +153,13 @@ <> ["", "## Programs", ""] <> links frontmatter = - setResource resource $ + applyGenerated opts . setResource resource $ okfCommon OkfCommon { commonType = "Shikumi App", commonTitle = Just appLabel, commonDescription = appDescription app, - commonTimestamp = timestamp + commonTimestamp = Nothing } okfDoc = OKFDocument {frontmatter, body = T.unlines bodyLines} pure (conceptFromDocument cid okfDoc) @@ -106,23 +169,24 @@ pure ("- " <> renderConceptLink pcid (fromMaybe (name doc) (title doc))) -- | All concepts for a manifest: the app concept first, then one per program. -generateBundle :: AppInfo -> Maybe Text -> ProgramManifest -> Either GenerateError [Concept] -generateBundle app timestamp manifest = do - appC <- appConcept app timestamp manifest - programCs <- traverse (programConcept app timestamp) (entries manifest) +generateBundle :: AppInfo -> GenerateOptions -> ProgramManifest -> Either GenerateError [Concept] +generateBundle app opts manifest = do + appC <- appConcept app opts manifest + programCs <- traverse (programConcept app opts) (entries manifest) pure (appC : programCs) --- | Generate the bundle and write it to @root@, then write its @index.md@ files. --- Returns the first error encountered (an invalid name before any IO, or an index --- write failure after the concept files are written). +-- | Generate the bundle and write it to @root@, then write its @index.md@ files +-- with the OKF version declaration from 'GenerateOptions'. Returns the first +-- error encountered (an invalid name before any IO, or an index write failure +-- after the concept files are written). writeProgramBundle :: - FilePath -> AppInfo -> Maybe Text -> ProgramManifest -> IO (Either GenerateError ()) -writeProgramBundle root app timestamp manifest = - case generateBundle app timestamp manifest of + FilePath -> AppInfo -> GenerateOptions -> ProgramManifest -> IO (Either GenerateError ()) +writeProgramBundle root app opts manifest = + case generateBundle app opts manifest of Left err -> pure (Left err) Right concepts -> do writeBundle root concepts - indexResult <- writeBundleIndexes root + indexResult <- writeBundleIndexesWith (okfVersion opts) root pure (first IndexWriteError indexResult) -- | Attach a @tags@ field only when there is at least one tag, so a program with @@ -130,3 +194,11 @@ applyTags :: [Text] -> Frontmatter -> Frontmatter applyTags [] frontmatter = frontmatter applyTags ts frontmatter = setTags ts frontmatter + +-- | Attach the @generated@ family when the caller asked for one. A caller who +-- passed 'Nothing' gets no key rather than an invented producer: OKF §5.2 makes +-- @generated.by@ a claim about who wrote the content, and this generator cannot +-- make that claim on someone else's behalf. +applyGenerated :: GenerateOptions -> Frontmatter -> Frontmatter +applyGenerated opts frontmatter = + maybe frontmatter (`setGenerated` frontmatter) (generated opts) diff --git a/test/Main.hs b/test/Main.hs --- a/test/Main.hs +++ b/test/Main.hs @@ -8,19 +8,54 @@ -- is thin and the declared metadata carries the documentation). module Main (main) where -import Control.Exception (IOException, catch) +import Control.Exception (IOException, catch, evaluate) +import Data.Foldable (toList) import Data.List (sort) import Data.Maybe (mapMaybe) import Data.Text (Text) import Data.Text qualified as T import GHC.Generics (Generic) -import Okf.Bundle (conceptIdOf, conceptResource, conceptType, walkBundle) +import Okf.Actor (Actor (ProcessActor)) +import Okf.Bundle + ( BundleInventory, + Concept, + bundleInventoryOfConcepts, + conceptDocument, + conceptGenerated, + conceptIdOf, + walkBundle, + ) import Okf.ConceptId (ConceptId, parseConceptId, renderConceptId) +import Okf.Document (Generated (..), OKFDocument (..), frontmatterLookup) import Okf.Graph (Edge (..), Graph (..), buildGraph) -import Okf.Validation (ValidationProfile (PermissiveConformance), validateBundle) +import Okf.Index + ( OkfVersion (..), + VersionDeclaration (VersionDeclared, VersionUndeclared), + readBundleVersion, + ) +import Okf.Markdown (computationBlocks) +import Okf.Profile + ( CompiledProfile, + compileProfile, + loadProfileFile, + validateProfile, + validateProfileVersion, + ) +import Okf.Validation + ( ValidationProfile (PermissiveConformance, StrictAuthoring), + validateBundle, + ) +import Paths_shikumi_okf (getDataFileName) import Shikumi.Adapter (ToPrompt) import Shikumi.Module (predict) -import Shikumi.Okf.Generate (generateBundle, writeProgramBundle) +import Shikumi.Okf.Generate + ( GenerateOptions (..), + defaultGenerateOptions, + defaultGeneratedBy, + generateBundle, + okfVersion02, + writeProgramBundle, + ) import Shikumi.Okf.Render (renderProgramBody) import Shikumi.Okf.Types ( AppInfo (..), @@ -196,8 +231,39 @@ -- --------------------------------------------------------------------------- main :: IO () -main = defaultMain tests +main = do + warmUpMarkdown + defaultMain tests +-- | Parse one Markdown document on the main thread before tasty forks. +-- +-- @cmark-gfm@ registers its core extensions lazily: @CMarkGFM.commonmarkToNode@ +-- calls @cmark_gfm_core_extensions_ensure_registered@ inside 'unsafePerformIO' +-- on every call, whatever extension list it was given, and that C function is +-- not thread-safe. Two tasty threads reaching it at once make +-- @cmark_register_node_flag@ print @flag initialization error@ and @abort()@ the +-- process — no test failure, just SIGABRT partway through the run. +-- +-- okf-core 0.5 made this reachable: it enabled footnotes and routed all three of +-- its parse sites through one options list, so validating a bundle now parses +-- far more Markdown than it used to, and this suite runs with @-N@. One parse +-- here does the registration once, single-threaded, so the tests below race over +-- an already-initialised library. +-- +-- __This is not redundant with the fork.__ @cabal.project@ pins +-- @shinzui/cmark-gfm-hs@, which fixes the registration properly, but that pin +-- governs builds of this repository only: a consumer building @shikumi-okf@ +-- from Hackage resolves stock @cmark-gfm@ and needs this warm-up. Delete it only +-- when the fix is released upstream. +-- +-- The defect is in the @cmark-gfm@ bindings, not in okf or shikumi. Full +-- mechanism: @mori://kivikakk/cmark-gfm-hs@, upstream-issues entry +-- @cmark-gfm-hs-unsafe-concurrent-extension-registration@. +warmUpMarkdown :: IO () +warmUpMarkdown = do + _ <- evaluate (length (computationBlocks "# Computation\n\n warm up\n")) + pure () + tests :: TestTree tests = testGroup @@ -226,42 +292,110 @@ testGroup "Generate" [ testCase "generated bundle has no validation errors" $ - case generateBundle demoApp Nothing demoManifest of + withConcepts $ \concepts -> + validateBundle PermissiveConformance declaredV02 (inventoryOf concepts) concepts @?= [], + testCase "one app concept plus one per program" $ + withConcepts $ \concepts -> do + let ids = map (renderConceptId . conceptIdOf) concepts + assertEqual + "concept ids" + ["apps/demo", "programs/qa", "programs/noop-summary", "programs/qa-polished"] + ids + ], + -- OKF v0.2 (okf-core 0.5) moved provenance from the v0.1 `timestamp` key + -- to the `generated` family and gave a bundle a way to declare the dialect + -- it targets. These pin both halves: what goes into every concept, and + -- what goes into the bundle root. + testGroup + "OKF v0.2" + [ testCase "every concept records process:shikumi-okf as its producer" $ + withConcepts $ \concepts -> + assertEqual + "generated family" + (map (const (Just (Generated (ProcessActor "shikumi-okf") Nothing))) concepts) + (map conceptGenerated concepts), + testCase "defaultGeneratedBy is the actor written" $ + defaultGeneratedBy @?= ProcessActor "shikumi-okf", + testCase "no concept carries the superseded v0.1 timestamp key" $ + withConcepts $ \concepts -> + assertEqual + "timestamp keys" + [] + [ renderConceptId (conceptIdOf c) + | c <- concepts, + let OKFDocument {frontmatter} = conceptDocument c, + Just _ <- [frontmatterLookup "timestamp" frontmatter] + ], + -- The strict pass is what asks for `generated`, and what reports a + -- v0.1 key surviving in a bundle that declares v0.2. A permissive run + -- would stay silent on both, so it could not notice this regressing. + testCase "strict validation of a v0.2-declaring bundle is clean" $ + withConcepts $ \concepts -> + validateBundle StrictAuthoring declaredV02 (inventoryOf concepts) concepts @?= [], + testCase "generated.at is omitted by default and written when supplied" $ do + let opts = + defaultGenerateOptions + { generated = Just (Generated (ProcessActor "shikumi-okf") (Just "2026-08-07T00:00:00Z")) + } + case generateBundle demoApp opts demoManifest of Left err -> fail ("generateBundle failed: " <> show err) Right concepts -> - validateBundle PermissiveConformance concepts @?= [], - testCase "one app concept plus one per program" $ - case generateBundle demoApp Nothing demoManifest of - Left err -> fail (show err) - Right concepts -> do - let ids = map (renderConceptId . conceptIdOf) concepts assertEqual - "concept ids" - ["apps/demo", "programs/qa", "programs/noop-summary", "programs/qa-polished"] - ids + "generated.at" + (map (const (Just "2026-08-07T00:00:00Z")) concepts) + (map ((generatedAt =<<) . conceptGenerated) concepts), + testCase "generated = Nothing writes no provenance at all" $ do + let opts = defaultGenerateOptions {generated = Nothing} + case generateBundle demoApp opts demoManifest of + Left err -> fail ("generateBundle failed: " <> show err) + Right concepts -> + assertEqual "generated family" [] (mapMaybe conceptGenerated concepts), + testCase "the written bundle root declares okf_version 0.2" $ do + root <- freshTempDir "shikumi-okf-version" + result <- writeProgramBundle root demoApp defaultGenerateOptions demoManifest + case result of + Left err -> fail ("writeProgramBundle failed: " <> show err) + Right () -> do + declared <- readBundleVersion root + declared @?= Right (VersionDeclared (OkfVersion 0 2)), + testCase "okfVersion02 is the version declared" $ + okfVersion02 @?= OkfVersion {okfVersionMajor = 0, okfVersionMinor = 2} ], testGroup - "Conformance" - -- Hermetic invariants matching profile/shikumi.dhall (the .dhall profile - -- itself is enforced end-to-end by `okf validate --profile-enforce`; this - -- asserts the same conventions in-process without dhall or a file path). - [ testCase "types and resource scheme match the shikumi profile" $ - case generateBundle demoApp Nothing demoManifest of - Left err -> fail (show err) - Right (appC : programCs) -> do - conceptType appC @?= "Shikumi App" - mapM_ (\c -> conceptType c @?= "Shikumi Program") programCs - let resources = map conceptResource (appC : programCs) - assertBool - "every resource uses the shikumi:// scheme" - (all (maybe False ("shikumi://" `T.isPrefixOf`)) resources) - Right [] -> fail "expected at least the app concept" + "Profile" + -- The real descriptor, not a Haskell restatement of it. This fails if + -- profile/shikumi.dhall stops type-checking against the okf-core schema, + -- stops compiling as a coherent profile, or stops describing what the + -- generator actually emits. + -- + -- This replaces a former "Conformance" group that re-asserted the + -- profile's conventions (concept types, `shikumi://` resource scheme) + -- in-process against a Haskell copy of them, and so could never notice + -- the descriptor itself going stale. Every convention it checked is now + -- expressed by the descriptor and enforced here against the real file. + [ testCase "profile/shikumi.dhall loads and compiles" $ do + _ <- loadAndCompileProfile + pure (), + testCase "generated bundle conforms to profile/shikumi.dhall" $ do + compiled <- loadAndCompileProfile + withConcepts $ \concepts -> + validateProfile PermissiveConformance compiled concepts @?= [], + -- The profile carries `requireBundleVersion = Some "0.2"`, which is a + -- separate entry point from validateProfile because it consults no + -- concepts. Both directions are pinned: an undeclared bundle must + -- deviate, or the requirement would be inert and untested. + testCase "the profile requires a v0.2 declaration, and ours satisfies it" $ do + compiled <- loadAndCompileProfile + validateProfileVersion declaredV02 compiled @?= [] + assertBool + "an undeclared bundle deviates" + (not (null (validateProfileVersion VersionUndeclared compiled))) ], testGroup "RoundTrip" [ testCase "app links to every program (graph edges)" $ do root <- freshTempDir "shikumi-okf-roundtrip" - result <- writeProgramBundle root demoApp Nothing demoManifest + result <- writeProgramBundle root demoApp defaultGenerateOptions demoManifest case result of Left err -> fail ("writeProgramBundle failed: " <> show err) Right () -> do @@ -289,6 +423,24 @@ -- Helpers -- --------------------------------------------------------------------------- +-- | Generate the demo bundle with the default options and hand the concepts to +-- an assertion, failing the test rather than pattern-matching at each call site. +withConcepts :: ([Concept] -> IO a) -> IO a +withConcepts assertion = + case generateBundle demoApp defaultGenerateOptions demoManifest of + Left err -> fail ("generateBundle failed: " <> show err) + Right concepts -> assertion concepts + +-- | What the generator declares, restated here so a test reads the same +-- declaration a reader of the bundle root would. +declaredV02 :: VersionDeclaration +declaredV02 = VersionDeclared okfVersion02 + +-- | The inventory an in-memory bundle can honestly report: its own concepts and +-- no non-Markdown file, because there is no directory holding one. +inventoryOf :: [Concept] -> BundleInventory +inventoryOf = bundleInventoryOfConcepts + -- | A clean temp directory (removed if it exists), so the round-trip test is -- idempotent across runs. freshTempDir :: FilePath -> IO FilePath @@ -298,6 +450,21 @@ removeDirectoryRecursive root `catch` \(_ :: IOException) -> pure () createDirectoryIfMissing True root pure root + +-- | Load and compile the shipped profile descriptor, failing the test with a +-- readable message at whichever stage breaks. The path is resolved through +-- Cabal's data-files mechanism, so it does not depend on the working directory +-- the test was launched from. +loadAndCompileProfile :: IO CompiledProfile +loadAndCompileProfile = do + path <- getDataFileName "profile/shikumi.dhall" + loaded <- loadProfileFile path + case loaded of + Left err -> fail ("could not load " <> path <> ": " <> T.unpack err) + Right spec -> + case compileProfile spec of + Left errs -> fail ("profile did not compile: " <> show (toList errs)) + Right compiled -> pure compiled expectConceptId :: Text -> IO ConceptId expectConceptId raw =