kioku-migrations 0.3.0.0 → 0.4.0.0
raw patch · 6 files changed
+1102/−18 lines, 6 filesdep ~keiro-migrationsdep ~kioku-migrationsdep ~kiroku-store-migrationsPVP ok
version bump matches the API change (PVP)
Dependency ranges changed: keiro-migrations, kioku-migrations, kiroku-store-migrations
API changes (from Hackage documentation)
Files
- CHANGELOG.md +57/−0
- kioku-migrations.cabal +6/−5
- migrations/0011-kioku-memory-space-partition.sql +226/−0
- migrations/0012-relocate-projections-to-kioku-schema.sql +127/−0
- migrations/manifest +2/−0
- test/Main.hs +684/−13
CHANGELOG.md view
@@ -1,5 +1,62 @@ # Changelog +## 0.4.0.0 — 2026-08-17++### Breaking Changes++- `0012-relocate-projections-to-kioku-schema.sql` moves the seven Kioku-owned projections out of+ the `kiroku` schema and into a dedicated `kioku` one, dropping the now-redundant name prefix:+ `kiroku.kioku_memories` becomes `kioku.memories`, and likewise for `sessions`, `turns`,+ `l1_watermarks`, `consolidation_decisions`, `scenes`, and `personas`. The shared Kiroku event+ store does not move. `ALTER TABLE ... SET SCHEMA` plus `... RENAME TO` preserve each table's+ OID, rows, indexes, constraints, owner, and grants; index and constraint names are deliberately+ unchanged. The `vector` extension is not touched.+- The migration accepts exactly two catalog states — seven ordinary source tables with no occupied+ target, or no source with seven ordinary targets — and raises before moving anything otherwise.+ It is transactional, so a refusal leaves the catalog exactly as it was found. There are no+ compatibility views: run it with writers stopped, before the new binary starts, and see+ `docs/user/upgrading-to-the-kioku-schema.md`.+- The composed fresh-install plan is now 53 migrations: kiroku 11, keiro 30, kioku 12.+- Moved to `keiro-migrations ^>=0.13.0.0` and `kiroku-store-migrations ^>=0.4.0.0`. Those bring+ eleven new framework migrations that Kioku composes but does not own: kiroku `0009` (the frozen+ `kiroku.subscription_checkpoints_v1` view), `0010` (replay-history retention leases) and `0011`,+ and keiro `0021`–`0030` (exact workflow discovery, then the schema-versioned projection rebuild,+ promotion, and external-read-guard machinery). Keiro `0024` renames a rebuild-group column, so+ complete or abandon any active catalog rebuild **before** upgrading.+- `kiroku-store-migrations` 0.4.0.0 corrects the payload of kiroku `0010`, which changes its+ checksum. A database that already applied `0010` from the withdrawn 0.3.2.0 or 0.3.2.1 fails+ every `up` and `verify` with `MigrationChecksumMismatch` until its ledger row is re-baselined+ with that package's `ledger-fixups/2026-08-16-rebaseline-0010-checksum.sql`; kiroku `0011` then+ converges its schema. A database that never reached `0010` — which is every database blocked by+ the PostgreSQL 17 defect this release fixes — needs neither, and applies the corrected `0010`+ as an ordinary pending migration.+++### Added++- `0011-kioku-memory-space-partition.sql` adds a non-null `memory_space_id` to `kioku_memories`,+ `kioku_sessions`, `kioku_turns`, `kioku_l1_watermarks`, `kioku_consolidation_decisions`,+ `kioku_scenes`, and `kioku_personas`. Existing rows are backfilled into the explicit+ `kioku_legacy` space; turns and watermarks derive theirs from their parent session, and the+ migration aborts if any derived row disagrees with a session that exists.++### Breaking Changes++- `kioku_scenes` and `kioku_personas` now have composite primary keys,+ `(memory_space_id, scene_id)` and `(memory_space_id, persona_id)`, and their scope-uniqueness+ constraints gained the space. Their ids are derived from the namespace and scope alone, which+ two spaces are allowed to share, so a single-column key let one space's upsert overwrite the+ other's row. Stored id strings are unchanged.+- Indexes led by a namespace or a scope were rebuilt partition-first and renamed:+ `kioku_memories_scope_idx` → `kioku_memories_space_scope_idx`, `kioku_memories_type_idx` →+ `kioku_memories_space_type_idx` (now covering namespace as well), `kioku_sessions_scope_idx`,+ `kioku_sessions_namespace_started_idx`, `kioku_sessions_namespace_focus_idx`,+ `kioku_sessions_awaiting_corr_idx`, and `kioku_consolidation_scope_idx` likewise.+ `kioku_memories_space_namespace_idx` is new. `kioku_scenes_scope_idx` is dropped as a strict+ prefix of the per-space unique index.+- The migration is forward-only and has no down step. Re-running it is a no-op. Rolling the+ application back is safe only while nothing has been written outside `kioku_legacy`.+ ## 0.3.0.0 — 2026-08-05 ### Breaking Changes
kioku-migrations.cabal view
@@ -1,6 +1,6 @@ cabal-version: 3.0 name: kioku-migrations-version: 0.3.0.0+version: 0.4.0.0 synopsis: Schema migrations for kioku description: Owns kioku schema evolution as a native pg-migrate component and composes@@ -58,8 +58,8 @@ , containers >=0.6 && <0.8 , hasql >=1.10 && <1.11 , hasql-transaction >=1.2 && <1.3- , keiro-migrations ^>=0.11.0.0- , kiroku-store-migrations ^>=0.3.0.0+ , keiro-migrations ^>=0.13.0.0+ , kiroku-store-migrations ^>=0.4.0.0 , pg-migrate ^>=1.1.0.0 , pg-migrate-embed ^>=1.1.0.0 , pg-migrate-import-codd ^>=1.1.0.0@@ -75,7 +75,7 @@ , base >=4.18 && <5 , ephemeral-pg ^>=0.2.2.0 , hasql >=1.10 && <1.11- , kioku-migrations ^>=0.3.0.0+ , kioku-migrations ^>=0.4.0.0 , pg-migrate-test-support ^>=1.1.0.0 , text >=2.0 && <2.2 @@ -91,8 +91,9 @@ build-depends: , base >=4.18 && <5 , hasql >=1.6- , kioku-migrations ^>=0.3.0.0+ , kioku-migrations ^>=0.4.0.0 , kioku-migrations:test-support+ , kiroku-store-migrations ^>=0.4.0.0 , pg-migrate ^>=1.1.0.0 , pg-migrate-embed ^>=1.1.0.0 , pg-migrate-import-codd ^>=1.1.0.0
+ migrations/0011-kioku-memory-space-partition.sql view
@@ -0,0 +1,226 @@+-- Migration: kioku-memory-space-partition+-- Created: 2026-08-06-15-10-00 UTC+--+-- Every kioku read-model row becomes attributable to exactly one memory space.+--+-- A memory space is the outer isolation boundary: two tenants who must never see each+-- other's data are separated by memory space, whereas two hosts sharing a database are+-- merely separated by namespace. Until this migration the read models had no such column,+-- so a query could only ever see every space at once. See+-- docs/adr/namespace-is-not-a-security-boundary.md.+--+-- The column is added in the only safe order: nullable, deterministic backfill, validation,+-- then NOT NULL, then the constraints and indexes that put the partition first. A row whose+-- space were NULL would have to be interpreted by every query, and the only two available+-- interpretations -- "invisible" and "visible everywhere" -- are both wrong. Absence of a+-- partition must never read as unrestricted access, so every pre-existing row is backfilled+-- into the single explicit space `kioku_legacy`. See+-- docs/adr/legacy-data-lands-in-one-explicit-space.md.+--+-- Turns and L1 watermarks derive their space from their parent session rather than being+-- assumed legacy, because that derivation is the rule that must hold forever; the fallback+-- for a turn whose session row is missing is the legacy space, and the validation block+-- below fails the whole migration if any turn or watermark ends up disagreeing with a+-- session that does exist.+--+-- Every statement is idempotent: the columns and indexes use IF NOT EXISTS, the constraints+-- are dropped before being re-added, and the backfill matches only rows that are still NULL.+SET search_path TO kiroku, pg_catalog;++-- Step 1: the column, nullable, on every table that holds partitioned data.++ALTER TABLE kioku_memories ADD COLUMN IF NOT EXISTS memory_space_id text;+ALTER TABLE kioku_sessions ADD COLUMN IF NOT EXISTS memory_space_id text;+ALTER TABLE kioku_turns ADD COLUMN IF NOT EXISTS memory_space_id text;+ALTER TABLE kioku_l1_watermarks ADD COLUMN IF NOT EXISTS memory_space_id text;+ALTER TABLE kioku_consolidation_decisions ADD COLUMN IF NOT EXISTS memory_space_id text;+ALTER TABLE kioku_scenes ADD COLUMN IF NOT EXISTS memory_space_id text;+ALTER TABLE kioku_personas ADD COLUMN IF NOT EXISTS memory_space_id text;++-- Step 2: the deterministic backfill.+--+-- Roots first. Every row that exists at this point predates memory spaces, so it belongs to+-- the legacy space by definition -- there is no other space it could have been written in.++UPDATE kioku_memories SET memory_space_id = 'kioku_legacy' WHERE memory_space_id IS NULL;+UPDATE kioku_sessions SET memory_space_id = 'kioku_legacy' WHERE memory_space_id IS NULL;+UPDATE kioku_consolidation_decisions SET memory_space_id = 'kioku_legacy' WHERE memory_space_id IS NULL;+UPDATE kioku_scenes SET memory_space_id = 'kioku_legacy' WHERE memory_space_id IS NULL;+UPDATE kioku_personas SET memory_space_id = 'kioku_legacy' WHERE memory_space_id IS NULL;++-- Derived rows inherit from the session they belong to. `kioku_turns.session_id` has no+-- foreign key, so an orphan turn is possible in principle; it falls back to the legacy+-- space, which is where its session would have been had it survived.++UPDATE kioku_turns t+ SET memory_space_id =+ COALESCE(+ (SELECT s.memory_space_id FROM kioku_sessions s WHERE s.session_id = t.session_id),+ 'kioku_legacy')+ WHERE t.memory_space_id IS NULL;++UPDATE kioku_l1_watermarks w+ SET memory_space_id =+ COALESCE(+ (SELECT s.memory_space_id FROM kioku_sessions s WHERE s.session_id = w.session_id),+ 'kioku_legacy')+ WHERE w.memory_space_id IS NULL;++-- Step 3: validation, before anything irreversible.+--+-- Abort rather than write a partition nobody can trust. The first check is the one that+-- matters on a fresh upgrade (nothing may be left unattributed); the second and third are+-- the derivation rule, which is vacuous today but is the invariant every later write must+-- preserve, and stating it here is what makes a future violation loud.+DO $$+DECLARE+ unattributed bigint;+ drifted bigint;+BEGIN+ SELECT (SELECT count(*) FROM kioku_memories WHERE memory_space_id IS NULL)+ + (SELECT count(*) FROM kioku_sessions WHERE memory_space_id IS NULL)+ + (SELECT count(*) FROM kioku_turns WHERE memory_space_id IS NULL)+ + (SELECT count(*) FROM kioku_l1_watermarks WHERE memory_space_id IS NULL)+ + (SELECT count(*) FROM kioku_consolidation_decisions WHERE memory_space_id IS NULL)+ + (SELECT count(*) FROM kioku_scenes WHERE memory_space_id IS NULL)+ + (SELECT count(*) FROM kioku_personas WHERE memory_space_id IS NULL)+ INTO unattributed;++ IF unattributed > 0 THEN+ RAISE EXCEPTION+ 'kioku memory-space backfill left % row(s) with no memory space; refusing to continue',+ unattributed;+ END IF;++ SELECT (SELECT count(*)+ FROM kioku_turns t+ JOIN kioku_sessions s ON s.session_id = t.session_id+ WHERE t.memory_space_id IS DISTINCT FROM s.memory_space_id)+ + (SELECT count(*)+ FROM kioku_l1_watermarks w+ JOIN kioku_sessions s ON s.session_id = w.session_id+ WHERE w.memory_space_id IS DISTINCT FROM s.memory_space_id)+ INTO drifted;++ IF drifted > 0 THEN+ RAISE EXCEPTION+ 'kioku memory-space backfill left % derived row(s) in a different space from their session',+ drifted;+ END IF;+END $$;++-- Step 4: make the column mandatory. An empty string is rejected too: it is not a space,+-- and allowing it would give "no space" a second spelling.++ALTER TABLE kioku_memories ALTER COLUMN memory_space_id SET NOT NULL;+ALTER TABLE kioku_sessions ALTER COLUMN memory_space_id SET NOT NULL;+ALTER TABLE kioku_turns ALTER COLUMN memory_space_id SET NOT NULL;+ALTER TABLE kioku_l1_watermarks ALTER COLUMN memory_space_id SET NOT NULL;+ALTER TABLE kioku_consolidation_decisions ALTER COLUMN memory_space_id SET NOT NULL;+ALTER TABLE kioku_scenes ALTER COLUMN memory_space_id SET NOT NULL;+ALTER TABLE kioku_personas ALTER COLUMN memory_space_id SET NOT NULL;++ALTER TABLE kioku_memories DROP CONSTRAINT IF EXISTS kioku_memories_space_present_check;+ALTER TABLE kioku_memories ADD CONSTRAINT kioku_memories_space_present_check+ CHECK (memory_space_id <> '');+ALTER TABLE kioku_sessions DROP CONSTRAINT IF EXISTS kioku_sessions_space_present_check;+ALTER TABLE kioku_sessions ADD CONSTRAINT kioku_sessions_space_present_check+ CHECK (memory_space_id <> '');+ALTER TABLE kioku_turns DROP CONSTRAINT IF EXISTS kioku_turns_space_present_check;+ALTER TABLE kioku_turns ADD CONSTRAINT kioku_turns_space_present_check+ CHECK (memory_space_id <> '');+ALTER TABLE kioku_l1_watermarks DROP CONSTRAINT IF EXISTS kioku_l1_watermarks_space_present_check;+ALTER TABLE kioku_l1_watermarks ADD CONSTRAINT kioku_l1_watermarks_space_present_check+ CHECK (memory_space_id <> '');+ALTER TABLE kioku_consolidation_decisions DROP CONSTRAINT IF EXISTS kioku_consolidation_decisions_space_present_check;+ALTER TABLE kioku_consolidation_decisions ADD CONSTRAINT kioku_consolidation_decisions_space_present_check+ CHECK (memory_space_id <> '');+ALTER TABLE kioku_scenes DROP CONSTRAINT IF EXISTS kioku_scenes_space_present_check;+ALTER TABLE kioku_scenes ADD CONSTRAINT kioku_scenes_space_present_check+ CHECK (memory_space_id <> '');+ALTER TABLE kioku_personas DROP CONSTRAINT IF EXISTS kioku_personas_space_present_check;+ALTER TABLE kioku_personas ADD CONSTRAINT kioku_personas_space_present_check+ CHECK (memory_space_id <> '');++-- Step 5: identities that were derived from a scope must now be derived from a space and a+-- scope together.+--+-- Scene and persona primary keys are computed from the namespace and scope alone+-- (Kioku.Distill.ScopeIdentity). Two memory spaces are allowed to use identical namespaces+-- and scopes -- that is the whole point of the partition -- so those ids collide across+-- spaces, and with a single-column primary key one space's scene would silently overwrite+-- another's through the upsert's ON CONFLICT clause.+--+-- The fix is a composite primary key rather than a new id derivation. Re-deriving the ids+-- would rewrite every scene and persona row and every plaintext mirror filename for a+-- cosmetic gain; the composite key makes the collision impossible while leaving already+-- persisted ids exactly as they are.++ALTER TABLE kioku_scenes DROP CONSTRAINT IF EXISTS kioku_scenes_pkey;+ALTER TABLE kioku_scenes ADD CONSTRAINT kioku_scenes_pkey+ PRIMARY KEY (memory_space_id, scene_id);++ALTER TABLE kioku_personas DROP CONSTRAINT IF EXISTS kioku_personas_pkey;+ALTER TABLE kioku_personas ADD CONSTRAINT kioku_personas_pkey+ PRIMARY KEY (memory_space_id, persona_id);++-- Scope uniqueness is now per space. NULLS NOT DISTINCT is preserved from the+-- schema-hardening migration: without it two global-scope rows in one space would both+-- satisfy the constraint, because SQL considers NULLs distinct.++ALTER TABLE kioku_scenes DROP CONSTRAINT IF EXISTS kioku_scenes_scope_scene_key_unique;+ALTER TABLE kioku_scenes ADD CONSTRAINT kioku_scenes_scope_scene_key_unique+ UNIQUE NULLS NOT DISTINCT (memory_space_id, namespace, scope_kind, scope_ref, scene_key);++ALTER TABLE kioku_personas DROP CONSTRAINT IF EXISTS kioku_personas_scope_unique;+ALTER TABLE kioku_personas ADD CONSTRAINT kioku_personas_scope_unique+ UNIQUE NULLS NOT DISTINCT (memory_space_id, namespace, scope_kind, scope_ref);++-- Step 6: lookup indexes, partition first.+--+-- Every index whose leading columns were a namespace or a scope is rebuilt with the memory+-- space in front of them, because every such query now carries a space predicate and the+-- space is the most selective column available. Indexes led by a globally unique id+-- (memory_id, session_id, parent_session_id, supersedes, superseded_by, content_hash) are+-- deliberately left alone: an id is already selective enough that prefixing it with the+-- space would buy nothing and cost a write on every insert. The space predicate is still in+-- those statements -- it is a filter there, not an access path.++DROP INDEX IF EXISTS kioku_memories_scope_idx;+CREATE INDEX IF NOT EXISTS kioku_memories_space_scope_idx+ ON kioku_memories (memory_space_id, namespace, scope_kind, scope_ref) WHERE status = 'active';++-- The old type index covered `memory_type` alone, while the query it exists for filters on+-- namespace and type together. Rebuilding it partition-first also fixes that.+DROP INDEX IF EXISTS kioku_memories_type_idx;+CREATE INDEX IF NOT EXISTS kioku_memories_space_type_idx+ ON kioku_memories (memory_space_id, namespace, memory_type) WHERE status = 'active';++CREATE INDEX IF NOT EXISTS kioku_memories_space_namespace_idx+ ON kioku_memories (memory_space_id, namespace, priority, created_at DESC) WHERE status = 'active';++DROP INDEX IF EXISTS kioku_sessions_scope_idx;+CREATE INDEX IF NOT EXISTS kioku_sessions_space_scope_idx+ ON kioku_sessions (memory_space_id, namespace, scope_kind, scope_ref);++DROP INDEX IF EXISTS kioku_sessions_namespace_started_idx;+CREATE INDEX IF NOT EXISTS kioku_sessions_space_namespace_started_idx+ ON kioku_sessions (memory_space_id, namespace, started_at DESC);++DROP INDEX IF EXISTS kioku_sessions_namespace_focus_idx;+CREATE INDEX IF NOT EXISTS kioku_sessions_space_namespace_focus_idx+ ON kioku_sessions (memory_space_id, namespace, focus, started_at DESC);++DROP INDEX IF EXISTS kioku_sessions_awaiting_corr_idx;+CREATE INDEX IF NOT EXISTS kioku_sessions_space_awaiting_corr_idx+ ON kioku_sessions (memory_space_id, namespace, awaiting_correlation_key)+ WHERE status = 'awaiting';++DROP INDEX IF EXISTS kioku_consolidation_scope_idx;+CREATE INDEX IF NOT EXISTS kioku_consolidation_space_scope_idx+ ON kioku_consolidation_decisions (memory_space_id, namespace, scope_kind, scope_ref);++-- The scene scope index is now a strict prefix of the unique constraint's index, exactly as+-- kioku_turns_session_idx was of UNIQUE (session_id, turn_index) before the schema-hardening+-- migration dropped it. Keeping it would be pure write amplification.+DROP INDEX IF EXISTS kioku_scenes_scope_idx;
+ migrations/0012-relocate-projections-to-kioku-schema.sql view
@@ -0,0 +1,127 @@+-- Migration: relocate-projections-to-kioku-schema+-- Created: 2026-08-07-18-20-00 UTC+--+-- Kioku's seven projection tables move out of Kiroku's schema and into their own.+--+-- Kioku deliberately shares the host application's Kiroku event store: that sharing is the+-- integration boundary, and this migration does not touch it. What it does change is where the+-- relations *Kioku* owns live. Until now they sat in `kiroku` alongside the event store's own+-- tables, distinguished only by a `kioku_` name prefix that nothing in the catalog enforces, so+-- a host that already ran Kiroku found another product's projections inside its event store's+-- namespace. After this migration every Kioku-owned relation is in the `kioku` schema, and the+-- prefix is dropped because the schema now supplies the namespace it used to stand in for. See+-- docs/adr/projections-live-in-the-kioku-schema.md.+--+-- kiroku.kioku_memories -> kioku.memories+-- kiroku.kioku_sessions -> kioku.sessions+-- kiroku.kioku_turns -> kioku.turns+-- kiroku.kioku_l1_watermarks -> kioku.l1_watermarks+-- kiroku.kioku_consolidation_decisions -> kioku.consolidation_decisions+-- kiroku.kioku_scenes -> kioku.scenes+-- kiroku.kioku_personas -> kioku.personas+--+-- This is a metadata operation, not a copy. `ALTER TABLE ... SET SCHEMA` and+-- `ALTER TABLE ... RENAME TO` both keep the table's object identity, so its rows, indexes,+-- constraints, owner, and table grants come along untouched and nothing is rebuilt. Index and+-- constraint names are deliberately *not* renamed: they travel by identity, and renaming them+-- would churn the catalog for no gain. None of the seven tables has a sequence, trigger, or+-- foreign key needing separate movement.+--+-- The `vector` extension is not moved and must not be. A PostgreSQL extension is a+-- database-wide object the host may share, so `ALTER EXTENSION` is out of scope here. The+-- memory table keeps its `vector` column and its HNSW index by identity, and the runtime's+-- `to_regtype('vector')` probe keeps resolving against the connection's search path exactly as+-- 0009-kioku-embedding-schema-heal.sql arranged.+--+-- Exactly two catalog states are accepted, and every other state aborts before a single table+-- moves. A partial or colliding layout is never repaired heuristically, because there is no way+-- to tell an interrupted upgrade apart from a host relation that happens to share a target name,+-- and guessing wrong would mean moving somebody else's table or writing into it. The migration+-- is transactional, so a refusal leaves the catalog exactly as it was found.+--+-- 1. all seven sources present as ordinary tables, no target name occupied -> move them+-- 2. no source name occupied at all, all seven targets present as ordinary tables -> no-op+--+-- The occupancy test uses `to_regclass`, which resolves any relation: a view, materialized view,+-- foreign table, sequence, or index sitting on a target name blocks the move, because all of+-- them share PostgreSQL's relation namespace and any one of them would make the rename fail.++CREATE SCHEMA IF NOT EXISTS kioku;++COMMENT ON SCHEMA kioku IS+ 'Kioku-owned read-model projections. Managed by pg-migrate component kioku.';++DO $$+DECLARE+ source_names CONSTANT text[] := ARRAY[+ 'kioku_memories',+ 'kioku_sessions',+ 'kioku_turns',+ 'kioku_l1_watermarks',+ 'kioku_consolidation_decisions',+ 'kioku_scenes',+ 'kioku_personas'+ ];+ target_names CONSTANT text[] := ARRAY[+ 'memories',+ 'sessions',+ 'turns',+ 'l1_watermarks',+ 'consolidation_decisions',+ 'scenes',+ 'personas'+ ];+ expected CONSTANT integer := array_length(source_names, 1);+ slot integer;+ source_relation regclass;+ target_relation regclass;+ relation_kind "char";+ present_sources integer := 0;+ ordinary_sources integer := 0;+ present_targets integer := 0;+ ordinary_targets integer := 0;+BEGIN+ -- Pass one counts. Nothing is altered until the whole layout has been classified, so a+ -- collision on the seventh table cannot be discovered after the first six have moved.+ FOR slot IN 1 .. expected LOOP+ source_relation := to_regclass('kiroku.' || quote_ident(source_names[slot]));+ target_relation := to_regclass('kioku.' || quote_ident(target_names[slot]));++ IF source_relation IS NOT NULL THEN+ present_sources := present_sources + 1;+ SELECT c.relkind INTO relation_kind+ FROM pg_catalog.pg_class c+ WHERE c.oid = source_relation;+ IF relation_kind = 'r' THEN+ ordinary_sources := ordinary_sources + 1;+ END IF;+ END IF;++ IF target_relation IS NOT NULL THEN+ present_targets := present_targets + 1;+ SELECT c.relkind INTO relation_kind+ FROM pg_catalog.pg_class c+ WHERE c.oid = target_relation;+ IF relation_kind = 'r' THEN+ ordinary_targets := ordinary_targets + 1;+ END IF;+ END IF;+ END LOOP;++ IF present_sources = expected AND ordinary_sources = expected AND present_targets = 0 THEN+ FOR slot IN 1 .. expected LOOP+ EXECUTE format('ALTER TABLE kiroku.%I SET SCHEMA kioku', source_names[slot]);+ EXECUTE format('ALTER TABLE kioku.%I RENAME TO %I', source_names[slot], target_names[slot]);+ END LOOP;+ ELSIF present_sources = 0 AND present_targets = expected AND ordinary_targets = expected THEN+ -- Already relocated. Re-running the body is a no-op by construction, not by accident.+ NULL;+ ELSE+ RAISE EXCEPTION+ 'refusing to relocate Kioku projections: expected either % ordinary kiroku.kioku_* tables '+ 'with no kioku.* target relation, or no kiroku.kioku_* relation with % ordinary kioku.* '+ 'tables; found % source relation(s) of which % ordinary, and % target relation(s) of '+ 'which % ordinary',+ expected, expected, present_sources, ordinary_sources, present_targets, ordinary_targets;+ END IF;+END $$;
migrations/manifest view
@@ -8,3 +8,5 @@ 0008-kioku-schema-hardening.sql 0009-kioku-embedding-schema-heal.sql 0010-kioku-scope-identity-recompute.sql+0011-kioku-memory-space-partition.sql+0012-relocate-projections-to-kioku-schema.sql
test/Main.hs view
@@ -13,6 +13,8 @@ import Control.Exception (bracket) import Data.Foldable (toList) import Data.Int (Int64)+import Data.List (sort)+import Data.List.NonEmpty (NonEmpty (..)) import Data.Maybe (mapMaybe) import Data.Text (Text) import Data.Text qualified as Text@@ -26,6 +28,7 @@ HistoryImportResult (..), MigrationId, MigrationOutcome (AlreadyApplied, AppliedNow),+ MigrationPlan, MigrationReport (..), MigrationResult (..), VerificationReport (..),@@ -33,6 +36,7 @@ defaultImportOptions, defaultRunOptions, migrationId,+ migrationPlan, runMigrationPlan, validateHistoryMappingTargets, verifyMigrationPlan,@@ -55,8 +59,9 @@ kiokuCoddHistoryMappings, ) import Kioku.Migrations.TestSupport (withBareDatabase, withKiokuMigratedDatabase)+import Kiroku.Store.Migrations qualified as KirokuMigrations import Test.Tasty (TestTree, defaultMain, testGroup)-import Test.Tasty.HUnit (Assertion, assertFailure, testCase, (@?=))+import Test.Tasty.HUnit (Assertion, assertBool, assertFailure, testCase, (@?=)) main :: IO () main = defaultMain tests@@ -74,7 +79,27 @@ testCase "the full migration chain applies to a fresh database" testFreshDatabase, testCase "the migration manifest is complete and valid" testManifestIntegrity, testCase "the pinned Codd history maps 30 known plan targets" testHistoryMappings,- testCase "the pre-cutover Codd cohort imports 30 rows and applies only eight forward migrations" testCoddCohortImport+ testCase "the pre-cutover Codd cohort imports 30 rows and applies only the forward migrations" testCoddCohortImport,+ testGroup+ "the memory-space partition migration"+ [ testCase "backfills every pre-partition row into the legacy space" testMemorySpaceBackfill,+ testCase "refuses to finish when a derived row disagrees with its session" testMemorySpaceDriftAborts,+ testCase "re-applying its body changes nothing" testMemorySpaceBackfillIdempotent+ ],+ testGroup+ "the projection schema relocation"+ [ testCase "a fresh database owns seven relations in kioku and none in kiroku" testRelocatedFreshLayout,+ testCase "a Kiroku-only ledger adopts Kioku without replaying Kiroku" testKirokuOnlyAdoption,+ testCase "a data-bearing upgrade keeps every row, OID, index, and grant" testRelocationPreservesTables,+ testCase "re-applying its body against the new layout changes nothing" testRelocationRerunIsNoOp,+ testGroup+ "a layout that is neither wholly old nor wholly new aborts and changes nothing"+ [ testCase "one table already moved" (assertRelocationAborts moveOneTableForward),+ testCase "one source table missing" (assertRelocationAborts dropOneSourceTable),+ testCase "a target name taken by a table" (assertRelocationAborts occupyTargetWithTable),+ testCase "a target name taken by a view" (assertRelocationAborts occupyTargetWithView)+ ]+ ] ] -- * The manifest guard@@ -164,18 +189,20 @@ ) ) --- | The migration's bytes as they were compiled into this binary, so the test exercises--- exactly what ships. @-- codd:@ directives are ordinary SQL comments, so the whole file--- runs as one script. loadRegistryBumpMigration :: IO Text-loadRegistryBumpMigration = do+loadRegistryBumpMigration =+ loadMigration "0006-kioku-session-readmodel-registry-bump.sql"++-- | A migration's bytes as they were compiled into this binary, so a test exercises exactly+-- what ships. @-- codd:@ directives are ordinary SQL comments, so a whole file runs as one+-- script.+loadMigration :: FilePath -> IO Text+loadMigration name = do result <- checkMigrationManifest "migrations/manifest" entries <- either (fail . show) pure result case lookup name (toList entries) of Nothing -> fail ("no manifest migration named " <> name) Just bytes -> pure (Text.Encoding.decodeUtf8 bytes)- where- name = "0006-kioku-session-readmodel-registry-bump.sql" -- * The fresh-database test @@ -194,6 +221,632 @@ E.noParams (D.singleRow (D.column (D.nonNullable D.bool))) +-- * The memory-space partition migration++-- | Prove the backfill on data that genuinely predates the partition.+--+-- The migrated database already has the column, so the test first puts it back the way it+-- was: 'undoMemorySpacePartition' drops every @memory_space_id@ (taking its indexes and+-- constraints with it) and restores the single-column scene and persona primary keys. Rows+-- inserted after that are indistinguishable from rows written by an older kioku, which is+-- what makes running the shipped migration bytes over them a test of the real upgrade+-- rather than of a mock of one.+testMemorySpaceBackfill :: Assertion+testMemorySpaceBackfill =+ withPrePartitionDatabase \conn -> do+ partition <- loadMigration memorySpacePartitionMigration+ run conn (Session.script partition)++ run conn (Session.statement () countMemorySpaces)+ >>= (@?= [("kioku_legacy", 8)])++ -- The turn whose session row is missing is the one case the derivation cannot answer+ -- from a parent, and it must still land somewhere explicit.+ run conn (Session.statement "t-orphan" selectTurnSpace)+ >>= (@?= Just "kioku_legacy")++ -- The partition-leading indexes and the composite scene/persona keys are what make the+ -- boundary enforceable rather than merely recorded.+ indexes <- run conn (Session.statement () selectKiokuIndexes)+ mapM_+ (\name -> assertBool (Text.unpack name <> " is missing") (name `elem` indexes))+ [ "kioku_memories_space_scope_idx",+ "kioku_memories_space_type_idx",+ "kioku_memories_space_namespace_idx",+ "kioku_sessions_space_scope_idx",+ "kioku_sessions_space_namespace_started_idx",+ "kioku_sessions_space_namespace_focus_idx",+ "kioku_sessions_space_awaiting_corr_idx",+ "kioku_consolidation_space_scope_idx"+ ]+ assertBool+ "kioku_scenes_scope_idx still exists; it duplicates the prefix of kioku_scenes_scope_scene_key_unique"+ ("kioku_scenes_scope_idx" `notElem` indexes)++ run conn (Session.statement "kioku_scenes" selectPrimaryKeyColumns)+ >>= (@?= ["memory_space_id", "scene_id"])+ run conn (Session.statement "kioku_personas" selectPrimaryKeyColumns)+ >>= (@?= ["memory_space_id", "persona_id"])++ -- Two spaces may now hold the same namespace, scope, and scene key. Before the composite+ -- key this pair collided on the scope-derived primary key and one silently overwrote the+ -- other through the upsert's ON CONFLICT clause.+ run conn (Session.script twoSpacesOneScopeKey)++-- | The derivation rule is a guard, not a comment: a turn that disagrees with its session+-- aborts the whole migration instead of quietly persisting a row in the wrong space.+--+-- Tampering after a first successful pass is how the state is reached, because the backfill+-- itself cannot produce it. On the second pass the @IS NULL@ updates match nothing and the+-- validation block is what runs.+testMemorySpaceDriftAborts :: Assertion+testMemorySpaceDriftAborts =+ withPrePartitionDatabase \conn -> do+ partition <- loadMigration memorySpacePartitionMigration+ run conn (Session.script partition)+ run conn (Session.script "UPDATE kiroku.kioku_turns SET memory_space_id = 'space_elsewhere' WHERE turn_id = 't-1'")++ result <- Connection.use conn (Session.script partition)+ case result of+ Right () -> assertFailure "the migration accepted a turn in a different space from its session"+ Left err ->+ assertBool+ ("expected a drift failure, got: " <> show err)+ ("different space from their session" `Text.isInfixOf` Text.pack (show err))++-- | Re-applying the body must be a no-op, which is what makes it safe to re-run after a+-- partially failed deployment.+testMemorySpaceBackfillIdempotent :: Assertion+testMemorySpaceBackfillIdempotent =+ withPrePartitionDatabase \conn -> do+ partition <- loadMigration memorySpacePartitionMigration+ run conn (Session.script partition)+ before <- run conn (Session.statement () kiokuSchemaSnapshot)+ spacesBefore <- run conn (Session.statement () countMemorySpaces)++ run conn (Session.script partition)+ run conn (Session.statement () kiokuSchemaSnapshot) >>= (@?= before)+ run conn (Session.statement () countMemorySpaces) >>= (@?= spacesBefore)++memorySpacePartitionMigration :: FilePath+memorySpacePartitionMigration = "0011-kioku-memory-space-partition.sql"++-- | A fully migrated database rolled back to the shape it had before memory spaces, then+-- seeded with one row in every partitioned table.+--+-- Migration 0011 predates the @kioku@ schema, so it names @kiroku.kioku_*@ throughout. The+-- database has to be walked back past 0012 before 0011 can be tested in the layout it was+-- written for; 'undoSchemaRelocation' is that step. It is a test fixture, not an operator+-- procedure — a released migration is never reversed in a real database.+withPrePartitionDatabase :: (Connection.Connection -> IO a) -> IO a+withPrePartitionDatabase use =+ withPreRelocationDatabase \conn -> do+ run conn (Session.script undoMemorySpacePartition)+ run conn (Session.script prePartitionRows)+ use conn++-- | @DROP COLUMN … CASCADE@ takes the column's indexes and constraints with it, including the+-- composite primary keys, which is why the single-column ones have to be put back by hand.+-- The rest of the DDL below is the pre-partition index set verbatim.+undoMemorySpacePartition :: Text+undoMemorySpacePartition =+ """+ ALTER TABLE kiroku.kioku_memories DROP COLUMN memory_space_id CASCADE;+ ALTER TABLE kiroku.kioku_sessions DROP COLUMN memory_space_id CASCADE;+ ALTER TABLE kiroku.kioku_turns DROP COLUMN memory_space_id CASCADE;+ ALTER TABLE kiroku.kioku_l1_watermarks DROP COLUMN memory_space_id CASCADE;+ ALTER TABLE kiroku.kioku_consolidation_decisions DROP COLUMN memory_space_id CASCADE;+ ALTER TABLE kiroku.kioku_scenes DROP COLUMN memory_space_id CASCADE;+ ALTER TABLE kiroku.kioku_personas DROP COLUMN memory_space_id CASCADE;++ ALTER TABLE kiroku.kioku_scenes ADD CONSTRAINT kioku_scenes_pkey PRIMARY KEY (scene_id);+ ALTER TABLE kiroku.kioku_personas ADD CONSTRAINT kioku_personas_pkey PRIMARY KEY (persona_id);+ ALTER TABLE kiroku.kioku_scenes ADD CONSTRAINT kioku_scenes_scope_scene_key_unique+ UNIQUE NULLS NOT DISTINCT (namespace, scope_kind, scope_ref, scene_key);+ ALTER TABLE kiroku.kioku_personas ADD CONSTRAINT kioku_personas_scope_unique+ UNIQUE NULLS NOT DISTINCT (namespace, scope_kind, scope_ref);++ CREATE INDEX kioku_memories_scope_idx+ ON kiroku.kioku_memories (namespace, scope_kind, scope_ref) WHERE status = 'active';+ CREATE INDEX kioku_memories_type_idx+ ON kiroku.kioku_memories (memory_type) WHERE status = 'active';+ CREATE INDEX kioku_sessions_scope_idx+ ON kiroku.kioku_sessions (namespace, scope_kind, scope_ref);+ CREATE INDEX kioku_sessions_namespace_started_idx+ ON kiroku.kioku_sessions (namespace, started_at DESC);+ CREATE INDEX kioku_sessions_namespace_focus_idx+ ON kiroku.kioku_sessions (namespace, focus, started_at DESC);+ CREATE INDEX kioku_sessions_awaiting_corr_idx+ ON kiroku.kioku_sessions (namespace, awaiting_correlation_key) WHERE status = 'awaiting';+ CREATE INDEX kioku_consolidation_scope_idx+ ON kiroku.kioku_consolidation_decisions (namespace, scope_kind, scope_ref);+ CREATE INDEX kioku_scenes_scope_idx+ ON kiroku.kioku_scenes (namespace, scope_kind, scope_ref);+ """++-- | Eight rows: one per partitioned table, plus a turn whose session does not exist.+prePartitionRows :: Text+prePartitionRows =+ """+ INSERT INTO kiroku.kioku_sessions (session_id, agent_id, focus, namespace, started_at)+ VALUES ('s-1', 'agent', 'focus', 'ns', now());++ INSERT INTO kiroku.kioku_turns (turn_id, session_id, turn_index, role, content, recorded_at)+ VALUES ('t-1', 's-1', 1, 'user', 'hello', now()),+ ('t-orphan', 's-vanished', 1, 'user', 'hello', now());++ INSERT INTO kiroku.kioku_l1_watermarks (session_id, last_turn_index) VALUES ('s-1', 1);++ INSERT INTO kiroku.kioku_memories+ (memory_id, agent_id, session_id, namespace, memory_type, content, created_at, updated_at)+ VALUES ('m-1', 'agent', 's-1', 'ns', 'fact', 'content', now(), now());++ INSERT INTO kiroku.kioku_consolidation_decisions+ (decision_id, session_id, namespace, candidate_content, decision)+ VALUES ('d-1', 's-1', 'ns', 'content', 'store');++ INSERT INTO kiroku.kioku_scenes (scene_id, namespace, scene_key, title, body_md, source_hash)+ VALUES ('kioku_scene:ns:default', 'ns', 'default', 'title', 'body', 'hash');++ INSERT INTO kiroku.kioku_personas (persona_id, namespace, body_md, source_hash)+ VALUES ('kioku_persona:ns', 'ns', 'body', 'hash');+ """++-- | The same scope-derived scene and persona ids, in a second space. Both inserts must+-- succeed; if either the primary key or the scope-uniqueness constraint had kept its old+-- shape, the second space would collide with the first.+twoSpacesOneScopeKey :: Text+twoSpacesOneScopeKey =+ """+ INSERT INTO kiroku.kioku_scenes+ (memory_space_id, scene_id, namespace, scene_key, title, body_md, source_hash)+ VALUES ('space_other', 'kioku_scene:ns:default', 'ns', 'default', 'title', 'body', 'hash');++ INSERT INTO kiroku.kioku_personas (memory_space_id, persona_id, namespace, body_md, source_hash)+ VALUES ('space_other', 'kioku_persona:ns', 'ns', 'body', 'hash');+ """++-- | Every partitioned row, grouped by the space it landed in.+countMemorySpaces :: Statement () [(Text, Int64)]+countMemorySpaces =+ preparable+ """+ SELECT memory_space_id, count(*)+ FROM (+ SELECT memory_space_id FROM kiroku.kioku_memories+ UNION ALL SELECT memory_space_id FROM kiroku.kioku_sessions+ UNION ALL SELECT memory_space_id FROM kiroku.kioku_turns+ UNION ALL SELECT memory_space_id FROM kiroku.kioku_l1_watermarks+ UNION ALL SELECT memory_space_id FROM kiroku.kioku_consolidation_decisions+ UNION ALL SELECT memory_space_id FROM kiroku.kioku_scenes+ UNION ALL SELECT memory_space_id FROM kiroku.kioku_personas+ ) AS partitioned+ GROUP BY memory_space_id+ ORDER BY memory_space_id+ """+ E.noParams+ (D.rowList ((,) <$> D.column (D.nonNullable D.text) <*> D.column (D.nonNullable D.int8)))++selectTurnSpace :: Statement Text (Maybe Text)+selectTurnSpace =+ preparable+ "SELECT memory_space_id FROM kiroku.kioku_turns WHERE turn_id = $1"+ (E.param (E.nonNullable E.text))+ (D.rowMaybe (D.column (D.nonNullable D.text)))++-- | @pg_indexes.indexname@ is a @name@, not a @text@; the cast is what lets hasql decode it.+selectKiokuIndexes :: Statement () [Text]+selectKiokuIndexes =+ preparable+ "SELECT indexname::text FROM pg_indexes WHERE schemaname = 'kiroku'"+ E.noParams+ (D.rowList (D.column (D.nonNullable D.text)))++selectPrimaryKeyColumns :: Statement Text [Text]+selectPrimaryKeyColumns =+ preparable+ """+ SELECT a.attname::text+ FROM pg_constraint c+ JOIN pg_class t ON t.oid = c.conrelid+ JOIN pg_namespace n ON n.oid = t.relnamespace+ CROSS JOIN LATERAL unnest(c.conkey) WITH ORDINALITY AS k(attnum, ordinality)+ JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = k.attnum+ WHERE c.contype = 'p' AND n.nspname = 'kiroku' AND t.relname = $1+ ORDER BY k.ordinality+ """+ (E.param (E.nonNullable E.text))+ (D.rowList (D.column (D.nonNullable D.text)))++kiokuSchemaSnapshot :: Statement () Text+kiokuSchemaSnapshot =+ preparable+ """+ SELECT md5(+ coalesce((SELECT string_agg(table_name || '.' || column_name || ':' || data_type || ':' || is_nullable,+ E'\n' ORDER BY table_name, ordinal_position)+ FROM information_schema.columns+ WHERE table_schema = 'kiroku' AND table_name LIKE 'kioku\\_%'), '')+ || coalesce((SELECT string_agg(indexname || ':' || indexdef, E'\n' ORDER BY indexname)+ FROM pg_indexes+ WHERE schemaname = 'kiroku' AND tablename LIKE 'kioku\\_%'), ''))+ """+ E.noParams+ (D.singleRow (D.column (D.nonNullable D.text)))++-- * The projection schema relocation++-- | The seven relations Kioku owns: the name migration 0011 left them under, and the name+-- migration 0012 puts them under. The schema changes with the name — @kiroku.kioku_memories@+-- becomes @kioku.memories@ — because the schema now supplies the namespace the prefix used to.+relocatedRelations :: [(Text, Text)]+relocatedRelations =+ [ ("kioku_memories", "memories"),+ ("kioku_sessions", "sessions"),+ ("kioku_turns", "turns"),+ ("kioku_l1_watermarks", "l1_watermarks"),+ ("kioku_consolidation_decisions", "consolidation_decisions"),+ ("kioku_scenes", "scenes"),+ ("kioku_personas", "personas")+ ]++relocationMigration :: FilePath+relocationMigration = "0012-relocate-projections-to-kioku-schema.sql"++-- | What 'projectionLayout' must report once the relocation has run: seven ordinary tables in+-- @kioku@, and nothing at all left behind in @kiroku@ under the old prefix.+targetProjectionLayout :: [(Text, Text, Text)]+targetProjectionLayout =+ [("kioku", target, "r") | target <- sort (snd <$> relocatedRelations)]++-- | Every relation that could occupy one of the fourteen names this migration cares about.+--+-- Indexes are excluded deliberately: they share PostgreSQL's relation namespace, so the+-- migration itself has to treat an index named @kioku.memories@ as a collision, but listing the+-- twenty-odd projection indexes here would drown the layout the assertions are about. The+-- relkind travels with each row so a view squatting on a target name is visible as a view.+projectionLayout :: Statement () [(Text, Text, Text)]+projectionLayout =+ preparable+ """+ SELECT n.nspname::text, c.relname::text, c.relkind::text+ FROM pg_catalog.pg_class c+ JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace+ WHERE c.relkind IN ('r', 'v', 'm', 'f', 'p')+ AND (n.nspname = 'kioku'+ OR (n.nspname = 'kiroku' AND c.relname LIKE 'kioku\\_%'))+ ORDER BY n.nspname COLLATE "C", c.relname COLLATE "C"+ """+ E.noParams+ ( D.rowList+ ( (,,)+ <$> D.column (D.nonNullable D.text)+ <*> D.column (D.nonNullable D.text)+ <*> D.column (D.nonNullable D.text)+ )+ )++-- | A fresh database must land in the target layout with nothing left in the event store's+-- schema. This is the whole point of the migration stated as a catalog fact.+testRelocatedFreshLayout :: Assertion+testRelocatedFreshLayout =+ withKiokuMigratedDatabase \connStr ->+ query connStr projectionLayout >>= (@?= targetProjectionLayout)++-- | The adoption case: a host that already runs Kiroku, and whose ledger therefore holds the+-- Kiroku component and nothing else.+--+-- pg-migrate identities are component-qualified, so Kioku's composed plan recognises those rows+-- as its own first component, verifies their checksums, and skips them. Nothing about the event+-- store is re-run or rewritten; only the missing Keiro and Kioku components apply. A host that+-- also has its own migration component is a different case on purpose — the standalone runner+-- rejects a ledger row outside its plan, and that host must compose one complete plan instead.+testKirokuOnlyAdoption :: Assertion+testKirokuOnlyAdoption =+ withBareDatabase \connStr -> do+ let settings = Settings.connectionString connStr+ kirokuOnly <- kirokuOnlyPlan+ full <- either (fail . show) pure kiokuMigrationPlan++ kirokuReport <- runMigrationPlan defaultRunOptions settings kirokuOnly >>= either (assertFailure . show) pure+ length (appliedNow kirokuReport) @?= 11+ ledgerBefore <- query connStr kirokuLedgerSnapshot++ adoption <- runMigrationPlan defaultRunOptions settings full >>= either (assertFailure . show) pure+ let MigrationReport {results = adoptionResults} = adoption+ length [() | MigrationResult {outcome = AlreadyApplied} <- toList adoptionResults] @?= 11+ length [() | MigrationResult {outcome = AppliedNow} <- toList adoptionResults] @?= 42++ -- Verified and skipped, never re-executed: the stored rows keep their checksums and their+ -- original application timestamps.+ query connStr kirokuLedgerSnapshot >>= (@?= ledgerBefore)++ -- Kioku adopted the host's event store rather than standing up a second one beside it.+ query connStr eventStoreRelationsExist >>= (@?= True)+ query connStr projectionLayout >>= (@?= targetProjectionLayout)++ verification <- verifyMigrationPlan defaultRunOptions settings full >>= either (assertFailure . show) pure+ let VerificationReport {issues = adoptionIssues, appliedMigrations, pendingMigrations, unknownMigrations} = verification+ adoptionIssues @?= []+ length appliedMigrations @?= 53+ pendingMigrations @?= []+ unknownMigrations @?= []++-- | The Kiroku component on its own, which is what an existing Kiroku host's database has+-- applied before it adopts Kioku.+kirokuOnlyPlan :: IO MigrationPlan+kirokuOnlyPlan = do+ component <- either (fail . show) pure KirokuMigrations.kirokuMigrations+ either (fail . show) pure (migrationPlan (component :| []))++kirokuLedgerSnapshot :: Statement () Text+kirokuLedgerSnapshot =+ preparable+ """+ SELECT coalesce(+ string_agg(migration || ':' || encode(checksum, 'hex') || ':' || status+ || ':' || started_at::text || ':' || coalesce(finished_at::text, ''),+ E'\n' ORDER BY position),+ '')+ FROM pgmigrate.migrations+ WHERE component = 'kiroku'+ """+ E.noParams+ (D.singleRow (D.column (D.nonNullable D.text)))++eventStoreRelationsExist :: Statement () Bool+eventStoreRelationsExist =+ preparable+ "SELECT to_regclass('kiroku.events') IS NOT NULL AND to_regclass('kiroku.streams') IS NOT NULL"+ E.noParams+ (D.singleRow (D.column (D.nonNullable D.bool)))++-- | The data-bearing upgrade. @ALTER TABLE … SET SCHEMA@ and @… RENAME TO@ are catalog metadata+-- edits, so the table keeps its object identity — and that is the claim worth asserting, because+-- a relocation implemented as create-copy-drop would pass a row-count check and still silently+-- rebuild every index and drop every grant.+testRelocationPreservesTables :: Assertion+testRelocationPreservesTables =+ withPreRelocationDatabase \conn -> do+ relocation <- loadMigration relocationMigration+ run conn (Session.script relocationSeedRows)++ oidsBefore <- run conn (Session.statement () oldLayoutTableOids)+ rowsBefore <- run conn (Session.statement () oldLayoutRowCounts)+ attachmentsBefore <- run conn (Session.statement () oldLayoutAttachments)+ length oidsBefore @?= 7+ rowsBefore @?= [(target, 1) | target <- sort (snd <$> relocatedRelations)]++ run conn (Session.script relocation)++ run conn (Session.statement () projectionLayout) >>= (@?= targetProjectionLayout)+ run conn (Session.statement () newLayoutTableOids) >>= (@?= oidsBefore)+ run conn (Session.statement () newLayoutRowCounts) >>= (@?= rowsBefore)+ run conn (Session.statement () newLayoutAttachments) >>= (@?= attachmentsBefore)++-- | Rerunning the body against the finished layout is the second of the migration's only two+-- accepted states, and it must not so much as touch a catalog row.+testRelocationRerunIsNoOp :: Assertion+testRelocationRerunIsNoOp =+ withKiokuMigratedDatabase \connStr ->+ withConnection connStr \conn -> do+ relocation <- loadMigration relocationMigration+ oidsBefore <- run conn (Session.statement () newLayoutTableOids)+ attachmentsBefore <- run conn (Session.statement () newLayoutAttachments)++ run conn (Session.script relocation)++ run conn (Session.statement () projectionLayout) >>= (@?= targetProjectionLayout)+ run conn (Session.statement () newLayoutTableOids) >>= (@?= oidsBefore)+ run conn (Session.statement () newLayoutAttachments) >>= (@?= attachmentsBefore)++-- | Every layout other than wholly-old and wholly-new is refused before a single table moves.+--+-- The migration is transactional, so \"refused\" has to mean the catalog is bit-for-bit what it+-- was before the attempt — not merely that an error was reported after a partial move.+assertRelocationAborts :: Text -> Assertion+assertRelocationAborts damage =+ withPreRelocationDatabase \conn -> do+ relocation <- loadMigration relocationMigration+ run conn (Session.script damage)+ before <- run conn (Session.statement () projectionLayout)++ result <- Connection.use conn (Session.script relocation)+ case result of+ Right () -> assertFailure "the relocation accepted a layout that is neither wholly old nor wholly new"+ Left err ->+ assertBool+ ("expected a layout refusal, got: " <> show err)+ ("refusing to relocate" `Text.isInfixOf` Text.pack (show err))++ run conn (Session.statement () projectionLayout) >>= (@?= before)++moveOneTableForward :: Text+moveOneTableForward =+ """+ ALTER TABLE kiroku.kioku_turns SET SCHEMA kioku;+ ALTER TABLE kioku.kioku_turns RENAME TO turns;+ """++dropOneSourceTable :: Text+dropOneSourceTable = "DROP TABLE kiroku.kioku_personas"++occupyTargetWithTable :: Text+occupyTargetWithTable = "CREATE TABLE kioku.memories (placeholder text)"++occupyTargetWithView :: Text+occupyTargetWithView = "CREATE VIEW kioku.scenes AS SELECT 1 AS placeholder"++-- | A fully migrated database put back into the layout migration 0011 left behind: the seven+-- tables in @kiroku@ under their @kioku_@ names. The empty @kioku@ schema stays, which is also+-- the state a host is in if it created the schema ahead of the upgrade.+withPreRelocationDatabase :: (Connection.Connection -> IO a) -> IO a+withPreRelocationDatabase use =+ withKiokuMigratedDatabase \connStr ->+ withConnection connStr \conn -> do+ run conn (Session.script undoSchemaRelocation)+ use conn++undoSchemaRelocation :: Text+undoSchemaRelocation =+ """+ ALTER TABLE kioku.memories RENAME TO kioku_memories;+ ALTER TABLE kioku.kioku_memories SET SCHEMA kiroku;+ ALTER TABLE kioku.sessions RENAME TO kioku_sessions;+ ALTER TABLE kioku.kioku_sessions SET SCHEMA kiroku;+ ALTER TABLE kioku.turns RENAME TO kioku_turns;+ ALTER TABLE kioku.kioku_turns SET SCHEMA kiroku;+ ALTER TABLE kioku.l1_watermarks RENAME TO kioku_l1_watermarks;+ ALTER TABLE kioku.kioku_l1_watermarks SET SCHEMA kiroku;+ ALTER TABLE kioku.consolidation_decisions RENAME TO kioku_consolidation_decisions;+ ALTER TABLE kioku.kioku_consolidation_decisions SET SCHEMA kiroku;+ ALTER TABLE kioku.scenes RENAME TO kioku_scenes;+ ALTER TABLE kioku.kioku_scenes SET SCHEMA kiroku;+ ALTER TABLE kioku.personas RENAME TO kioku_personas;+ ALTER TABLE kioku.kioku_personas SET SCHEMA kiroku;+ """++-- | One row in each of the seven relations, in the pre-relocation layout. Each carries a memory+-- space, because this database is already past migration 0011.+relocationSeedRows :: Text+relocationSeedRows =+ """+ INSERT INTO kiroku.kioku_sessions+ (memory_space_id, session_id, agent_id, focus, namespace, started_at)+ VALUES ('space_a', 's-1', 'agent', 'focus', 'ns', now());++ INSERT INTO kiroku.kioku_turns+ (memory_space_id, turn_id, session_id, turn_index, role, content, recorded_at)+ VALUES ('space_a', 't-1', 's-1', 1, 'user', 'hello', now());++ INSERT INTO kiroku.kioku_l1_watermarks (memory_space_id, session_id, last_turn_index)+ VALUES ('space_a', 's-1', 1);++ INSERT INTO kiroku.kioku_memories+ (memory_space_id, memory_id, agent_id, session_id, namespace, memory_type,+ content, created_at, updated_at)+ VALUES ('space_a', 'm-1', 'agent', 's-1', 'ns', 'fact', 'content', now(), now());++ INSERT INTO kiroku.kioku_consolidation_decisions+ (memory_space_id, decision_id, session_id, namespace, candidate_content, decision)+ VALUES ('space_a', 'd-1', 's-1', 'ns', 'content', 'store');++ INSERT INTO kiroku.kioku_scenes+ (memory_space_id, scene_id, namespace, scene_key, title, body_md, source_hash)+ VALUES ('space_a', 'kioku_scene:ns:default', 'ns', 'default', 'title', 'body', 'hash');++ INSERT INTO kiroku.kioku_personas+ (memory_space_id, persona_id, namespace, body_md, source_hash)+ VALUES ('space_a', 'kioku_persona:ns', 'ns', 'body', 'hash');+ """++oldLayoutTableOids :: Statement () [(Text, Int64)]+oldLayoutTableOids =+ preparable+ """+ SELECT substr(c.relname, 7)::text, c.oid::int8+ FROM pg_catalog.pg_class c+ JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace+ WHERE n.nspname = 'kiroku' AND c.relkind = 'r' AND c.relname LIKE 'kioku\\_%'+ ORDER BY substr(c.relname, 7) COLLATE "C"+ """+ E.noParams+ (D.rowList ((,) <$> D.column (D.nonNullable D.text) <*> D.column (D.nonNullable D.int8)))++newLayoutTableOids :: Statement () [(Text, Int64)]+newLayoutTableOids =+ preparable+ """+ SELECT c.relname::text, c.oid::int8+ FROM pg_catalog.pg_class c+ JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace+ WHERE n.nspname = 'kioku' AND c.relkind = 'r'+ ORDER BY c.relname COLLATE "C"+ """+ E.noParams+ (D.rowList ((,) <$> D.column (D.nonNullable D.text) <*> D.column (D.nonNullable D.int8)))++oldLayoutRowCounts :: Statement () [(Text, Int64)]+oldLayoutRowCounts =+ preparable+ """+ SELECT relation, row_count+ FROM (+ SELECT 'consolidation_decisions'::text AS relation, count(*) AS row_count FROM kiroku.kioku_consolidation_decisions+ UNION ALL SELECT 'l1_watermarks'::text, count(*) FROM kiroku.kioku_l1_watermarks+ UNION ALL SELECT 'memories'::text, count(*) FROM kiroku.kioku_memories+ UNION ALL SELECT 'personas'::text, count(*) FROM kiroku.kioku_personas+ UNION ALL SELECT 'scenes'::text, count(*) FROM kiroku.kioku_scenes+ UNION ALL SELECT 'sessions'::text, count(*) FROM kiroku.kioku_sessions+ UNION ALL SELECT 'turns'::text, count(*) FROM kiroku.kioku_turns+ ) AS counts+ ORDER BY relation COLLATE "C"+ """+ E.noParams+ (D.rowList ((,) <$> D.column (D.nonNullable D.text) <*> D.column (D.nonNullable D.int8)))++newLayoutRowCounts :: Statement () [(Text, Int64)]+newLayoutRowCounts =+ preparable+ """+ SELECT relation, row_count+ FROM (+ SELECT 'consolidation_decisions'::text AS relation, count(*) AS row_count FROM kioku.consolidation_decisions+ UNION ALL SELECT 'l1_watermarks'::text, count(*) FROM kioku.l1_watermarks+ UNION ALL SELECT 'memories'::text, count(*) FROM kioku.memories+ UNION ALL SELECT 'personas'::text, count(*) FROM kioku.personas+ UNION ALL SELECT 'scenes'::text, count(*) FROM kioku.scenes+ UNION ALL SELECT 'sessions'::text, count(*) FROM kioku.sessions+ UNION ALL SELECT 'turns'::text, count(*) FROM kioku.turns+ ) AS counts+ ORDER BY relation COLLATE "C"+ """+ E.noParams+ (D.rowList ((,) <$> D.column (D.nonNullable D.text) <*> D.column (D.nonNullable D.int8)))++-- | Everything PostgreSQL is supposed to carry along with a table when it changes schema and+-- name: its owner, its table grants, the names of its indexes, and the full text of its+-- constraints. None of Kioku's seven constraints names a schema, so their definitions are+-- expected to be identical on both sides of the move rather than merely similar.+tableAttachmentsSql :: Text -> Text -> Text+tableAttachmentsSql schema shortName =+ "SELECT "+ <> shortName+ <> "::text,\n\+ \ (pg_catalog.pg_get_userbyid(c.relowner)\n\+ \ || '|' || coalesce(c.relacl::text, '')\n\+ \ || '|' || coalesce((SELECT string_agg(i.relname, ',' ORDER BY i.relname COLLATE \"C\")\n\+ \ FROM pg_catalog.pg_index x\n\+ \ JOIN pg_catalog.pg_class i ON i.oid = x.indexrelid\n\+ \ WHERE x.indrelid = c.oid), '')\n\+ \ || '|' || coalesce((SELECT string_agg(con.conname || '=' || pg_catalog.pg_get_constraintdef(con.oid),\n\+ \ ',' ORDER BY con.conname COLLATE \"C\")\n\+ \ FROM pg_catalog.pg_constraint con\n\+ \ WHERE con.conrelid = c.oid), ''))::text\n\+ \FROM pg_catalog.pg_class c\n\+ \JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace\n\+ \WHERE n.nspname = '"+ <> schema+ <> "' AND c.relkind = 'r'"+ <> (if schema == "kiroku" then " AND c.relname LIKE 'kioku\\_%'" else "")+ <> "\nORDER BY "+ <> shortName+ <> " COLLATE \"C\""++oldLayoutAttachments :: Statement () [(Text, Text)]+oldLayoutAttachments =+ preparable+ (tableAttachmentsSql "kiroku" "substr(c.relname, 7)")+ E.noParams+ (D.rowList ((,) <$> D.column (D.nonNullable D.text) <*> D.column (D.nonNullable D.text)))++newLayoutAttachments :: Statement () [(Text, Text)]+newLayoutAttachments =+ preparable+ (tableAttachmentsSql "kioku" "c.relname")+ E.noParams+ (D.rowList ((,) <$> D.column (D.nonNullable D.text) <*> D.column (D.nonNullable D.text)))+ -- * The downstream Codd cutover rehearsal -- | Exercise the operator runbook against the exact 30 historical migration@@ -260,18 +913,19 @@ let MigrationReport {results = migratedResults} = migrated appliedNow migrated @?= expectedForwardMigrationIds length [() | MigrationResult {outcome = AlreadyApplied} <- toList migratedResults] @?= 30- query connStr forwardMigrationEffectCountStatement >>= (@?= 5)+ query connStr forwardMigrationEffectCountStatement >>= (@?= 6)+ query connStr projectionLayout >>= (@?= targetProjectionLayout) verification <- verifyMigrationPlan defaultRunOptions settings plan >>= either (assertFailure . show) pure let VerificationReport {issues = verificationIssues, appliedMigrations, pendingMigrations, unknownMigrations} = verification verificationIssues @?= []- length appliedMigrations @?= 38+ length appliedMigrations @?= 53 pendingMigrations @?= [] unknownMigrations @?= [] repeated <- runMigrationPlan defaultRunOptions settings plan >>= either (assertFailure . show) pure let MigrationReport {results = repeatedResults} = repeated- length [() | MigrationResult {outcome = AlreadyApplied} <- toList repeatedResults] @?= 38+ length [() | MigrationResult {outcome = AlreadyApplied} <- toList repeatedResults] @?= 53 length [() | MigrationResult {outcome = AppliedNow} <- toList repeatedResults] @?= 0 fixtureMigrationNames :: Text -> [FilePath]@@ -339,12 +993,27 @@ expectRight <$> [ migrationId "kiroku" "0007-stream-truncate-before", migrationId "kiroku" "0008-schema-management-comment",+ migrationId "kiroku" "0009",+ migrationId "kiroku" "0010",+ migrationId "kiroku" "0011", migrationId "keiro" "0015-keiro-outbox-claim-order-index", migrationId "keiro" "0016-keiro-inbox-drop-received-idx", migrationId "keiro" "0017-schema-management-comment", migrationId "keiro" "0018", migrationId "keiro" "0019-keiro-snapshots-state-shape-hash",- migrationId "keiro" "0020-keiro-workflow-children-failure-reason"+ migrationId "keiro" "0020-keiro-workflow-children-failure-reason",+ migrationId "keiro" "0021-keiro-workflows-exact-discovery",+ migrationId "keiro" "0022",+ migrationId "keiro" "0023",+ migrationId "keiro" "0024",+ migrationId "keiro" "0025",+ migrationId "keiro" "0026",+ migrationId "keiro" "0027",+ migrationId "keiro" "0028",+ migrationId "keiro" "0029",+ migrationId "keiro" "0030",+ migrationId "kioku" "0011-kioku-memory-space-partition",+ migrationId "kioku" "0012-relocate-projections-to-kioku-schema" ] expectRight :: (Show error) => Either error value -> value@@ -389,11 +1058,13 @@ (EXISTS (SELECT 1 FROM information_schema.columns WHERE table_schema = 'kiroku' AND table_name = 'streams' AND column_name = 'truncate_before'))::int + (coalesce(obj_description(to_regnamespace('kiroku'), 'pg_namespace'), '') =- 'Managed by pg-migrate component kiroku through 0008-schema-management-comment')::int+ 'Managed by pg-migrate component kiroku through 0011')::int + (to_regclass('keiro.keiro_outbox_claim_order_idx') IS NOT NULL)::int + (to_regclass('keiro.keiro_inbox_received_idx') IS NULL)::int + (coalesce(obj_description(to_regnamespace('keiro'), 'pg_namespace'), '') = 'Managed by pg-migrate component keiro through 0017-schema-management-comment')::int+ + (to_regclass('kioku.memories') IS NOT NULL+ AND to_regclass('kiroku.kioku_memories') IS NULL)::int )::bigint """ E.noParams