diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,62 @@ # Changelog +## 0.4.0.0 — 2026-08-17 + +### Breaking Changes + +- `0012-relocate-projections-to-kioku-schema.sql` moves the seven Kioku-owned projections out of + the `kiroku` schema and into a dedicated `kioku` one, dropping the now-redundant name prefix: + `kiroku.kioku_memories` becomes `kioku.memories`, and likewise for `sessions`, `turns`, + `l1_watermarks`, `consolidation_decisions`, `scenes`, and `personas`. The shared Kiroku event + store does not move. `ALTER TABLE ... SET SCHEMA` plus `... RENAME TO` preserve each table's + OID, rows, indexes, constraints, owner, and grants; index and constraint names are deliberately + unchanged. The `vector` extension is not touched. +- The migration accepts exactly two catalog states — seven ordinary source tables with no occupied + target, or no source with seven ordinary targets — and raises before moving anything otherwise. + It is transactional, so a refusal leaves the catalog exactly as it was found. There are no + compatibility views: run it with writers stopped, before the new binary starts, and see + `docs/user/upgrading-to-the-kioku-schema.md`. +- The composed fresh-install plan is now 53 migrations: kiroku 11, keiro 30, kioku 12. +- Moved to `keiro-migrations ^>=0.13.0.0` and `kiroku-store-migrations ^>=0.4.0.0`. Those bring + eleven new framework migrations that Kioku composes but does not own: kiroku `0009` (the frozen + `kiroku.subscription_checkpoints_v1` view), `0010` (replay-history retention leases) and `0011`, + and keiro `0021`–`0030` (exact workflow discovery, then the schema-versioned projection rebuild, + promotion, and external-read-guard machinery). Keiro `0024` renames a rebuild-group column, so + complete or abandon any active catalog rebuild **before** upgrading. +- `kiroku-store-migrations` 0.4.0.0 corrects the payload of kiroku `0010`, which changes its + checksum. A database that already applied `0010` from the withdrawn 0.3.2.0 or 0.3.2.1 fails + every `up` and `verify` with `MigrationChecksumMismatch` until its ledger row is re-baselined + with that package's `ledger-fixups/2026-08-16-rebaseline-0010-checksum.sql`; kiroku `0011` then + converges its schema. A database that never reached `0010` — which is every database blocked by + the PostgreSQL 17 defect this release fixes — needs neither, and applies the corrected `0010` + as an ordinary pending migration. + + +### Added + +- `0011-kioku-memory-space-partition.sql` adds a non-null `memory_space_id` to `kioku_memories`, + `kioku_sessions`, `kioku_turns`, `kioku_l1_watermarks`, `kioku_consolidation_decisions`, + `kioku_scenes`, and `kioku_personas`. Existing rows are backfilled into the explicit + `kioku_legacy` space; turns and watermarks derive theirs from their parent session, and the + migration aborts if any derived row disagrees with a session that exists. + +### Breaking Changes + +- `kioku_scenes` and `kioku_personas` now have composite primary keys, + `(memory_space_id, scene_id)` and `(memory_space_id, persona_id)`, and their scope-uniqueness + constraints gained the space. Their ids are derived from the namespace and scope alone, which + two spaces are allowed to share, so a single-column key let one space's upsert overwrite the + other's row. Stored id strings are unchanged. +- Indexes led by a namespace or a scope were rebuilt partition-first and renamed: + `kioku_memories_scope_idx` → `kioku_memories_space_scope_idx`, `kioku_memories_type_idx` → + `kioku_memories_space_type_idx` (now covering namespace as well), `kioku_sessions_scope_idx`, + `kioku_sessions_namespace_started_idx`, `kioku_sessions_namespace_focus_idx`, + `kioku_sessions_awaiting_corr_idx`, and `kioku_consolidation_scope_idx` likewise. + `kioku_memories_space_namespace_idx` is new. `kioku_scenes_scope_idx` is dropped as a strict + prefix of the per-space unique index. +- The migration is forward-only and has no down step. Re-running it is a no-op. Rolling the + application back is safe only while nothing has been written outside `kioku_legacy`. + ## 0.3.0.0 — 2026-08-05 ### Breaking Changes diff --git a/kioku-migrations.cabal b/kioku-migrations.cabal --- a/kioku-migrations.cabal +++ b/kioku-migrations.cabal @@ -1,6 +1,6 @@ cabal-version: 3.0 name: kioku-migrations -version: 0.3.0.0 +version: 0.4.0.0 synopsis: Schema migrations for kioku description: Owns kioku schema evolution as a native pg-migrate component and composes @@ -58,8 +58,8 @@ , containers >=0.6 && <0.8 , hasql >=1.10 && <1.11 , hasql-transaction >=1.2 && <1.3 - , keiro-migrations ^>=0.11.0.0 - , kiroku-store-migrations ^>=0.3.0.0 + , keiro-migrations ^>=0.13.0.0 + , kiroku-store-migrations ^>=0.4.0.0 , pg-migrate ^>=1.1.0.0 , pg-migrate-embed ^>=1.1.0.0 , pg-migrate-import-codd ^>=1.1.0.0 @@ -75,7 +75,7 @@ , base >=4.18 && <5 , ephemeral-pg ^>=0.2.2.0 , hasql >=1.10 && <1.11 - , kioku-migrations ^>=0.3.0.0 + , kioku-migrations ^>=0.4.0.0 , pg-migrate-test-support ^>=1.1.0.0 , text >=2.0 && <2.2 @@ -91,8 +91,9 @@ build-depends: , base >=4.18 && <5 , hasql >=1.6 - , kioku-migrations ^>=0.3.0.0 + , kioku-migrations ^>=0.4.0.0 , kioku-migrations:test-support + , kiroku-store-migrations ^>=0.4.0.0 , pg-migrate ^>=1.1.0.0 , pg-migrate-embed ^>=1.1.0.0 , pg-migrate-import-codd ^>=1.1.0.0 diff --git a/migrations/0011-kioku-memory-space-partition.sql b/migrations/0011-kioku-memory-space-partition.sql new file mode 100644 --- /dev/null +++ b/migrations/0011-kioku-memory-space-partition.sql @@ -0,0 +1,226 @@ +-- Migration: kioku-memory-space-partition +-- Created: 2026-08-06-15-10-00 UTC +-- +-- Every kioku read-model row becomes attributable to exactly one memory space. +-- +-- A memory space is the outer isolation boundary: two tenants who must never see each +-- other's data are separated by memory space, whereas two hosts sharing a database are +-- merely separated by namespace. Until this migration the read models had no such column, +-- so a query could only ever see every space at once. See +-- docs/adr/namespace-is-not-a-security-boundary.md. +-- +-- The column is added in the only safe order: nullable, deterministic backfill, validation, +-- then NOT NULL, then the constraints and indexes that put the partition first. A row whose +-- space were NULL would have to be interpreted by every query, and the only two available +-- interpretations -- "invisible" and "visible everywhere" -- are both wrong. Absence of a +-- partition must never read as unrestricted access, so every pre-existing row is backfilled +-- into the single explicit space `kioku_legacy`. See +-- docs/adr/legacy-data-lands-in-one-explicit-space.md. +-- +-- Turns and L1 watermarks derive their space from their parent session rather than being +-- assumed legacy, because that derivation is the rule that must hold forever; the fallback +-- for a turn whose session row is missing is the legacy space, and the validation block +-- below fails the whole migration if any turn or watermark ends up disagreeing with a +-- session that does exist. +-- +-- Every statement is idempotent: the columns and indexes use IF NOT EXISTS, the constraints +-- are dropped before being re-added, and the backfill matches only rows that are still NULL. +SET search_path TO kiroku, pg_catalog; + +-- Step 1: the column, nullable, on every table that holds partitioned data. + +ALTER TABLE kioku_memories ADD COLUMN IF NOT EXISTS memory_space_id text; +ALTER TABLE kioku_sessions ADD COLUMN IF NOT EXISTS memory_space_id text; +ALTER TABLE kioku_turns ADD COLUMN IF NOT EXISTS memory_space_id text; +ALTER TABLE kioku_l1_watermarks ADD COLUMN IF NOT EXISTS memory_space_id text; +ALTER TABLE kioku_consolidation_decisions ADD COLUMN IF NOT EXISTS memory_space_id text; +ALTER TABLE kioku_scenes ADD COLUMN IF NOT EXISTS memory_space_id text; +ALTER TABLE kioku_personas ADD COLUMN IF NOT EXISTS memory_space_id text; + +-- Step 2: the deterministic backfill. +-- +-- Roots first. Every row that exists at this point predates memory spaces, so it belongs to +-- the legacy space by definition -- there is no other space it could have been written in. + +UPDATE kioku_memories SET memory_space_id = 'kioku_legacy' WHERE memory_space_id IS NULL; +UPDATE kioku_sessions SET memory_space_id = 'kioku_legacy' WHERE memory_space_id IS NULL; +UPDATE kioku_consolidation_decisions SET memory_space_id = 'kioku_legacy' WHERE memory_space_id IS NULL; +UPDATE kioku_scenes SET memory_space_id = 'kioku_legacy' WHERE memory_space_id IS NULL; +UPDATE kioku_personas SET memory_space_id = 'kioku_legacy' WHERE memory_space_id IS NULL; + +-- Derived rows inherit from the session they belong to. `kioku_turns.session_id` has no +-- foreign key, so an orphan turn is possible in principle; it falls back to the legacy +-- space, which is where its session would have been had it survived. + +UPDATE kioku_turns t + SET memory_space_id = + COALESCE( + (SELECT s.memory_space_id FROM kioku_sessions s WHERE s.session_id = t.session_id), + 'kioku_legacy') + WHERE t.memory_space_id IS NULL; + +UPDATE kioku_l1_watermarks w + SET memory_space_id = + COALESCE( + (SELECT s.memory_space_id FROM kioku_sessions s WHERE s.session_id = w.session_id), + 'kioku_legacy') + WHERE w.memory_space_id IS NULL; + +-- Step 3: validation, before anything irreversible. +-- +-- Abort rather than write a partition nobody can trust. The first check is the one that +-- matters on a fresh upgrade (nothing may be left unattributed); the second and third are +-- the derivation rule, which is vacuous today but is the invariant every later write must +-- preserve, and stating it here is what makes a future violation loud. +DO $$ +DECLARE + unattributed bigint; + drifted bigint; +BEGIN + SELECT (SELECT count(*) FROM kioku_memories WHERE memory_space_id IS NULL) + + (SELECT count(*) FROM kioku_sessions WHERE memory_space_id IS NULL) + + (SELECT count(*) FROM kioku_turns WHERE memory_space_id IS NULL) + + (SELECT count(*) FROM kioku_l1_watermarks WHERE memory_space_id IS NULL) + + (SELECT count(*) FROM kioku_consolidation_decisions WHERE memory_space_id IS NULL) + + (SELECT count(*) FROM kioku_scenes WHERE memory_space_id IS NULL) + + (SELECT count(*) FROM kioku_personas WHERE memory_space_id IS NULL) + INTO unattributed; + + IF unattributed > 0 THEN + RAISE EXCEPTION + 'kioku memory-space backfill left % row(s) with no memory space; refusing to continue', + unattributed; + END IF; + + SELECT (SELECT count(*) + FROM kioku_turns t + JOIN kioku_sessions s ON s.session_id = t.session_id + WHERE t.memory_space_id IS DISTINCT FROM s.memory_space_id) + + (SELECT count(*) + FROM kioku_l1_watermarks w + JOIN kioku_sessions s ON s.session_id = w.session_id + WHERE w.memory_space_id IS DISTINCT FROM s.memory_space_id) + INTO drifted; + + IF drifted > 0 THEN + RAISE EXCEPTION + 'kioku memory-space backfill left % derived row(s) in a different space from their session', + drifted; + END IF; +END $$; + +-- Step 4: make the column mandatory. An empty string is rejected too: it is not a space, +-- and allowing it would give "no space" a second spelling. + +ALTER TABLE kioku_memories ALTER COLUMN memory_space_id SET NOT NULL; +ALTER TABLE kioku_sessions ALTER COLUMN memory_space_id SET NOT NULL; +ALTER TABLE kioku_turns ALTER COLUMN memory_space_id SET NOT NULL; +ALTER TABLE kioku_l1_watermarks ALTER COLUMN memory_space_id SET NOT NULL; +ALTER TABLE kioku_consolidation_decisions ALTER COLUMN memory_space_id SET NOT NULL; +ALTER TABLE kioku_scenes ALTER COLUMN memory_space_id SET NOT NULL; +ALTER TABLE kioku_personas ALTER COLUMN memory_space_id SET NOT NULL; + +ALTER TABLE kioku_memories DROP CONSTRAINT IF EXISTS kioku_memories_space_present_check; +ALTER TABLE kioku_memories ADD CONSTRAINT kioku_memories_space_present_check + CHECK (memory_space_id <> ''); +ALTER TABLE kioku_sessions DROP CONSTRAINT IF EXISTS kioku_sessions_space_present_check; +ALTER TABLE kioku_sessions ADD CONSTRAINT kioku_sessions_space_present_check + CHECK (memory_space_id <> ''); +ALTER TABLE kioku_turns DROP CONSTRAINT IF EXISTS kioku_turns_space_present_check; +ALTER TABLE kioku_turns ADD CONSTRAINT kioku_turns_space_present_check + CHECK (memory_space_id <> ''); +ALTER TABLE kioku_l1_watermarks DROP CONSTRAINT IF EXISTS kioku_l1_watermarks_space_present_check; +ALTER TABLE kioku_l1_watermarks ADD CONSTRAINT kioku_l1_watermarks_space_present_check + CHECK (memory_space_id <> ''); +ALTER TABLE kioku_consolidation_decisions DROP CONSTRAINT IF EXISTS kioku_consolidation_decisions_space_present_check; +ALTER TABLE kioku_consolidation_decisions ADD CONSTRAINT kioku_consolidation_decisions_space_present_check + CHECK (memory_space_id <> ''); +ALTER TABLE kioku_scenes DROP CONSTRAINT IF EXISTS kioku_scenes_space_present_check; +ALTER TABLE kioku_scenes ADD CONSTRAINT kioku_scenes_space_present_check + CHECK (memory_space_id <> ''); +ALTER TABLE kioku_personas DROP CONSTRAINT IF EXISTS kioku_personas_space_present_check; +ALTER TABLE kioku_personas ADD CONSTRAINT kioku_personas_space_present_check + CHECK (memory_space_id <> ''); + +-- Step 5: identities that were derived from a scope must now be derived from a space and a +-- scope together. +-- +-- Scene and persona primary keys are computed from the namespace and scope alone +-- (Kioku.Distill.ScopeIdentity). Two memory spaces are allowed to use identical namespaces +-- and scopes -- that is the whole point of the partition -- so those ids collide across +-- spaces, and with a single-column primary key one space's scene would silently overwrite +-- another's through the upsert's ON CONFLICT clause. +-- +-- The fix is a composite primary key rather than a new id derivation. Re-deriving the ids +-- would rewrite every scene and persona row and every plaintext mirror filename for a +-- cosmetic gain; the composite key makes the collision impossible while leaving already +-- persisted ids exactly as they are. + +ALTER TABLE kioku_scenes DROP CONSTRAINT IF EXISTS kioku_scenes_pkey; +ALTER TABLE kioku_scenes ADD CONSTRAINT kioku_scenes_pkey + PRIMARY KEY (memory_space_id, scene_id); + +ALTER TABLE kioku_personas DROP CONSTRAINT IF EXISTS kioku_personas_pkey; +ALTER TABLE kioku_personas ADD CONSTRAINT kioku_personas_pkey + PRIMARY KEY (memory_space_id, persona_id); + +-- Scope uniqueness is now per space. NULLS NOT DISTINCT is preserved from the +-- schema-hardening migration: without it two global-scope rows in one space would both +-- satisfy the constraint, because SQL considers NULLs distinct. + +ALTER TABLE kioku_scenes DROP CONSTRAINT IF EXISTS kioku_scenes_scope_scene_key_unique; +ALTER TABLE kioku_scenes ADD CONSTRAINT kioku_scenes_scope_scene_key_unique + UNIQUE NULLS NOT DISTINCT (memory_space_id, namespace, scope_kind, scope_ref, scene_key); + +ALTER TABLE kioku_personas DROP CONSTRAINT IF EXISTS kioku_personas_scope_unique; +ALTER TABLE kioku_personas ADD CONSTRAINT kioku_personas_scope_unique + UNIQUE NULLS NOT DISTINCT (memory_space_id, namespace, scope_kind, scope_ref); + +-- Step 6: lookup indexes, partition first. +-- +-- Every index whose leading columns were a namespace or a scope is rebuilt with the memory +-- space in front of them, because every such query now carries a space predicate and the +-- space is the most selective column available. Indexes led by a globally unique id +-- (memory_id, session_id, parent_session_id, supersedes, superseded_by, content_hash) are +-- deliberately left alone: an id is already selective enough that prefixing it with the +-- space would buy nothing and cost a write on every insert. The space predicate is still in +-- those statements -- it is a filter there, not an access path. + +DROP INDEX IF EXISTS kioku_memories_scope_idx; +CREATE INDEX IF NOT EXISTS kioku_memories_space_scope_idx + ON kioku_memories (memory_space_id, namespace, scope_kind, scope_ref) WHERE status = 'active'; + +-- The old type index covered `memory_type` alone, while the query it exists for filters on +-- namespace and type together. Rebuilding it partition-first also fixes that. +DROP INDEX IF EXISTS kioku_memories_type_idx; +CREATE INDEX IF NOT EXISTS kioku_memories_space_type_idx + ON kioku_memories (memory_space_id, namespace, memory_type) WHERE status = 'active'; + +CREATE INDEX IF NOT EXISTS kioku_memories_space_namespace_idx + ON kioku_memories (memory_space_id, namespace, priority, created_at DESC) WHERE status = 'active'; + +DROP INDEX IF EXISTS kioku_sessions_scope_idx; +CREATE INDEX IF NOT EXISTS kioku_sessions_space_scope_idx + ON kioku_sessions (memory_space_id, namespace, scope_kind, scope_ref); + +DROP INDEX IF EXISTS kioku_sessions_namespace_started_idx; +CREATE INDEX IF NOT EXISTS kioku_sessions_space_namespace_started_idx + ON kioku_sessions (memory_space_id, namespace, started_at DESC); + +DROP INDEX IF EXISTS kioku_sessions_namespace_focus_idx; +CREATE INDEX IF NOT EXISTS kioku_sessions_space_namespace_focus_idx + ON kioku_sessions (memory_space_id, namespace, focus, started_at DESC); + +DROP INDEX IF EXISTS kioku_sessions_awaiting_corr_idx; +CREATE INDEX IF NOT EXISTS kioku_sessions_space_awaiting_corr_idx + ON kioku_sessions (memory_space_id, namespace, awaiting_correlation_key) + WHERE status = 'awaiting'; + +DROP INDEX IF EXISTS kioku_consolidation_scope_idx; +CREATE INDEX IF NOT EXISTS kioku_consolidation_space_scope_idx + ON kioku_consolidation_decisions (memory_space_id, namespace, scope_kind, scope_ref); + +-- The scene scope index is now a strict prefix of the unique constraint's index, exactly as +-- kioku_turns_session_idx was of UNIQUE (session_id, turn_index) before the schema-hardening +-- migration dropped it. Keeping it would be pure write amplification. +DROP INDEX IF EXISTS kioku_scenes_scope_idx; diff --git a/migrations/0012-relocate-projections-to-kioku-schema.sql b/migrations/0012-relocate-projections-to-kioku-schema.sql new file mode 100644 --- /dev/null +++ b/migrations/0012-relocate-projections-to-kioku-schema.sql @@ -0,0 +1,127 @@ +-- Migration: relocate-projections-to-kioku-schema +-- Created: 2026-08-07-18-20-00 UTC +-- +-- Kioku's seven projection tables move out of Kiroku's schema and into their own. +-- +-- Kioku deliberately shares the host application's Kiroku event store: that sharing is the +-- integration boundary, and this migration does not touch it. What it does change is where the +-- relations *Kioku* owns live. Until now they sat in `kiroku` alongside the event store's own +-- tables, distinguished only by a `kioku_` name prefix that nothing in the catalog enforces, so +-- a host that already ran Kiroku found another product's projections inside its event store's +-- namespace. After this migration every Kioku-owned relation is in the `kioku` schema, and the +-- prefix is dropped because the schema now supplies the namespace it used to stand in for. See +-- docs/adr/projections-live-in-the-kioku-schema.md. +-- +-- kiroku.kioku_memories -> kioku.memories +-- kiroku.kioku_sessions -> kioku.sessions +-- kiroku.kioku_turns -> kioku.turns +-- kiroku.kioku_l1_watermarks -> kioku.l1_watermarks +-- kiroku.kioku_consolidation_decisions -> kioku.consolidation_decisions +-- kiroku.kioku_scenes -> kioku.scenes +-- kiroku.kioku_personas -> kioku.personas +-- +-- This is a metadata operation, not a copy. `ALTER TABLE ... SET SCHEMA` and +-- `ALTER TABLE ... RENAME TO` both keep the table's object identity, so its rows, indexes, +-- constraints, owner, and table grants come along untouched and nothing is rebuilt. Index and +-- constraint names are deliberately *not* renamed: they travel by identity, and renaming them +-- would churn the catalog for no gain. None of the seven tables has a sequence, trigger, or +-- foreign key needing separate movement. +-- +-- The `vector` extension is not moved and must not be. A PostgreSQL extension is a +-- database-wide object the host may share, so `ALTER EXTENSION` is out of scope here. The +-- memory table keeps its `vector` column and its HNSW index by identity, and the runtime's +-- `to_regtype('vector')` probe keeps resolving against the connection's search path exactly as +-- 0009-kioku-embedding-schema-heal.sql arranged. +-- +-- Exactly two catalog states are accepted, and every other state aborts before a single table +-- moves. A partial or colliding layout is never repaired heuristically, because there is no way +-- to tell an interrupted upgrade apart from a host relation that happens to share a target name, +-- and guessing wrong would mean moving somebody else's table or writing into it. The migration +-- is transactional, so a refusal leaves the catalog exactly as it was found. +-- +-- 1. all seven sources present as ordinary tables, no target name occupied -> move them +-- 2. no source name occupied at all, all seven targets present as ordinary tables -> no-op +-- +-- The occupancy test uses `to_regclass`, which resolves any relation: a view, materialized view, +-- foreign table, sequence, or index sitting on a target name blocks the move, because all of +-- them share PostgreSQL's relation namespace and any one of them would make the rename fail. + +CREATE SCHEMA IF NOT EXISTS kioku; + +COMMENT ON SCHEMA kioku IS + 'Kioku-owned read-model projections. Managed by pg-migrate component kioku.'; + +DO $$ +DECLARE + source_names CONSTANT text[] := ARRAY[ + 'kioku_memories', + 'kioku_sessions', + 'kioku_turns', + 'kioku_l1_watermarks', + 'kioku_consolidation_decisions', + 'kioku_scenes', + 'kioku_personas' + ]; + target_names CONSTANT text[] := ARRAY[ + 'memories', + 'sessions', + 'turns', + 'l1_watermarks', + 'consolidation_decisions', + 'scenes', + 'personas' + ]; + expected CONSTANT integer := array_length(source_names, 1); + slot integer; + source_relation regclass; + target_relation regclass; + relation_kind "char"; + present_sources integer := 0; + ordinary_sources integer := 0; + present_targets integer := 0; + ordinary_targets integer := 0; +BEGIN + -- Pass one counts. Nothing is altered until the whole layout has been classified, so a + -- collision on the seventh table cannot be discovered after the first six have moved. + FOR slot IN 1 .. expected LOOP + source_relation := to_regclass('kiroku.' || quote_ident(source_names[slot])); + target_relation := to_regclass('kioku.' || quote_ident(target_names[slot])); + + IF source_relation IS NOT NULL THEN + present_sources := present_sources + 1; + SELECT c.relkind INTO relation_kind + FROM pg_catalog.pg_class c + WHERE c.oid = source_relation; + IF relation_kind = 'r' THEN + ordinary_sources := ordinary_sources + 1; + END IF; + END IF; + + IF target_relation IS NOT NULL THEN + present_targets := present_targets + 1; + SELECT c.relkind INTO relation_kind + FROM pg_catalog.pg_class c + WHERE c.oid = target_relation; + IF relation_kind = 'r' THEN + ordinary_targets := ordinary_targets + 1; + END IF; + END IF; + END LOOP; + + IF present_sources = expected AND ordinary_sources = expected AND present_targets = 0 THEN + FOR slot IN 1 .. expected LOOP + EXECUTE format('ALTER TABLE kiroku.%I SET SCHEMA kioku', source_names[slot]); + EXECUTE format('ALTER TABLE kioku.%I RENAME TO %I', source_names[slot], target_names[slot]); + END LOOP; + ELSIF present_sources = 0 AND present_targets = expected AND ordinary_targets = expected THEN + -- Already relocated. Re-running the body is a no-op by construction, not by accident. + NULL; + ELSE + RAISE EXCEPTION + 'refusing to relocate Kioku projections: expected either % ordinary kiroku.kioku_* tables ' + 'with no kioku.* target relation, or no kiroku.kioku_* relation with % ordinary kioku.* ' + 'tables; found % source relation(s) of which % ordinary, and % target relation(s) of ' + 'which % ordinary', + expected, expected, present_sources, ordinary_sources, present_targets, ordinary_targets; + END IF; +END $$; diff --git a/migrations/manifest b/migrations/manifest --- a/migrations/manifest +++ b/migrations/manifest @@ -8,3 +8,5 @@ 0008-kioku-schema-hardening.sql 0009-kioku-embedding-schema-heal.sql 0010-kioku-scope-identity-recompute.sql +0011-kioku-memory-space-partition.sql +0012-relocate-projections-to-kioku-schema.sql diff --git a/test/Main.hs b/test/Main.hs --- a/test/Main.hs +++ b/test/Main.hs @@ -13,6 +13,8 @@ import Control.Exception (bracket) import Data.Foldable (toList) import Data.Int (Int64) +import Data.List (sort) +import Data.List.NonEmpty (NonEmpty (..)) import Data.Maybe (mapMaybe) import Data.Text (Text) import Data.Text qualified as Text @@ -26,6 +28,7 @@ HistoryImportResult (..), MigrationId, MigrationOutcome (AlreadyApplied, AppliedNow), + MigrationPlan, MigrationReport (..), MigrationResult (..), VerificationReport (..), @@ -33,6 +36,7 @@ defaultImportOptions, defaultRunOptions, migrationId, + migrationPlan, runMigrationPlan, validateHistoryMappingTargets, verifyMigrationPlan, @@ -55,8 +59,9 @@ kiokuCoddHistoryMappings, ) import Kioku.Migrations.TestSupport (withBareDatabase, withKiokuMigratedDatabase) +import Kiroku.Store.Migrations qualified as KirokuMigrations import Test.Tasty (TestTree, defaultMain, testGroup) -import Test.Tasty.HUnit (Assertion, assertFailure, testCase, (@?=)) +import Test.Tasty.HUnit (Assertion, assertBool, assertFailure, testCase, (@?=)) main :: IO () main = defaultMain tests @@ -74,7 +79,27 @@ testCase "the full migration chain applies to a fresh database" testFreshDatabase, testCase "the migration manifest is complete and valid" testManifestIntegrity, testCase "the pinned Codd history maps 30 known plan targets" testHistoryMappings, - testCase "the pre-cutover Codd cohort imports 30 rows and applies only eight forward migrations" testCoddCohortImport + testCase "the pre-cutover Codd cohort imports 30 rows and applies only the forward migrations" testCoddCohortImport, + testGroup + "the memory-space partition migration" + [ testCase "backfills every pre-partition row into the legacy space" testMemorySpaceBackfill, + testCase "refuses to finish when a derived row disagrees with its session" testMemorySpaceDriftAborts, + testCase "re-applying its body changes nothing" testMemorySpaceBackfillIdempotent + ], + testGroup + "the projection schema relocation" + [ testCase "a fresh database owns seven relations in kioku and none in kiroku" testRelocatedFreshLayout, + testCase "a Kiroku-only ledger adopts Kioku without replaying Kiroku" testKirokuOnlyAdoption, + testCase "a data-bearing upgrade keeps every row, OID, index, and grant" testRelocationPreservesTables, + testCase "re-applying its body against the new layout changes nothing" testRelocationRerunIsNoOp, + testGroup + "a layout that is neither wholly old nor wholly new aborts and changes nothing" + [ testCase "one table already moved" (assertRelocationAborts moveOneTableForward), + testCase "one source table missing" (assertRelocationAborts dropOneSourceTable), + testCase "a target name taken by a table" (assertRelocationAborts occupyTargetWithTable), + testCase "a target name taken by a view" (assertRelocationAborts occupyTargetWithView) + ] + ] ] -- * The manifest guard @@ -164,18 +189,20 @@ ) ) --- | The migration's bytes as they were compiled into this binary, so the test exercises --- exactly what ships. @-- codd:@ directives are ordinary SQL comments, so the whole file --- runs as one script. loadRegistryBumpMigration :: IO Text -loadRegistryBumpMigration = do +loadRegistryBumpMigration = + loadMigration "0006-kioku-session-readmodel-registry-bump.sql" + +-- | A migration's bytes as they were compiled into this binary, so a test exercises exactly +-- what ships. @-- codd:@ directives are ordinary SQL comments, so a whole file runs as one +-- script. +loadMigration :: FilePath -> IO Text +loadMigration name = do result <- checkMigrationManifest "migrations/manifest" entries <- either (fail . show) pure result case lookup name (toList entries) of Nothing -> fail ("no manifest migration named " <> name) Just bytes -> pure (Text.Encoding.decodeUtf8 bytes) - where - name = "0006-kioku-session-readmodel-registry-bump.sql" -- * The fresh-database test @@ -194,6 +221,632 @@ E.noParams (D.singleRow (D.column (D.nonNullable D.bool))) +-- * The memory-space partition migration + +-- | Prove the backfill on data that genuinely predates the partition. +-- +-- The migrated database already has the column, so the test first puts it back the way it +-- was: 'undoMemorySpacePartition' drops every @memory_space_id@ (taking its indexes and +-- constraints with it) and restores the single-column scene and persona primary keys. Rows +-- inserted after that are indistinguishable from rows written by an older kioku, which is +-- what makes running the shipped migration bytes over them a test of the real upgrade +-- rather than of a mock of one. +testMemorySpaceBackfill :: Assertion +testMemorySpaceBackfill = + withPrePartitionDatabase \conn -> do + partition <- loadMigration memorySpacePartitionMigration + run conn (Session.script partition) + + run conn (Session.statement () countMemorySpaces) + >>= (@?= [("kioku_legacy", 8)]) + + -- The turn whose session row is missing is the one case the derivation cannot answer + -- from a parent, and it must still land somewhere explicit. + run conn (Session.statement "t-orphan" selectTurnSpace) + >>= (@?= Just "kioku_legacy") + + -- The partition-leading indexes and the composite scene/persona keys are what make the + -- boundary enforceable rather than merely recorded. + indexes <- run conn (Session.statement () selectKiokuIndexes) + mapM_ + (\name -> assertBool (Text.unpack name <> " is missing") (name `elem` indexes)) + [ "kioku_memories_space_scope_idx", + "kioku_memories_space_type_idx", + "kioku_memories_space_namespace_idx", + "kioku_sessions_space_scope_idx", + "kioku_sessions_space_namespace_started_idx", + "kioku_sessions_space_namespace_focus_idx", + "kioku_sessions_space_awaiting_corr_idx", + "kioku_consolidation_space_scope_idx" + ] + assertBool + "kioku_scenes_scope_idx still exists; it duplicates the prefix of kioku_scenes_scope_scene_key_unique" + ("kioku_scenes_scope_idx" `notElem` indexes) + + run conn (Session.statement "kioku_scenes" selectPrimaryKeyColumns) + >>= (@?= ["memory_space_id", "scene_id"]) + run conn (Session.statement "kioku_personas" selectPrimaryKeyColumns) + >>= (@?= ["memory_space_id", "persona_id"]) + + -- Two spaces may now hold the same namespace, scope, and scene key. Before the composite + -- key this pair collided on the scope-derived primary key and one silently overwrote the + -- other through the upsert's ON CONFLICT clause. + run conn (Session.script twoSpacesOneScopeKey) + +-- | The derivation rule is a guard, not a comment: a turn that disagrees with its session +-- aborts the whole migration instead of quietly persisting a row in the wrong space. +-- +-- Tampering after a first successful pass is how the state is reached, because the backfill +-- itself cannot produce it. On the second pass the @IS NULL@ updates match nothing and the +-- validation block is what runs. +testMemorySpaceDriftAborts :: Assertion +testMemorySpaceDriftAborts = + withPrePartitionDatabase \conn -> do + partition <- loadMigration memorySpacePartitionMigration + run conn (Session.script partition) + run conn (Session.script "UPDATE kiroku.kioku_turns SET memory_space_id = 'space_elsewhere' WHERE turn_id = 't-1'") + + result <- Connection.use conn (Session.script partition) + case result of + Right () -> assertFailure "the migration accepted a turn in a different space from its session" + Left err -> + assertBool + ("expected a drift failure, got: " <> show err) + ("different space from their session" `Text.isInfixOf` Text.pack (show err)) + +-- | Re-applying the body must be a no-op, which is what makes it safe to re-run after a +-- partially failed deployment. +testMemorySpaceBackfillIdempotent :: Assertion +testMemorySpaceBackfillIdempotent = + withPrePartitionDatabase \conn -> do + partition <- loadMigration memorySpacePartitionMigration + run conn (Session.script partition) + before <- run conn (Session.statement () kiokuSchemaSnapshot) + spacesBefore <- run conn (Session.statement () countMemorySpaces) + + run conn (Session.script partition) + run conn (Session.statement () kiokuSchemaSnapshot) >>= (@?= before) + run conn (Session.statement () countMemorySpaces) >>= (@?= spacesBefore) + +memorySpacePartitionMigration :: FilePath +memorySpacePartitionMigration = "0011-kioku-memory-space-partition.sql" + +-- | A fully migrated database rolled back to the shape it had before memory spaces, then +-- seeded with one row in every partitioned table. +-- +-- Migration 0011 predates the @kioku@ schema, so it names @kiroku.kioku_*@ throughout. The +-- database has to be walked back past 0012 before 0011 can be tested in the layout it was +-- written for; 'undoSchemaRelocation' is that step. It is a test fixture, not an operator +-- procedure — a released migration is never reversed in a real database. +withPrePartitionDatabase :: (Connection.Connection -> IO a) -> IO a +withPrePartitionDatabase use = + withPreRelocationDatabase \conn -> do + run conn (Session.script undoMemorySpacePartition) + run conn (Session.script prePartitionRows) + use conn + +-- | @DROP COLUMN … CASCADE@ takes the column's indexes and constraints with it, including the +-- composite primary keys, which is why the single-column ones have to be put back by hand. +-- The rest of the DDL below is the pre-partition index set verbatim. +undoMemorySpacePartition :: Text +undoMemorySpacePartition = + """ + ALTER TABLE kiroku.kioku_memories DROP COLUMN memory_space_id CASCADE; + ALTER TABLE kiroku.kioku_sessions DROP COLUMN memory_space_id CASCADE; + ALTER TABLE kiroku.kioku_turns DROP COLUMN memory_space_id CASCADE; + ALTER TABLE kiroku.kioku_l1_watermarks DROP COLUMN memory_space_id CASCADE; + ALTER TABLE kiroku.kioku_consolidation_decisions DROP COLUMN memory_space_id CASCADE; + ALTER TABLE kiroku.kioku_scenes DROP COLUMN memory_space_id CASCADE; + ALTER TABLE kiroku.kioku_personas DROP COLUMN memory_space_id CASCADE; + + ALTER TABLE kiroku.kioku_scenes ADD CONSTRAINT kioku_scenes_pkey PRIMARY KEY (scene_id); + ALTER TABLE kiroku.kioku_personas ADD CONSTRAINT kioku_personas_pkey PRIMARY KEY (persona_id); + ALTER TABLE kiroku.kioku_scenes ADD CONSTRAINT kioku_scenes_scope_scene_key_unique + UNIQUE NULLS NOT DISTINCT (namespace, scope_kind, scope_ref, scene_key); + ALTER TABLE kiroku.kioku_personas ADD CONSTRAINT kioku_personas_scope_unique + UNIQUE NULLS NOT DISTINCT (namespace, scope_kind, scope_ref); + + CREATE INDEX kioku_memories_scope_idx + ON kiroku.kioku_memories (namespace, scope_kind, scope_ref) WHERE status = 'active'; + CREATE INDEX kioku_memories_type_idx + ON kiroku.kioku_memories (memory_type) WHERE status = 'active'; + CREATE INDEX kioku_sessions_scope_idx + ON kiroku.kioku_sessions (namespace, scope_kind, scope_ref); + CREATE INDEX kioku_sessions_namespace_started_idx + ON kiroku.kioku_sessions (namespace, started_at DESC); + CREATE INDEX kioku_sessions_namespace_focus_idx + ON kiroku.kioku_sessions (namespace, focus, started_at DESC); + CREATE INDEX kioku_sessions_awaiting_corr_idx + ON kiroku.kioku_sessions (namespace, awaiting_correlation_key) WHERE status = 'awaiting'; + CREATE INDEX kioku_consolidation_scope_idx + ON kiroku.kioku_consolidation_decisions (namespace, scope_kind, scope_ref); + CREATE INDEX kioku_scenes_scope_idx + ON kiroku.kioku_scenes (namespace, scope_kind, scope_ref); + """ + +-- | Eight rows: one per partitioned table, plus a turn whose session does not exist. +prePartitionRows :: Text +prePartitionRows = + """ + INSERT INTO kiroku.kioku_sessions (session_id, agent_id, focus, namespace, started_at) + VALUES ('s-1', 'agent', 'focus', 'ns', now()); + + INSERT INTO kiroku.kioku_turns (turn_id, session_id, turn_index, role, content, recorded_at) + VALUES ('t-1', 's-1', 1, 'user', 'hello', now()), + ('t-orphan', 's-vanished', 1, 'user', 'hello', now()); + + INSERT INTO kiroku.kioku_l1_watermarks (session_id, last_turn_index) VALUES ('s-1', 1); + + INSERT INTO kiroku.kioku_memories + (memory_id, agent_id, session_id, namespace, memory_type, content, created_at, updated_at) + VALUES ('m-1', 'agent', 's-1', 'ns', 'fact', 'content', now(), now()); + + INSERT INTO kiroku.kioku_consolidation_decisions + (decision_id, session_id, namespace, candidate_content, decision) + VALUES ('d-1', 's-1', 'ns', 'content', 'store'); + + INSERT INTO kiroku.kioku_scenes (scene_id, namespace, scene_key, title, body_md, source_hash) + VALUES ('kioku_scene:ns:default', 'ns', 'default', 'title', 'body', 'hash'); + + INSERT INTO kiroku.kioku_personas (persona_id, namespace, body_md, source_hash) + VALUES ('kioku_persona:ns', 'ns', 'body', 'hash'); + """ + +-- | The same scope-derived scene and persona ids, in a second space. Both inserts must +-- succeed; if either the primary key or the scope-uniqueness constraint had kept its old +-- shape, the second space would collide with the first. +twoSpacesOneScopeKey :: Text +twoSpacesOneScopeKey = + """ + INSERT INTO kiroku.kioku_scenes + (memory_space_id, scene_id, namespace, scene_key, title, body_md, source_hash) + VALUES ('space_other', 'kioku_scene:ns:default', 'ns', 'default', 'title', 'body', 'hash'); + + INSERT INTO kiroku.kioku_personas (memory_space_id, persona_id, namespace, body_md, source_hash) + VALUES ('space_other', 'kioku_persona:ns', 'ns', 'body', 'hash'); + """ + +-- | Every partitioned row, grouped by the space it landed in. +countMemorySpaces :: Statement () [(Text, Int64)] +countMemorySpaces = + preparable + """ + SELECT memory_space_id, count(*) + FROM ( + SELECT memory_space_id FROM kiroku.kioku_memories + UNION ALL SELECT memory_space_id FROM kiroku.kioku_sessions + UNION ALL SELECT memory_space_id FROM kiroku.kioku_turns + UNION ALL SELECT memory_space_id FROM kiroku.kioku_l1_watermarks + UNION ALL SELECT memory_space_id FROM kiroku.kioku_consolidation_decisions + UNION ALL SELECT memory_space_id FROM kiroku.kioku_scenes + UNION ALL SELECT memory_space_id FROM kiroku.kioku_personas + ) AS partitioned + GROUP BY memory_space_id + ORDER BY memory_space_id + """ + E.noParams + (D.rowList ((,) <$> D.column (D.nonNullable D.text) <*> D.column (D.nonNullable D.int8))) + +selectTurnSpace :: Statement Text (Maybe Text) +selectTurnSpace = + preparable + "SELECT memory_space_id FROM kiroku.kioku_turns WHERE turn_id = $1" + (E.param (E.nonNullable E.text)) + (D.rowMaybe (D.column (D.nonNullable D.text))) + +-- | @pg_indexes.indexname@ is a @name@, not a @text@; the cast is what lets hasql decode it. +selectKiokuIndexes :: Statement () [Text] +selectKiokuIndexes = + preparable + "SELECT indexname::text FROM pg_indexes WHERE schemaname = 'kiroku'" + E.noParams + (D.rowList (D.column (D.nonNullable D.text))) + +selectPrimaryKeyColumns :: Statement Text [Text] +selectPrimaryKeyColumns = + preparable + """ + SELECT a.attname::text + FROM pg_constraint c + JOIN pg_class t ON t.oid = c.conrelid + JOIN pg_namespace n ON n.oid = t.relnamespace + CROSS JOIN LATERAL unnest(c.conkey) WITH ORDINALITY AS k(attnum, ordinality) + JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = k.attnum + WHERE c.contype = 'p' AND n.nspname = 'kiroku' AND t.relname = $1 + ORDER BY k.ordinality + """ + (E.param (E.nonNullable E.text)) + (D.rowList (D.column (D.nonNullable D.text))) + +kiokuSchemaSnapshot :: Statement () Text +kiokuSchemaSnapshot = + preparable + """ + SELECT md5( + coalesce((SELECT string_agg(table_name || '.' || column_name || ':' || data_type || ':' || is_nullable, + E'\n' ORDER BY table_name, ordinal_position) + FROM information_schema.columns + WHERE table_schema = 'kiroku' AND table_name LIKE 'kioku\\_%'), '') + || coalesce((SELECT string_agg(indexname || ':' || indexdef, E'\n' ORDER BY indexname) + FROM pg_indexes + WHERE schemaname = 'kiroku' AND tablename LIKE 'kioku\\_%'), '')) + """ + E.noParams + (D.singleRow (D.column (D.nonNullable D.text))) + +-- * The projection schema relocation + +-- | The seven relations Kioku owns: the name migration 0011 left them under, and the name +-- migration 0012 puts them under. The schema changes with the name — @kiroku.kioku_memories@ +-- becomes @kioku.memories@ — because the schema now supplies the namespace the prefix used to. +relocatedRelations :: [(Text, Text)] +relocatedRelations = + [ ("kioku_memories", "memories"), + ("kioku_sessions", "sessions"), + ("kioku_turns", "turns"), + ("kioku_l1_watermarks", "l1_watermarks"), + ("kioku_consolidation_decisions", "consolidation_decisions"), + ("kioku_scenes", "scenes"), + ("kioku_personas", "personas") + ] + +relocationMigration :: FilePath +relocationMigration = "0012-relocate-projections-to-kioku-schema.sql" + +-- | What 'projectionLayout' must report once the relocation has run: seven ordinary tables in +-- @kioku@, and nothing at all left behind in @kiroku@ under the old prefix. +targetProjectionLayout :: [(Text, Text, Text)] +targetProjectionLayout = + [("kioku", target, "r") | target <- sort (snd <$> relocatedRelations)] + +-- | Every relation that could occupy one of the fourteen names this migration cares about. +-- +-- Indexes are excluded deliberately: they share PostgreSQL's relation namespace, so the +-- migration itself has to treat an index named @kioku.memories@ as a collision, but listing the +-- twenty-odd projection indexes here would drown the layout the assertions are about. The +-- relkind travels with each row so a view squatting on a target name is visible as a view. +projectionLayout :: Statement () [(Text, Text, Text)] +projectionLayout = + preparable + """ + SELECT n.nspname::text, c.relname::text, c.relkind::text + FROM pg_catalog.pg_class c + JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace + WHERE c.relkind IN ('r', 'v', 'm', 'f', 'p') + AND (n.nspname = 'kioku' + OR (n.nspname = 'kiroku' AND c.relname LIKE 'kioku\\_%')) + ORDER BY n.nspname COLLATE "C", c.relname COLLATE "C" + """ + E.noParams + ( D.rowList + ( (,,) + <$> D.column (D.nonNullable D.text) + <*> D.column (D.nonNullable D.text) + <*> D.column (D.nonNullable D.text) + ) + ) + +-- | A fresh database must land in the target layout with nothing left in the event store's +-- schema. This is the whole point of the migration stated as a catalog fact. +testRelocatedFreshLayout :: Assertion +testRelocatedFreshLayout = + withKiokuMigratedDatabase \connStr -> + query connStr projectionLayout >>= (@?= targetProjectionLayout) + +-- | The adoption case: a host that already runs Kiroku, and whose ledger therefore holds the +-- Kiroku component and nothing else. +-- +-- pg-migrate identities are component-qualified, so Kioku's composed plan recognises those rows +-- as its own first component, verifies their checksums, and skips them. Nothing about the event +-- store is re-run or rewritten; only the missing Keiro and Kioku components apply. A host that +-- also has its own migration component is a different case on purpose — the standalone runner +-- rejects a ledger row outside its plan, and that host must compose one complete plan instead. +testKirokuOnlyAdoption :: Assertion +testKirokuOnlyAdoption = + withBareDatabase \connStr -> do + let settings = Settings.connectionString connStr + kirokuOnly <- kirokuOnlyPlan + full <- either (fail . show) pure kiokuMigrationPlan + + kirokuReport <- runMigrationPlan defaultRunOptions settings kirokuOnly >>= either (assertFailure . show) pure + length (appliedNow kirokuReport) @?= 11 + ledgerBefore <- query connStr kirokuLedgerSnapshot + + adoption <- runMigrationPlan defaultRunOptions settings full >>= either (assertFailure . show) pure + let MigrationReport {results = adoptionResults} = adoption + length [() | MigrationResult {outcome = AlreadyApplied} <- toList adoptionResults] @?= 11 + length [() | MigrationResult {outcome = AppliedNow} <- toList adoptionResults] @?= 42 + + -- Verified and skipped, never re-executed: the stored rows keep their checksums and their + -- original application timestamps. + query connStr kirokuLedgerSnapshot >>= (@?= ledgerBefore) + + -- Kioku adopted the host's event store rather than standing up a second one beside it. + query connStr eventStoreRelationsExist >>= (@?= True) + query connStr projectionLayout >>= (@?= targetProjectionLayout) + + verification <- verifyMigrationPlan defaultRunOptions settings full >>= either (assertFailure . show) pure + let VerificationReport {issues = adoptionIssues, appliedMigrations, pendingMigrations, unknownMigrations} = verification + adoptionIssues @?= [] + length appliedMigrations @?= 53 + pendingMigrations @?= [] + unknownMigrations @?= [] + +-- | The Kiroku component on its own, which is what an existing Kiroku host's database has +-- applied before it adopts Kioku. +kirokuOnlyPlan :: IO MigrationPlan +kirokuOnlyPlan = do + component <- either (fail . show) pure KirokuMigrations.kirokuMigrations + either (fail . show) pure (migrationPlan (component :| [])) + +kirokuLedgerSnapshot :: Statement () Text +kirokuLedgerSnapshot = + preparable + """ + SELECT coalesce( + string_agg(migration || ':' || encode(checksum, 'hex') || ':' || status + || ':' || started_at::text || ':' || coalesce(finished_at::text, ''), + E'\n' ORDER BY position), + '') + FROM pgmigrate.migrations + WHERE component = 'kiroku' + """ + E.noParams + (D.singleRow (D.column (D.nonNullable D.text))) + +eventStoreRelationsExist :: Statement () Bool +eventStoreRelationsExist = + preparable + "SELECT to_regclass('kiroku.events') IS NOT NULL AND to_regclass('kiroku.streams') IS NOT NULL" + E.noParams + (D.singleRow (D.column (D.nonNullable D.bool))) + +-- | The data-bearing upgrade. @ALTER TABLE … SET SCHEMA@ and @… RENAME TO@ are catalog metadata +-- edits, so the table keeps its object identity — and that is the claim worth asserting, because +-- a relocation implemented as create-copy-drop would pass a row-count check and still silently +-- rebuild every index and drop every grant. +testRelocationPreservesTables :: Assertion +testRelocationPreservesTables = + withPreRelocationDatabase \conn -> do + relocation <- loadMigration relocationMigration + run conn (Session.script relocationSeedRows) + + oidsBefore <- run conn (Session.statement () oldLayoutTableOids) + rowsBefore <- run conn (Session.statement () oldLayoutRowCounts) + attachmentsBefore <- run conn (Session.statement () oldLayoutAttachments) + length oidsBefore @?= 7 + rowsBefore @?= [(target, 1) | target <- sort (snd <$> relocatedRelations)] + + run conn (Session.script relocation) + + run conn (Session.statement () projectionLayout) >>= (@?= targetProjectionLayout) + run conn (Session.statement () newLayoutTableOids) >>= (@?= oidsBefore) + run conn (Session.statement () newLayoutRowCounts) >>= (@?= rowsBefore) + run conn (Session.statement () newLayoutAttachments) >>= (@?= attachmentsBefore) + +-- | Rerunning the body against the finished layout is the second of the migration's only two +-- accepted states, and it must not so much as touch a catalog row. +testRelocationRerunIsNoOp :: Assertion +testRelocationRerunIsNoOp = + withKiokuMigratedDatabase \connStr -> + withConnection connStr \conn -> do + relocation <- loadMigration relocationMigration + oidsBefore <- run conn (Session.statement () newLayoutTableOids) + attachmentsBefore <- run conn (Session.statement () newLayoutAttachments) + + run conn (Session.script relocation) + + run conn (Session.statement () projectionLayout) >>= (@?= targetProjectionLayout) + run conn (Session.statement () newLayoutTableOids) >>= (@?= oidsBefore) + run conn (Session.statement () newLayoutAttachments) >>= (@?= attachmentsBefore) + +-- | Every layout other than wholly-old and wholly-new is refused before a single table moves. +-- +-- The migration is transactional, so \"refused\" has to mean the catalog is bit-for-bit what it +-- was before the attempt — not merely that an error was reported after a partial move. +assertRelocationAborts :: Text -> Assertion +assertRelocationAborts damage = + withPreRelocationDatabase \conn -> do + relocation <- loadMigration relocationMigration + run conn (Session.script damage) + before <- run conn (Session.statement () projectionLayout) + + result <- Connection.use conn (Session.script relocation) + case result of + Right () -> assertFailure "the relocation accepted a layout that is neither wholly old nor wholly new" + Left err -> + assertBool + ("expected a layout refusal, got: " <> show err) + ("refusing to relocate" `Text.isInfixOf` Text.pack (show err)) + + run conn (Session.statement () projectionLayout) >>= (@?= before) + +moveOneTableForward :: Text +moveOneTableForward = + """ + ALTER TABLE kiroku.kioku_turns SET SCHEMA kioku; + ALTER TABLE kioku.kioku_turns RENAME TO turns; + """ + +dropOneSourceTable :: Text +dropOneSourceTable = "DROP TABLE kiroku.kioku_personas" + +occupyTargetWithTable :: Text +occupyTargetWithTable = "CREATE TABLE kioku.memories (placeholder text)" + +occupyTargetWithView :: Text +occupyTargetWithView = "CREATE VIEW kioku.scenes AS SELECT 1 AS placeholder" + +-- | A fully migrated database put back into the layout migration 0011 left behind: the seven +-- tables in @kiroku@ under their @kioku_@ names. The empty @kioku@ schema stays, which is also +-- the state a host is in if it created the schema ahead of the upgrade. +withPreRelocationDatabase :: (Connection.Connection -> IO a) -> IO a +withPreRelocationDatabase use = + withKiokuMigratedDatabase \connStr -> + withConnection connStr \conn -> do + run conn (Session.script undoSchemaRelocation) + use conn + +undoSchemaRelocation :: Text +undoSchemaRelocation = + """ + ALTER TABLE kioku.memories RENAME TO kioku_memories; + ALTER TABLE kioku.kioku_memories SET SCHEMA kiroku; + ALTER TABLE kioku.sessions RENAME TO kioku_sessions; + ALTER TABLE kioku.kioku_sessions SET SCHEMA kiroku; + ALTER TABLE kioku.turns RENAME TO kioku_turns; + ALTER TABLE kioku.kioku_turns SET SCHEMA kiroku; + ALTER TABLE kioku.l1_watermarks RENAME TO kioku_l1_watermarks; + ALTER TABLE kioku.kioku_l1_watermarks SET SCHEMA kiroku; + ALTER TABLE kioku.consolidation_decisions RENAME TO kioku_consolidation_decisions; + ALTER TABLE kioku.kioku_consolidation_decisions SET SCHEMA kiroku; + ALTER TABLE kioku.scenes RENAME TO kioku_scenes; + ALTER TABLE kioku.kioku_scenes SET SCHEMA kiroku; + ALTER TABLE kioku.personas RENAME TO kioku_personas; + ALTER TABLE kioku.kioku_personas SET SCHEMA kiroku; + """ + +-- | One row in each of the seven relations, in the pre-relocation layout. Each carries a memory +-- space, because this database is already past migration 0011. +relocationSeedRows :: Text +relocationSeedRows = + """ + INSERT INTO kiroku.kioku_sessions + (memory_space_id, session_id, agent_id, focus, namespace, started_at) + VALUES ('space_a', 's-1', 'agent', 'focus', 'ns', now()); + + INSERT INTO kiroku.kioku_turns + (memory_space_id, turn_id, session_id, turn_index, role, content, recorded_at) + VALUES ('space_a', 't-1', 's-1', 1, 'user', 'hello', now()); + + INSERT INTO kiroku.kioku_l1_watermarks (memory_space_id, session_id, last_turn_index) + VALUES ('space_a', 's-1', 1); + + INSERT INTO kiroku.kioku_memories + (memory_space_id, memory_id, agent_id, session_id, namespace, memory_type, + content, created_at, updated_at) + VALUES ('space_a', 'm-1', 'agent', 's-1', 'ns', 'fact', 'content', now(), now()); + + INSERT INTO kiroku.kioku_consolidation_decisions + (memory_space_id, decision_id, session_id, namespace, candidate_content, decision) + VALUES ('space_a', 'd-1', 's-1', 'ns', 'content', 'store'); + + INSERT INTO kiroku.kioku_scenes + (memory_space_id, scene_id, namespace, scene_key, title, body_md, source_hash) + VALUES ('space_a', 'kioku_scene:ns:default', 'ns', 'default', 'title', 'body', 'hash'); + + INSERT INTO kiroku.kioku_personas + (memory_space_id, persona_id, namespace, body_md, source_hash) + VALUES ('space_a', 'kioku_persona:ns', 'ns', 'body', 'hash'); + """ + +oldLayoutTableOids :: Statement () [(Text, Int64)] +oldLayoutTableOids = + preparable + """ + SELECT substr(c.relname, 7)::text, c.oid::int8 + FROM pg_catalog.pg_class c + JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace + WHERE n.nspname = 'kiroku' AND c.relkind = 'r' AND c.relname LIKE 'kioku\\_%' + ORDER BY substr(c.relname, 7) COLLATE "C" + """ + E.noParams + (D.rowList ((,) <$> D.column (D.nonNullable D.text) <*> D.column (D.nonNullable D.int8))) + +newLayoutTableOids :: Statement () [(Text, Int64)] +newLayoutTableOids = + preparable + """ + SELECT c.relname::text, c.oid::int8 + FROM pg_catalog.pg_class c + JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace + WHERE n.nspname = 'kioku' AND c.relkind = 'r' + ORDER BY c.relname COLLATE "C" + """ + E.noParams + (D.rowList ((,) <$> D.column (D.nonNullable D.text) <*> D.column (D.nonNullable D.int8))) + +oldLayoutRowCounts :: Statement () [(Text, Int64)] +oldLayoutRowCounts = + preparable + """ + SELECT relation, row_count + FROM ( + SELECT 'consolidation_decisions'::text AS relation, count(*) AS row_count FROM kiroku.kioku_consolidation_decisions + UNION ALL SELECT 'l1_watermarks'::text, count(*) FROM kiroku.kioku_l1_watermarks + UNION ALL SELECT 'memories'::text, count(*) FROM kiroku.kioku_memories + UNION ALL SELECT 'personas'::text, count(*) FROM kiroku.kioku_personas + UNION ALL SELECT 'scenes'::text, count(*) FROM kiroku.kioku_scenes + UNION ALL SELECT 'sessions'::text, count(*) FROM kiroku.kioku_sessions + UNION ALL SELECT 'turns'::text, count(*) FROM kiroku.kioku_turns + ) AS counts + ORDER BY relation COLLATE "C" + """ + E.noParams + (D.rowList ((,) <$> D.column (D.nonNullable D.text) <*> D.column (D.nonNullable D.int8))) + +newLayoutRowCounts :: Statement () [(Text, Int64)] +newLayoutRowCounts = + preparable + """ + SELECT relation, row_count + FROM ( + SELECT 'consolidation_decisions'::text AS relation, count(*) AS row_count FROM kioku.consolidation_decisions + UNION ALL SELECT 'l1_watermarks'::text, count(*) FROM kioku.l1_watermarks + UNION ALL SELECT 'memories'::text, count(*) FROM kioku.memories + UNION ALL SELECT 'personas'::text, count(*) FROM kioku.personas + UNION ALL SELECT 'scenes'::text, count(*) FROM kioku.scenes + UNION ALL SELECT 'sessions'::text, count(*) FROM kioku.sessions + UNION ALL SELECT 'turns'::text, count(*) FROM kioku.turns + ) AS counts + ORDER BY relation COLLATE "C" + """ + E.noParams + (D.rowList ((,) <$> D.column (D.nonNullable D.text) <*> D.column (D.nonNullable D.int8))) + +-- | Everything PostgreSQL is supposed to carry along with a table when it changes schema and +-- name: its owner, its table grants, the names of its indexes, and the full text of its +-- constraints. None of Kioku's seven constraints names a schema, so their definitions are +-- expected to be identical on both sides of the move rather than merely similar. +tableAttachmentsSql :: Text -> Text -> Text +tableAttachmentsSql schema shortName = + "SELECT " + <> shortName + <> "::text,\n\ + \ (pg_catalog.pg_get_userbyid(c.relowner)\n\ + \ || '|' || coalesce(c.relacl::text, '')\n\ + \ || '|' || coalesce((SELECT string_agg(i.relname, ',' ORDER BY i.relname COLLATE \"C\")\n\ + \ FROM pg_catalog.pg_index x\n\ + \ JOIN pg_catalog.pg_class i ON i.oid = x.indexrelid\n\ + \ WHERE x.indrelid = c.oid), '')\n\ + \ || '|' || coalesce((SELECT string_agg(con.conname || '=' || pg_catalog.pg_get_constraintdef(con.oid),\n\ + \ ',' ORDER BY con.conname COLLATE \"C\")\n\ + \ FROM pg_catalog.pg_constraint con\n\ + \ WHERE con.conrelid = c.oid), ''))::text\n\ + \FROM pg_catalog.pg_class c\n\ + \JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace\n\ + \WHERE n.nspname = '" + <> schema + <> "' AND c.relkind = 'r'" + <> (if schema == "kiroku" then " AND c.relname LIKE 'kioku\\_%'" else "") + <> "\nORDER BY " + <> shortName + <> " COLLATE \"C\"" + +oldLayoutAttachments :: Statement () [(Text, Text)] +oldLayoutAttachments = + preparable + (tableAttachmentsSql "kiroku" "substr(c.relname, 7)") + E.noParams + (D.rowList ((,) <$> D.column (D.nonNullable D.text) <*> D.column (D.nonNullable D.text))) + +newLayoutAttachments :: Statement () [(Text, Text)] +newLayoutAttachments = + preparable + (tableAttachmentsSql "kioku" "c.relname") + E.noParams + (D.rowList ((,) <$> D.column (D.nonNullable D.text) <*> D.column (D.nonNullable D.text))) + -- * The downstream Codd cutover rehearsal -- | Exercise the operator runbook against the exact 30 historical migration @@ -260,18 +913,19 @@ let MigrationReport {results = migratedResults} = migrated appliedNow migrated @?= expectedForwardMigrationIds length [() | MigrationResult {outcome = AlreadyApplied} <- toList migratedResults] @?= 30 - query connStr forwardMigrationEffectCountStatement >>= (@?= 5) + query connStr forwardMigrationEffectCountStatement >>= (@?= 6) + query connStr projectionLayout >>= (@?= targetProjectionLayout) verification <- verifyMigrationPlan defaultRunOptions settings plan >>= either (assertFailure . show) pure let VerificationReport {issues = verificationIssues, appliedMigrations, pendingMigrations, unknownMigrations} = verification verificationIssues @?= [] - length appliedMigrations @?= 38 + length appliedMigrations @?= 53 pendingMigrations @?= [] unknownMigrations @?= [] repeated <- runMigrationPlan defaultRunOptions settings plan >>= either (assertFailure . show) pure let MigrationReport {results = repeatedResults} = repeated - length [() | MigrationResult {outcome = AlreadyApplied} <- toList repeatedResults] @?= 38 + length [() | MigrationResult {outcome = AlreadyApplied} <- toList repeatedResults] @?= 53 length [() | MigrationResult {outcome = AppliedNow} <- toList repeatedResults] @?= 0 fixtureMigrationNames :: Text -> [FilePath] @@ -339,12 +993,27 @@ expectRight <$> [ migrationId "kiroku" "0007-stream-truncate-before", migrationId "kiroku" "0008-schema-management-comment", + migrationId "kiroku" "0009", + migrationId "kiroku" "0010", + migrationId "kiroku" "0011", migrationId "keiro" "0015-keiro-outbox-claim-order-index", migrationId "keiro" "0016-keiro-inbox-drop-received-idx", migrationId "keiro" "0017-schema-management-comment", migrationId "keiro" "0018", migrationId "keiro" "0019-keiro-snapshots-state-shape-hash", - migrationId "keiro" "0020-keiro-workflow-children-failure-reason" + migrationId "keiro" "0020-keiro-workflow-children-failure-reason", + migrationId "keiro" "0021-keiro-workflows-exact-discovery", + migrationId "keiro" "0022", + migrationId "keiro" "0023", + migrationId "keiro" "0024", + migrationId "keiro" "0025", + migrationId "keiro" "0026", + migrationId "keiro" "0027", + migrationId "keiro" "0028", + migrationId "keiro" "0029", + migrationId "keiro" "0030", + migrationId "kioku" "0011-kioku-memory-space-partition", + migrationId "kioku" "0012-relocate-projections-to-kioku-schema" ] expectRight :: (Show error) => Either error value -> value @@ -389,11 +1058,13 @@ (EXISTS (SELECT 1 FROM information_schema.columns WHERE table_schema = 'kiroku' AND table_name = 'streams' AND column_name = 'truncate_before'))::int + (coalesce(obj_description(to_regnamespace('kiroku'), 'pg_namespace'), '') = - 'Managed by pg-migrate component kiroku through 0008-schema-management-comment')::int + 'Managed by pg-migrate component kiroku through 0011')::int + (to_regclass('keiro.keiro_outbox_claim_order_idx') IS NOT NULL)::int + (to_regclass('keiro.keiro_inbox_received_idx') IS NULL)::int + (coalesce(obj_description(to_regnamespace('keiro'), 'pg_namespace'), '') = 'Managed by pg-migrate component keiro through 0017-schema-management-comment')::int + + (to_regclass('kioku.memories') IS NOT NULL + AND to_regclass('kiroku.kioku_memories') IS NULL)::int )::bigint """ E.noParams