diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,59 @@ [Keep a Changelog](https://keepachangelog.com/), and the package follows the [Haskell Package Versioning Policy](https://pvp.haskell.org/). +## Unreleased + +## 0.12.0.0 — 2026-08-14 + +### Breaking Changes + +- Requires `kiroku-store >=0.7 && <0.8` and + `kiroku-store-migrations ^>=0.3.2.0`. Kiroku migration `0010` supplies the renewable + history-retention lease required by online replay. + +### New Features + +- Migration `0030.sql` makes the external-read guard lock the lifecycle row before + re-reading mutable contract metadata under a matching shared lock. A reader queued + behind promotion therefore receives retryable `KR001` when its statement snapshot + crosses the serving epoch; its next statement validates and reads the rebound + generation from one committed authority. +- Migration `0029.sql` adds bounded, run-scoped async-dedup staging, persisted + admission/preparation evidence, and private database-clock deadline helpers for + schema-versioned writer-fence and cumulative promotion-lock attempts. + +- Migration `0028.sql` permits an implementation-backed compatibility wrapper + to retain a zero-argument public signature while a breaking projection shape + is promoted. Keyed contracts still require a named, versioned private + implementation; all-row contracts remain Keiro-backed. +- Migration `0027.sql` adds private external-read contract and managed-object + registries plus the fixed `keiro_read.guard_external_read_v1` security-definer + guard. The guard takes a shared lifecycle lock, consumes persisted + `reads_allowed` and serving compatibility, revokes `PUBLIC`, and raises stable + `KR001`, `KR002`, or `KR003` SQLSTATEs. +- Migration `0026.sql` publishes the frozen, owner-rights + `keiro_read.projection_group_status_v1` relation and private derived cursor authority. + The PostgreSQL 18 schema verifier now fingerprints ordered public-view column + signatures, and migration tests prove narrow-reader privileges and v1 null semantics. +- Migration `0025.sql` persists projection revisions, target generations, serving + epochs and availability, versioned run/cutover policy, relation/schema evidence, + promotion-object mappings, replay-source progress, and Kiroku retention handles. + +- Migration `0024.sql` renames projection rebuild group + `catalog_fingerprint` to `slice_fingerprint` and records + `group_slice_fingerprint` on every rebuild run. This is a clean pre-0.12 + cutover: complete or abandon active catalog rebuilds before upgrading; stale + group fingerprints are recovered through the library adoption API. + +- Migration `0021-keiro-workflows-exact-discovery.sql`. Widens + `keiro_workflows_active_idx` to `(status, wake_after)` so both arms of keiro's + exact workflow-discovery predicate are index-served, and returns every + pre-existing `suspended` instance to `running` once so it is re-examined + through the new suspend/wake arbitration. Forward-only and idempotent. + +### Other Changes + +- The source distribution now includes the BSD-3-Clause license file. ## [0.11.0.0] - 2026-08-05 No changes this release. Released with the package set for the `keiro-dsl` diff --git a/LICENSE b/LICENSE new file mode 100644 --- /dev/null +++ b/LICENSE @@ -0,0 +1,28 @@ +BSD 3-Clause License + +Copyright (c) 2026, Nadeem Bitar + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its contributors + may be used to endorse or promote products derived from this software without + specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS +OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT INCLUDING +NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN +IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # Keiro database migrations `keiro-migrations` exports a native `pg-migrate` component named `keiro`. The -component owns sixteen embedded SQL migrations and declares one dependency, +component owns thirty embedded SQL migrations and declares one dependency, `kiroku`. Applications compose Kiroku first and Keiro second; Keiro never embeds or copies Kiroku's SQL. @@ -53,7 +53,7 @@ ## Importing the shared Codd ledger `Keiro.Migrations.History.Codd` combines Kiroku's seven mappings with Keiro's -sixteen mappings. `frameworkCoddSourceConfig` selects both histories from the +sixteen legacy mappings. `frameworkCoddSourceConfig` selects both histories from the shared `codd.sql_migrations` or legacy `codd_schema.sql_migrations` ledger, checks the original SHA-256 manifests and payload bytes, and produces one atomic history import: diff --git a/expected-schema/native/keiro-v18.txt b/expected-schema/native/keiro-v18.txt --- a/expected-schema/native/keiro-v18.txt +++ b/expected-schema/native/keiro-v18.txt @@ -18,6 +18,27 @@ column keiro_dead_letters.source_event_id uuid not null column keiro_dead_letters.source_global_position bigint not null column keiro_dead_letters.target_stream_name text not null +column keiro_external_read_contracts.argument_names text[] not null default ARRAY[]::text[] +column keiro_external_read_contracts.argument_types text[] not null default ARRAY[]::text[] +column keiro_external_read_contracts.compatible_revision_ids text[] not null +column keiro_external_read_contracts.contract_id text not null +column keiro_external_read_contracts.contract_kind text not null +column keiro_external_read_contracts.contract_version integer not null +column keiro_external_read_contracts.created_at timestamp with time zone not null default now() +column keiro_external_read_contracts.definition_hash text not null +column keiro_external_read_contracts.group_id text not null +column keiro_external_read_contracts.immutable_signature_hash text not null +column keiro_external_read_contracts.private_implementation text +column keiro_external_read_contracts.private_implementation_version integer +column keiro_external_read_contracts.public_function_name text not null +column keiro_external_read_contracts.query_model_id text not null +column keiro_external_read_contracts.result_shape_hash text not null +column keiro_external_read_contracts.result_type text not null +column keiro_external_read_contracts.retired_at timestamp with time zone +column keiro_external_read_contracts.serving_shape_hash text +column keiro_external_read_contracts.state text not null +column keiro_external_read_contracts.surface_generation integer not null +column keiro_external_read_contracts.updated_at timestamp with time zone not null default now() column keiro_inbox.attempt_count bigint not null default 0 column keiro_inbox.attributes jsonb column keiro_inbox.causation_id uuid @@ -48,6 +69,19 @@ column keiro_inbox.status text not null default 'processing'::text column keiro_inbox.traceparent text column keiro_inbox.tracestate text +column keiro_managed_read_objects.contract_id text +column keiro_managed_read_objects.contract_version integer +column keiro_managed_read_objects.created_at timestamp with time zone not null default now() +column keiro_managed_read_objects.definition_hash text not null +column keiro_managed_read_objects.managed_by text not null +column keiro_managed_read_objects.object_kind text not null +column keiro_managed_read_objects.object_name text not null +column keiro_managed_read_objects.object_schema text not null +column keiro_managed_read_objects.object_signature text not null default ''::text +column keiro_managed_read_objects.retired_at timestamp with time zone +column keiro_managed_read_objects.state text not null +column keiro_managed_read_objects.surface_generation integer not null +column keiro_managed_read_objects.updated_at timestamp with time zone not null default now() column keiro_outbox.attempt_count bigint not null default 0 column keiro_outbox.attributes jsonb column keiro_outbox.causation_id uuid @@ -80,8 +114,123 @@ column keiro_projection_dedup.applied_at timestamp with time zone not null default now() column keiro_projection_dedup.event_id uuid not null column keiro_projection_dedup.projection_name text not null +column keiro_projection_group_cursors.group_id text not null +column keiro_projection_group_cursors.position_basis text not null +column keiro_projection_group_cursors.subscription_names text[] not null default ARRAY[]::text[] +column keiro_projection_group_cursors.updated_at timestamp with time zone not null default now() +column keiro_projection_rebuild_adapters.adapter_order integer not null +column keiro_projection_rebuild_adapters.apply_count bigint not null default 0 +column keiro_projection_rebuild_adapters.completed_through bigint +column keiro_projection_rebuild_adapters.evaluation_count bigint not null default 0 +column keiro_projection_rebuild_adapters.projection_id text not null +column keiro_projection_rebuild_adapters.run_id text not null +column keiro_projection_rebuild_adapters.source_id text not null +column keiro_projection_rebuild_adapters.updated_at timestamp with time zone not null default now() +column keiro_projection_rebuild_dedup_stage.event_id uuid not null +column keiro_projection_rebuild_dedup_stage.global_position bigint not null +column keiro_projection_rebuild_dedup_stage.projection_name text not null +column keiro_projection_rebuild_dedup_stage.run_id text not null +column keiro_projection_rebuild_dedup_stage.subscription_name text not null +column keiro_projection_rebuild_groups.active_run_id text +column keiro_projection_rebuild_groups.completed_at timestamp with time zone +column keiro_projection_rebuild_groups.created_at timestamp with time zone not null default now() +column keiro_projection_rebuild_groups.failed_at timestamp with time zone +column keiro_projection_rebuild_groups.failure_code text +column keiro_projection_rebuild_groups.failure_detail text +column keiro_projection_rebuild_groups.group_id text not null +column keiro_projection_rebuild_groups.reads_allowed boolean not null default true +column keiro_projection_rebuild_groups.request_reason text +column keiro_projection_rebuild_groups.requested_by text +column keiro_projection_rebuild_groups.serving_epoch bigint not null default 0 +column keiro_projection_rebuild_groups.serving_revision_id text +column keiro_projection_rebuild_groups.slice_fingerprint text not null +column keiro_projection_rebuild_groups.started_at timestamp with time zone +column keiro_projection_rebuild_groups.status text not null +column keiro_projection_rebuild_groups.updated_at timestamp with time zone not null default now() +column keiro_projection_rebuild_groups.writes_allowed boolean not null default true +column keiro_projection_rebuild_promotion_objects.canonical_name text not null +column keiro_projection_rebuild_promotion_objects.generation_name text not null +column keiro_projection_rebuild_promotion_objects.object_kind text not null +column keiro_projection_rebuild_promotion_objects.object_order integer not null +column keiro_projection_rebuild_promotion_objects.run_id text not null +column keiro_projection_rebuild_promotion_objects.target_id text not null +column keiro_projection_rebuild_run_targets.candidate_generation_id uuid not null +column keiro_projection_rebuild_run_targets.run_id text not null +column keiro_projection_rebuild_run_targets.target_id text not null +column keiro_projection_rebuild_run_targets.target_mode text not null +column keiro_projection_rebuild_runs.abandoned_at timestamp with time zone +column keiro_projection_rebuild_runs.candidate_revision_id text +column keiro_projection_rebuild_runs.captured_head bigint not null +column keiro_projection_rebuild_runs.catalog_fingerprint text not null +column keiro_projection_rebuild_runs.contract_fingerprint text not null +column keiro_projection_rebuild_runs.cutover_lock_timeout_ms bigint +column keiro_projection_rebuild_runs.cutover_threshold bigint +column keiro_projection_rebuild_runs.dedup_provisional_head bigint +column keiro_projection_rebuild_runs.failed_at timestamp with time zone +column keiro_projection_rebuild_runs.failure_code text +column keiro_projection_rebuild_runs.failure_detail text +column keiro_projection_rebuild_runs.failure_position bigint +column keiro_projection_rebuild_runs.failure_projection_id text +column keiro_projection_rebuild_runs.failure_source_id text +column keiro_projection_rebuild_runs.group_id text not null +column keiro_projection_rebuild_runs.group_slice_fingerprint text not null +column keiro_projection_rebuild_runs.history_retention_expires_at timestamp with time zone +column keiro_projection_rebuild_runs.history_retention_lease_id uuid +column keiro_projection_rebuild_runs.history_retention_lease_owner text +column keiro_projection_rebuild_runs.history_retention_protected_through bigint +column keiro_projection_rebuild_runs.history_retention_released_at timestamp with time zone +column keiro_projection_rebuild_runs.history_retention_renewed_at timestamp with time zone +column keiro_projection_rebuild_runs.page_size integer not null +column keiro_projection_rebuild_runs.promoted_at timestamp with time zone +column keiro_projection_rebuild_runs.promotion_dedup_limit bigint +column keiro_projection_rebuild_runs.promotion_prepared_at timestamp with time zone +column keiro_projection_rebuild_runs.rebuild_mode text not null default 'offline'::text +column keiro_projection_rebuild_runs.run_id text not null +column keiro_projection_rebuild_runs.runner_format text not null +column keiro_projection_rebuild_runs.started_at timestamp with time zone not null default now() +column keiro_projection_rebuild_runs.status text not null default 'running'::text +column keiro_projection_rebuild_runs.updated_at timestamp with time zone not null default now() +column keiro_projection_rebuild_runs.verified_at timestamp with time zone +column keiro_projection_rebuild_sources.category text +column keiro_projection_rebuild_sources.cursor_position bigint not null default 0 +column keiro_projection_rebuild_sources.event_count bigint not null default 0 +column keiro_projection_rebuild_sources.exhausted_through bigint +column keiro_projection_rebuild_sources.run_id text not null +column keiro_projection_rebuild_sources.source_id text not null +column keiro_projection_rebuild_sources.source_scope text not null +column keiro_projection_rebuild_sources.target_position bigint not null +column keiro_projection_rebuild_sources.updated_at timestamp with time zone not null default now() +column keiro_projection_rebuild_verifications.completed_at timestamp with time zone +column keiro_projection_rebuild_verifications.detail text +column keiro_projection_rebuild_verifications.run_id text not null +column keiro_projection_rebuild_verifications.status text not null default 'pending'::text +column keiro_projection_rebuild_verifications.verification_id text not null +column keiro_projection_rebuild_verifications.verification_version text not null +column keiro_projection_revisions.group_id text not null +column keiro_projection_revisions.group_slice_fingerprint text not null +column keiro_projection_revisions.registered_at timestamp with time zone not null default now() +column keiro_projection_revisions.revision_id text not null +column keiro_projection_revisions.updated_at timestamp with time zone not null default now() +column keiro_projection_target_generations.created_at timestamp with time zone not null default now() +column keiro_projection_target_generations.created_by_run_id text +column keiro_projection_target_generations.dropped_at timestamp with time zone +column keiro_projection_target_generations.expected_shape_id text not null +column keiro_projection_target_generations.generation_id uuid not null +column keiro_projection_target_generations.group_id text not null +column keiro_projection_target_generations.lifecycle text not null +column keiro_projection_target_generations.observed_catalog_snapshot text not null +column keiro_projection_target_generations.observed_shape_fingerprint text not null +column keiro_projection_target_generations.relation_name text not null +column keiro_projection_target_generations.relation_oid bigint not null +column keiro_projection_target_generations.retired_at timestamp with time zone +column keiro_projection_target_generations.revision_id text not null +column keiro_projection_target_generations.schema_name text not null +column keiro_projection_target_generations.schema_version text not null +column keiro_projection_target_generations.served_at timestamp with time zone +column keiro_projection_target_generations.target_id text not null column keiro_read_models.last_built_at timestamp with time zone column keiro_read_models.name text not null +column keiro_read_models.rebuild_group_id text not null column keiro_read_models.shape_hash text not null column keiro_read_models.status text not null column keiro_read_models.updated_at timestamp with time zone not null default now() @@ -166,6 +315,31 @@ constraint keiro_dead_letters.keiro_dead_letters_source_event_id_not_null NOT NULL source_event_id constraint keiro_dead_letters.keiro_dead_letters_source_global_position_not_null NOT NULL source_global_position constraint keiro_dead_letters.keiro_dead_letters_target_stream_name_not_null NOT NULL target_stream_name +constraint keiro_external_read_contracts.keiro_external_read_contracts_argument_names_not_null NOT NULL argument_names +constraint keiro_external_read_contracts.keiro_external_read_contracts_argument_types_not_null NOT NULL argument_types +constraint keiro_external_read_contracts.keiro_external_read_contracts_compatible_revision_ids_not_null NOT NULL compatible_revision_ids +constraint keiro_external_read_contracts.keiro_external_read_contracts_contract_id_not_null NOT NULL contract_id +constraint keiro_external_read_contracts.keiro_external_read_contracts_contract_kind_not_null NOT NULL contract_kind +constraint keiro_external_read_contracts.keiro_external_read_contracts_contract_version_not_null NOT NULL contract_version +constraint keiro_external_read_contracts.keiro_external_read_contracts_created_at_not_null NOT NULL created_at +constraint keiro_external_read_contracts.keiro_external_read_contracts_definition_hash_not_null NOT NULL definition_hash +constraint keiro_external_read_contracts.keiro_external_read_contracts_group_id_fkey FOREIGN KEY (group_id) REFERENCES keiro.keiro_projection_rebuild_groups(group_id) +constraint keiro_external_read_contracts.keiro_external_read_contracts_group_id_not_null NOT NULL group_id +constraint keiro_external_read_contracts.keiro_external_read_contracts_identity_chk CHECK (((length(contract_id) > 0) AND (contract_version > 0) AND (length(query_model_id) > 0) AND (length(group_id) > 0) AND (length(public_function_name) > 0) AND (length(result_type) > 0) AND (length(result_shape_hash) > 0) AND (length(immutable_signature_hash) > 0) AND (length(definition_hash) > 0) AND (surface_generation > 0) AND (cardinality(compatible_revision_ids) > 0) AND (array_position(compatible_revision_ids, NULL::text) IS NULL) AND (array_position(argument_names, NULL::text) IS NULL) AND (array_position(argument_types, NULL::text) IS NULL) AND (cardinality(argument_names) = cardinality(argument_types)))) +constraint keiro_external_read_contracts.keiro_external_read_contracts_immutable_signature_hash_not_null NOT NULL immutable_signature_hash +constraint keiro_external_read_contracts.keiro_external_read_contracts_implementation_chk CHECK ((((contract_kind = 'all-rows'::text) AND (cardinality(argument_names) = 0) AND (private_implementation IS NULL) AND (private_implementation_version IS NULL)) OR ((contract_kind = 'keyed'::text) AND (private_implementation IS NOT NULL) AND (length(private_implementation) > 0) AND (private_implementation_version > 0)))) +constraint keiro_external_read_contracts.keiro_external_read_contracts_kind_chk CHECK ((contract_kind = ANY (ARRAY['all-rows'::text, 'keyed'::text]))) +constraint keiro_external_read_contracts.keiro_external_read_contracts_pkey PRIMARY KEY (contract_id, contract_version) +constraint keiro_external_read_contracts.keiro_external_read_contracts_public_function_name_argument_key UNIQUE (public_function_name, argument_types) +constraint keiro_external_read_contracts.keiro_external_read_contracts_public_function_name_not_null NOT NULL public_function_name +constraint keiro_external_read_contracts.keiro_external_read_contracts_query_model_id_not_null NOT NULL query_model_id +constraint keiro_external_read_contracts.keiro_external_read_contracts_result_shape_hash_not_null NOT NULL result_shape_hash +constraint keiro_external_read_contracts.keiro_external_read_contracts_result_type_not_null NOT NULL result_type +constraint keiro_external_read_contracts.keiro_external_read_contracts_retired_chk CHECK ((((state = 'retired'::text) AND (retired_at IS NOT NULL)) OR ((state <> 'retired'::text) AND (retired_at IS NULL)))) +constraint keiro_external_read_contracts.keiro_external_read_contracts_state_chk CHECK ((state = ANY (ARRAY['candidate'::text, 'active'::text, 'pending-retirement'::text, 'retired'::text]))) +constraint keiro_external_read_contracts.keiro_external_read_contracts_state_not_null NOT NULL state +constraint keiro_external_read_contracts.keiro_external_read_contracts_surface_generation_not_null NOT NULL surface_generation +constraint keiro_external_read_contracts.keiro_external_read_contracts_updated_at_not_null NOT NULL updated_at constraint keiro_inbox.keiro_inbox_attempt_count_not_null NOT NULL attempt_count constraint keiro_inbox.keiro_inbox_content_type_not_null NOT NULL content_type constraint keiro_inbox.keiro_inbox_dedupe_key_not_null NOT NULL dedupe_key @@ -174,6 +348,23 @@ constraint keiro_inbox.keiro_inbox_received_at_not_null NOT NULL received_at constraint keiro_inbox.keiro_inbox_source_not_null NOT NULL source constraint keiro_inbox.keiro_inbox_status_not_null NOT NULL status +constraint keiro_managed_read_objects.keiro_managed_read_objects_contract_id_contract_version_fkey FOREIGN KEY (contract_id, contract_version) REFERENCES keiro.keiro_external_read_contracts(contract_id, contract_version) +constraint keiro_managed_read_objects.keiro_managed_read_objects_created_at_not_null NOT NULL created_at +constraint keiro_managed_read_objects.keiro_managed_read_objects_definition_hash_not_null NOT NULL definition_hash +constraint keiro_managed_read_objects.keiro_managed_read_objects_identity_chk CHECK (((length(object_schema) > 0) AND (length(object_name) > 0) AND (length(object_kind) > 0) AND (length(managed_by) > 0) AND (length(definition_hash) > 0) AND (surface_generation > 0) AND (((contract_id IS NULL) AND (contract_version IS NULL)) OR ((contract_id IS NOT NULL) AND (contract_version IS NOT NULL))))) +constraint keiro_managed_read_objects.keiro_managed_read_objects_kind_chk CHECK ((object_kind = ANY (ARRAY['guard-function'::text, 'binding-view'::text, 'wrapper-function'::text, 'contract-type'::text]))) +constraint keiro_managed_read_objects.keiro_managed_read_objects_managed_by_not_null NOT NULL managed_by +constraint keiro_managed_read_objects.keiro_managed_read_objects_object_kind_not_null NOT NULL object_kind +constraint keiro_managed_read_objects.keiro_managed_read_objects_object_name_not_null NOT NULL object_name +constraint keiro_managed_read_objects.keiro_managed_read_objects_object_schema_not_null NOT NULL object_schema +constraint keiro_managed_read_objects.keiro_managed_read_objects_object_signature_not_null NOT NULL object_signature +constraint keiro_managed_read_objects.keiro_managed_read_objects_owner_chk CHECK ((managed_by = ANY (ARRAY['keiro'::text, 'consumer'::text]))) +constraint keiro_managed_read_objects.keiro_managed_read_objects_pkey PRIMARY KEY (object_schema, object_name, object_kind, object_signature) +constraint keiro_managed_read_objects.keiro_managed_read_objects_retired_chk CHECK ((((state = 'retired'::text) AND (retired_at IS NOT NULL)) OR ((state <> 'retired'::text) AND (retired_at IS NULL)))) +constraint keiro_managed_read_objects.keiro_managed_read_objects_state_chk CHECK ((state = ANY (ARRAY['active'::text, 'pending-retirement'::text, 'retired'::text]))) +constraint keiro_managed_read_objects.keiro_managed_read_objects_state_not_null NOT NULL state +constraint keiro_managed_read_objects.keiro_managed_read_objects_surface_generation_not_null NOT NULL surface_generation +constraint keiro_managed_read_objects.keiro_managed_read_objects_updated_at_not_null NOT NULL updated_at constraint keiro_outbox.keiro_outbox_attempt_count_not_null NOT NULL attempt_count constraint keiro_outbox.keiro_outbox_content_type_not_null NOT NULL content_type constraint keiro_outbox.keiro_outbox_created_at_not_null NOT NULL created_at @@ -194,8 +385,148 @@ constraint keiro_projection_dedup.keiro_projection_dedup_event_id_not_null NOT NULL event_id constraint keiro_projection_dedup.keiro_projection_dedup_pkey PRIMARY KEY (projection_name, event_id) constraint keiro_projection_dedup.keiro_projection_dedup_projection_name_not_null NOT NULL projection_name +constraint keiro_projection_group_cursors.keiro_projection_group_cursors_basis_chk CHECK ((position_basis = ANY (ARRAY['append'::text, 'checkpoint'::text, 'unmanaged'::text]))) +constraint keiro_projection_group_cursors.keiro_projection_group_cursors_group_id_fkey FOREIGN KEY (group_id) REFERENCES keiro.keiro_projection_rebuild_groups(group_id) ON DELETE CASCADE +constraint keiro_projection_group_cursors.keiro_projection_group_cursors_group_id_not_null NOT NULL group_id +constraint keiro_projection_group_cursors.keiro_projection_group_cursors_pkey PRIMARY KEY (group_id) +constraint keiro_projection_group_cursors.keiro_projection_group_cursors_position_basis_not_null NOT NULL position_basis +constraint keiro_projection_group_cursors.keiro_projection_group_cursors_subscription_names_chk CHECK (((array_position(subscription_names, NULL::text) IS NULL) AND (((position_basis = 'checkpoint'::text) AND (cardinality(subscription_names) > 0)) OR ((position_basis = ANY (ARRAY['append'::text, 'unmanaged'::text])) AND (cardinality(subscription_names) = 0))))) +constraint keiro_projection_group_cursors.keiro_projection_group_cursors_subscription_names_not_null NOT NULL subscription_names +constraint keiro_projection_group_cursors.keiro_projection_group_cursors_updated_at_not_null NOT NULL updated_at +constraint keiro_projection_rebuild_adapters.keiro_projection_rebuild_adapters_adapter_order_not_null NOT NULL adapter_order +constraint keiro_projection_rebuild_adapters.keiro_projection_rebuild_adapters_apply_count_not_null NOT NULL apply_count +constraint keiro_projection_rebuild_adapters.keiro_projection_rebuild_adapters_completed_chk CHECK (((completed_through IS NULL) OR (completed_through >= 0))) +constraint keiro_projection_rebuild_adapters.keiro_projection_rebuild_adapters_counts_chk CHECK (((evaluation_count >= 0) AND (apply_count >= 0) AND (apply_count <= evaluation_count))) +constraint keiro_projection_rebuild_adapters.keiro_projection_rebuild_adapters_evaluation_count_not_null NOT NULL evaluation_count +constraint keiro_projection_rebuild_adapters.keiro_projection_rebuild_adapters_order_chk CHECK ((adapter_order >= 0)) +constraint keiro_projection_rebuild_adapters.keiro_projection_rebuild_adapters_pkey PRIMARY KEY (run_id, source_id, projection_id) +constraint keiro_projection_rebuild_adapters.keiro_projection_rebuild_adapters_projection_id_not_null NOT NULL projection_id +constraint keiro_projection_rebuild_adapters.keiro_projection_rebuild_adapters_run_id_adapter_order_key UNIQUE (run_id, adapter_order) +constraint keiro_projection_rebuild_adapters.keiro_projection_rebuild_adapters_run_id_not_null NOT NULL run_id +constraint keiro_projection_rebuild_adapters.keiro_projection_rebuild_adapters_run_id_source_id_fkey FOREIGN KEY (run_id, source_id) REFERENCES keiro.keiro_projection_rebuild_sources(run_id, source_id) ON DELETE CASCADE +constraint keiro_projection_rebuild_adapters.keiro_projection_rebuild_adapters_source_id_not_null NOT NULL source_id +constraint keiro_projection_rebuild_adapters.keiro_projection_rebuild_adapters_updated_at_not_null NOT NULL updated_at +constraint keiro_projection_rebuild_dedup_stage.keiro_projection_rebuild_dedup_stage_event_id_not_null NOT NULL event_id +constraint keiro_projection_rebuild_dedup_stage.keiro_projection_rebuild_dedup_stage_global_position_not_null NOT NULL global_position +constraint keiro_projection_rebuild_dedup_stage.keiro_projection_rebuild_dedup_stage_names_chk CHECK (((length(subscription_name) > 0) AND (length(projection_name) > 0))) +constraint keiro_projection_rebuild_dedup_stage.keiro_projection_rebuild_dedup_stage_pkey PRIMARY KEY (run_id, projection_name, event_id) +constraint keiro_projection_rebuild_dedup_stage.keiro_projection_rebuild_dedup_stage_position_chk CHECK ((global_position > 0)) +constraint keiro_projection_rebuild_dedup_stage.keiro_projection_rebuild_dedup_stage_projection_name_not_null NOT NULL projection_name +constraint keiro_projection_rebuild_dedup_stage.keiro_projection_rebuild_dedup_stage_run_id_fkey FOREIGN KEY (run_id) REFERENCES keiro.keiro_projection_rebuild_runs(run_id) ON DELETE CASCADE +constraint keiro_projection_rebuild_dedup_stage.keiro_projection_rebuild_dedup_stage_run_id_not_null NOT NULL run_id +constraint keiro_projection_rebuild_dedup_stage.keiro_projection_rebuild_dedup_stage_subscription_name_not_null NOT NULL subscription_name +constraint keiro_projection_rebuild_groups.keiro_projection_rebuild_groups_active_run_chk CHECK ((((status = ANY (ARRAY['live'::text, 'serving-versioned'::text])) AND (active_run_id IS NULL)) OR ((status = ANY (ARRAY['rebuilding'::text, 'failed'::text, 'rebuilding-versioned'::text, 'cutover-versioned'::text, 'failed-versioned'::text])) AND (active_run_id IS NOT NULL)))) +constraint keiro_projection_rebuild_groups.keiro_projection_rebuild_groups_availability_chk CHECK ((((status = 'live'::text) AND reads_allowed AND writes_allowed) OR ((status = ANY (ARRAY['rebuilding'::text, 'failed'::text])) AND (NOT reads_allowed) AND (NOT writes_allowed)) OR ((status = ANY (ARRAY['serving-versioned'::text, 'rebuilding-versioned'::text])) AND reads_allowed AND writes_allowed) OR ((status = ANY (ARRAY['cutover-versioned'::text, 'failed-versioned'::text])) AND reads_allowed AND (NOT writes_allowed)))) +constraint keiro_projection_rebuild_groups.keiro_projection_rebuild_groups_catalog_fingerprint_not_null NOT NULL slice_fingerprint +constraint keiro_projection_rebuild_groups.keiro_projection_rebuild_groups_created_at_not_null NOT NULL created_at +constraint keiro_projection_rebuild_groups.keiro_projection_rebuild_groups_group_id_not_null NOT NULL group_id +constraint keiro_projection_rebuild_groups.keiro_projection_rebuild_groups_pkey PRIMARY KEY (group_id) +constraint keiro_projection_rebuild_groups.keiro_projection_rebuild_groups_reads_allowed_not_null NOT NULL reads_allowed +constraint keiro_projection_rebuild_groups.keiro_projection_rebuild_groups_revision_phase_chk CHECK ((((status = ANY (ARRAY['live'::text, 'rebuilding'::text, 'failed'::text])) AND (serving_revision_id IS NULL) AND (serving_epoch = 0)) OR ((status = ANY (ARRAY['serving-versioned'::text, 'rebuilding-versioned'::text, 'cutover-versioned'::text, 'failed-versioned'::text])) AND (serving_revision_id IS NOT NULL) AND (serving_epoch >= 0)))) +constraint keiro_projection_rebuild_groups.keiro_projection_rebuild_groups_serving_epoch_not_null NOT NULL serving_epoch +constraint keiro_projection_rebuild_groups.keiro_projection_rebuild_groups_serving_revision_fk FOREIGN KEY (group_id, serving_revision_id) REFERENCES keiro.keiro_projection_revisions(group_id, revision_id) +constraint keiro_projection_rebuild_groups.keiro_projection_rebuild_groups_status_chk CHECK ((status = ANY (ARRAY['live'::text, 'rebuilding'::text, 'failed'::text, 'serving-versioned'::text, 'rebuilding-versioned'::text, 'cutover-versioned'::text, 'failed-versioned'::text]))) +constraint keiro_projection_rebuild_groups.keiro_projection_rebuild_groups_status_not_null NOT NULL status +constraint keiro_projection_rebuild_groups.keiro_projection_rebuild_groups_updated_at_not_null NOT NULL updated_at +constraint keiro_projection_rebuild_groups.keiro_projection_rebuild_groups_writes_allowed_not_null NOT NULL writes_allowed +constraint keiro_projection_rebuild_promotion_objects.keiro_projection_rebuild_prom_run_id_target_id_canonical_na_key UNIQUE (run_id, target_id, canonical_name) +constraint keiro_projection_rebuild_promotion_objects.keiro_projection_rebuild_prom_run_id_target_id_generation_n_key UNIQUE (run_id, target_id, generation_name) +constraint keiro_projection_rebuild_promotion_objects.keiro_projection_rebuild_promotion_obj_generation_name_not_null NOT NULL generation_name +constraint keiro_projection_rebuild_promotion_objects.keiro_projection_rebuild_promotion_obje_canonical_name_not_null NOT NULL canonical_name +constraint keiro_projection_rebuild_promotion_objects.keiro_projection_rebuild_promotion_object_object_order_not_null NOT NULL object_order +constraint keiro_projection_rebuild_promotion_objects.keiro_projection_rebuild_promotion_object_run_id_target_id_fkey FOREIGN KEY (run_id, target_id) REFERENCES keiro.keiro_projection_rebuild_run_targets(run_id, target_id) ON DELETE CASCADE +constraint keiro_projection_rebuild_promotion_objects.keiro_projection_rebuild_promotion_objects_kind_chk CHECK ((object_kind = ANY (ARRAY['index'::text, 'constraint'::text, 'owned-sequence'::text]))) +constraint keiro_projection_rebuild_promotion_objects.keiro_projection_rebuild_promotion_objects_names_chk CHECK (((length(generation_name) > 0) AND (length(canonical_name) > 0))) +constraint keiro_projection_rebuild_promotion_objects.keiro_projection_rebuild_promotion_objects_object_kind_not_null NOT NULL object_kind +constraint keiro_projection_rebuild_promotion_objects.keiro_projection_rebuild_promotion_objects_order_chk CHECK ((object_order >= 0)) +constraint keiro_projection_rebuild_promotion_objects.keiro_projection_rebuild_promotion_objects_pkey PRIMARY KEY (run_id, target_id, object_order) +constraint keiro_projection_rebuild_promotion_objects.keiro_projection_rebuild_promotion_objects_run_id_not_null NOT NULL run_id +constraint keiro_projection_rebuild_promotion_objects.keiro_projection_rebuild_promotion_objects_target_id_not_null NOT NULL target_id +constraint keiro_projection_rebuild_run_targets.keiro_projection_rebuild_run__candidate_generation_id_run__fkey FOREIGN KEY (candidate_generation_id, run_id, target_id) REFERENCES keiro.keiro_projection_target_generations(generation_id, created_by_run_id, target_id) +constraint keiro_projection_rebuild_run_targets.keiro_projection_rebuild_run_t_candidate_generation_id_not_null NOT NULL candidate_generation_id +constraint keiro_projection_rebuild_run_targets.keiro_projection_rebuild_run_target_candidate_generation_id_key UNIQUE (candidate_generation_id) +constraint keiro_projection_rebuild_run_targets.keiro_projection_rebuild_run_targets_mode_chk CHECK ((target_mode = ANY (ARRAY['application'::text, 'clone'::text]))) +constraint keiro_projection_rebuild_run_targets.keiro_projection_rebuild_run_targets_pkey PRIMARY KEY (run_id, target_id) +constraint keiro_projection_rebuild_run_targets.keiro_projection_rebuild_run_targets_run_id_fkey FOREIGN KEY (run_id) REFERENCES keiro.keiro_projection_rebuild_runs(run_id) ON DELETE CASCADE +constraint keiro_projection_rebuild_run_targets.keiro_projection_rebuild_run_targets_run_id_not_null NOT NULL run_id +constraint keiro_projection_rebuild_run_targets.keiro_projection_rebuild_run_targets_target_id_not_null NOT NULL target_id +constraint keiro_projection_rebuild_run_targets.keiro_projection_rebuild_run_targets_target_mode_not_null NOT NULL target_mode +constraint keiro_projection_rebuild_runs.keiro_projection_rebuild_runs_abandoned_chk CHECK ((((status = 'abandoned'::text) AND (abandoned_at IS NOT NULL)) OR ((status <> 'abandoned'::text) AND (abandoned_at IS NULL)))) +constraint keiro_projection_rebuild_runs.keiro_projection_rebuild_runs_candidate_revision_fk FOREIGN KEY (group_id, candidate_revision_id) REFERENCES keiro.keiro_projection_revisions(group_id, revision_id) +constraint keiro_projection_rebuild_runs.keiro_projection_rebuild_runs_captured_head_not_null NOT NULL captured_head +constraint keiro_projection_rebuild_runs.keiro_projection_rebuild_runs_catalog_fingerprint_not_null NOT NULL catalog_fingerprint +constraint keiro_projection_rebuild_runs.keiro_projection_rebuild_runs_contract_fingerprint_not_null NOT NULL contract_fingerprint +constraint keiro_projection_rebuild_runs.keiro_projection_rebuild_runs_failure_chk CHECK ((((status = 'failed'::text) AND (failure_code IS NOT NULL) AND (failed_at IS NOT NULL)) OR ((status <> 'failed'::text) AND (failure_code IS NULL) AND (failed_at IS NULL)))) +constraint keiro_projection_rebuild_runs.keiro_projection_rebuild_runs_group_id_fkey FOREIGN KEY (group_id) REFERENCES keiro.keiro_projection_rebuild_groups(group_id) +constraint keiro_projection_rebuild_runs.keiro_projection_rebuild_runs_group_id_not_null NOT NULL group_id +constraint keiro_projection_rebuild_runs.keiro_projection_rebuild_runs_group_slice_fingerprint_not_null NOT NULL group_slice_fingerprint +constraint keiro_projection_rebuild_runs.keiro_projection_rebuild_runs_head_chk CHECK ((captured_head >= 0)) +constraint keiro_projection_rebuild_runs.keiro_projection_rebuild_runs_mode_chk CHECK ((((rebuild_mode = 'offline'::text) AND (candidate_revision_id IS NULL) AND (cutover_threshold IS NULL) AND (cutover_lock_timeout_ms IS NULL) AND (promotion_dedup_limit IS NULL) AND (dedup_provisional_head IS NULL) AND (promotion_prepared_at IS NULL) AND (history_retention_lease_id IS NULL) AND (history_retention_lease_owner IS NULL) AND (history_retention_protected_through IS NULL) AND (history_retention_expires_at IS NULL) AND (history_retention_renewed_at IS NULL) AND (history_retention_released_at IS NULL)) OR ((rebuild_mode = 'versioned'::text) AND (candidate_revision_id IS NOT NULL) AND (cutover_threshold IS NOT NULL) AND (cutover_threshold >= 0) AND (cutover_lock_timeout_ms IS NOT NULL) AND (cutover_lock_timeout_ms > 0) AND (promotion_dedup_limit IS NOT NULL) AND (promotion_dedup_limit > 0) AND ((dedup_provisional_head IS NULL) OR (dedup_provisional_head >= 0)) AND (history_retention_lease_id IS NOT NULL) AND (history_retention_lease_owner IS NOT NULL) AND (length(history_retention_lease_owner) > 0) AND (history_retention_protected_through IS NOT NULL) AND (history_retention_protected_through >= 0) AND (history_retention_expires_at IS NOT NULL) AND (history_retention_renewed_at IS NOT NULL)))) +constraint keiro_projection_rebuild_runs.keiro_projection_rebuild_runs_page_size_chk CHECK ((page_size > 0)) +constraint keiro_projection_rebuild_runs.keiro_projection_rebuild_runs_page_size_not_null NOT NULL page_size +constraint keiro_projection_rebuild_runs.keiro_projection_rebuild_runs_pkey PRIMARY KEY (run_id) +constraint keiro_projection_rebuild_runs.keiro_projection_rebuild_runs_rebuild_mode_not_null NOT NULL rebuild_mode +constraint keiro_projection_rebuild_runs.keiro_projection_rebuild_runs_run_id_not_null NOT NULL run_id +constraint keiro_projection_rebuild_runs.keiro_projection_rebuild_runs_runner_format_not_null NOT NULL runner_format +constraint keiro_projection_rebuild_runs.keiro_projection_rebuild_runs_started_at_not_null NOT NULL started_at +constraint keiro_projection_rebuild_runs.keiro_projection_rebuild_runs_status_chk CHECK ((status = ANY (ARRAY['running'::text, 'failed'::text, 'verified'::text, 'cutover'::text, 'promoted'::text, 'abandoned'::text]))) +constraint keiro_projection_rebuild_runs.keiro_projection_rebuild_runs_status_not_null NOT NULL status +constraint keiro_projection_rebuild_runs.keiro_projection_rebuild_runs_updated_at_not_null NOT NULL updated_at +constraint keiro_projection_rebuild_sources.keiro_projection_rebuild_sources_count_chk CHECK ((event_count >= 0)) +constraint keiro_projection_rebuild_sources.keiro_projection_rebuild_sources_cursor_chk CHECK (((cursor_position >= 0) AND (cursor_position <= target_position))) +constraint keiro_projection_rebuild_sources.keiro_projection_rebuild_sources_cursor_position_not_null NOT NULL cursor_position +constraint keiro_projection_rebuild_sources.keiro_projection_rebuild_sources_event_count_not_null NOT NULL event_count +constraint keiro_projection_rebuild_sources.keiro_projection_rebuild_sources_exhausted_chk CHECK (((exhausted_through IS NULL) OR (exhausted_through = target_position))) +constraint keiro_projection_rebuild_sources.keiro_projection_rebuild_sources_pkey PRIMARY KEY (run_id, source_id) +constraint keiro_projection_rebuild_sources.keiro_projection_rebuild_sources_run_id_fkey FOREIGN KEY (run_id) REFERENCES keiro.keiro_projection_rebuild_runs(run_id) ON DELETE CASCADE +constraint keiro_projection_rebuild_sources.keiro_projection_rebuild_sources_run_id_not_null NOT NULL run_id +constraint keiro_projection_rebuild_sources.keiro_projection_rebuild_sources_scope_chk CHECK ((((source_scope = 'all'::text) AND (category IS NULL)) OR ((source_scope = 'category'::text) AND (category IS NOT NULL)))) +constraint keiro_projection_rebuild_sources.keiro_projection_rebuild_sources_source_id_not_null NOT NULL source_id +constraint keiro_projection_rebuild_sources.keiro_projection_rebuild_sources_source_scope_not_null NOT NULL source_scope +constraint keiro_projection_rebuild_sources.keiro_projection_rebuild_sources_target_chk CHECK ((target_position >= 0)) +constraint keiro_projection_rebuild_sources.keiro_projection_rebuild_sources_target_position_not_null NOT NULL target_position +constraint keiro_projection_rebuild_sources.keiro_projection_rebuild_sources_updated_at_not_null NOT NULL updated_at +constraint keiro_projection_rebuild_verifications.keiro_projection_rebuild_verifica_verification_version_not_null NOT NULL verification_version +constraint keiro_projection_rebuild_verifications.keiro_projection_rebuild_verifications_completion_chk CHECK ((((status = 'pending'::text) AND (completed_at IS NULL)) OR ((status = ANY (ARRAY['passed'::text, 'failed'::text])) AND (completed_at IS NOT NULL)))) +constraint keiro_projection_rebuild_verifications.keiro_projection_rebuild_verifications_pkey PRIMARY KEY (run_id, verification_id) +constraint keiro_projection_rebuild_verifications.keiro_projection_rebuild_verifications_run_id_fkey FOREIGN KEY (run_id) REFERENCES keiro.keiro_projection_rebuild_runs(run_id) ON DELETE CASCADE +constraint keiro_projection_rebuild_verifications.keiro_projection_rebuild_verifications_run_id_not_null NOT NULL run_id +constraint keiro_projection_rebuild_verifications.keiro_projection_rebuild_verifications_status_chk CHECK ((status = ANY (ARRAY['pending'::text, 'passed'::text, 'failed'::text]))) +constraint keiro_projection_rebuild_verifications.keiro_projection_rebuild_verifications_status_not_null NOT NULL status +constraint keiro_projection_rebuild_verifications.keiro_projection_rebuild_verifications_verification_id_not_null NOT NULL verification_id +constraint keiro_projection_revisions.keiro_projection_revisions_group_id_fkey FOREIGN KEY (group_id) REFERENCES keiro.keiro_projection_rebuild_groups(group_id) +constraint keiro_projection_revisions.keiro_projection_revisions_group_id_not_null NOT NULL group_id +constraint keiro_projection_revisions.keiro_projection_revisions_group_slice_fingerprint_not_null NOT NULL group_slice_fingerprint +constraint keiro_projection_revisions.keiro_projection_revisions_identity_chk CHECK (((length(group_id) > 0) AND (length(revision_id) > 0) AND (length(group_slice_fingerprint) > 0))) +constraint keiro_projection_revisions.keiro_projection_revisions_pkey PRIMARY KEY (group_id, revision_id) +constraint keiro_projection_revisions.keiro_projection_revisions_registered_at_not_null NOT NULL registered_at +constraint keiro_projection_revisions.keiro_projection_revisions_revision_id_key UNIQUE (revision_id) +constraint keiro_projection_revisions.keiro_projection_revisions_revision_id_not_null NOT NULL revision_id +constraint keiro_projection_revisions.keiro_projection_revisions_updated_at_not_null NOT NULL updated_at +constraint keiro_projection_target_generations.keiro_projection_target_gen_observed_shape_fingerprint_not_null NOT NULL observed_shape_fingerprint +constraint keiro_projection_target_generations.keiro_projection_target_gene_observed_catalog_snapshot_not_null NOT NULL observed_catalog_snapshot +constraint keiro_projection_target_generations.keiro_projection_target_gener_generation_id_created_by_run__key UNIQUE (generation_id, created_by_run_id, target_id) +constraint keiro_projection_target_generations.keiro_projection_target_generations_created_at_not_null NOT NULL created_at +constraint keiro_projection_target_generations.keiro_projection_target_generations_created_by_run_id_fkey FOREIGN KEY (created_by_run_id) REFERENCES keiro.keiro_projection_rebuild_runs(run_id) +constraint keiro_projection_target_generations.keiro_projection_target_generations_expected_shape_id_not_null NOT NULL expected_shape_id +constraint keiro_projection_target_generations.keiro_projection_target_generations_generation_id_not_null NOT NULL generation_id +constraint keiro_projection_target_generations.keiro_projection_target_generations_group_id_not_null NOT NULL group_id +constraint keiro_projection_target_generations.keiro_projection_target_generations_group_id_revision_id_fkey FOREIGN KEY (group_id, revision_id) REFERENCES keiro.keiro_projection_revisions(group_id, revision_id) +constraint keiro_projection_target_generations.keiro_projection_target_generations_identity_chk CHECK (((length(target_id) > 0) AND (length(schema_name) > 0) AND (length(relation_name) > 0) AND (relation_oid > 0) AND (length(schema_version) > 0) AND (length(expected_shape_id) > 0) AND (length(observed_shape_fingerprint) > 0))) +constraint keiro_projection_target_generations.keiro_projection_target_generations_lifecycle_chk CHECK ((lifecycle = ANY (ARRAY['staging'::text, 'serving'::text, 'retired'::text, 'dropped'::text]))) +constraint keiro_projection_target_generations.keiro_projection_target_generations_lifecycle_not_null NOT NULL lifecycle +constraint keiro_projection_target_generations.keiro_projection_target_generations_pkey PRIMARY KEY (generation_id) +constraint keiro_projection_target_generations.keiro_projection_target_generations_relation_name_not_null NOT NULL relation_name +constraint keiro_projection_target_generations.keiro_projection_target_generations_relation_oid_not_null NOT NULL relation_oid +constraint keiro_projection_target_generations.keiro_projection_target_generations_revision_id_not_null NOT NULL revision_id +constraint keiro_projection_target_generations.keiro_projection_target_generations_schema_name_not_null NOT NULL schema_name +constraint keiro_projection_target_generations.keiro_projection_target_generations_schema_version_not_null NOT NULL schema_version +constraint keiro_projection_target_generations.keiro_projection_target_generations_target_id_not_null NOT NULL target_id +constraint keiro_projection_target_generations.keiro_projection_target_generations_transition_evidence_chk CHECK ((((lifecycle = 'staging'::text) AND (created_by_run_id IS NOT NULL) AND (served_at IS NULL) AND (retired_at IS NULL) AND (dropped_at IS NULL)) OR ((lifecycle = 'serving'::text) AND (served_at IS NOT NULL) AND (retired_at IS NULL) AND (dropped_at IS NULL)) OR ((lifecycle = 'retired'::text) AND (served_at IS NOT NULL) AND (retired_at IS NOT NULL) AND (dropped_at IS NULL)) OR ((lifecycle = 'dropped'::text) AND (dropped_at IS NOT NULL) AND (((served_at IS NULL) AND (retired_at IS NULL)) OR ((served_at IS NOT NULL) AND (retired_at IS NOT NULL)))))) constraint keiro_read_models.keiro_read_models_name_not_null NOT NULL name constraint keiro_read_models.keiro_read_models_pkey PRIMARY KEY (name) +constraint keiro_read_models.keiro_read_models_rebuild_group_fk FOREIGN KEY (rebuild_group_id) REFERENCES keiro.keiro_projection_rebuild_groups(group_id) +constraint keiro_read_models.keiro_read_models_rebuild_group_id_not_null NOT NULL rebuild_group_id constraint keiro_read_models.keiro_read_models_shape_hash_not_null NOT NULL shape_hash constraint keiro_read_models.keiro_read_models_status_not_null NOT NULL status constraint keiro_read_models.keiro_read_models_updated_at_not_null NOT NULL updated_at @@ -258,9 +589,14 @@ index keiro_dead_letters_dispatcher_created_at_idx CREATE INDEX keiro_dead_letters_dispatcher_created_at_idx ON keiro.keiro_dead_letters USING btree (dispatcher_name, created_at DESC) index keiro_dead_letters_dispatcher_name_source_event_id_emit_ind_key CREATE UNIQUE INDEX keiro_dead_letters_dispatcher_name_source_event_id_emit_ind_key ON keiro.keiro_dead_letters USING btree (dispatcher_name, source_event_id, emit_index) index keiro_dead_letters_pkey CREATE UNIQUE INDEX keiro_dead_letters_pkey ON keiro.keiro_dead_letters USING btree (dead_letter_id) +index keiro_external_read_contracts_group_idx CREATE INDEX keiro_external_read_contracts_group_idx ON keiro.keiro_external_read_contracts USING btree (group_id, state) +index keiro_external_read_contracts_pkey CREATE UNIQUE INDEX keiro_external_read_contracts_pkey ON keiro.keiro_external_read_contracts USING btree (contract_id, contract_version) +index keiro_external_read_contracts_public_function_name_argument_key CREATE UNIQUE INDEX keiro_external_read_contracts_public_function_name_argument_key ON keiro.keiro_external_read_contracts USING btree (public_function_name, argument_types) index keiro_inbox_backlog_idx CREATE INDEX keiro_inbox_backlog_idx ON keiro.keiro_inbox USING btree (status) WHERE (status = ANY (ARRAY['processing'::text, 'failed'::text])) index keiro_inbox_completed_idx CREATE INDEX keiro_inbox_completed_idx ON keiro.keiro_inbox USING btree (completed_at) WHERE (status = 'completed'::text) index keiro_inbox_pkey CREATE UNIQUE INDEX keiro_inbox_pkey ON keiro.keiro_inbox USING btree (source, dedupe_key) +index keiro_managed_read_objects_contract_idx CREATE INDEX keiro_managed_read_objects_contract_idx ON keiro.keiro_managed_read_objects USING btree (contract_id, contract_version) +index keiro_managed_read_objects_pkey CREATE UNIQUE INDEX keiro_managed_read_objects_pkey ON keiro.keiro_managed_read_objects USING btree (object_schema, object_name, object_kind, object_signature) index keiro_outbox_claim_order_idx CREATE INDEX keiro_outbox_claim_order_idx ON keiro.keiro_outbox USING btree (created_at, outbox_id) WHERE (status = ANY (ARRAY['pending'::text, 'failed'::text])) index keiro_outbox_head_of_line_idx CREATE INDEX keiro_outbox_head_of_line_idx ON keiro.keiro_outbox USING btree (source, message_key, created_at) WHERE ((status <> ALL (ARRAY['sent'::text, 'dead'::text])) AND (message_key IS NOT NULL)) index keiro_outbox_pending_idx CREATE INDEX keiro_outbox_pending_idx ON keiro.keiro_outbox USING btree (status, next_attempt_at, created_at) @@ -270,7 +606,29 @@ index keiro_outbox_source_order_idx CREATE INDEX keiro_outbox_source_order_idx ON keiro.keiro_outbox USING btree (source, created_at, outbox_id) WHERE (status <> ALL (ARRAY['sent'::text, 'dead'::text])) index keiro_projection_dedup_applied_at_idx CREATE INDEX keiro_projection_dedup_applied_at_idx ON keiro.keiro_projection_dedup USING btree (applied_at) index keiro_projection_dedup_pkey CREATE UNIQUE INDEX keiro_projection_dedup_pkey ON keiro.keiro_projection_dedup USING btree (projection_name, event_id) +index keiro_projection_group_cursors_pkey CREATE UNIQUE INDEX keiro_projection_group_cursors_pkey ON keiro.keiro_projection_group_cursors USING btree (group_id) +index keiro_projection_rebuild_adapters_pkey CREATE UNIQUE INDEX keiro_projection_rebuild_adapters_pkey ON keiro.keiro_projection_rebuild_adapters USING btree (run_id, source_id, projection_id) +index keiro_projection_rebuild_adapters_run_id_adapter_order_key CREATE UNIQUE INDEX keiro_projection_rebuild_adapters_run_id_adapter_order_key ON keiro.keiro_projection_rebuild_adapters USING btree (run_id, adapter_order) +index keiro_projection_rebuild_dedup_stage_pkey CREATE UNIQUE INDEX keiro_projection_rebuild_dedup_stage_pkey ON keiro.keiro_projection_rebuild_dedup_stage USING btree (run_id, projection_name, event_id) +index keiro_projection_rebuild_dedup_stage_run_position_idx CREATE INDEX keiro_projection_rebuild_dedup_stage_run_position_idx ON keiro.keiro_projection_rebuild_dedup_stage USING btree (run_id, subscription_name, global_position) +index keiro_projection_rebuild_groups_pkey CREATE UNIQUE INDEX keiro_projection_rebuild_groups_pkey ON keiro.keiro_projection_rebuild_groups USING btree (group_id) +index keiro_projection_rebuild_prom_run_id_target_id_canonical_na_key CREATE UNIQUE INDEX keiro_projection_rebuild_prom_run_id_target_id_canonical_na_key ON keiro.keiro_projection_rebuild_promotion_objects USING btree (run_id, target_id, canonical_name) +index keiro_projection_rebuild_prom_run_id_target_id_generation_n_key CREATE UNIQUE INDEX keiro_projection_rebuild_prom_run_id_target_id_generation_n_key ON keiro.keiro_projection_rebuild_promotion_objects USING btree (run_id, target_id, generation_name) +index keiro_projection_rebuild_promotion_objects_pkey CREATE UNIQUE INDEX keiro_projection_rebuild_promotion_objects_pkey ON keiro.keiro_projection_rebuild_promotion_objects USING btree (run_id, target_id, object_order) +index keiro_projection_rebuild_run_target_candidate_generation_id_key CREATE UNIQUE INDEX keiro_projection_rebuild_run_target_candidate_generation_id_key ON keiro.keiro_projection_rebuild_run_targets USING btree (candidate_generation_id) +index keiro_projection_rebuild_run_targets_pkey CREATE UNIQUE INDEX keiro_projection_rebuild_run_targets_pkey ON keiro.keiro_projection_rebuild_run_targets USING btree (run_id, target_id) +index keiro_projection_rebuild_runs_group_idx CREATE INDEX keiro_projection_rebuild_runs_group_idx ON keiro.keiro_projection_rebuild_runs USING btree (group_id, started_at DESC) +index keiro_projection_rebuild_runs_pkey CREATE UNIQUE INDEX keiro_projection_rebuild_runs_pkey ON keiro.keiro_projection_rebuild_runs USING btree (run_id) +index keiro_projection_rebuild_sources_pkey CREATE UNIQUE INDEX keiro_projection_rebuild_sources_pkey ON keiro.keiro_projection_rebuild_sources USING btree (run_id, source_id) +index keiro_projection_rebuild_verifications_pkey CREATE UNIQUE INDEX keiro_projection_rebuild_verifications_pkey ON keiro.keiro_projection_rebuild_verifications USING btree (run_id, verification_id) +index keiro_projection_revisions_pkey CREATE UNIQUE INDEX keiro_projection_revisions_pkey ON keiro.keiro_projection_revisions USING btree (group_id, revision_id) +index keiro_projection_revisions_revision_id_key CREATE UNIQUE INDEX keiro_projection_revisions_revision_id_key ON keiro.keiro_projection_revisions USING btree (revision_id) +index keiro_projection_target_gener_generation_id_created_by_run__key CREATE UNIQUE INDEX keiro_projection_target_gener_generation_id_created_by_run__key ON keiro.keiro_projection_target_generations USING btree (generation_id, created_by_run_id, target_id) +index keiro_projection_target_generations_pkey CREATE UNIQUE INDEX keiro_projection_target_generations_pkey ON keiro.keiro_projection_target_generations USING btree (generation_id) +index keiro_projection_target_generations_relation_idx CREATE UNIQUE INDEX keiro_projection_target_generations_relation_idx ON keiro.keiro_projection_target_generations USING btree (schema_name, relation_name) WHERE (lifecycle <> 'dropped'::text) +index keiro_projection_target_generations_serving_idx CREATE UNIQUE INDEX keiro_projection_target_generations_serving_idx ON keiro.keiro_projection_target_generations USING btree (group_id, target_id) WHERE (lifecycle = 'serving'::text) index keiro_read_models_pkey CREATE UNIQUE INDEX keiro_read_models_pkey ON keiro.keiro_read_models USING btree (name) +index keiro_read_models_rebuild_group_idx CREATE INDEX keiro_read_models_rebuild_group_idx ON keiro.keiro_read_models USING btree (rebuild_group_id) index keiro_snapshots_compat_idx CREATE INDEX keiro_snapshots_compat_idx ON keiro.keiro_snapshots USING btree (stream_id, state_codec_version, regfile_shape_hash, stream_version DESC) index keiro_snapshots_pkey CREATE UNIQUE INDEX keiro_snapshots_pkey ON keiro.keiro_snapshots USING btree (stream_id) index keiro_subscription_shards_lease_idx CREATE INDEX keiro_subscription_shards_lease_idx ON keiro.keiro_subscription_shards USING btree (subscription_name, lease_expires_at) @@ -283,14 +641,27 @@ index keiro_workflow_children_running_idx CREATE INDEX keiro_workflow_children_running_idx ON keiro.keiro_workflow_children USING btree (status) WHERE (status = 'running'::text) index keiro_workflow_steps_pkey CREATE UNIQUE INDEX keiro_workflow_steps_pkey ON keiro.keiro_workflow_steps USING btree (workflow_id, workflow_name, generation, step_name) index keiro_workflow_steps_workflow_idx CREATE INDEX keiro_workflow_steps_workflow_idx ON keiro.keiro_workflow_steps USING btree (workflow_id, workflow_name, generation) -index keiro_workflows_active_idx CREATE INDEX keiro_workflows_active_idx ON keiro.keiro_workflows USING btree (status) WHERE (status = ANY (ARRAY['running'::text, 'suspended'::text])) +index keiro_workflows_active_idx CREATE INDEX keiro_workflows_active_idx ON keiro.keiro_workflows USING btree (status, wake_after) WHERE (status = ANY (ARRAY['running'::text, 'suspended'::text])) index keiro_workflows_gc_idx CREATE INDEX keiro_workflows_gc_idx ON keiro.keiro_workflows USING btree (status, completed_at) index keiro_workflows_pkey CREATE UNIQUE INDEX keiro_workflows_pkey ON keiro.keiro_workflows USING btree (workflow_id, workflow_name) table keiro_awakeables kind=r table keiro_dead_letters kind=r +table keiro_external_read_contracts kind=r table keiro_inbox kind=r +table keiro_managed_read_objects kind=r table keiro_outbox kind=r table keiro_projection_dedup kind=r +table keiro_projection_group_cursors kind=r +table keiro_projection_rebuild_adapters kind=r +table keiro_projection_rebuild_dedup_stage kind=r +table keiro_projection_rebuild_groups kind=r +table keiro_projection_rebuild_promotion_objects kind=r +table keiro_projection_rebuild_run_targets kind=r +table keiro_projection_rebuild_runs kind=r +table keiro_projection_rebuild_sources kind=r +table keiro_projection_rebuild_verifications kind=r +table keiro_projection_revisions kind=r +table keiro_projection_target_generations kind=r table keiro_read_models kind=r table keiro_snapshots kind=r table keiro_subscription_shards kind=r @@ -298,3 +669,4 @@ table keiro_workflow_children kind=r table keiro_workflow_steps kind=r table keiro_workflows kind=r +view keiro_read.projection_group_status_v1 columns=1:group_id text,2:lifecycle_phase text,3:reads_allowed boolean,4:writes_allowed boolean,5:serving_revision_id text,6:serving_epoch bigint,7:serving_position_basis text,8:serving_applied_position bigint,9:active_run_id text,10:candidate_revision_id text,11:candidate_rebuild_position bigint,12:candidate_rebuild_head bigint,13:query_models text[],14:rebuild_started_at timestamp with time zone,15:last_promoted_at timestamp with time zone,16:failed_at timestamp with time zone,17:failure_code text,18:failure_detail text diff --git a/keiro-migrations.cabal b/keiro-migrations.cabal --- a/keiro-migrations.cabal +++ b/keiro-migrations.cabal @@ -1,12 +1,13 @@ cabal-version: 3.0 name: keiro-migrations -version: 0.11.0.0 +version: 0.12.0.0 synopsis: Schema migrations for keiro description: Embedded PostgreSQL schema migrations and a migration runner for the Keiro event-sourcing framework. license: BSD-3-Clause +license-file: LICENSE author: Nadeem Bitar maintainer: nadeem@gmail.com copyright: 2026 Nadeem Bitar @@ -332,7 +333,7 @@ , bytestring >=0.11 && <0.13 , containers >=0.6 && <0.8 , hasql >=1.10 && <1.11 - , kiroku-store-migrations ^>=0.3.0.0 + , kiroku-store-migrations ^>=0.3.2.0 , pg-migrate ^>=1.1.0.0 , pg-migrate-embed ^>=1.1.0.0 , pg-migrate-import-codd ^>=1.1.0.0 @@ -364,7 +365,7 @@ , bytestring >=0.11 && <0.13 , hasql >=1.10 && <1.11 , keiro-migrations - , kiroku-store-migrations ^>=0.3.0.0 + , kiroku-store-migrations ^>=0.3.2.0 , optparse-applicative >=0.17 && <0.20 , pg-migrate ^>=1.1.0.0 , pg-migrate-cli ^>=1.1.0.0 @@ -404,7 +405,7 @@ , hasql >=1.10 && <1.11 , hspec >=2.10 && <2.12 , keiro-migrations - , kiroku-store-migrations ^>=0.3.0.0 + , kiroku-store-migrations ^>=0.3.2.0 , pg-migrate ^>=1.1.0.0 , pg-migrate-import-codd ^>=1.1.0.0 , pg-migrate-test-support ^>=1.1.0.0 @@ -437,8 +438,8 @@ , hasql-pool >=1.2 && <1.5 , hspec >=2.10 , keiro-migrations - , kiroku-store >=0.3 && <0.4 - , kiroku-store-migrations ^>=0.3.0.0 + , kiroku-store >=0.7 && <0.8 + , kiroku-store-migrations ^>=0.3.2.0 , lens >=5.2 && <5.4 , temporary , text >=2.0 && <2.2 diff --git a/migrations.native.lock b/migrations.native.lock --- a/migrations.native.lock +++ b/migrations.native.lock @@ -18,3 +18,13 @@ be7e510421f983ac2e446ae655d000b810b19cf2667f328dd7caf41a2101d0b6 0018.sql 7a0d0280d8d54f811c33b4bec525d78fe8f172de92b62e4745dcdd217c656209 0019-keiro-snapshots-state-shape-hash.sql c125e8c32a0848854d94bce1245f349cf7f479326a1bf540e9dba9e222b75a4f 0020-keiro-workflow-children-failure-reason.sql +fa59bb3930e541b55b92f22b4662b800f560a48c65af9edbb43089b33ba43cb2 0021-keiro-workflows-exact-discovery.sql +f06ed6b5a3de431f3f9de55f7776eb86549432317bc728ef848fd246db67c0ce 0022.sql +04cbd376eb9509cb202ec0caa43e6646957d10c241511829b225d072cfd92641 0023.sql +faad6bd740cda45378645be1d3fe0b1754c4f1e9ac6a02e9e935bb5f6e304f95 0024.sql +343f0852de6cddf85ab7a69bbba7d07396427877a78b0f7f97f9835d86c4deaf 0025.sql +cc167230c3765bd2de986cc1dc0db447d4f186a3e6b2dbbdac7ea2cabe6cb2b3 0026.sql +bb725945d567c9446e67db65535daf2017dd1b6facb3277c2b85e7429b4e95f7 0027.sql +524e90271704112722bbfe0fc54f1c8dcc06b1f9e2ce1821e54dc232d434104f 0028.sql +40f62a18adf0457fc6d35e880eca8be1922f22a2fd8c893c13d59537fe941719 0029.sql +0e0b847088d178290e77913224df82435b86c00062feb47c79351ed014a605a1 0030.sql diff --git a/migrations/0021-keiro-workflows-exact-discovery.sql b/migrations/0021-keiro-workflows-exact-discovery.sql new file mode 100644 --- /dev/null +++ b/migrations/0021-keiro-workflows-exact-discovery.sql @@ -0,0 +1,21 @@ +-- Exact discovery: the resume worker now returns a workflow only when its +-- instance row says it has progress to make — status 'running', or status +-- 'suspended' with a due wake_after hint. The status-only active index cannot +-- serve the second arm, so replace it with one that also carries wake_after. +-- The index predicate is unchanged, so the swap is transparent to every other +-- reader of the active set. +DROP INDEX IF EXISTS keiro.keiro_workflows_active_idx; +CREATE INDEX IF NOT EXISTS keiro_workflows_active_idx + ON keiro.keiro_workflows (status, wake_after) + WHERE status IN ('running', 'suspended'); + +-- Legacy suspensions predate exact discovery: before this change a 'suspended' +-- row was re-examined on every pass, so a workflow could be parked on a promise +-- that was already cancelled, or on a wake whose row-lifecycle transition was +-- never reflected in the instance row, and nothing would ever flip it again. +-- Return every suspended instance to the runnable pool once. The next pass +-- re-examines each and re-suspends it through the new suspend/wake arbitration, +-- which is what makes the row's status trustworthy from then on. The cost is one +-- extra replay per legacy suspended instance, once. +UPDATE keiro.keiro_workflows SET status = 'running', updated_at = now() +WHERE status = 'suspended'; diff --git a/migrations/0022.sql b/migrations/0022.sql new file mode 100644 --- /dev/null +++ b/migrations/0022.sql @@ -0,0 +1,80 @@ +-- projection rebuild groups + +CREATE TABLE IF NOT EXISTS keiro.keiro_projection_rebuild_groups ( + group_id TEXT PRIMARY KEY, + catalog_fingerprint TEXT NOT NULL, + status TEXT NOT NULL, + active_run_id TEXT, + requested_by TEXT, + request_reason TEXT, + started_at TIMESTAMPTZ, + completed_at TIMESTAMPTZ, + failed_at TIMESTAMPTZ, + failure_code TEXT, + failure_detail TEXT, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + CONSTRAINT keiro_projection_rebuild_groups_status_chk + CHECK (status IN ('live', 'rebuilding', 'failed')), + CONSTRAINT keiro_projection_rebuild_groups_active_run_chk + CHECK ( + (status = 'live' AND active_run_id IS NULL) + OR (status IN ('rebuilding', 'failed') AND active_run_id IS NOT NULL) + ) +); + +ALTER TABLE keiro.keiro_read_models + ADD COLUMN IF NOT EXISTS rebuild_group_id TEXT; + +-- Every pre-catalog read model becomes a deterministic singleton group. A +-- non-live legacy state stays fenced: rebuilding remains rebuilding, while +-- paused, abandoned, and unknown values become failed with the raw state in +-- failure_detail. The catalog registration path may later adopt a matching live +-- singleton into an explicitly named catalog group. +INSERT INTO keiro.keiro_projection_rebuild_groups ( + group_id, + catalog_fingerprint, + status, + active_run_id, + started_at, + failed_at, + failure_code, + failure_detail, + created_at, + updated_at +) +SELECT + '$legacy-read-model:' || name, + '$legacy-unmanaged', + CASE status + WHEN 'live' THEN 'live' + WHEN 'rebuilding' THEN 'rebuilding' + ELSE 'failed' + END, + CASE + WHEN status = 'live' THEN NULL + ELSE '$legacy-read-model:' || name + END, + CASE WHEN status = 'rebuilding' THEN updated_at ELSE NULL END, + CASE WHEN status NOT IN ('live', 'rebuilding') THEN updated_at ELSE NULL END, + CASE WHEN status NOT IN ('live', 'rebuilding') THEN 'legacy-read-model-state' ELSE NULL END, + CASE WHEN status NOT IN ('live', 'rebuilding') THEN status ELSE NULL END, + updated_at, + updated_at +FROM keiro.keiro_read_models +ON CONFLICT (group_id) DO NOTHING; + +UPDATE keiro.keiro_read_models +SET rebuild_group_id = '$legacy-read-model:' || name +WHERE rebuild_group_id IS NULL; + +ALTER TABLE keiro.keiro_read_models + ALTER COLUMN rebuild_group_id SET NOT NULL; + +ALTER TABLE keiro.keiro_read_models + ADD CONSTRAINT keiro_read_models_rebuild_group_fk + FOREIGN KEY (rebuild_group_id) + REFERENCES keiro.keiro_projection_rebuild_groups (group_id); + +CREATE INDEX IF NOT EXISTS keiro_read_models_rebuild_group_idx + ON keiro.keiro_read_models (rebuild_group_id); diff --git a/migrations/0023.sql b/migrations/0023.sql new file mode 100644 --- /dev/null +++ b/migrations/0023.sql @@ -0,0 +1,104 @@ +-- projection replay progress + +CREATE TABLE keiro.keiro_projection_rebuild_runs ( + run_id TEXT PRIMARY KEY, + group_id TEXT NOT NULL + REFERENCES keiro.keiro_projection_rebuild_groups (group_id), + catalog_fingerprint TEXT NOT NULL, + contract_fingerprint TEXT NOT NULL, + runner_format TEXT NOT NULL, + captured_head BIGINT NOT NULL, + page_size INTEGER NOT NULL, + status TEXT NOT NULL DEFAULT 'running', + started_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + failed_at TIMESTAMPTZ, + verified_at TIMESTAMPTZ, + promoted_at TIMESTAMPTZ, + failure_code TEXT, + failure_detail TEXT, + failure_source_id TEXT, + failure_projection_id TEXT, + failure_position BIGINT, + CONSTRAINT keiro_projection_rebuild_runs_status_chk + CHECK (status IN ('running', 'failed', 'verified', 'promoted')), + CONSTRAINT keiro_projection_rebuild_runs_head_chk + CHECK (captured_head >= 0), + CONSTRAINT keiro_projection_rebuild_runs_page_size_chk + CHECK (page_size > 0), + CONSTRAINT keiro_projection_rebuild_runs_failure_chk + CHECK ( + (status = 'failed' AND failure_code IS NOT NULL AND failed_at IS NOT NULL) + OR (status <> 'failed' AND failure_code IS NULL AND failed_at IS NULL) + ) +); + +CREATE TABLE keiro.keiro_projection_rebuild_sources ( + run_id TEXT NOT NULL + REFERENCES keiro.keiro_projection_rebuild_runs (run_id) ON DELETE CASCADE, + source_id TEXT NOT NULL, + source_scope TEXT NOT NULL, + category TEXT, + cursor_position BIGINT NOT NULL DEFAULT 0, + target_position BIGINT NOT NULL, + exhausted_through BIGINT, + event_count BIGINT NOT NULL DEFAULT 0, + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (run_id, source_id), + CONSTRAINT keiro_projection_rebuild_sources_scope_chk + CHECK ( + (source_scope = 'all' AND category IS NULL) + OR (source_scope = 'category' AND category IS NOT NULL) + ), + CONSTRAINT keiro_projection_rebuild_sources_cursor_chk + CHECK (cursor_position >= 0 AND cursor_position <= target_position), + CONSTRAINT keiro_projection_rebuild_sources_target_chk + CHECK (target_position >= 0), + CONSTRAINT keiro_projection_rebuild_sources_exhausted_chk + CHECK (exhausted_through IS NULL OR exhausted_through = target_position), + CONSTRAINT keiro_projection_rebuild_sources_count_chk + CHECK (event_count >= 0) +); + +CREATE TABLE keiro.keiro_projection_rebuild_adapters ( + run_id TEXT NOT NULL, + source_id TEXT NOT NULL, + projection_id TEXT NOT NULL, + adapter_order INTEGER NOT NULL, + evaluation_count BIGINT NOT NULL DEFAULT 0, + apply_count BIGINT NOT NULL DEFAULT 0, + completed_through BIGINT, + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (run_id, source_id, projection_id), + UNIQUE (run_id, adapter_order), + FOREIGN KEY (run_id, source_id) + REFERENCES keiro.keiro_projection_rebuild_sources (run_id, source_id) + ON DELETE CASCADE, + CONSTRAINT keiro_projection_rebuild_adapters_order_chk + CHECK (adapter_order >= 0), + CONSTRAINT keiro_projection_rebuild_adapters_counts_chk + CHECK (evaluation_count >= 0 AND apply_count >= 0 AND apply_count <= evaluation_count), + CONSTRAINT keiro_projection_rebuild_adapters_completed_chk + CHECK (completed_through IS NULL OR completed_through >= 0) +); + +CREATE TABLE keiro.keiro_projection_rebuild_verifications ( + run_id TEXT NOT NULL + REFERENCES keiro.keiro_projection_rebuild_runs (run_id) ON DELETE CASCADE, + verification_id TEXT NOT NULL, + verification_version TEXT NOT NULL, + status TEXT NOT NULL DEFAULT 'pending', + detail TEXT, + completed_at TIMESTAMPTZ, + PRIMARY KEY (run_id, verification_id), + CONSTRAINT keiro_projection_rebuild_verifications_status_chk + CHECK (status IN ('pending', 'passed', 'failed')), + CONSTRAINT keiro_projection_rebuild_verifications_completion_chk + CHECK ( + (status = 'pending' AND completed_at IS NULL) + OR (status IN ('passed', 'failed') AND completed_at IS NOT NULL) + ) +); + +CREATE INDEX keiro_projection_rebuild_runs_group_idx + ON keiro.keiro_projection_rebuild_runs (group_id, started_at DESC); diff --git a/migrations/0024.sql b/migrations/0024.sql new file mode 100644 --- /dev/null +++ b/migrations/0024.sql @@ -0,0 +1,13 @@ +-- projection rebuild slice fingerprints + +-- Group identity becomes the group's own catalog slice; runs carry the slice +-- they were begun under for epoch-consistency joins. The run catalog +-- fingerprint remains begin-stamped provenance. +ALTER TABLE keiro.keiro_projection_rebuild_groups + RENAME COLUMN catalog_fingerprint TO slice_fingerprint; + +ALTER TABLE keiro.keiro_projection_rebuild_runs + ADD COLUMN group_slice_fingerprint TEXT NOT NULL DEFAULT '$pre-canonical'; + +ALTER TABLE keiro.keiro_projection_rebuild_runs + ALTER COLUMN group_slice_fingerprint DROP DEFAULT; diff --git a/migrations/0025.sql b/migrations/0025.sql new file mode 100644 --- /dev/null +++ b/migrations/0025.sql @@ -0,0 +1,266 @@ +-- schema-versioned projection rebuild lifecycle + +-- The existing status column remains the lifecycle phase so a pre-feature +-- runtime sees every version-managed value as unknown and fails closed. +ALTER TABLE keiro.keiro_projection_rebuild_groups + DROP CONSTRAINT keiro_projection_rebuild_groups_status_chk, + DROP CONSTRAINT keiro_projection_rebuild_groups_active_run_chk, + ADD COLUMN serving_revision_id TEXT, + ADD COLUMN serving_epoch BIGINT NOT NULL DEFAULT 0, + ADD COLUMN reads_allowed BOOLEAN NOT NULL DEFAULT TRUE, + ADD COLUMN writes_allowed BOOLEAN NOT NULL DEFAULT TRUE; + +UPDATE keiro.keiro_projection_rebuild_groups +SET reads_allowed = (status = 'live'), + writes_allowed = (status = 'live'); + +ALTER TABLE keiro.keiro_projection_rebuild_groups + ADD CONSTRAINT keiro_projection_rebuild_groups_status_chk + CHECK ( + status IN ( + 'live', + 'rebuilding', + 'failed', + 'serving-versioned', + 'rebuilding-versioned', + 'cutover-versioned', + 'failed-versioned' + ) + ), + ADD CONSTRAINT keiro_projection_rebuild_groups_active_run_chk + CHECK ( + (status IN ('live', 'serving-versioned') AND active_run_id IS NULL) + OR ( + status IN ( + 'rebuilding', + 'failed', + 'rebuilding-versioned', + 'cutover-versioned', + 'failed-versioned' + ) + AND active_run_id IS NOT NULL + ) + ), + ADD CONSTRAINT keiro_projection_rebuild_groups_revision_phase_chk + CHECK ( + ( + status IN ('live', 'rebuilding', 'failed') + AND serving_revision_id IS NULL + AND serving_epoch = 0 + ) + OR ( + status IN ( + 'serving-versioned', + 'rebuilding-versioned', + 'cutover-versioned', + 'failed-versioned' + ) + AND serving_revision_id IS NOT NULL + AND serving_epoch >= 0 + ) + ), + ADD CONSTRAINT keiro_projection_rebuild_groups_availability_chk + CHECK ( + (status = 'live' AND reads_allowed AND writes_allowed) + OR (status IN ('rebuilding', 'failed') AND NOT reads_allowed AND NOT writes_allowed) + OR (status IN ('serving-versioned', 'rebuilding-versioned') AND reads_allowed AND writes_allowed) + OR (status IN ('cutover-versioned', 'failed-versioned') AND reads_allowed AND NOT writes_allowed) + ); + +CREATE TABLE keiro.keiro_projection_revisions ( + group_id TEXT NOT NULL + REFERENCES keiro.keiro_projection_rebuild_groups (group_id), + revision_id TEXT NOT NULL, + group_slice_fingerprint TEXT NOT NULL, + registered_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (group_id, revision_id), + UNIQUE (revision_id), + CONSTRAINT keiro_projection_revisions_identity_chk + CHECK ( + length(group_id) > 0 + AND length(revision_id) > 0 + AND length(group_slice_fingerprint) > 0 + ) +); + +ALTER TABLE keiro.keiro_projection_rebuild_groups + ADD CONSTRAINT keiro_projection_rebuild_groups_serving_revision_fk + FOREIGN KEY (group_id, serving_revision_id) + REFERENCES keiro.keiro_projection_revisions (group_id, revision_id); + +ALTER TABLE keiro.keiro_projection_rebuild_runs + DROP CONSTRAINT keiro_projection_rebuild_runs_status_chk, + DROP CONSTRAINT keiro_projection_rebuild_runs_failure_chk, + ADD COLUMN rebuild_mode TEXT NOT NULL DEFAULT 'offline', + ADD COLUMN candidate_revision_id TEXT, + ADD COLUMN cutover_threshold BIGINT, + ADD COLUMN cutover_lock_timeout_ms BIGINT, + ADD COLUMN history_retention_lease_id UUID, + ADD COLUMN history_retention_lease_owner TEXT, + ADD COLUMN history_retention_protected_through BIGINT, + ADD COLUMN history_retention_expires_at TIMESTAMPTZ, + ADD COLUMN history_retention_renewed_at TIMESTAMPTZ, + ADD COLUMN history_retention_released_at TIMESTAMPTZ, + ADD COLUMN abandoned_at TIMESTAMPTZ, + ADD CONSTRAINT keiro_projection_rebuild_runs_status_chk + CHECK (status IN ('running', 'failed', 'verified', 'cutover', 'promoted', 'abandoned')), + ADD CONSTRAINT keiro_projection_rebuild_runs_failure_chk + CHECK ( + (status = 'failed' AND failure_code IS NOT NULL AND failed_at IS NOT NULL) + OR (status <> 'failed' AND failure_code IS NULL AND failed_at IS NULL) + ), + ADD CONSTRAINT keiro_projection_rebuild_runs_abandoned_chk + CHECK ( + (status = 'abandoned' AND abandoned_at IS NOT NULL) + OR (status <> 'abandoned' AND abandoned_at IS NULL) + ), + ADD CONSTRAINT keiro_projection_rebuild_runs_mode_chk + CHECK ( + ( + rebuild_mode = 'offline' + AND candidate_revision_id IS NULL + AND cutover_threshold IS NULL + AND cutover_lock_timeout_ms IS NULL + AND history_retention_lease_id IS NULL + AND history_retention_lease_owner IS NULL + AND history_retention_protected_through IS NULL + AND history_retention_expires_at IS NULL + AND history_retention_renewed_at IS NULL + AND history_retention_released_at IS NULL + ) + OR ( + rebuild_mode = 'versioned' + AND candidate_revision_id IS NOT NULL + AND cutover_threshold IS NOT NULL + AND cutover_threshold >= 0 + AND cutover_lock_timeout_ms IS NOT NULL + AND cutover_lock_timeout_ms > 0 + AND history_retention_lease_id IS NOT NULL + AND history_retention_lease_owner IS NOT NULL + AND length(history_retention_lease_owner) > 0 + AND history_retention_protected_through IS NOT NULL + AND history_retention_protected_through >= 0 + AND history_retention_expires_at IS NOT NULL + AND history_retention_renewed_at IS NOT NULL + ) + ), + ADD CONSTRAINT keiro_projection_rebuild_runs_candidate_revision_fk + FOREIGN KEY (group_id, candidate_revision_id) + REFERENCES keiro.keiro_projection_revisions (group_id, revision_id); + +CREATE TABLE keiro.keiro_projection_target_generations ( + generation_id UUID PRIMARY KEY, + group_id TEXT NOT NULL, + target_id TEXT NOT NULL, + revision_id TEXT NOT NULL, + schema_name TEXT NOT NULL, + relation_name TEXT NOT NULL, + relation_oid BIGINT NOT NULL, + schema_version TEXT NOT NULL, + expected_shape_id TEXT NOT NULL, + observed_shape_fingerprint TEXT NOT NULL, + observed_catalog_snapshot TEXT NOT NULL, + lifecycle TEXT NOT NULL, + created_by_run_id TEXT + REFERENCES keiro.keiro_projection_rebuild_runs (run_id), + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + served_at TIMESTAMPTZ, + retired_at TIMESTAMPTZ, + dropped_at TIMESTAMPTZ, + UNIQUE (generation_id, created_by_run_id, target_id), + FOREIGN KEY (group_id, revision_id) + REFERENCES keiro.keiro_projection_revisions (group_id, revision_id), + CONSTRAINT keiro_projection_target_generations_identity_chk + CHECK ( + length(target_id) > 0 + AND length(schema_name) > 0 + AND length(relation_name) > 0 + AND relation_oid > 0 + AND length(schema_version) > 0 + AND length(expected_shape_id) > 0 + AND length(observed_shape_fingerprint) > 0 + ), + CONSTRAINT keiro_projection_target_generations_lifecycle_chk + CHECK (lifecycle IN ('staging', 'serving', 'retired', 'dropped')), + CONSTRAINT keiro_projection_target_generations_transition_evidence_chk + CHECK ( + ( + lifecycle = 'staging' + AND created_by_run_id IS NOT NULL + AND served_at IS NULL + AND retired_at IS NULL + AND dropped_at IS NULL + ) + OR ( + lifecycle = 'serving' + AND served_at IS NOT NULL + AND retired_at IS NULL + AND dropped_at IS NULL + ) + OR ( + lifecycle = 'retired' + AND served_at IS NOT NULL + AND retired_at IS NOT NULL + AND dropped_at IS NULL + ) + OR ( + lifecycle = 'dropped' + AND dropped_at IS NOT NULL + AND ( + (served_at IS NULL AND retired_at IS NULL) + OR (served_at IS NOT NULL AND retired_at IS NOT NULL) + ) + ) + ) +); + +CREATE UNIQUE INDEX keiro_projection_target_generations_serving_idx + ON keiro.keiro_projection_target_generations (group_id, target_id) + WHERE lifecycle = 'serving'; + +CREATE UNIQUE INDEX keiro_projection_target_generations_relation_idx + ON keiro.keiro_projection_target_generations (schema_name, relation_name) + WHERE lifecycle <> 'dropped'; + +CREATE TABLE keiro.keiro_projection_rebuild_run_targets ( + run_id TEXT NOT NULL + REFERENCES keiro.keiro_projection_rebuild_runs (run_id) ON DELETE CASCADE, + target_id TEXT NOT NULL, + target_mode TEXT NOT NULL, + candidate_generation_id UUID NOT NULL, + PRIMARY KEY (run_id, target_id), + UNIQUE (candidate_generation_id), + FOREIGN KEY (candidate_generation_id, run_id, target_id) + REFERENCES keiro.keiro_projection_target_generations + (generation_id, created_by_run_id, target_id), + CONSTRAINT keiro_projection_rebuild_run_targets_mode_chk + CHECK (target_mode IN ('application', 'clone')) +); + +CREATE TABLE keiro.keiro_projection_rebuild_promotion_objects ( + run_id TEXT NOT NULL, + target_id TEXT NOT NULL, + object_order INTEGER NOT NULL, + object_kind TEXT NOT NULL, + generation_name TEXT NOT NULL, + canonical_name TEXT NOT NULL, + PRIMARY KEY (run_id, target_id, object_order), + UNIQUE (run_id, target_id, generation_name), + UNIQUE (run_id, target_id, canonical_name), + FOREIGN KEY (run_id, target_id) + REFERENCES keiro.keiro_projection_rebuild_run_targets (run_id, target_id) + ON DELETE CASCADE, + CONSTRAINT keiro_projection_rebuild_promotion_objects_order_chk + CHECK (object_order >= 0), + CONSTRAINT keiro_projection_rebuild_promotion_objects_kind_chk + CHECK (object_kind IN ('index', 'constraint', 'owned-sequence')), + CONSTRAINT keiro_projection_rebuild_promotion_objects_names_chk + CHECK (length(generation_name) > 0 AND length(canonical_name) > 0) +); + +COMMENT ON TABLE keiro.keiro_projection_revisions IS + 'Registered executable/schema revision identities for projection rebuild groups.'; + +COMMENT ON TABLE keiro.keiro_projection_target_generations IS + 'Durable physical projection target identity and schema evidence across staging, serving, retirement, and drop.'; diff --git a/migrations/0026.sql b/migrations/0026.sql new file mode 100644 --- /dev/null +++ b/migrations/0026.sql @@ -0,0 +1,149 @@ +-- public projection-group status contract + +-- Persist the cursor authority selected by catalog registration. A row exists +-- for every group so an inline projection (append) is distinguishable from a +-- legacy or not-yet-reconciled group (unmanaged). Checkpoint subscription +-- names are private registration metadata; only their conservative floor is +-- published below. +CREATE TABLE keiro.keiro_projection_group_cursors ( + group_id TEXT PRIMARY KEY + REFERENCES keiro.keiro_projection_rebuild_groups (group_id) ON DELETE CASCADE, + position_basis TEXT NOT NULL, + subscription_names TEXT[] NOT NULL DEFAULT ARRAY[]::TEXT[], + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + CONSTRAINT keiro_projection_group_cursors_basis_chk + CHECK (position_basis IN ('append', 'checkpoint', 'unmanaged')), + CONSTRAINT keiro_projection_group_cursors_subscription_names_chk + CHECK ( + array_position(subscription_names, NULL) IS NULL + AND ( + (position_basis = 'checkpoint' AND cardinality(subscription_names) > 0) + OR ( + position_basis IN ('append', 'unmanaged') + AND cardinality(subscription_names) = 0 + ) + ) + ) +); + +INSERT INTO keiro.keiro_projection_group_cursors + (group_id, position_basis, subscription_names) +SELECT group_id, 'unmanaged', ARRAY[]::TEXT[] +FROM keiro.keiro_projection_rebuild_groups +ON CONFLICT (group_id) DO NOTHING; + +CREATE SCHEMA IF NOT EXISTS keiro_read; +REVOKE ALL ON SCHEMA keiro_read FROM PUBLIC; + +-- This owner-rights view is the only supported SQL surface for an external +-- projection reader. It intentionally depends only on Keiro-owned relations +-- and Kiroku's versioned owner relation, never Kiroku's private tables. +CREATE VIEW keiro_read.projection_group_status_v1 +WITH (security_barrier = true, security_invoker = false) AS +WITH checkpoint_floors AS ( + SELECT + cursors.group_id, + CASE + WHEN cursors.position_basis = 'checkpoint' + AND count(DISTINCT checkpoints.subscription_name) + = cardinality(cursors.subscription_names)::BIGINT + THEN min(checkpoints.checkpoint_position) + ELSE NULL + END AS serving_applied_position + FROM keiro.keiro_projection_group_cursors AS cursors + LEFT JOIN LATERAL unnest(cursors.subscription_names) + AS required_subscription(subscription_name) + ON true + LEFT JOIN kiroku.subscription_checkpoints_v1 AS checkpoints + ON checkpoints.subscription_name = required_subscription.subscription_name + GROUP BY cursors.group_id, cursors.position_basis, cursors.subscription_names +), candidate_progress AS ( + SELECT + sources.run_id, + min(sources.cursor_position) AS candidate_rebuild_position + FROM keiro.keiro_projection_rebuild_sources AS sources + GROUP BY sources.run_id +), query_models AS ( + SELECT + models.rebuild_group_id AS group_id, + array_agg(models.name ORDER BY models.name) AS query_models + FROM keiro.keiro_read_models AS models + GROUP BY models.rebuild_group_id +) +SELECT + groups.group_id, + groups.status AS lifecycle_phase, + groups.reads_allowed, + groups.writes_allowed, + groups.serving_revision_id, + groups.serving_epoch, + coalesce(cursors.position_basis, 'unmanaged') AS serving_position_basis, + CASE + WHEN groups.reads_allowed AND cursors.position_basis = 'checkpoint' + THEN checkpoint_floors.serving_applied_position + ELSE NULL + END AS serving_applied_position, + groups.active_run_id, + runs.candidate_revision_id, + candidate_progress.candidate_rebuild_position, + runs.captured_head AS candidate_rebuild_head, + coalesce(query_models.query_models, ARRAY[]::TEXT[]) AS query_models, + runs.started_at AS rebuild_started_at, + groups.completed_at AS last_promoted_at, + groups.failed_at, + groups.failure_code, + groups.failure_detail +FROM keiro.keiro_projection_rebuild_groups AS groups +LEFT JOIN keiro.keiro_projection_group_cursors AS cursors + ON cursors.group_id = groups.group_id +LEFT JOIN checkpoint_floors + ON checkpoint_floors.group_id = groups.group_id +LEFT JOIN keiro.keiro_projection_rebuild_runs AS runs + ON runs.run_id = groups.active_run_id +LEFT JOIN candidate_progress + ON candidate_progress.run_id = runs.run_id +LEFT JOIN query_models + ON query_models.group_id = groups.group_id; + +REVOKE ALL ON keiro_read.projection_group_status_v1 FROM PUBLIC; + +COMMENT ON SCHEMA keiro_read IS + 'Versioned, owner-rights read contracts for out-of-process Keiro consumers.'; +COMMENT ON VIEW keiro_read.projection_group_status_v1 IS + 'Frozen v1 projection-group status contract. Grant schema USAGE and view SELECT only.'; +COMMENT ON COLUMN keiro_read.projection_group_status_v1.group_id IS + 'Stable rebuild-group identifier and row key.'; +COMMENT ON COLUMN keiro_read.projection_group_status_v1.lifecycle_phase IS + 'Diagnostic lifecycle phase; readers must use reads_allowed for availability.'; +COMMENT ON COLUMN keiro_read.projection_group_status_v1.reads_allowed IS + 'Authoritative permission to serve reads from the current generation.'; +COMMENT ON COLUMN keiro_read.projection_group_status_v1.writes_allowed IS + 'Whether live projection writes may currently target the serving generation.'; +COMMENT ON COLUMN keiro_read.projection_group_status_v1.serving_revision_id IS + 'Served schema revision, or NULL for legacy/offline groups without versioned generations.'; +COMMENT ON COLUMN keiro_read.projection_group_status_v1.serving_epoch IS + 'Monotonic cache-invalidation epoch for changes to the served generation.'; +COMMENT ON COLUMN keiro_read.projection_group_status_v1.serving_position_basis IS + 'append, checkpoint, or unmanaged; determines how serving_applied_position is interpreted.'; +COMMENT ON COLUMN keiro_read.projection_group_status_v1.serving_applied_position IS + 'Conservative floor across every registered checkpoint subscription/member; NULL until all subscriptions are present, and always NULL for append or unmanaged groups.'; +COMMENT ON COLUMN keiro_read.projection_group_status_v1.active_run_id IS + 'Active rebuild run, including a retained failed run, or NULL when no rebuild is active.'; +COMMENT ON COLUMN keiro_read.projection_group_status_v1.candidate_revision_id IS + 'Versioned candidate revision for the active run, or NULL for offline/no active run.'; +COMMENT ON COLUMN keiro_read.projection_group_status_v1.candidate_rebuild_position IS + 'Conservative floor of persisted source cursors for the active run, or NULL before sources exist/no active run.'; +COMMENT ON COLUMN keiro_read.projection_group_status_v1.candidate_rebuild_head IS + 'Head captured by the active rebuild run, or NULL when no run is active.'; +COMMENT ON COLUMN keiro_read.projection_group_status_v1.query_models IS + 'Sorted query-model names owned by the group; empty when none are registered.'; +COMMENT ON COLUMN keiro_read.projection_group_status_v1.rebuild_started_at IS + 'Start time of the active rebuild run, or NULL when no run is active.'; +COMMENT ON COLUMN keiro_read.projection_group_status_v1.last_promoted_at IS + 'Most recent group promotion completion time, or NULL before the first promotion.'; +COMMENT ON COLUMN keiro_read.projection_group_status_v1.failed_at IS + 'Group failure time, or NULL outside a failed lifecycle.'; +COMMENT ON COLUMN keiro_read.projection_group_status_v1.failure_code IS + 'Stable machine-readable group failure code, or NULL outside a failed lifecycle.'; +COMMENT ON COLUMN keiro_read.projection_group_status_v1.failure_detail IS + 'Operator-facing group failure detail, or NULL outside a failed lifecycle.'; diff --git a/migrations/0027.sql b/migrations/0027.sql new file mode 100644 --- /dev/null +++ b/migrations/0027.sql @@ -0,0 +1,230 @@ +-- managed external read contracts + +CREATE TABLE keiro.keiro_external_read_contracts ( + contract_id TEXT NOT NULL, + contract_version INTEGER NOT NULL, + query_model_id TEXT NOT NULL, + group_id TEXT NOT NULL + REFERENCES keiro.keiro_projection_rebuild_groups (group_id), + public_function_name TEXT NOT NULL, + contract_kind TEXT NOT NULL, + argument_names TEXT[] NOT NULL DEFAULT ARRAY[]::TEXT[], + argument_types TEXT[] NOT NULL DEFAULT ARRAY[]::TEXT[], + result_type TEXT NOT NULL, + result_shape_hash TEXT NOT NULL, + serving_shape_hash TEXT, + compatible_revision_ids TEXT[] NOT NULL, + private_implementation TEXT, + private_implementation_version INTEGER, + immutable_signature_hash TEXT NOT NULL, + definition_hash TEXT NOT NULL, + surface_generation INTEGER NOT NULL, + state TEXT NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + retired_at TIMESTAMPTZ, + PRIMARY KEY (contract_id, contract_version), + UNIQUE (public_function_name, argument_types), + CONSTRAINT keiro_external_read_contracts_identity_chk + CHECK ( + length(contract_id) > 0 + AND contract_version > 0 + AND length(query_model_id) > 0 + AND length(group_id) > 0 + AND length(public_function_name) > 0 + AND length(result_type) > 0 + AND length(result_shape_hash) > 0 + AND length(immutable_signature_hash) > 0 + AND length(definition_hash) > 0 + AND surface_generation > 0 + AND cardinality(compatible_revision_ids) > 0 + AND array_position(compatible_revision_ids, NULL) IS NULL + AND array_position(argument_names, NULL) IS NULL + AND array_position(argument_types, NULL) IS NULL + AND cardinality(argument_names) = cardinality(argument_types) + ), + CONSTRAINT keiro_external_read_contracts_kind_chk + CHECK (contract_kind IN ('all-rows', 'keyed')), + CONSTRAINT keiro_external_read_contracts_implementation_chk + CHECK ( + ( + contract_kind = 'all-rows' + AND cardinality(argument_names) = 0 + AND private_implementation IS NULL + AND private_implementation_version IS NULL + ) + OR ( + contract_kind = 'keyed' + AND cardinality(argument_names) > 0 + AND private_implementation IS NOT NULL + AND length(private_implementation) > 0 + AND private_implementation_version > 0 + ) + ), + CONSTRAINT keiro_external_read_contracts_state_chk + CHECK (state IN ('candidate', 'active', 'pending-retirement', 'retired')), + CONSTRAINT keiro_external_read_contracts_retired_chk + CHECK ( + (state = 'retired' AND retired_at IS NOT NULL) + OR (state <> 'retired' AND retired_at IS NULL) + ) +); + +CREATE INDEX keiro_external_read_contracts_group_idx + ON keiro.keiro_external_read_contracts (group_id, state); + +CREATE TABLE keiro.keiro_managed_read_objects ( + object_schema TEXT NOT NULL, + object_name TEXT NOT NULL, + object_kind TEXT NOT NULL, + object_signature TEXT NOT NULL DEFAULT '', + contract_id TEXT, + contract_version INTEGER, + managed_by TEXT NOT NULL, + definition_hash TEXT NOT NULL, + surface_generation INTEGER NOT NULL, + state TEXT NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + retired_at TIMESTAMPTZ, + PRIMARY KEY (object_schema, object_name, object_kind, object_signature), + FOREIGN KEY (contract_id, contract_version) + REFERENCES keiro.keiro_external_read_contracts (contract_id, contract_version), + CONSTRAINT keiro_managed_read_objects_identity_chk + CHECK ( + length(object_schema) > 0 + AND length(object_name) > 0 + AND length(object_kind) > 0 + AND length(managed_by) > 0 + AND length(definition_hash) > 0 + AND surface_generation > 0 + AND ( + (contract_id IS NULL AND contract_version IS NULL) + OR (contract_id IS NOT NULL AND contract_version IS NOT NULL) + ) + ), + CONSTRAINT keiro_managed_read_objects_kind_chk + CHECK (object_kind IN ('guard-function', 'binding-view', 'wrapper-function', 'contract-type')), + CONSTRAINT keiro_managed_read_objects_owner_chk + CHECK (managed_by IN ('keiro', 'consumer')), + CONSTRAINT keiro_managed_read_objects_state_chk + CHECK (state IN ('active', 'pending-retirement', 'retired')), + CONSTRAINT keiro_managed_read_objects_retired_chk + CHECK ( + (state = 'retired' AND retired_at IS NOT NULL) + OR (state <> 'retired' AND retired_at IS NULL) + ) +); + +CREATE INDEX keiro_managed_read_objects_contract_idx + ON keiro.keiro_managed_read_objects (contract_id, contract_version); + +CREATE FUNCTION keiro_read.guard_external_read_v1( + requested_contract_id TEXT, + requested_contract_version INTEGER +) +RETURNS VOID +LANGUAGE plpgsql +SECURITY DEFINER +SET search_path = pg_catalog +AS $guard$ +DECLARE + contract_group_id TEXT; + contract_state TEXT; + contract_shape_hash TEXT; + contract_serving_shape_hash TEXT; + contract_compatible_revisions TEXT[]; + group_reads_allowed BOOLEAN; + group_serving_revision TEXT; +BEGIN + SELECT + contracts.group_id, + contracts.state, + contracts.result_shape_hash, + contracts.serving_shape_hash, + contracts.compatible_revision_ids + INTO + contract_group_id, + contract_state, + contract_shape_hash, + contract_serving_shape_hash, + contract_compatible_revisions + FROM keiro.keiro_external_read_contracts AS contracts + WHERE contracts.contract_id = requested_contract_id + AND contracts.contract_version = requested_contract_version; + + IF NOT FOUND OR contract_state IN ('pending-retirement', 'retired') THEN + RAISE EXCEPTION USING + ERRCODE = 'KR002', + MESSAGE = 'external read contract is unknown or retired', + DETAIL = pg_catalog.format( + 'contract=%s version=%s', + requested_contract_id, + requested_contract_version + ); + END IF; + + SELECT groups.reads_allowed, groups.serving_revision_id + INTO group_reads_allowed, group_serving_revision + FROM keiro.keiro_projection_rebuild_groups AS groups + WHERE groups.group_id = contract_group_id + FOR SHARE; + + IF NOT FOUND THEN + RAISE EXCEPTION USING + ERRCODE = 'KR002', + MESSAGE = 'external read contract group is unavailable', + DETAIL = pg_catalog.format( + 'group=%s contract=%s version=%s', + contract_group_id, + requested_contract_id, + requested_contract_version + ); + END IF; + + IF NOT group_reads_allowed THEN + RAISE EXCEPTION USING + ERRCODE = 'KR001', + MESSAGE = 'projection is temporarily unavailable', + DETAIL = pg_catalog.format( + 'group=%s contract=%s version=%s serving_revision=%s', + contract_group_id, + requested_contract_id, + requested_contract_version, + coalesce(group_serving_revision, '') + ); + END IF; + + IF contract_state <> 'active' + OR group_serving_revision IS NULL + OR NOT (group_serving_revision = ANY(contract_compatible_revisions)) + OR contract_serving_shape_hash IS DISTINCT FROM contract_shape_hash THEN + RAISE EXCEPTION USING + ERRCODE = 'KR003', + MESSAGE = 'external read contract is incompatible with the serving revision', + DETAIL = pg_catalog.format( + 'group=%s contract=%s version=%s serving_revision=%s', + contract_group_id, + requested_contract_id, + requested_contract_version, + coalesce(group_serving_revision, '') + ); + END IF; +END +$guard$; + +REVOKE ALL ON FUNCTION keiro_read.guard_external_read_v1(TEXT, INTEGER) FROM PUBLIC; + +INSERT INTO keiro.keiro_managed_read_objects + (object_schema, object_name, object_kind, object_signature, + managed_by, definition_hash, surface_generation, state) +VALUES + ('keiro_read', 'guard_external_read_v1', 'guard-function', 'text,integer', + 'keiro', 'migration-0027', 1, 'active'); + +COMMENT ON TABLE keiro.keiro_external_read_contracts IS + 'Private lifecycle and compatibility metadata for versioned external read functions.'; +COMMENT ON TABLE keiro.keiro_managed_read_objects IS + 'Object-level ownership, generation, and definition hashes for reconciled external read surfaces.'; +COMMENT ON FUNCTION keiro_read.guard_external_read_v1(TEXT, INTEGER) IS + 'Shared-lock availability and serving-revision compatibility guard for managed external reads.'; diff --git a/migrations/0028.sql b/migrations/0028.sql new file mode 100644 --- /dev/null +++ b/migrations/0028.sql @@ -0,0 +1,23 @@ +-- permit implementation-backed compatibility wrappers with stable zero-argument signatures + +ALTER TABLE keiro.keiro_external_read_contracts + DROP CONSTRAINT keiro_external_read_contracts_implementation_chk, + ADD CONSTRAINT keiro_external_read_contracts_implementation_chk + CHECK ( + ( + contract_kind = 'all-rows' + AND cardinality(argument_names) = 0 + AND private_implementation IS NULL + AND private_implementation_version IS NULL + ) + OR ( + contract_kind = 'keyed' + AND private_implementation IS NOT NULL + AND length(private_implementation) > 0 + AND private_implementation_version > 0 + ) + ); + +COMMENT ON CONSTRAINT keiro_external_read_contracts_implementation_chk + ON keiro.keiro_external_read_contracts IS + 'All-row contracts are Keiro-backed; keyed contracts delegate to a versioned consumer implementation and may retain a zero-argument public compatibility signature.'; diff --git a/migrations/0029.sql b/migrations/0029.sql new file mode 100644 --- /dev/null +++ b/migrations/0029.sql @@ -0,0 +1,218 @@ +-- Bound schema-versioned promotion attempts and stage async redelivery evidence. + +ALTER TABLE keiro.keiro_projection_rebuild_runs + ADD COLUMN promotion_dedup_limit BIGINT, + ADD COLUMN dedup_provisional_head BIGINT, + ADD COLUMN promotion_prepared_at TIMESTAMPTZ; + +UPDATE keiro.keiro_projection_rebuild_runs +SET promotion_dedup_limit = 1000000 +WHERE rebuild_mode = 'versioned'; + +ALTER TABLE keiro.keiro_projection_rebuild_runs + DROP CONSTRAINT keiro_projection_rebuild_runs_mode_chk, + ADD CONSTRAINT keiro_projection_rebuild_runs_mode_chk + CHECK ( + ( + rebuild_mode = 'offline' + AND candidate_revision_id IS NULL + AND cutover_threshold IS NULL + AND cutover_lock_timeout_ms IS NULL + AND promotion_dedup_limit IS NULL + AND dedup_provisional_head IS NULL + AND promotion_prepared_at IS NULL + AND history_retention_lease_id IS NULL + AND history_retention_lease_owner IS NULL + AND history_retention_protected_through IS NULL + AND history_retention_expires_at IS NULL + AND history_retention_renewed_at IS NULL + AND history_retention_released_at IS NULL + ) + OR ( + rebuild_mode = 'versioned' + AND candidate_revision_id IS NOT NULL + AND cutover_threshold IS NOT NULL + AND cutover_threshold >= 0 + AND cutover_lock_timeout_ms IS NOT NULL + AND cutover_lock_timeout_ms > 0 + AND promotion_dedup_limit IS NOT NULL + AND promotion_dedup_limit > 0 + AND (dedup_provisional_head IS NULL OR dedup_provisional_head >= 0) + AND history_retention_lease_id IS NOT NULL + AND history_retention_lease_owner IS NOT NULL + AND length(history_retention_lease_owner) > 0 + AND history_retention_protected_through IS NOT NULL + AND history_retention_protected_through >= 0 + AND history_retention_expires_at IS NOT NULL + AND history_retention_renewed_at IS NOT NULL + ) + ); + +CREATE TABLE keiro.keiro_projection_rebuild_dedup_stage ( + run_id TEXT NOT NULL + REFERENCES keiro.keiro_projection_rebuild_runs (run_id) ON DELETE CASCADE, + subscription_name TEXT NOT NULL, + projection_name TEXT NOT NULL, + event_id UUID NOT NULL, + global_position BIGINT NOT NULL, + PRIMARY KEY (run_id, projection_name, event_id), + CONSTRAINT keiro_projection_rebuild_dedup_stage_names_chk + CHECK (length(subscription_name) > 0 AND length(projection_name) > 0), + CONSTRAINT keiro_projection_rebuild_dedup_stage_position_chk + CHECK (global_position > 0) +); + +CREATE INDEX keiro_projection_rebuild_dedup_stage_run_position_idx + ON keiro.keiro_projection_rebuild_dedup_stage + (run_id, subscription_name, global_position); + +-- Each helper catches only lock/deadline failures. That keeps the transaction +-- usable long enough for Keiro to return a typed retryable lifecycle outcome; +-- every other SQL error remains an ordinary database failure. +CREATE FUNCTION keiro.keiro_try_projection_cutover_fence_v1( + requested_run_id TEXT, + requested_contract TEXT, + attempt_deadline TIMESTAMPTZ +) +RETURNS TEXT +LANGUAGE plpgsql +SECURITY INVOKER +SET search_path = pg_catalog, pg_temp +AS $$ +DECLARE + remaining_ms BIGINT; + fenced BOOLEAN; +BEGIN + remaining_ms := ceil(extract(epoch FROM (attempt_deadline - clock_timestamp())) * 1000); + IF remaining_ms <= 0 THEN + RETURN 'deadline-exceeded'; + END IF; + PERFORM set_config('lock_timeout', remaining_ms::text || 'ms', true); + PERFORM set_config('statement_timeout', remaining_ms::text || 'ms', true); + BEGIN + UPDATE keiro.keiro_projection_rebuild_groups AS groups + SET status = 'cutover-versioned', writes_allowed = FALSE, updated_at = now() + FROM keiro.keiro_projection_rebuild_runs AS runs + WHERE runs.run_id = requested_run_id + AND runs.contract_fingerprint = requested_contract + AND runs.status = 'running' + AND groups.group_id = runs.group_id + AND groups.status = 'rebuilding-versioned' + AND groups.active_run_id = runs.run_id + AND NOT EXISTS ( + SELECT 1 FROM keiro.keiro_projection_rebuild_sources AS sources + WHERE sources.run_id = runs.run_id + AND sources.exhausted_through IS DISTINCT FROM sources.target_position + ) + RETURNING TRUE INTO fenced; + EXCEPTION + WHEN lock_not_available OR query_canceled THEN + RETURN 'deadline-exceeded'; + END; + IF coalesce(fenced, FALSE) THEN + RETURN 'fenced'; + END IF; + RETURN 'not-ready'; +END; +$$; + +CREATE FUNCTION keiro.keiro_try_projection_promotion_lock_v1( + requested_run_id TEXT, + requested_contract TEXT, + attempt_deadline TIMESTAMPTZ +) +RETURNS TEXT +LANGUAGE plpgsql +SECURITY INVOKER +SET search_path = pg_catalog, pg_temp +AS $$ +DECLARE + remaining_ms BIGINT; + locked_run_id TEXT; +BEGIN + remaining_ms := ceil(extract(epoch FROM (attempt_deadline - clock_timestamp())) * 1000); + IF remaining_ms <= 0 THEN + RETURN 'deadline-exceeded'; + END IF; + PERFORM set_config('lock_timeout', remaining_ms::text || 'ms', true); + PERFORM set_config('statement_timeout', remaining_ms::text || 'ms', true); + BEGIN + SELECT runs.run_id INTO locked_run_id + FROM keiro.keiro_projection_rebuild_runs AS runs + JOIN keiro.keiro_projection_rebuild_groups AS groups + ON groups.group_id = runs.group_id + WHERE runs.run_id = requested_run_id + AND runs.contract_fingerprint = requested_contract + AND runs.status = 'cutover' + AND runs.promotion_prepared_at IS NOT NULL + AND groups.status = 'cutover-versioned' + AND groups.active_run_id = runs.run_id + AND NOT EXISTS ( + SELECT 1 FROM keiro.keiro_projection_rebuild_sources AS sources + WHERE sources.run_id = runs.run_id + AND sources.exhausted_through IS DISTINCT FROM sources.target_position + ) + FOR UPDATE OF runs, groups; + EXCEPTION + WHEN lock_not_available OR query_canceled THEN + RETURN 'deadline-exceeded'; + END; + IF locked_run_id IS NOT NULL THEN + RETURN 'locked'; + END IF; + RETURN 'not-ready'; +END; +$$; + +CREATE FUNCTION keiro.keiro_try_projection_relation_locks_v1( + relation_oids BIGINT[], + attempt_deadline TIMESTAMPTZ +) +RETURNS BOOLEAN +LANGUAGE plpgsql +SECURITY INVOKER +SET search_path = pg_catalog, pg_temp +AS $$ +DECLARE + remaining_ms BIGINT; + relation_list TEXT; + resolved_count BIGINT; +BEGIN + remaining_ms := ceil(extract(epoch FROM (attempt_deadline - clock_timestamp())) * 1000); + IF remaining_ms <= 0 THEN + RETURN FALSE; + END IF; + + SELECT + string_agg(format('%I.%I', namespaces.nspname, relations.relname), ', ' ORDER BY requested.ordinality), + count(*) + INTO relation_list, resolved_count + FROM unnest(relation_oids) WITH ORDINALITY AS requested(relation_oid, ordinality) + JOIN pg_class AS relations ON relations.oid = requested.relation_oid + JOIN pg_namespace AS namespaces ON namespaces.oid = relations.relnamespace; + + IF resolved_count IS DISTINCT FROM cardinality(relation_oids) OR relation_list IS NULL THEN + RAISE EXCEPTION 'one or more promotion relations no longer exist'; + END IF; + + -- Object resolution is part of the attempt. Recompute immediately before the + -- cumulative lock so time already spent under the group lock is not restored. + remaining_ms := ceil(extract(epoch FROM (attempt_deadline - clock_timestamp())) * 1000); + IF remaining_ms <= 0 THEN + RETURN FALSE; + END IF; + PERFORM set_config('lock_timeout', remaining_ms::text || 'ms', true); + PERFORM set_config('statement_timeout', remaining_ms::text || 'ms', true); + BEGIN + EXECUTE 'LOCK TABLE ' || relation_list || ' IN ACCESS EXCLUSIVE MODE'; + EXCEPTION + WHEN lock_not_available OR query_canceled THEN + RETURN FALSE; + END; + RETURN TRUE; +END; +$$; + +REVOKE ALL ON FUNCTION keiro.keiro_try_projection_cutover_fence_v1(TEXT, TEXT, TIMESTAMPTZ) FROM PUBLIC; +REVOKE ALL ON FUNCTION keiro.keiro_try_projection_promotion_lock_v1(TEXT, TEXT, TIMESTAMPTZ) FROM PUBLIC; +REVOKE ALL ON FUNCTION keiro.keiro_try_projection_relation_locks_v1(BIGINT[], TIMESTAMPTZ) FROM PUBLIC; diff --git a/migrations/0030.sql b/migrations/0030.sql new file mode 100644 --- /dev/null +++ b/migrations/0030.sql @@ -0,0 +1,145 @@ +-- make guarded external reads revalidate contract state after the lifecycle lock + +CREATE OR REPLACE FUNCTION keiro_read.guard_external_read_v1( + requested_contract_id TEXT, + requested_contract_version INTEGER +) +RETURNS VOID +LANGUAGE plpgsql +SECURITY DEFINER +SET search_path = pg_catalog +AS $guard$ +DECLARE + contract_group_id TEXT; + contract_state TEXT; + contract_shape_hash TEXT; + contract_serving_shape_hash TEXT; + contract_compatible_revisions TEXT[]; + group_reads_allowed BOOLEAN; + group_serving_revision TEXT; + group_serving_epoch_before BIGINT; + group_serving_epoch BIGINT; +BEGIN + -- Contract-to-group ownership is immutable. Read only that routing fact before + -- locking, then re-read every mutable contract fact after the group lock. + SELECT contracts.group_id, groups.serving_epoch + INTO contract_group_id, group_serving_epoch_before + FROM keiro.keiro_external_read_contracts AS contracts + JOIN keiro.keiro_projection_rebuild_groups AS groups + ON groups.group_id = contracts.group_id + WHERE contracts.contract_id = requested_contract_id + AND contracts.contract_version = requested_contract_version; + + IF NOT FOUND THEN + RAISE EXCEPTION USING + ERRCODE = 'KR002', + MESSAGE = 'external read contract is unknown or retired', + DETAIL = pg_catalog.format( + 'contract=%s version=%s', + requested_contract_id, + requested_contract_version + ); + END IF; + + SELECT groups.reads_allowed, groups.serving_revision_id, groups.serving_epoch + INTO group_reads_allowed, group_serving_revision, group_serving_epoch + FROM keiro.keiro_projection_rebuild_groups AS groups + WHERE groups.group_id = contract_group_id + FOR SHARE; + + IF NOT FOUND THEN + RAISE EXCEPTION USING + ERRCODE = 'KR002', + MESSAGE = 'external read contract group is unavailable', + DETAIL = pg_catalog.format( + 'group=%s contract=%s version=%s', + contract_group_id, + requested_contract_id, + requested_contract_version + ); + END IF; + + -- A promotion updates the group and reconciles this row in one transaction. + -- The locking re-read observes that committed row after waiting for promotion, + -- rather than retaining the statement's pre-promotion metadata snapshot. + SELECT + contracts.state, + contracts.result_shape_hash, + contracts.serving_shape_hash, + contracts.compatible_revision_ids + INTO + contract_state, + contract_shape_hash, + contract_serving_shape_hash, + contract_compatible_revisions + FROM keiro.keiro_external_read_contracts AS contracts + WHERE contracts.contract_id = requested_contract_id + AND contracts.contract_version = requested_contract_version + AND contracts.group_id = contract_group_id + FOR SHARE; + + IF NOT FOUND OR contract_state IN ('pending-retirement', 'retired') THEN + RAISE EXCEPTION USING + ERRCODE = 'KR002', + MESSAGE = 'external read contract is unknown or retired', + DETAIL = pg_catalog.format( + 'contract=%s version=%s', + requested_contract_id, + requested_contract_version + ); + END IF; + + -- PostgreSQL fixes the caller statement snapshot before entering this + -- function. A locking read can observe the newly committed group row after + -- waiting, but the later projection query cannot safely adopt every catalog + -- and relation change in that old snapshot. Refuse the crossed epoch so the + -- caller retries as a new statement instead of returning stale or empty data. + IF group_serving_epoch IS DISTINCT FROM group_serving_epoch_before THEN + RAISE EXCEPTION USING + ERRCODE = 'KR001', + MESSAGE = 'projection generation changed while the external read waited', + DETAIL = pg_catalog.format( + 'group=%s contract=%s version=%s serving_epoch_before=%s serving_epoch=%s', + contract_group_id, + requested_contract_id, + requested_contract_version, + group_serving_epoch_before, + group_serving_epoch + ); + END IF; + + IF NOT group_reads_allowed THEN + RAISE EXCEPTION USING + ERRCODE = 'KR001', + MESSAGE = 'projection is temporarily unavailable', + DETAIL = pg_catalog.format( + 'group=%s contract=%s version=%s serving_revision=%s', + contract_group_id, + requested_contract_id, + requested_contract_version, + coalesce(group_serving_revision, '') + ); + END IF; + + IF contract_state <> 'active' + OR group_serving_revision IS NULL + OR NOT (group_serving_revision = ANY(contract_compatible_revisions)) + OR contract_serving_shape_hash IS DISTINCT FROM contract_shape_hash THEN + RAISE EXCEPTION USING + ERRCODE = 'KR003', + MESSAGE = 'external read contract is incompatible with the serving revision', + DETAIL = pg_catalog.format( + 'group=%s contract=%s version=%s serving_revision=%s', + contract_group_id, + requested_contract_id, + requested_contract_version, + coalesce(group_serving_revision, '') + ); + END IF; +END +$guard$; + +REVOKE ALL ON FUNCTION keiro_read.guard_external_read_v1(TEXT, INTEGER) FROM PUBLIC; + +COMMENT ON FUNCTION keiro_read.guard_external_read_v1(TEXT, INTEGER) IS + 'Shared-lock availability, post-lock contract revalidation, and crossed-epoch retry fencing for managed external reads.'; diff --git a/migrations/manifest b/migrations/manifest --- a/migrations/manifest +++ b/migrations/manifest @@ -18,3 +18,13 @@ 0018.sql 0019-keiro-snapshots-state-shape-hash.sql 0020-keiro-workflow-children-failure-reason.sql +0021-keiro-workflows-exact-discovery.sql +0022.sql +0023.sql +0024.sql +0025.sql +0026.sql +0027.sql +0028.sql +0029.sql +0030.sql diff --git a/src/Keiro/Migrations/SchemaCheck.hs b/src/Keiro/Migrations/SchemaCheck.hs --- a/src/Keiro/Migrations/SchemaCheck.hs +++ b/src/Keiro/Migrations/SchemaCheck.hs @@ -101,17 +101,19 @@ <> actualDefinition <> ")" --- | Read a sorted canonical snapshot of tables, columns, constraints, and indexes. +-- | Read a sorted canonical snapshot of tables, columns, constraints, indexes, +-- and ordered view column signatures in one schema. snapshotSchema :: Text -> Session Text snapshotSchema schema = Text.unlines <$> Session.statement schema schemaSnapshotStatement --- | PostgreSQL 18 snapshot generated from the complete embedded migration plan. +-- | PostgreSQL 18 snapshot regenerated through migration 0030. expectedSchemaSnapshot :: Text expectedSchemaSnapshot = $(embedTextFile "expected-schema/native/keiro-v18.txt") --- | Compare the live @keiro@ schema with the embedded PostgreSQL 18 snapshot. +-- | Compare the live @keiro@ and @keiro_read@ schemas with the embedded +-- PostgreSQL 18 snapshot. verifyExpectedSchema :: Settings.Settings -> IO (Either MigrationError [SchemaDrift]) @@ -135,7 +137,10 @@ serverVersionNumber <- Session.statement () serverVersionStatement let majorVersion = fromIntegral serverVersionNumber `div` 10000 if majorVersion == (18 :: Int) - then Right <$> snapshotSchema "keiro" + then do + privateSchema <- snapshotSchema "keiro" + publicSchema <- snapshotSchema "keiro_read" + pure (Right (privateSchema <> publicSchema)) else pure (Left (UnsupportedPostgresVersion majorVersion)) snapshotObjects :: Text -> Map Text (Text, Text) @@ -212,6 +217,23 @@ JOIN pg_namespace n ON n.oid = ct.relnamespace WHERE n.nspname = $1 AND configured.search_path = 'pg_catalog' + UNION ALL + SELECT 'view' || E'\t' || n.nspname || '.' || c.relname || E'\t' + || 'columns=' + || string_agg( + a.attnum::text || ':' || a.attname || ' ' + || format_type(a.atttypid, a.atttypmod), + ',' ORDER BY a.attnum + ) AS line + FROM pg_class c + JOIN pg_namespace n ON n.oid = c.relnamespace + JOIN pg_attribute a ON a.attrelid = c.oid + WHERE n.nspname = $1 + AND c.relkind = 'v' + AND a.attnum > 0 + AND NOT a.attisdropped + AND configured.search_path = 'pg_catalog' + GROUP BY n.nspname, c.relname ) snapshot ORDER BY line COLLATE "C" """ diff --git a/test/Lint.hs b/test/Lint.hs --- a/test/Lint.hs +++ b/test/Lint.hs @@ -20,6 +20,7 @@ -- "-- pg-migrate: no-transaction" leading comment, which review gates. data LintConfig = LintConfig { requiredQualifier :: Text, + additionalQualifiers :: [Text], exemptFiles :: [FilePath] } deriving stock (Eq, Show) @@ -38,21 +39,34 @@ body = Text.Encoding.decodeUtf8With lenientDecode bytes statements = map Text.strip . Text.splitOn ";" $ stripCommentLines body - requiredLower = Text.toCaseFold (requiredQualifier config) + allowedQualifiers = + Text.toCaseFold <$> (requiredQualifier config : additionalQualifiers config) searchPathViolation file body = [ "migration body mentions search_path: " <> Text.pack file - | "search_path" `Text.isInfixOf` Text.toCaseFold (stripCommentLines body) + | let normalized = Text.toCaseFold (stripCommentLines body), + "search_path" `Text.isInfixOf` normalized, + not (onlyHardenedSearchPathClauses normalized) ] + onlyHardenedSearchPathClauses body = + "search_path" `Text.isInfixOf` body + && not + ( "search_path" + `Text.isInfixOf` Text.replace "set search_path = pg_catalog" "" body + ) + statementViolations file statement = case statementTarget statement of Nothing -> [] Just target - | requiredLower `Text.isPrefixOf` Text.toCaseFold (cleanTarget target) -> [] + | any + (`Text.isPrefixOf` Text.toCaseFold (cleanTarget target)) + allowedQualifiers -> + [] | otherwise -> [ "migration DDL target is not qualified with " - <> requiredQualifier config + <> Text.intercalate " or " (requiredQualifier config : additionalQualifiers config) <> " in " <> Text.pack file <> ": " @@ -66,6 +80,10 @@ targetAfter ["create", "table"] wordsOriginal | lower `startsWithWords` ["alter", "table"] = targetAfter ["alter", "table"] wordsOriginal + | lower `startsWithWords` ["create", "view"] = + targetAfter ["create", "view"] wordsOriginal + | lower `startsWithWords` ["create", "or", "replace", "view"] = + targetAfter ["create", "or", "replace", "view"] wordsOriginal | lower `startsWithWords` ["drop", "index"] = targetAfter ["drop", "index"] wordsOriginal | lower `startsWithWords` ["create", "index"] = diff --git a/test/Main.hs b/test/Main.hs --- a/test/Main.hs +++ b/test/Main.hs @@ -9,9 +9,10 @@ import Data.ByteString qualified as ByteString import Data.Either (isLeft) import Data.Foldable (toList) -import Data.Int (Int64) +import Data.Int (Int32, Int64) import Data.List (findIndex, sort, (\\)) import Data.List.NonEmpty (NonEmpty (..)) +import Data.List.NonEmpty qualified as NonEmpty import Data.Map.Strict qualified as Map import Data.Set qualified as Set import Data.Text (Text) @@ -53,7 +54,7 @@ main :: IO () main = hspec $ do describe "native Keiro migration definition" $ do - it "tracks twenty native files in manifest order" $ do + it "tracks thirty native files in manifest order" $ do directory <- findMigrationsDirectory manifest <- Text.lines <$> Text.IO.readFile (directory "manifest") manifest `shouldBe` Text.pack <$> nativeMigrationFiles @@ -66,7 +67,7 @@ bytes <- ByteString.readFile (directory nativeName) lookup legacyName lockEntries `shouldBe` Just (checksumText bytes) - it "builds component keiro with dependency kiroku and twenty migrations" $ do + it "builds component keiro with dependency kiroku and thirty migrations" $ do plan <- requirePlan let PlanDescription components = planDescription plan case toList components of @@ -75,11 +76,11 @@ ] -> do componentNameText kirokuName `shouldBe` "kiroku" kirokuDependencies `shouldBe` mempty - length kirokuEntries `shouldBe` 8 + length kirokuEntries `shouldBe` 10 componentNameText keiroName `shouldBe` "keiro" dependencyName <- requireRight (componentName "kiroku") keiroDependencies `shouldBe` Set.singleton dependencyName - length keiroEntries `shouldBe` 20 + length keiroEntries `shouldBe` 30 actual -> expectationFailure ("unexpected plan description: " <> show actual) validateHistoryMappingTargets plan frameworkCoddHistoryMappings `shouldBe` Right () @@ -125,7 +126,12 @@ ) describe "migration body lint" $ do - let config = LintConfig {requiredQualifier = "keiro.", exemptFiles = []} + let config = + LintConfig + { requiredQualifier = "keiro.", + additionalQualifiers = ["keiro_read."], + exemptFiles = [] + } it "flags an unqualified DDL target" $ do let violations = @@ -142,13 +148,28 @@ [("9999-fixture.sql", "SET search_path TO keiro;")] `shouldSatisfy` (not . null) + it "accepts a fixed pg_catalog search path on a security-definer function" $ do + lintViolations + config + [ ( "9999-fixture.sql", + "CREATE FUNCTION keiro_read.safe_v1() RETURNS void LANGUAGE sql SECURITY DEFINER SET search_path = pg_catalog AS 'SELECT';" + ) + ] + `shouldBe` [] + + it "accepts a versioned view in the public read schema" $ do + lintViolations + config + [("9999-fixture.sql", "CREATE VIEW keiro_read.status_v1 AS SELECT 1;")] + `shouldBe` [] + it "ignores comment-only mentions" $ do lintViolations config [("9999-fixture.sql", "-- Never set search_path in a migration.\nSELECT 1;")] `shouldBe` [] - it "passes all 20 embedded native bodies" $ do + it "passes all 30 embedded native bodies" $ do lintViolations config (toList embeddedMigrationEntries) `shouldBe` [] describe "startup handshake" $ do @@ -162,7 +183,7 @@ plan >>= requireRight Keiro.pendingMigrations handshake `shouldBe` planMigrationIds plan - length (Keiro.pendingMigrations handshake) `shouldBe` 28 + length (Keiro.pendingMigrations handshake) `shouldBe` 40 Keiro.ledgerIssues handshake `shouldBe` [] handshakePassed handshake `shouldBe` False @@ -187,8 +208,8 @@ _ <- runMigrationPlan defaultRunOptions settings kirokuOnly >>= requireRight handshake <- missingMigrations defaultRunOptions provider plan >>= requireRight - Keiro.pendingMigrations handshake `shouldBe` drop 8 (planMigrationIds plan) - length (Keiro.pendingMigrations handshake) `shouldBe` 20 + Keiro.pendingMigrations handshake `shouldBe` drop 10 (planMigrationIds plan) + length (Keiro.pendingMigrations handshake) `shouldBe` 30 Keiro.ledgerIssues handshake `shouldBe` [] handshakePassed handshake `shouldBe` False @@ -220,7 +241,9 @@ snapshotPath <- findNativeExpectedSchema regenerate <- maybe False (const True) <$> lookupEnv "KEIRO_REGENERATE_EXPECTED_SCHEMA" result <- withMigratedDatabase plan $ \connection -> do - actual <- useSession connection (snapshotSchema "keiro") + privateSchema <- useSession connection (snapshotSchema "keiro") + publicSchema <- useSession connection (snapshotSchema "keiro_read") + let actual = privateSchema <> publicSchema if regenerate then do Text.IO.writeFile snapshotPath actual @@ -244,6 +267,7 @@ ( Session.script """ DROP INDEX keiro.keiro_outbox_pending_idx; + DROP VIEW keiro_read.projection_group_status_v1; ALTER TABLE keiro.keiro_outbox ALTER COLUMN correlation_id TYPE character varying(64) USING correlation_id::text; @@ -257,6 +281,9 @@ rendered `shouldSatisfy` any (Text.isInfixOf "keiro_outbox.correlation_id") + rendered + `shouldSatisfy` any + (Text.isInfixOf "keiro_read.projection_group_status_v1") describe "fresh native databases" $ do it "applies Kiroku then Keiro, verifies strictly, and is repeatable" $ do @@ -265,16 +292,51 @@ assertSchema connection let provider = providerFor connection rerun <- runMigrationPlanWith defaultRunOptions provider plan >>= requireRight - reportOutcomes rerun `shouldBe` replicate 28 AlreadyApplied + reportOutcomes rerun `shouldBe` replicate 40 AlreadyApplied verified <- verifyMigrationPlanWith defaultRunOptions provider plan >>= requireRight case verified of VerificationReport verificationIssues applied pending unknown -> do verificationIssues `shouldBe` [] - length applied `shouldBe` 28 + length applied `shouldBe` 40 pending `shouldBe` [] unknown `shouldBe` [] either (expectationFailure . show) pure result + it "publishes the frozen status shape through an isolated reader grant" $ do + plan <- requirePlan + result <- withMigratedDatabase plan $ \connection -> do + useSession connection (Session.script projectionStatusFixtureSql) + columns <- + useSession connection (Session.statement () projectionStatusColumnsStatement) + columns + `shouldBe` [ ("group_id", "text"), + ("lifecycle_phase", "text"), + ("reads_allowed", "boolean"), + ("writes_allowed", "boolean"), + ("serving_revision_id", "text"), + ("serving_epoch", "bigint"), + ("serving_position_basis", "text"), + ("serving_applied_position", "bigint"), + ("active_run_id", "text"), + ("candidate_revision_id", "text"), + ("candidate_rebuild_position", "bigint"), + ("candidate_rebuild_head", "bigint"), + ("query_models", "text[]"), + ("rebuild_started_at", "timestamp with time zone"), + ("last_promoted_at", "timestamp with time zone"), + ("failed_at", "timestamp with time zone"), + ("failure_code", "text"), + ("failure_detail", "text") + ] + contractHealthy <- + useSession connection (Session.statement () projectionStatusContractStatement) + contractHealthy `shouldBe` True + withProjectionReaderRole connection $ do + readerHealthy <- + useSession connection (Session.statement () projectionReaderFactsStatement) + readerHealthy `shouldBe` True + either (expectationFailure . show) pure result + it "serializes concurrent composed applies" $ do plan <- requirePlan withKeiroPg $ \database -> do @@ -284,8 +346,134 @@ (runMigrationPlan defaultRunOptions settings plan >>= requireRight) (runMigrationPlan defaultRunOptions settings plan >>= requireRight) sort [reportOutcomes first, reportOutcomes second] - `shouldBe` sort [replicate 28 AppliedNow, replicate 28 AlreadyApplied] + `shouldBe` sort [replicate 40 AppliedNow, replicate 40 AlreadyApplied] + it "upgrades singleton read-model rows into deterministic rebuild groups" $ do + fullPlan <- requirePlan + kiroku <- requireRight Kiroku.kirokuMigrations + priorKeiro <- + requireRight + ( migrationComponentFromEmbeddedSql + "keiro" + (Set.singleton "kiroku") + (NonEmpty.fromList (take 21 (toList embeddedMigrationEntries))) + ) + priorPlan <- requireRight (frameworkMigrationPlan kiroku priorKeiro) + withKeiroPg $ \database -> do + let settings = Pg.connectionSettings database + _ <- runMigrationPlan defaultRunOptions settings priorPlan >>= requireRight + withConnection settings $ \connection -> + useSession connection (Session.script legacyReadModelFixtureSql) + report <- runMigrationPlan defaultRunOptions settings fullPlan >>= requireRight + Prelude.drop 31 (reportOutcomes report) `shouldBe` replicate 9 AppliedNow + withConnection settings $ \connection -> do + rows <- useSession connection (Session.statement () legacyGroupUpgradeStatement) + rows + `shouldBe` [ ("legacy-abandoned", 9, "shape-abandoned", "abandoned", "$legacy-read-model:legacy-abandoned", "failed", Just "$legacy-read-model:legacy-abandoned", Just "abandoned"), + ("legacy-live", 7, "shape-live", "live", "$legacy-read-model:legacy-live", "live", Nothing, Nothing), + ("legacy-rebuilding", 8, "shape-rebuilding", "rebuilding", "$legacy-read-model:legacy-rebuilding", "rebuilding", Just "$legacy-read-model:legacy-rebuilding", Nothing) + ] + + it "0024 stamps in-flight rebuild runs with the pre-canonical sentinel" $ do + fullPlan <- requirePlan + kiroku <- requireRight Kiroku.kirokuMigrations + priorKeiro <- + requireRight + ( migrationComponentFromEmbeddedSql + "keiro" + (Set.singleton "kiroku") + (NonEmpty.fromList (take 23 (toList embeddedMigrationEntries))) + ) + priorPlan <- requireRight (frameworkMigrationPlan kiroku priorKeiro) + withKeiroPg $ \database -> do + let settings = Pg.connectionSettings database + _ <- runMigrationPlan defaultRunOptions settings priorPlan >>= requireRight + withConnection settings $ \connection -> + useSession connection (Session.script preCanonicalRebuildFixtureSql) + report <- runMigrationPlan defaultRunOptions settings fullPlan >>= requireRight + Prelude.drop 33 (reportOutcomes report) `shouldBe` replicate 7 AppliedNow + withConnection settings $ \connection -> do + rows <- useSession connection (Session.statement () preCanonicalRebuildShapeStatement) + rows + `shouldBe` [ ("upgrade-failed", Text.replicate 64 "b", "upgrade-run-failed", "$pre-canonical"), + ("upgrade-rebuilding", Text.replicate 64 "a", "upgrade-run-live", "$pre-canonical") + ] + + it "0026 gives pre-existing groups explicit fail-safe cursor authority" $ do + fullPlan <- requirePlan + kiroku <- requireRight Kiroku.kirokuMigrations + priorKeiro <- + requireRight + ( migrationComponentFromEmbeddedSql + "keiro" + (Set.singleton "kiroku") + (NonEmpty.fromList (take 25 (toList embeddedMigrationEntries))) + ) + priorPlan <- requireRight (frameworkMigrationPlan kiroku priorKeiro) + withKeiroPg $ \database -> do + let settings = Pg.connectionSettings database + _ <- runMigrationPlan defaultRunOptions settings priorPlan >>= requireRight + withConnection settings $ \connection -> + useSession connection (Session.script preStatusContractFixtureSql) + report <- runMigrationPlan defaultRunOptions settings fullPlan >>= requireRight + Prelude.drop 35 (reportOutcomes report) `shouldBe` replicate 5 AppliedNow + withConnection settings $ \connection -> do + facts <- useSession connection (Session.statement () preStatusContractFactsStatement) + facts `shouldBe` ("unmanaged", 0, "unmanaged", True, True) + + it "0030 upgrades the guard in place with post-lock epoch fencing" $ do + fullPlan <- requirePlan + kiroku <- requireRight Kiroku.kirokuMigrations + priorKeiro <- + requireRight + ( migrationComponentFromEmbeddedSql + "keiro" + (Set.singleton "kiroku") + (NonEmpty.fromList (take 29 (toList embeddedMigrationEntries))) + ) + priorPlan <- requireRight (frameworkMigrationPlan kiroku priorKeiro) + withKeiroPg $ \database -> do + let settings = Pg.connectionSettings database + _ <- runMigrationPlan defaultRunOptions settings priorPlan >>= requireRight + beforeOid <- + withConnection settings $ \connection -> + useSession connection (Session.statement () guardFunctionOidStatement) + report <- runMigrationPlan defaultRunOptions settings fullPlan >>= requireRight + Prelude.drop 39 (reportOutcomes report) `shouldBe` [AppliedNow] + withConnection settings $ \connection -> do + (afterOid, epochFenced, publicRevoked) <- + useSession connection (Session.statement () upgradedGuardFactsStatement) + afterOid `shouldBe` beforeOid + epochFenced `shouldBe` True + publicRevoked `shouldBe` True + + it "enforces replay source, adapter, and verification membership constraints" $ do + plan <- requirePlan + result <- withMigratedDatabase plan $ \connection -> do + useSession connection (Session.script replayProgressFixtureSql) + invalidScope <- + Connection.use + connection + ( Session.script + "INSERT INTO keiro.keiro_projection_rebuild_sources (run_id, source_id, source_scope, target_position) VALUES ('constraint-run', 'bad-scope', 'category', 0)" + ) + invalidScope `shouldSatisfy` isLeft + missingSource <- + Connection.use + connection + ( Session.script + "INSERT INTO keiro.keiro_projection_rebuild_adapters (run_id, source_id, projection_id, adapter_order) VALUES ('constraint-run', 'missing', 'projection', 0)" + ) + missingSource `shouldSatisfy` isLeft + duplicateVerification <- + Connection.use + connection + ( Session.script + "INSERT INTO keiro.keiro_projection_rebuild_verifications (run_id, verification_id, verification_version) VALUES ('constraint-run', 'verify', 'v2')" + ) + duplicateVerification `shouldSatisfy` isLeft + either (expectationFailure . show) pure result + describe "codd-ledger preflight" $ do it "blocks a current codd ledger before native history exists" $ assertBlockedCoddPreflight "codd" @@ -418,9 +606,9 @@ up <- runMigrationPlan defaultRunOptions settings plan >>= requireRight reportOutcomes up `shouldBe` replicate 7 AlreadyApplied - <> [AppliedNow] + <> replicate 3 AppliedNow <> replicate 16 AlreadyApplied - <> replicate 4 AppliedNow + <> replicate 14 AppliedNow verifiedAfterUp <- verifyMigrationPlan defaultRunOptions settings plan >>= requireRight @@ -429,6 +617,326 @@ verificationIssues `shouldBe` [] withConnection settings assertSchema +legacyReadModelFixtureSql :: Text +legacyReadModelFixtureSql = + """ + INSERT INTO keiro.keiro_read_models + (name, version, shape_hash, last_built_at, status, updated_at) + VALUES + ('legacy-live', 7, 'shape-live', '2026-01-01 00:00:00+00', 'live', '2026-01-01 00:00:00+00'), + ('legacy-rebuilding', 8, 'shape-rebuilding', '2026-01-01 00:00:00+00', 'rebuilding', '2026-01-02 00:00:00+00'), + ('legacy-abandoned', 9, 'shape-abandoned', '2026-01-01 00:00:00+00', 'abandoned', '2026-01-03 00:00:00+00'); + """ + +projectionStatusFixtureSql :: Text +projectionStatusFixtureSql = + """ + INSERT INTO keiro.keiro_projection_rebuild_groups + (group_id, slice_fingerprint, status, active_run_id, + reads_allowed, writes_allowed, started_at) + VALUES + ('status-contract-group', 'slice-v1:status-contract', 'rebuilding', + 'status-contract-run', false, false, '2026-08-13 12:00:00+00'); + + INSERT INTO keiro.keiro_projection_group_cursors + (group_id, position_basis, subscription_names) + VALUES ('status-contract-group', 'append', ARRAY[]::TEXT[]); + + INSERT INTO keiro.keiro_projection_rebuild_runs + (run_id, group_id, catalog_fingerprint, group_slice_fingerprint, + contract_fingerprint, runner_format, captured_head, page_size) + VALUES + ('status-contract-run', 'status-contract-group', 'catalog-status-contract', + 'slice-v1:status-contract', 'contract-v2:status-contract', + 'keiro/projection-replay/v2', 10, 100); + + INSERT INTO keiro.keiro_projection_rebuild_sources + (run_id, source_id, source_scope, category, cursor_position, target_position) + VALUES + ('status-contract-run', 'orders', 'category', 'orders', 7, 10), + ('status-contract-run', 'customers', 'category', 'customers', 4, 10); + + INSERT INTO keiro.keiro_read_models + (name, version, shape_hash, status, rebuild_group_id) + VALUES + ('orders.summary', 1, 'shape-summary', 'rebuilding', 'status-contract-group'), + ('orders.by-customer', 1, 'shape-by-customer', 'rebuilding', 'status-contract-group'); + """ + +projectionStatusColumnsStatement :: Statement () [(Text, Text)] +projectionStatusColumnsStatement = + Statement.preparable + """ + SELECT attribute.attname::TEXT, format_type(attribute.atttypid, attribute.atttypmod) + FROM pg_catalog.pg_attribute AS attribute + JOIN pg_catalog.pg_class AS relation + ON relation.oid = attribute.attrelid + JOIN pg_catalog.pg_namespace AS namespace + ON namespace.oid = relation.relnamespace + WHERE namespace.nspname = 'keiro_read' + AND relation.relname = 'projection_group_status_v1' + AND attribute.attnum > 0 + AND NOT attribute.attisdropped + ORDER BY attribute.attnum + """ + Encoders.noParams + ( Decoders.rowList + ( (,) + <$> column Decoders.text + <*> column Decoders.text + ) + ) + where + column = Decoders.column . Decoders.nonNullable + +projectionStatusContractStatement :: Statement () Bool +projectionStatusContractStatement = + Statement.preparable + """ + SELECT + lifecycle_phase = 'rebuilding' + AND NOT reads_allowed + AND NOT writes_allowed + AND serving_revision_id IS NULL + AND serving_epoch = 0 + AND serving_position_basis = 'append' + AND serving_applied_position IS NULL + AND active_run_id = 'status-contract-run' + AND candidate_revision_id IS NULL + AND candidate_rebuild_position = 4 + AND candidate_rebuild_head = 10 + AND query_models = ARRAY['orders.by-customer', 'orders.summary']::TEXT[] + AND rebuild_started_at IS NOT NULL + AND last_promoted_at IS NULL + AND failed_at IS NULL + AND failure_code IS NULL + AND failure_detail IS NULL + FROM keiro_read.projection_group_status_v1 + WHERE group_id = 'status-contract-group' + """ + Encoders.noParams + (Decoders.singleRow (Decoders.column (Decoders.nonNullable Decoders.bool))) + +projectionReaderFactsStatement :: Statement () Bool +projectionReaderFactsStatement = + Statement.preparable + """ + SELECT + has_schema_privilege(current_user, 'keiro_read', 'USAGE') + AND has_table_privilege( + current_user, + 'keiro_read.projection_group_status_v1', + 'SELECT' + ) + AND NOT has_schema_privilege(current_user, 'keiro', 'USAGE') + AND NOT has_schema_privilege(current_user, 'kiroku', 'USAGE') + AND ( + SELECT count(*) = 1 + FROM keiro_read.projection_group_status_v1 + WHERE group_id = 'status-contract-group' + ) + """ + Encoders.noParams + (Decoders.singleRow (Decoders.column (Decoders.nonNullable Decoders.bool))) + +backendPidStatement :: Statement () Int32 +backendPidStatement = + Statement.preparable + "SELECT pg_backend_pid()" + Encoders.noParams + (Decoders.singleRow (Decoders.column (Decoders.nonNullable Decoders.int4))) + +withProjectionReaderRole :: Connection.Connection -> IO value -> IO value +withProjectionReaderRole connection action = do + backendPid <- useSession connection (Session.statement () backendPidStatement) + let roleName = "keiro_projection_reader_test_" <> Text.pack (show backendPid) + install = + Text.unlines + [ "CREATE ROLE " <> roleName <> " NOLOGIN;", + "GRANT USAGE ON SCHEMA keiro_read TO " <> roleName <> ";", + "GRANT SELECT ON keiro_read.projection_group_status_v1 TO " <> roleName <> ";", + "SET ROLE " <> roleName <> ";" + ] + cleanup = + Text.unlines + [ "RESET ROLE;", + "REVOKE ALL ON keiro_read.projection_group_status_v1 FROM " <> roleName <> ";", + "REVOKE ALL ON SCHEMA keiro_read FROM " <> roleName <> ";", + "DROP ROLE " <> roleName <> ";" + ] + useSession connection (Session.script install) + action `finally` useSession connection (Session.script cleanup) + +replayProgressFixtureSql :: Text +replayProgressFixtureSql = + """ + INSERT INTO keiro.keiro_projection_rebuild_groups + (group_id, slice_fingerprint, status, active_run_id, reads_allowed, writes_allowed) + VALUES ('constraint-group', 'slice-v1:fixture', 'rebuilding', 'constraint-run', false, false); + INSERT INTO keiro.keiro_projection_rebuild_runs + (run_id, group_id, catalog_fingerprint, group_slice_fingerprint, contract_fingerprint, + runner_format, captured_head, page_size) + VALUES + ('constraint-run', 'constraint-group', 'catalog-fingerprint', + 'slice-v1:fixture', 'contract-v2:fixture', 'keiro/projection-replay/v2', 0, 10); + INSERT INTO keiro.keiro_projection_rebuild_sources + (run_id, source_id, source_scope, category, target_position) + VALUES ('constraint-run', 'source', 'category', 'orders', 0); + INSERT INTO keiro.keiro_projection_rebuild_verifications + (run_id, verification_id, verification_version) + VALUES ('constraint-run', 'verify', 'v1'); + """ + +preCanonicalRebuildFixtureSql :: Text +preCanonicalRebuildFixtureSql = + """ + INSERT INTO keiro.keiro_projection_rebuild_groups + (group_id, catalog_fingerprint, status, active_run_id, requested_by, request_reason, started_at) + VALUES + ('upgrade-rebuilding', repeat('a', 64), 'rebuilding', 'upgrade-run-live', 'ops', 'mid-rebuild upgrade fixture', now()), + ('upgrade-failed', repeat('b', 64), 'failed', 'upgrade-run-failed', 'ops', 'abandoned before upgrade', now()); + UPDATE keiro.keiro_projection_rebuild_groups + SET failed_at = now(), failure_code = 'operator.abandoned', failure_detail = 'abandoned with the old binary' + WHERE group_id = 'upgrade-failed'; + INSERT INTO keiro.keiro_projection_rebuild_runs + (run_id, group_id, catalog_fingerprint, contract_fingerprint, runner_format, captured_head, page_size, status) + VALUES + ('upgrade-run-live', 'upgrade-rebuilding', repeat('a', 64), 'contract-v2:' || repeat('c', 64), 'keiro/projection-replay/v2', 6, 100, 'running'); + INSERT INTO keiro.keiro_projection_rebuild_runs + (run_id, group_id, catalog_fingerprint, contract_fingerprint, runner_format, captured_head, page_size, status, failed_at, failure_code, failure_detail) + VALUES + ('upgrade-run-failed', 'upgrade-failed', repeat('b', 64), 'contract-v2:' || repeat('d', 64), 'keiro/projection-replay/v2', 6, 100, 'failed', now(), 'operator.abandoned', 'abandoned with the old binary'); + """ + +preStatusContractFixtureSql :: Text +preStatusContractFixtureSql = + """ + INSERT INTO keiro.keiro_projection_rebuild_groups + (group_id, slice_fingerprint, status, reads_allowed, writes_allowed) + VALUES + ('upgrade-status-group', 'slice-v6:upgrade-status', 'live', true, true); + """ + +preStatusContractFactsStatement :: Statement () (Text, Int32, Text, Bool, Bool) +preStatusContractFactsStatement = + Statement.preparable + """ + SELECT cursors.position_basis, + cardinality(cursors.subscription_names)::integer, + status.serving_position_basis, + status.serving_applied_position IS NULL, + status.reads_allowed + FROM keiro.keiro_projection_group_cursors AS cursors + JOIN keiro_read.projection_group_status_v1 AS status + ON status.group_id = cursors.group_id + WHERE cursors.group_id = 'upgrade-status-group' + """ + Encoders.noParams + ( Decoders.singleRow + ( (,,,,) + <$> Decoders.column (Decoders.nonNullable Decoders.text) + <*> Decoders.column (Decoders.nonNullable Decoders.int4) + <*> Decoders.column (Decoders.nonNullable Decoders.text) + <*> Decoders.column (Decoders.nonNullable Decoders.bool) + <*> Decoders.column (Decoders.nonNullable Decoders.bool) + ) + ) + +guardFunctionOidStatement :: Statement () Int64 +guardFunctionOidStatement = + Statement.preparable + """ + SELECT 'keiro_read.guard_external_read_v1(text,integer)'::regprocedure::oid::bigint + """ + Encoders.noParams + (Decoders.singleRow (Decoders.column (Decoders.nonNullable Decoders.int8))) + +upgradedGuardFactsStatement :: Statement () (Int64, Bool, Bool) +upgradedGuardFactsStatement = + Statement.preparable + """ + SELECT procedures.oid::bigint, + pg_catalog.pg_get_functiondef(procedures.oid) + LIKE '%group_serving_epoch_before%', + NOT pg_catalog.has_function_privilege( + 'public', + procedures.oid, + 'EXECUTE' + ) + FROM pg_catalog.pg_proc AS procedures + WHERE procedures.oid = + 'keiro_read.guard_external_read_v1(text,integer)'::regprocedure + """ + Encoders.noParams + ( Decoders.singleRow + ( (,,) + <$> Decoders.column (Decoders.nonNullable Decoders.int8) + <*> Decoders.column (Decoders.nonNullable Decoders.bool) + <*> Decoders.column (Decoders.nonNullable Decoders.bool) + ) + ) + +preCanonicalRebuildShapeStatement :: Statement () [(Text, Text, Text, Text)] +preCanonicalRebuildShapeStatement = + Statement.preparable + """ + SELECT groups.group_id, + groups.slice_fingerprint, + runs.run_id, + runs.group_slice_fingerprint + FROM keiro.keiro_projection_rebuild_groups AS groups + JOIN keiro.keiro_projection_rebuild_runs AS runs + ON runs.group_id = groups.group_id + WHERE groups.group_id IN ('upgrade-rebuilding', 'upgrade-failed') + ORDER BY groups.group_id + """ + Encoders.noParams + ( Decoders.rowList + ( (,,,) + <$> column Decoders.text + <*> column Decoders.text + <*> column Decoders.text + <*> column Decoders.text + ) + ) + where + column = Decoders.column . Decoders.nonNullable + +legacyGroupUpgradeStatement :: + Statement () [(Text, Int64, Text, Text, Text, Text, Maybe Text, Maybe Text)] +legacyGroupUpgradeStatement = + Statement.preparable + """ + SELECT rm.name, + rm.version, + rm.shape_hash, + rm.status, + rm.rebuild_group_id, + rg.status, + rg.active_run_id, + rg.failure_detail + FROM keiro.keiro_read_models AS rm + JOIN keiro.keiro_projection_rebuild_groups AS rg + ON rg.group_id = rm.rebuild_group_id + ORDER BY rm.name + """ + Encoders.noParams + ( Decoders.rowList + ( (,,,,,,,) + <$> column Decoders.text + <*> column Decoders.int8 + <*> column Decoders.text + <*> column Decoders.text + <*> column Decoders.text + <*> column Decoders.text + <*> nullableColumn Decoders.text + <*> nullableColumn Decoders.text + ) + ) + where + column = Decoders.column . Decoders.nonNullable + nullableColumn = Decoders.column . Decoders.nullable + assertBlockedCoddPreflight :: Text -> Expectation assertBlockedCoddPreflight sourceSchema = withKeiroPg $ \database -> do @@ -469,14 +977,14 @@ up <- runMigrationPlan defaultRunOptions settings plan >>= requireRight reportOutcomes up `shouldBe` replicate 7 AlreadyApplied - <> [AppliedNow] + <> replicate 3 AppliedNow <> replicate 16 AlreadyApplied - <> [AppliedNow, AppliedNow, AppliedNow, AppliedNow] + <> replicate 14 AppliedNow verifiedAfterCanaries <- verifyMigrationPlan defaultRunOptions settings plan >>= requireRight case verifiedAfterCanaries of VerificationReport verificationIssues _ _ _ -> verificationIssues `shouldBe` [] rerun <- runMigrationPlan defaultRunOptions settings plan >>= requireRight - reportOutcomes rerun `shouldBe` replicate 28 AlreadyApplied + reportOutcomes rerun `shouldBe` replicate 40 AlreadyApplied second <- importCoddHistory defaultImportOptions config provider plan frameworkCoddHistoryMappings >>= requireRight @@ -486,7 +994,7 @@ sourceRows <- useSession connection (Session.statement () (sourceRowCountStatement sourceSchema)) sourceRows `shouldBe` 23 facts <- useSession connection (Session.statement () importFactsStatement) - facts `shouldBe` (28, 23, True) + facts `shouldBe` (40, 23, True) postCoddImportPendingIssues :: IO [VerificationIssue] postCoddImportPendingIssues = @@ -497,10 +1005,22 @@ pendingNames = [ ("kiroku", "0008-schema-management-comment"), + ("kiroku", "0009"), + ("kiroku", "0010"), ("keiro", "0017-schema-management-comment"), ("keiro", "0018"), ("keiro", "0019-keiro-snapshots-state-shape-hash"), - ("keiro", "0020-keiro-workflow-children-failure-reason") + ("keiro", "0020-keiro-workflow-children-failure-reason"), + ("keiro", "0021-keiro-workflows-exact-discovery"), + ("keiro", "0022"), + ("keiro", "0023"), + ("keiro", "0024"), + ("keiro", "0025"), + ("keiro", "0026"), + ("keiro", "0027"), + ("keiro", "0028"), + ("keiro", "0029"), + ("keiro", "0030") ] assertPoisonedLedger :: Settings.Settings -> Expectation @@ -533,7 +1053,17 @@ "0017-schema-management-comment.sql", "0018.sql", "0019-keiro-snapshots-state-shape-hash.sql", - "0020-keiro-workflow-children-failure-reason.sql" + "0020-keiro-workflow-children-failure-reason.sql", + "0021-keiro-workflows-exact-discovery.sql", + "0022.sql", + "0023.sql", + "0024.sql", + "0025.sql", + "0026.sql", + "0027.sql", + "0028.sql", + "0029.sql", + "0030.sql" ] findMigrationsDirectory :: IO FilePath @@ -709,11 +1239,16 @@ (to_regnamespace('kiroku') IS NOT NULL), (to_regclass('kiroku.events') IS NOT NULL), (to_regnamespace('keiro') IS NOT NULL), + (to_regnamespace('keiro_read') IS NOT NULL), (to_regclass('keiro.keiro_inbox') IS NOT NULL), (to_regclass('keiro.keiro_outbox') IS NOT NULL), (to_regclass('keiro.keiro_timers') IS NOT NULL), (to_regclass('keiro.keiro_workflows') IS NOT NULL), - (obj_description(to_regnamespace('kiroku'), 'pg_namespace') = 'Managed by pg-migrate component kiroku through 0008-schema-management-comment'), + (to_regclass('keiro.keiro_projection_group_cursors') IS NOT NULL), + (to_regclass('keiro_read.projection_group_status_v1') IS NOT NULL), + (obj_description(to_regnamespace('keiro_read'), 'pg_namespace') = + 'Versioned, owner-rights read contracts for out-of-process Keiro consumers.'), + (obj_description(to_regnamespace('kiroku'), 'pg_namespace') = 'Managed by pg-migrate component kiroku through 0010'), (obj_description(to_regnamespace('keiro'), 'pg_namespace') = 'Managed by pg-migrate component keiro through 0017-schema-management-comment') ) AS checks(ok) """