diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,145 @@ --- +## [0.17.1] - unreleased + +Development version. Opens the 0.17.x line — the first release built entirely against +published 0.17.0 hosts (Hackage / PyPI / Maven Central), with the `hostOverrides` shims +that were needed while 0.16.1 lacked the effect type and I/O primitives removed. + +### New features + +- **Generated term-reference DSLs** ([#467](https://github.com/CategoricalData/hydra/issues/467)): + kernel *term* modules now project generated DSL surfaces, completing the story begun with + type-module DSLs and the generated primitive wrappers (`hydra.dsl.lib.*`, 0.17.0). A + signature read-back in the DSL pass reuses the main pass's inference, and a demand-curated + set of 19 kernel term modules (plus the json/test/extract modules) emits one typed, + rename-safe reference per definition into every target language — e.g. Haskell + `Hydra.Dsl.Strip.deannotateType`, Java `hydra.dsl.Strip.deannotateType(...)`, Python + `hydra.dsl.strip.deannotate_type(...)`. The Java and Python coder sources were pivoted onto + the generated references, retiring ~700 stringly-typed inline `var("hydra....")` references + and shrinking Python's hand registry to the non-generatable remainder (byte-identical + generated output before and after). Follow-ups: + [#555](https://github.com/CategoricalData/hydra/issues/555) (typed references to derived + encode/decode/show functions via `TypedName`), + [#556](https://github.com/CategoricalData/hydra/issues/556) (non-kernel packages). +- **Package validation** ([#574](https://github.com/CategoricalData/hydra/issues/574), + [#575](https://github.com/CategoricalData/hydra/issues/575)): a translingual `hydra.validate.*` + framework carries per-package `ValidationProfile`s (error/warning rules, error/warning caps) and a + catalog of checks — undeclared-package-dependency detection (now a fatal gate, #574), + definition-name convention (PascalCase), definition ordering, documentation completeness, and + conflicting-variant-name detection — layered on the existing core term/type well-formedness rules. + A kernel-strict / others-relaxed policy keeps `/sync` green fleet-wide while the documentation + backlog drains. See the new [Validation](https://github.com/CategoricalData/hydra/wiki/Validation) + wiki page for the full per-package constraint specification. +- **DSL-authoring registration guards** ([#554](https://github.com/CategoricalData/hydra/issues/554)): + fail-fast authoring-time validation for the host-native DSL sources — every DSL-authoring module in + Java/`hydra-jvm`, Python, and Scala now asserts its definitions are fully registered + (`Defs.checkComplete` / `check_complete`), backed by a Haskell kernel-source registration-completeness + scanner, so a definition added to a source file but omitted from its `definitions` list fails loudly + instead of silently vanishing from the generated output. +- **`hydra-build` package** ([#546](https://github.com/CategoricalData/hydra/issues/546), + [#529](https://github.com/CategoricalData/hydra/issues/529), + [#530](https://github.com/CategoricalData/hydra/issues/530)): the build-orchestration logic that had + lived only in host scripts is now translingual Hydra. Pure module-list utilities became + `hydra.build.modules` (#529) and the orphan-reconcile decision became `hydra.build.reconcile` (#530), + each with a translingual test suite; the `hydra.build.*` modules were then extracted into a dedicated + `hydra-build` package (#546), separate from the kernel. +- **Metadata sidecar + content/version-addressed freshness** + ([#415](https://github.com/CategoricalData/hydra/issues/415)): dist manifests now carry a + `moduleFormatVersion` field, and the generator's freshness basis is content/version-addressed — the + groundwork for reproducible, cache-correct incremental generation. +- **Structured generation-provenance record** + ([#523](https://github.com/CategoricalData/hydra/issues/523)): each generation run now records a + structured `Generation` provenance entry (digest v2), replacing the ad-hoc shell threading of + generation metadata. +- **Worker + issue hierarchy** ([#557](https://github.com/CategoricalData/hydra/issues/557)): a + principled agent/worker framework — roles and parentage mirroring the GitHub issue tree, a + cross-worktree message transport with a verify-after-copy protocol, a blocking-aware design-question + (`decisions/`) channel, an issue-proposal queue with the `/issues` command, an autonomy-dial banner, + and cross-machine staging coordination. Contributor-facing process only; no change to generated code. + +### API and usability + +- **DSL usability passes** ([#430](https://github.com/CategoricalData/hydra/issues/430), + [#553](https://github.com/CategoricalData/hydra/issues/553)): reviewed the host-native Java, Python, + and Scala DSL surfaces for ergonomics and cross-host parity (documented def shapes, added a Scala + projection helper), bringing the three authoring dialects into closer alignment. + +### Fixed + +Bug fixes landed this cycle, grouped by area (see each issue for detail): + +- **TypeScript** — reactivated `hydra.lib.*` primitive tests surfaced encoder gaps: lazy show-calling + let bindings and compact bare-string `LiteralType` expansion in the test converter + ([#564](https://github.com/CategoricalData/hydra/issues/564)), + `binaryToBytes` registration + base64 binary decode in `decodeUtf8` + ([#561](https://github.com/CategoricalData/hydra/issues/561)), + `hydra.test.build.*` emission after the #546 split + ([#570](https://github.com/CategoricalData/hydra/issues/570)), and `HYDRA_DEFAULT_IMPLS` support in + the test runner ([#536](https://github.com/CategoricalData/hydra/issues/536)). +- **Tests and CI** — restored the default-primitive-implementation testing solution + ([#388](https://github.com/CategoricalData/hydra/issues/388), + [#549](https://github.com/CategoricalData/hydra/issues/549)), reactivated previously-ignored kernel + test cases ([#513](https://github.com/CategoricalData/hydra/issues/513), + [#550](https://github.com/CategoricalData/hydra/issues/550)), wired the regression harnesses into CI + and split them out of the monolithic Haskell job for timeout headroom + ([#535](https://github.com/CategoricalData/hydra/issues/535), + [#563](https://github.com/CategoricalData/hydra/issues/563)), added Scala/Go/Lisp + self-containment gates to `verify-distribution.sh` + ([#537](https://github.com/CategoricalData/hydra/issues/537)), a manifest-JSON-tracked CI check + ([#541](https://github.com/CategoricalData/hydra/issues/541)), a per-package test-digest freshness + fix ([#551](https://github.com/CategoricalData/hydra/issues/551)), overlay-file restoration in + `test-stale-output-prune.sh` ([#558](https://github.com/CategoricalData/hydra/issues/558)), and a + sync-freshness fingerprint that now sees host-native DSL sources + ([#562](https://github.com/CategoricalData/hydra/issues/562)). +- **Kernel and coders** — Java binary-literal signed-byte emission + ([#528](https://github.com/CategoricalData/hydra/issues/528)), `hydra.json.Decode.fromJson` + parametric-type (Application/Forall) support + ([#531](https://github.com/CategoricalData/hydra/issues/531)), stale 2-param `Coder` shape in + Java/Scala test bootstrap ([#552](https://github.com/CategoricalData/hydra/issues/552)), and a + kernel dead-code/TODO cleanup ([#538](https://github.com/CategoricalData/hydra/issues/538)). +- **Lib and build** — `hydra.lib.files` copy/removeDirectory/status overlays across all nine hosts + ([#525](https://github.com/CategoricalData/hydra/issues/525)), `hydra.build.*` regeneration into the + Lisp dialects ([#544](https://github.com/CategoricalData/hydra/issues/544)), Scheme maps + last-writer-wins in `union` ([#545](https://github.com/CategoricalData/hydra/issues/545)), + standalone-kernel SHA dependency ([#542](https://github.com/CategoricalData/hydra/issues/542)), + the lexicon's empty Types section ([#539](https://github.com/CategoricalData/hydra/issues/539)), and + a duplicated generated-file header in `generate-head-haskell-build.py` + ([#540](https://github.com/CategoricalData/hydra/issues/540)). + +- **TypeScript overlay-import rewrite** ([#565](https://github.com/CategoricalData/hydra/issues/565)): + the TypeScript coder's lib overlay-import rewrite no longer captures `hydra.lib.defaults`, which is a + generated (not overlaid) module; excluding it fixes spurious import references in generated TypeScript. + +- **Lexicon generation** (`bin/regenerate-lexicon.sh`): the generator had been failing + silently since the encoder/decoder modules moved to in-memory synthesis + ([#448](https://github.com/CategoricalData/hydra/issues/448)) — the script masked the + failure and left a stale `docs/hydra-lexicon.txt` missing the four effectful primitive + libraries (`hydra.lib.{effects,files,system,text}`). The synthesized coder modules are + now included in the lexicon universe, the script fails loudly, and the regenerated + lexicon covers all 267 primitives across 17 library modules. +- **`bin/prepare-release.sh`**: the non-blocking Java quality check invoked Gradle + subprojects (`:hydra-rdf4j:test` etc.) that were removed in + [#511](https://github.com/CategoricalData/hydra/issues/511), so it always warned without + checking anything; it now runs the `dist/java/hydra-pg` package tests (the current home + of the former binding tests). +- **`.gitignore`**: the ignore rules for the overlaid hand-written Haskell runtime still + named pre-[#501](https://github.com/CategoricalData/hydra/issues/501) paths, so 41 + duplicate copies under `dist/haskell/` had become tracked; the rules now match the + `Hydra/Overlay/` location and the duplicates are untracked. + +### Documentation + +- Corrected the 0.17.0 Maven coordinates throughout (Java artifacts publish under group + `net.fortytwo.hydra.java`, Scala under `net.fortytwo.hydra.scala`, + [#519](https://github.com/CategoricalData/hydra/issues/519)); added the npm and Scala + release channels to the README; added TypeScript/Scala/Lisp sections to Getting started; + repointed stale pre-#418/#501 DSL paths in the architecture docs; added GitHub issue and + PR templates. + +--- + ## [0.17.0] - 2026-07-01 Major themes: first-class support for effects and I/O in the kernel, TypeScript and diff --git a/hydra-pg.cabal b/hydra-pg.cabal --- a/hydra-pg.cabal +++ b/hydra-pg.cabal @@ -1,11 +1,11 @@ cabal-version: 1.12 --- This file has been generated from package.yaml by hpack version 0.39.1. +-- This file has been generated from package.yaml by hpack version 0.38.1. -- -- see: https://github.com/sol/hpack name: hydra-pg -version: 0.17.0 +version: 0.17.1 synopsis: Hydra's property-graph (TinkerPop/Gremlin) model and coder support description: Hydra is an implementation of the LambdaGraph data model, which takes advantage of an isomorphism between labeled hypergraphs and typed lambda calculus: in Hydra, "graphs are programs, and programs are graphs". Property graph support for Hydra category: Data @@ -46,6 +46,8 @@ Hydra.Graphviz.Serde Hydra.Neo4j.Model Hydra.Neo4j.Pg + Hydra.Overlay.Haskell.Dsl.Pg.Mappings + Hydra.Overlay.Haskell.Dsl.Pg.Schemas Hydra.Pg.Coder Hydra.Pg.Graphson.Coder Hydra.Pg.Graphson.Construct @@ -75,7 +77,7 @@ base >=4.19.0 && <4.22 , bytestring >=0.11.5 && <0.13 , containers >=0.6.7 && <0.8 - , hydra-kernel ==0.17.0 - , hydra-rdf ==0.17.0 + , hydra-kernel ==0.17.1 + , hydra-rdf ==0.17.1 , scientific >=0.3.7 && <0.4 default-language: Haskell2010 diff --git a/src/main/haskell/Hydra/Overlay/Haskell/Dsl/Pg/Mappings.hs b/src/main/haskell/Hydra/Overlay/Haskell/Dsl/Pg/Mappings.hs new file mode 100644 --- /dev/null +++ b/src/main/haskell/Hydra/Overlay/Haskell/Dsl/Pg/Mappings.hs @@ -0,0 +1,65 @@ +-- | A DSL for defining mappings between relational data and property graphs in Hydra. +module Hydra.Overlay.Haskell.Dsl.Pg.Mappings where + +import Hydra.Core +import Hydra.Typed +import Hydra.Rewriting +import Hydra.Pg.Model +import Hydra.Overlay.Haskell.Dsl.Typed.Phantoms + +import qualified Data.Map as M + +-- * Helpers + +temporaryTableName = "(((temp)))" + +replaceTableName tableName = rewriteTerm $ \recurse term -> case term of + TermProject (Projection tname fname) -> if unName tname == temporaryTableName + then TermProject $ Projection (Name tableName) fname + else term + _ -> recurse term + +-- * Graph construction functions + +-- | Create a mapping function that selects a specified column from a table +column :: String -> TypedTerm (r -> Maybe a) +column sourceColumn = project (Name temporaryTableName) (Name sourceColumn) + +-- | Create an edge with the given label, ID, source vertex, target vertex, and properties +-- Example: edge "social_network.csv" "knows" (constant unit) (column "person_1") (column "person_2") +-- [property "since" $ column "knows_since"] +edge :: String -> String -> TypedTerm e -> TypedTerm vo -> TypedTerm vi -> [Property Term] -> Edge Term +edge tableName label (TypedTerm idValue) (TypedTerm outValue) (TypedTerm inValue) propertiesList = Edge + (EdgeLabel label) + (replaceTableName tableName idValue) + (replaceTableName tableName outValue) + (replaceTableName tableName inValue) + (M.fromList $ fmap (\(Property k v) -> (k, replaceTableName tableName v)) propertiesList) + +-- | Create an edge with a unit ID (when the edge id isn't significant) +-- Example: simpleEdge "social_network.csv" "knows" (column "person_1") (column "person_2") +-- [property "since" $ column "knows_since"] +edgeNoId :: String -> String -> TypedTerm vo -> TypedTerm vi -> [Property Term] -> Edge Term +edgeNoId tableName label = edge tableName label (constant unit) + +-- | Create a lazy graph from vertices and edges, without enforcing id uniqueness +-- Example: graph [person1, person2] [knows] +-- Where: +-- person1 = vertex "Person" (string "p1") [property "name" (string "John")] +-- person2 = vertex "Person" (string "p2") [property "name" (string "Jane")] +-- knows = edge "knows" (string "e1") person1 person2 [property "since" (int32 2020)] +graph :: [Vertex v] -> [Edge v] -> LazyGraph v +graph = LazyGraph + +-- | Create a property with the given key and value +-- Example: property "name" (string "John") +property :: String -> TypedTerm v -> Property Term +property key (TypedTerm value) = Property (PropertyKey key) value + +-- | Create a vertex derived from the given table with the given label, ID, and properties +-- Example: vertex "people.csv" "Person" personId [property "name" (column "full_name"), property "age" (column "age)] +vertex :: String -> String -> TypedTerm v -> [Property Term] -> Vertex Term +vertex tableName label (TypedTerm idValue) propertiesList = Vertex + (VertexLabel label) + (replaceTableName tableName idValue) + (M.fromList $ fmap (\(Property k v) -> (k, replaceTableName tableName v)) propertiesList) diff --git a/src/main/haskell/Hydra/Overlay/Haskell/Dsl/Pg/Schemas.hs b/src/main/haskell/Hydra/Overlay/Haskell/Dsl/Pg/Schemas.hs new file mode 100644 --- /dev/null +++ b/src/main/haskell/Hydra/Overlay/Haskell/Dsl/Pg/Schemas.hs @@ -0,0 +1,76 @@ +-- | A DSL for constructing property graph schemas and instances in Hydra. +module Hydra.Overlay.Haskell.Dsl.Pg.Schemas where + +import Hydra.Pg.Model +import Hydra.Core +import qualified Hydra.Overlay.Haskell.Dsl.Types as Types +import qualified Hydra.Overlay.Haskell.Dsl.Terms as Terms + +import qualified Data.Map as M + +-- * Type-level functions (for schema definitions) + +-- | Create a property type definition with the given key and type +-- Example: propertyType "name" Types.string +propertyType :: String -> t -> PropertyType t +propertyType key typ = PropertyType (PropertyKey key) typ False + +-- | Mark a property type as required +-- Example: required (propertyType "name" Types.string) +required :: PropertyType t -> PropertyType t +required (PropertyType key typ _) = PropertyType key typ True + +-- | Define a vertex type with the given label, ID type, and property types +-- Example: vertexType "Person" Types.string [required (propertyType "name" Types.string), propertyType "age" Types.int32] +vertexType :: String -> t -> [PropertyType t] -> VertexType t +vertexType label id properties = VertexType (VertexLabel label) id properties + +-- | Define an edge type with the given label, ID type, source vertex label, target vertex label, and property types +-- Example: edgeType "KNOWS" Types.string "Person" "Person" [propertyType "since" Types.int32] +edgeType :: String -> t -> String -> String -> [PropertyType t] -> EdgeType t +edgeType label id outLabel inLabel properties = EdgeType (EdgeLabel label) id (VertexLabel outLabel) (VertexLabel inLabel) properties + +-- | Define a simple edge type with a unit ID type +-- Example: simpleEdgeType "KNOWS" "Person" "Person" [propertyType "since" Types.int32] +simpleEdgeType :: String -> String -> String -> [PropertyType Type] -> EdgeType Type +simpleEdgeType label = edgeType label Types.unit + +-- | Create a graph schema from vertex types and edge types +-- Example: schema [personType] [knowsType] +-- Where: +-- personType = vertexType "Person" Types.string [required (propertyType "name" Types.string)] +-- knowsType = edgeType "KNOWS" Types.string "Person" "Person" [propertyType "since" Types.int32] +schema :: [VertexType t] -> [EdgeType t] -> GraphSchema t +schema vtypes etypes = GraphSchema + (M.fromList $ fmap (\vt -> (vertexTypeLabel vt, vt)) vtypes) + (M.fromList $ fmap (\et -> (edgeTypeLabel et, et)) etypes) + +-- * Term-level functions (for graph instances) + +-- | Create a property with the given key and value +-- Example: property "name" (Terms.string "John") +property :: String -> v -> Property v +property key value = Property (PropertyKey key) value + +-- | Create a vertex with the given label, ID, and properties +-- Example: vertex "Person" (Terms.string "p1") [property "name" (Terms.string "John"), property "age" (Terms.int32 42)] +vertex :: String -> v -> [Property v] -> Vertex v +vertex label idValue propertiesList = Vertex (VertexLabel label) idValue $ + M.fromList $ fmap (\(Property k v) -> (k, v)) propertiesList + +-- | Create an edge with the given label, ID, source vertex, target vertex, and properties +-- Example: edge "KNOWS" (Terms.string "e1") person1 person2 [property "since" (Terms.int32 2020)] +edge :: String -> v -> v -> v -> [Property v] -> Edge v +edge label idValue outValue inValue propertiesList = Edge (EdgeLabel label) idValue outValue inValue $ + M.fromList $ fmap (\(Property k v) -> (k, v)) propertiesList + +-- | Create a graph from vertices and edges +-- Example: graph [person1, person2] [knows] +-- Where: +-- person1 = vertex "Person" (Terms.string "p1") [property "name" (Terms.string "John")] +-- person2 = vertex "Person" (Terms.string "p2") [property "name" (Terms.string "Jane")] +-- knows = edge "KNOWS" (Terms.string "e1") person1 person2 [property "since" (Terms.int32 2020)] +graph :: Ord v => [Vertex v] -> [Edge v] -> Graph v +graph vertices edges = Graph + (M.fromList $ fmap (\v -> (vertexId v, v)) vertices) + (M.fromList $ fmap (\e -> (edgeId e, e)) edges)