hydra-0.13.0: README.md

# Hydra-Haskell

Hydra is a functional programming language based on the [LambdaGraph](https://bit.ly/lg-kgc2024) data model,
exploring an isomorphism between typed lambda calculus and labeled hypergraphs.
See the main Hydra [README](https://github.com/CategoricalData/hydra) for project overview and use cases.

This package contains Hydra's **Haskell implementation**, which serves as the bootstrapping implementation for the entire Hydra project.
Releases are available [on Hackage](https://hackage.haskell.org/package/hydra).

## Code Organization

Hydra-Haskell uses the **src/main vs src/gen-main** separation pattern (see [Code organization wiki page](https://github.com/CategoricalData/hydra/wiki/Code-organization) for details).

- **`src/main/haskell/`** - Hand-written source code
  - `Hydra/Dsl/` - DSL syntax definitions
  - `Hydra/Sources/` - Kernel specifications written in the DSL
  - `Hydra/Lib/` - Native primitive implementations
  - `Hydra/Generation.hs` - Code generation utilities

- **`src/gen-main/haskell/`** - Generated code
  - `Hydra/Core.hs`, `Hydra/Graph.hs`, etc. - Complete kernel implementation
  - Generated by running `writeHaskell` in GHCi

- **`src/gen-test/haskell/`** - Generated test code
  - Kernel tests generated from test sources using `writeHaskell`
  - Generation tests created using language-specific test generators
  - See [Testing](#test) section for details

Haskell serves as the **bootstrapping implementation** - the DSL sources here generate code for Java, Python, and other languages.

## Documentation

For comprehensive documentation about Hydra's architecture, type system, and implementation details, see:

- **[Concepts](https://github.com/CategoricalData/hydra/wiki/Concepts)** - Core concepts: Type, Term, Graph, Flow monad, primitives, coders
- **[Implementation](https://github.com/CategoricalData/hydra/blob/main/docs/src/implementation.md)** - Detailed guide covering type modules, DSLs, primitives, coders, and the bootstrap process
- **[DSL Guide](https://github.com/CategoricalData/hydra/blob/main/docs/src/dsl-guide.md)** - Comprehensive guide to Hydra's domain-specific languages
- **[Code Organization](https://github.com/CategoricalData/hydra/wiki/Code-organization)** - The src/main vs src/gen-main pattern
- **[Testing](https://github.com/CategoricalData/hydra/wiki/Testing)** - Common test suite and language-specific testing
- **[Developer Recipes](https://github.com/CategoricalData/hydra/blob/main/docs/src/recipes/index.md)** - Step-by-step guides for extending Hydra

This README focuses on practical instructions for building, testing, and generating code with Hydra-Haskell.

## Build

Haskell is Hydra's **bootstrapping language**. The entire Hydra kernel is written using Haskell-based domain-specific languages (DSLs):
- **DSL syntax**: [src/main/haskell/Hydra/Dsl](https://github.com/CategoricalData/hydra/tree/main/hydra-haskell/src/main/haskell/Hydra/Dsl) - Specify the syntax for Hydra programs written in Haskell
- **DSL-based sources**: [src/main/haskell/Hydra/Sources](https://github.com/CategoricalData/hydra/tree/main/hydra-haskell/src/main/haskell/Hydra/Sources) - Type definitions and core logic written in Haskell DSL
- **Generated code**: [src/gen-main/haskell](https://github.com/CategoricalData/hydra/tree/main/hydra-haskell/src/gen-main/haskell) - Haskell code generated from DSL sources
- **Primitives**: [src/main/haskell/Hydra/Lib](https://github.com/CategoricalData/hydra/tree/main/hydra-haskell/src/main/haskell/Hydra/Lib) - Hydra's standard libraries of primitive functions and constants, implemented in Haskell. These libraries are registered in [Libraries.hs](https://github.com/CategoricalData/hydra/tree/main/hydra-haskell/src/main/haskell/Hydra/Sources/Libraries.hs).

The DSL sources are also used to generate Java and Python implementations, ensuring parity across each Hydra language variant.

### Build and REPL

First, install the [Haskell Tool Stack](https://docs.haskellstack.org/en/stable):

```bash
# On macOS
brew install haskell-stack

# Other platforms: see https://docs.haskellstack.org/en/stable/install_and_upgrade/
```

Then build and enter the GHCi REPL:

```bash
stack ghci hydra:lib
```

Note: The `hydra:lib` target is required to avoid loading test modules. Running `stack ghci` without a target will load all components including test dependencies, which you may not need.

## Test

Run all tests:

```bash
stack test
```

For interactive testing with access to test utilities:

```bash
stack ghci hydra:lib hydra:hydra-test
```

Then in the REPL, you can run individual tests:

```haskell
Test.Hspec.hspec Hydra.TestSuiteSpec.spec
```

See the [Testing wiki page](https://github.com/CategoricalData/hydra/wiki/Testing) for details on Hydra's common test suite, which ensures parity across all Hydra language variants.

## Code generation

Hydra is **self-hosting**: it can generate its own source code from DSL definitions.

### Generate Haskell code

Enter the GHCi REPL and import the generation module:

```haskell
import Hydra.Generation
```

Generate all main modules provided in Hydra-Haskell:

```haskell
-- First arg: output directory
-- Second arg: universe modules (for dependency resolution)
-- Third arg: modules to generate
writeHaskell "src/gen-main/haskell" mainModules mainModules
```

Generate only the Hydra kernel (excluding other Hydra-Haskell artifacts like the Haskell coder, the JSON coder, etc.):

```haskell
writeHaskell "src/gen-main/haskell" kernelModules kernelModules
```

Generate specific main modules:

```haskell
-- For specific modules, include dependencies in the universe
writeHaskell "src/gen-main/haskell" mainModules [haskellLanguageModule, haskellCoderModule]
```

Generate kernel test modules:

```haskell
let allModules = mainModules ++ testModules
writeHaskell "src/gen-test/haskell" allModules baseTestModules
```

Or use the convenience script:

```bash
./bin/update-kernel-tests.sh
```

This generates tests that validate the Hydra kernel implementation.

Generate language-specific generation tests (tests that verify code generation to Haskell):

```bash
./bin/update-generation-tests.sh
```

These two types of tests serve different purposes:
- **Kernel tests**: Validate core Hydra functionality (type checking, reduction, etc.)
- **Generation tests**: Verify that Hydra can correctly generate code in target languages

Note `src/gen-test` as opposed to `src/gen-main`.

### Generate code for other languages

Java and Python code generation is handled by the [Hydra-Ext](https://github.com/CategoricalData/hydra/tree/main/hydra-ext) package.
From the `hydra-ext` directory:

```bash
cd ../hydra-ext && stack ghci
```

Then in GHCi:

```haskell
import Hydra.Ext.Generation

-- Generate Python kernel
-- Second arg: universe modules (for dependency resolution)
-- Third arg: modules to generate
writePython "../hydra-python/src/gen-main/python" kernelModules kernelModules

-- Generate Java kernel
writeJava "../hydra-java/src/gen-main/java" kernelModules kernelModules
```

And similar for test artifacts. See the Hydra-Ext README for more details.
Hydra-Ext also includes coders for many other languages and formats: Avro, Protobuf, C++, C#, Scala, GraphQL, JSON Schema, RDF, and more.

## Working with Hydra

### Core types

Some of the fundamental types in Hydra are:

- **`Type`** - Represents the structure of data (literals, records, unions, functions, etc.)
- **`Term`** - Represents data or computation (instances of types)
- **`Binding`** - A named binding (name + term + type scheme), also called an *element*.
- **`Graph`** - A collection of elements with an environment, types, primitives, and schema graph
- **`Module`** - A namespace containing elements with dependencies on other modules

These are defined in [Hydra/Sources/Kernel/Types](https://github.com/CategoricalData/hydra/tree/main/hydra-haskell/src/main/haskell/Hydra/Sources/Kernel/Types)
and code-generated into [Hydra.Core](https://github.com/CategoricalData/hydra/blob/main/hydra-haskell/src/gen-main/haskell/Hydra/Core.hs),
[Hydra.Graph](https://github.com/CategoricalData/hydra/blob/main/hydra-haskell/src/gen-main/haskell/Hydra/Graph.hs), and
[Hydra.Module](https://github.com/CategoricalData/hydra/blob/main/hydra-haskell/src/gen-main/haskell/Hydra/Module.hs).

See [Concepts](https://github.com/CategoricalData/hydra/wiki/Concepts) for detailed explanations.

### The Flow monad

Transformations in Hydra use the `Flow` monad for:
- State management (graph context)
- Error handling
- Logging and tracing

```haskell
type Flow s a = s -> Trace -> FlowState s a
```

Common usage pattern:

```haskell
import Hydra.Compute (Flow, FlowState, unFlow, emptyTrace)

-- Create a Flow
myComputation :: Flow Graph String
myComputation = pure "result"

-- Execute it
let state = unFlow myComputation graph emptyTrace
case flowStateValue state of
  Just result -> putStrLn result
  Nothing -> print (flowStateMessages $ flowStateTrace state)
```

### Coders and adapters

**Coders** are bidirectional transformations:

```haskell
data Coder s1 s2 v1 v2 = Coder {
  coderEncode :: v1 -> Flow s2 v2,
  coderDecode :: v2 -> Flow s1 v1
}
```

**Adapters** also transform types, enabling schema evolution and language mapping.

See the [Implementation wiki](https://github.com/CategoricalData/hydra/blob/main/docs/src/implementation.md#cross-language-compilation-coders) for details.

### DSLs

Hydra provides multiple domain-specific languages for constructing types and terms:

**Untyped DSLs** ([Hydra/Dsl/Types.hs](https://github.com/CategoricalData/hydra/blob/main/hydra-haskell/src/main/haskell/Hydra/Dsl/Types.hs),
[Hydra/Dsl/Terms.hs](https://github.com/CategoricalData/hydra/blob/main/hydra-haskell/src/main/haskell/Hydra/Dsl/Terms.hs)):
```haskell
import qualified Hydra.Dsl.Types as Types
import qualified Hydra.Dsl.Terms as Terms

personType = Types.record [
  "name" >: string,
  "age" >: int32]

alice = Terms.record [
  "name" >: Terms.string "Alice",
  "age" >: Terms.int32 30]
```

**Phantom-typed DSLs** ([Hydra/Dsl/Meta/Phantoms.hs](https://github.com/CategoricalData/hydra/blob/main/hydra-haskell/src/main/haskell/Hydra/Dsl/Meta/Phantoms.hs)) - Compile-time type safety:
```haskell
import Hydra.Dsl.Meta.Phantoms

safeFn :: TTerm (Int -> String)
safeFn = lambda "x" (Strings.toUpper (var "x"))  -- Type-checked at compile time
```

**Library DSLs** ([Hydra/Dsl/Meta/Lib](https://github.com/CategoricalData/hydra/tree/main/hydra-haskell/src/main/haskell/Hydra/Dsl/Meta/Lib)) - Wrappers for primitive functions:
```haskell
import Hydra.Dsl.Meta.Lib.Lists as Lists
import Hydra.Dsl.Meta.Lib.Strings as Strings

example = Lists.map (Strings.toUpper) (list ["hello", "world"])
```

See the [DSL system section](https://github.com/CategoricalData/hydra/blob/main/docs/src/implementation.md#dsl-system) in the Implementation wiki for comprehensive coverage.

### JSON and YAML serialization

Serialize Hydra terms to JSON or YAML:

```haskell
import Hydra.Kernel
import Hydra.Codegen
import Hydra.Langs.Json.Serde
import Hydra.Dsl.Terms as Terms

-- Get the Hydra core graph
let g = hydraCoreGraph
let flow = fromFlowIo g

-- Define a type
let typ = TypeVariable _Precision

-- Create a term of that type
let term = Terms.inject _Precision (Field _Precision_bits $ Terms.int32 64)

-- Create a JSON coder for the type
coder <- flow $ jsonStringCoder typ

-- Encode to JSON
result <- flow (coderEncode coder term)
putStrLn result
```

## Self-hosting demonstration

Hydra-Haskell is a [self-hosting compiler](https://en.wikipedia.org/wiki/Self-hosting_(compilers)) - it can generate its own source code.

Complete self-hosting cycle:

```bash
# Generate all kernel code
stack ghci hydra:lib
import Hydra.Sources.All
import Hydra.Generation
writeHaskell "src/gen-main/haskell" mainModules mainModules
:q

# Generate kernel tests
./bin/update-kernel-tests.sh

# Generate language-specific generation tests
./bin/update-generation-tests.sh

# Run all tests
stack test
```

Alternatively, use GHCi for all generation steps:

```bash
stack ghci hydra:lib
import Hydra.Sources.All
import Hydra.Generation
writeHaskell "src/gen-main/haskell" mainModules mainModules
let allModules = mainModules ++ testModules
writeHaskell "src/gen-test/haskell" allModules baseTestModules
:q
stack test
```

The generated code includes:
- All core types (Type, Term, Graph, Module, etc.)
- Type inference and checking
- Term reduction and rewriting
- Coders and adapters
- Primitive functions (signatures only; implementations are in Hydra/Lib)

What remains hand-written:
- `Hydra.Lib` - Native primitive implementations
- `Hydra.Sources` - DSL-based specifications (input to code generation)
- `Hydra.Dsl` - DSL syntax
- `Hydra.Generation` - I/O and generation utilities
- Test runners

See the [Bootstrap process](https://github.com/CategoricalData/hydra/blob/main/docs/src/implementation.md#the-bootstrap-process) section for details on extending Hydra.
For example:
- [Adding new primitive functions](https://github.com/CategoricalData/hydra/blob/main/docs/src/recipes/adding-primitives.md)
- [Extending Hydra Core](https://github.com/CategoricalData/hydra/blob/main/docs/src/recipes/extending-hydra-core.md)

## Troubleshooting

### Stack version warnings

If you see warnings like:
```
Stack has not been tested with GHC versions above 9.0, and using 9.10.2, this may fail
Stack has not been tested with Cabal versions above 3.4, but version 3.12.1.0 was found, this may fail
```

**Solution:** Update Stack to the latest version:
```bash
stack upgrade
# or on macOS with Homebrew:
brew upgrade stack
```

**Explanation:** These are compatibility warnings that appear when using newer GHC/Cabal versions. If your builds complete successfully, the warnings are harmless - Stack works fine with newer versions even before official testing. Upgrading Stack eliminates the warnings.