diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,10 +1,19 @@ # Changelog +## 0.2.0.0 — 2026-06-05 + +### Milestone: `import {}` Evaluates + +- **`import {}` now evaluates to WHNF** — the top-level package set resolves and enumerates ~23,000 attributes, and a package evaluates through the full stdenv bootstrap down to a derivation: `(import {}).hello.drvPath` yields a `/nix/store/…-hello-2.12.1.drv` path. (Derivation-hash parity with upstream Nix and on-Windows building are the next fronts — this milestone is evaluation, not yet building.) +- **Fix: `inherit ;` de Bruijn level in positional let/rec blocks** — `inherit x;` (without `from`) in a positional `let`/`rec` block desugars to `x = x;` with the right-hand side resolved against the *outer* scope, so it refers to the enclosing `x` rather than the binding being defined. But the desugared thunk is evaluated at runtime in the *inner* (let/rec) env, which adds one parent-chain level the outer resolution did not count — so every resolved level was short by one. This produced `nn_env_lookup_resolved: idx out of bounds` whenever an outer *lexical* variable was inherited inside a positional block — first triggered by perl's `inherit version;` (argument-set slot 7) nested in a 4-binding `let`. Fixed by shifting the desugared `inherit` RHS up one de Bruijn level in `Nix.Expr.Resolve`. (`inherit (from) …` was already correct: its RHS resolves against the inner scope.) +- **Fix: Hackage build — ship C headers in the sdist** — the `cbits/*.h` headers are `#include`d by the C sources (via `include-dirs: cbits`) but were never listed in the cabal file, so `cabal sdist` omitted them and the Hackage build failed with `nn_*.h: No such file or directory`. Added `extra-source-files: cbits/*.h`. A regression introduced in 0.1.9.0 (the first release to ship the C data layer); CI never caught it because CI builds the working tree, not the sdist. Verified by building from a freshly-extracted sdist tarball in isolation. +- 593 tests, `-Werror` clean, ormolu clean, hlint clean + ## 0.1.9.0 — 2026-06-05 ### nixpkgs Evaluation: Lazy Derivations -- **Fix: `import {}` infinite recursion — lazy `derivation` over `derivationStrict`** — `builtins.derivation` was eager: forcing a derivation to WHNF forced its entire env/input closure. This turned nixpkgs' lazy `perl` ↔ `libxcrypt` dependency cycle into a blackhole — `perl`'s `assert (libxcrypt != null)` forced `libxcrypt`'s build, which forced `perl` mid-construction. Restored Nix's two-tier model: the eager `builtins.derivationStrict` primop (identical content-hashing, renamed from the old `builtins.derivation` body) plus a lazy `derivation` wrapper over it (mirroring `corepkgs/derivation.nix`). Forcing a derivation to WHNF now yields its attribute spine without computing `drvPath`/`outPath` or forcing its inputs, so referencing a package no longer builds its closure. No C changes; `drvPath`/`outPath` hashes unchanged. The full stdenv bootstrap and package construction (perl, libxcrypt, binutils) now evaluate. +- **Fix: `import {}` infinite recursion — lazy `derivation` over `derivationStrict`** — `builtins.derivation` was eager: forcing a derivation to WHNF forced its entire env/input closure. This turned nixpkgs' lazy `perl` ↔ `libxcrypt` dependency cycle into a blackhole — `perl`'s `assert (libxcrypt != null)` forced `libxcrypt`'s build, which forced `perl` mid-construction. Restored Nix's two-tier model: the eager `builtins.derivationStrict` primop (identical content-hashing, renamed from the old `builtins.derivation` body) plus a lazy `derivation` wrapper over it (mirroring `corepkgs/derivation.nix`). Forcing a derivation to WHNF now yields its attribute spine without computing `drvPath`/`outPath` or forcing its inputs, so referencing a package no longer builds its closure. No C changes; `drvPath`/`outPath` hashes unchanged. The stdenv bootstrap now progresses through derivation construction instead of looping (next blocker: a variable-slot bug while evaluating perl's `@`-pattern). ### Technical Audit + C Data Layer Polish diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -1,273 +1,106 @@

nova-nix

-

Windows-Native Nix

-

A from-scratch Nix implementation in Haskell + C99. Parser, lazy evaluator, content-addressed store, derivation builder, binary substituter. Runs natively on Windows, macOS, and Linux. No WSL. No Cygwin.

-

Try It · CLI · Architecture · Performance · Modules · Roadmap

-

+

A Windows-native Nix, from scratch.

+

Parser, lazy evaluator, content-addressed store, derivation builder, and binary-cache substituter — in Haskell and C99. Runs natively on Windows, macOS, and Linux. No WSL, no Cygwin.

[![CI](https://github.com/Novavero-AI/nova-nix/actions/workflows/ci.yml/badge.svg)](https://github.com/Novavero-AI/nova-nix/actions/workflows/ci.yml) [![Hackage](https://img.shields.io/hackage/v/nova-nix.svg)](https://hackage.haskell.org/package/nova-nix) -![Haskell](https://img.shields.io/badge/haskell-GHC%209.8-purple) +![GHC](https://img.shields.io/badge/GHC-9.8-purple) ![License](https://img.shields.io/badge/license-BSD--3--Clause-blue) -

--- -## What is nova-nix? +## Status -| Layer | What it does | -|-------|-------------| -| **Parser** | Hand-rolled recursive descent. 14 precedence levels, 19 AST constructors, all Nix syntax including ``, `${expr}` keys, indented strings. | -| **Evaluator** | Bytecode-compiled lazy evaluation. Thunk memoization with blackhole detection. Knot-tying for recursive `let`/`rec`. Polymorphic via `MonadEval`. | -| **109 Builtins** | Arithmetic, strings, lists, attrsets, higher-order (`map`, `filter`, `foldl'`, `sort`, `genList`, `concatMap`, `mapAttrs`), JSON, hashing, regex, version parsing, `tryEval`, `deepSeq`, `genericClosure`, string contexts, IO (`import`, `readFile`, `pathExists`, `derivation`, `fetchurl`, `fetchTarball`, `fetchGit`), and more. | -| **C99 Data Layer** | 9 arena-allocated C modules for interned symbols, sorted attrsets, thunks, envs, lists, bytecode, lambdas. All eval data off the GHC heap. | -| **Store** | Content-addressed `/nix/store` (or `C:\nix\store`). SQLite metadata, reference scanning, read-only enforcement. | -| **Builder** | Dependency graph via Kahn's toposort. Binary cache substitution before local builds. Multi-output, reference scanning, store registration. | -| **Substituter** | HTTP binary cache protocol. Narinfo parsing, Ed25519 verification, NAR download/unpack. Priority-ordered multi-cache. Built on [nova-cache](https://github.com/Novavero-AI/nova-cache). | +nova-nix evaluates real nixpkgs. `import {}` resolves the top-level package set (~23,000 attributes) to weak head normal form, and a package evaluates through the stdenv bootstrap down to a derivation: -Every module is pure by default. IO at the boundaries only. +```console +$ NIX_PATH=nixpkgs=/path/to/nixpkgs \ + nova-nix eval --expr '(import { system = "x86_64-linux"; }).hello.drvPath' +"/nix/store/azg0gjls29w3sii2kjp4c1v85ka5alnp-hello-2.12.1.drv" +``` ---- +It targets the Nix 2.24 language. Evaluation is the milestone reached so far; confirming derivation-hash parity with upstream Nix, and *building* on Windows, are the work ahead. -## Try It +## Quickstart ```bash git clone https://github.com/Novavero-AI/nova-nix.git cd nova-nix -cabal run nova-nix -- --strict eval test.nix -``` - -``` -{ count = 6; greeting = "Hello, nova-nix!"; items = [ 2 4 6 8 10 ]; nested = { a = 1; b = 2; c = 4; }; types = { attrs = "set"; int = "int"; list = "list"; string = "string"; }; } -``` - -That's `let` bindings, `rec` attrs, lambdas, `builtins.map`, `builtins.typeOf`, and arithmetic — parsed, lazily evaluated, and printed. On Windows, macOS, or Linux. - ---- - -## CLI - -```bash -nova-nix eval FILE.nix # Evaluate a .nix file -nova-nix eval --expr 'EXPR' # Evaluate an inline expression -nova-nix build FILE.nix # Build a derivation -nova-nix --nix-path nixpkgs=/path eval FILE # Add search paths (repeatable) +cabal build ``` -```bash +```console $ nova-nix eval --expr '1 + 2' 3 -$ nova-nix eval --expr 'builtins.map (x: x * x) [1 2 3 4 5]' +$ nova-nix eval --expr 'builtins.map (x: x * x) [ 1 2 3 4 5 ]' [ 1 4 9 16 25 ] -$ NIX_PATH=nixpkgs=/path/to/nixpkgs nova-nix eval --expr '(import ).trivial.version' -"24.11pre-git" -``` - -```bash -$ nova-nix build hello.nix -/nix/store/abc...-hello -``` - -The `build` command evaluates, extracts the derivation, builds the dependency graph, checks binary caches, builds locally, and registers outputs. - ---- - -## Architecture - -``` - Pure Core (no IO) - +------------------------------------------------+ - | | - | Parser --> Expr.Types --> Eval --> Builtins | - | | | | - | Expr.Resolve Eval.Types | - | Expr.ClosureTrim Eval.Operator | - | Parser.Lexer Eval.Compile | - | Parser.Expr Eval.StringInterp | - | | | - | Derivation --> Hash | - | | | - | Store.Path DependencyGraph | - | | - +------------------------------------------------+ - | - IO Boundary (thin) - +------------------------------------------------+ - | Eval.IO Store.DB Store Builder Substituter | - +------------------------------------------------+ - | - C99 Data Layer (off GHC heap) - +------------------------------------------------+ - | nn_symbol nn_attrset nn_thunk nn_env | - | nn_list nn_ctxstr nn_bytecode nn_lambda | - | nn_arena | - +------------------------------------------------+ +$ nova-nix eval FILE.nix # evaluate a file +$ nova-nix build FILE.nix # build a derivation +$ nova-nix --nix-path nixpkgs=/path eval FILE.nix ``` -**Key design decisions:** - -- **Haskell owns eval logic, C99 owns data layout.** The evaluator stays in Haskell. Data structures (attr sets, thunks, envs, values) live in C. Haskell calls C to create, query, and mutate data. C never calls back into Haskell. -- **Bytecode-compiled evaluation.** The 19-constructor Expr AST compiles to a flat 24-opcode bytecode array. The bytecode evaluator (`evalBytecode`) is the sole dispatch path. The AST is GC'd after compilation. -- **MonadEval typeclass.** `PureEval` (newtype over `Either Text`) for deterministic testing. `EvalIO` for real filesystem access. Same `eval` function, different effects. -- **Arena allocation.** Thunks, attr sets, envs, and lambdas are arena-allocated in C. Bulk free at eval end, zero per-object GC overhead. The GHC heap holds only control flow and `StablePtr` handles. -- **Knot-tying via Haskell laziness.** Recursive `let` and `rec {}` use two-phase construction: allocate slots, create env, fill slots. Thunks capture environments lazily via `StablePtr` so they can reference not-yet-filled arrays. -- **Lazy derivations.** `derivation` is a lazy wrapper over the eager `derivationStrict` primop (mirroring Nix's `corepkgs/derivation.nix`). Forcing a derivation to WHNF yields its attribute spine without computing `drvPath`/`outPath` or forcing its input closure — so referencing a package (`pkg ? out`, `assert (dep != null)`) is free. -- **String context propagation.** Every `VStr` carries a `StringContext` tracking store path references. The `derivationStrict` primop collects all context into `drvInputDrvs`/`drvInputSrcs`. - ---- - -## Performance - -The C99 data layer moves all evaluation data off the GHC heap: - -| Metric | Pure Haskell | After C Data Layer | Change | -|--------|-------------|-------------------|--------| -| Max residency | 69.7 MB | 6.25 MB | **-91%** | -| GC productivity | 1.6% | 56.3% | **+35x** | -| Total memory | ~200 MB | 26 MB | **-87%** | - -Measured on a stress test with 100k attribute sets, recursive computations, list operations, and overlay patterns. - -**nixpkgs status:** `import ` evaluates correctly (450 attributes). `lib.fix`, `lib.extends`, `lib.makeExtensible`, `lib.evalModules`, and `lib.systems.elaborate` all work. The full stdenv bootstrap and package construction — perl, libxcrypt, binutils — now evaluate: `derivation` is a lazy wrapper over the eager `derivationStrict` primop (matching C++ Nix), so referencing a derivation no longer forces its build closure. Full `import {}` is currently blocked on a variable-resolution bug in named-formal-set (`@`-pattern) argument slots. - ---- - -## Windows Native - -nova-nix runs natively on Windows — no compatibility layers, no translation. - -| Platform Difference | How nova-nix Handles It | -|---------------------|------------------------| -| No `fork`/`exec` | Win32 `CreateProcess` via `System.Process` | -| No symlinks by default | Developer Mode symlinks; junction point / copy fallback | -| No `/nix/store` | `C:\nix\store` — all paths parameterized, never hardcoded | -| Case-insensitive FS | Content-addressed hashes make collisions impossible | -| 260-char path limit | `\\?\` extended-length prefix (32K chars) | -| No bash | Builder ships `bash.exe` in the store (from MSYS2, same as Git for Windows) | -| stdenv bootstrap | Windows stdenv with MinGW GCC + MSYS2 coreutils in `C:\nix\store` | - ---- - -## Modules - -### Parser - -| Module | Purpose | -|--------|---------| -| `Nix.Expr.Types` | Complete Nix AST — 18 expression constructors, atoms, formals, operators, string parts | -| `Nix.Expr.Resolve` | De Bruijn-style variable resolution — replaces `EVar` with `EResolvedVar` at parse time | -| `Nix.Expr.ClosureTrim` | Closure trimming — determines free variables per lambda/with to minimize captured env size | -| `Nix.Parser` | Hand-rolled recursive descent parser + lexer, source position tracking | -| `Nix.Parser.Lexer` | Tokenizer — integers, floats, strings with interpolation, paths, URIs, search paths, operators/keywords | -| `Nix.Parser.Expr` | Expression parser — 14 precedence levels, left/right/non-associative operators | - -### Evaluator - -| Module | Purpose | -|--------|---------| -| `Nix.Eval` | Bytecode evaluator — 24-opcode dispatch, thunk forcing, env operations, 109-builtin dispatch. Polymorphic via `MonadEval` | -| `Nix.Eval.Types` | `NixValue` (12 constructors), `Thunk` (C arena cell), `Env` (C-native struct), `AttrSet` (C sorted arrays), `MonadEval` typeclass | -| `Nix.Eval.Compile` | Bytecode compiler — Expr AST to flat `nn_bytecode` instruction array | -| `Nix.Eval.Operator` | Arithmetic with float promotion, deep structural equality, floored division | -| `Nix.Eval.StringInterp` | String interpolation, `coerceToString`, indented string stripping, float formatting | -| `Nix.Eval.Context` | String context construction, queries, and extraction for derivation building | -| `Nix.Eval.IO` | IO evaluation monad — filesystem access, import cache, process execution, per-thunk memoization with blackhole detection | -| `Nix.Builtins` | 109 builtins, search path plumbing, top-level builtin exposure | +## How it works -### C99 Data Layer +Six layers — Haskell for logic, C99 for data: -| Module | Purpose | -|--------|---------| -| `nn_symbol` | FNV-1a hash table for string interning — O(1) comparison | -| `nn_attrset` | Sorted arrays with binary search — O(log n) lookup, merge-join union | -| `nn_thunk` | 16-byte arena cells — 10 value tags for inline scalars and C-native types, blackhole detection | -| `nn_env` | 40-byte arena structs — slots, lazy scope, parent chain, with-scopes, resolved variable lookup | -| `nn_list` | Contiguous thunk pointer arrays | -| `nn_ctxstr` | Context-bearing strings with interned StorePath fields | -| `nn_bytecode` | Flat instruction array — 24 opcodes, 16-byte `nn_op_t` + data buffer | -| `nn_lambda` | Lambda closures — env ptr, body bytecode index, formals | -| `nn_arena` | Unified lifecycle — batch StablePtr cleanup, coordinated init/destroy | +1. **Parser** (`Nix.Parser`, `Nix.Expr`) — hand-rolled recursive descent; the full Nix grammar to a 19-constructor AST. +2. **Evaluator** (`Nix.Eval`) — the AST compiles to a flat 24-opcode bytecode that a lazy, thunk-memoizing evaluator runs. Recursive `let`/`rec` are knot-tied; reference cycles are caught by blackhole detection. The evaluator is polymorphic over its effect via `MonadEval` — `PureEval` for tests, `EvalIO` for the real thing. +3. **Data layer** (`cbits/nn_*.c`) — nine arena-allocated C99 modules (interned symbols, sorted attrsets, thunks, environments, lists, context strings, bytecode, lambdas) hold evaluation data off the GHC heap. Haskell calls C to create and query it; C never calls back. +4. **Store** (`Nix.Store`) — content-addressed `/nix/store` (`C:\nix\store` on Windows) with SQLite metadata and reference scanning. +5. **Builder** (`Nix.Builder`) — dependency graph, topological sort, and binary-cache substitution before local builds. +6. **Substituter** (`Nix.Substituter`) — the HTTP binary-cache protocol: narinfo parsing, Ed25519 verification, NAR download and unpack. Built on [nova-cache](https://github.com/Novavero-AI/nova-cache). -### Store + Builder +Two decisions shape the rest. **Haskell owns evaluation, C owns data layout** — bulk eval state lives off the GHC heap, so large evaluations don't thrash the collector. And **`derivation` is a lazy wrapper over the eager `derivationStrict` primop** (as in Nix's `corepkgs/derivation.nix`), so referencing a package never forces its build closure. -| Module | Purpose | -|--------|---------| -| `Nix.Derivation` | Derivation type, ATerm serialization + parsing, platform detection | -| `Nix.Store.Path` | Store path types — `StoreDir`, `StorePath`, Windows/Unix support | -| `Nix.Store.DB` | SQLite store database — WAL mode, path registration, reference queries | -| `Nix.Store` | `addToStore`, `scanReferences`, `setReadOnly`, `writeDrv` | -| `Nix.Builder` | Dependency graph, topological sort, binary cache substitution, local build | -| `Nix.DependencyGraph` | BFS graph construction, Kahn's toposort, cycle detection | -| `Nix.Hash` | Derivation hashing (`DrvHash`), SHA-256, truncated base-32, shared hash utilities | -| `Nix.Substituter` | HTTP binary cache — narinfo, Ed25519 verification, NAR download/unpack | +## Windows-native ---- +| Unix assumption | How nova-nix handles it | +|---|---| +| `/nix/store` | `C:\nix\store` — every path is parameterized, never hardcoded | +| `fork`/`exec` | `CreateProcess` via `System.Process` | +| Symlinks | Developer-mode symlinks, with junction / copy fallback | +| 260-char path limit | `\\?\` extended-length prefixes | +| A system `bash` | the builder ships `bash.exe` in the store (MSYS2) | ## Roadmap -### Done +**Done** — parser, lazy bytecode evaluator, the Nix `builtins` set, the C99 data layer, content-addressed store, derivation builder, binary-cache substituter, and `import {}` evaluation through to `hello.drvPath`. -- [x] Full Nix parser (14 precedence levels, all syntax forms) -- [x] Lazy bytecode evaluator (24 opcodes, thunk memoization, blackhole detection) -- [x] 109 builtins (matching real Nix spec) -- [x] C99 data layer (9 modules, all eval data off GHC heap) -- [x] Content-addressed store with SQLite metadata -- [x] Derivation builder with dependency resolution -- [x] Binary cache substituter with Ed25519 verification -- [x] String context tracking and propagation -- [x] nixpkgs lib layer evaluation (`lib.fix`, `lib.extends`, `lib.evalModules`) +**Next** -### Next +- Derivation-hash parity — confirm computed `drvPath`/`outPath` byte-match upstream Nix across nixpkgs. +- Windows stdenv — MinGW GCC + MSYS2 coreutils bootstrap for native builds. +- Substituter — XZ decompression and real NAR hashing (currently uncompressed-only, with a placeholder hash). -- [ ] Full `import {}` — bootstrap + package construction evaluate (lazy `derivation`); blocked on a named-formal-set (`@`-pattern) variable-slot bug -- [ ] `--system` flag — override `builtins.currentSystem` for cross-platform evaluation -- [ ] Windows stdenv — MinGW GCC + MSYS2 coreutils bootstrap for native Windows builds +## Library usage -### Long-Term +```haskell +import Nix.Parser (parseNix) +import Nix.Eval (PureEval (..), eval) +import Nix.Builtins (builtinEnv) -- [ ] `nova-nix shell` — enter a development shell -- [ ] `nova-nix repl` — interactive evaluator -- [ ] Nix daemon protocol compatibility -- [ ] XZ decompression for binary cache downloads +main :: IO () +main = case parseNix "" "let x = 5; in x * 2 + 1" of + Left err -> print err + Right expr -> print (runPureEval (eval (builtinEnv 0 []) expr)) -- Right (VInt 11) +``` ---- +The evaluator is polymorphic over `MonadEval`: `PureEval` for deterministic tests, `EvalIO` for filesystem access. -## Build & Test +## Build & test ```bash -cabal build # Build library + CLI -cabal test # Run all 593 tests -cabal build --ghc-options="-Werror" # Warnings as errors (CI default) +cabal build +cabal test ``` Requires GHC 9.8+ and cabal-install 3.10+. --- -## Library Usage - -```haskell -import Nix.Parser (parseNix) -import Nix.Eval (eval, NixValue(..), PureEval(..)) -import Nix.Builtins (builtinEnv) - -main :: IO () -main = do - case parseNix "" "let x = 5; y = x * 2; in y + 1" of - Left err -> print err - Right expr -> case runPureEval (eval (builtinEnv 0 []) expr) of - Left err -> putStrLn ("Error: " ++ show err) - Right val -> print val -- VInt 11 -``` - -The evaluator is polymorphic via `MonadEval` — `PureEval` for pure tests, `EvalIO` for filesystem access. - ---- - -

- BSD-3-Clause · Novavero AI -

+

BSD-3-Clause · Novavero AI

diff --git a/cbits/nn_arena.h b/cbits/nn_arena.h new file mode 100644 --- /dev/null +++ b/cbits/nn_arena.h @@ -0,0 +1,33 @@ +/* + * nn_arena.h — Unified arena lifecycle and StablePtr cleanup. + * + * Provides batch collection of StablePtr payloads from the thunk arena + * for efficient cleanup. After M6 bytecode integration, PENDING thunks + * hold (bc_idx, StablePtr Env) — the Expr is eliminated, replaced by + * a bytecode index. COMPUTED/NN_VALUE_PTR thunks hold StablePtr NixValue. + * + * Inline scalar/C-pointer payloads (INT, FLOAT, BOOL, NULL, STR, + * PATH, LIST, ATTRS, CTXSTR) are skipped. + * + * Used by Haskell-side Arena.hs to free all StablePtrs before + * destroying the sub-arenas (thunk, env, symbol). + */ + +#ifndef NN_ARENA_H +#define NN_ARENA_H + +#include + +/* Bulk-free all tracked attribute sets (called during arena teardown). */ +void nn_attrset_free_all(void); + +/* Count how many thunk payloads are StablePtrs (need freeing). + * Iterates all thunks, checks state + val_tag. */ +uint32_t nn_arena_stableptr_count(void); + +/* Collect StablePtr payloads into `output`. Writes at most `max_count` + * pointers. Returns the number actually written. + * Caller must allocate the output buffer. */ +uint32_t nn_arena_collect_stableptrs(void **output, uint32_t max_count); + +#endif /* NN_ARENA_H */ diff --git a/cbits/nn_assert.h b/cbits/nn_assert.h new file mode 100644 --- /dev/null +++ b/cbits/nn_assert.h @@ -0,0 +1,24 @@ +#ifndef NN_ASSERT_H +#define NN_ASSERT_H + +#include +#include + +/* + * Debug-mode bounds checking. Compiles to nothing under NDEBUG. + * Use for array accesses, index validation, and invariant checks. + */ +#ifdef NDEBUG +#define NN_ASSERT(cond, msg) ((void)0) +#else +#define NN_ASSERT(cond, msg) \ + do { \ + if (!(cond)) { \ + fprintf(stderr, "NN_ASSERT failed: %s\n at %s:%d\n", \ + (msg), __FILE__, __LINE__); \ + abort(); \ + } \ + } while (0) +#endif + +#endif /* NN_ASSERT_H */ diff --git a/cbits/nn_attrset.h b/cbits/nn_attrset.h new file mode 100644 --- /dev/null +++ b/cbits/nn_attrset.h @@ -0,0 +1,108 @@ +/* + * nn_attrset.h — Sorted symbol-keyed attribute set for nova-nix. + * + * Replaces Haskell's Data.Map.Strict (48 bytes per Bin node, pointer- + * chasing balanced tree) with a contiguous sorted array of (symbol, slot) + * pairs. Binary search for O(log n) lookup; linear scan for iteration. + * + * Construction is two-phase: + * 1. nn_attrset_new(cap) + nn_attrset_insert() — append unsorted + * 2. nn_attrset_freeze() — sort by symbol ID, enable binary search + * + * After freeze, the set is immutable (values can be updated in-place + * but keys cannot change). This matches Nix semantics: attribute sets + * are constructed once and never gain/lose keys. + * + * Values are opaque void* — Haskell passes StablePtr Thunk. The C + * side never dereferences them. + * + * Lifecycle: caller owns the nn_attrset_t* and must call nn_attrset_free. + * Values (StablePtrs) are NOT freed by nn_attrset_free — Haskell owns them. + */ + +#ifndef NN_ATTRSET_H +#define NN_ATTRSET_H + +#include "nn_symbol.h" +#include +#include + +/* --- Types --- */ + +/* Opaque attribute set handle. */ +typedef struct nn_attrset nn_attrset_t; + +/* --- Lifecycle --- */ + +/* Allocate a new empty attribute set with space for `capacity` entries. + * Returns NULL on allocation failure. */ +nn_attrset_t *nn_attrset_new(uint32_t capacity); + +/* Free the attribute set structure. Does NOT free the value pointers + * (Haskell owns those via StablePtr). */ +void nn_attrset_free(nn_attrset_t *set); + +/* --- Construction (before freeze) --- */ + +/* Append a key-value pair. Keys need not be sorted or unique at this + * stage — duplicates are resolved at freeze time (last insert wins, + * matching Nix // merge semantics). + * Grows the internal arrays if capacity is exceeded. */ +void nn_attrset_insert(nn_attrset_t *set, nn_symbol_t key, void *value); + +/* Sort entries by symbol ID and deduplicate (last writer wins). + * Must be called exactly once before any lookup/keys/values access. + * After freeze, no more inserts are allowed. */ +void nn_attrset_freeze(nn_attrset_t *set); + +/* --- Query (after freeze) --- */ + +/* Look up a key by symbol. Returns the value pointer, or NULL if + * the key is not present. O(log n) binary search. */ +void *nn_attrset_lookup(const nn_attrset_t *set, nn_symbol_t key); + +/* Return the index of a key, or -1 if not found. + * Useful for updating value slots in place. */ +int32_t nn_attrset_index(const nn_attrset_t *set, nn_symbol_t key); + +/* Update the value at a known index. Index must be valid (0 <= idx < count). + * Used for lazy materialization: slot starts NULL, gets filled on first force. */ +void nn_attrset_set_value(nn_attrset_t *set, uint32_t idx, void *value); + +/* Return the value at a known index. */ +void *nn_attrset_get_value(const nn_attrset_t *set, uint32_t idx); + +/* Return the key at a known index. */ +nn_symbol_t nn_attrset_get_key(const nn_attrset_t *set, uint32_t idx); + +/* --- Bulk access --- */ + +/* Number of entries (after freeze: unique keys). */ +uint32_t nn_attrset_size(const nn_attrset_t *set); + +/* Direct pointer to the sorted keys array (count elements). + * Valid until the set is freed. */ +const nn_symbol_t *nn_attrset_keys_ptr(const nn_attrset_t *set); + +/* --- Lifecycle: bulk cleanup --- */ + +/* Free all tracked attribute sets at once (arena-style cleanup). + * Every set created via nn_attrset_new is automatically tracked. + * Call this once at evaluation end, before destroying sub-arenas. */ +void nn_attrset_free_all(void); + +/* --- Set operations --- */ + +/* Create a new set that is the right-biased union of `a` and `b` + * (keys in `b` override `a`). Both inputs must be frozen. + * The result is returned frozen. Caller owns the result. */ +nn_attrset_t *nn_attrset_union(const nn_attrset_t *a, const nn_attrset_t *b); + +/* Create a new set with the given keys removed. + * Input must be frozen. Result is returned frozen. */ +nn_attrset_t *nn_attrset_remove_keys( + const nn_attrset_t *set, + const nn_symbol_t *keys, + uint32_t key_count); + +#endif /* NN_ATTRSET_H */ diff --git a/cbits/nn_bytecode.h b/cbits/nn_bytecode.h new file mode 100644 --- /dev/null +++ b/cbits/nn_bytecode.h @@ -0,0 +1,150 @@ +/* + * nn_bytecode.h -- Flat bytecode representation for Nix expressions. + * + * Compiles the 19-constructor Expr AST into a contiguous instruction + * array (nn_op_t[]) plus a variable-length data buffer (uint32_t[]). + * Each Expr node maps to exactly one instruction. Children are + * referenced by their index in the instruction array (post-order + * traversal guarantees children < parent). Variable-length data + * (string parts, bindings, formals, capture info, attr paths) is + * stored in the data buffer, referenced by offset from instructions. + * + * Goal: eliminate StablePtr (Expr, Env) in pending thunks. After + * compilation, pending thunks hold (bc_idx, nn_env_t*) -- both + * C-native, zero GHC heap pressure. + * + * Lifecycle: nn_bytecode_init() before evaluation, nn_bytecode_destroy() + * after. Not thread-safe -- single-threaded evaluation only. + */ + +#ifndef NN_BYTECODE_H +#define NN_BYTECODE_H + +#include + +/* --- Opcodes (one per Expr constructor) --- */ + +#define NN_OP_LIT_INT 0 +#define NN_OP_LIT_FLOAT 1 +#define NN_OP_LIT_BOOL 2 +#define NN_OP_LIT_NULL 3 +#define NN_OP_LIT_URI 4 +#define NN_OP_LIT_PATH 5 +#define NN_OP_STR 6 +#define NN_OP_IND_STR 7 +#define NN_OP_VAR 8 +#define NN_OP_WITH_VAR 9 +#define NN_OP_RESOLVED_VAR 10 +#define NN_OP_ATTRS 11 +#define NN_OP_LIST 12 +#define NN_OP_SELECT 13 +#define NN_OP_HAS_ATTR 14 +#define NN_OP_APP 15 +#define NN_OP_LAMBDA 16 +#define NN_OP_LET 17 +#define NN_OP_IF 18 +#define NN_OP_WITH 19 +#define NN_OP_ASSERT 20 +#define NN_OP_UNARY 21 +#define NN_OP_BINARY 22 +#define NN_OP_SEARCH_PATH 23 + +/* --- UnaryOp flags (NN_OP_UNARY) --- */ + +#define NN_UNARY_NOT 0 +#define NN_UNARY_NEGATE 1 + +/* --- BinaryOp flags (NN_OP_BINARY) --- */ + +#define NN_BINARY_ADD 0 +#define NN_BINARY_SUB 1 +#define NN_BINARY_MUL 2 +#define NN_BINARY_DIV 3 +#define NN_BINARY_AND 4 +#define NN_BINARY_OR 5 +#define NN_BINARY_IMPL 6 +#define NN_BINARY_EQ 7 +#define NN_BINARY_NEQ 8 +#define NN_BINARY_LT 9 +#define NN_BINARY_LTE 10 +#define NN_BINARY_GT 11 +#define NN_BINARY_GTE 12 +#define NN_BINARY_CONCAT 13 +#define NN_BINARY_UPDATE 14 + +/* --- Formal type flags (NN_OP_LAMBDA) --- */ + +#define NN_FORMAL_NAME 0 +#define NN_FORMAL_SET 1 +#define NN_FORMAL_NAMED_SET 2 + +/* --- String part tags (data buffer) --- */ + +#define NN_STRPART_LIT 0 +#define NN_STRPART_INTERP 1 + +/* --- Binding type tags (data buffer) --- */ + +#define NN_BIND_NAMED 0 +#define NN_BIND_INHERIT 1 + +/* --- CaptureInfo type tags (data buffer) --- */ + +#define NN_CAPTURE_NONE 0 +#define NN_CAPTURE_SLOTS 1 +#define NN_CAPTURE_WITH_SCOPES 2 + +/* --- Attr path element tags (data buffer) --- */ + +#define NN_ATTRKEY_STATIC 0 +#define NN_ATTRKEY_DYNAMIC 1 + +/* --- Instruction (16 bytes, naturally aligned) --- */ + +typedef struct nn_op { + uint8_t opcode; /* Which Expr constructor */ + uint8_t flags; /* Sub-type: BinaryOp, UnaryOp, formal type, etc. */ + uint16_t short_arg; /* Small immediate: count, bool flag, etc. */ + uint32_t arg1; /* Child index, symbol, data offset, lo32, etc. */ + uint32_t arg2; /* Child index, symbol, hi32, etc. */ + uint32_t arg3; /* Child index, data offset, etc. */ +} nn_op_t; + +/* --- Lifecycle --- */ + +/* Initialize the global bytecode store. op_capacity is initial + * instruction slots (0 = default 65536). data_capacity is initial + * data buffer slots (0 = default 131072). Both grow automatically. */ +void nn_bytecode_init(uint32_t op_capacity, uint32_t data_capacity); + +/* Destroy the global bytecode store, freeing all memory. */ +void nn_bytecode_destroy(void); + +/* --- Emit --- */ + +/* Append one instruction. Returns the instruction index. */ +uint32_t nn_bc_emit(uint8_t opcode, uint8_t flags, uint16_t short_arg, + uint32_t arg1, uint32_t arg2, uint32_t arg3); + +/* Append one uint32 to the data buffer. Returns the data offset. */ +uint32_t nn_bc_emit_data(uint32_t value); + +/* --- Read instructions --- */ + +uint8_t nn_bc_opcode(uint32_t idx); +uint8_t nn_bc_flags(uint32_t idx); +uint16_t nn_bc_short_arg(uint32_t idx); +uint32_t nn_bc_arg1(uint32_t idx); +uint32_t nn_bc_arg2(uint32_t idx); +uint32_t nn_bc_arg3(uint32_t idx); + +/* --- Read data --- */ + +uint32_t nn_bc_data(uint32_t offset); + +/* --- Diagnostics --- */ + +uint32_t nn_bc_op_count(void); +uint32_t nn_bc_data_count(void); + +#endif /* NN_BYTECODE_H */ diff --git a/cbits/nn_ctxstr.h b/cbits/nn_ctxstr.h new file mode 100644 --- /dev/null +++ b/cbits/nn_ctxstr.h @@ -0,0 +1,91 @@ +/* + * nn_ctxstr.h — Context-bearing strings for nova-nix. + * + * Nix strings carry a StringContext: a set of store path references + * that track derivation dependencies. Context-free strings use + * nn_thunk tag 4 (bare nn_symbol_t). Context-bearing strings use + * tag 8 (nn_ctxstr_t pointer). + * + * Each context element is one of: + * SCPlain — plain store path reference (inputSrcs) + * SCDrvOutput — derivation output (.drv path + output name) + * SCAllOutputs — all outputs of a derivation (drvPath itself) + * + * StorePaths are represented as two interned symbols (hash + name). + * + * Memory: each nn_ctxstr_t is a single contiguous allocation via + * C99 flexible array member. Headers are tracked globally for + * bulk cleanup via nn_ctxstr_free_all(). + */ + +#ifndef NN_CTXSTR_H +#define NN_CTXSTR_H + +#include + +/* --- Context element tags --- */ + +#define NN_SCE_PLAIN 0 +#define NN_SCE_DRV_OUTPUT 1 +#define NN_SCE_ALL_OUTPUTS 2 + +/* --- Types --- */ + +/* A single string context element: store path reference with kind tag. + * sp_hash and sp_name are interned nn_symbol_t values representing + * the StorePath's hash and name fields. output is an interned symbol + * for the derivation output name (only valid when tag == DRV_OUTPUT, + * 0 otherwise). */ +typedef struct nn_sce { + uint8_t tag; + uint32_t sp_hash; + uint32_t sp_name; + uint32_t output; +} nn_sce_t; + +/* A string with context. text is an interned nn_symbol_t for the + * string content. ctx_count is the number of context elements. + * ctx[] is a C99 flexible array of context elements, sorted by + * (tag, sp_hash, sp_name, output) for deterministic comparison. + * + * Single contiguous allocation: header + elements. */ +typedef struct nn_ctxstr { + uint32_t text; + uint16_t ctx_count; + nn_sce_t ctx[]; +} nn_ctxstr_t; + +/* --- Lifecycle --- */ + +/* Allocate a new nn_ctxstr_t with space for ctx_count elements. + * Elements are uninitialized — caller must fill via nn_ctxstr_set_*. + * Tracked globally for bulk cleanup. */ +nn_ctxstr_t *nn_ctxstr_new(uint32_t text, uint16_t ctx_count); + +/* Free all tracked nn_ctxstr_t allocations. Called during arena + * teardown, after thunk iteration but before env/symbol cleanup. */ +void nn_ctxstr_free_all(void); + +/* --- Element setters --- */ + +void nn_ctxstr_set_plain(nn_ctxstr_t *s, uint16_t idx, + uint32_t sp_hash, uint32_t sp_name); + +void nn_ctxstr_set_drv_output(nn_ctxstr_t *s, uint16_t idx, + uint32_t sp_hash, uint32_t sp_name, + uint32_t output); + +void nn_ctxstr_set_all_outputs(nn_ctxstr_t *s, uint16_t idx, + uint32_t sp_hash, uint32_t sp_name); + +/* --- Accessors --- */ + +uint32_t nn_ctxstr_text(const nn_ctxstr_t *s); +uint16_t nn_ctxstr_ctx_count(const nn_ctxstr_t *s); + +uint8_t nn_ctxstr_elem_tag(const nn_ctxstr_t *s, uint16_t idx); +uint32_t nn_ctxstr_elem_hash(const nn_ctxstr_t *s, uint16_t idx); +uint32_t nn_ctxstr_elem_name(const nn_ctxstr_t *s, uint16_t idx); +uint32_t nn_ctxstr_elem_output(const nn_ctxstr_t *s, uint16_t idx); + +#endif /* NN_CTXSTR_H */ diff --git a/cbits/nn_env.h b/cbits/nn_env.h new file mode 100644 --- /dev/null +++ b/cbits/nn_env.h @@ -0,0 +1,106 @@ +/* + * nn_env.h — C-native evaluation environments. + * + * Nix evaluation creates millions of environments, each with a + * variable-size array of slot pointers, optional lazy scope (attr set), + * optional parent pointer, and a with-scopes chain. + * + * All nn_env_t structs and their backing arrays are arena-allocated + * from a page-based bump allocator. Nothing is individually freed — + * nn_env_destroy() tears down everything at eval end. + * + * Lifecycle: nn_env_init() before evaluation, nn_env_destroy() after. + * Not thread-safe — single-threaded evaluation only. + */ + +#ifndef NN_ENV_H +#define NN_ENV_H + +#include + +/* --- Env struct --- */ + +/* 48 bytes on 64-bit (with padding after slot_count and with_count). + * Could be reduced to 40 bytes by reordering fields, but the current + * layout groups related fields logically and matches access patterns. */ +typedef struct nn_env { + void **slots; /* nn_thunk_t* array (or NULL for 0 slots) */ + uint32_t slot_count; /* 4 bytes + 4 pad to align lazy_scope */ + void *lazy_scope; /* nn_attrset_t* or NULL */ + struct nn_env *parent; /* nn_env_t* or NULL */ + void **with_scopes; /* array of nn_attrset_t* (or NULL) */ + uint16_t with_count; /* 2 bytes + 6 pad to struct alignment */ +} nn_env_t; + +/* --- Lifecycle --- */ + +/* Initialize the global env allocator. Must be called once before + * any nn_env_* calls (except nn_env_destroy). */ +void nn_env_init(void); + +/* Destroy the global env allocator, freeing all page memory. + * All pointers returned by any nn_env_* function become invalid. */ +void nn_env_destroy(void); + +/* --- Slot allocation (unchanged from original API) --- */ + +/* Allocate an array of `count` void* pointers. O(1) amortized. + * All slots initialized to NULL. Returns NULL if count is 0. */ +void **nn_env_alloc_slots(uint32_t count); + +/* --- Env constructors (all arena-allocated) --- */ + +/* Return a pointer to the global empty env (all fields zero/NULL). + * Valid until nn_env_destroy(). Do not modify the returned struct. */ +nn_env_t *nn_env_empty(void); + +/* Full constructor: set all fields explicitly. */ +nn_env_t *nn_env_new(void **slots, uint32_t slot_count, + void *lazy_scope, nn_env_t *parent, + void **with_scopes, uint16_t with_count); + +/* Child env with positional slots, inheriting parent's with-scopes. + * lazy_scope = NULL. */ +nn_env_t *nn_env_from_slots(void **slots, uint32_t slot_count, + nn_env_t *parent); + +/* Copy base env with a new with-scope prepended (innermost position). + * Allocates a new with-scopes array of size (base->with_count + 1). */ +nn_env_t *nn_env_push_with(nn_env_t *base, void *scope); + +/* Minimal env: slots only, no parent, no with-scopes, no lazy scope. */ +nn_env_t *nn_env_new_minimal(void **slots, uint32_t slot_count); + +/* --- Accessors --- */ + +void **nn_env_slots(const nn_env_t *env); +uint32_t nn_env_slot_count(const nn_env_t *env); +void *nn_env_lazy_scope(const nn_env_t *env); +nn_env_t *nn_env_parent(const nn_env_t *env); +void **nn_env_with_scopes(const nn_env_t *env); +uint16_t nn_env_with_count(const nn_env_t *env); + +/* --- Lookup --- */ + +/* Resolved variable lookup: walk `level` parent hops, then read + * slots[idx]. Returns the nn_thunk_t* at that position. + * Caller must ensure level/idx are in bounds (guaranteed by + * the resolution pass). */ +void *nn_env_lookup_resolved(const nn_env_t *env, int level, int idx); + +/* Walk parent chain to the root env (parent == NULL), return its + * lazy_scope. Returns NULL if the root has no lazy scope. */ +void *nn_env_root_scope(const nn_env_t *env); + +/* --- With-scopes array allocation --- */ + +/* Allocate an arena array of `count` void* pointers for with-scopes. + * All entries initialized to NULL. Returns NULL if count is 0. */ +void **nn_env_alloc_with_scopes(uint16_t count); + +/* --- Diagnostics --- */ + +/* Total bytes allocated across all pages (for memory tracking). */ +uint64_t nn_env_bytes_allocated(void); + +#endif /* NN_ENV_H */ diff --git a/cbits/nn_lambda.h b/cbits/nn_lambda.h new file mode 100644 --- /dev/null +++ b/cbits/nn_lambda.h @@ -0,0 +1,93 @@ +/* + * nn_lambda.h — Lambda closure structs for nova-nix. + * + * Stores lambda closures entirely in C: captured environment (nn_env_t*), + * body bytecode index, and formal parameter specification. Replaces + * StablePtr NixValue for VLambda on the GHC heap (~25% of remaining + * heap after M6) with a C-native tag 9 in the thunk system. + * + * Formals are stored as a type tag (Name/Set/NamedSet) plus an array + * of (symbol, has_default, default_bc_idx) entries for set patterns. + * Interned symbols avoid any string allocation — pointer equality + * for name comparison. + * + * Lambda structs are malloc'd and tracked for bulk cleanup via + * nn_lambda_free_all(). Formal entry arrays use malloc (small, typ. + * 1-20 entries). All freed at evaluation end. + * + * Lifecycle: constructed during evaluation, freed via + * nn_lambda_free_all() at arena teardown. Not thread-safe. + */ + +#ifndef NN_LAMBDA_H +#define NN_LAMBDA_H + +#include + +/* Forward declarations */ +struct nn_env; + +/* --- Formals type tags --- */ + +#define NN_FORMALS_NAME 0 /* x: body */ +#define NN_FORMALS_SET 1 /* { a, b, ... }: body */ +#define NN_FORMALS_NAMED_SET 2 /* x@{ a, b, ... }: body */ + +/* --- Types --- */ + +/* A single formal entry: name + optional default bytecode index. */ +typedef struct nn_formal_entry { + uint32_t name_sym; /* interned symbol ID */ + uint32_t has_default; /* 0 = required, 1 = has default */ + uint32_t default_bc_idx; /* bytecode index of default (valid if has_default) */ +} nn_formal_entry_t; + +/* Lambda closure: env + body + formals specification. + * Packed for minimal size (pointers first, then uint32s, then small fields). + * 28 bytes on 64-bit (no padding). */ +typedef struct nn_lambda { + struct nn_env *env; /* captured closure environment */ + nn_formal_entry_t *entries; /* formal entries (malloc'd, or NULL) */ + uint32_t body_bc_idx; /* bytecode index of lambda body */ + uint32_t name_sym; /* symbol for Name/NamedSet binding */ + uint16_t formal_count; /* number of formal entries */ + uint8_t formals_type; /* NN_FORMALS_NAME/SET/NAMED_SET */ + uint8_t allow_extra; /* 1 if ellipsis (...) present */ +} nn_lambda_t; + +/* --- Lifecycle --- */ + +/* Allocate a new lambda closure. The entries array is allocated + * internally (formal_count entries). Fill entries via + * nn_lambda_set_entry() after construction. + * Returns NULL on allocation failure. */ +nn_lambda_t *nn_lambda_new(struct nn_env *env, uint32_t body_bc_idx, + uint8_t formals_type, uint32_t name_sym, + uint8_t allow_extra, uint16_t formal_count); + +/* Set a formal entry at the given index (for construction). + * Index must be < formal_count. */ +void nn_lambda_set_entry(nn_lambda_t *lam, uint16_t idx, + uint32_t name_sym, uint32_t has_default, + uint32_t default_bc_idx); + +/* Free all tracked lambda structs (arena-style cleanup). + * Also frees all entries arrays. + * Call once at evaluation end. */ +void nn_lambda_free_all(void); + +/* --- Accessors --- */ + +struct nn_env *nn_lambda_env(const nn_lambda_t *lam); +uint32_t nn_lambda_body(const nn_lambda_t *lam); +uint8_t nn_lambda_formals_type(const nn_lambda_t *lam); +uint32_t nn_lambda_name_sym(const nn_lambda_t *lam); +uint8_t nn_lambda_allow_extra(const nn_lambda_t *lam); +uint16_t nn_lambda_formal_count(const nn_lambda_t *lam); + +/* Read formal entry fields by index (must be < formal_count). */ +uint32_t nn_lambda_entry_name(const nn_lambda_t *lam, uint16_t idx); +uint32_t nn_lambda_entry_has_default(const nn_lambda_t *lam, uint16_t idx); +uint32_t nn_lambda_entry_default(const nn_lambda_t *lam, uint16_t idx); + +#endif /* NN_LAMBDA_H */ diff --git a/cbits/nn_list.h b/cbits/nn_list.h new file mode 100644 --- /dev/null +++ b/cbits/nn_list.h @@ -0,0 +1,61 @@ +/* + * nn_list.h — Contiguous array of thunk pointers for nova-nix lists. + * + * Replaces Haskell's [Thunk] (linked-list cons cells, ~48 bytes per + * element on the GHC heap) with a flat C array of nn_thunk_t* pointers + * (8 bytes per element, cache-friendly layout). + * + * The items array is allocated via nn_env_alloc_slots (page-based bump + * allocator). The nn_list_t header is malloc'd and tracked for bulk + * cleanup via nn_list_free_all(). Both are freed at evaluation end. + * + * Lists are immutable after construction: allocate with nn_list_new, + * fill via nn_list_set, then read with nn_list_get/nn_list_count. + * + * Lifecycle: constructed during evaluation, freed via nn_list_free_all() + * at arena teardown. Not thread-safe — single-threaded evaluation only. + */ + +#ifndef NN_LIST_H +#define NN_LIST_H + +#include + +/* Forward declaration — nn_thunk_t is defined in nn_thunk.h */ +struct nn_thunk; + +/* --- Types --- */ + +/* A list: contiguous array of thunk pointers with a count. */ +typedef struct nn_list { + struct nn_thunk **items; /* contiguous array (page-allocated) */ + uint32_t count; /* number of elements */ +} nn_list_t; + +/* --- Lifecycle --- */ + +/* Allocate a new list with space for `count` thunk pointers. + * The items array is allocated via the env page allocator (O(1)). + * The header is malloc'd and tracked for bulk cleanup. + * Returns NULL on allocation failure or if count is 0. */ +nn_list_t *nn_list_new(uint32_t count); + +/* Free all tracked list headers at once (arena-style cleanup). + * Items arrays are freed by nn_env_destroy (page allocator). + * Call this once at evaluation end. */ +void nn_list_free_all(void); + +/* --- Access --- */ + +/* Number of elements in the list. */ +uint32_t nn_list_count(const nn_list_t *list); + +/* Get the thunk pointer at the given index. + * Index must be < nn_list_count(list). */ +struct nn_thunk *nn_list_get(const nn_list_t *list, uint32_t index); + +/* Set the thunk pointer at the given index (for construction). + * Index must be < nn_list_count(list). */ +void nn_list_set(nn_list_t *list, uint32_t index, struct nn_thunk *thunk); + +#endif /* NN_LIST_H */ diff --git a/cbits/nn_symbol.h b/cbits/nn_symbol.h new file mode 100644 --- /dev/null +++ b/cbits/nn_symbol.h @@ -0,0 +1,71 @@ +/* + * nn_symbol.h — Interned string symbols for nova-nix. + * + * Attribute names are the most repeated data in Nix evaluation. + * nixpkgs uses ~8k unique names across 30k+ packages, but each name + * appears hundreds of times. Interning deduplicates storage and + * replaces O(n) string comparison with O(1) integer comparison. + * + * Symbols are uint32_t indices into a global table. The table owns + * all string data in a contiguous arena (no per-string malloc). + * Lookup uses open-addressing with FNV-1a hashing. + * + * Lifecycle: nn_symbol_init() before evaluation, nn_symbol_destroy() + * after. Not thread-safe — single-threaded evaluation only. + */ + +#ifndef NN_SYMBOL_H +#define NN_SYMBOL_H + +#include +#include + +/* --- Types --- */ + +/* An interned symbol: index into the global symbol table. + * 0 is reserved as the invalid/empty sentinel. */ +typedef uint32_t nn_symbol_t; + +/* Invalid symbol constant. */ +#define NN_SYMBOL_INVALID ((nn_symbol_t)0) + +/* --- Lifecycle --- */ + +/* Initialize the global symbol table. Must be called once before + * any nn_symbol_intern() calls. initial_capacity is the expected + * number of unique symbols (hint for pre-allocation; 0 uses default). */ +void nn_symbol_init(uint32_t initial_capacity); + +/* Destroy the global symbol table, freeing all memory. + * All nn_symbol_t values become invalid after this call. */ +void nn_symbol_destroy(void); + +/* --- Core API --- */ + +/* Intern a string, returning its symbol. If the string was already + * interned, returns the existing symbol (O(1) amortized). + * The input string is copied — the caller may free it after this call. + * str must not be NULL. len is the byte length (not null-terminated). */ +nn_symbol_t nn_symbol_intern(const char *str, size_t len); + +/* Return the string data for a symbol. The pointer is valid until + * nn_symbol_destroy(). Returns NULL for NN_SYMBOL_INVALID. */ +const char *nn_symbol_text(nn_symbol_t sym); + +/* Return the byte length of a symbol's string. + * Returns 0 for NN_SYMBOL_INVALID. */ +size_t nn_symbol_len(nn_symbol_t sym); + +/* --- Comparison --- */ + +/* Symbols are equal iff their uint32_t values are equal. + * No function call needed — just use == directly. + * This macro exists for documentation purposes. */ +#define nn_symbol_eq(a, b) ((a) == (b)) + +/* --- Diagnostics --- */ + +/* Return the number of unique symbols currently interned. */ +uint32_t nn_symbol_count(void); + +#endif /* NN_SYMBOL_H */ diff --git a/cbits/nn_thunk.h b/cbits/nn_thunk.h new file mode 100644 --- /dev/null +++ b/cbits/nn_thunk.h @@ -0,0 +1,200 @@ +/* + * nn_thunk.h — Arena-allocated thunk memoization cells for nova-nix. + * + * Thunks are the core memoization mechanism in Nix evaluation. Each + * thunk starts as PENDING (holding a bytecode index + C environment + * pointer) and transitions to COMPUTED (holding the result value) on + * first force. A BLACKHOLE state detects infinite recursion. + * + * Replaces Haskell's IORef ThunkCell (~56 bytes on GHC heap, all GC- + * traced) with a 16-byte C struct in an arena (invisible to GHC GC). + * For 8M thunks: ~450 MB less GHC heap pressure. + * + * Arena: chunked bump allocator. Thunks are never individually freed — + * the entire arena is destroyed at evaluation end. Pointers into the + * arena remain valid until nn_thunk_destroy(). + * + * PENDING payloads are StablePtr Env (~16 bytes on GHC heap). The + * StablePtr is necessary for knot-tying: deferred env construction in + * recursive lets/attrs requires Haskell laziness. Freed on force. + * COMPUTED payloads are either inline scalars (int/float/bool/null via + * val_tag 0-3), C-native pointers (str/path/list/attrs/ctxstr/lambda + * via val_tag 4-9), or StablePtrs (for VBuiltin/VDrv via tag 255). + * + * Lifecycle: nn_thunk_init() before evaluation, nn_thunk_destroy() + * after. Not thread-safe — single-threaded evaluation only. + */ + +#ifndef NN_THUNK_H +#define NN_THUNK_H + +#include + +/* --- Thunk states --- */ + +#define NN_THUNK_PENDING 0 +#define NN_THUNK_COMPUTED 1 +#define NN_THUNK_BLACKHOLE 2 + +/* --- Value tags (valid when state == COMPUTED) --- */ + +#define NN_VALUE_INT 0 /* payload = (void*)(intptr_t)int64_value */ +#define NN_VALUE_FLOAT 1 /* payload = memcpy'd double */ +#define NN_VALUE_BOOL 2 /* payload = (void*)(intptr_t)(0 or 1) */ +#define NN_VALUE_NULL 3 /* payload = NULL */ +#define NN_VALUE_STR 4 /* payload = (void*)(intptr_t)nn_symbol_t (no context) */ +#define NN_VALUE_PATH 5 /* payload = (void*)(intptr_t)nn_symbol_t */ +#define NN_VALUE_LIST 6 /* payload = nn_list_t* */ +#define NN_VALUE_ATTRS 7 /* payload = nn_attrset_t* */ +#define NN_VALUE_CTXSTR 8 /* payload = nn_ctxstr_t* (string with context) */ +#define NN_VALUE_LAMBDA 9 /* payload = nn_lambda_t* (lambda closure) */ +#define NN_VALUE_PTR 255 /* payload = StablePtr NixValue (complex types) */ + +/* --- Types --- */ + +/* A thunk: mutable cell holding either a pending expression or a + * computed value. 16 bytes: state(1) + val_tag(1) + _pad(2) + + * bc_idx(4) + payload(8). val_tag is valid only when state == COMPUTED. + * + * PENDING: bc_idx = bytecode instruction index + * payload = StablePtr Env (Haskell-side, for knot-tying) + * COMPUTED: val_tag selects interpretation of payload: + * INT/BOOL: (intptr_t) cast of scalar + * FLOAT: memcpy'd double + * NULL: ignored + * PTR: StablePtr to Haskell NixValue (complex types) + * BLACKHOLE: bc_idx/payload = stale from PENDING (do not dereference) */ +typedef struct nn_thunk { + uint8_t state; + uint8_t val_tag; + uint16_t _pad; + uint32_t bc_idx; /* PENDING: bytecode index; COMPUTED: 0 */ + void *payload; +} nn_thunk_t; + +/* Nix integers are int64_t, stored inline via (intptr_t) cast into payload. + * This requires pointers to be at least 64 bits. 32-bit targets would + * silently truncate large integers. C99-compatible compile-time check: */ +typedef char nn_thunk_ptr_size_check_[(sizeof(void *) >= sizeof(int64_t)) ? 1 : -1]; + +/* --- Lifecycle --- */ + +/* Initialize the global thunk arena. initial_capacity is the number + * of thunks to pre-allocate per block (0 uses default: 65536 = 1 MB). + * Must be called once before any nn_thunk_new() calls. */ +void nn_thunk_init(uint32_t initial_capacity); + +/* Destroy the global thunk arena, freeing all block memory. + * Does NOT free payload StablePtrs — caller must iterate and free + * them first (via nn_thunk_count/nn_thunk_get). + * All nn_thunk_t pointers become invalid after this call. */ +void nn_thunk_destroy(void); + +/* --- Allocation --- */ + +/* Allocate a new PENDING thunk from the arena (legacy StablePtr path). + * pending_data is an opaque pointer (StablePtr to Haskell value). + * Sets bc_idx = 0 to distinguish from bytecode thunks. + * Returns a pointer into arena memory, valid until nn_thunk_destroy(). */ +nn_thunk_t *nn_thunk_new(void *pending_data); + +/* Allocate a new PENDING thunk with a bytecode index + env payload. + * bc_idx is the root instruction index in the bytecode store. + * env_ptr is a StablePtr Env from Haskell (required for knot-tying; + * raw Ptr would force the env at allocation time, breaking laziness). */ +nn_thunk_t *nn_thunk_new_bc(uint32_t bc_idx, void *env_ptr); + +/* Read the bytecode index from a PENDING thunk. */ +uint32_t nn_thunk_get_bc_idx(const nn_thunk_t *thunk); + +/* Set the payload of a thunk. Used for deferred env fixup in + * knot-tying (rec attrs, let, formal defaults): allocate the thunk + * with NULL payload first, then fill the env pointer once the + * knot-tied env is constructed. */ +void nn_thunk_set_payload(nn_thunk_t *thunk, void *payload); + +/* Allocate a new pre-COMPUTED thunk from the arena (StablePtr payload). + * value is an opaque pointer (StablePtr to Haskell NixValue). + * val_tag is set to NN_VALUE_PTR. */ +nn_thunk_t *nn_thunk_new_computed(void *value); + +/* Allocate pre-COMPUTED thunks with inline scalar values (no StablePtr). */ +nn_thunk_t *nn_thunk_new_computed_int(int64_t value); +nn_thunk_t *nn_thunk_new_computed_float(double value); +nn_thunk_t *nn_thunk_new_computed_bool(uint8_t value); +nn_thunk_t *nn_thunk_new_computed_null(void); + +/* Allocate pre-COMPUTED thunks with C-native complex values (no StablePtr). */ +nn_thunk_t *nn_thunk_new_computed_str(uint32_t symbol); +nn_thunk_t *nn_thunk_new_computed_path(uint32_t symbol); +nn_thunk_t *nn_thunk_new_computed_list(void *list); +nn_thunk_t *nn_thunk_new_computed_attrs(void *attrset); +nn_thunk_t *nn_thunk_new_computed_ctxstr(void *ctxstr); +nn_thunk_t *nn_thunk_new_computed_lambda(void *lambda); + +/* --- State queries --- */ + +/* Read the current state (NN_THUNK_PENDING/COMPUTED/BLACKHOLE). */ +uint8_t nn_thunk_state(const nn_thunk_t *thunk); + +/* Read the payload pointer. Meaning depends on state + val_tag. */ +void *nn_thunk_payload(const nn_thunk_t *thunk); + +/* Read the value tag (valid when state == COMPUTED). */ +uint8_t nn_thunk_value_tag(const nn_thunk_t *thunk); + +/* Read inline scalar values from a COMPUTED thunk. */ +int64_t nn_thunk_get_int(const nn_thunk_t *thunk); +double nn_thunk_get_float(const nn_thunk_t *thunk); +uint8_t nn_thunk_get_bool(const nn_thunk_t *thunk); + +/* Read C-native complex values from a COMPUTED thunk. */ +uint32_t nn_thunk_get_str(const nn_thunk_t *thunk); +uint32_t nn_thunk_get_path(const nn_thunk_t *thunk); +void *nn_thunk_get_list(const nn_thunk_t *thunk); +void *nn_thunk_get_attrs(const nn_thunk_t *thunk); +void *nn_thunk_get_ctxstr(const nn_thunk_t *thunk); +void *nn_thunk_get_lambda(const nn_thunk_t *thunk); + +/* --- State transitions --- */ + +/* Transition PENDING -> BLACKHOLE (thunk is being evaluated). + * Payload remains unchanged (caller reads it before this call). + * Returns 1 on success, 0 if thunk is not PENDING. */ +int nn_thunk_mark_blackhole(nn_thunk_t *thunk); + +/* Set a non-COMPUTED thunk to COMPUTED with a StablePtr value (NN_VALUE_PTR). + * Accepts PENDING or BLACKHOLE state (skips blackhole for direct memoization). + * Returns the old payload (pending StablePtr for caller to free). + * Returns NULL if thunk is already COMPUTED. */ +void *nn_thunk_set_computed(nn_thunk_t *thunk, void *value); + +/* Set a non-COMPUTED thunk to COMPUTED with inline scalar values. + * Returns the old payload (pending StablePtr for caller to free). + * Returns NULL if thunk is already COMPUTED. */ +void *nn_thunk_set_computed_int(nn_thunk_t *thunk, int64_t value); +void *nn_thunk_set_computed_float(nn_thunk_t *thunk, double value); +void *nn_thunk_set_computed_bool(nn_thunk_t *thunk, uint8_t value); +void *nn_thunk_set_computed_null(nn_thunk_t *thunk); + +/* Set a non-COMPUTED thunk to COMPUTED with C-native complex values. + * Returns the old payload (pending StablePtr for caller to free). + * Returns NULL if thunk is already COMPUTED. */ +void *nn_thunk_set_computed_str(nn_thunk_t *thunk, uint32_t symbol); +void *nn_thunk_set_computed_path(nn_thunk_t *thunk, uint32_t symbol); +void *nn_thunk_set_computed_list(nn_thunk_t *thunk, void *list); +void *nn_thunk_set_computed_attrs(nn_thunk_t *thunk, void *attrset); +void *nn_thunk_set_computed_ctxstr(nn_thunk_t *thunk, void *ctxstr); +void *nn_thunk_set_computed_lambda(nn_thunk_t *thunk, void *lambda); + +/* --- Arena diagnostics / cleanup iteration --- */ + +/* Total thunks allocated across all blocks. */ +uint32_t nn_thunk_count(void); + +/* Get a thunk by global index (0-based, across all blocks). + * Used for StablePtr cleanup iteration before nn_thunk_destroy(). + * Index must be < nn_thunk_count(). */ +nn_thunk_t *nn_thunk_get(uint32_t index); + +#endif /* NN_THUNK_H */ diff --git a/nova-nix.cabal b/nova-nix.cabal --- a/nova-nix.cabal +++ b/nova-nix.cabal @@ -1,17 +1,18 @@ cabal-version: 3.0 name: nova-nix -version: 0.1.9.0 +version: 0.2.0.0 synopsis: Windows-native Nix implementation in Haskell and C99 description: - A Windows-native implementation of the Nix package manager — Haskell for - evaluation logic, a C99 data layer for off-heap evaluation data. Runs - natively on Windows, macOS, and Linux — no WSL, no MSYS2, no Cygwin - required. Evaluates .nix files, builds derivations, manages a - content-addressed store, and substitutes pre-built binaries from remote - caches (nova-cache, cache.nixos.org). + A from-scratch implementation of the Nix package manager that runs + natively on Windows, macOS, and Linux — no WSL or Cygwin. A + Haskell evaluator handles parsing and lazy evaluation, backed by a C99 + data layer that keeps evaluation data off the GHC heap. It evaluates + Nix expressions (including real nixpkgs @lib@), computes derivations and + content-addressed store paths, and includes a derivation builder and + binary-cache substituter. - Built on top of @nova-cache@ for NAR serialization, narinfo handling, - Ed25519 signing, and binary substitution. + Built on @nova-cache@ for NAR serialization, narinfo handling, and + Ed25519-signed binary substitution. license: BSD-3-Clause license-file: LICENSE @@ -26,6 +27,12 @@ extra-doc-files: CHANGELOG.md README.md +-- C data-layer headers. These are #included by the cbits/*.c sources +-- (via include-dirs: cbits) but are not auto-added to the sdist by the +-- c-sources field, so they must be listed here or the Hackage build +-- fails with "nn_*.h: No such file or directory". +extra-source-files: + cbits/*.h data-files: nix/fetchurl.nix data-dir: data diff --git a/src/Nix/Expr/Resolve.hs b/src/Nix/Expr/Resolve.hs --- a/src/Nix/Expr/Resolve.hs +++ b/src/Nix/Expr/Resolve.hs @@ -221,9 +221,18 @@ resolveLetBinding _ innerStack (Inherit (Just fromExpr) names) = [Inherit (Just (resolve innerStack fromExpr)) names] resolveLetBinding outerStack _ (Inherit Nothing names) = - -- Desugar: inherit x y; → x = x; y = y; - -- RHS resolves against outer scope (before the let bindings). - [NamedBinding [StaticKey name] (resolve outerStack (EVar name)) | name <- names] + -- Desugar @inherit x y;@ → @x = x; y = y;@, resolving each RHS against the + -- outer scope so it names the enclosing binding, not the one defined here. + -- 'shiftInheritLevel' corrects for the inner env the resulting thunk runs in. + [NamedBinding [StaticKey name] (shiftInheritLevel (resolve outerStack (EVar name))) | name <- names] + +-- | A desugared positional @inherit@ RHS is resolved against the outer scope +-- but evaluated in the inner (let\/rec) env — one extra parent-chain hop — so +-- its de Bruijn level is one too shallow. Bump it. 'EVar'\/'EWithVar' are +-- name-based and need no adjustment. +shiftInheritLevel :: Expr -> Expr +shiftInheritLevel (EResolvedVar level idx) = EResolvedVar (level + 1) idx +shiftInheritLevel other = other -- | Resolve variables inside formal default expressions. resolveFormalsDefaults :: [ScopeEntry] -> Formals -> Formals