rtk-0.12: CHANGELOG.md

# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.12] - 2026-07-02

### Changed
- Widened GHC support: the boot-library bounds now span GHC 9.4 through 9.14
  (`base < 4.23`, `template-haskell < 2.25`, `containers < 0.9`,
  `time < 1.16`). The endpoints are CI-tested — the pinned 9.4.7 toolchain
  runs the full battery and the newest-GHC canary builds `-Werror` and runs
  the cabal suites on 9.14.1. 0.11 capped `base` at 4.19 (GHC 9.6), which
  made the package uninstallable on current compilers and failed Hackage's
  own doc builder (GHC 9.8). Version bounds are now also stated once per
  dependency instead of repeated across components, per Hackage guidance —
  the executable's own stale `base < 4.19` copy is exactly the drift this
  prevents

### Added
- Block layout for generated pretty-printers (task 9b), the "pretty" layer on
  top of 9a's correct-not-pretty printer. `rtk --generate-pp --pp-layout=block`
  emits a `<Name>PP.hs` that indents and line-breaks bracket-structured
  languages so the output reads like hand-written source; `--pp-layout=flat`
  (the default) keeps 9a's one-line-per-construct output byte-for-byte.
  Indentation is derived purely structurally from *block lists* — statement /
  declaration lists (an `STMany` with no separator over a statement-shaped
  element, or a `;`-separated list) — which wrap their elements in an
  indented, line-broken block. The flanking delimiters (`{` `}` for C/Java,
  `begin` `end` for PL/0) need no special handling and there is no bracket
  charset baked into rtk: they render inline and the list supplies the indent,
  so a grammar with no block lists simply degrades to flat. The generated
  module stays dependency-free (it carries an in-module ~20-line layout
  engine; no `pretty`/`prettyprinter` pulled into users' code).

  Correctness is inherited and unbreakable: layout is whitespace, every corpus
  grammar ignores whitespace, so block layout cannot change the parse —
  `parse (print ast) == ast` holds by construction. The round-trip test
  (`make test-pp`) gains block-mode cases (a small bracket grammar `block.pg`
  and the c-compiler tutorial `c.pg`) that stay green, and `block.pg` opts into
  a block-mode PP golden (`test/golden/block/BlockPP.hs`, gated through
  `TestSupport.ppGoldenGrammars`) so the indented output is pinned and
  reviewable. A forward guard (`GenPP.layoutSensitive`, `TODO(#95)`) falls back
  to flat should a layout-sensitive (offside) lexer ever land (task 12); it is
  a no-op today.

  Honest scope — this is heuristic readability for bracket/terminator
  languages, not universal beauty. `c.pg` and `block.pg` read idiomatically;
  PL/0 reads well bar a minor wart (its top-level `procedure` list, being a
  no-separator block list, indents one level); Java's statement/class bodies
  indent correctly but its top-level preamble (package/import/type decls,
  which are not a single broken list) runs together and multi-line doc-comment
  tokens disturb the indentation, so Java is best left on flat for now. All of
  these still round-trip green in block mode — only quality varies, never
  correctness. Alignment, operator-aware wrapping and configurable width are
  explicitly out of scope (later opt-in layers).
- Pretty-printer generation (task 9), the "emit" third of "Rewrite ToolKit".
  `rtk --generate-pp <grammar> <out>` emits an opt-in fifth artifact
  `<Name>PP.hs`: a module of `pp<Type>` functions that turn the AST RTK
  generates for a grammar back into source text. `--debug-pp-spec` dumps the
  printer to stdout. The generated module depends only on `base` (it imports
  its grammar's `<Name>Parser` AST and `Data.List`) — no rtk imports, no new
  packages, the same dependency discipline as the generated lexer/parser/
  quasi-quoter. The flag is off by default, so the default invocation is
  byte-for-byte unchanged and non-users see zero golden churn.

  v1 is correctness-first, not pretty: exactly one space between tokens, no
  indentation or alignment, and parenthesization is whatever the grammar's
  own structure produces (no precedence/paren-insertion engine). The single
  guarantee is the semantic round-trip `parse (print ast) == ast` — NOT
  byte-faithful reproduction; comments and the original whitespace are lost
  (the AST is lossy). That guarantee is enforced by a generated round-trip
  test over a corpus (`make test-pp`, wired into CI): it parses each fragment,
  prints, reparses, and asserts equality. 7b's position-transparent `RtkPos`
  makes the reparsed AST (different source positions) compare `==` to the
  original with no stripping, so under-parenthesization or a dropped token
  becomes a failing test rather than a silently wrong program. The grammars
  `p` and `sandbox` opt in to a PP golden snapshot (`<Name>PP.hs` in
  `test/golden/`, gated through `TestSupport.ppGoldenGrammars` and compiled
  by `make test-compile-goldens`); every other grammar stays opted out.

  Dogfooding capstone: the printer RTK generates for its own grammar language
  (`grammar.pg`) compiles and round-trips representative grammar ASTs — string
  literals, regexes, typed/func/lifted rules, repetition/option/delimiter
  forms and an imports block — so **RTK prints its own language**. Known v1
  limitation: `grammar.pg`'s *own source* does not fully round-trip, because
  its `str`/`regexplit`/`bigstr` rules use repetition over an
  un-parenthesized alternation (e.g. `([^\\'] | backslash .)*`), which a
  structural printer without a paren engine re-associates on reparse. This is
  precisely the under-parenthesization the round-trip oracle is built to
  catch; auto-parenthesization is a deliberately separate later task.

  Supersedes #84 (predated the 8a–8c AST migration and pursued a "smart"
  layout-hint design out of scope for the correctness-first v1).

### Changed
- The pipeline computes directly over the generated AST (task 8c): the
  historic `InitialGrammar`/`IRule`/`IClause` types — the hand-written
  parsed-grammar representation dating back to the original `Parser.y` —
  are retired. String-literal normalization, clause normalization and the
  lexical-clause translation in GenX now consume `GrammarParser`'s
  `Grammar`/`Rule`/`Clause` (compiled from the `test/golden/grammar/`
  snapshot), the same types the compiled-in `GrammarQQ` quotes produce, so
  normalization-level rewrites can be expressed as quasi-quoted patterns
  (task 8d). The hand-written reference front end stays as the equivalence
  oracle: `Parser.y`'s actions construct generated-AST values (same
  constructors, spines and first-symbol positions), and the AST-equality
  suite now also projects and compares the position of every node, since
  `RtkPos` is equality-transparent. The `ASTAdapter` conversion layer is
  replaced by `src/generated/Frontend.hs` (parse entry points, generated-AST
  helpers, and the now-shared token-text cleanup: both lexers keep raw token
  text and a single `cleanGrammarTokens` pass strips delimiters and
  processes escapes after parsing). Generated artifacts are byte-identical
  for every corpus grammar. The bootstrap coupling is now structural — a
  grammar.pg change that reshapes the AST breaks the in-tree build until the
  snapshot is re-accepted and the pipeline's matches are fixed; see
  "Changing the grammar language" in BOOTSTRAP.md
- Normalization diagnostics point at the offending CLAUSE instead of the
  rule header: the generated AST carries a source position on every node,
  so errors like "a lifted (,) clause cannot be mixed with other clauses",
  "repetition over the lexical rule ...", and the constructor-label checks
  (case, reserved prefixes, duplicates — including the "first used at"
  cross-reference) now report the line and column of the clause or label
  itself, still with the rule named as context

### Added
- Rewrites as quasi-quoted patterns (task 8d) — the headline: the "rewrite
  toolkit" finally means it. Every generated `<Name>QQ` now comes with a
  documented, tested rewrite recipe ("Rewriting parsed Java" in
  docs/java-quasi-quotation-tests.md, exercised by `make
  test-java-rewrite`): a rewrite is an ordinary function whose match arms
  are quasi-quoted patterns and whose results are quasi-quoted
  expressions, applied to every node of any AST value with SYB's
  `everywhere`/`extT` — the packages generated code already depends on, so
  there is no new API surface and no change to generated artifacts. The
  same patterns drive queries via `everything`/`mkQ`. rtk eats this
  cooking: its own pipeline now matches clause shapes with its own
  compiled-in quoter — `Frontend.altElems`/`seqElems` flatten the `'|'`
  and juxtaposition spines by matching `[clause| $cl1 | $cl2 |]` /
  `[clause| $cl1 $cl2 |]`, `StringLiterals` picks string literals out of
  syntax rules with `[clause| $StrLit:s |]`, and `Normalize`'s
  repetition/option shaping matches `[clause| $cl1 * ~ $cl2 |]`-style
  patterns — each conversion landed golden-neutral (artifacts
  byte-identical, bootstrap fixed point intact). Steps where the
  constructor spelling reads better (wildcard tests, scalar-`Name`
  binders, state-threading plumbing) deliberately stay plain; the
  conversion record and the discovered pattern-matching boundaries live
  in docs/qq-grammar-rewrites-plan.md §8d
- Named constructors (task 8a): an alternative may carry a leading label —
  `Expr = Add: Expr '+' Term | Term ;` — that names its generated AST
  constructor (`Add RtkPos Expr Term`) instead of the positional
  `Ctr__<Rule>__<index>` default, so code and quasi-quote patterns written
  against generated ASTs survive alternative reordering and insertion. The
  label binds tighter than `|`, scopes over one alternative's sequence, and
  works inside parenthesized groups (naming the extracted group's
  constructor, including under repetition through the list-proxy
  machinery). Unlabeled alternatives keep today's generated names
  byte-for-byte. Misuse is rejected with positioned diagnostics: names
  must start uppercase, the generated-name prefixes `Ctr__`/`Anti_` are
  reserved, explicit names must be unique across the whole grammar (all
  constructors share one generated Haskell module, and the shared-type
  constructor deduplication in GenAST would otherwise silently merge
  duplicates), and lifted (`,Rule`) alternatives — which produce no
  constructor — cannot be named, nor can anything in a lexical rule.
  Surface syntax chosen over a trailing `@ctor(Name)` annotation after
  proving LALR feasibility: the labeled grammar.pg's generated parser has
  0 happy conflicts (as before) and the reference `Parser.y` keeps exactly
  its 4 pre-existing state-6 reduce/reduce conflicts
- The generated quasi-quoter for RTK's own grammar language (`GrammarQQ`) is
  compiled into rtk, beside `GrammarLexer`/`GrammarParser` in the snapshot
  front-end component — possible since 0.11 made generated quasi-quoters
  dependency-light (`template-haskell`, a GHC boot library, is the only
  dependency the quasi-quoter adds). Grammar fragments can now be quoted
  in-tree, and the unit suite smoke-tests it: `[clause| Name * |]` parses at
  rtk's own compile time, `$cl`-style metavariables splice and bind through
  the `Anti_*` constructors in both expression and pattern context. The
  quotes produce the generated AST types, not the pipeline's
  `InitialGrammar`; migrating the pipeline onto the generated AST so
  normalization rewrites can be quasi-quoted is the follow-up (tasks 8c/8d
  in docs/qq-grammar-rewrites-plan.md)
- PL/0 compiler tutorial (`tutorials/pl0-compiler/`): the language of Brian
  Callahan's "Let's write a compiler" tutorial series, at the series'
  parts 1-3 ("validator") milestone. Baseline Wirth PL/0 — empty statements
  via `StatementOpt`, unary signs, `{ ... }` comments — generated from
  `pl0.pg` into a conflict-free Happy parser that is byte-identical under
  both front ends. Ships a validator driver (`Main.hs`), a valid/invalid
  test corpus with positioned-diagnostic checks (`run_tests.sh`), and a
  quasi-quotation test suite (`TestQQ.hs`) exercising construction,
  splicing, pattern matching with metavariables (a miniature PL/0 → C code
  generator) and SYB-based AST rewrite rules; wired into CI via
  `make -C tutorials/pl0-compiler test`
- Lisp interpreter tutorial (`tutorials/lisp-interpreter/`): Peter
  Norvig's lis.py ported to RTK — the reader is generated from a 15-line
  grammar (`scheme.pg`), and the interpreter's special-form dispatch and
  macro expansion (`let`, `when`, `unless`, `and`, `or`) are written as
  quasi-quotation patterns; `make -C tutorials/lisp-interpreter test`
  (wired into CI) runs Norvig's own test cases plus the examples, and the
  tutorial's README is a section-by-section walkthrough against the
  original essay

### Changed
- grammar.pg names every constructor-producing alternative (`RuleSimple`,
  `Star`, `Labeled`, `Ref`, ...), so the generated grammar front end's AST
  reads like prose and `src/generated/ASTAdapter.hs` no longer references
  any positional `Ctr__*` constructor — the ergonomics prerequisite for
  retiring `InitialGrammar` (task 8c). The two `?`-rules whose
  empty/present alternatives are synthesized (and therefore unnameable) —
  `ImportsOpt` and `OptDelim` — are restructured away: the imports block is
  inlined into `Grammar`'s two alternatives (`GrammarDef`/`GrammarImports`)
  and the `~` delimiter into `Star`/`StarDelim`/`Plus`/`PlusDelim`. The
  accepted language is unchanged; only grammar.pg's own AST shape moved

### Fixed
- The reference lexer (`Lexer.x`) now accepts underscores in identifiers,
  as grammar.pg's `id = [a-zA-Z][A-Za-z0-9_]*` has always specified; it
  used to stop at the first `_` with a lexical error while the generated
  (default) front end accepted the name — an invisible front-end
  divergence until label names like `Mk_1` exercised it

## [0.11] - 2026-06-12

Highlights: RTK is now self-hosting by default — grammar files are parsed
with the front end RTK generated from its own grammar description,
`test-grammars/grammar.pg` is the authoritative definition of the grammar
language, and the hand-written front end is a reference oracle selected by
`--use-handwritten`. Errors are structured, positioned diagnostics rendered
in GNU one-line style; generated ASTs carry equality-transparent source
positions. A reworked test architecture — golden snapshots, dual-front-end
equivalence, a compile gate over every snapshot — enforces all of it.
Quasi-quoters gained a `$$` escape and honest context errors; packaging
gained PVP bounds; CI a pinned, cached toolchain.

### Added
- Early-warning CI job (`ghc-latest`): builds with `-Werror` and runs the
  cabal test suites once against the newest GHC series the CI runner image
  ships (currently 9.14), relaxing only rtk's own bounds on the GHC boot
  libraries. Warnings a new GHC promotes into `-Wall` (or turns on by
  default) now surface on every push instead of at the next toolchain bump;
  the pinned primary job stays authoritative. rtk's own code is kept
  warning-clean portably: pattern matches instead of `head` (9.8 warns on
  every `head`/`tail` use, `-Wx-partial`), and the no-op `Typeable`
  derivings dropped (9.14 promotes `-Wderiving-typeable` into `-Wall`;
  explicit `Typeable` deriving has been meaningless since GHC 7.10). The
  compiled-in generated snapshot gets a scoped, GHC-gated `-Wno-x-partial`
  (generated code is not edited to placate
  linters - the lexer generator emitting `drop 1` instead of `tail` is
  tracked as follow-up work, since it churns every golden)
- Provenance banner: every generated artifact (`<Name>Lexer.x`,
  `<Name>Parser.y`, `<Name>QQ.hs`) now opens with `-- Generated by RTK from
  grammar '<Name>'. Do not edit by hand.` The banner deliberately names the
  grammar rather than the grammar file (a path would make the output - and
  the golden snapshots - depend on where the grammar lives) and carries no
  rtk version (a version would couple every release bump to a full golden
  churn)
- `make test-compile-goldens`, wired into CI after the cabal test suites:
  runs `alex -g` and `happy --ghc` over every checked-in golden
  `<Name>Lexer.x`/`<Name>Parser.y` pair and typechecks the result with
  `ghc -fno-code`. The golden suite diffs text only and sat green while the
  debug-test and t1 snapshots did not compile; the gate closes that
  test-architecture hole (the `<Name>QQ.hs` goldens are gated too, since
  generated quasi-quoters dropped their regex dependency — see Fixed)
- Source positions in generated ASTs: every constructor that generated
  parsers build (except the quasi-quotation-only `Anti_*` splice
  constructors) now stores the position of its alternative's first symbol
  in a leading `RtkPos` field. `RtkPos` is transparent for equality and
  ordering (`==` always holds, `compare` is always `EQ`), so ASTs that
  differ only in source positions compare equal — in particular a
  quasi-quote parsed at compile time still matches the same construct
  parsed at run time, and quasi-quote *patterns* wildcard every position
  field while *expressions* may embed them. Payload-carrying `%token`
  bindings now bind the whole positioned token, with generated `tkVal_*`
  extractors recovering the payload in semantic actions; a generated
  `RtkPosOf` class projects the position of any symbol (tokens,
  nonterminals, lists, optionals). Generated code stays dependency-free.
  The self-hosted front end now maps the rule constructors' positions into
  `getIRulePos` (previously `Nothing` under `--use-generated`), so
  pipeline diagnostics carry real `FILE:LINE:COL:` positions with both
  front ends, and the dual-front-end AST equality suite compares positions
  too. The hand-written parser's position for the `'.' id ':' …` rule form
  moved from the identifier to the leading dot (a rule's position is where
  the rule starts), aligning it with first-symbol capture; no corpus
  grammar uses that form
- `--debug-rule RULENAME` is back, this time with a real implementation
  (it was removed together with the other never-implemented placeholder
  flags): it traces one rule through the pipeline, printing only that
  rule's representation after each stage — its mentions in the token
  stream (with positions), the matching `IRule`s after parsing and after
  string normalization (making literal-to-`!tok_*` rewrites visible), and
  the matching rule groups/lexical rules after clause normalization and
  constructor filling — instead of full-grammar dumps. `--expand-rule`
  shows a rule's final expanded form; this flag shows its evolution. When
  the rule is missing at a stage the trace says so and lists up to five
  case-insensitive near matches (normalization renames things — `Rule_N`,
  `ListElem_*`, `tok_*`); a rule found at no stage at all fails the run
  with exit code 1, so typos don't go unnoticed in scripts. Composes with
  `--debug-stage` (stop early) and works under `--use-generated`, where
  the token stage is internal to the generated front end and the trace
  starts after parsing
- Self-hosting milestone (Prototype 2 closed): `rtk --use-generated` parses
  grammar files with the lexer/parser RTK generated from
  `test-grammars/grammar.pg`. The generated modules are compiled into rtk
  straight from the checked-in golden snapshot (`test/golden/grammar/`, the
  bootstrap stage produced by the previous rtk binary, kept current by `make
  accept-golden`), and `src/generated/ASTAdapter.hs` converts the generated
  AST to the hand-written `InitialGrammar`; everything after parsing is the
  same shared pipeline. The golden suite now runs every grammar in
  `test-grammars/` through BOTH front ends and requires byte-identical
  artifacts, and the unit suite asserts AST equality (positions stripped)
  for every grammar. The fixed point holds: `rtk --use-generated
  test-grammars/grammar.pg out/` regenerates `test/golden/grammar/` exactly.
  Accepted divergences (documented in BOOTSTRAP.md): errors carry positions
  in the message text rather than structured diagnostics, `getIRulePos` is
  not captured (both converge in task 7b), no nested `/* /* */ */` comments
  (#25), no concatenation of adjacent `"""…"""` blocks. The harness also
  surfaced two constructs the hand-written parser accepts beyond
  grammar.pg's own definition of the language — empty alternatives
  (`Gd = | ExpI ;` in haskell.pg) and redundant parentheses kept as
  semantic grouping (`(ImportStatement)*` in java.pg, `(A B) C` in t1.pg) —
  so those three grammars are pinned in the test suites
  (`frontEndDivergentGrammars`) with a guard that fails once they stop
  diverging; resolving which front end defines the language is follow-up
  work
- Generated parsers, lexers and quasi-quoters report errors with `Either`
  instead of throwing (the deferred "Stage G" of the diagnostics
  migration): generated parsers use `%monad { Either String }`, so
  `parse<Name> :: [PosToken] -> Either String <AST>` mirrors the shape of
  the hand-written grammar parser; generated lexers expose
  `scanTokens :: String -> Either String [PosToken]` (`alexScanTokens`
  stays as a throwing compatibility wrapper); and the generated
  quasi-quoters route lexer, parser and unknown-metavariable failures
  through `fail` in `TH.Q`, so a bad quasi-quote is now a positioned GHC
  compile error at the splice site instead of a runtime crash during
  Template Haskell expansion. Generated code stays dependency-free
- Duplicate rule definitions are rejected: defining the same rule name twice
  in a grammar is now a normalization error carrying both source positions
  (e.g. `g.pg:3:1: error: in rule 'Foo': rule 'Foo' is defined more than
  once (first definition at line 2, column 1)`), instead of the definitions
  being silently merged into one rule group with extra alternatives
  (closes #20). `test-grammars/debug-test.pg`, the one grammar that relied
  on this, now defines `IfStatement` once with `|` alternatives
- A `$$` escape in quasi-quote bodies: `$$name` now produces the literal
  text `$name` instead of being rewritten as a metavariable (or rejected).
  Previously the generated quasi-quoter rewrote `$ident` everywhere in a
  quote body — including inside the quoted language's own string literals —
  mangling programs. Each `$$` pair directly before a metavariable stands
  for one literal `$` (so `$$$x` is a literal `$` followed by the
  metavariable `$x`). The unknown-metavariable error now names the
  offending `$name`, lists the known shortcuts and points at the `$$`
  escape (see `docs/why-qq-limitations.md`)
- Hackage packaging hygiene: PVP version bounds on all dependencies and on
  the `alex`/`happy` build tools, `Tested-With` now lists GHC 9.4.7 and
  9.6.4 (the versions actually exercised locally and in CI), and the
  packages required by RTK-generated code are documented in the README and
  the cabal description
- Structured diagnostics: grammar errors are now `Diagnostic` values
  (message plus optional source position and context) threaded through
  `Either`, instead of being thrown with `error`. The grammar-processing
  pipeline functions (`scanTokens`, `parse`, `normalizeTopLevelClauses`,
  `genX`/`genY`/`genAST`/`genQ`) are total for user-caused failures
- One-line GNU-style error reporting: `rtk` prints `FILE:LINE:COL: error:
  MESSAGE` to stderr and exits 1 for a bad grammar, with no Haskell exception
  or call stack (closes #23 and #31)
- A lifted (`,`) clause under `*`/`+`/`?` (e.g. `Foo = ,Bar* ;`) is now
  rejected with a clear error during normalization, instead of silently
  generating broken code
- A reference to an unknown rule now names both the unknown rule and the type
  that references it
- Source positions on tokens: the lexer now returns `PosToken` values
  (token plus line/column), both in the hand-written lexer and in all
  generated lexers
- Parse errors report line, column and a human-readable description of the
  unexpected token (e.g. `Parse error at line 2, column 1: unexpected
  identifier 'Foo'`); generated parsers report positions too instead of
  dumping the remaining token list
- Errors at end of input carry the position where the input ended, in both
  the hand-written and generated parsers
- Grammar normalization errors name the offending rule and its source
  position (e.g. `Grammar error in rule 'Foo' (at line 2, column 1): ...`)
- Lexer-generation errors name the lexical rule they occur in

### Changed
- **RTK is now self-hosting by default — the headline of this release.**
  Grammar files are parsed with the front end RTK generated from its own
  grammar description, and `test-grammars/grammar.pg` is the authoritative
  definition of the grammar language. The hand-written `Lexer.x`/`Parser.y`
  are demoted to a reference oracle selected with the new
  `--use-handwritten` flag (`--use-generated` remains accepted and is now a
  no-op); the equivalence harness keeps holding both front ends to identical
  artifacts and equal ASTs for every corpus grammar, and the bootstrap fixed
  point now holds for a *default* invocation: `rtk test-grammars/grammar.pg
  out/` reproduces `test/golden/grammar/` byte-for-byte. Changes to the
  grammar language land in grammar.pg (plus regenerated goldens) first; the
  hand-written files follow only to keep the harness green. See BOOTSTRAP.md
- Front-end error parity (prerequisite of the default flip): generated
  lexers and parsers now encode failures as machine-splittable
  `LINE:COL:message` (previously prose like `Parse error at line L, column
  C: …`), the shared `Diagnostics.diagnosticFromPositioned` splits the
  encoding into a positioned diagnostic, and rtk renders the same GNU-style
  `FILE:LINE:COL: error: …` line under both front ends — for lexical errors
  the stderr line is identical character for character; for parse errors the
  position is identical and only the token wording differs (generated
  parsers render tokens generically). Generated quasi-quoters re-render the
  encoding human-readably (`line L, column C: …`) in their `fail` path, as
  do the standalone test drivers. Every golden `<Name>Lexer.x`/`<Name>Parser.y`
  changed uniformly with the new error templates
- The two historic front-end divergences are resolved and the
  pinned-divergence list (`TestSupport.frontEndDivergentGrammars`) is empty:
  the hand-written reference parser now defines the same language as
  grammar.pg — it rejects empty alternatives (`R = | X ;`) and lifts
  redundant parenthesis groups exactly like grammar.pg's
  `Clause5 = '(' ,Clause ')'` (single-leaf groups collapse, pure-sequence
  groups merge at the head of a sequence, alternation groups merge as the
  first alternative); haskell.pg's `Gd = | ExpI ;` was rewritten as the
  equivalent `Gd = ExpI? ;`. java.pg, t1.pg and haskell.pg artifacts changed
  accordingly (fewer proxy sub-rules, e.g. java.pg's `(ImportStatement)*`
  now generates a plain list rule)
- The core pipeline data types (`InitialGrammar`, `IClause`,
  `NormalGrammar`, …) moved from the footer of the hand-written `Parser.y`
  into the new front-end-agnostic `Syntax` module; a demoted parser no
  longer owns the project's core types. Pure move, no behavior change
- `--debug-tokens` in the default (generated) front end now dumps the
  generated lexer's token stream (it previously printed nothing under
  `--use-generated`)
- The Java grammar parses exactly ONE `ModifierList` per declaration. The
  class/interface/enum/`@interface` rules no longer begin with their own
  nullable list; `TypeDeclRest` (ex-`NestedTypeDeclaration`, now used at the
  top level too) carries everything after the modifiers, and the list lives
  on the enclosing rule: `TypeDeclaration = OptDocComment ModifierList
  TypeDeclRest`, `FieldDeclaration = OptDocComment ModifierList
  (MemberDeclaration | TypeDeclRest | StaticInitializer) | ';'`. The nested
  nullable lists used to produce 14 shift/reduce conflicts (after the outer
  list, every modifier/annotation token chose between extending it and
  epsilon-starting the inner one) whose shift resolution parsed
  `public class A {}` with `public` on the OUTER list and a confusing,
  always-empty inner list; that always-empty `ModifierList` field is gone
  from the class/interface/enum/annotation AST constructors, and the
  generated quasi-quoter renames `nestedTypeDeclaration` to `typeDeclRest`.
  Together with the restored dangling-else conflict (see Fixed) the
  generated `JavaParser.y` is down to 18 shift/reduce + 0 reduce/reduce
  conflicts (from 32 + 0), and java.pg now opens with a complete inventory
  of all 18 - each family with its automaton items and why the shift is the
  correct Java reading (dangling else; catch/finally attach to the nearest
  try; member id vs empty TypeParameters; greedy CompoundName `.`;
  bracket-list `[` shifts; `<` commits to type arguments; the QQ bootstrap
  dummy bracket) - replacing the stale "~14"/"~13"/"32" conflict comments.
  The commons-lang corpus success sets are unchanged (14/16 main/tests
  files), so the parser blacklists stay as they are
- The makefile test pipeline now runs alex with `-g` (the GHC backend), so
  generated lexers in `test-out/` compile as compact string-encoded tables
  instead of ~half-a-million-line pattern matches. GHC's compile of the
  Java lexer drops from minutes to seconds with identical token streams
  (verified against the `test-lex-java` goldens). The library's own lexers
  already used `-g` via cabal's preprocessor; this closes the gap for the
  makefile-driven tests (#27)
- The `$Type:var` splice alternative is now attached to a minimal set of
  rules of a shared-type group instead of to every rule. The normalizer
  builds the group's lift graph (rule A → rule B when B has an alternative
  that is exactly the single nonterminal A, ignoring nullable clauses) and
  greedily picks attach points whose unit-closure covers every rule the
  grammar demands from some position; a splice token reduces at an attach
  point and climbs the chain to the level its position requires. For
  java.pg's 18-rule Expression chain the single attach point is
  `PrimaryNoPostfix`, which removes the 806 reduce/reduce conflicts
  (of 883) the per-rule alternatives used to cause, along with the three
  "rule ... is unused" happy warnings, and makes the parse of a splice
  independent of conflict-resolution accidents. Types declared by a
  single rule are unaffected. A
  group whose demanded rules cannot be covered from one attach point gets
  several; if their closures overlap, the overlap can reintroduce
  reduce/reduce conflicts between the splice reductions (inherent to such
  grammars; normalization has no warning channel to report them)

### Removed
- The `--debug-format` option: its only honest format was the `pretty`
  default - `compact` merely stripped newlines into an unreadable single
  line (and the advertised `json`/`tree` formats had already been removed
  as never-implemented). Debug output is always pretty-printed now;
  `DEBUG_OPTIONS.md` and `test-debug-options.sh` follow
- The orphan `PrintGrammar` module: a dead pretty-printing draft predating
  the current pipeline, listed in the cabal file but imported by nothing
  (pretty-printing returns properly as a tracked feature)
- The makefile's `Windows_NT` branch: its `dist/build/rtk/rtk.exe` was the
  cabal-v1 path, dead for years - the makefile is Unix-only and now says so
- The superseded textual bootstrap comparison: `compare-bootstrap.sh`, the
  `make test-bootstrap` target and its informational CI step. Behavioral
  equivalence of the two front ends is enforced by the golden/unit harness
  on every test run; textual identity of the generated `.x`/`.y` with the
  hand-written ones was never the goal
- Unimplemented CLI options that were advertised in `--help` but had no
  effect: `--debug-rule`, `--compare-stages`, `--memory-stats`,
  `--debug-output-dir`, `--debug-log`, `--interactive`, the placeholder
  `json`/`tree` debug formats, and the `--use-generated` stub that only
  printed an error

### Fixed
- Generated quasi-quoters' `quoteType`/`quoteDec` no longer lie: they used
  to silently return dummies (`TH.ListT` / `[]`), so a quote in a type or
  declaration context compiled into nonsense. Both now `fail` in `TH.Q`,
  making the misuse a GHC compile error at the splice site ("this
  quasi-quoter cannot be used in a type/declaration context")
- A `$name` metavariable at the end of a line (or of the whole quote body)
  silently stayed unrewritten, failing with a confusing parse error at the
  `$`: the generated quasi-quoter's metavariable scan used regex-posix,
  whose default newline option made the terminator class never match `\n`.
  The scan is now a plain character function with the documented semantics
  preserved (`$$` escapes, explicit `$Type:name` passthrough, the
  unknown-metavariable error). Generated QQ modules consequently import no
  regex packages at all — `regex-posix`/`regex-base` (which were never
  declared rtk dependencies and compiled only as transitive flukes) are
  gone from every user's generated-code dependency footprint, and the
  README/cabal generated-code dependency lists shrank accordingly
- Writing into a missing output directory no longer escapes as a raw
  `IOException`: `rtk g.pg /tmp/does/not/exist` now creates the directory
  and succeeds; IO failures that remain (no permission, a file where the
  directory should be) are rendered as the usual one-line GNU-style
  diagnostic and exit 1
- A literal backslash inside a `[...]` character class is now emitted
  Alex-escaped (`\` → `\\`) by the lexer generator; it used to pass through
  raw, where Alex set syntax reads it as an escape, forcing grammars onto
  the `[\x5C]` hex spelling (issue #95's smaller finding). The `\n \t \r \f
  \v` pairs that token post-processing preserves keep passing through bare.
  java.pg drops its hex workarounds for a `backslash` charset macro and a
  plain `[^\"\\\n\r]` negated class; generated Java token streams are
  byte-identical (the `test-lex-java` goldens are untouched). One
  grammar-lexer limitation remains, documented at the macro definition: a
  class ending in a backslash must also end its source line, because `\]`
  lexes as an escape pair
- `GenAST`'s unreachable backstop for lifted clauses claimed "lifted rules
  are not yet implemented" - a missing feature it is not; reworded to the
  `Internal error (GenAST): …` convention (Normalize rejects or filters
  every lifted position before AST generation)
- A type declared only through rule annotations (`Thing : Item = …` with no
  rule named `Thing`) could not be referenced: a plain `S = Thing ;`, a list
  element `Thing* ~ ','`, or merely the QQ start wrapper (which references
  every public type by name) failed generation with `reference to unknown
  rule 'Thing'`. rtk now synthesizes the cover rule that grammar authors
  wrote by hand for this (java.pg's
  `Expression : Expression = AssignmentExpression ;`), in lifted form —
  `Thing : Thing = ,Item1 | ,Item2 ;` — so the type gets a nonterminal
  without an extra AST constructor. Synthesis is demand-driven (a grammar
  that never references the bare type is generated byte-identically) and
  happens on the parsed grammar before normalization, so the cover takes
  part in the splice attach-point machinery like a hand-written one: the
  `$Thing:var` alternative lands on one annotated rule and climbs to the
  type through the cover, and the covered type gets a top-level quoter.
  LALR conflicts of the synthesized cover are exactly those of its
  hand-written equivalent (overlapping FIRST sets among the annotated
  rules remain the grammar's own, and happy still reports them). The
  remaining unresolvable case — referencing a lexical rule's *value* type,
  which declares no syntax type — keeps a diagnostic that now explains the
  type-named-rule convention. New corpus grammar `test-grammars/i14.pg`
  pins all reference shapes in the golden suite and compile gate, and
  `make test-i14-qq` (wired into CI) exercises quoters and splices against
  the covered types (#14)
- A grammar whose start rule is a repetition (`Start = Item* ;`) generated a
  parser that did not compile: the QQ start-wrapper alternatives extended
  the start group as a `data` declaration while the repetition typed the
  same nonterminal as the alias `[Item]` (plus a dummy-token shift/reduce
  conflict against the empty list). Such an alias start group now skips the
  QQ entry-point machinery entirely — no dummy tokens, wrapper alternatives
  or top-level quoters, since an alias has no constructor to project a quote
  through — while element-level `$Type:var` splices keep working. Groups
  merely *referenced* by a wrapper alternative (grammar.pg's
  `RuleList = Rule*`) were always well-typed and are unchanged. The
  debug-test golden, uncompilable until now, is regenerated (#34)
- Repetition over a terminal silently generated uncompilable code:
  `Foo = 'x'* ;` emitted a degenerate self-recursive `data Foo` against
  list-building parser actions, and `Foo = num* ;` an invalid lowercase
  `data num`. Decided as by-design and rejected during normalization with a
  positioned diagnostic: a list element must be a syntax rule, because the
  element type hosts the list's splice constructor — wrap the terminal in a
  syntax rule and repeat that rule instead. A lifted `(,X)` element under
  `*`/`+` is rejected the same way instead of crashing the generator (#28)
- The synthesized QQ start wrapper is now emitted directly adjacent to the
  rule it wraps: list-element proxy rules could separate the two same-named
  rule blocks in the generated `.y`, which happy rejects outright
  ("Multiple rules for 'A'", t1.pg's golden was affected)
- A brace-less nested `if` parses again: `IfStatement`'s then-branch was
  `StatementWithoutIf`, so valid Java like `if (a) if (b) f(); else g();`
  was rejected with "unexpected 'if'". The then-branch is a full `Statement`
  and `StatementWithoutIf` is inlined away (`Statement` absorbs its
  alternatives; nothing else referenced it). This re-surfaces the classic
  dangling-else shift/reduce conflict, which happy resolves by shifting:
  `else` binds to the NEAREST `if`, exactly the JLS 14.5 rule (verified on
  the printed AST; regression coverage via
  `test-grammars/java/test-nested-if.java` / `make test-java-nested-if` and
  a quasi-quotation construction case). The exclusion never even avoided
  that conflict - it already existed through loop bodies in then-position,
  e.g. `if (a) while (b) if (c) f(); else g();` - it only rejected the
  direct nested form
- The Java grammar's last structural LALR ambiguity - deciding between a
  type and an expression after a `CompoundName` at statement start or after
  `(` - is resolved JLS-style (`happy` now reports 0 reduce/reduce
  conflicts on the generated `JavaParser.y`, down from 2). Types and
  declarators take only empty bracket pairs (`Dims`/`NonEmptyDims`,
  anchored on the unreduced name so `[` is a plain shift); only array
  creation takes sizing expressions; `CastExpression` uses the JLS trick
  (`'(' Expression ')' UnaryExpressionNotPlusMinus` for reference casts
  plus explicit primitive/generic/array alternatives); and array access is
  anchored on `CompoundName` in `PrimaryNoPostfix`. `a[0] = 1;`,
  `x = (a);`, `x = (Foo) y;`, `x = (List<String>) y;`, `((Map) c).get(k);`
  and `new int[3][];` now parse; JLS-invalid `int[3] x;` declarations are
  now rejected. Known residual limitation (shared by LALR Java parsers
  generally): `(a < b)` as a parenthesized primary mis-commits to a
  generic cast on `<` (conditions like `if (a < b)` are unaffected)
- `make test-p` works again: the target had a hand-rolled recipe that
  compiled the driver with bare `ghc` outside the cabal package environment,
  failing with `Could not find module 'Data.Generics'` in clean
  environments. It now uses the generic `make-test-rule` like its sibling
  targets (`cabal exec -- ghc --make -itest-out ...`); the driver binary is
  consequently named `test-out/p-main` instead of `test-out/p-rtk`. The
  compile failure had masked a latent bug in the driver itself:
  `p-main.hs`'s `subst` bound `e1 = subst id e1 i` etc., self-referential
  recursive lets that shadow the pattern variables and diverge (`<<loop>>`)
  on the first fold expression, and it had no base case. The lets now bind
  fresh names and a catch-all clause terminates the recursion, so the
  driver prints the substituted AST
- `'\f'` and `'\v'` escapes in grammar string and `[...]` regex literals now
  reach the generated lexer as bare Alex escapes; previously token
  post-processing (`unBackQuote`) stripped them to the literal letters
  `f`/`v` even though `GenX.isAlexEscape` was ready to emit them, so the two
  escape sets disagreed. The preserved set (`\n \t \r \f \v`) is now pinned
  to `isAlexEscape` by a unit test
- `test-grammars/haskell.pg` is self-consistent again: minimal `Pat` and
  `QOp` rules were added for the previously dangling references, so RTK
  generation no longer aborts on it (progress on issue #30; the grammar is
  still far from full Haskell). The haskell special cases in the test suites
  were dropped and golden snapshots added
- A grammar whose first rule is lexical (or has a data-type annotation
  different from the rule name) no longer crashes with `fromJust: Nothing`;
  it reports that the first rule must be a syntax rule, or resolves the
  annotated type correctly
- Internal `fromJust` calls in code generation replaced with descriptive
  internal-error messages
- Invalid clauses in lexical-rule macros now abort generation with an error
  instead of writing the error text into the generated lexer
- User-facing errors no longer print a GHC call stack
- `--debug-stage` now exits with a success status after stopping at the
  requested stage instead of reporting failure via `error`
- `--profile-stages` timings now force each stage's result to normal form,
  so per-stage durations are no longer skewed by lazy evaluation

### Documentation
- Replaced the README "Grammar Format" example, which used Happy-style
  semantic actions that RTK cannot parse, with a verified `.pg` example
- Removed the stale `Claude.MD` (a case-colliding near-duplicate of
  `CLAUDE.md`; its Quick Reference table was folded into `CLAUDE.md`) and the
  stray root `test-simple-return.java` duplicate

## [0.10] - 2025-12-03

### Added
- MIT license with generated code exemption
- Full Java grammar support with comprehensive parsing tests
- Quasi-quotation support for embedding parsed syntax in Haskell
- Debug options for grammar development and troubleshooting
- Bootstrap self-hosting capability (RTK can parse its own grammar format)

### Fixed
- Alex escape sequence generation in GenX.hs
- Java grammar lexer patterns for complete test coverage

## [0.9] - Initial Development

### Added
- Core grammar specification format (.pg files)
- Alex lexer generation (GenX.hs)
- Happy parser generation (GenY.hs)
- AST generation (GenAST.hs)
- Quasi-quotation generation (GenQ.hs)
- Grammar normalization and transformation