pandoc-filter-indent 0.3.2.0 → 0.3.3.0

raw patch · 21 files changed

+1494/−682 lines, 21 filesdep +ghc-lib-parserdep ~HaTeXdep ~blaze-htmldep ~blaze-markup

Dependencies added: ghc-lib-parser

Dependency ranges changed: HaTeX, blaze-html, blaze-markup, containers, ghc-syntax-highlighter, optics-core, optics-th, optparse-applicative, pandoc-types, skylighting, text

Files

• ChangeLog.md +56/−1
• DESCRIPTION.md +558/−0
• LICENSE +1/−1
• README.md +60/−513
• app/Main.hs +9/−8
• pandoc-filter-indent.cabal +43/−39
• src/Alignment.hs +12/−12
• src/Filter.hs +0/−1
• src/FindColumns.hs +15/−16
• src/Render/ColSpan.hs +2/−3
• src/Render/Debug.hs +1/−1
• src/Render/HTML.hs +4/−1
• src/Render/Latex.hs +123/−28
• src/Token.hs +13/−13
• src/Token/Haskell.hs +15/−1
• src/Token/Skylighting.hs +109/−29
• src/Tuples.hs +1/−0
• src/Util.hs +2/−3
• test/LambdaSpec.hs +298/−0
• test/Spec.hs +84/−12
• test/UnicodeOperatorSpec.hs +88/−0

@@ -1,6 +1,61 @@ # Changelog for pandoc-filter-indent -## Unreleased changes+## 0.3.3.0 Nov 17 2025++### Unicode and Operator Support+  - Add comprehensive support for GHC UnicodeSyntax extension operators+    - Unicode lambda (λ) now correctly renders as `\lambda`+    - All GHC UnicodeSyntax operators now supported: ∷, ⇒, →, ←, ⤚, ⤙, ⤜, ⤛, ★, ∀, ⦇, ⦈, ⟦, ⟧, ⊸+    - ASCII and Unicode versions of operators produce identical LaTeX output+  - Fix unicode lambda (λ) character handling in both Haskell and Skylighting tokenizers+    - Add `splitUnicodeLambda` to handle tokens like "λx" → ["λ", "x"]+  - Fix lambda expressions rendering in LaTeX math mode+  - Fix backslash operators (\, \\, \/, /\, etc.) rendering+  - Add preprocessing for Skylighting tokenizer to handle split backslash operators+  - Add `fixBracketOperators` to merge GHC bracket operators: (|, |), [|, |]+  - >< as Cartesian product (\times)+  - :-> as \longmapsto+  - **Design choice**: `*` renders as `\times` (multiplication) since modern Haskell uses `Type` for kinds+    - Unicode `★` renders as `\star` for legacy kind annotations+    - Tokenizers cannot distinguish between kind and multiplication contexts+    - TODO: Make this configurable for projects with legacy kind syntax++### Tokenizer Differences and Solutions+  - **Root cause identified**: Haskell and Skylighting tokenizers produce different token types+    - Haskell tokenizer: most operators → TOther+    - Skylighting tokenizer: some operators → TOperator (e.g., *, >-, -<, >>-, -<<)+    - Unicode ∀: TKeyword (Haskell) vs TOther (Skylighting)+    - Unicode λ: TVar (Haskell) vs TOther (Skylighting)+  - **Solution**: Introduced `OpOrOther` pattern synonym using PatternSynonyms extension+    - Eliminates code duplication when handling same operator from both tokenizers+    - Cleaner syntax: `formatToken (OpOrOther "->") = mathop "to"`++### Bug Fixes+  - Fix `extraColumns` and `tableColumns` deduplication mismatch+    - Both now use text column position deduplication to avoid gaps in table column indices+    - Fixes `prop_tableColumns` test failure on inputs like "\n a"+  - Rename subscripts to subAndSuperscripts with proper superscript support++### Testing+  - Add comprehensive test suite for lambda expressions and backslash operators+  - Add unit tests for all GHC UnicodeSyntax operators (UnicodeOperatorSpec)+    - Tests verify ASCII and Unicode versions produce identical LaTeX output+    - Tests now fail on error instead of just warning (enforced with error function)+  - Add tokenizer comparison tests (TokenizerTest)+  - Refactor test suite: extract common test pattern into helper function+  - Improve test readability: reduce code duplication in property tests++### Code Quality+  - Code cleanup: remove unused LANGUAGE pragmas, add missing pragmas+  - Code cleanup: use newtype for single-field records, use concatMap and elemIndex+  - Code cleanup: remove redundant $ operators and brackets+  - Code cleanup: fix pattern matching style (otherwise -> _)+  - Code cleanup: use hPrint instead of hPutStrLn . show++### Documentation+  - Add UNICODE_OPERATORS.md explaining Unicode operator support, design decisions, and limitations+  - Document tokenizer differences and solutions+  - Document the `*` vs `★` rendering choice with rationale  ## Release history 

@@ -0,0 +1,558 @@+---+title: "Code typesetting made simple"+subtitle: "Pandoc filter"+description: |+  To build this document use:++  > pandoc README.md  -o README.pdf --filter=pandoc-filter-indent++date: |+  `\today`{=latex}+abstract: |+  Program code has become a prime medium for communicating important algorithmic and mathematical ideas, as+  indicated by unwavering popularity of functional pearls and multitude blog posts using literate programming style+  to illustrate key ideas.  However the literate programming systems are either appallingly complex or provide+  only limited functionality to emphasise code structure.  We propose extremely simple code typesetting tool that+  is also a Pandoc filter, and can thus be used to improve comprehensibility of the code. It is also simple enough+  to be provided as a literate program within this submission. While it is processing Haskell code, we show how it+  can be easily adapted to typeset Python, Java or C/C++.+author: Michał J. Gajda++#  - name: Michał J. Gajda+#    email: mjgajda@migamake.com+#    orcid:       "0000-0001-7820-3906"+#    affiliation: 1+affiliation:+  institution: "Migamake Pte Ltd"+  email:        mjgajda@migamake.com+  url:         "https://migamake.com"+review: true+numbersections: true+header-includes:+  - |-2+    \usepackage{graphicx}+    \DeclareUnicodeCharacter{03B1}{\ensuremath{\alpha{}}}++prologue:+  - |-2+    \usepackage{graphicx}+    \DeclareUnicodeCharacter{03B1}{\ensuremath{\alpha{}}}+    \renewcommand{\longeq}{\scalebox{1.7}[1]{=}}++---++# Introduction++Program code has become a prime medium for communicating important algorithmic and mathematical ideas, as+indicated by unwavering popularity of functional pearls and multitude blog posts using literate programming style+to illustrate key ideas.  However the literate code typesetting systems like lhs2TEX[@lhs2tex]+are either appallingly complex, language-specific, or provide only limited functionality to emphasise code structure.++We propose a simple approach that can be used not just for code in languages with syntax defined by indentation-defined+(like Haskell or Python), but also for code that has been formatted by indentation (like many Java and C/C++ projects mandate).++# Algorithm++We use `pandoc` to process only code fragments in otherwise unprocessed literate program+or article with code excerpts.++We detect layout boundaries from tokens:++(1) we note an column indent for each line^[This step is implemented using GHC API.],+then++(2) mark a start of each operator (like `::`, `=`, or `>>=`) by column.+Our second step is that of transposing a list of `Line = [(Column, Token)]`+into per-column list of indentations `[(Line, Token)]`.++(3) If any token is present after more than a single space, we also mark its beginning+as indentation boundary.++(4) After sorting columns by line number, we mark these columns that have a consistent+presence along consecutive lines as indentation anchors.++(5) Additionally we mark the leftmost indent as a same indentation barrier+as long as it follows the nesting order.++(6) For postprocessing, we escape the text according to target text processing+engine syntax, translate common operators to their ligatures (see appendix),+and output the text with layout boundaries.++We also plan to implement support for pointing to code fragments+with TikZ target marks[@tikz] at this stage^[With syntax of `{->p1-}` standing for `\tikzmark{p1}`].++We also give user an option to align code fragments without regards for operators,+in case the source programming language is not yet supported.++## Integration++This solution is provided as a `pandoc`[@pandoc] filter so that it is integrated+into standard Markdown-based processing engines with its numerous plugins+that allow inclusion of GraphViz graphs, tables from `.csv` content+or citation referencing.++# Examples++## Layout++The example input is here:+```{.haskell}+class Eq      a+   => Compare a where+  compare :: a -> a -> Ordering+  (>=)    :: a -> a -> Bool+```++After splitting input sections into separated code blocks+and further by `\n\n` markers, we get the following layout boundaries+detected.++```+cl.ass |Eq  .   . |a.+  . => |Comp.are. |a. wh.er.e+  |comp.are |:: |a. |-> |a |-> |Ordering+  |(>=).    |:: |a. |-> |a |-> |Bool+```+Here `|` marks an indentation boundary in a given line and column, whereas `.` is filler (no marker) to keep the columns indented.++First you sort by column (`Data.List.sortBy`), then within each column (`Data.List.groupBy`)+you compare token type that starts there.+As a result we see the following list of column boundaries^[Counting from column 1 and line 1.]:+```{.haskell}+[(3 -- column+ ,[3,4]) -- lines where it applies+ (7, -- column+ ,[1,2]) -- lines where it applies+,(11+ ,[3,4])+,(14+ ,[3,4])+,(15+ ,[1,2])+,(16,+  [3,4])+,(18,+  [3,4])+,(21,+  [3,4])+]+```++Another example input follows, to illustrate that we only take account+of tokens that start after at least one space:++```haskell+(\x -> x)+```+The following detection would be **wrong**:+```+|(|\|x |-|> |x)+```+Instead we detect layout boundaries as follows:+```+(\x |-> |x)+```++## Generating \LaTeX{} or HTML output++When generating \LaTeX{} or HTML output, we simply+assign a list of columns to each span of code tex in line.+This can be implemented using multicolumn marking in both output languages.+Consider the above example with columns numbered at starting character:++```+cl.ass |Eq  .   . |a.+  . => |Comp.are. |a. wh.er.e+  |comp.are |:: |a. |-> |a |-> |Ordering+  |(>=).    |:: |a. |-> |a |-> |Bool+  1    2    3   4 5 6   7  8   9+```+Alignment assignment is here:+```+cl.ass >Eq  .   . |a.+  . => >Comp.are. |a. wh.er.e+  |comp.are |:: |a. |-> |a |-> ^Ordering+  |(>=).    |:: |a. |-> |a |-> ^Bool+```+Note the use of `>` instead of `|` at the boundary of right aligned block.+We also use `^` at the right boundary of center-aligned.++That means that we produce code like this for the first line:+```latex+\multicolumn{2}{r}{class}+\multicolumn{2}{l}{Eq}+\multicolumn{6}{l}{a}+```+General syntax of+`\multicolumn`^[See [Overleaf tutorial](https://www.overleaf.com/learn/latex/tables#Combining_rows_and_columns)+if you do not know how \LaTeX{} tables work.] has three arguments, each enclosed with braces (`{}`):+1. Number of columns in the cell, for example `{2}` or `{6}`.+2. Alignment of text in the cell:+    - `{l}` for left,+    - `{r}` right,+    - `{c}` for centered.+3. The text in the cell. For example `{class}`++First two columns end at `>`. So `\multicolumn` has parameter `{2}` to indicate that the cell spans two columns (just like [`colspan="2"` in HTML](https://www.w3schools.com/tags/att_td_colspan.asp).)+`Eq ` spans another two columns (3rd and 4th column) so again it is parameter `{2}`.++You basically compute column span this way:+* Start a new column with `colspan=1` for other column markers (`<` and `|`)+* Add `+1` to the current column span for every `.`.++The third column would have code like this:+```latex+\multicolumns{1}{l}{ }+\multicolumn{2}{l}{compare}+\multicolumn{1}{c}{::}+\multicolumn{2}{l}{a}+\multicolumn{1}{c}{->}+\multicolumn{1}{l}{a}+\multicolumn{1}{c}{->}+\multicolumn{1}{l}{Ordering}+```+That is the `class` takes columns 0 and 1,+`Eq` takes columns 2 and 3, and `a` goes from column 4 til the end in column 10.++This can be easily converted to HTML table:+```+<td colspan="2" style="text-align:right">class</td>+<td colspan="2" style="text-align:left" >Eq</td>+<td colspan="2" style="text-align:left" >a</td>+```+## Pandoc filter interface++Main executable is a `pandoc` filter.+You get `pandoc` input stream, and replace+[`CodeBlock` blocks](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Block)+there with `Raw "latex"` \LaTeX{} blocks.  It is these block elements of ADT that should contain the \LaTeX{} code+Pandoc will build the document for you, and do it better than you would.+Below is a modified [example from `pandoc` documentation](https://texblog.org/2012/12/21/multi-column-and-multi-row-cells-in-latex-tables/)+for making a `pandoc` filter executable:++```{.haskell}+module Main where++import Text.Pandoc.JSON++main :: IO ()+main = toJSONFilter blockFormatter++blockFormatter :: Block -> Block+blockFormatter (CodeBlock attrs content) | isHaskell attrs =+  haskellCodeFormatter :: Text -> Text+  Raw "latex" $ haskellCodeFormatter content+blockFormatter  x = x+```++In the executable above, you write `haskellCodeFormatter` function+that takes Haskell code, and returns \LaTeX{} code _fragment_ like this:++```{.latex}+\begin{array}+\multicol{2}{..}+\end{array}+```++## Deciding between output formats++We need to check Pandoc meta before issuing `walk`+in order to check what is the output format.+In case it is \LaTeX{} or PDF, then we produce \LaTeX{} raw code fragments.+In case output format is any other, then we produce HTML table.++## Passing options to the filter++There are three ways that `pandoc` filter should accept the options:++1. On the command line `--debug`.+2. As `pandoc` metadata elements+   ([`YAML`](https://en.wikipedia.org/wiki/YAML)+    [at the start of the document](https://pandoc.org/MANUAL.html#extension-yaml_metadata_block)):+    ` pandoc-filter-indent: debug=true`+3. As attributes of the code block: ` ```{.haskell debug=true}`++All options should be available through all three option channels.+See tutorial on `optparse-applicative` to define CLI options.+To allow `YAML` option definition, add `FromJSON` instance.+For concatenating options from different sources use+this [tutorial on option merging](https://chrispenner.ca/posts/hkd-options).++There are following options that will be processed:++* `lexer` -- select a language lexer:+  - `haskell` -- default lexer, GHC+  - `indent`  -- indent only lexer: table column starts when whitespace characters at beginning of the line ends+  - `spaces`  -- space-only lexer: table column starts at the end of block of two or more whitespace characters+  - `python3` -- Python lexer (see below in @sec:python-lexer)+* `debug` -- add option that shows table columns (for debugging layout)+* `underbar` -- do not escape `_`, but instead use \LaTeX{}/HTML subscript til the end of the token:+   `method_agile` becomes `method_{agile}` in \LaTeX{}, `x_i_j` becomes `x_{i,j}` in \LaTeX{}+* table alignment (\LaTeX{} output only):+  - `array` -- use `array` environment, the default+  - `polytable` -- use `polytable` environment+* code alignment:+  - center environment -- default for `array`+  - `left` justify entire code environment -- for convenience when mentioning code inline.+* output mode -- is selected by the `pandoc` metadata only+  - `latex` -- `Raw "latex"` for `tex`, `pdf`, or `beamer` output+  - `html` -- `Raw "html"` for `html`, `slidy`, `slideous`, `s5`, or `revealjs` outputs+  - `table` -- for all other outputs we produce a `Table` block++## Appendix: operator symbol replacement++Code should be given a table of possible token replacements+depending on type of the symbol.++1. We implement greek unicode replacements for single-character type variables:+```+(TypeVar "a") -> α+```+++2. We also replace the common operators with \LaTeX{} symbols commonly used for this purpose:++| Input token      | \LaTeX{} code | Rendering       |+|:-----------------|:--------------|:---------------:|+| `Operator "="`   | `\scalebox{1.7}[1]{=}`     | ＝ |+| `Operator "<>"`  | `\diamond`    | $\diamond{}$    |+| `Operator ">="`  | `\geq`        | $\geq{}$        |+| `Operator "<="`  | `\geq`        | $\leq{}$        |+| `Operator "/="`  | `\ne`         | $\ne{}$         |+| `Operator "==>"` | `\Rightarrow` | $\Rightarrow{}$ |+| `Operator "\/"`  | `\ne`         | $\bigvee{}$     |+| `Operator "/\"`  | `\ne`         | $\bigwedge{}$   |+| `Operator "."`   | `\cdot`       | $\cdot$         |+| `Operator ".."`  | `\cdots`       | $\cdots$         |+| `Operator "elem"`| `\in`         | $\in$            |+| `Operator ">>"`  | `\gg`         | $\gg$            |+| `Operator "<<"`  | `\ll`         | $\ll$            |+| `Operator "~="`  | `\approx`     | $\approx$          |+| `Operator "~"`   | `\sim`             |   $\sim$          |+| `Operator "<->"` | `\leftrightarrow` | $\leftrightarrow{}$         |+| `Operator "<|>"` | `\updownarrow`    | $\updownarrow{}$            |+| `Operator ">>>"` | `\ggg`            | $\ggg{}$         |+| `Operator "<<<"` | `\lll`            | $\lll{}$         |+| `Operator "||"`  | `\parallel`       | $\parallel{}$         |+| `Operator ">>="` | `\gg\joinrel=`             | $\gg\joinrel=$         |+| `Operator "|>"`  | `\triangleright`  | $\triangleright{}$ |+| `Operator "-<"`  | `\prec`           | $\prec{}$       |+| `Operator "<-"`  | `\gets`           | $\gets{}$       |+| `Operator "|"`   | `\vert{}`         | $\vert{}$       |+| `Operator "\\"`  | `\setminus`       | $\setminus{}$       |+| `Var "bottom"`   | `\bot`           | $\bot$            |+| `Var "top"`      | `\top`              | $\top$            |+| `Var "not"`      | `\neg`              | $\neg$            |+| `Var "mempty"`   | `\emptyset{}`     | $\emptyset{}$   |+| `Var "forall"`   | `\forall{}`       | $\forall{}$   |++3. For HTML we only replace these with either HTML entities that indicate these symbols:++```+  (Operator ">") -> &gt;+```++*Unreplaced* content should be escaped.++In order to allow easy implementation we will need alternate debugging+executable `lexer` that will just print the output of the lexer+from the code on the input:++```+[Operator "=", Var "mempty", ...]+```++For finding the right symbol replacements use:++1. [Guide to lhs2TeX](https://hackage.haskell.org/package/lhs2tex-1.18/src/doc/Guide2.pdf) is a good reference on how to format Haskell code symbols.+   See code examples laid out in this document. I expect you to find the mapping for the most commonly used operators:+   - lambda sign token: `\` shown as $\lambda$,+   - equals sign `=`,+   - function type `->`,+   - type sign `::`,+   - and operators in standard type classes:+     * `Control.Monad`: `>>=`, `>>`, `>=>` +     * `Control.Alternative`: `<|>`+     * `Control.Functor`: `<$>`+     * `Control.Applicative`: `<*>`, `<$>`, `<*`, and `*>`+     * `Data.Semigroup.<>` shown as $\diamond$+     * `Data.Ord`: `/=`, `>=`, `<=`, `>`, `<`+     * `Num`: `+`, `-`, `*`, `/`,+     * `System.FilePath`: `</>`, `<.>`+     * `Test.QuickCheck`: `==>` shown as $\Rightarrow$+     * `Control.Arrow`: `>>>`, `***`, `&&&`, `<+>`, `^<<`, `<<^`, `>>^`, `^>>`+     * `Control.Lens`: `^.`, `^.=`.+     * arrow notation: `-<`, `>-`.+   - convert single-letter type variables (and only type variables) to greek letters:+     * `a` to `\alpha` shown as $\alpha$+     * `b` to `\beta` shown as $\beta$+     * etc.++2. [The Comprehensive LATEX Symbol List](http://tug.ctan.org/info/symbols/comprehensive/symbols-a4.pdf) is a good reference of \LaTeX{} symbol names.+   See section _3. Mathematical symbols_.++## Formatting different token types++Since we want to use `\begin{array}` environment in \LaTeX{},+we should encompass the token characters with different \LaTeX{} operators.++* variables with `\textrm{var}`+* type variables with `\mathit{tyvar}`+* others with `\textrm{}` as well+* Haskell keywords with `\textit`+This mapping should be easily changed in a single place in code.++_Please let me know if there is a question about any other token types!_++## Formatting inline code++To format inline code we lookup `inline-code` in YAML metadata,+and append it to attributes of each inline code fragment.+By default it is `haskell`.++## Safe escaping++In order to safely escape strings, we keep token type+next to original token text over entire pipeline+until rendering of code fragment as raw \LaTeX{} or HTML:+```{.haskell}+type Token... = (MyTok, Text, ...)+```++### Pandoc filter connection++Using [`Text.Pandoc.Walk`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Walk.html)+interface we can easily implement the filter.++Filter main is a function like:+```{.haskell}+import Text.Pandoc.JSON+import Text.Pandoc.Walk++main :: IO ()+main = toJSONFilter (ourPandocWalk :: Walkable [a] Pandoc => ToJSONFilter (a -> IO [a])+```++Then we make `ourPandocWalk` to be a function that:++1. Matches [`Meta`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Meta)+to check if we are targetting \LaTeX{} or HTML.+2. Finds [`CodeBlock`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Block) and leaves everything else as-is.+3. Generates `\table` as [`RawBlock`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Block)+   of \LaTeX{} output.++Options **shall** be parsed **per `CodeBlock`**.+First parameter to each `CodeBlock` are attributes ([`Attr`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Attr)).+These attributes should be parsed as options++#### Option attributes++Per-`CodeBlock` attributes to be handled:+* `lexer=indent` or `lexer=haskell`+* ignore `CodeBlock` that does not have attribute:+  - `lexer=`+  - or `.haskell`++Global `Meta` attributes to be handled:+* output format:+  - \LaTeX{} or PDF -- produce \LaTeX{} `RawBlock`+  - HTML -- produce HTML `RawBlock`+  - or all others -- produce `Table`++### TikZ marks++TikZ marks are useful for pointing to fragments of the generated code.++You should just look for comments with syntax:+`{->markName-}` and convert them to a raw+\LaTeX{} string `\tikzmark{markName}`.++For HTML, it generates `<span id="markName" />` which you+can then draw to with [a convenient JavaScript](https://stackoverflow.com/questions/554167/drawing-arrows-on-an-html-page-to-visualize-semantic-links-between-textual-spans#623770).++### Alternate lexers++Please note that using alternate lexers+disables token replacement!+This is important, since the token replacement+to \LaTeX{} special symbols is language-specific.++#### Indent only++Indent only is simplest to implement,+just start a column after initial whitespace in each line ends.++#### Space only++There should be an option to use alternate lexer.+It is driven solely by indentation.++1. Cell break is delimited by a starting indent.+2. Or column of **consecutive** spaces that occur in more than one line,+   **and** at least one line has at least **one more** space before it.++Example input:+```{.python}+# n is size of heap+def heapify(arr, n, i):+    largest = i  # Initialize largest as root+    l = 2 * i + 1     # left = 2*i + 1+    r = 2 * i + 2     # right = 2*i + 2+```++Example column division:+```+# n is size of heap+def |heapify(arr, n, i).:+    |largest = i  # Ini.tialize largest as root+    |l = 2 * i + 1     |# left = 2*i + 1+    |r = 2 * i + 2     |# right = 2*i + 2+```++For highlighting we will later connect [`skylighting`](https://hackage.haskell.org/package/skylighting)+like Pandoc does natively.++#### Python3 lexer {#sec:python-lexer}++Very easy to support, just add package [`language-python`](https://hackage.haskell.org/package/language-python-0.5.6/docs/Language-Python-Version3-Lexer.html) to dependencies.+The token type is different, but we only ever compare it by equality.  ++# Used tools++## Finding Haskell tokens++We find Haskell tokens with GHC-lib that uses GHC parser itself.+For simplicity we use [`ghc-syntax-highlighter`](https://hackage.haskell.org/package/ghc-syntax-highlighter-0.0.6.0/docs/GHC-SyntaxHighlighter.html)+We readd locations after the fact, since it gives you pure tokens with text and spaces,+or pure locations (without text).++```+tokenizeHaskell :: Text -> Maybe [(Token, Text)]+```++## \LaTeX{} output++For escaping text in TeX we use [HaTeX](http://hackage.haskell.org/package/HaTeX-3.5/docs/Text-LaTeX-Base-Render.html#t:Render)[@hatex]++We render tables inline tables with `multicolumn` (not `polytable` like lhs2TeX[@lhs2tex] does.)++We do not use [HaTeX table support](https://hackage.haskell.org/package/HaTeX-3.22.2.0/docs/Text-LaTeX-Packages-Multirow.html) yet.++## Debugging++Pandoc can automatically detect output format.+In order to get \LaTeX{} and HTML output just run:++```sh+pandoc input.md --filter=pandoc-filter-indent -o output.tex+pandoc input.md --filter=pandoc-filter-indent -o output.pdf+```++For debugging indentation use `text` output format:+```+pandoc input.md --filter=pandoc-filter-indent -o output.txt+```+In text mode, it renders indent boundaries as `|` and `^`.++Note that the filter *does not* touch the text outside code blocks.+It can add necessary \LaTeX{} headers or HTML styles in meta `headers-include`.

@@ -1,4 +1,4 @@-Copyright Michał J. Gajda (c) 2020+Copyright Michał J. Gajda (c) 2020-2023  All rights reserved. 

@@ -1,556 +1,103 @@-----title: "Code typesetting made simple"-subtitle: "Pandoc filter"-description: |-  To build this document use:--  > pandoc README.md  -o README.pdf --filter=pandoc-filter-indent--date: |-  `\today`{=latex}-abstract: |-  Program code has become a prime medium for communicating important algorithmic and mathematical ideas, as-  indicated by unwavering popularity of functional pearls and multitude blog posts using literate programming style-  to illustrate key ideas.  However the literate programming systems are either appallingly complex or provide-  only limited functionality to emphasise code structure.  We propose extremely simple code typesetting tool that-  is also a Pandoc filter, and can thus be used to improve comprehensibility of the code. It is also simple enough-  to be provided as a literate program within this submission. While it is processing Haskell code, we show how it-  can be easily adapted to typeset Python, Java or C/C++.-author: Michał J. Gajda--#  - name: Michał J. Gajda-#    email: mjgajda@migamake.com-#    orcid:       "0000-0001-7820-3906"-#    affiliation: 1-affiliation:-  institution: "Migamake Pte Ltd"-  email:        mjgajda@migamake.com-  url:         "https://migamake.com"-review: true-numbersections: true-header-includes:-  - |-2-    \usepackage{graphicx}-    \DeclareUnicodeCharacter{03B1}{\ensuremath{\alpha{}}}--prologue:-  - |-2-    \usepackage{graphicx}-    \DeclareUnicodeCharacter{03B1}{\ensuremath{\alpha{}}}-    \renewcommand{\longeq}{\scalebox{1.7}[1]{=}}-------# Introduction--Program code has become a prime medium for communicating important algorithmic and mathematical ideas, as-indicated by unwavering popularity of functional pearls and multitude blog posts using literate programming style-to illustrate key ideas.  However the literate code typesetting systems like lhs2TEX[@lhs2tex]-are either appallingly complex, language-specific, or provide only limited functionality to emphasise code structure.--We propose a simple approach that can be used not just for code in languages with syntax defined by indentation-defined-(like Haskell or Python), but also for code that has been formatted by indentation (like many Java and C/C++ projects mandate).--# Algorithm--We use `pandoc` to process only code fragments in otherwise unprocessed literate program-or article with code excerpts.--We detect layout boundaries from tokens:--(1) we note an column indent for each line^[This step is implemented using GHC API.],-then--(2) mark a start of each operator (like `::`, `=`, or `>>=`) by column.-Our second step is that of transposing a list of `Line = [(Column, Token)]`-into per-column list of indentations `[(Line, Token)]`.--(3) If any token is present after more than a single space, we also mark its beginning-as indentation boundary.+# pandoc-filter-indent -(4) After sorting columns by line number, we mark these columns that have a consistent-presence along consecutive lines as indentation anchors.+Pandoc filter that intelligently typesets code by detecting indentation boundaries and aligning operators. -(5) Additionally we mark the leftmost indent as a same indentation barrier-as long as it follows the nesting order.+## Quick Start -(6) For postprocessing, we escape the text according to target text processing-engine syntax, translate common operators to their ligatures (see appendix),-and output the text with layout boundaries.+Install via Stack: -We also plan to implement support for pointing to code fragments-with TikZ target marks[@tikz] at this stage^[With syntax of `{->p1-}` standing for `\tikzmark{p1}`].+```bash+stack install pandoc-filter-indent+``` -We also give user an option to align code fragments without regards for operators,-in case the source programming language is not yet supported.+Use with Pandoc: -## Integration+```bash+pandoc --filter pandoc-filter-indent input.md -o output.pdf+pandoc --filter pandoc-filter-indent input.md -o output.html+``` -This solution is provided as a `pandoc`[@pandoc] filter so that it is integrated-into standard Markdown-based processing engines with its numerous plugins-that allow inclusion of GraphViz graphs, tables from `.csv` content-or citation referencing.+## What it does -# Examples+Transforms code blocks into well-aligned, tabular layouts that highlight code structure. Instead of plain monospaced code, you get properly aligned operators, function signatures, and nested structures. -## Layout+**Input:** -The example input is here:+````markdown ```{.haskell} class Eq      a    => Compare a where   compare :: a -> a -> Ordering   (>=)    :: a -> a -> Bool ```--After splitting input sections into separated code blocks-and further by `\n\n` markers, we get the following layout boundaries-detected.--```-cl.ass |Eq  .   . |a.-  . => |Comp.are. |a. wh.er.e-  |comp.are |:: |a. |-> |a |-> |Ordering-  |(>=).    |:: |a. |-> |a |-> |Bool-```-Here `|` marks an indentation boundary in a given line and column, whereas `.` is filler (no marker) to keep the columns indented.--First you sort by column (`Data.List.sortBy`), then within each column (`Data.List.groupBy`)-you compare token type that starts there.-As a result we see the following list of column boundaries^[Counting from column 1 and line 1.]:-```{.haskell}-[(3 -- column- ,[3,4]) -- lines where it applies- (7, -- column- ,[1,2]) -- lines where it applies-,(11- ,[3,4])-,(14- ,[3,4])-,(15- ,[1,2])-,(16,-  [3,4])-,(18,-  [3,4])-,(21,-  [3,4])-]-```--Another example input follows, to illustrate that we only take account-of tokens that start after at least one space:--```haskell-(\x -> x)-```-The following detection would be **wrong**:-```-|(|\|x |-|> |x)-```-Instead we detect layout boundaries as follows:-```-(\x |-> |x)-```--## Generating \LaTeX{} or HTML output--When generating \LaTeX{} or HTML output, we simply-assign a list of columns to each span of code tex in line.-This can be implemented using multicolumn marking in both output languages.-Consider the above example with columns numbered at starting character:--```-cl.ass |Eq  .   . |a.-  . => |Comp.are. |a. wh.er.e-  |comp.are |:: |a. |-> |a |-> |Ordering-  |(>=).    |:: |a. |-> |a |-> |Bool-  1    2    3   4 5 6   7  8   9-```-Alignment assignment is here:-```-cl.ass >Eq  .   . |a.-  . => >Comp.are. |a. wh.er.e-  |comp.are |:: |a. |-> |a |-> ^Ordering-  |(>=).    |:: |a. |-> |a |-> ^Bool-```-Note the use of `>` instead of `|` at the boundary of right aligned block.-We also use `^` at the right boundary of center-aligned.--That means that we produce code like this for the first line:-```latex-\multicolumn{2}{r}{class}-\multicolumn{2}{l}{Eq}-\multicolumn{6}{l}{a}-```-General syntax of-`\multicolumn`^[See [Overleaf tutorial](https://www.overleaf.com/learn/latex/tables#Combining_rows_and_columns)-if you do not know how \LaTeX{} tables work.] has three arguments, each enclosed with braces (`{}`):-1. Number of columns in the cell, for example `{2}` or `{6}`.-2. Alignment of text in the cell:-    - `{l}` for left,-    - `{r}` right,-    - `{c}` for centered.-3. The text in the cell. For example `{class}`--First two columns end at `>`. So `\multicolumn` has parameter `{2}` to indicate that the cell spans two columns (just like [`colspan="2"` in HTML](https://www.w3schools.com/tags/att_td_colspan.asp).)-`Eq ` spans another two columns (3rd and 4th column) so again it is parameter `{2}`.--You basically compute column span this way:-* Start a new column with `colspan=1` for other column markers (`<` and `|`)-* Add `+1` to the current column span for every `.`.--The third column would have code like this:-```latex-\multicolumns{1}{l}{ }-\multicolumn{2}{l}{compare}-\multicolumn{1}{c}{::}-\multicolumn{2}{l}{a}-\multicolumn{1}{c}{->}-\multicolumn{1}{l}{a}-\multicolumn{1}{c}{->}-\multicolumn{1}{l}{Ordering}-```-That is the `class` takes columns 0 and 1,-`Eq` takes columns 2 and 3, and `a` goes from column 4 til the end in column 10.--This can be easily converted to HTML table:-```-<td colspan="2" style="text-align:right">class</td>-<td colspan="2" style="text-align:left" >Eq</td>-<td colspan="2" style="text-align:left" >a</td>-```-## Pandoc filter interface--Main executable is a `pandoc` filter.-You get `pandoc` input stream, and replace-[`CodeBlock` blocks](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Block)-there with `Raw "latex"` \LaTeX{} blocks.  It is these block elements of ADT that should contain the \LaTeX{} code-Pandoc will build the document for you, and do it better than you would.-Below is a modified [example from `pandoc` documentation](https://texblog.org/2012/12/21/multi-column-and-multi-row-cells-in-latex-tables/)-for making a `pandoc` filter executable:--```{.haskell}-module Main where--import Text.Pandoc.JSON--main :: IO ()-main = toJSONFilter blockFormatter--blockFormatter :: Block -> Block-blockFormatter (CodeBlock attrs content) | isHaskell attrs =-  haskellCodeFormatter :: Text -> Text-  Raw "latex" $ haskellCodeFormatter content-blockFormatter  x = x-```--In the executable above, you write `haskellCodeFormatter` function-that takes Haskell code, and returns \LaTeX{} code _fragment_ like this:--```{.latex}-\begin{array}-\multicol{2}{..}-\end{array}-```--## Deciding between output formats--We need to check Pandoc meta before issuing `walk`-in order to check what is the output format.-In case it is \LaTeX{} or PDF, then we produce \LaTeX{} raw code fragments.-In case output format is any other, then we produce HTML table.--## Passing options to the filter--There are three ways that `pandoc` filter should accept the options:--1. On the command line `--debug`.-2. As `pandoc` metadata elements-   ([`YAML`](https://en.wikipedia.org/wiki/YAML)-    [at the start of the document](https://pandoc.org/MANUAL.html#extension-yaml_metadata_block)):-    ` pandoc-filter-indent: debug=true`-3. As attributes of the code block: ` ```{.haskell debug=true}`--All options should be available through all three option channels.-See tutorial on `optparse-applicative` to define CLI options.-To allow `YAML` option definition, add `FromJSON` instance.-For concatenating options from different sources use-this [tutorial on option merging](https://chrispenner.ca/posts/hkd-options).--There are following options that will be processed:--* `lexer` -- select a language lexer:-  - `haskell` -- default lexer, GHC-  - `indent`  -- indent only lexer: table column starts when whitespace characters at beginning of the line ends-  - `spaces`  -- space-only lexer: table column starts at the end of block of two or more whitespace characters-  - `python3` -- Python lexer (see below in @sec:python-lexer)-* `debug` -- add option that shows table columns (for debugging layout)-* `underbar` -- do not escape `_`, but instead use \LaTeX{}/HTML subscript til the end of the token:-   `method_agile` becomes `method_{agile}` in \LaTeX{}, `x_i_j` becomes `x_{i,j}` in \LaTeX{}-* table alignment (\LaTeX{} output only):-  - `array` -- use `array` environment, the default-  - `polytable` -- use `polytable` environment-* code alignment:-  - center environment -- default for `array`-  - `left` justify entire code environment -- for convenience when mentioning code inline.-* output mode -- is selected by the `pandoc` metadata only-  - `latex` -- `Raw "latex"` for `tex`, `pdf`, or `beamer` output-  - `html` -- `Raw "html"` for `html`, `slidy`, `slideous`, `s5`, or `revealjs` outputs-  - `table` -- for all other outputs we produce a `Table` block--## Appendix: operator symbol replacement--Code should be given a table of possible token replacements-depending on type of the symbol.--1. We implement greek unicode replacements for single-character type variables:-```-(TypeVar "a") -> α-```---2. We also replace the common operators with \LaTeX{} symbols commonly used for this purpose:--| Input token      | \LaTeX{} code | Rendering       |-|:-----------------|:--------------|:---------------:|-| `Operator "="`   | `\scalebox{1.7}[1]{=}`     | $\scalebox{1.7}[1]{=}$ |-| `Operator "<>"`  | `\diamond`    | $\diamond{}$    |-| `Operator ">="`  | `\geq`        | $\geq{}$        |-| `Operator "<="`  | `\geq`        | $\leq{}$        |-| `Operator "/="`  | `\ne`         | $\ne{}$         |-| `Operator "==>"` | `\Rightarrow` | $\Rightarrow{}$ |-| `Operator "\/"`  | `\ne`         | $\bigvee{}$     |-| `Operator "/\"`  | `\ne`         | $\bigwedge{}$   |-| `Operator "."`   | `\cdot`       | $\cdot$         |-| `Operator "elem"`| `\in`         | $\in$            |-| `Operator ">>"`  | `\gg`         | $\gg$            |-| `Operator "<<"`  | `\ll`         | $\ll$            |-| `Operator "~="`  | `\approx`     | $\approx$          |-| `Operator "~"`   | `\sim`             |   $\sim$          |-| `Operator "<->"` | `\leftrightarrow` | $\leftrightarrow{}$         |-| `Operator ">>>"` | `\ggg`            | $\ggg{}$         |-| `Operator "<<<"` | `\lll`            | $\lll{}$         |-| `Operator "||"`  | `\parallel`       | $\parallel{}$         |-| `Operator ">>="` | `\ne`             | $\gg\joinrel=$         |-| `Operator "|>"`  | `\triangleright`  | $\triangleright{}$ |-| `Operator "-<"`  | `\prec`           | $\prec{}$       |-| `Operator "<-"`  | `\gets`           | $\gets{}$       |-| `Operator "|"`   | `\vert{}`         | $\vert{}$       |-| `Operator "\\"`  | `\setminus`       | $\setminus{}$       |-| `Var "bottom"`   | `\bot`           | $\bot$            |-| `Var "top"`      | `\top`              | $\top$            |-| `Var "not"`      | `\neg`              | $\neg$            |-| `Var "mempty"`   | `\emptyset{}`     | $\emptyset{}$   |-| `Var "forall"`   | `\forall{}`       | $\forall{}$   |--3. For HTML we only replace these with either HTML entities that indicate these symbols:--```-  (Operator ">") -> &gt;-```--*Unreplaced* content should be escaped.--In order to allow easy implementation we will need alternate debugging-executable `lexer` that will just print the output of the lexer-from the code on the input:--```-[Operator "=", Var "mempty", ...]-```--For finding the right symbol replacements use:--1. [Guide to lhs2TeX](https://hackage.haskell.org/package/lhs2tex-1.18/src/doc/Guide2.pdf) is a good reference on how to format Haskell code symbols.-   See code examples laid out in this document. I expect you to find the mapping for the most commonly used operators:-   - lambda sign token: `\` shown as $\lambda$,-   - equals sign `=`,-   - function type `->`,-   - type sign `::`,-   - and operators in standard type classes:-     * `Control.Monad`: `>>=`, `>>`, `>=>` -     * `Control.Alternative`: `<|>`-     * `Control.Functor`: `<$>`-     * `Control.Applicative`: `<*>`, `<$>`, `<*`, and `*>`-     * `Data.Semigroup.<>` shown as $\diamond$-     * `Data.Ord`: `/=`, `>=`, `<=`, `>`, `<`-     * `Num`: `+`, `-`, `*`, `/`,-     * `System.FilePath`: `</>`, `<.>`-     * `Test.QuickCheck`: `==>` shown as $\Rightarrow$-     * `Control.Arrow`: `>>>`, `***`, `&&&`, `<+>`, `^<<`, `<<^`, `>>^`, `^>>`-     * `Control.Lens`: `^.`, `^.=`.-     * arrow notation: `-<`, `>-`.-   - convert single-letter type variables (and only type variables) to greek letters:-     * `a` to `\alpha` shown as $\alpha$-     * `b` to `\beta` shown as $\beta$-     * etc.--2. [The Comprehensive LATEX Symbol List](http://tug.ctan.org/info/symbols/comprehensive/symbols-a4.pdf) is a good reference of \LaTeX{} symbol names.-   See section _3. Mathematical symbols_.--## Formatting different token types--Since we want to use `\begin{array}` environment in \LaTeX{},-we should encompass the token characters with different \LaTeX{} operators.--* variables with `\textrm{var}`-* type variables with `\mathit{tyvar}`-* others with `\textrm{}` as well-* Haskell keywords with `\textit`-This mapping should be easily changed in a single place in code.--_Please let me know if there is a question about any other token types!_--## Formatting inline code+```` -To format inline code we lookup `inline-code` in YAML metadata,-and append it to attributes of each inline code fragment.-By default it is `haskell`.+**Output:** Rendered with operators vertically aligned in clean tables, using mathematical symbols (→, ⇒, ≥) in LaTeX/PDF output. -## Safe escaping+## Features -In order to safely escape strings, we keep token type-next to original token text over entire pipeline-until rendering of code fragment as raw \LaTeX{} or HTML:-```{.haskell}-type Token... = (MyTok, Text, ...)-```+- **Smart alignment** - Detects and aligns operators (`::`, `=`, `->`, etc.) using GHC lexer+- **Multiple languages** - Haskell (GHC), Python, or generic indentation-based formatting+- **Flexible output** - LaTeX tables, HTML tables, or Pandoc native tables+- **Symbol beautification** - Converts operators to mathematical symbols (optional)+- **Lightweight** - Simple Pandoc filter, easy to integrate -### Pandoc filter connection+## Usage -Using [`Text.Pandoc.Walk`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Walk.html)-interface we can easily implement the filter.+Mark code blocks with language attribute: -Filter main is a function like:+````markdown ```{.haskell}-import Text.Pandoc.JSON-import Text.Pandoc.Walk--main :: IO ()-main = toJSONFilter (ourPandocWalk :: Walkable [a] Pandoc => ToJSONFilter (a -> IO [a])+-- your Haskell code ```--Then we make `ourPandocWalk` to be a function that:--1. Matches [`Meta`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Meta)-to check if we are targetting \LaTeX{} or HTML.-2. Finds [`CodeBlock`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Block) and leaves everything else as-is.-3. Generates `\table` as [`RawBlock`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Block)-   of \LaTeX{} output.--Options **shall** be parsed **per `CodeBlock`**.-First parameter to each `CodeBlock` are attributes ([`Attr`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Attr)).-These attributes should be parsed as options--#### Option attributes--Per-`CodeBlock` attributes to be handled:-* `lexer=indent` or `lexer=haskell`-* ignore `CodeBlock` that does not have attribute:-  - `lexer=`-  - or `.haskell`--Global `Meta` attributes to be handled:-* output format:-  - \LaTeX{} or PDF -- produce \LaTeX{} `RawBlock`-  - HTML -- produce HTML `RawBlock`-  - or all others -- produce `Table`--### TikZ marks--TikZ marks are useful for pointing to fragments of the generated code.--You should just look for comments with syntax:-`{->markName-}` and convert them to a raw-\LaTeX{} string `\tikzmark{markName}`.--For HTML, it generates `<span id="markName" />` which you-can then draw to with [a convenient JavaScript](https://stackoverflow.com/questions/554167/drawing-arrows-on-an-html-page-to-visualize-semantic-links-between-textual-spans#623770).--### Alternate lexers--Please note that using alternate lexers-disables token replacement!-This is important, since the token replacement-to \LaTeX{} special symbols is language-specific.--#### Indent only--Indent only is simplest to implement,-just start a column after initial whitespace in each line ends.--#### Space only--There should be an option to use alternate lexer.-It is driven solely by indentation.+```` -1. Cell break is delimited by a starting indent.-2. Or column of **consecutive** spaces that occur in more than one line,-   **and** at least one line has at least **one more** space before it.+Or use alternate lexers: -Example input:+````markdown ```{.python}-# n is size of heap-def heapify(arr, n, i):-    largest = i  # Initialize largest as root-    l = 2 * i + 1     # left = 2*i + 1-    r = 2 * i + 2     # right = 2*i + 2-```--Example column division:-```-# n is size of heap-def |heapify(arr, n, i).:-    |largest = i  # Ini.tialize largest as root-    |l = 2 * i + 1     |# left = 2*i + 1-    |r = 2 * i + 2     |# right = 2*i + 2+# your Python code ```+```` -For highlighting we will later connect [`skylighting`](https://hackage.haskell.org/package/skylighting)-like Pandoc does natively.+For generic indentation-based alignment: -#### Python3 lexer {#sec:python-lexer}+````markdown+```{.haskell lexer=indent}+-- any indented code+```+```` -Very easy to support, just add package [`language-python`](https://hackage.haskell.org/package/language-python-0.5.6/docs/Language-Python-Version3-Lexer.html) to dependencies.-The token type is different, but we only ever compare it by equality.  +## Options -# Used tools+Configure via code block attributes: -## Finding Haskell tokens+- `lexer=haskell` - GHC lexer (default)+- `lexer=python3` - Python lexer+- `lexer=indent` - Indent-only alignment+- `lexer=spaces` - Space-based alignment+- `debug=true` - Show column boundaries for debugging -We find Haskell tokens with GHC-lib that uses GHC parser itself.-For simplicity we use [`ghc-syntax-highlighter`](https://hackage.haskell.org/package/ghc-syntax-highlighter-0.0.6.0/docs/GHC-SyntaxHighlighter.html)-We readd locations after the fact, since it gives you pure tokens with text and spaces,-or pure locations (without text).+Example: -```-tokenizeHaskell :: Text -> Maybe [(Token, Text)]+````markdown+```{.haskell debug=true}+yourCode :: Here ```--## \LaTeX{} output--For escaping text in TeX we use [HaTeX](http://hackage.haskell.org/package/HaTeX-3.5/docs/Text-LaTeX-Base-Render.html#t:Render)[@hatex]+```` -We render tables inline tables with `multicolumn` (not `polytable` like lhs2TeX[@lhs2tex] does.)+## Full Documentation -We do not use [HaTeX table support](https://hackage.haskell.org/package/HaTeX-3.22.2.0/docs/Text-LaTeX-Packages-Multirow.html) yet.+For detailed algorithm explanation, implementation guide, and advanced usage, see [DESCRIPTION.md](DESCRIPTION.md). -## Debugging+**See also:** [Complete Unicode operator symbol mapping](DESCRIPTION.md#appendix-operator-symbol-replacement) for all supported operator conversions (→, ⇒, ≥, ⊥, ∅, etc.). -Pandoc can automatically detect output format.-In order to get \LaTeX{} and HTML output just run:+## Links -```sh-pandoc input.md --filter=pandoc-filter-indent -o output.tex-pandoc input.md --filter=pandoc-filter-indent -o output.pdf-```+- **GitHub:** <https://github.com/migamake/pandoc-filter-typeset-code>+- **Hackage:** <https://hackage.haskell.org/package/pandoc-filter-indent>+- **Issues:** <https://github.com/migamake/pandoc-filter-typeset-code/issues> -For debugging indentation use `text` output format:-```-pandoc input.md --filter=pandoc-filter-indent -o output.txt-```-In text mode, it renders indent boundaries as `|` and `^`.+## License -Note that the filter *does not* touch the text outside code blocks.-It can add necessary \LaTeX{} headers or HTML styles in meta `headers-include`.+BSD3 - see LICENSE file

@@ -1,6 +1,5 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE ViewPatterns      #-}-{-# LANGUAGE RecordWildCards   #-} {-# LANGUAGE NamedFieldPuns    #-} module Main where @@ -22,7 +21,7 @@ import           System.IO import Debug.Trace(trace) -data Options = Options {+newtype Options = Options {     inlineSyntax :: Text   } deriving (Show) @@ -36,10 +35,12 @@ --   Reads format option, metadata, --   and calls `walk` with `blockFormatter`. runner :: Maybe Format -> Pandoc -> IO Pandoc-runner (fromMaybe (Format "text") -> format) input@(Pandoc (Meta meta) ast) = do-    hPutStrLn stderr $ show opts-    hPutStrLn stderr $ show $ Map.lookup "header-includes" meta-    let top = if format `elem` [Format "latex", Format "tex"]+runner (fromMaybe (Format "text") -> format) input@(Pandoc (Meta meta) ast) =+  do+    hPutStrLn stderr $ "Output format: " <> show format+    hPrint stderr opts+    hPrint stderr (Map.lookup "header-includes" meta)+    let top = if format `elem` [Format "latex", Format "tex", Format "beamer"]                  then Pandoc (Meta $ Map.alter modifyIncludes "header-includes" meta) ast                  else input     return $ walk (blockFormatter opts format) top@@ -82,8 +83,8 @@                -> Format -- ^ Output format, defaults to "plain" if not found                -> Block                -> Block-blockFormatter _opts format (CodeBlock attrs content) = -  case fmap findColumns $ getTokenizer attrs content of+blockFormatter _opts format (CodeBlock attrs content) =+  case findColumns <$> getTokenizer attrs content of     Just processed -> renderBlock format attrs processed     Nothing        -> CodeBlock          attrs content -- fallback -- Do not touch other blocks than 'CodeBock'

@@ -1,18 +1,18 @@ cabal-version: 1.12 --- This file has been generated from package.yaml by hpack version 0.33.0.+-- This file has been generated from package.yaml by hpack version 0.38.0. -- -- see: https://github.com/sol/hpack------ hash: 1d8f275838382e4b48fbe6b67fb6036026bdfd772402682271ab1034e1ae5201  name:           pandoc-filter-indent-version:        0.3.2.0+version:        0.3.3.0 synopsis:       Pandoc filter formatting Haskell code fragments using GHC lexer.-description:    Formats marked code fragments, and allows `pandoc` to safely process rest of your literate program:+description:    Formats marked code fragments, and allows `pandoc` to safely process rest of your literate program.+                Mark code blocks with .haskell attribute:                 .                 > ```{.haskell}-                .+                > -- your code here+                > ```                 .                 Usage:                 .@@ -28,10 +28,10 @@                 and creates tabular code structures from indentation.                 It uses GHC lexer to assure that latest features are always parsed correctly.                 .-                Please see the README on GitHub at <https://github.com/mgajda/pandoc-filter-indent#readme>+                Please see the README on GitHub at <https://github.com/migamake/pandoc-filter-typeset-code#readme> category:       Text-homepage:       https://github.com/mgajda/pandoc-filter-indent#readme-bug-reports:    https://github.com/mgajda/pandoc-filter-indent/issues+homepage:       https://github.com/migamake/pandoc-filter-typeset-code#readme+bug-reports:    https://github.com/migamake/pandoc-filter-typeset-code/issues author:         Michał J. Gajda maintainer:     mjgajda@migamake.com copyright:      AllRightsReserved@@ -40,11 +40,12 @@ build-type:     Simple extra-source-files:     README.md+    DESCRIPTION.md     ChangeLog.md  source-repository head   type: git-  location: https://github.com/mgajda/pandoc-filter-indent+  location: https://github.com/migamake/pandoc-filter-typeset-code  library   exposed-modules:@@ -66,16 +67,16 @@   hs-source-dirs:       src   build-depends:-      HaTeX+      HaTeX >=3.23.0 && <4     , base >=4.7 && <5-    , blaze-html-    , blaze-markup-    , ghc-syntax-highlighter-    , optics-core-    , optics-th-    , pandoc-types-    , skylighting-    , text+    , blaze-html >=0.9.2 && <1+    , blaze-markup >=0.8.3 && <1+    , ghc-syntax-highlighter >=0.0.12 && <1+    , optics-core >=0.4.1 && <1+    , optics-th >=0.4.1 && <1+    , pandoc-types >=1.23.1 && <2+    , skylighting >=0.14.7 && <1+    , text >=2.0.2 && <3   default-language: Haskell2010  executable pandoc-filter-indent@@ -86,41 +87,44 @@       app   ghc-options: -threaded -rtsopts -with-rtsopts=-N   build-depends:-      HaTeX+      HaTeX >=3.23.0 && <4     , base >=4.7 && <5-    , blaze-html-    , blaze-markup-    , containers-    , ghc-syntax-highlighter-    , optics-core-    , optics-th-    , optparse-applicative+    , blaze-html >=0.9.2 && <1+    , blaze-markup >=0.8.3 && <1+    , containers >=0.6.7 && <1+    , ghc-syntax-highlighter >=0.0.12 && <1+    , optics-core >=0.4.1 && <1+    , optics-th >=0.4.1 && <1+    , optparse-applicative >=0.19.0 && <1     , pandoc-filter-indent-    , pandoc-types-    , skylighting-    , text+    , pandoc-types >=1.23.1 && <2+    , skylighting >=0.14.7 && <1+    , text >=2.0.2 && <3   default-language: Haskell2010  test-suite pandoc-filter-indent-test   type: exitcode-stdio-1.0   main-is: Spec.hs   other-modules:+      LambdaSpec+      UnicodeOperatorSpec       Paths_pandoc_filter_indent   hs-source-dirs:       test   ghc-options: -threaded -rtsopts -with-rtsopts=-N   build-depends:-      HaTeX+      HaTeX >=3.23.0 && <4     , QuickCheck     , base >=4.7 && <5-    , blaze-html-    , blaze-markup-    , ghc-syntax-highlighter-    , optics-core-    , optics-th+    , blaze-html >=0.9.2 && <1+    , blaze-markup >=0.8.3 && <1+    , ghc-lib-parser+    , ghc-syntax-highlighter >=0.0.12 && <1+    , optics-core >=0.4.1 && <1+    , optics-th >=0.4.1 && <1     , pandoc-filter-indent-    , pandoc-types+    , pandoc-types >=1.23.1 && <2     , quickcheck-text-    , skylighting-    , text+    , skylighting >=0.14.7 && <1+    , text >=2.0.2 && <3   default-language: Haskell2010

@@ -4,26 +4,26 @@ --   from tuples used in analysis. module Alignment where -import Data.Text (Text)-import Data.Tuple.Optics-import Optics.Lens+import Data.Text         ( Text )+import Data.Tuple.Optics ( Field1(..), Field3(..) )+import Optics.Lens       ( Lens' )  import Token ( MyLoc, MyTok )  -- | Datatype to present columns with alignment requirements. data Align =-    ALeft-  | ACenter-  | AIndent -- indentation spacing+    ALeft   -- ^ Align to the left of the cell+  | ACenter -- ^ Align to the center of the cell+  | AIndent -- ^ Indentation spacing: whitespace with minimum width to preserve   deriving (Eq, Ord, Show)  -- | Records tokenized and converted to common token format.-type Processed = (MyTok -- ^ Token type-                 ,MyLoc -- ^ Token location-                 ,Text  -- ^ Text content-                 ,Maybe Int -- ^ Indent column-                 ,Maybe (Align -- ^ Alignment mark-                        ,Int)  -- ^ Alignment column+type Processed = (MyTok        -- Token type+                 ,MyLoc        -- Token location+                 ,Text         -- Text content+                 ,Maybe Int    -- Indent column+                 ,Maybe (Align -- Alignment mark+                        ,Int)  -- Alignment column                  )   

@@ -1,7 +1,6 @@ {-# LANGUAGE AllowAmbiguousTypes   #-} {-# LANGUAGE OverloadedStrings     #-} {-# LANGUAGE PartialTypeSignatures #-}-{-# LANGUAGE ViewPatterns          #-} {-# LANGUAGE FlexibleInstances     #-} {-# LANGUAGE FlexibleContexts      #-} -- | Filtering a single code block.

@@ -1,7 +1,7 @@ {-# LANGUAGE OverloadedStrings     #-} {-# LANGUAGE PartialTypeSignatures #-} {-# LANGUAGE ViewPatterns          #-}-{-# LANGUAGE TypeSynonymInstances  #-}+{-# LANGUAGE ExplicitForAll        #-} {-# LANGUAGE FlexibleInstances     #-} {-# LANGUAGE FlexibleContexts      #-} -- | Find alignment columns.@@ -12,7 +12,7 @@ import Data.Function  (on) import Data.String    (fromString, IsString) import Data.Text      (Text)-import Data.List      (groupBy, sortBy, sort, group, findIndex)+import Data.List      (groupBy, sortBy, sort, group, elemIndex) import Prelude hiding (getLine) import Optics.Core import Data.Tuple.Optics@@ -26,15 +26,15 @@  -- | Records tokenized and converted to common token format. type Unanalyzed = (MyTok, MyLoc, Text-                  , Maybe Int -- ^ Indent for entire line+                  , Maybe Int -- Indent for entire line                   )  -- | Records aligned, but without column numbers yet.-type Aligned   = (MyTok       -- ^ token type-                 ,MyLoc       -- ^ text start location-                 ,Text        -- ^ text content-                 ,Maybe Int   -- ^ indent in this column-                 ,Maybe Align -- ^ alignment+type Aligned   = (MyTok       -- token type+                 ,MyLoc       -- text start location+                 ,Text        -- text content+                 ,Maybe Int   -- indent in this column+                 ,Maybe Align -- alignment                  )  -- * Definitions of fields accessible at many stages@@ -55,11 +55,9 @@       columnIndices     . withExtraColumns     . sortBy (compare `on` getLineCol)-    . concat-    . map markBoundaries+    . concatMap markBoundaries     . grouping getCol-    . concat-    . map addLineIndent+    . concatMap addLineIndent     . grouping getLine  -- | Extract both line and column where a token starts.@@ -69,9 +67,8 @@ -- | Mark alignment boundaries. markBoundaries :: [Unanalyzed] -> [Aligned] markBoundaries = map markIndent-               . concat                -- . (\b -> trace ("ALIGNED: " <> show b) b)-               . map alignBlock+               . concatMap alignBlock                -- . (\b -> trace ("BLOCKS: " <> show b) b)                . blocks                . sortBy (compare `on` getLine)@@ -111,11 +108,13 @@     apply entry                             = entry `annex` Nothing  -- | Compute all alignment columns existing and their positions in the text column space.+--   Note: Deduplicates by text column position only (not by alignment type)+--   to ensure consistency with tableColumns and avoid gaps in table column indices. extraColumns :: Field2 a a  MyLoc         MyLoc              => Field5 a a (Maybe Align) (Maybe Align)              => [a] -> [(Int, Maybe Align)] extraColumns =-    nubSorted+    nubSortedBy (compare `on` fst)  -- Deduplicate by text column position only   . filter hasAlignment   . map extract   where@@ -186,7 +185,7 @@         mod (Just  a) =              Just (a, colIndex)           where-            colIndex = case findIndex (==getCol aligned) markerColumns of+            colIndex = case elemIndex (getCol aligned) markerColumns of                          Nothing -> error $ "Did not find the index for column " <> show (getCol aligned) <> " within " <> show markerColumns                          Just i  -> i 

@@ -33,7 +33,7 @@             $ sortBy  (compare `on` getLineCol) ps   where     maxCol :: Int-    maxCol = maximum $ map getAlignCol $ ps+    maxCol = maximum $ map getAlignCol ps     sameColSpan :: Processed -> Processed -> Bool     sameColSpan tok1 tok2 = case view alignPos tok2 of                               Nothing | getLine tok1 == getLine tok2 -> True@@ -53,8 +53,7 @@ numColSpans :: [Processed] -> Int numColSpans ps = case colspansPerLine of                    []   -> 1-                   n:ns -> assert (all (n==) ns)-                         $ n+                   n:ns -> assert (all (n==) ns) n   where     colspansPerLine :: [Int]     colspansPerLine = map (sum . map (\(_,c,_) -> c)) . colspans $ ps

@@ -50,7 +50,7 @@         alignMarker  = case view alignPos tok of                          Just (ACenter, _) -> "^"                          Just _            -> "|"-                         otherwise         -> ""+                         _                 -> ""  -- | Text content with markers for markers inside it. textWithMarkers tColumns nextCol tok = 

@@ -80,6 +80,7 @@ formatToken (TOperator,"|>"       ) = "⊳" formatToken (TOperator,"<>"       ) = "⋄" formatToken (TOperator,"=>"       ) = "⇒"+formatToken (TOther,   "="        ) = "＝" formatToken (TOperator,"->"       ) = "→" formatToken (TOperator,"|->"      ) = "↦" formatToken (TVar     ,"undefined") = "⊥"@@ -105,6 +106,8 @@ formatToken (TOperator,"-<"       ) = "≺" formatToken (TOperator,">-"       ) = "≻" formatToken (TOperator,"!="       ) = "≠"+formatToken (TOperator,"=="       ) = "="+formatToken (TOperator,"="        ) = "＝" formatToken (TOperator,"\\/"      ) = "⋁" formatToken (TOperator,"/\\"      ) = "⋀" formatToken (TOperator,"~"        ) = "∼"@@ -113,7 +116,7 @@ formatToken (TKeyword,  kwd       ) = b $ toHtml kwd formatToken (TVar,      v         ) = i $ toHtml v formatToken (TCons,     v         ) = span (toHtml v)-                                      ! style ("font-variant: small-caps;")+                                      ! style "font-variant: small-caps;" formatToken (TTikz mark,_         ) = span ""                                       ! id (toValue mark) formatToken (_,         txt       ) = toHtml txt

@@ -1,19 +1,57 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE ViewPatterns      #-}+{-# LANGUAGE PatternSynonyms   #-} -- | Render analyzed input into LaTeX table.-module Render.Latex(latexFromColSpans, latexInline, latexPackages, subscripts) where+module Render.Latex(latexFromColSpans, latexInline, latexPackages, subAndSuperscripts) where  import           Data.Text(Text) import qualified Data.Text as T-import           Data.Char(isAlpha)+import           Data.Char(isAlpha, isAsciiLower, isAsciiUpper, isDigit)  import           Text.LaTeX.Base.Syntax(protectText)+import           Data.Maybe (fromMaybe)  import           Alignment ( Align(..) ) import           Render.Common(TokensWithColSpan) import           Token(MyTok(..)) import           Util(unbrace) +-- | Protect special LaTeX characters for use in math mode.+--   Handles Haskell operators with backslashes that Skylighting doesn't tokenize properly.+protectTextMath :: Text -> Text+protectTextMath = processText+  where+    processText :: Text -> Text+    processText t+      | T.null t = ""+      -- Handle specific multi-character operators first+      | "\\\\" `T.isPrefixOf` t = "\\setminus " <> processText (T.drop 2 t)+      | "\\/" `T.isPrefixOf` t = "\\land " <> processText (T.drop 2 t)+      | "/\\" `T.isPrefixOf` t = "\\lor " <> processText (T.drop 2 t)+      -- Handle lambda: backslash followed by letter/digit+      | Just ('\\', rest) <- T.uncons t+      , Just (nextChar, _) <- T.uncons rest+      , isAlphaNum nextChar = "\\lambda " <> processText rest+      -- Handle standalone backslash followed by operator chars+      | Just ('\\', rest) <- T.uncons t = "\\backslash " <> processText rest+      -- Handle other special chars+      | otherwise =+          let (c, rest) = fromMaybe (' ', "") $ T.uncons t+          in escapeChar c <> processText rest++    escapeChar :: Char -> Text+    escapeChar '#'  = "\\#"+    escapeChar '$'  = "\\$"+    escapeChar '%'  = "\\%"+    escapeChar '&'  = "\\&"+    escapeChar '_'  = "\\_"+    escapeChar '{'  = "\\{"+    escapeChar '}'  = "\\}"+    escapeChar c    = T.singleton c++    isAlphaNum :: Char -> Bool+    isAlphaNum c = isAsciiLower c || isAsciiUpper c || isDigit c+ -- | Given a number of table columns, --   and a list of lists of colspans for each table row, --   return raw LaTeX code.@@ -60,7 +98,7 @@ -- | Preprocesses functions converted to operator syntax and joins them into a single token. -- FIXME: deduplicate preformatTokens []                                                     = []-preformatTokens ((TOperator,"`"):(TVar, "elem"):(TOperator, "`"):rest) = (TOperator, "elem"):preformatTokens rest+preformatTokens ((TOther, "`"):(TVar, "elem"):(TOther, "`"):rest) = (TOperator, "elem"):preformatTokens rest preformatTokens (a                                              :rest) =  a                 :preformatTokens rest  @@ -72,22 +110,48 @@              . preformatTokens  -- | Add subscripts and superscripts to variable names.---   `_` is subscript, and `__` is superscript.vim -subscripts :: Text -> Text-subscripts ""  = " "-subscripts "_" = "\\_"-subscripts t   = segments t+--   "_" is subscript, and "__" is superscript.+--   Superscripts nest their remaining content recursively.+subAndSuperscripts :: Text -> Text+subAndSuperscripts ""  = " "+subAndSuperscripts "_" = "\\_"+subAndSuperscripts t   = case T.splitOn "_" t of+  [] -> ""+  (x:xs) -> x <> processSegments xs   where-    segments = foldr1 addSubscript . T.splitOn "_"-    -- | Tags the second argument with superscript "^" or subscript "_"-    addSubscript :: Text -> Text -> Text-    addSubscript t tsub = mconcat [t, "\\textsubscript{", tsub, "}"]+    -- Process segments after splitting by "_"+    -- Empty string indicates double underscore (superscript)+    processSegments :: [Text] -> Text+    processSegments [] = ""+    processSegments ("":rest) = case rest of+      [] -> ""  -- Trailing "__"+      _  -> let remainingText = T.intercalate "_" rest+            in "\\textsuperscript{" <> subAndSuperscripts remainingText <> "}"+    processSegments (x:xs) = "\\textsubscript{" <> x <> "}" <> processSegments xs +-- | Pattern for operator-like tokens (TOperator or TOther).+--   Both tokenizers produce different token types for the same operators:+--   - Haskell tokenizer: most operators → TOther+--   - Skylighting tokenizer: some operators → TOperator (e.g., *, >-, -<, >>-, -<<)+--   This pattern synonym eliminates duplication when handling the same operator from both tokenizers.+pattern OpOrOther :: Text -> (MyTok, Text)+pattern OpOrOther txt <- (matchOpOrOther -> Just txt)++matchOpOrOther :: (MyTok, Text) -> Maybe Text+matchOpOrOther (TOperator, txt) = Just txt+matchOpOrOther (TOther,    txt) = Just txt+matchOpOrOther _                = Nothing+ -- Workaround with joinEscapedOperators til w consider spaces only. -- | Render a simple token. formatToken :: (MyTok, Text) -> Text formatToken (TOperator,unbrace -> Just op) = "(" <> formatToken (TOperator, op) <> ")"+-- GHC UnicodeSyntax extension operators+-- See: https://downloads.haskell.org/ghc/latest/docs/users_guide/exts/unicode_syntax.html+-- Note: ∀ can be TKeyword (Haskell) or TOther (Skylighting) formatToken (TKeyword, "forall") = mathop "forall"+formatToken (TKeyword, "∀"     ) = mathop "forall"+formatToken (OpOrOther "∀") = mathop "forall" --formatToken (TVar,     "mempty") = mathop "emptyset" formatToken (TVar,     "bottom") = mathop "bot" formatToken (TVar,  "undefined") = mathop "perp"@@ -102,6 +166,8 @@ formatToken (TOperator,"\\/"   ) = mathop "land" formatToken (TOperator,"\\|/"  ) = mathop "downarrow" formatToken (TOperator,"\\||/" ) = mathop "Downarrow"+formatToken (TOperator,"/|\\"  ) = mathop "uparrow"+formatToken (TOperator,"/||\\" ) = mathop "Uparrow" formatToken (TOperator,"~>"    ) = mathop "leadsto" formatToken (TOperator,"|="    ) = mathop "models" formatToken (TCons    ,"Natural") = "\\mathbb{N}"@@ -115,19 +181,44 @@ formatToken (TOperator,">>>"   ) = mathop "ggg" formatToken (TOperator,"<<"    ) = mathop "ll" formatToken (TOperator,"<<<"   ) = mathop "lll"-formatToken (TOther,   "λ"     ) = mathop "lambda" -- Haskell only?---formatToken (TOperator,"-<"    ) = mathop "prec"-formatToken (TOther,   "-<"    ) = mathop "prec"-formatToken (TOther,   ">-"    ) = mathop "succ"-formatToken (TOperator,"\\\\"  ) = mathop "setminus"-formatToken (TOther   ,"<-"    ) = mathop "gets"+formatToken (TOperator,"\\\\"  ) = mathop "setminus" -- MUST come before single backslash!+formatToken (OpOrOther "\\") = mathop "lambda" -- Lambda (both tokenizers)+formatToken (OpOrOther "λ")  = mathop "lambda" -- Unicode lambda (TOther from Skylighting)+formatToken (TVar,     "λ"     ) = mathop "lambda" -- Unicode lambda (TVar from Haskell tokenizer)+formatToken (OpOrOther "-<") = mathop "prec" -- Left arrow-tail+formatToken (OpOrOther "⤙")  = mathop "prec" -- Unicode -<+formatToken (OpOrOther ">-") = mathop "succ" -- Right arrow-tail+formatToken (OpOrOther "⤚")  = mathop "succ" -- Unicode >-+formatToken (OpOrOther "<-") = mathop "gets" -- Left arrow+formatToken (OpOrOther "←")  = mathop "gets" -- Unicode <- formatToken (TOperator,">="    ) = mathop "geq" formatToken (TOperator,"<="    ) = mathop "leq" formatToken (TOperator,"!="    ) = mathop "ne"-formatToken (TOperator,"<->"   ) = mathop "leftrightarrow"---formatToken (TOperator,"->"    ) = mathop "to"-formatToken (TOther,   "->"    ) = mathop "to"-formatToken (TOther,   "=>"    ) = mathop "Rightarrow"+formatToken (TOperator,"<->"   ) = mathop "updownarrow"+formatToken (TOperator,"<|>"   ) = mathop "leftrightarrow"+formatToken (OpOrOther "->") = mathop "to"+formatToken (OpOrOther "→")  = mathop "to"      -- Unicode ->+formatToken (OpOrOther "=>") = mathop "Rightarrow"+formatToken (OpOrOther "⇒")  = mathop "Rightarrow" -- Unicode =>+formatToken (OpOrOther "::") = mathop ":"       -- Type annotation+formatToken (OpOrOther "∷")  = mathop ":"       -- Unicode ::+-- Note: * is rendered as \times (multiplication) since modern Haskell uses Type for kinds+-- TODO: Make this configurable for legacy code that uses * for kinds+formatToken (OpOrOther "*")  = mathop "times"   -- Multiplication (star for old kind syntax)+formatToken (OpOrOther "★")  = mathop "star"    -- Unicode kind star+formatToken (OpOrOther ">>-") = mathop "rr"     -- Right double arrow-tail+formatToken (OpOrOther "⤜")  = mathop "rr"      -- Unicode >>-+formatToken (OpOrOther "-<<") = mathop "ll"     -- Left double arrow-tail+formatToken (OpOrOther "⤛")  = mathop "ll"      -- Unicode -<<+formatToken (OpOrOther "⊸")  = mathop "multimap" -- Unicode linear arrow %1->+formatToken (OpOrOther "(|") = mathop "llparenthesis" -- Parallel array bracket (stmaryrd)+formatToken (OpOrOther "⦇")  = mathop "llparenthesis" -- Unicode (|+formatToken (OpOrOther "|)") = mathop "rrparenthesis" -- Parallel array bracket (stmaryrd)+formatToken (OpOrOther "⦈")  = mathop "rrparenthesis" -- Unicode |)+formatToken (OpOrOther "[|") = mathop "llbracket" -- Quasiquote bracket (stmaryrd)+formatToken (OpOrOther "⟦")  = mathop "llbracket" -- Unicode [|+formatToken (OpOrOther "|]") = mathop "rrbracket" -- Quasiquote bracket (stmaryrd)+formatToken (OpOrOther "⟧")  = mathop "rrbracket" -- Unicode |] formatToken (TOperator,"==>"   ) = mathop "implies" formatToken (TOperator,"|->"   ) = mathop "mapsto" formatToken (TOperator,"|=>"   ) = mathop "Mapsto" -- requires stmaryrd@@ -137,6 +228,8 @@ formatToken (TOperator,"elem"  ) = mathop "in" formatToken (TOperator,"~"     ) = mathop "sim" formatToken (TOperator,"~="    ) = mathop "approx"+formatToken (TOperator,"><"    ) = mathop "times"+formatToken (TOperator,":->"   ) = mathop "longmapsto" formatToken (TVar,     "a"     ) = mathop "alpha" formatToken (TVar,     "b"     ) = mathop "beta" formatToken (TVar,     "c"     ) = mathop "gamma"@@ -145,16 +238,16 @@ formatToken (TVar,     "k"     ) = mathop "kappa" formatToken (TVar,     "n"     ) = mathop "nu" formatToken (TVar,     "m"     ) = mathop "mu"-formatToken (TVar,     "s"     ) = mathop "sigma"-formatToken (TVar,     "o"     ) = mathop "omega"+formatToken (TVar,     "sigma" ) = mathop "sigma"+formatToken (TVar,     "omega" ) = mathop "omega" formatToken (TVar,     "pi"    ) = mathop "pi" formatToken (TVar,     "tau"   ) = mathop "tau" formatToken (TVar,     "rho"   ) = mathop "rho"-formatToken (TVar    , txt     ) | T.any isAlpha txt = "\\textit{" <> subscripts txt <> "}"+formatToken (TVar    , txt     ) | T.any isAlpha txt = "\\textit{" <> subAndSuperscripts txt <> "}" formatToken (TVar,     txt     ) = "\\textit{"     <> protectText txt  <> "}"-formatToken (TNum    , kwd     ) = protectText kwd +formatToken (TNum    , kwd     ) = protectText kwd formatToken (TKeyword, kwd     ) = "\\textbf{"     <> protectText kwd  <> "}"-formatToken (TCons,    cons    ) = "\\textsc{"     <> subscripts cons <> "}"+formatToken (TCons,    cons    ) = "\\textsc{"     <> subAndSuperscripts cons <> "}" --formatToken (TOperator,"\\"    ) = mathop "lambda" formatToken (TTikz mark,_      ) = mathop $ "tikzMark{" <> mark <> "}" --formatToken (TOther,   "`"     ) = mathop "textasciigrave"@@ -167,8 +260,10 @@ formatToken (TOther,   "["     ) = protectText "[" formatToken (TOther,   "}"     ) = protectText "}" formatToken (TOther,   "{"     ) = protectText "{"+--formatToken (TOther,   "="     ) = "\\scalebox{1.7}{" <> mathop "=" <> "}"+formatToken (TOther,   "="     ) = "=\\joinrel=" -- formatToken (TBlank,   txt  ) = "\\textit{\\textcolor{gray}{" <> protectText txt <> "}}"-formatToken (_,  txt           ) = "\\textrm{"     <> protectText txt  <> "}"+formatToken (_,  txt           ) = "\\textrm{"     <> protectTextMath txt  <> "}"  mathop :: Text -> Text mathop code = "\\" <> code

@@ -8,7 +8,7 @@  import           Data.Text(Text) import qualified Data.Text as T-import           Optics.TH+import           Optics.TH ( makeLenses )  -- * Common tokens and locations --   We keep them here, so we can translate output from tokenizers to common format.@@ -23,21 +23,21 @@  -- | Token just classifies to blank, operator, and the style class data MyTok =-    TBlank-  | TOperator-  | TKeyword-  | TCons-  | TVar-  | TNum-  | TOther-  | TString-  | TTikz  Text -- TikZmark in a comment+    TBlank      -- ^ Whitespace or comments+  | TOperator   -- ^ Operators+  | TKeyword    -- ^ Language-specific keywords+  | TCons       -- ^ Constructors+  | TVar        -- ^ Variables, function names+  | TNum        -- ^ Numbers+  | TOther      -- ^ Other tokens+  | TString     -- ^ String constants+  | TTikz  Text -- ^ TikZmark in a comment   deriving (Eq, Ord, Show)  -- | Records tokenized and converted to common token format.-type Tokenized = (MyTok -- ^ Token type-                 ,MyLoc -- ^ Starting location for the token-                 ,Text  -- ^ text value of the token+type Tokenized = (MyTok -- Token type+                 ,MyLoc -- Starting location for the token+                 ,Text  -- text value of the token                  )  -- | Unpack a Haskell comment with a TikZ mark indicator.

@@ -30,15 +30,29 @@ tokenizer  = fmap ( joinEscapedOperators                   . splitTokens                   . restoreLocations+                  . splitUnicodeLambda                   . fmap recognizeToken)            . tokenizeHaskell +-- | Split tokens that start with unicode lambda (λ) into separate tokens.+--   E.g., "λx" becomes ["λ", "x"]+--   Note: λ is not officially part of GHC's UnicodeSyntax extension,+--   but is widely used as a de facto standard for backslash in lambdas.+splitUnicodeLambda :: [(MyTok, Text)] -> [(MyTok, Text)]+splitUnicodeLambda [] = []+splitUnicodeLambda ((tok, txt):rest)+  | "λ" `T.isPrefixOf` txt && T.length txt > 1 =+      (TOther, "λ") : splitUnicodeLambda ((tok, T.drop 1 txt):rest)+  | otherwise = (tok, txt) : splitUnicodeLambda rest+ -- | Recognize token using both token type from `ghc-lib`, --   and text content. --   Only TikZ marks are recognized by looking up text content. recognizeToken :: (Token, Text) -> (MyTok, Text) recognizeToken (SymbolTok,  "\\") = -- special treatment for lambda   (TOther,               "λ"    )+recognizeToken (SymbolTok,  "λ") = -- Unicode lambda (de facto standard)+  (TOther,               "λ"    ) recognizeToken (CommentTok, tokText@(unTikzMark -> Just mark)) =   (TTikz mark,           tokText) recognizeToken (tokType, tokText) =@@ -106,7 +120,7 @@         withNewLines = fmap (<>"\n") (init split)                     <> [last split]         withLocs :: [Text] -> [(MyTok, MyLoc, Text)]-        withLocs (l:ls) = (TBlank, set mark True $ loc, l)+        withLocs (l:ls) = (TBlank, set mark True loc, l)                         : zipWith mkEntry [line+1..] ls         mkEntry :: Int -> Text -> (MyTok, MyLoc, Text)         mkEntry i t = (TBlank, MyLoc i 1 True, t)

@@ -1,15 +1,15 @@ {-# LANGUAGE OverloadedStrings     #-} {-# LANGUAGE PartialTypeSignatures #-}-{-# LANGUAGE ViewPatterns          #-} {-# LANGUAGE FlexibleContexts      #-} -- | Skylighting code tokenizer-module Token.Skylighting(lookupTokenizer, tokenizer) where+module Token.Skylighting(lookupTokenizer, tokenizer, fixBackslashOperators) where  import Control.Arrow(first) import Text.Pandoc.JSON () import Text.Pandoc.Definition ()-import Data.Maybe(listToMaybe, catMaybes)+import Data.Maybe(listToMaybe, mapMaybe) import Data.String (IsString)+import Data.Char(isAsciiLower, isAsciiUpper, isDigit) import Data.Text (Text) import qualified Data.Text as T import Prelude hiding(getLine)@@ -23,13 +23,15 @@  import Token ( MyLoc(MyLoc), MyTok(..), unTikzMark, mark ) +rightToMaybe :: Either a b -> Maybe b rightToMaybe (Left  err   ) = Nothing rightToMaybe (Right result) = Just result +-- | Looks up the tokenizer from Skylighting preset library by the short name of the language.+--   Picks the first match. lookupTokenizer :: [Text] -> Maybe Syntax lookupTokenizer  = listToMaybe-                 . catMaybes-                 . fmap (Sky.syntaxByShortName Sky.defaultSyntaxMap)+                 . mapMaybe (Sky.syntaxByShortName Sky.defaultSyntaxMap)  -- * Haskell tokenizer frontend -- | Attempt to tokenize input,@@ -41,44 +43,58 @@           -> Maybe [(MyTok, MyLoc, Text)] tokenizer syntax =     fmap ( joinEscapedOperators+         . fixBackslashOperators  -- Fix Skylighting's incorrect splitting of backslash operators+         . fixBracketOperators    -- Fix Skylighting's splitting of bracket operators like (|, |), [|, |]          . splitTokens          . restoreLocations+         . splitUnicodeLambda     -- Split unicode lambda from following characters          . recognizeTokens )     . rightToMaybe     . Sky.tokenize tokenizerOpts syntax   where     tokenizerOpts = Sky.TokenizerConfig Sky.defaultSyntaxMap False +-- | Split tokens that start with unicode lambda (λ) into separate tokens.+--   E.g., "λx" becomes ["λ", "x"]+--   Note: λ is not officially part of GHC's UnicodeSyntax extension,+--   but is widely used as a de facto standard for backslash in lambdas.+splitUnicodeLambda :: [[(MyTok, Text)]] -> [[(MyTok, Text)]]+splitUnicodeLambda = map (concatMap splitToken)+  where+    splitToken (tok, txt)+      | "λ" `T.isPrefixOf` txt && T.length txt > 1 =+          [(TOther, "λ"), (tok, T.drop 1 txt)]+      | otherwise = [(tok, txt)]+ -- | Recognize tokens from all source lines. recognizeTokens :: [SourceLine] -> [[(MyTok, Text)]] recognizeTokens  = map $ map $ first skyTok  -- | Convert token type of `ghc-lib` into tokens recognized by the filter. skyTok :: TokenType -> MyTok-skyTok FloatTok  = TNum-skyTok DecValTok  = TNum-skyTok BaseNTok  = TNum-skyTok StringTok = TString-skyTok CharTok = TString-skyTok FunctionTok = TVar-skyTok AttributeTok = TBlank+skyTok FloatTok          = TNum+skyTok DecValTok         = TNum+skyTok BaseNTok          = TNum+skyTok StringTok         = TString+skyTok CharTok           = TString+skyTok FunctionTok       = TVar+skyTok AttributeTok      = TBlank skyTok VerbatimStringTok = TString-skyTok SpecialStringTok = TCons-skyTok ConstantTok = TCons-skyTok KeywordTok = TKeyword-skyTok BuiltInTok = TKeyword-skyTok PreprocessorTok = TBlank-skyTok CommentTok = TBlank-skyTok DocumentationTok = TBlank-skyTok CommentTok = TBlank-skyTok OperatorTok = TOperator-skyTok SpecialCharTok = TOperator-skyTok RegionMarkerTok = TOperator-skyTok AnnotationTok = TBlank-skyTok ControlFlowTok = TKeyword-skyTok VariableTok = TVar-skyTok DataTypeTok = TCons-skyTok other     = TOther+skyTok SpecialStringTok  = TCons+skyTok ConstantTok       = TCons+skyTok KeywordTok        = TKeyword+skyTok BuiltInTok        = TKeyword+skyTok PreprocessorTok   = TBlank+skyTok CommentTok        = TBlank+skyTok DocumentationTok  = TBlank+skyTok OperatorTok       = TOperator+skyTok SpecialCharTok    = TOperator+skyTok RegionMarkerTok   = TOperator+skyTok AnnotationTok     = TBlank+skyTok ControlFlowTok    = TKeyword+skyTok VariableTok       = TVar+skyTok DataTypeTok       = TCons+skyTok other             = TOther  -- FIXME: generalize for GHC tokenizer and Skylighting -- | Restore locations@@ -122,7 +138,7 @@         withNewLines = fmap (<>"\n") (init split)                     <> [last split]         withLocs :: [Text] -> [(MyTok, MyLoc, Text)]-        withLocs (l:ls) = (TBlank, set mark True $ loc, l)+        withLocs (l:ls) = (TBlank, set mark True loc, l)                         : zipWith mkEntry [line+1..] ls         mkEntry :: Int -> Text -> (MyTok, MyLoc, Text)         mkEntry i t = (TBlank, MyLoc i 1 True, t)@@ -132,6 +148,70 @@  unmark :: Field2 a a MyLoc MyLoc => a -> a unmark = set (_2 % mark) False++-- | Fix Skylighting's tokenization of bracket operators used in GHC extensions.+--   Skylighting splits operators like (|, |), [|, |] which should be single tokens.+--   This function merges them back together.+fixBracketOperators :: [(MyTok, MyLoc, Text)] -> [(MyTok, MyLoc, Text)]+fixBracketOperators [] = []+-- (| parallel array bracket+fixBracketOperators ((TOther, loc, "("):(TOperator, _, "|"):remaining) =+  (TOperator, loc, "(|") : fixBracketOperators remaining+-- |) parallel array bracket+fixBracketOperators ((TOperator, loc, "|"):(TOther, _, ")"):remaining) =+  (TOperator, loc, "|)") : fixBracketOperators remaining+-- [| quasiquote bracket+fixBracketOperators ((TOther, loc, "["):(TOperator, _, "|"):remaining) =+  (TOperator, loc, "[|") : fixBracketOperators remaining+-- |] quasiquote bracket+fixBracketOperators ((TOperator, loc, "|"):(TOther, _, "]"):remaining) =+  (TOperator, loc, "|]") : fixBracketOperators remaining+-- Default case+fixBracketOperators (tok:rest) =+  tok : fixBracketOperators rest++-- | Fix Skylighting's incorrect tokenization of backslash operators.+--   Skylighting incorrectly splits operators like \+, \>, \/, etc.+--   This function merges them back together to match Haskell tokenizer behavior.+fixBackslashOperators :: [(MyTok, MyLoc, Text)] -> [(MyTok, MyLoc, Text)]+fixBackslashOperators [] = []+-- Standalone backslash followed by backslash -> \\ set difference operator+fixBackslashOperators ((TOther, loc, "\\"):(TOther, _, "\\"):remaining) =+  (TOperator, loc, "\\\\") : fixBackslashOperators remaining+-- Standalone backslash followed by operator symbol -> merge as operator (\+, \>, etc.)+fixBackslashOperators ((TOther, loc, "\\"):(TOperator, _, op):remaining) =+  (TOperator, loc, "\\" <> op) : fixBackslashOperators remaining+-- Standalone backslash followed by variable -> lambda, keep separate+fixBackslashOperators (tok1@(TOther, _, "\\"):rest@((TVar, _, _):_)) =+  tok1 : fixBackslashOperators rest+-- Standalone backslash followed by alphanumeric TOther -> lambda, keep separate+fixBackslashOperators (tok1@(TOther, loc1, "\\"):tok2@(TOther, loc2, txt):rest)+  | not (T.null txt) && isAlphaStart txt =+  tok1 : fixBackslashOperators (tok2:rest)+-- Token ending with backslash followed by / -> merge as \/ operator+fixBackslashOperators ((TOther, loc, txt):(TOperator, _, "/"):remaining)+  | "\\" `T.isSuffixOf` txt =+  (TOperator, loc, T.init txt <> "\\/") : fixBackslashOperators remaining+-- Operator / followed by token starting with backslash -> merge as /\ operator+fixBackslashOperators ((TOperator, loc, "/"):(TOther, _, txt):remaining)+  | "\\" `T.isPrefixOf` txt =+  (TOperator, loc, "/\\") : fixBackslashOperators ((TOther, loc, T.tail txt):remaining)+-- Token starting with backslash followed by dot (lambda notation like \f.)+fixBackslashOperators (tok1@(TOther, _, txt):tok2@(TOperator, _, "."):rest)+  | "\\" `T.isPrefixOf` txt && T.length txt > 1 && isAlphaRest (T.tail txt) =+  tok1 : tok2 : fixBackslashOperators rest+-- Default case - keep token as is+fixBackslashOperators (tok:rest) =+  tok : fixBackslashOperators rest++-- Helper functions for fixBackslashOperators+isAlphaStart :: Text -> Bool+isAlphaStart txt = case T.uncons txt of+  Just (c, _) -> isAsciiLower c || isAsciiUpper c+  Nothing -> False++isAlphaRest :: Text -> Bool+isAlphaRest = T.all (\c -> isAsciiLower c || isAsciiUpper c || isDigit c)  -- FIXME: use no-indent-mark instead. joinEscapedOperators :: (Eq c, IsString c, Semigroup c) => [(MyTok, MyLoc, c)] -> [(MyTok, MyLoc, c)]

@@ -1,4 +1,5 @@ {-# LANGUAGE FunctionalDependencies #-}+{-# LANGUAGE MultiParamTypeClasses  #-} {-# LANGUAGE FlexibleInstances      #-} -- | Tuple utilities. module Tuples where

@@ -1,9 +1,6 @@ {-# LANGUAGE AllowAmbiguousTypes   #-} {-# LANGUAGE OverloadedStrings     #-} {-# LANGUAGE PartialTypeSignatures #-}-{-# LANGUAGE ViewPatterns          #-}-{-# LANGUAGE TupleSections         #-}-{-# LANGUAGE TypeSynonymInstances  #-} {-# LANGUAGE FlexibleInstances     #-} {-# LANGUAGE FlexibleContexts      #-} -- | Shared utilities that may be moved to upstream libraries.@@ -52,6 +49,8 @@ --   and end in closing brace. unbrace txt | T.head txt == '(' && T.last txt == ')' && T.length txt > 2 = Just $ T.tail $ T.init txt unbrace _ = Nothing++brace txt = mconcat ["(", txt, ")"]  -- | Preprocess tokens before formatting --   in order to detect tokens like functions converted to operator syntax.

@@ -0,0 +1,298 @@+{-# LANGUAGE OverloadedStrings #-}+-- | Tests for tokenizer consistency on backslash operators and lambda expressions.+--   Compares behavior between Haskell tokenizer and Skylighting tokenizer.+module LambdaSpec where++import           Test.QuickCheck+import qualified Data.Text as T+import           Data.Text (Text)+import           Data.Maybe (fromMaybe)+import           Control.Monad (liftM2, liftM3)++import qualified Token.Haskell as Haskell+import qualified Token.Skylighting as Sky+import           Token (MyTok(..), MyLoc(..))+import           Render.Latex (latexInline)++-- * QuickCheck Generators for Haskell Code++-- | Generate valid Haskell variable names (lowercase start)+genVarName :: Gen Text+genVarName = do+  first <- elements ['a'..'z']+  rest <- resize 5 $ listOf (elements $ ['a'..'z'] ++ ['A'..'Z'] ++ ['0'..'9'] ++ ['\''])+  return $ T.pack (first:rest)++-- | Generate simple Haskell expressions+genSimpleExpr :: Gen Text+genSimpleExpr = frequency+  [ (5, genVarName)+  , (2, fmap (T.pack . show) (arbitrary :: Gen Int))+  , (1, liftM2 (\a b -> a <> " + " <> b) genVarName genVarName)+  , (1, liftM2 (\a b -> a <> " * " <> b) genVarName genVarName)+  ]++-- | Generate lambda expressions with single parameter+-- Examples: \x -> x, \y -> y + 1+genLambdaSimple :: Gen Text+genLambdaSimple = do+  var <- genVarName+  body <- genSimpleExpr+  return $ "\\" <> var <> " -> " <> body++-- | Generate lambda expressions with multiple parameters+-- Examples: \x y -> x + y+genLambdaMulti :: Gen Text+genLambdaMulti = do+  n <- choose (2, 3)+  vars <- vectorOf n genVarName+  body <- genSimpleExpr+  return $ "\\" <> T.intercalate " " vars <> " -> " <> body++-- | Generate nested lambda expressions (curried functions)+-- Examples: \x -> \y -> x + y+genLambdaNested :: Gen Text+genLambdaNested = do+  n <- choose (2, 3)+  vars <- vectorOf n genVarName+  body <- genSimpleExpr+  -- Generate \x -> \y -> body+  let mkLambda v rest = "\\" <> v <> " -> " <> rest+  return $ foldr mkLambda body vars++-- | Generate lambda expression with lambda calculus notation (using dots)+-- Examples: \f. \x. f x+genLambdaDot :: Gen Text+genLambdaDot = do+  n <- choose (2, 3)+  vars <- vectorOf n genVarName+  body <- genSimpleExpr+  -- Generate \f. \x. body+  let parts = map (\v -> "\\" <> v <> ".") vars ++ [body]+  return $ T.intercalate " " parts++-- | Generate any lambda expression+genLambda :: Gen Text+genLambda = frequency+  [ (5, genLambdaSimple)+  , (3, genLambdaMulti)+  , (2, genLambdaNested)+  , (2, genLambdaDot)+  ]++-- | Generate set difference expressions+-- Examples: xs \\ ys, [1,2] \\ [2]+genSetDiff :: Gen Text+genSetDiff = do+  a <- genVarName+  b <- genVarName+  return $ a <> " \\\\ " <> b++-- * Unit Tests for fixBackslashOperators preprocessing++mkLoc :: MyLoc+mkLoc = MyLoc 1 1 True++-- Helper to compare tokens ignoring location details+tokensEq :: [(MyTok, MyLoc, Text)] -> [(MyTok, MyLoc, Text)] -> Bool+tokensEq xs ys = map (\(t,_,txt) -> (t,txt)) xs == map (\(t,_,txt) -> (t,txt)) ys++-- | Property: Standalone \ followed by + should merge to \++prop_fix_backslash_plus :: Property+prop_fix_backslash_plus =+  let input = [(TOther, mkLoc, "\\"), (TOperator, mkLoc, "+")]+      expected = [(TOperator, mkLoc, "\\+")]+  in Sky.fixBackslashOperators input `tokensEq` expected === True++-- | Property: Standalone \ followed by > should merge to \>+prop_fix_backslash_gt :: Property+prop_fix_backslash_gt =+  let input = [(TOther, mkLoc, "\\"), (TOperator, mkLoc, ">")]+      expected = [(TOperator, mkLoc, "\\>")]+  in Sky.fixBackslashOperators input `tokensEq` expected === True++-- | Property: Two backslashes should merge to \\+prop_fix_double_backslash :: Property+prop_fix_double_backslash =+  let input = [(TOther, mkLoc, "\\"), (TOther, mkLoc, "\\")]+      expected = [(TOperator, mkLoc, "\\\\")]+  in Sky.fixBackslashOperators input `tokensEq` expected === True++-- | Property: Backslash followed by variable should stay separate (lambda)+prop_fix_lambda_variable :: Property+prop_fix_lambda_variable =+  let input = [(TOther, mkLoc, "\\"), (TVar, mkLoc, "x")]+      expected = [(TOther, mkLoc, "\\"), (TVar, mkLoc, "x")]+  in Sky.fixBackslashOperators input `tokensEq` expected === True++-- | Property: Token ending with \ followed by / should merge to \/+prop_fix_backslash_slash :: Property+prop_fix_backslash_slash =+  let input = [(TOther, mkLoc, "a \\"), (TOperator, mkLoc, "/")]+      expected = [(TOperator, mkLoc, "a \\/")]+  in Sky.fixBackslashOperators input `tokensEq` expected === True++-- | Property: Backslash in token should stay (lambda)+prop_fix_lambda_in_token :: Property+prop_fix_lambda_in_token =+  let input = [(TOther, mkLoc, "\\x")]+      expected = [(TOther, mkLoc, "\\x")]+  in Sky.fixBackslashOperators input `tokensEq` expected === True++-- * Unit Tests++-- | Test cases for backslash operators and lambda expressions.+--   Tests tokenizer consistency on constructs that Skylighting often mishandles.+backslashOperatorTestCases :: [(String, Text, String)]+backslashOperatorTestCases =+  [ ("simple lambda", "\\x -> x", "lambda")+  , ("lambda at start", "\\x -> case x of", "lambda")+  , ("multiple lambdas", "\\f. \\x. f (f x)", "lambda")+  , ("curried lambda", "\\x -> \\y -> x + y", "lambda")+  , ("lambda in expression", "map (\\x -> x + 1) xs", "lambda")+  , ("unicode lambda", "λx -> x", "lambda")+  , ("set difference", "a \\\\ b", "setminus")+  , ("set diff in list", "xs \\\\ ys", "setminus")+  , ("or operator", "a \\/ b", "land")        -- \/ is logical and+  , ("and operator", "a /\\ b", "lor")        -- /\ is logical or+  , ("backslash operator +", "\\+ 1", "backslash")+  , ("backslash operator >", "\\> x", "backslash")+  ]++-- | Helper function to test both tokenizers with a predicate+--   Reduces duplication in property tests+testBothTokenizers :: Text -> String -> (Text -> Property) -> Property+testBothTokenizers code descr check =+  let toLatex tokens = latexInline $ map (\(t,_,txt) -> (t, txt)) tokens+      haskellResult = fmap toLatex (Haskell.tokenizer code)+      skyResult = Sky.lookupTokenizer ["haskell"] >>= \syntax ->+                    fmap toLatex (Sky.tokenizer syntax code)+  in counterexample descr $+     case (haskellResult, skyResult) of+       (Just h, Just s) ->+         counterexample ("Haskell output: " ++ T.unpack h) $+         counterexample ("Skylighting output: " ++ T.unpack s) $+         label "both tokenized" $+         check h .&&. check s+       (Just h, Nothing) ->+         counterexample ("Haskell output: " ++ T.unpack h) $+         label "only haskell" $+         check h+       (Nothing, Just s) ->+         counterexample ("Skylighting output: " ++ T.unpack s) $+         label "only skylighting" $+         check s+       _ -> label "neither tokenized" $ property True++-- | Property: Test that both tokenizers produce expected LaTeX commands+prop_backslash_operators :: Property+prop_backslash_operators =+  forAll (elements backslashOperatorTestCases) $ \(name, code, expected) ->+    let expectedCmd = "\\" <> T.pack expected+        hasExpected txt = property $ expectedCmd `T.isInfixOf` txt+        descr = "Test: " ++ name ++ "\nCode: " ++ T.unpack code ++ "\nExpected: " ++ expected+    in testBothTokenizers code descr hasExpected++-- * Property Tests++-- | Property: Single backslash in lambda should always render as \lambda+prop_lambda_renders_correctly :: Property+prop_lambda_renders_correctly =+  forAll genLambda $ \lambdaExpr ->+    let hasLambda output = property $ "\\lambda" `T.isInfixOf` output+        descr = "Lambda expression: " ++ T.unpack lambdaExpr+    in testBothTokenizers lambdaExpr descr hasLambda++-- | Property: Both tokenizers should produce the same number of \lambda commands+prop_tokenizers_agree_on_lambda_count :: Property+prop_tokenizers_agree_on_lambda_count =+  forAll genLambda $ \lambdaExpr ->+    let haskellResult = Haskell.tokenizer lambdaExpr >>= \tokens ->+          Just $ latexInline $ map (\(t,_,txt) -> (t, txt)) tokens++        skyResult = Sky.lookupTokenizer ["haskell"] >>= \syntax ->+          Sky.tokenizer syntax lambdaExpr >>= \tokens ->+            Just $ latexInline $ map (\(t,_,txt) -> (t, txt)) tokens++        countLambdas txt = T.count "\\lambda" txt++    in counterexample ("Lambda expression: " ++ T.unpack lambdaExpr) $+       case (haskellResult, skyResult) of+         (Just h, Just s) ->+           let hCount = countLambdas h+               sCount = countLambdas s+           in counterexample ("Haskell output: " ++ T.unpack h) $+              counterexample ("Skylighting output: " ++ T.unpack s) $+              counterexample ("Haskell lambdas: " ++ show hCount) $+              counterexample ("Skylighting lambdas: " ++ show sCount) $+              label ("lambdas: " ++ show hCount) $+              hCount === sCount+         _ -> label "tokenization failed" $ property True++-- | Property: No output should contain \textbackslash or similar problematic commands+prop_no_textbackslash :: Property+prop_no_textbackslash =+  forAll genLambda $ \lambdaExpr ->+    let hasBadBackslash txt = "\\textbackslash" `T.isInfixOf` txt ||+                              "\\backslashx" `T.isInfixOf` txt ||+                              "\\backslash{}" `T.isInfixOf` txt+        noBadBackslash txt = property $ not (hasBadBackslash txt)+        descr = "Lambda expression: " ++ T.unpack lambdaExpr+    in testBothTokenizers lambdaExpr descr noBadBackslash++-- | Property: Double backslash should render as \setminus, not lambda+prop_setdiff_not_lambda :: Property+prop_setdiff_not_lambda =+  forAll genSetDiff $ \expr ->+    let hasSetminus txt = "\\setminus" `T.isInfixOf` txt+        hasLambda txt = "\\lambda" `T.isInfixOf` txt+        checkSetdiff txt = property $ hasSetminus txt && not (hasLambda txt)+        descr = "Set difference expression: " ++ T.unpack expr+    in testBothTokenizers expr descr checkSetdiff++-- | Property: Tokenizers should handle the same code similarly+prop_tokenizers_compatible :: Property+prop_tokenizers_compatible =+  forAll genLambda $ \code ->+    let haskellToks = Haskell.tokenizer code+        skyToks = Sky.lookupTokenizer ["haskell"] >>= \syntax ->+                    Sky.tokenizer syntax code++        -- Extract just the token types and texts (ignoring locations)+        simplify = map (\(t,_,txt) -> (t, txt))++    in counterexample ("Code: " ++ T.unpack code) $+       case (haskellToks, skyToks) of+         (Just h, Just s) ->+           let hSimple = simplify h+               sSimple = simplify s+               hOutput = latexInline hSimple+               sOutput = latexInline sSimple+           in counterexample ("Haskell tokens: " ++ show (length hSimple)) $+              counterexample ("Skylighting tokens: " ++ show (length sSimple)) $+              counterexample ("Haskell output: " ++ T.unpack hOutput) $+              counterexample ("Skylighting output: " ++ T.unpack sOutput) $+              label "both tokenized" $+              property True -- Just ensure both can tokenize+         _ -> label "tokenization failed" $ property True++-- | Main test runner+runAllTests :: IO ()+runAllTests = do+  putStrLn "\n=== fixBackslashOperators Unit Tests ==="+  quickCheck prop_fix_backslash_plus+  quickCheck prop_fix_backslash_gt+  quickCheck prop_fix_double_backslash+  quickCheck prop_fix_lambda_variable+  quickCheck prop_fix_backslash_slash+  quickCheck prop_fix_lambda_in_token++  putStrLn "\n=== Backslash Operator Tests ==="+  quickCheckWith stdArgs { maxSuccess = length backslashOperatorTestCases } prop_backslash_operators++  putStrLn "\n=== Lambda Property Tests ==="+  quickCheckWith stdArgs { maxSuccess = 100 } prop_lambda_renders_correctly+  quickCheckWith stdArgs { maxSuccess = 100 } prop_tokenizers_agree_on_lambda_count+  quickCheckWith stdArgs { maxSuccess = 100 } prop_no_textbackslash+  quickCheckWith stdArgs { maxSuccess = 50 } prop_setdiff_not_lambda+  quickCheckWith stdArgs { maxSuccess = 100 } prop_tokenizers_compatible

@@ -1,4 +1,5 @@ {-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE FlexibleContexts #-} -- | Test suite module Main where @@ -14,20 +15,68 @@             ,MyLoc(..)             ,Tokenized) import Token.Haskell(tokenizer)-import Render.Latex(subscripts)+import Render.Latex(subAndSuperscripts) import qualified Render.Debug import FindColumns import Render.ColSpan import Alignment(Processed) import GHC.Stack +import qualified LambdaSpec+import qualified UnicodeOperatorSpec++-- * Property tests for subAndSuperscripts++-- | Property: Single underscore should produce \textsubscript+prop_single_underscore_is_subscript :: Property+prop_single_underscore_is_subscript =+  forAll (listOf1 $ elements ['a'..'z']) $ \prefix ->+  forAll (listOf1 $ elements ['a'..'z']) $ \suffix ->+    let input = T.pack prefix <> "_" <> T.pack suffix+        output = subAndSuperscripts input+    in counterexample ("Input: " ++ T.unpack input) $+       counterexample ("Output: " ++ T.unpack output) $+       "\\textsubscript{" `T.isInfixOf` output++-- | Property: Double underscore should produce \textsuperscript+prop_double_underscore_is_superscript :: Property+prop_double_underscore_is_superscript =+  forAll (listOf1 $ elements ['a'..'z']) $ \prefix ->+  forAll (listOf1 $ elements ['a'..'z']) $ \suffix ->+    let input = T.pack prefix <> "__" <> T.pack suffix+        output = subAndSuperscripts input+    in counterexample ("Input: " ++ T.unpack input) $+       counterexample ("Output: " ++ T.unpack output) $+       "\\textsuperscript{" `T.isInfixOf` output++-- | Property: Output should not contain raw underscores (except escaped \_)+prop_no_raw_underscores :: Property+prop_no_raw_underscores =+  forAll (resize 20 $ listOf (elements (['a'..'z'] ++ ['_']))) $ \str ->+    not (null str) && str /= "_" ==>+      let input = T.pack str+          output = subAndSuperscripts input+          -- Count underscores that aren't part of \_ escape+          hasRawUnderscore = T.any (== '_') $ T.replace "\\_" "" output+      in counterexample ("Input: " ++ T.unpack input) $+         counterexample ("Output: " ++ T.unpack output) $+         label "processed" $+         not hasRawUnderscore++-- | Property: Empty input or single underscore should be handled correctly+prop_edge_cases :: Property+prop_edge_cases = conjoin+  [ subAndSuperscripts "" === " "+  , subAndSuperscripts "_" === "\\_"+  ]+ prop_tokenizer str = case tokenizer str of-                       Nothing -> label "cannot lex" $ True+                       Nothing -> label "cannot lex" True                        Just t  -> label "lexed" $ length t <= T.length str  prop_debug_renderer_text_length input =     case debug of-      Nothing -> label "cannot lex" $ True+      Nothing -> label "cannot lex" True       Just  t -> label "lexed"                $ all (uncurry (<=))                $ zip (extract input)@@ -40,7 +89,7 @@  prop_colspans input =     case debug of-      Nothing -> label "cannot lex" $ True+      Nothing -> label "cannot lex" True       Just  t -> label "lexed" $                  sameNumber t   where@@ -53,14 +102,14 @@  prop_tableColumns input = T.any (not . Data.Char.isSpace) input ==>   case tokenizer input of-    Nothing     -> label "cannot lex" $ True+    Nothing     -> label "cannot lex" True     Just tokens | all ((TBlank==) . view _1) tokens-                -> label "only blanks" $ True-    Just tokens -> +                -> label "only blanks" True+    Just tokens ->       case findColumns tokens of-        [] -> label "empty" $ True+        [] -> label "empty" True         t  -> case sumColSpans t of-                []  -> label "no colspans"  $ True+                []  -> label "no colspans" True                 c:_ -> label "tableColumns" $ c == length (tableColumns t)   where     extract :: T.Text -> [Int]@@ -78,12 +127,35 @@  main :: IO () main = do-    (subscripts "alpha_beta")        `shouldBe` ("alpha\\textsubscript{beta}")-    (subscripts "alpha__beta")       `shouldBe` ("alpha\\textsuperscript{beta}")-    (subscripts "alpha__gamma_beta") `shouldBe` ("alpha\\textsuperscript{gamma\\textsubscript{beta}}")+    putStrLn "=== Subscript/Superscript Tests ==="+    subAndSuperscripts "alpha_beta"        `shouldBe` "alpha\\textsubscript{beta}"+    subAndSuperscripts "alpha__beta"       `shouldBe` "alpha\\textsuperscript{beta}"+    subAndSuperscripts "alpha__gamma_beta" `shouldBe` "alpha\\textsuperscript{gamma\\textsubscript{beta}}"++    putStrLn "\n=== Example Problems ==="     problem " a\na"     problem "--"     problem "a\n a"+    problem "\n a"   -- prop_tableColumns failure case 1+    problem " a\na"  -- prop_tableColumns failure case 2 (duplicate but documented)++    putStrLn "\n=== Lambda Expression Tests ==="+    LambdaSpec.runAllTests++    putStrLn "\n=== GHC UnicodeSyntax Operator Tests ==="+    UnicodeOperatorSpec.runAllTests++    putStrLn "\n=== SubAndSuperscripts Property Tests ==="+    putStrLn "  Testing single underscore → subscript"+    quickCheckWith stdArgs { maxSuccess = 100 } prop_single_underscore_is_subscript+    putStrLn "  Testing double underscore → superscript"+    quickCheckWith stdArgs { maxSuccess = 100 } prop_double_underscore_is_superscript+    putStrLn "  Testing no raw underscores in output"+    quickCheckWith stdArgs { maxSuccess = 100 } prop_no_raw_underscores+    putStrLn "  Testing edge cases"+    quickCheck prop_edge_cases++    putStrLn "\n=== General Property Tests ==="     quickCheck   prop_tokenizer     quickCheck   prop_debug_renderer_text_length     quickCheck $ withMaxSuccess 10000  prop_colspans -- is it sufficient?

@@ -0,0 +1,88 @@+{-# LANGUAGE OverloadedStrings #-}+-- | Unit tests for GHC UnicodeSyntax extension operators+-- | See: https://downloads.haskell.org/ghc/latest/docs/users_guide/exts/unicode_syntax.html+module UnicodeOperatorSpec (runAllTests) where++import qualified Data.Text as T+import Data.Text (Text)+import Data.Maybe (fromMaybe)++import qualified Token.Haskell as Haskell+import qualified Token.Skylighting as Sky+import Render.Latex (latexInline)++-- | Test that ASCII and Unicode versions produce identical LaTeX output+testOperatorPair :: (String, Text, Text, Text) -> IO ()+testOperatorPair (desc, asciiOp, unicodeOp, expectedLatex) = do+  putStrLn $ "  Testing: " ++ desc++  let haskellAscii = renderHaskell asciiOp+      haskellUnicode = renderHaskell unicodeOp+      skyAscii = renderSkylighting asciiOp+      skyUnicode = renderSkylighting unicodeOp++  -- Check ASCII versions produce expected LaTeX+  checkResult "Haskell (ASCII)" haskellAscii expectedLatex+  checkResult "Skylighting (ASCII)" skyAscii expectedLatex++  -- Check Unicode versions produce the same LaTeX as ASCII+  checkResult "Haskell (Unicode)" haskellUnicode expectedLatex+  checkResult "Skylighting (Unicode)" skyUnicode expectedLatex++  -- Verify ASCII and Unicode produce identical output+  if haskellAscii == haskellUnicode+    then putStrLn "    ✓ Haskell: ASCII and Unicode match"+    else error $ "    ✗ Haskell: ASCII (" ++ T.unpack haskellAscii +++                 ") != Unicode (" ++ T.unpack haskellUnicode ++ ")"++  if skyAscii == skyUnicode+    then putStrLn "    ✓ Skylighting: ASCII and Unicode match"+    else error $ "    ✗ Skylighting: ASCII (" ++ T.unpack skyAscii +++                 ") != Unicode (" ++ T.unpack skyUnicode ++ ")"++  putStrLn ""++checkResult :: String -> Text -> Text -> IO ()+checkResult label actual expected+  | expected `T.isInfixOf` actual =+      putStrLn $ "    ✓ " ++ label ++ ": contains " ++ T.unpack expected+  | otherwise =+      error $ "    ✗ " ++ label ++ ": expected " ++ T.unpack expected +++              " but got " ++ T.unpack actual++renderHaskell :: Text -> Text+renderHaskell code = fromMaybe "" $ do+  tokens <- Haskell.tokenizer code+  return $ latexInline $ map (\(t,_,txt) -> (t, txt)) tokens++renderSkylighting :: Text -> Text+renderSkylighting code = fromMaybe "" $ do+  syntax <- Sky.lookupTokenizer ["haskell"]+  tokens <- Sky.tokenizer syntax code+  return $ latexInline $ map (\(t,_,txt) -> (t, txt)) tokens++-- | Test cases: (description, ASCII operator, Unicode operator, expected LaTeX command)+unicodeOperatorTests :: [(String, Text, Text, Text)]+unicodeOperatorTests =+  [ ("forall", "forall", "∀", "\\forall")+  , ("type annotation", "::", "∷", "\\:")+  , ("class constraint", "=>", "⇒", "\\Rightarrow")+  , ("function type", "->", "→", "\\to")+  , ("bind/generator", "<-", "←", "\\gets")+  , ("right arrow-tail", ">-", "⤚", "\\succ")+  , ("left arrow-tail", "-<", "⤙", "\\prec")+  , ("right double arrow-tail", ">>-", "⤜", "\\rr")+  , ("left double arrow-tail", "-<<", "⤛", "\\ll")+  , ("unicode kind star", "★", "★", "\\star")  -- Unicode only (ASCII * is multiplication)+  , ("linear arrow", "⊸", "⊸", "\\multimap")  -- Unicode only+  , ("parallel array open", "(|", "⦇", "\\llparenthesis")+  , ("parallel array close", "|)", "⦈", "\\rrparenthesis")+  , ("quasiquote open", "[|", "⟦", "\\llbracket")+  , ("quasiquote close", "|]", "⟧", "\\rrbracket")+  ]++runAllTests :: IO ()+runAllTests = do+  putStrLn "\n=== GHC UnicodeSyntax Operator Tests ==="+  mapM_ testOperatorPair unicodeOperatorTests+  putStrLn "All Unicode operator tests passed! ✓"