diff --git a/ChangeLog.md b/ChangeLog.md --- a/ChangeLog.md +++ b/ChangeLog.md @@ -1,6 +1,61 @@ # Changelog for pandoc-filter-indent -## Unreleased changes +## 0.3.3.0 Nov 17 2025 + +### Unicode and Operator Support + - Add comprehensive support for GHC UnicodeSyntax extension operators + - Unicode lambda (λ) now correctly renders as `\lambda` + - All GHC UnicodeSyntax operators now supported: ∷, ⇒, →, ←, ⤚, ⤙, ⤜, ⤛, ★, ∀, ⦇, ⦈, ⟦, ⟧, ⊸ + - ASCII and Unicode versions of operators produce identical LaTeX output + - Fix unicode lambda (λ) character handling in both Haskell and Skylighting tokenizers + - Add `splitUnicodeLambda` to handle tokens like "λx" → ["λ", "x"] + - Fix lambda expressions rendering in LaTeX math mode + - Fix backslash operators (\, \\, \/, /\, etc.) rendering + - Add preprocessing for Skylighting tokenizer to handle split backslash operators + - Add `fixBracketOperators` to merge GHC bracket operators: (|, |), [|, |] + - >< as Cartesian product (\times) + - :-> as \longmapsto + - **Design choice**: `*` renders as `\times` (multiplication) since modern Haskell uses `Type` for kinds + - Unicode `★` renders as `\star` for legacy kind annotations + - Tokenizers cannot distinguish between kind and multiplication contexts + - TODO: Make this configurable for projects with legacy kind syntax + +### Tokenizer Differences and Solutions + - **Root cause identified**: Haskell and Skylighting tokenizers produce different token types + - Haskell tokenizer: most operators → TOther + - Skylighting tokenizer: some operators → TOperator (e.g., *, >-, -<, >>-, -<<) + - Unicode ∀: TKeyword (Haskell) vs TOther (Skylighting) + - Unicode λ: TVar (Haskell) vs TOther (Skylighting) + - **Solution**: Introduced `OpOrOther` pattern synonym using PatternSynonyms extension + - Eliminates code duplication when handling same operator from both tokenizers + - Cleaner syntax: `formatToken (OpOrOther "->") = mathop "to"` + +### Bug Fixes + - Fix `extraColumns` and `tableColumns` deduplication mismatch + - Both now use text column position deduplication to avoid gaps in table column indices + - Fixes `prop_tableColumns` test failure on inputs like "\n a" + - Rename subscripts to subAndSuperscripts with proper superscript support + +### Testing + - Add comprehensive test suite for lambda expressions and backslash operators + - Add unit tests for all GHC UnicodeSyntax operators (UnicodeOperatorSpec) + - Tests verify ASCII and Unicode versions produce identical LaTeX output + - Tests now fail on error instead of just warning (enforced with error function) + - Add tokenizer comparison tests (TokenizerTest) + - Refactor test suite: extract common test pattern into helper function + - Improve test readability: reduce code duplication in property tests + +### Code Quality + - Code cleanup: remove unused LANGUAGE pragmas, add missing pragmas + - Code cleanup: use newtype for single-field records, use concatMap and elemIndex + - Code cleanup: remove redundant $ operators and brackets + - Code cleanup: fix pattern matching style (otherwise -> _) + - Code cleanup: use hPrint instead of hPutStrLn . show + +### Documentation + - Add UNICODE_OPERATORS.md explaining Unicode operator support, design decisions, and limitations + - Document tokenizer differences and solutions + - Document the `*` vs `★` rendering choice with rationale ## Release history diff --git a/DESCRIPTION.md b/DESCRIPTION.md new file mode 100644 --- /dev/null +++ b/DESCRIPTION.md @@ -0,0 +1,558 @@ +--- +title: "Code typesetting made simple" +subtitle: "Pandoc filter" +description: | + To build this document use: + + > pandoc README.md -o README.pdf --filter=pandoc-filter-indent + +date: | + `\today`{=latex} +abstract: | + Program code has become a prime medium for communicating important algorithmic and mathematical ideas, as + indicated by unwavering popularity of functional pearls and multitude blog posts using literate programming style + to illustrate key ideas. However the literate programming systems are either appallingly complex or provide + only limited functionality to emphasise code structure. We propose extremely simple code typesetting tool that + is also a Pandoc filter, and can thus be used to improve comprehensibility of the code. It is also simple enough + to be provided as a literate program within this submission. While it is processing Haskell code, we show how it + can be easily adapted to typeset Python, Java or C/C++. +author: Michał J. Gajda + +# - name: Michał J. Gajda +# email: mjgajda@migamake.com +# orcid: "0000-0001-7820-3906" +# affiliation: 1 +affiliation: + institution: "Migamake Pte Ltd" + email: mjgajda@migamake.com + url: "https://migamake.com" +review: true +numbersections: true +header-includes: + - |-2 + \usepackage{graphicx} + \DeclareUnicodeCharacter{03B1}{\ensuremath{\alpha{}}} + +prologue: + - |-2 + \usepackage{graphicx} + \DeclareUnicodeCharacter{03B1}{\ensuremath{\alpha{}}} + \renewcommand{\longeq}{\scalebox{1.7}[1]{=}} + +--- + +# Introduction + +Program code has become a prime medium for communicating important algorithmic and mathematical ideas, as +indicated by unwavering popularity of functional pearls and multitude blog posts using literate programming style +to illustrate key ideas. However the literate code typesetting systems like lhs2TEX[@lhs2tex] +are either appallingly complex, language-specific, or provide only limited functionality to emphasise code structure. + +We propose a simple approach that can be used not just for code in languages with syntax defined by indentation-defined +(like Haskell or Python), but also for code that has been formatted by indentation (like many Java and C/C++ projects mandate). + +# Algorithm + +We use `pandoc` to process only code fragments in otherwise unprocessed literate program +or article with code excerpts. + +We detect layout boundaries from tokens: + +(1) we note an column indent for each line^[This step is implemented using GHC API.], +then + +(2) mark a start of each operator (like `::`, `=`, or `>>=`) by column. +Our second step is that of transposing a list of `Line = [(Column, Token)]` +into per-column list of indentations `[(Line, Token)]`. + +(3) If any token is present after more than a single space, we also mark its beginning +as indentation boundary. + +(4) After sorting columns by line number, we mark these columns that have a consistent +presence along consecutive lines as indentation anchors. + +(5) Additionally we mark the leftmost indent as a same indentation barrier +as long as it follows the nesting order. + +(6) For postprocessing, we escape the text according to target text processing +engine syntax, translate common operators to their ligatures (see appendix), +and output the text with layout boundaries. + +We also plan to implement support for pointing to code fragments +with TikZ target marks[@tikz] at this stage^[With syntax of `{->p1-}` standing for `\tikzmark{p1}`]. + +We also give user an option to align code fragments without regards for operators, +in case the source programming language is not yet supported. + +## Integration + +This solution is provided as a `pandoc`[@pandoc] filter so that it is integrated +into standard Markdown-based processing engines with its numerous plugins +that allow inclusion of GraphViz graphs, tables from `.csv` content +or citation referencing. + +# Examples + +## Layout + +The example input is here: +```{.haskell} +class Eq a + => Compare a where + compare :: a -> a -> Ordering + (>=) :: a -> a -> Bool +``` + +After splitting input sections into separated code blocks +and further by `\n\n` markers, we get the following layout boundaries +detected. + +``` +cl.ass |Eq . . |a. + . => |Comp.are. |a. wh.er.e + |comp.are |:: |a. |-> |a |-> |Ordering + |(>=). |:: |a. |-> |a |-> |Bool +``` +Here `|` marks an indentation boundary in a given line and column, whereas `.` is filler (no marker) to keep the columns indented. + +First you sort by column (`Data.List.sortBy`), then within each column (`Data.List.groupBy`) +you compare token type that starts there. +As a result we see the following list of column boundaries^[Counting from column 1 and line 1.]: +```{.haskell} +[(3 -- column + ,[3,4]) -- lines where it applies + (7, -- column + ,[1,2]) -- lines where it applies +,(11 + ,[3,4]) +,(14 + ,[3,4]) +,(15 + ,[1,2]) +,(16, + [3,4]) +,(18, + [3,4]) +,(21, + [3,4]) +] +``` + +Another example input follows, to illustrate that we only take account +of tokens that start after at least one space: + +```haskell +(\x -> x) +``` +The following detection would be **wrong**: +``` +|(|\|x |-|> |x) +``` +Instead we detect layout boundaries as follows: +``` +(\x |-> |x) +``` + +## Generating \LaTeX{} or HTML output + +When generating \LaTeX{} or HTML output, we simply +assign a list of columns to each span of code tex in line. +This can be implemented using multicolumn marking in both output languages. +Consider the above example with columns numbered at starting character: + +``` +cl.ass |Eq . . |a. + . => |Comp.are. |a. wh.er.e + |comp.are |:: |a. |-> |a |-> |Ordering + |(>=). |:: |a. |-> |a |-> |Bool + 1 2 3 4 5 6 7 8 9 +``` +Alignment assignment is here: +``` +cl.ass >Eq . . |a. + . => >Comp.are. |a. wh.er.e + |comp.are |:: |a. |-> |a |-> ^Ordering + |(>=). |:: |a. |-> |a |-> ^Bool +``` +Note the use of `>` instead of `|` at the boundary of right aligned block. +We also use `^` at the right boundary of center-aligned. + +That means that we produce code like this for the first line: +```latex +\multicolumn{2}{r}{class} +\multicolumn{2}{l}{Eq} +\multicolumn{6}{l}{a} +``` +General syntax of +`\multicolumn`^[See [Overleaf tutorial](https://www.overleaf.com/learn/latex/tables#Combining_rows_and_columns) +if you do not know how \LaTeX{} tables work.] has three arguments, each enclosed with braces (`{}`): +1. Number of columns in the cell, for example `{2}` or `{6}`. +2. Alignment of text in the cell: + - `{l}` for left, + - `{r}` right, + - `{c}` for centered. +3. The text in the cell. For example `{class}` + +First two columns end at `>`. So `\multicolumn` has parameter `{2}` to indicate that the cell spans two columns (just like [`colspan="2"` in HTML](https://www.w3schools.com/tags/att_td_colspan.asp).) +`Eq ` spans another two columns (3rd and 4th column) so again it is parameter `{2}`. + +You basically compute column span this way: +* Start a new column with `colspan=1` for other column markers (`<` and `|`) +* Add `+1` to the current column span for every `.`. + +The third column would have code like this: +```latex +\multicolumns{1}{l}{ } +\multicolumn{2}{l}{compare} +\multicolumn{1}{c}{::} +\multicolumn{2}{l}{a} +\multicolumn{1}{c}{->} +\multicolumn{1}{l}{a} +\multicolumn{1}{c}{->} +\multicolumn{1}{l}{Ordering} +``` +That is the `class` takes columns 0 and 1, +`Eq` takes columns 2 and 3, and `a` goes from column 4 til the end in column 10. + +This can be easily converted to HTML table: +``` +class +Eq +a +``` +## Pandoc filter interface + +Main executable is a `pandoc` filter. +You get `pandoc` input stream, and replace +[`CodeBlock` blocks](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Block) +there with `Raw "latex"` \LaTeX{} blocks. It is these block elements of ADT that should contain the \LaTeX{} code +Pandoc will build the document for you, and do it better than you would. +Below is a modified [example from `pandoc` documentation](https://texblog.org/2012/12/21/multi-column-and-multi-row-cells-in-latex-tables/) +for making a `pandoc` filter executable: + +```{.haskell} +module Main where + +import Text.Pandoc.JSON + +main :: IO () +main = toJSONFilter blockFormatter + +blockFormatter :: Block -> Block +blockFormatter (CodeBlock attrs content) | isHaskell attrs = + haskellCodeFormatter :: Text -> Text + Raw "latex" $ haskellCodeFormatter content +blockFormatter x = x +``` + +In the executable above, you write `haskellCodeFormatter` function +that takes Haskell code, and returns \LaTeX{} code _fragment_ like this: + +```{.latex} +\begin{array} +\multicol{2}{..} +\end{array} +``` + +## Deciding between output formats + +We need to check Pandoc meta before issuing `walk` +in order to check what is the output format. +In case it is \LaTeX{} or PDF, then we produce \LaTeX{} raw code fragments. +In case output format is any other, then we produce HTML table. + +## Passing options to the filter + +There are three ways that `pandoc` filter should accept the options: + +1. On the command line `--debug`. +2. As `pandoc` metadata elements + ([`YAML`](https://en.wikipedia.org/wiki/YAML) + [at the start of the document](https://pandoc.org/MANUAL.html#extension-yaml_metadata_block)): + ` pandoc-filter-indent: debug=true` +3. As attributes of the code block: ` ```{.haskell debug=true}` + +All options should be available through all three option channels. +See tutorial on `optparse-applicative` to define CLI options. +To allow `YAML` option definition, add `FromJSON` instance. +For concatenating options from different sources use +this [tutorial on option merging](https://chrispenner.ca/posts/hkd-options). + +There are following options that will be processed: + +* `lexer` -- select a language lexer: + - `haskell` -- default lexer, GHC + - `indent` -- indent only lexer: table column starts when whitespace characters at beginning of the line ends + - `spaces` -- space-only lexer: table column starts at the end of block of two or more whitespace characters + - `python3` -- Python lexer (see below in @sec:python-lexer) +* `debug` -- add option that shows table columns (for debugging layout) +* `underbar` -- do not escape `_`, but instead use \LaTeX{}/HTML subscript til the end of the token: + `method_agile` becomes `method_{agile}` in \LaTeX{}, `x_i_j` becomes `x_{i,j}` in \LaTeX{} +* table alignment (\LaTeX{} output only): + - `array` -- use `array` environment, the default + - `polytable` -- use `polytable` environment +* code alignment: + - center environment -- default for `array` + - `left` justify entire code environment -- for convenience when mentioning code inline. +* output mode -- is selected by the `pandoc` metadata only + - `latex` -- `Raw "latex"` for `tex`, `pdf`, or `beamer` output + - `html` -- `Raw "html"` for `html`, `slidy`, `slideous`, `s5`, or `revealjs` outputs + - `table` -- for all other outputs we produce a `Table` block + +## Appendix: operator symbol replacement + +Code should be given a table of possible token replacements +depending on type of the symbol. + +1. We implement greek unicode replacements for single-character type variables: +``` +(TypeVar "a") -> α +``` + + +2. We also replace the common operators with \LaTeX{} symbols commonly used for this purpose: + +| Input token | \LaTeX{} code | Rendering | +|:-----------------|:--------------|:---------------:| +| `Operator "="` | `\scalebox{1.7}[1]{=}` | ＝ | +| `Operator "<>"` | `\diamond` | $\diamond{}$ | +| `Operator ">="` | `\geq` | $\geq{}$ | +| `Operator "<="` | `\geq` | $\leq{}$ | +| `Operator "/="` | `\ne` | $\ne{}$ | +| `Operator "==>"` | `\Rightarrow` | $\Rightarrow{}$ | +| `Operator "\/"` | `\ne` | $\bigvee{}$ | +| `Operator "/\"` | `\ne` | $\bigwedge{}$ | +| `Operator "."` | `\cdot` | $\cdot$ | +| `Operator ".."` | `\cdots` | $\cdots$ | +| `Operator "elem"`| `\in` | $\in$ | +| `Operator ">>"` | `\gg` | $\gg$ | +| `Operator "<<"` | `\ll` | $\ll$ | +| `Operator "~="` | `\approx` | $\approx$ | +| `Operator "~"` | `\sim` | $\sim$ | +| `Operator "<->"` | `\leftrightarrow` | $\leftrightarrow{}$ | +| `Operator "<|>"` | `\updownarrow` | $\updownarrow{}$ | +| `Operator ">>>"` | `\ggg` | $\ggg{}$ | +| `Operator "<<<"` | `\lll` | $\lll{}$ | +| `Operator "||"` | `\parallel` | $\parallel{}$ | +| `Operator ">>="` | `\gg\joinrel=` | $\gg\joinrel=$ | +| `Operator "|>"` | `\triangleright` | $\triangleright{}$ | +| `Operator "-<"` | `\prec` | $\prec{}$ | +| `Operator "<-"` | `\gets` | $\gets{}$ | +| `Operator "|"` | `\vert{}` | $\vert{}$ | +| `Operator "\\"` | `\setminus` | $\setminus{}$ | +| `Var "bottom"` | `\bot` | $\bot$ | +| `Var "top"` | `\top` | $\top$ | +| `Var "not"` | `\neg` | $\neg$ | +| `Var "mempty"` | `\emptyset{}` | $\emptyset{}$ | +| `Var "forall"` | `\forall{}` | $\forall{}$ | + +3. For HTML we only replace these with either HTML entities that indicate these symbols: + +``` + (Operator ">") -> > +``` + +*Unreplaced* content should be escaped. + +In order to allow easy implementation we will need alternate debugging +executable `lexer` that will just print the output of the lexer +from the code on the input: + +``` +[Operator "=", Var "mempty", ...] +``` + +For finding the right symbol replacements use: + +1. [Guide to lhs2TeX](https://hackage.haskell.org/package/lhs2tex-1.18/src/doc/Guide2.pdf) is a good reference on how to format Haskell code symbols. + See code examples laid out in this document. I expect you to find the mapping for the most commonly used operators: + - lambda sign token: `\` shown as $\lambda$, + - equals sign `=`, + - function type `->`, + - type sign `::`, + - and operators in standard type classes: + * `Control.Monad`: `>>=`, `>>`, `>=>` + * `Control.Alternative`: `<|>` + * `Control.Functor`: `<$>` + * `Control.Applicative`: `<*>`, `<$>`, `<*`, and `*>` + * `Data.Semigroup.<>` shown as $\diamond$ + * `Data.Ord`: `/=`, `>=`, `<=`, `>`, `<` + * `Num`: `+`, `-`, `*`, `/`, + * `System.FilePath`: ``, `<.>` + * `Test.QuickCheck`: `==>` shown as $\Rightarrow$ + * `Control.Arrow`: `>>>`, `***`, `&&&`, `<+>`, `^<<`, `<<^`, `>>^`, `^>>` + * `Control.Lens`: `^.`, `^.=`. + * arrow notation: `-<`, `>-`. + - convert single-letter type variables (and only type variables) to greek letters: + * `a` to `\alpha` shown as $\alpha$ + * `b` to `\beta` shown as $\beta$ + * etc. + +2. [The Comprehensive LATEX Symbol List](http://tug.ctan.org/info/symbols/comprehensive/symbols-a4.pdf) is a good reference of \LaTeX{} symbol names. + See section _3. Mathematical symbols_. + +## Formatting different token types + +Since we want to use `\begin{array}` environment in \LaTeX{}, +we should encompass the token characters with different \LaTeX{} operators. + +* variables with `\textrm{var}` +* type variables with `\mathit{tyvar}` +* others with `\textrm{}` as well +* Haskell keywords with `\textit` +This mapping should be easily changed in a single place in code. + +_Please let me know if there is a question about any other token types!_ + +## Formatting inline code + +To format inline code we lookup `inline-code` in YAML metadata, +and append it to attributes of each inline code fragment. +By default it is `haskell`. + +## Safe escaping + +In order to safely escape strings, we keep token type +next to original token text over entire pipeline +until rendering of code fragment as raw \LaTeX{} or HTML: +```{.haskell} +type Token... = (MyTok, Text, ...) +``` + +### Pandoc filter connection + +Using [`Text.Pandoc.Walk`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Walk.html) +interface we can easily implement the filter. + +Filter main is a function like: +```{.haskell} +import Text.Pandoc.JSON +import Text.Pandoc.Walk + +main :: IO () +main = toJSONFilter (ourPandocWalk :: Walkable [a] Pandoc => ToJSONFilter (a -> IO [a]) +``` + +Then we make `ourPandocWalk` to be a function that: + +1. Matches [`Meta`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Meta) +to check if we are targetting \LaTeX{} or HTML. +2. Finds [`CodeBlock`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Block) and leaves everything else as-is. +3. Generates `\table` as [`RawBlock`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Block) + of \LaTeX{} output. + +Options **shall** be parsed **per `CodeBlock`**. +First parameter to each `CodeBlock` are attributes ([`Attr`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Attr)). +These attributes should be parsed as options + +#### Option attributes + +Per-`CodeBlock` attributes to be handled: +* `lexer=indent` or `lexer=haskell` +* ignore `CodeBlock` that does not have attribute: + - `lexer=` + - or `.haskell` + +Global `Meta` attributes to be handled: +* output format: + - \LaTeX{} or PDF -- produce \LaTeX{} `RawBlock` + - HTML -- produce HTML `RawBlock` + - or all others -- produce `Table` + +### TikZ marks + +TikZ marks are useful for pointing to fragments of the generated code. + +You should just look for comments with syntax: +`{->markName-}` and convert them to a raw +\LaTeX{} string `\tikzmark{markName}`. + +For HTML, it generates `` which you +can then draw to with [a convenient JavaScript](https://stackoverflow.com/questions/554167/drawing-arrows-on-an-html-page-to-visualize-semantic-links-between-textual-spans#623770). + +### Alternate lexers + +Please note that using alternate lexers +disables token replacement! +This is important, since the token replacement +to \LaTeX{} special symbols is language-specific. + +#### Indent only + +Indent only is simplest to implement, +just start a column after initial whitespace in each line ends. + +#### Space only + +There should be an option to use alternate lexer. +It is driven solely by indentation. + +1. Cell break is delimited by a starting indent. +2. Or column of **consecutive** spaces that occur in more than one line, + **and** at least one line has at least **one more** space before it. + +Example input: +```{.python} +# n is size of heap +def heapify(arr, n, i): + largest = i # Initialize largest as root + l = 2 * i + 1 # left = 2*i + 1 + r = 2 * i + 2 # right = 2*i + 2 +``` + +Example column division: +``` +# n is size of heap +def |heapify(arr, n, i).: + |largest = i # Ini.tialize largest as root + |l = 2 * i + 1 |# left = 2*i + 1 + |r = 2 * i + 2 |# right = 2*i + 2 +``` + +For highlighting we will later connect [`skylighting`](https://hackage.haskell.org/package/skylighting) +like Pandoc does natively. + +#### Python3 lexer {#sec:python-lexer} + +Very easy to support, just add package [`language-python`](https://hackage.haskell.org/package/language-python-0.5.6/docs/Language-Python-Version3-Lexer.html) to dependencies. +The token type is different, but we only ever compare it by equality. + +# Used tools + +## Finding Haskell tokens + +We find Haskell tokens with GHC-lib that uses GHC parser itself. +For simplicity we use [`ghc-syntax-highlighter`](https://hackage.haskell.org/package/ghc-syntax-highlighter-0.0.6.0/docs/GHC-SyntaxHighlighter.html) +We readd locations after the fact, since it gives you pure tokens with text and spaces, +or pure locations (without text). + +``` +tokenizeHaskell :: Text -> Maybe [(Token, Text)] +``` + +## \LaTeX{} output + +For escaping text in TeX we use [HaTeX](http://hackage.haskell.org/package/HaTeX-3.5/docs/Text-LaTeX-Base-Render.html#t:Render)[@hatex] + +We render tables inline tables with `multicolumn` (not `polytable` like lhs2TeX[@lhs2tex] does.) + +We do not use [HaTeX table support](https://hackage.haskell.org/package/HaTeX-3.22.2.0/docs/Text-LaTeX-Packages-Multirow.html) yet. + +## Debugging + +Pandoc can automatically detect output format. +In order to get \LaTeX{} and HTML output just run: + +```sh +pandoc input.md --filter=pandoc-filter-indent -o output.tex +pandoc input.md --filter=pandoc-filter-indent -o output.pdf +``` + +For debugging indentation use `text` output format: +``` +pandoc input.md --filter=pandoc-filter-indent -o output.txt +``` +In text mode, it renders indent boundaries as `|` and `^`. + +Note that the filter *does not* touch the text outside code blocks. +It can add necessary \LaTeX{} headers or HTML styles in meta `headers-include`. diff --git a/LICENSE b/LICENSE --- a/LICENSE +++ b/LICENSE @@ -1,4 +1,4 @@ -Copyright Michał J. Gajda (c) 2020 +Copyright Michał J. Gajda (c) 2020-2023 All rights reserved. diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -1,556 +1,103 @@ ---- -title: "Code typesetting made simple" -subtitle: "Pandoc filter" -description: | - To build this document use: - - > pandoc README.md -o README.pdf --filter=pandoc-filter-indent - -date: | - `\today`{=latex} -abstract: | - Program code has become a prime medium for communicating important algorithmic and mathematical ideas, as - indicated by unwavering popularity of functional pearls and multitude blog posts using literate programming style - to illustrate key ideas. However the literate programming systems are either appallingly complex or provide - only limited functionality to emphasise code structure. We propose extremely simple code typesetting tool that - is also a Pandoc filter, and can thus be used to improve comprehensibility of the code. It is also simple enough - to be provided as a literate program within this submission. While it is processing Haskell code, we show how it - can be easily adapted to typeset Python, Java or C/C++. -author: Michał J. Gajda - -# - name: Michał J. Gajda -# email: mjgajda@migamake.com -# orcid: "0000-0001-7820-3906" -# affiliation: 1 -affiliation: - institution: "Migamake Pte Ltd" - email: mjgajda@migamake.com - url: "https://migamake.com" -review: true -numbersections: true -header-includes: - - |-2 - \usepackage{graphicx} - \DeclareUnicodeCharacter{03B1}{\ensuremath{\alpha{}}} - -prologue: - - |-2 - \usepackage{graphicx} - \DeclareUnicodeCharacter{03B1}{\ensuremath{\alpha{}}} - \renewcommand{\longeq}{\scalebox{1.7}[1]{=}} - ---- - -# Introduction - -Program code has become a prime medium for communicating important algorithmic and mathematical ideas, as -indicated by unwavering popularity of functional pearls and multitude blog posts using literate programming style -to illustrate key ideas. However the literate code typesetting systems like lhs2TEX[@lhs2tex] -are either appallingly complex, language-specific, or provide only limited functionality to emphasise code structure. - -We propose a simple approach that can be used not just for code in languages with syntax defined by indentation-defined -(like Haskell or Python), but also for code that has been formatted by indentation (like many Java and C/C++ projects mandate). - -# Algorithm - -We use `pandoc` to process only code fragments in otherwise unprocessed literate program -or article with code excerpts. - -We detect layout boundaries from tokens: - -(1) we note an column indent for each line^[This step is implemented using GHC API.], -then - -(2) mark a start of each operator (like `::`, `=`, or `>>=`) by column. -Our second step is that of transposing a list of `Line = [(Column, Token)]` -into per-column list of indentations `[(Line, Token)]`. - -(3) If any token is present after more than a single space, we also mark its beginning -as indentation boundary. +# pandoc-filter-indent -(4) After sorting columns by line number, we mark these columns that have a consistent -presence along consecutive lines as indentation anchors. +Pandoc filter that intelligently typesets code by detecting indentation boundaries and aligning operators. -(5) Additionally we mark the leftmost indent as a same indentation barrier -as long as it follows the nesting order. +## Quick Start -(6) For postprocessing, we escape the text according to target text processing -engine syntax, translate common operators to their ligatures (see appendix), -and output the text with layout boundaries. +Install via Stack: -We also plan to implement support for pointing to code fragments -with TikZ target marks[@tikz] at this stage^[With syntax of `{->p1-}` standing for `\tikzmark{p1}`]. +```bash +stack install pandoc-filter-indent +``` -We also give user an option to align code fragments without regards for operators, -in case the source programming language is not yet supported. +Use with Pandoc: -## Integration +```bash +pandoc --filter pandoc-filter-indent input.md -o output.pdf +pandoc --filter pandoc-filter-indent input.md -o output.html +``` -This solution is provided as a `pandoc`[@pandoc] filter so that it is integrated -into standard Markdown-based processing engines with its numerous plugins -that allow inclusion of GraphViz graphs, tables from `.csv` content -or citation referencing. +## What it does -# Examples +Transforms code blocks into well-aligned, tabular layouts that highlight code structure. Instead of plain monospaced code, you get properly aligned operators, function signatures, and nested structures. -## Layout +**Input:** -The example input is here: +````markdown ```{.haskell} class Eq a => Compare a where compare :: a -> a -> Ordering (>=) :: a -> a -> Bool ``` - -After splitting input sections into separated code blocks -and further by `\n\n` markers, we get the following layout boundaries -detected. - -``` -cl.ass |Eq . . |a. - . => |Comp.are. |a. wh.er.e - |comp.are |:: |a. |-> |a |-> |Ordering - |(>=). |:: |a. |-> |a |-> |Bool -``` -Here `|` marks an indentation boundary in a given line and column, whereas `.` is filler (no marker) to keep the columns indented. - -First you sort by column (`Data.List.sortBy`), then within each column (`Data.List.groupBy`) -you compare token type that starts there. -As a result we see the following list of column boundaries^[Counting from column 1 and line 1.]: -```{.haskell} -[(3 -- column - ,[3,4]) -- lines where it applies - (7, -- column - ,[1,2]) -- lines where it applies -,(11 - ,[3,4]) -,(14 - ,[3,4]) -,(15 - ,[1,2]) -,(16, - [3,4]) -,(18, - [3,4]) -,(21, - [3,4]) -] -``` - -Another example input follows, to illustrate that we only take account -of tokens that start after at least one space: - -```haskell -(\x -> x) -``` -The following detection would be **wrong**: -``` -|(|\|x |-|> |x) -``` -Instead we detect layout boundaries as follows: -``` -(\x |-> |x) -``` - -## Generating \LaTeX{} or HTML output - -When generating \LaTeX{} or HTML output, we simply -assign a list of columns to each span of code tex in line. -This can be implemented using multicolumn marking in both output languages. -Consider the above example with columns numbered at starting character: - -``` -cl.ass |Eq . . |a. - . => |Comp.are. |a. wh.er.e - |comp.are |:: |a. |-> |a |-> |Ordering - |(>=). |:: |a. |-> |a |-> |Bool - 1 2 3 4 5 6 7 8 9 -``` -Alignment assignment is here: -``` -cl.ass >Eq . . |a. - . => >Comp.are. |a. wh.er.e - |comp.are |:: |a. |-> |a |-> ^Ordering - |(>=). |:: |a. |-> |a |-> ^Bool -``` -Note the use of `>` instead of `|` at the boundary of right aligned block. -We also use `^` at the right boundary of center-aligned. - -That means that we produce code like this for the first line: -```latex -\multicolumn{2}{r}{class} -\multicolumn{2}{l}{Eq} -\multicolumn{6}{l}{a} -``` -General syntax of -`\multicolumn`^[See [Overleaf tutorial](https://www.overleaf.com/learn/latex/tables#Combining_rows_and_columns) -if you do not know how \LaTeX{} tables work.] has three arguments, each enclosed with braces (`{}`): -1. Number of columns in the cell, for example `{2}` or `{6}`. -2. Alignment of text in the cell: - - `{l}` for left, - - `{r}` right, - - `{c}` for centered. -3. The text in the cell. For example `{class}` - -First two columns end at `>`. So `\multicolumn` has parameter `{2}` to indicate that the cell spans two columns (just like [`colspan="2"` in HTML](https://www.w3schools.com/tags/att_td_colspan.asp).) -`Eq ` spans another two columns (3rd and 4th column) so again it is parameter `{2}`. - -You basically compute column span this way: -* Start a new column with `colspan=1` for other column markers (`<` and `|`) -* Add `+1` to the current column span for every `.`. - -The third column would have code like this: -```latex -\multicolumns{1}{l}{ } -\multicolumn{2}{l}{compare} -\multicolumn{1}{c}{::} -\multicolumn{2}{l}{a} -\multicolumn{1}{c}{->} -\multicolumn{1}{l}{a} -\multicolumn{1}{c}{->} -\multicolumn{1}{l}{Ordering} -``` -That is the `class` takes columns 0 and 1, -`Eq` takes columns 2 and 3, and `a` goes from column 4 til the end in column 10. - -This can be easily converted to HTML table: -``` -class -Eq -a -``` -## Pandoc filter interface - -Main executable is a `pandoc` filter. -You get `pandoc` input stream, and replace -[`CodeBlock` blocks](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Block) -there with `Raw "latex"` \LaTeX{} blocks. It is these block elements of ADT that should contain the \LaTeX{} code -Pandoc will build the document for you, and do it better than you would. -Below is a modified [example from `pandoc` documentation](https://texblog.org/2012/12/21/multi-column-and-multi-row-cells-in-latex-tables/) -for making a `pandoc` filter executable: - -```{.haskell} -module Main where - -import Text.Pandoc.JSON - -main :: IO () -main = toJSONFilter blockFormatter - -blockFormatter :: Block -> Block -blockFormatter (CodeBlock attrs content) | isHaskell attrs = - haskellCodeFormatter :: Text -> Text - Raw "latex" $ haskellCodeFormatter content -blockFormatter x = x -``` - -In the executable above, you write `haskellCodeFormatter` function -that takes Haskell code, and returns \LaTeX{} code _fragment_ like this: - -```{.latex} -\begin{array} -\multicol{2}{..} -\end{array} -``` - -## Deciding between output formats - -We need to check Pandoc meta before issuing `walk` -in order to check what is the output format. -In case it is \LaTeX{} or PDF, then we produce \LaTeX{} raw code fragments. -In case output format is any other, then we produce HTML table. - -## Passing options to the filter - -There are three ways that `pandoc` filter should accept the options: - -1. On the command line `--debug`. -2. As `pandoc` metadata elements - ([`YAML`](https://en.wikipedia.org/wiki/YAML) - [at the start of the document](https://pandoc.org/MANUAL.html#extension-yaml_metadata_block)): - ` pandoc-filter-indent: debug=true` -3. As attributes of the code block: ` ```{.haskell debug=true}` - -All options should be available through all three option channels. -See tutorial on `optparse-applicative` to define CLI options. -To allow `YAML` option definition, add `FromJSON` instance. -For concatenating options from different sources use -this [tutorial on option merging](https://chrispenner.ca/posts/hkd-options). - -There are following options that will be processed: - -* `lexer` -- select a language lexer: - - `haskell` -- default lexer, GHC - - `indent` -- indent only lexer: table column starts when whitespace characters at beginning of the line ends - - `spaces` -- space-only lexer: table column starts at the end of block of two or more whitespace characters - - `python3` -- Python lexer (see below in @sec:python-lexer) -* `debug` -- add option that shows table columns (for debugging layout) -* `underbar` -- do not escape `_`, but instead use \LaTeX{}/HTML subscript til the end of the token: - `method_agile` becomes `method_{agile}` in \LaTeX{}, `x_i_j` becomes `x_{i,j}` in \LaTeX{} -* table alignment (\LaTeX{} output only): - - `array` -- use `array` environment, the default - - `polytable` -- use `polytable` environment -* code alignment: - - center environment -- default for `array` - - `left` justify entire code environment -- for convenience when mentioning code inline. -* output mode -- is selected by the `pandoc` metadata only - - `latex` -- `Raw "latex"` for `tex`, `pdf`, or `beamer` output - - `html` -- `Raw "html"` for `html`, `slidy`, `slideous`, `s5`, or `revealjs` outputs - - `table` -- for all other outputs we produce a `Table` block - -## Appendix: operator symbol replacement - -Code should be given a table of possible token replacements -depending on type of the symbol. - -1. We implement greek unicode replacements for single-character type variables: -``` -(TypeVar "a") -> α -``` - - -2. We also replace the common operators with \LaTeX{} symbols commonly used for this purpose: - -| Input token | \LaTeX{} code | Rendering | -|:-----------------|:--------------|:---------------:| -| `Operator "="` | `\scalebox{1.7}[1]{=}` | $\scalebox{1.7}[1]{=}$ | -| `Operator "<>"` | `\diamond` | $\diamond{}$ | -| `Operator ">="` | `\geq` | $\geq{}$ | -| `Operator "<="` | `\geq` | $\leq{}$ | -| `Operator "/="` | `\ne` | $\ne{}$ | -| `Operator "==>"` | `\Rightarrow` | $\Rightarrow{}$ | -| `Operator "\/"` | `\ne` | $\bigvee{}$ | -| `Operator "/\"` | `\ne` | $\bigwedge{}$ | -| `Operator "."` | `\cdot` | $\cdot$ | -| `Operator "elem"`| `\in` | $\in$ | -| `Operator ">>"` | `\gg` | $\gg$ | -| `Operator "<<"` | `\ll` | $\ll$ | -| `Operator "~="` | `\approx` | $\approx$ | -| `Operator "~"` | `\sim` | $\sim$ | -| `Operator "<->"` | `\leftrightarrow` | $\leftrightarrow{}$ | -| `Operator ">>>"` | `\ggg` | $\ggg{}$ | -| `Operator "<<<"` | `\lll` | $\lll{}$ | -| `Operator "||"` | `\parallel` | $\parallel{}$ | -| `Operator ">>="` | `\ne` | $\gg\joinrel=$ | -| `Operator "|>"` | `\triangleright` | $\triangleright{}$ | -| `Operator "-<"` | `\prec` | $\prec{}$ | -| `Operator "<-"` | `\gets` | $\gets{}$ | -| `Operator "|"` | `\vert{}` | $\vert{}$ | -| `Operator "\\"` | `\setminus` | $\setminus{}$ | -| `Var "bottom"` | `\bot` | $\bot$ | -| `Var "top"` | `\top` | $\top$ | -| `Var "not"` | `\neg` | $\neg$ | -| `Var "mempty"` | `\emptyset{}` | $\emptyset{}$ | -| `Var "forall"` | `\forall{}` | $\forall{}$ | - -3. For HTML we only replace these with either HTML entities that indicate these symbols: - -``` - (Operator ">") -> > -``` - -*Unreplaced* content should be escaped. - -In order to allow easy implementation we will need alternate debugging -executable `lexer` that will just print the output of the lexer -from the code on the input: - -``` -[Operator "=", Var "mempty", ...] -``` - -For finding the right symbol replacements use: - -1. [Guide to lhs2TeX](https://hackage.haskell.org/package/lhs2tex-1.18/src/doc/Guide2.pdf) is a good reference on how to format Haskell code symbols. - See code examples laid out in this document. I expect you to find the mapping for the most commonly used operators: - - lambda sign token: `\` shown as $\lambda$, - - equals sign `=`, - - function type `->`, - - type sign `::`, - - and operators in standard type classes: - * `Control.Monad`: `>>=`, `>>`, `>=>` - * `Control.Alternative`: `<|>` - * `Control.Functor`: `<$>` - * `Control.Applicative`: `<*>`, `<$>`, `<*`, and `*>` - * `Data.Semigroup.<>` shown as $\diamond$ - * `Data.Ord`: `/=`, `>=`, `<=`, `>`, `<` - * `Num`: `+`, `-`, `*`, `/`, - * `System.FilePath`: ``, `<.>` - * `Test.QuickCheck`: `==>` shown as $\Rightarrow$ - * `Control.Arrow`: `>>>`, `***`, `&&&`, `<+>`, `^<<`, `<<^`, `>>^`, `^>>` - * `Control.Lens`: `^.`, `^.=`. - * arrow notation: `-<`, `>-`. - - convert single-letter type variables (and only type variables) to greek letters: - * `a` to `\alpha` shown as $\alpha$ - * `b` to `\beta` shown as $\beta$ - * etc. - -2. [The Comprehensive LATEX Symbol List](http://tug.ctan.org/info/symbols/comprehensive/symbols-a4.pdf) is a good reference of \LaTeX{} symbol names. - See section _3. Mathematical symbols_. - -## Formatting different token types - -Since we want to use `\begin{array}` environment in \LaTeX{}, -we should encompass the token characters with different \LaTeX{} operators. - -* variables with `\textrm{var}` -* type variables with `\mathit{tyvar}` -* others with `\textrm{}` as well -* Haskell keywords with `\textit` -This mapping should be easily changed in a single place in code. - -_Please let me know if there is a question about any other token types!_ - -## Formatting inline code +```` -To format inline code we lookup `inline-code` in YAML metadata, -and append it to attributes of each inline code fragment. -By default it is `haskell`. +**Output:** Rendered with operators vertically aligned in clean tables, using mathematical symbols (→, ⇒, ≥) in LaTeX/PDF output. -## Safe escaping +## Features -In order to safely escape strings, we keep token type -next to original token text over entire pipeline -until rendering of code fragment as raw \LaTeX{} or HTML: -```{.haskell} -type Token... = (MyTok, Text, ...) -``` +- **Smart alignment** - Detects and aligns operators (`::`, `=`, `->`, etc.) using GHC lexer +- **Multiple languages** - Haskell (GHC), Python, or generic indentation-based formatting +- **Flexible output** - LaTeX tables, HTML tables, or Pandoc native tables +- **Symbol beautification** - Converts operators to mathematical symbols (optional) +- **Lightweight** - Simple Pandoc filter, easy to integrate -### Pandoc filter connection +## Usage -Using [`Text.Pandoc.Walk`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Walk.html) -interface we can easily implement the filter. +Mark code blocks with language attribute: -Filter main is a function like: +````markdown ```{.haskell} -import Text.Pandoc.JSON -import Text.Pandoc.Walk - -main :: IO () -main = toJSONFilter (ourPandocWalk :: Walkable [a] Pandoc => ToJSONFilter (a -> IO [a]) +-- your Haskell code ``` - -Then we make `ourPandocWalk` to be a function that: - -1. Matches [`Meta`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Meta) -to check if we are targetting \LaTeX{} or HTML. -2. Finds [`CodeBlock`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Block) and leaves everything else as-is. -3. Generates `\table` as [`RawBlock`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Block) - of \LaTeX{} output. - -Options **shall** be parsed **per `CodeBlock`**. -First parameter to each `CodeBlock` are attributes ([`Attr`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Attr)). -These attributes should be parsed as options - -#### Option attributes - -Per-`CodeBlock` attributes to be handled: -* `lexer=indent` or `lexer=haskell` -* ignore `CodeBlock` that does not have attribute: - - `lexer=` - - or `.haskell` - -Global `Meta` attributes to be handled: -* output format: - - \LaTeX{} or PDF -- produce \LaTeX{} `RawBlock` - - HTML -- produce HTML `RawBlock` - - or all others -- produce `Table` - -### TikZ marks - -TikZ marks are useful for pointing to fragments of the generated code. - -You should just look for comments with syntax: -`{->markName-}` and convert them to a raw -\LaTeX{} string `\tikzmark{markName}`. - -For HTML, it generates `` which you -can then draw to with [a convenient JavaScript](https://stackoverflow.com/questions/554167/drawing-arrows-on-an-html-page-to-visualize-semantic-links-between-textual-spans#623770). - -### Alternate lexers - -Please note that using alternate lexers -disables token replacement! -This is important, since the token replacement -to \LaTeX{} special symbols is language-specific. - -#### Indent only - -Indent only is simplest to implement, -just start a column after initial whitespace in each line ends. - -#### Space only - -There should be an option to use alternate lexer. -It is driven solely by indentation. +```` -1. Cell break is delimited by a starting indent. -2. Or column of **consecutive** spaces that occur in more than one line, - **and** at least one line has at least **one more** space before it. +Or use alternate lexers: -Example input: +````markdown ```{.python} -# n is size of heap -def heapify(arr, n, i): - largest = i # Initialize largest as root - l = 2 * i + 1 # left = 2*i + 1 - r = 2 * i + 2 # right = 2*i + 2 -``` - -Example column division: -``` -# n is size of heap -def |heapify(arr, n, i).: - |largest = i # Ini.tialize largest as root - |l = 2 * i + 1 |# left = 2*i + 1 - |r = 2 * i + 2 |# right = 2*i + 2 +# your Python code ``` +```` -For highlighting we will later connect [`skylighting`](https://hackage.haskell.org/package/skylighting) -like Pandoc does natively. +For generic indentation-based alignment: -#### Python3 lexer {#sec:python-lexer} +````markdown +```{.haskell lexer=indent} +-- any indented code +``` +```` -Very easy to support, just add package [`language-python`](https://hackage.haskell.org/package/language-python-0.5.6/docs/Language-Python-Version3-Lexer.html) to dependencies. -The token type is different, but we only ever compare it by equality. +## Options -# Used tools +Configure via code block attributes: -## Finding Haskell tokens +- `lexer=haskell` - GHC lexer (default) +- `lexer=python3` - Python lexer +- `lexer=indent` - Indent-only alignment +- `lexer=spaces` - Space-based alignment +- `debug=true` - Show column boundaries for debugging -We find Haskell tokens with GHC-lib that uses GHC parser itself. -For simplicity we use [`ghc-syntax-highlighter`](https://hackage.haskell.org/package/ghc-syntax-highlighter-0.0.6.0/docs/GHC-SyntaxHighlighter.html) -We readd locations after the fact, since it gives you pure tokens with text and spaces, -or pure locations (without text). +Example: -``` -tokenizeHaskell :: Text -> Maybe [(Token, Text)] +````markdown +```{.haskell debug=true} +yourCode :: Here ``` - -## \LaTeX{} output - -For escaping text in TeX we use [HaTeX](http://hackage.haskell.org/package/HaTeX-3.5/docs/Text-LaTeX-Base-Render.html#t:Render)[@hatex] +```` -We render tables inline tables with `multicolumn` (not `polytable` like lhs2TeX[@lhs2tex] does.) +## Full Documentation -We do not use [HaTeX table support](https://hackage.haskell.org/package/HaTeX-3.22.2.0/docs/Text-LaTeX-Packages-Multirow.html) yet. +For detailed algorithm explanation, implementation guide, and advanced usage, see [DESCRIPTION.md](DESCRIPTION.md). -## Debugging +**See also:** [Complete Unicode operator symbol mapping](DESCRIPTION.md#appendix-operator-symbol-replacement) for all supported operator conversions (→, ⇒, ≥, ⊥, ∅, etc.). -Pandoc can automatically detect output format. -In order to get \LaTeX{} and HTML output just run: +## Links -```sh -pandoc input.md --filter=pandoc-filter-indent -o output.tex -pandoc input.md --filter=pandoc-filter-indent -o output.pdf -``` +- **GitHub:** +- **Hackage:** +- **Issues:** -For debugging indentation use `text` output format: -``` -pandoc input.md --filter=pandoc-filter-indent -o output.txt -``` -In text mode, it renders indent boundaries as `|` and `^`. +## License -Note that the filter *does not* touch the text outside code blocks. -It can add necessary \LaTeX{} headers or HTML styles in meta `headers-include`. +BSD3 - see LICENSE file diff --git a/app/Main.hs b/app/Main.hs --- a/app/Main.hs +++ b/app/Main.hs @@ -1,6 +1,5 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE ViewPatterns #-} -{-# LANGUAGE RecordWildCards #-} {-# LANGUAGE NamedFieldPuns #-} module Main where @@ -22,7 +21,7 @@ import System.IO import Debug.Trace(trace) -data Options = Options { +newtype Options = Options { inlineSyntax :: Text } deriving (Show) @@ -36,10 +35,12 @@ -- Reads format option, metadata, -- and calls `walk` with `blockFormatter`. runner :: Maybe Format -> Pandoc -> IO Pandoc -runner (fromMaybe (Format "text") -> format) input@(Pandoc (Meta meta) ast) = do - hPutStrLn stderr $ show opts - hPutStrLn stderr $ show $ Map.lookup "header-includes" meta - let top = if format `elem` [Format "latex", Format "tex"] +runner (fromMaybe (Format "text") -> format) input@(Pandoc (Meta meta) ast) = + do + hPutStrLn stderr $ "Output format: " <> show format + hPrint stderr opts + hPrint stderr (Map.lookup "header-includes" meta) + let top = if format `elem` [Format "latex", Format "tex", Format "beamer"] then Pandoc (Meta $ Map.alter modifyIncludes "header-includes" meta) ast else input return $ walk (blockFormatter opts format) top @@ -82,8 +83,8 @@ -> Format -- ^ Output format, defaults to "plain" if not found -> Block -> Block -blockFormatter _opts format (CodeBlock attrs content) = - case fmap findColumns $ getTokenizer attrs content of +blockFormatter _opts format (CodeBlock attrs content) = + case findColumns <$> getTokenizer attrs content of Just processed -> renderBlock format attrs processed Nothing -> CodeBlock attrs content -- fallback -- Do not touch other blocks than 'CodeBock' diff --git a/pandoc-filter-indent.cabal b/pandoc-filter-indent.cabal --- a/pandoc-filter-indent.cabal +++ b/pandoc-filter-indent.cabal @@ -1,18 +1,18 @@ cabal-version: 1.12 --- This file has been generated from package.yaml by hpack version 0.33.0. +-- This file has been generated from package.yaml by hpack version 0.38.0. -- -- see: https://github.com/sol/hpack --- --- hash: 1d8f275838382e4b48fbe6b67fb6036026bdfd772402682271ab1034e1ae5201 name: pandoc-filter-indent -version: 0.3.2.0 +version: 0.3.3.0 synopsis: Pandoc filter formatting Haskell code fragments using GHC lexer. -description: Formats marked code fragments, and allows `pandoc` to safely process rest of your literate program: +description: Formats marked code fragments, and allows `pandoc` to safely process rest of your literate program. + Mark code blocks with .haskell attribute: . > ```{.haskell} - . + > -- your code here + > ``` . Usage: . @@ -28,10 +28,10 @@ and creates tabular code structures from indentation. It uses GHC lexer to assure that latest features are always parsed correctly. . - Please see the README on GitHub at + Please see the README on GitHub at category: Text -homepage: https://github.com/mgajda/pandoc-filter-indent#readme -bug-reports: https://github.com/mgajda/pandoc-filter-indent/issues +homepage: https://github.com/migamake/pandoc-filter-typeset-code#readme +bug-reports: https://github.com/migamake/pandoc-filter-typeset-code/issues author: Michał J. Gajda maintainer: mjgajda@migamake.com copyright: AllRightsReserved @@ -40,11 +40,12 @@ build-type: Simple extra-source-files: README.md + DESCRIPTION.md ChangeLog.md source-repository head type: git - location: https://github.com/mgajda/pandoc-filter-indent + location: https://github.com/migamake/pandoc-filter-typeset-code library exposed-modules: @@ -66,16 +67,16 @@ hs-source-dirs: src build-depends: - HaTeX + HaTeX >=3.23.0 && <4 , base >=4.7 && <5 - , blaze-html - , blaze-markup - , ghc-syntax-highlighter - , optics-core - , optics-th - , pandoc-types - , skylighting - , text + , blaze-html >=0.9.2 && <1 + , blaze-markup >=0.8.3 && <1 + , ghc-syntax-highlighter >=0.0.12 && <1 + , optics-core >=0.4.1 && <1 + , optics-th >=0.4.1 && <1 + , pandoc-types >=1.23.1 && <2 + , skylighting >=0.14.7 && <1 + , text >=2.0.2 && <3 default-language: Haskell2010 executable pandoc-filter-indent @@ -86,41 +87,44 @@ app ghc-options: -threaded -rtsopts -with-rtsopts=-N build-depends: - HaTeX + HaTeX >=3.23.0 && <4 , base >=4.7 && <5 - , blaze-html - , blaze-markup - , containers - , ghc-syntax-highlighter - , optics-core - , optics-th - , optparse-applicative + , blaze-html >=0.9.2 && <1 + , blaze-markup >=0.8.3 && <1 + , containers >=0.6.7 && <1 + , ghc-syntax-highlighter >=0.0.12 && <1 + , optics-core >=0.4.1 && <1 + , optics-th >=0.4.1 && <1 + , optparse-applicative >=0.19.0 && <1 , pandoc-filter-indent - , pandoc-types - , skylighting - , text + , pandoc-types >=1.23.1 && <2 + , skylighting >=0.14.7 && <1 + , text >=2.0.2 && <3 default-language: Haskell2010 test-suite pandoc-filter-indent-test type: exitcode-stdio-1.0 main-is: Spec.hs other-modules: + LambdaSpec + UnicodeOperatorSpec Paths_pandoc_filter_indent hs-source-dirs: test ghc-options: -threaded -rtsopts -with-rtsopts=-N build-depends: - HaTeX + HaTeX >=3.23.0 && <4 , QuickCheck , base >=4.7 && <5 - , blaze-html - , blaze-markup - , ghc-syntax-highlighter - , optics-core - , optics-th + , blaze-html >=0.9.2 && <1 + , blaze-markup >=0.8.3 && <1 + , ghc-lib-parser + , ghc-syntax-highlighter >=0.0.12 && <1 + , optics-core >=0.4.1 && <1 + , optics-th >=0.4.1 && <1 , pandoc-filter-indent - , pandoc-types + , pandoc-types >=1.23.1 && <2 , quickcheck-text - , skylighting - , text + , skylighting >=0.14.7 && <1 + , text >=2.0.2 && <3 default-language: Haskell2010 diff --git a/src/Alignment.hs b/src/Alignment.hs --- a/src/Alignment.hs +++ b/src/Alignment.hs @@ -4,26 +4,26 @@ -- from tuples used in analysis. module Alignment where -import Data.Text (Text) -import Data.Tuple.Optics -import Optics.Lens +import Data.Text ( Text ) +import Data.Tuple.Optics ( Field1(..), Field3(..) ) +import Optics.Lens ( Lens' ) import Token ( MyLoc, MyTok ) -- | Datatype to present columns with alignment requirements. data Align = - ALeft - | ACenter - | AIndent -- indentation spacing + ALeft -- ^ Align to the left of the cell + | ACenter -- ^ Align to the center of the cell + | AIndent -- ^ Indentation spacing: whitespace with minimum width to preserve deriving (Eq, Ord, Show) -- | Records tokenized and converted to common token format. -type Processed = (MyTok -- ^ Token type - ,MyLoc -- ^ Token location - ,Text -- ^ Text content - ,Maybe Int -- ^ Indent column - ,Maybe (Align -- ^ Alignment mark - ,Int) -- ^ Alignment column +type Processed = (MyTok -- Token type + ,MyLoc -- Token location + ,Text -- Text content + ,Maybe Int -- Indent column + ,Maybe (Align -- Alignment mark + ,Int) -- Alignment column ) diff --git a/src/Filter.hs b/src/Filter.hs --- a/src/Filter.hs +++ b/src/Filter.hs @@ -1,7 +1,6 @@ {-# LANGUAGE AllowAmbiguousTypes #-} {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE PartialTypeSignatures #-} -{-# LANGUAGE ViewPatterns #-} {-# LANGUAGE FlexibleInstances #-} {-# LANGUAGE FlexibleContexts #-} -- | Filtering a single code block. diff --git a/src/FindColumns.hs b/src/FindColumns.hs --- a/src/FindColumns.hs +++ b/src/FindColumns.hs @@ -1,7 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE PartialTypeSignatures #-} {-# LANGUAGE ViewPatterns #-} -{-# LANGUAGE TypeSynonymInstances #-} +{-# LANGUAGE ExplicitForAll #-} {-# LANGUAGE FlexibleInstances #-} {-# LANGUAGE FlexibleContexts #-} -- | Find alignment columns. @@ -12,7 +12,7 @@ import Data.Function (on) import Data.String (fromString, IsString) import Data.Text (Text) -import Data.List (groupBy, sortBy, sort, group, findIndex) +import Data.List (groupBy, sortBy, sort, group, elemIndex) import Prelude hiding (getLine) import Optics.Core import Data.Tuple.Optics @@ -26,15 +26,15 @@ -- | Records tokenized and converted to common token format. type Unanalyzed = (MyTok, MyLoc, Text - , Maybe Int -- ^ Indent for entire line + , Maybe Int -- Indent for entire line ) -- | Records aligned, but without column numbers yet. -type Aligned = (MyTok -- ^ token type - ,MyLoc -- ^ text start location - ,Text -- ^ text content - ,Maybe Int -- ^ indent in this column - ,Maybe Align -- ^ alignment +type Aligned = (MyTok -- token type + ,MyLoc -- text start location + ,Text -- text content + ,Maybe Int -- indent in this column + ,Maybe Align -- alignment ) -- * Definitions of fields accessible at many stages @@ -55,11 +55,9 @@ columnIndices . withExtraColumns . sortBy (compare `on` getLineCol) - . concat - . map markBoundaries + . concatMap markBoundaries . grouping getCol - . concat - . map addLineIndent + . concatMap addLineIndent . grouping getLine -- | Extract both line and column where a token starts. @@ -69,9 +67,8 @@ -- | Mark alignment boundaries. markBoundaries :: [Unanalyzed] -> [Aligned] markBoundaries = map markIndent - . concat -- . (\b -> trace ("ALIGNED: " <> show b) b) - . map alignBlock + . concatMap alignBlock -- . (\b -> trace ("BLOCKS: " <> show b) b) . blocks . sortBy (compare `on` getLine) @@ -111,11 +108,13 @@ apply entry = entry `annex` Nothing -- | Compute all alignment columns existing and their positions in the text column space. +-- Note: Deduplicates by text column position only (not by alignment type) +-- to ensure consistency with tableColumns and avoid gaps in table column indices. extraColumns :: Field2 a a MyLoc MyLoc => Field5 a a (Maybe Align) (Maybe Align) => [a] -> [(Int, Maybe Align)] extraColumns = - nubSorted + nubSortedBy (compare `on` fst) -- Deduplicate by text column position only . filter hasAlignment . map extract where @@ -186,7 +185,7 @@ mod (Just a) = Just (a, colIndex) where - colIndex = case findIndex (==getCol aligned) markerColumns of + colIndex = case elemIndex (getCol aligned) markerColumns of Nothing -> error $ "Did not find the index for column " <> show (getCol aligned) <> " within " <> show markerColumns Just i -> i diff --git a/src/Render/ColSpan.hs b/src/Render/ColSpan.hs --- a/src/Render/ColSpan.hs +++ b/src/Render/ColSpan.hs @@ -33,7 +33,7 @@ $ sortBy (compare `on` getLineCol) ps where maxCol :: Int - maxCol = maximum $ map getAlignCol $ ps + maxCol = maximum $ map getAlignCol ps sameColSpan :: Processed -> Processed -> Bool sameColSpan tok1 tok2 = case view alignPos tok2 of Nothing | getLine tok1 == getLine tok2 -> True @@ -53,8 +53,7 @@ numColSpans :: [Processed] -> Int numColSpans ps = case colspansPerLine of [] -> 1 - n:ns -> assert (all (n==) ns) - $ n + n:ns -> assert (all (n==) ns) n where colspansPerLine :: [Int] colspansPerLine = map (sum . map (\(_,c,_) -> c)) . colspans $ ps diff --git a/src/Render/Debug.hs b/src/Render/Debug.hs --- a/src/Render/Debug.hs +++ b/src/Render/Debug.hs @@ -50,7 +50,7 @@ alignMarker = case view alignPos tok of Just (ACenter, _) -> "^" Just _ -> "|" - otherwise -> "" + _ -> "" -- | Text content with markers for markers inside it. textWithMarkers tColumns nextCol tok = diff --git a/src/Render/HTML.hs b/src/Render/HTML.hs --- a/src/Render/HTML.hs +++ b/src/Render/HTML.hs @@ -80,6 +80,7 @@ formatToken (TOperator,"|>" ) = "⊳" formatToken (TOperator,"<>" ) = "⋄" formatToken (TOperator,"=>" ) = "⇒" +formatToken (TOther, "=" ) = "＝" formatToken (TOperator,"->" ) = "→" formatToken (TOperator,"|->" ) = "↦" formatToken (TVar ,"undefined") = "⊥" @@ -105,6 +106,8 @@ formatToken (TOperator,"-<" ) = "≺" formatToken (TOperator,">-" ) = "≻" formatToken (TOperator,"!=" ) = "≠" +formatToken (TOperator,"==" ) = "=" +formatToken (TOperator,"=" ) = "＝" formatToken (TOperator,"\\/" ) = "⋁" formatToken (TOperator,"/\\" ) = "⋀" formatToken (TOperator,"~" ) = "∼" @@ -113,7 +116,7 @@ formatToken (TKeyword, kwd ) = b $ toHtml kwd formatToken (TVar, v ) = i $ toHtml v formatToken (TCons, v ) = span (toHtml v) - ! style ("font-variant: small-caps;") + ! style "font-variant: small-caps;" formatToken (TTikz mark,_ ) = span "" ! id (toValue mark) formatToken (_, txt ) = toHtml txt diff --git a/src/Render/Latex.hs b/src/Render/Latex.hs --- a/src/Render/Latex.hs +++ b/src/Render/Latex.hs @@ -1,19 +1,57 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE ViewPatterns #-} +{-# LANGUAGE PatternSynonyms #-} -- | Render analyzed input into LaTeX table. -module Render.Latex(latexFromColSpans, latexInline, latexPackages, subscripts) where +module Render.Latex(latexFromColSpans, latexInline, latexPackages, subAndSuperscripts) where import Data.Text(Text) import qualified Data.Text as T -import Data.Char(isAlpha) +import Data.Char(isAlpha, isAsciiLower, isAsciiUpper, isDigit) import Text.LaTeX.Base.Syntax(protectText) +import Data.Maybe (fromMaybe) import Alignment ( Align(..) ) import Render.Common(TokensWithColSpan) import Token(MyTok(..)) import Util(unbrace) +-- | Protect special LaTeX characters for use in math mode. +-- Handles Haskell operators with backslashes that Skylighting doesn't tokenize properly. +protectTextMath :: Text -> Text +protectTextMath = processText + where + processText :: Text -> Text + processText t + | T.null t = "" + -- Handle specific multi-character operators first + | "\\\\" `T.isPrefixOf` t = "\\setminus " <> processText (T.drop 2 t) + | "\\/" `T.isPrefixOf` t = "\\land " <> processText (T.drop 2 t) + | "/\\" `T.isPrefixOf` t = "\\lor " <> processText (T.drop 2 t) + -- Handle lambda: backslash followed by letter/digit + | Just ('\\', rest) <- T.uncons t + , Just (nextChar, _) <- T.uncons rest + , isAlphaNum nextChar = "\\lambda " <> processText rest + -- Handle standalone backslash followed by operator chars + | Just ('\\', rest) <- T.uncons t = "\\backslash " <> processText rest + -- Handle other special chars + | otherwise = + let (c, rest) = fromMaybe (' ', "") $ T.uncons t + in escapeChar c <> processText rest + + escapeChar :: Char -> Text + escapeChar '#' = "\\#" + escapeChar '$' = "\\$" + escapeChar '%' = "\\%" + escapeChar '&' = "\\&" + escapeChar '_' = "\\_" + escapeChar '{' = "\\{" + escapeChar '}' = "\\}" + escapeChar c = T.singleton c + + isAlphaNum :: Char -> Bool + isAlphaNum c = isAsciiLower c || isAsciiUpper c || isDigit c + -- | Given a number of table columns, -- and a list of lists of colspans for each table row, -- return raw LaTeX code. @@ -60,7 +98,7 @@ -- | Preprocesses functions converted to operator syntax and joins them into a single token. -- FIXME: deduplicate preformatTokens [] = [] -preformatTokens ((TOperator,"`"):(TVar, "elem"):(TOperator, "`"):rest) = (TOperator, "elem"):preformatTokens rest +preformatTokens ((TOther, "`"):(TVar, "elem"):(TOther, "`"):rest) = (TOperator, "elem"):preformatTokens rest preformatTokens (a :rest) = a :preformatTokens rest @@ -72,22 +110,48 @@ . preformatTokens -- | Add subscripts and superscripts to variable names. --- `_` is subscript, and `__` is superscript.vim -subscripts :: Text -> Text -subscripts "" = " " -subscripts "_" = "\\_" -subscripts t = segments t +-- "_" is subscript, and "__" is superscript. +-- Superscripts nest their remaining content recursively. +subAndSuperscripts :: Text -> Text +subAndSuperscripts "" = " " +subAndSuperscripts "_" = "\\_" +subAndSuperscripts t = case T.splitOn "_" t of + [] -> "" + (x:xs) -> x <> processSegments xs where - segments = foldr1 addSubscript . T.splitOn "_" - -- | Tags the second argument with superscript "^" or subscript "_" - addSubscript :: Text -> Text -> Text - addSubscript t tsub = mconcat [t, "\\textsubscript{", tsub, "}"] + -- Process segments after splitting by "_" + -- Empty string indicates double underscore (superscript) + processSegments :: [Text] -> Text + processSegments [] = "" + processSegments ("":rest) = case rest of + [] -> "" -- Trailing "__" + _ -> let remainingText = T.intercalate "_" rest + in "\\textsuperscript{" <> subAndSuperscripts remainingText <> "}" + processSegments (x:xs) = "\\textsubscript{" <> x <> "}" <> processSegments xs +-- | Pattern for operator-like tokens (TOperator or TOther). +-- Both tokenizers produce different token types for the same operators: +-- - Haskell tokenizer: most operators → TOther +-- - Skylighting tokenizer: some operators → TOperator (e.g., *, >-, -<, >>-, -<<) +-- This pattern synonym eliminates duplication when handling the same operator from both tokenizers. +pattern OpOrOther :: Text -> (MyTok, Text) +pattern OpOrOther txt <- (matchOpOrOther -> Just txt) + +matchOpOrOther :: (MyTok, Text) -> Maybe Text +matchOpOrOther (TOperator, txt) = Just txt +matchOpOrOther (TOther, txt) = Just txt +matchOpOrOther _ = Nothing + -- Workaround with joinEscapedOperators til w consider spaces only. -- | Render a simple token. formatToken :: (MyTok, Text) -> Text formatToken (TOperator,unbrace -> Just op) = "(" <> formatToken (TOperator, op) <> ")" +-- GHC UnicodeSyntax extension operators +-- See: https://downloads.haskell.org/ghc/latest/docs/users_guide/exts/unicode_syntax.html +-- Note: ∀ can be TKeyword (Haskell) or TOther (Skylighting) formatToken (TKeyword, "forall") = mathop "forall" +formatToken (TKeyword, "∀" ) = mathop "forall" +formatToken (OpOrOther "∀") = mathop "forall" --formatToken (TVar, "mempty") = mathop "emptyset" formatToken (TVar, "bottom") = mathop "bot" formatToken (TVar, "undefined") = mathop "perp" @@ -102,6 +166,8 @@ formatToken (TOperator,"\\/" ) = mathop "land" formatToken (TOperator,"\\|/" ) = mathop "downarrow" formatToken (TOperator,"\\||/" ) = mathop "Downarrow" +formatToken (TOperator,"/|\\" ) = mathop "uparrow" +formatToken (TOperator,"/||\\" ) = mathop "Uparrow" formatToken (TOperator,"~>" ) = mathop "leadsto" formatToken (TOperator,"|=" ) = mathop "models" formatToken (TCons ,"Natural") = "\\mathbb{N}" @@ -115,19 +181,44 @@ formatToken (TOperator,">>>" ) = mathop "ggg" formatToken (TOperator,"<<" ) = mathop "ll" formatToken (TOperator,"<<<" ) = mathop "lll" -formatToken (TOther, "λ" ) = mathop "lambda" -- Haskell only? ---formatToken (TOperator,"-<" ) = mathop "prec" -formatToken (TOther, "-<" ) = mathop "prec" -formatToken (TOther, ">-" ) = mathop "succ" -formatToken (TOperator,"\\\\" ) = mathop "setminus" -formatToken (TOther ,"<-" ) = mathop "gets" +formatToken (TOperator,"\\\\" ) = mathop "setminus" -- MUST come before single backslash! +formatToken (OpOrOther "\\") = mathop "lambda" -- Lambda (both tokenizers) +formatToken (OpOrOther "λ") = mathop "lambda" -- Unicode lambda (TOther from Skylighting) +formatToken (TVar, "λ" ) = mathop "lambda" -- Unicode lambda (TVar from Haskell tokenizer) +formatToken (OpOrOther "-<") = mathop "prec" -- Left arrow-tail +formatToken (OpOrOther "⤙") = mathop "prec" -- Unicode -< +formatToken (OpOrOther ">-") = mathop "succ" -- Right arrow-tail +formatToken (OpOrOther "⤚") = mathop "succ" -- Unicode >- +formatToken (OpOrOther "<-") = mathop "gets" -- Left arrow +formatToken (OpOrOther "←") = mathop "gets" -- Unicode <- formatToken (TOperator,">=" ) = mathop "geq" formatToken (TOperator,"<=" ) = mathop "leq" formatToken (TOperator,"!=" ) = mathop "ne" -formatToken (TOperator,"<->" ) = mathop "leftrightarrow" ---formatToken (TOperator,"->" ) = mathop "to" -formatToken (TOther, "->" ) = mathop "to" -formatToken (TOther, "=>" ) = mathop "Rightarrow" +formatToken (TOperator,"<->" ) = mathop "updownarrow" +formatToken (TOperator,"<|>" ) = mathop "leftrightarrow" +formatToken (OpOrOther "->") = mathop "to" +formatToken (OpOrOther "→") = mathop "to" -- Unicode -> +formatToken (OpOrOther "=>") = mathop "Rightarrow" +formatToken (OpOrOther "⇒") = mathop "Rightarrow" -- Unicode => +formatToken (OpOrOther "::") = mathop ":" -- Type annotation +formatToken (OpOrOther "∷") = mathop ":" -- Unicode :: +-- Note: * is rendered as \times (multiplication) since modern Haskell uses Type for kinds +-- TODO: Make this configurable for legacy code that uses * for kinds +formatToken (OpOrOther "*") = mathop "times" -- Multiplication (star for old kind syntax) +formatToken (OpOrOther "★") = mathop "star" -- Unicode kind star +formatToken (OpOrOther ">>-") = mathop "rr" -- Right double arrow-tail +formatToken (OpOrOther "⤜") = mathop "rr" -- Unicode >>- +formatToken (OpOrOther "-<<") = mathop "ll" -- Left double arrow-tail +formatToken (OpOrOther "⤛") = mathop "ll" -- Unicode -<< +formatToken (OpOrOther "⊸") = mathop "multimap" -- Unicode linear arrow %1-> +formatToken (OpOrOther "(|") = mathop "llparenthesis" -- Parallel array bracket (stmaryrd) +formatToken (OpOrOther "⦇") = mathop "llparenthesis" -- Unicode (| +formatToken (OpOrOther "|)") = mathop "rrparenthesis" -- Parallel array bracket (stmaryrd) +formatToken (OpOrOther "⦈") = mathop "rrparenthesis" -- Unicode |) +formatToken (OpOrOther "[|") = mathop "llbracket" -- Quasiquote bracket (stmaryrd) +formatToken (OpOrOther "⟦") = mathop "llbracket" -- Unicode [| +formatToken (OpOrOther "|]") = mathop "rrbracket" -- Quasiquote bracket (stmaryrd) +formatToken (OpOrOther "⟧") = mathop "rrbracket" -- Unicode |] formatToken (TOperator,"==>" ) = mathop "implies" formatToken (TOperator,"|->" ) = mathop "mapsto" formatToken (TOperator,"|=>" ) = mathop "Mapsto" -- requires stmaryrd @@ -137,6 +228,8 @@ formatToken (TOperator,"elem" ) = mathop "in" formatToken (TOperator,"~" ) = mathop "sim" formatToken (TOperator,"~=" ) = mathop "approx" +formatToken (TOperator,"><" ) = mathop "times" +formatToken (TOperator,":->" ) = mathop "longmapsto" formatToken (TVar, "a" ) = mathop "alpha" formatToken (TVar, "b" ) = mathop "beta" formatToken (TVar, "c" ) = mathop "gamma" @@ -145,16 +238,16 @@ formatToken (TVar, "k" ) = mathop "kappa" formatToken (TVar, "n" ) = mathop "nu" formatToken (TVar, "m" ) = mathop "mu" -formatToken (TVar, "s" ) = mathop "sigma" -formatToken (TVar, "o" ) = mathop "omega" +formatToken (TVar, "sigma" ) = mathop "sigma" +formatToken (TVar, "omega" ) = mathop "omega" formatToken (TVar, "pi" ) = mathop "pi" formatToken (TVar, "tau" ) = mathop "tau" formatToken (TVar, "rho" ) = mathop "rho" -formatToken (TVar , txt ) | T.any isAlpha txt = "\\textit{" <> subscripts txt <> "}" +formatToken (TVar , txt ) | T.any isAlpha txt = "\\textit{" <> subAndSuperscripts txt <> "}" formatToken (TVar, txt ) = "\\textit{" <> protectText txt <> "}" -formatToken (TNum , kwd ) = protectText kwd +formatToken (TNum , kwd ) = protectText kwd formatToken (TKeyword, kwd ) = "\\textbf{" <> protectText kwd <> "}" -formatToken (TCons, cons ) = "\\textsc{" <> subscripts cons <> "}" +formatToken (TCons, cons ) = "\\textsc{" <> subAndSuperscripts cons <> "}" --formatToken (TOperator,"\\" ) = mathop "lambda" formatToken (TTikz mark,_ ) = mathop $ "tikzMark{" <> mark <> "}" --formatToken (TOther, "`" ) = mathop "textasciigrave" @@ -167,8 +260,10 @@ formatToken (TOther, "[" ) = protectText "[" formatToken (TOther, "}" ) = protectText "}" formatToken (TOther, "{" ) = protectText "{" +--formatToken (TOther, "=" ) = "\\scalebox{1.7}{" <> mathop "=" <> "}" +formatToken (TOther, "=" ) = "=\\joinrel=" -- formatToken (TBlank, txt ) = "\\textit{\\textcolor{gray}{" <> protectText txt <> "}}" -formatToken (_, txt ) = "\\textrm{" <> protectText txt <> "}" +formatToken (_, txt ) = "\\textrm{" <> protectTextMath txt <> "}" mathop :: Text -> Text mathop code = "\\" <> code diff --git a/src/Token.hs b/src/Token.hs --- a/src/Token.hs +++ b/src/Token.hs @@ -8,7 +8,7 @@ import Data.Text(Text) import qualified Data.Text as T -import Optics.TH +import Optics.TH ( makeLenses ) -- * Common tokens and locations -- We keep them here, so we can translate output from tokenizers to common format. @@ -23,21 +23,21 @@ -- | Token just classifies to blank, operator, and the style class data MyTok = - TBlank - | TOperator - | TKeyword - | TCons - | TVar - | TNum - | TOther - | TString - | TTikz Text -- TikZmark in a comment + TBlank -- ^ Whitespace or comments + | TOperator -- ^ Operators + | TKeyword -- ^ Language-specific keywords + | TCons -- ^ Constructors + | TVar -- ^ Variables, function names + | TNum -- ^ Numbers + | TOther -- ^ Other tokens + | TString -- ^ String constants + | TTikz Text -- ^ TikZmark in a comment deriving (Eq, Ord, Show) -- | Records tokenized and converted to common token format. -type Tokenized = (MyTok -- ^ Token type - ,MyLoc -- ^ Starting location for the token - ,Text -- ^ text value of the token +type Tokenized = (MyTok -- Token type + ,MyLoc -- Starting location for the token + ,Text -- text value of the token ) -- | Unpack a Haskell comment with a TikZ mark indicator. diff --git a/src/Token/Haskell.hs b/src/Token/Haskell.hs --- a/src/Token/Haskell.hs +++ b/src/Token/Haskell.hs @@ -30,15 +30,29 @@ tokenizer = fmap ( joinEscapedOperators . splitTokens . restoreLocations + . splitUnicodeLambda . fmap recognizeToken) . tokenizeHaskell +-- | Split tokens that start with unicode lambda (λ) into separate tokens. +-- E.g., "λx" becomes ["λ", "x"] +-- Note: λ is not officially part of GHC's UnicodeSyntax extension, +-- but is widely used as a de facto standard for backslash in lambdas. +splitUnicodeLambda :: [(MyTok, Text)] -> [(MyTok, Text)] +splitUnicodeLambda [] = [] +splitUnicodeLambda ((tok, txt):rest) + | "λ" `T.isPrefixOf` txt && T.length txt > 1 = + (TOther, "λ") : splitUnicodeLambda ((tok, T.drop 1 txt):rest) + | otherwise = (tok, txt) : splitUnicodeLambda rest + -- | Recognize token using both token type from `ghc-lib`, -- and text content. -- Only TikZ marks are recognized by looking up text content. recognizeToken :: (Token, Text) -> (MyTok, Text) recognizeToken (SymbolTok, "\\") = -- special treatment for lambda (TOther, "λ" ) +recognizeToken (SymbolTok, "λ") = -- Unicode lambda (de facto standard) + (TOther, "λ" ) recognizeToken (CommentTok, tokText@(unTikzMark -> Just mark)) = (TTikz mark, tokText) recognizeToken (tokType, tokText) = @@ -106,7 +120,7 @@ withNewLines = fmap (<>"\n") (init split) <> [last split] withLocs :: [Text] -> [(MyTok, MyLoc, Text)] - withLocs (l:ls) = (TBlank, set mark True $ loc, l) + withLocs (l:ls) = (TBlank, set mark True loc, l) : zipWith mkEntry [line+1..] ls mkEntry :: Int -> Text -> (MyTok, MyLoc, Text) mkEntry i t = (TBlank, MyLoc i 1 True, t) diff --git a/src/Token/Skylighting.hs b/src/Token/Skylighting.hs --- a/src/Token/Skylighting.hs +++ b/src/Token/Skylighting.hs @@ -1,15 +1,15 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE PartialTypeSignatures #-} -{-# LANGUAGE ViewPatterns #-} {-# LANGUAGE FlexibleContexts #-} -- | Skylighting code tokenizer -module Token.Skylighting(lookupTokenizer, tokenizer) where +module Token.Skylighting(lookupTokenizer, tokenizer, fixBackslashOperators) where import Control.Arrow(first) import Text.Pandoc.JSON () import Text.Pandoc.Definition () -import Data.Maybe(listToMaybe, catMaybes) +import Data.Maybe(listToMaybe, mapMaybe) import Data.String (IsString) +import Data.Char(isAsciiLower, isAsciiUpper, isDigit) import Data.Text (Text) import qualified Data.Text as T import Prelude hiding(getLine) @@ -23,13 +23,15 @@ import Token ( MyLoc(MyLoc), MyTok(..), unTikzMark, mark ) +rightToMaybe :: Either a b -> Maybe b rightToMaybe (Left err ) = Nothing rightToMaybe (Right result) = Just result +-- | Looks up the tokenizer from Skylighting preset library by the short name of the language. +-- Picks the first match. lookupTokenizer :: [Text] -> Maybe Syntax lookupTokenizer = listToMaybe - . catMaybes - . fmap (Sky.syntaxByShortName Sky.defaultSyntaxMap) + . mapMaybe (Sky.syntaxByShortName Sky.defaultSyntaxMap) -- * Haskell tokenizer frontend -- | Attempt to tokenize input, @@ -41,44 +43,58 @@ -> Maybe [(MyTok, MyLoc, Text)] tokenizer syntax = fmap ( joinEscapedOperators + . fixBackslashOperators -- Fix Skylighting's incorrect splitting of backslash operators + . fixBracketOperators -- Fix Skylighting's splitting of bracket operators like (|, |), [|, |] . splitTokens . restoreLocations + . splitUnicodeLambda -- Split unicode lambda from following characters . recognizeTokens ) . rightToMaybe . Sky.tokenize tokenizerOpts syntax where tokenizerOpts = Sky.TokenizerConfig Sky.defaultSyntaxMap False +-- | Split tokens that start with unicode lambda (λ) into separate tokens. +-- E.g., "λx" becomes ["λ", "x"] +-- Note: λ is not officially part of GHC's UnicodeSyntax extension, +-- but is widely used as a de facto standard for backslash in lambdas. +splitUnicodeLambda :: [[(MyTok, Text)]] -> [[(MyTok, Text)]] +splitUnicodeLambda = map (concatMap splitToken) + where + splitToken (tok, txt) + | "λ" `T.isPrefixOf` txt && T.length txt > 1 = + [(TOther, "λ"), (tok, T.drop 1 txt)] + | otherwise = [(tok, txt)] + -- | Recognize tokens from all source lines. recognizeTokens :: [SourceLine] -> [[(MyTok, Text)]] recognizeTokens = map $ map $ first skyTok -- | Convert token type of `ghc-lib` into tokens recognized by the filter. skyTok :: TokenType -> MyTok -skyTok FloatTok = TNum -skyTok DecValTok = TNum -skyTok BaseNTok = TNum -skyTok StringTok = TString -skyTok CharTok = TString -skyTok FunctionTok = TVar -skyTok AttributeTok = TBlank +skyTok FloatTok = TNum +skyTok DecValTok = TNum +skyTok BaseNTok = TNum +skyTok StringTok = TString +skyTok CharTok = TString +skyTok FunctionTok = TVar +skyTok AttributeTok = TBlank skyTok VerbatimStringTok = TString -skyTok SpecialStringTok = TCons -skyTok ConstantTok = TCons -skyTok KeywordTok = TKeyword -skyTok BuiltInTok = TKeyword -skyTok PreprocessorTok = TBlank -skyTok CommentTok = TBlank -skyTok DocumentationTok = TBlank -skyTok CommentTok = TBlank -skyTok OperatorTok = TOperator -skyTok SpecialCharTok = TOperator -skyTok RegionMarkerTok = TOperator -skyTok AnnotationTok = TBlank -skyTok ControlFlowTok = TKeyword -skyTok VariableTok = TVar -skyTok DataTypeTok = TCons -skyTok other = TOther +skyTok SpecialStringTok = TCons +skyTok ConstantTok = TCons +skyTok KeywordTok = TKeyword +skyTok BuiltInTok = TKeyword +skyTok PreprocessorTok = TBlank +skyTok CommentTok = TBlank +skyTok DocumentationTok = TBlank +skyTok OperatorTok = TOperator +skyTok SpecialCharTok = TOperator +skyTok RegionMarkerTok = TOperator +skyTok AnnotationTok = TBlank +skyTok ControlFlowTok = TKeyword +skyTok VariableTok = TVar +skyTok DataTypeTok = TCons +skyTok other = TOther -- FIXME: generalize for GHC tokenizer and Skylighting -- | Restore locations @@ -122,7 +138,7 @@ withNewLines = fmap (<>"\n") (init split) <> [last split] withLocs :: [Text] -> [(MyTok, MyLoc, Text)] - withLocs (l:ls) = (TBlank, set mark True $ loc, l) + withLocs (l:ls) = (TBlank, set mark True loc, l) : zipWith mkEntry [line+1..] ls mkEntry :: Int -> Text -> (MyTok, MyLoc, Text) mkEntry i t = (TBlank, MyLoc i 1 True, t) @@ -132,6 +148,70 @@ unmark :: Field2 a a MyLoc MyLoc => a -> a unmark = set (_2 % mark) False + +-- | Fix Skylighting's tokenization of bracket operators used in GHC extensions. +-- Skylighting splits operators like (|, |), [|, |] which should be single tokens. +-- This function merges them back together. +fixBracketOperators :: [(MyTok, MyLoc, Text)] -> [(MyTok, MyLoc, Text)] +fixBracketOperators [] = [] +-- (| parallel array bracket +fixBracketOperators ((TOther, loc, "("):(TOperator, _, "|"):remaining) = + (TOperator, loc, "(|") : fixBracketOperators remaining +-- |) parallel array bracket +fixBracketOperators ((TOperator, loc, "|"):(TOther, _, ")"):remaining) = + (TOperator, loc, "|)") : fixBracketOperators remaining +-- [| quasiquote bracket +fixBracketOperators ((TOther, loc, "["):(TOperator, _, "|"):remaining) = + (TOperator, loc, "[|") : fixBracketOperators remaining +-- |] quasiquote bracket +fixBracketOperators ((TOperator, loc, "|"):(TOther, _, "]"):remaining) = + (TOperator, loc, "|]") : fixBracketOperators remaining +-- Default case +fixBracketOperators (tok:rest) = + tok : fixBracketOperators rest + +-- | Fix Skylighting's incorrect tokenization of backslash operators. +-- Skylighting incorrectly splits operators like \+, \>, \/, etc. +-- This function merges them back together to match Haskell tokenizer behavior. +fixBackslashOperators :: [(MyTok, MyLoc, Text)] -> [(MyTok, MyLoc, Text)] +fixBackslashOperators [] = [] +-- Standalone backslash followed by backslash -> \\ set difference operator +fixBackslashOperators ((TOther, loc, "\\"):(TOther, _, "\\"):remaining) = + (TOperator, loc, "\\\\") : fixBackslashOperators remaining +-- Standalone backslash followed by operator symbol -> merge as operator (\+, \>, etc.) +fixBackslashOperators ((TOther, loc, "\\"):(TOperator, _, op):remaining) = + (TOperator, loc, "\\" <> op) : fixBackslashOperators remaining +-- Standalone backslash followed by variable -> lambda, keep separate +fixBackslashOperators (tok1@(TOther, _, "\\"):rest@((TVar, _, _):_)) = + tok1 : fixBackslashOperators rest +-- Standalone backslash followed by alphanumeric TOther -> lambda, keep separate +fixBackslashOperators (tok1@(TOther, loc1, "\\"):tok2@(TOther, loc2, txt):rest) + | not (T.null txt) && isAlphaStart txt = + tok1 : fixBackslashOperators (tok2:rest) +-- Token ending with backslash followed by / -> merge as \/ operator +fixBackslashOperators ((TOther, loc, txt):(TOperator, _, "/"):remaining) + | "\\" `T.isSuffixOf` txt = + (TOperator, loc, T.init txt <> "\\/") : fixBackslashOperators remaining +-- Operator / followed by token starting with backslash -> merge as /\ operator +fixBackslashOperators ((TOperator, loc, "/"):(TOther, _, txt):remaining) + | "\\" `T.isPrefixOf` txt = + (TOperator, loc, "/\\") : fixBackslashOperators ((TOther, loc, T.tail txt):remaining) +-- Token starting with backslash followed by dot (lambda notation like \f.) +fixBackslashOperators (tok1@(TOther, _, txt):tok2@(TOperator, _, "."):rest) + | "\\" `T.isPrefixOf` txt && T.length txt > 1 && isAlphaRest (T.tail txt) = + tok1 : tok2 : fixBackslashOperators rest +-- Default case - keep token as is +fixBackslashOperators (tok:rest) = + tok : fixBackslashOperators rest + +-- Helper functions for fixBackslashOperators +isAlphaStart :: Text -> Bool +isAlphaStart txt = case T.uncons txt of + Just (c, _) -> isAsciiLower c || isAsciiUpper c + Nothing -> False + +isAlphaRest :: Text -> Bool +isAlphaRest = T.all (\c -> isAsciiLower c || isAsciiUpper c || isDigit c) -- FIXME: use no-indent-mark instead. joinEscapedOperators :: (Eq c, IsString c, Semigroup c) => [(MyTok, MyLoc, c)] -> [(MyTok, MyLoc, c)] diff --git a/src/Tuples.hs b/src/Tuples.hs --- a/src/Tuples.hs +++ b/src/Tuples.hs @@ -1,4 +1,5 @@ {-# LANGUAGE FunctionalDependencies #-} +{-# LANGUAGE MultiParamTypeClasses #-} {-# LANGUAGE FlexibleInstances #-} -- | Tuple utilities. module Tuples where diff --git a/src/Util.hs b/src/Util.hs --- a/src/Util.hs +++ b/src/Util.hs @@ -1,9 +1,6 @@ {-# LANGUAGE AllowAmbiguousTypes #-} {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE PartialTypeSignatures #-} -{-# LANGUAGE ViewPatterns #-} -{-# LANGUAGE TupleSections #-} -{-# LANGUAGE TypeSynonymInstances #-} {-# LANGUAGE FlexibleInstances #-} {-# LANGUAGE FlexibleContexts #-} -- | Shared utilities that may be moved to upstream libraries. @@ -52,6 +49,8 @@ -- and end in closing brace. unbrace txt | T.head txt == '(' && T.last txt == ')' && T.length txt > 2 = Just $ T.tail $ T.init txt unbrace _ = Nothing + +brace txt = mconcat ["(", txt, ")"] -- | Preprocess tokens before formatting -- in order to detect tokens like functions converted to operator syntax. diff --git a/test/LambdaSpec.hs b/test/LambdaSpec.hs new file mode 100644 --- /dev/null +++ b/test/LambdaSpec.hs @@ -0,0 +1,298 @@ +{-# LANGUAGE OverloadedStrings #-} +-- | Tests for tokenizer consistency on backslash operators and lambda expressions. +-- Compares behavior between Haskell tokenizer and Skylighting tokenizer. +module LambdaSpec where + +import Test.QuickCheck +import qualified Data.Text as T +import Data.Text (Text) +import Data.Maybe (fromMaybe) +import Control.Monad (liftM2, liftM3) + +import qualified Token.Haskell as Haskell +import qualified Token.Skylighting as Sky +import Token (MyTok(..), MyLoc(..)) +import Render.Latex (latexInline) + +-- * QuickCheck Generators for Haskell Code + +-- | Generate valid Haskell variable names (lowercase start) +genVarName :: Gen Text +genVarName = do + first <- elements ['a'..'z'] + rest <- resize 5 $ listOf (elements $ ['a'..'z'] ++ ['A'..'Z'] ++ ['0'..'9'] ++ ['\'']) + return $ T.pack (first:rest) + +-- | Generate simple Haskell expressions +genSimpleExpr :: Gen Text +genSimpleExpr = frequency + [ (5, genVarName) + , (2, fmap (T.pack . show) (arbitrary :: Gen Int)) + , (1, liftM2 (\a b -> a <> " + " <> b) genVarName genVarName) + , (1, liftM2 (\a b -> a <> " * " <> b) genVarName genVarName) + ] + +-- | Generate lambda expressions with single parameter +-- Examples: \x -> x, \y -> y + 1 +genLambdaSimple :: Gen Text +genLambdaSimple = do + var <- genVarName + body <- genSimpleExpr + return $ "\\" <> var <> " -> " <> body + +-- | Generate lambda expressions with multiple parameters +-- Examples: \x y -> x + y +genLambdaMulti :: Gen Text +genLambdaMulti = do + n <- choose (2, 3) + vars <- vectorOf n genVarName + body <- genSimpleExpr + return $ "\\" <> T.intercalate " " vars <> " -> " <> body + +-- | Generate nested lambda expressions (curried functions) +-- Examples: \x -> \y -> x + y +genLambdaNested :: Gen Text +genLambdaNested = do + n <- choose (2, 3) + vars <- vectorOf n genVarName + body <- genSimpleExpr + -- Generate \x -> \y -> body + let mkLambda v rest = "\\" <> v <> " -> " <> rest + return $ foldr mkLambda body vars + +-- | Generate lambda expression with lambda calculus notation (using dots) +-- Examples: \f. \x. f x +genLambdaDot :: Gen Text +genLambdaDot = do + n <- choose (2, 3) + vars <- vectorOf n genVarName + body <- genSimpleExpr + -- Generate \f. \x. body + let parts = map (\v -> "\\" <> v <> ".") vars ++ [body] + return $ T.intercalate " " parts + +-- | Generate any lambda expression +genLambda :: Gen Text +genLambda = frequency + [ (5, genLambdaSimple) + , (3, genLambdaMulti) + , (2, genLambdaNested) + , (2, genLambdaDot) + ] + +-- | Generate set difference expressions +-- Examples: xs \\ ys, [1,2] \\ [2] +genSetDiff :: Gen Text +genSetDiff = do + a <- genVarName + b <- genVarName + return $ a <> " \\\\ " <> b + +-- * Unit Tests for fixBackslashOperators preprocessing + +mkLoc :: MyLoc +mkLoc = MyLoc 1 1 True + +-- Helper to compare tokens ignoring location details +tokensEq :: [(MyTok, MyLoc, Text)] -> [(MyTok, MyLoc, Text)] -> Bool +tokensEq xs ys = map (\(t,_,txt) -> (t,txt)) xs == map (\(t,_,txt) -> (t,txt)) ys + +-- | Property: Standalone \ followed by + should merge to \+ +prop_fix_backslash_plus :: Property +prop_fix_backslash_plus = + let input = [(TOther, mkLoc, "\\"), (TOperator, mkLoc, "+")] + expected = [(TOperator, mkLoc, "\\+")] + in Sky.fixBackslashOperators input `tokensEq` expected === True + +-- | Property: Standalone \ followed by > should merge to \> +prop_fix_backslash_gt :: Property +prop_fix_backslash_gt = + let input = [(TOther, mkLoc, "\\"), (TOperator, mkLoc, ">")] + expected = [(TOperator, mkLoc, "\\>")] + in Sky.fixBackslashOperators input `tokensEq` expected === True + +-- | Property: Two backslashes should merge to \\ +prop_fix_double_backslash :: Property +prop_fix_double_backslash = + let input = [(TOther, mkLoc, "\\"), (TOther, mkLoc, "\\")] + expected = [(TOperator, mkLoc, "\\\\")] + in Sky.fixBackslashOperators input `tokensEq` expected === True + +-- | Property: Backslash followed by variable should stay separate (lambda) +prop_fix_lambda_variable :: Property +prop_fix_lambda_variable = + let input = [(TOther, mkLoc, "\\"), (TVar, mkLoc, "x")] + expected = [(TOther, mkLoc, "\\"), (TVar, mkLoc, "x")] + in Sky.fixBackslashOperators input `tokensEq` expected === True + +-- | Property: Token ending with \ followed by / should merge to \/ +prop_fix_backslash_slash :: Property +prop_fix_backslash_slash = + let input = [(TOther, mkLoc, "a \\"), (TOperator, mkLoc, "/")] + expected = [(TOperator, mkLoc, "a \\/")] + in Sky.fixBackslashOperators input `tokensEq` expected === True + +-- | Property: Backslash in token should stay (lambda) +prop_fix_lambda_in_token :: Property +prop_fix_lambda_in_token = + let input = [(TOther, mkLoc, "\\x")] + expected = [(TOther, mkLoc, "\\x")] + in Sky.fixBackslashOperators input `tokensEq` expected === True + +-- * Unit Tests + +-- | Test cases for backslash operators and lambda expressions. +-- Tests tokenizer consistency on constructs that Skylighting often mishandles. +backslashOperatorTestCases :: [(String, Text, String)] +backslashOperatorTestCases = + [ ("simple lambda", "\\x -> x", "lambda") + , ("lambda at start", "\\x -> case x of", "lambda") + , ("multiple lambdas", "\\f. \\x. f (f x)", "lambda") + , ("curried lambda", "\\x -> \\y -> x + y", "lambda") + , ("lambda in expression", "map (\\x -> x + 1) xs", "lambda") + , ("unicode lambda", "λx -> x", "lambda") + , ("set difference", "a \\\\ b", "setminus") + , ("set diff in list", "xs \\\\ ys", "setminus") + , ("or operator", "a \\/ b", "land") -- \/ is logical and + , ("and operator", "a /\\ b", "lor") -- /\ is logical or + , ("backslash operator +", "\\+ 1", "backslash") + , ("backslash operator >", "\\> x", "backslash") + ] + +-- | Helper function to test both tokenizers with a predicate +-- Reduces duplication in property tests +testBothTokenizers :: Text -> String -> (Text -> Property) -> Property +testBothTokenizers code descr check = + let toLatex tokens = latexInline $ map (\(t,_,txt) -> (t, txt)) tokens + haskellResult = fmap toLatex (Haskell.tokenizer code) + skyResult = Sky.lookupTokenizer ["haskell"] >>= \syntax -> + fmap toLatex (Sky.tokenizer syntax code) + in counterexample descr $ + case (haskellResult, skyResult) of + (Just h, Just s) -> + counterexample ("Haskell output: " ++ T.unpack h) $ + counterexample ("Skylighting output: " ++ T.unpack s) $ + label "both tokenized" $ + check h .&&. check s + (Just h, Nothing) -> + counterexample ("Haskell output: " ++ T.unpack h) $ + label "only haskell" $ + check h + (Nothing, Just s) -> + counterexample ("Skylighting output: " ++ T.unpack s) $ + label "only skylighting" $ + check s + _ -> label "neither tokenized" $ property True + +-- | Property: Test that both tokenizers produce expected LaTeX commands +prop_backslash_operators :: Property +prop_backslash_operators = + forAll (elements backslashOperatorTestCases) $ \(name, code, expected) -> + let expectedCmd = "\\" <> T.pack expected + hasExpected txt = property $ expectedCmd `T.isInfixOf` txt + descr = "Test: " ++ name ++ "\nCode: " ++ T.unpack code ++ "\nExpected: " ++ expected + in testBothTokenizers code descr hasExpected + +-- * Property Tests + +-- | Property: Single backslash in lambda should always render as \lambda +prop_lambda_renders_correctly :: Property +prop_lambda_renders_correctly = + forAll genLambda $ \lambdaExpr -> + let hasLambda output = property $ "\\lambda" `T.isInfixOf` output + descr = "Lambda expression: " ++ T.unpack lambdaExpr + in testBothTokenizers lambdaExpr descr hasLambda + +-- | Property: Both tokenizers should produce the same number of \lambda commands +prop_tokenizers_agree_on_lambda_count :: Property +prop_tokenizers_agree_on_lambda_count = + forAll genLambda $ \lambdaExpr -> + let haskellResult = Haskell.tokenizer lambdaExpr >>= \tokens -> + Just $ latexInline $ map (\(t,_,txt) -> (t, txt)) tokens + + skyResult = Sky.lookupTokenizer ["haskell"] >>= \syntax -> + Sky.tokenizer syntax lambdaExpr >>= \tokens -> + Just $ latexInline $ map (\(t,_,txt) -> (t, txt)) tokens + + countLambdas txt = T.count "\\lambda" txt + + in counterexample ("Lambda expression: " ++ T.unpack lambdaExpr) $ + case (haskellResult, skyResult) of + (Just h, Just s) -> + let hCount = countLambdas h + sCount = countLambdas s + in counterexample ("Haskell output: " ++ T.unpack h) $ + counterexample ("Skylighting output: " ++ T.unpack s) $ + counterexample ("Haskell lambdas: " ++ show hCount) $ + counterexample ("Skylighting lambdas: " ++ show sCount) $ + label ("lambdas: " ++ show hCount) $ + hCount === sCount + _ -> label "tokenization failed" $ property True + +-- | Property: No output should contain \textbackslash or similar problematic commands +prop_no_textbackslash :: Property +prop_no_textbackslash = + forAll genLambda $ \lambdaExpr -> + let hasBadBackslash txt = "\\textbackslash" `T.isInfixOf` txt || + "\\backslashx" `T.isInfixOf` txt || + "\\backslash{}" `T.isInfixOf` txt + noBadBackslash txt = property $ not (hasBadBackslash txt) + descr = "Lambda expression: " ++ T.unpack lambdaExpr + in testBothTokenizers lambdaExpr descr noBadBackslash + +-- | Property: Double backslash should render as \setminus, not lambda +prop_setdiff_not_lambda :: Property +prop_setdiff_not_lambda = + forAll genSetDiff $ \expr -> + let hasSetminus txt = "\\setminus" `T.isInfixOf` txt + hasLambda txt = "\\lambda" `T.isInfixOf` txt + checkSetdiff txt = property $ hasSetminus txt && not (hasLambda txt) + descr = "Set difference expression: " ++ T.unpack expr + in testBothTokenizers expr descr checkSetdiff + +-- | Property: Tokenizers should handle the same code similarly +prop_tokenizers_compatible :: Property +prop_tokenizers_compatible = + forAll genLambda $ \code -> + let haskellToks = Haskell.tokenizer code + skyToks = Sky.lookupTokenizer ["haskell"] >>= \syntax -> + Sky.tokenizer syntax code + + -- Extract just the token types and texts (ignoring locations) + simplify = map (\(t,_,txt) -> (t, txt)) + + in counterexample ("Code: " ++ T.unpack code) $ + case (haskellToks, skyToks) of + (Just h, Just s) -> + let hSimple = simplify h + sSimple = simplify s + hOutput = latexInline hSimple + sOutput = latexInline sSimple + in counterexample ("Haskell tokens: " ++ show (length hSimple)) $ + counterexample ("Skylighting tokens: " ++ show (length sSimple)) $ + counterexample ("Haskell output: " ++ T.unpack hOutput) $ + counterexample ("Skylighting output: " ++ T.unpack sOutput) $ + label "both tokenized" $ + property True -- Just ensure both can tokenize + _ -> label "tokenization failed" $ property True + +-- | Main test runner +runAllTests :: IO () +runAllTests = do + putStrLn "\n=== fixBackslashOperators Unit Tests ===" + quickCheck prop_fix_backslash_plus + quickCheck prop_fix_backslash_gt + quickCheck prop_fix_double_backslash + quickCheck prop_fix_lambda_variable + quickCheck prop_fix_backslash_slash + quickCheck prop_fix_lambda_in_token + + putStrLn "\n=== Backslash Operator Tests ===" + quickCheckWith stdArgs { maxSuccess = length backslashOperatorTestCases } prop_backslash_operators + + putStrLn "\n=== Lambda Property Tests ===" + quickCheckWith stdArgs { maxSuccess = 100 } prop_lambda_renders_correctly + quickCheckWith stdArgs { maxSuccess = 100 } prop_tokenizers_agree_on_lambda_count + quickCheckWith stdArgs { maxSuccess = 100 } prop_no_textbackslash + quickCheckWith stdArgs { maxSuccess = 50 } prop_setdiff_not_lambda + quickCheckWith stdArgs { maxSuccess = 100 } prop_tokenizers_compatible diff --git a/test/Spec.hs b/test/Spec.hs --- a/test/Spec.hs +++ b/test/Spec.hs @@ -1,4 +1,5 @@ {-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE FlexibleContexts #-} -- | Test suite module Main where @@ -14,20 +15,68 @@ ,MyLoc(..) ,Tokenized) import Token.Haskell(tokenizer) -import Render.Latex(subscripts) +import Render.Latex(subAndSuperscripts) import qualified Render.Debug import FindColumns import Render.ColSpan import Alignment(Processed) import GHC.Stack +import qualified LambdaSpec +import qualified UnicodeOperatorSpec + +-- * Property tests for subAndSuperscripts + +-- | Property: Single underscore should produce \textsubscript +prop_single_underscore_is_subscript :: Property +prop_single_underscore_is_subscript = + forAll (listOf1 $ elements ['a'..'z']) $ \prefix -> + forAll (listOf1 $ elements ['a'..'z']) $ \suffix -> + let input = T.pack prefix <> "_" <> T.pack suffix + output = subAndSuperscripts input + in counterexample ("Input: " ++ T.unpack input) $ + counterexample ("Output: " ++ T.unpack output) $ + "\\textsubscript{" `T.isInfixOf` output + +-- | Property: Double underscore should produce \textsuperscript +prop_double_underscore_is_superscript :: Property +prop_double_underscore_is_superscript = + forAll (listOf1 $ elements ['a'..'z']) $ \prefix -> + forAll (listOf1 $ elements ['a'..'z']) $ \suffix -> + let input = T.pack prefix <> "__" <> T.pack suffix + output = subAndSuperscripts input + in counterexample ("Input: " ++ T.unpack input) $ + counterexample ("Output: " ++ T.unpack output) $ + "\\textsuperscript{" `T.isInfixOf` output + +-- | Property: Output should not contain raw underscores (except escaped \_) +prop_no_raw_underscores :: Property +prop_no_raw_underscores = + forAll (resize 20 $ listOf (elements (['a'..'z'] ++ ['_']))) $ \str -> + not (null str) && str /= "_" ==> + let input = T.pack str + output = subAndSuperscripts input + -- Count underscores that aren't part of \_ escape + hasRawUnderscore = T.any (== '_') $ T.replace "\\_" "" output + in counterexample ("Input: " ++ T.unpack input) $ + counterexample ("Output: " ++ T.unpack output) $ + label "processed" $ + not hasRawUnderscore + +-- | Property: Empty input or single underscore should be handled correctly +prop_edge_cases :: Property +prop_edge_cases = conjoin + [ subAndSuperscripts "" === " " + , subAndSuperscripts "_" === "\\_" + ] + prop_tokenizer str = case tokenizer str of - Nothing -> label "cannot lex" $ True + Nothing -> label "cannot lex" True Just t -> label "lexed" $ length t <= T.length str prop_debug_renderer_text_length input = case debug of - Nothing -> label "cannot lex" $ True + Nothing -> label "cannot lex" True Just t -> label "lexed" $ all (uncurry (<=)) $ zip (extract input) @@ -40,7 +89,7 @@ prop_colspans input = case debug of - Nothing -> label "cannot lex" $ True + Nothing -> label "cannot lex" True Just t -> label "lexed" $ sameNumber t where @@ -53,14 +102,14 @@ prop_tableColumns input = T.any (not . Data.Char.isSpace) input ==> case tokenizer input of - Nothing -> label "cannot lex" $ True + Nothing -> label "cannot lex" True Just tokens | all ((TBlank==) . view _1) tokens - -> label "only blanks" $ True - Just tokens -> + -> label "only blanks" True + Just tokens -> case findColumns tokens of - [] -> label "empty" $ True + [] -> label "empty" True t -> case sumColSpans t of - [] -> label "no colspans" $ True + [] -> label "no colspans" True c:_ -> label "tableColumns" $ c == length (tableColumns t) where extract :: T.Text -> [Int] @@ -78,12 +127,35 @@ main :: IO () main = do - (subscripts "alpha_beta") `shouldBe` ("alpha\\textsubscript{beta}") - (subscripts "alpha__beta") `shouldBe` ("alpha\\textsuperscript{beta}") - (subscripts "alpha__gamma_beta") `shouldBe` ("alpha\\textsuperscript{gamma\\textsubscript{beta}}") + putStrLn "=== Subscript/Superscript Tests ===" + subAndSuperscripts "alpha_beta" `shouldBe` "alpha\\textsubscript{beta}" + subAndSuperscripts "alpha__beta" `shouldBe` "alpha\\textsuperscript{beta}" + subAndSuperscripts "alpha__gamma_beta" `shouldBe` "alpha\\textsuperscript{gamma\\textsubscript{beta}}" + + putStrLn "\n=== Example Problems ===" problem " a\na" problem "--" problem "a\n a" + problem "\n a" -- prop_tableColumns failure case 1 + problem " a\na" -- prop_tableColumns failure case 2 (duplicate but documented) + + putStrLn "\n=== Lambda Expression Tests ===" + LambdaSpec.runAllTests + + putStrLn "\n=== GHC UnicodeSyntax Operator Tests ===" + UnicodeOperatorSpec.runAllTests + + putStrLn "\n=== SubAndSuperscripts Property Tests ===" + putStrLn " Testing single underscore → subscript" + quickCheckWith stdArgs { maxSuccess = 100 } prop_single_underscore_is_subscript + putStrLn " Testing double underscore → superscript" + quickCheckWith stdArgs { maxSuccess = 100 } prop_double_underscore_is_superscript + putStrLn " Testing no raw underscores in output" + quickCheckWith stdArgs { maxSuccess = 100 } prop_no_raw_underscores + putStrLn " Testing edge cases" + quickCheck prop_edge_cases + + putStrLn "\n=== General Property Tests ===" quickCheck prop_tokenizer quickCheck prop_debug_renderer_text_length quickCheck $ withMaxSuccess 10000 prop_colspans -- is it sufficient? diff --git a/test/UnicodeOperatorSpec.hs b/test/UnicodeOperatorSpec.hs new file mode 100644 --- /dev/null +++ b/test/UnicodeOperatorSpec.hs @@ -0,0 +1,88 @@ +{-# LANGUAGE OverloadedStrings #-} +-- | Unit tests for GHC UnicodeSyntax extension operators +-- | See: https://downloads.haskell.org/ghc/latest/docs/users_guide/exts/unicode_syntax.html +module UnicodeOperatorSpec (runAllTests) where + +import qualified Data.Text as T +import Data.Text (Text) +import Data.Maybe (fromMaybe) + +import qualified Token.Haskell as Haskell +import qualified Token.Skylighting as Sky +import Render.Latex (latexInline) + +-- | Test that ASCII and Unicode versions produce identical LaTeX output +testOperatorPair :: (String, Text, Text, Text) -> IO () +testOperatorPair (desc, asciiOp, unicodeOp, expectedLatex) = do + putStrLn $ " Testing: " ++ desc + + let haskellAscii = renderHaskell asciiOp + haskellUnicode = renderHaskell unicodeOp + skyAscii = renderSkylighting asciiOp + skyUnicode = renderSkylighting unicodeOp + + -- Check ASCII versions produce expected LaTeX + checkResult "Haskell (ASCII)" haskellAscii expectedLatex + checkResult "Skylighting (ASCII)" skyAscii expectedLatex + + -- Check Unicode versions produce the same LaTeX as ASCII + checkResult "Haskell (Unicode)" haskellUnicode expectedLatex + checkResult "Skylighting (Unicode)" skyUnicode expectedLatex + + -- Verify ASCII and Unicode produce identical output + if haskellAscii == haskellUnicode + then putStrLn " ✓ Haskell: ASCII and Unicode match" + else error $ " ✗ Haskell: ASCII (" ++ T.unpack haskellAscii ++ + ") != Unicode (" ++ T.unpack haskellUnicode ++ ")" + + if skyAscii == skyUnicode + then putStrLn " ✓ Skylighting: ASCII and Unicode match" + else error $ " ✗ Skylighting: ASCII (" ++ T.unpack skyAscii ++ + ") != Unicode (" ++ T.unpack skyUnicode ++ ")" + + putStrLn "" + +checkResult :: String -> Text -> Text -> IO () +checkResult label actual expected + | expected `T.isInfixOf` actual = + putStrLn $ " ✓ " ++ label ++ ": contains " ++ T.unpack expected + | otherwise = + error $ " ✗ " ++ label ++ ": expected " ++ T.unpack expected ++ + " but got " ++ T.unpack actual + +renderHaskell :: Text -> Text +renderHaskell code = fromMaybe "" $ do + tokens <- Haskell.tokenizer code + return $ latexInline $ map (\(t,_,txt) -> (t, txt)) tokens + +renderSkylighting :: Text -> Text +renderSkylighting code = fromMaybe "" $ do + syntax <- Sky.lookupTokenizer ["haskell"] + tokens <- Sky.tokenizer syntax code + return $ latexInline $ map (\(t,_,txt) -> (t, txt)) tokens + +-- | Test cases: (description, ASCII operator, Unicode operator, expected LaTeX command) +unicodeOperatorTests :: [(String, Text, Text, Text)] +unicodeOperatorTests = + [ ("forall", "forall", "∀", "\\forall") + , ("type annotation", "::", "∷", "\\:") + , ("class constraint", "=>", "⇒", "\\Rightarrow") + , ("function type", "->", "→", "\\to") + , ("bind/generator", "<-", "←", "\\gets") + , ("right arrow-tail", ">-", "⤚", "\\succ") + , ("left arrow-tail", "-<", "⤙", "\\prec") + , ("right double arrow-tail", ">>-", "⤜", "\\rr") + , ("left double arrow-tail", "-<<", "⤛", "\\ll") + , ("unicode kind star", "★", "★", "\\star") -- Unicode only (ASCII * is multiplication) + , ("linear arrow", "⊸", "⊸", "\\multimap") -- Unicode only + , ("parallel array open", "(|", "⦇", "\\llparenthesis") + , ("parallel array close", "|)", "⦈", "\\rrparenthesis") + , ("quasiquote open", "[|", "⟦", "\\llbracket") + , ("quasiquote close", "|]", "⟧", "\\rrbracket") + ] + +runAllTests :: IO () +runAllTests = do + putStrLn "\n=== GHC UnicodeSyntax Operator Tests ===" + mapM_ testOperatorPair unicodeOperatorTests + putStrLn "All Unicode operator tests passed! ✓"