pandoc-filter-indent 0.1.0.0 → 0.2.0.0
raw patch · 12 files changed
+179/−181 lines, 12 files
Files
- ChangeLog.md +4/−0
- README.md +82/−73
- app/Main.hs +3/−15
- pandoc-filter-indent.cabal +2/−2
- src/Alignment.hs +2/−8
- src/Render/ColSpan.hs +9/−23
- src/Render/Debug.hs +7/−13
- src/Render/HTML.hs +30/−27
- src/Render/Latex.hs +1/−1
- src/Token.hs +23/−4
- src/Token/Haskell.hs +8/−2
- src/Util.hs +8/−13
ChangeLog.md view
@@ -3,3 +3,7 @@ ## Unreleased changes 0.0.1 Dec 1 2020 - Initial release supporting LaTeX and HTML output++0.0.2 Dec 2 2020+ - Preliminary support for Tikzmarks+ - Updated README
README.md view
@@ -1,7 +1,11 @@ --- title: "Code typesetting made simple"-subtitle: "Project description"-author: "Michał J. Gajda"+subtitle: "Pandoc filter"+description: |+ To build this document use:++ > pandoc README.md -o README.pdf --filter=pandoc-filter-indent+ date: | `\today`{=latex} abstract: |@@ -12,11 +16,12 @@ is also a Pandoc filter, and can thus be used to improve comprehensibility of the code. It is also simple enough to be provided as a literate program within this submission. While it is processing Haskell code, we show how it can be easily adapted to typeset Python, Java or C/C++.-author:- - name: Michał J. Gajda- email: mjgajda@migamake.com- orcid: "0000-0001-7820-3906"- affiliation: 1+author: Michał J. Gajda++# - name: Michał J. Gajda+# email: mjgajda@migamake.com+# orcid: "0000-0001-7820-3906"+# affiliation: 1 affiliation: institution: "Migamake Pte Ltd" email: mjgajda@migamake.com@@ -24,16 +29,16 @@ review: true numbersections: true header-includes:- - |- \usepackage{amssymb}- \usepackage{graphicx}- \DeclareUnicodeCharacter{03B1}{\ensuremath{\alpha{}}}- \newcommand{\longeq}{\scalebox{1.7}[1]{=}}-prologue: |- \usepackage{amssymb}- \usepackage{graphicx}- \DeclareUnicodeCharacter{03B1}{\ensuremath{\alpha{}}}- \newcommand{\longeq}{\scalebox{1.7}[1]{=}}+ - |-2+ \usepackage{graphicx}+ \DeclareUnicodeCharacter{03B1}{\ensuremath{\alpha{}}}++prologue:+ - |-2+ \usepackage{graphicx}+ \DeclareUnicodeCharacter{03B1}{\ensuremath{\alpha{}}}+ \renewcommand{\longeq}{\scalebox{1.7}[1]{=}}+ --- # Introduction@@ -51,19 +56,28 @@ We use `pandoc` to process only code fragments in otherwise unprocessed literate program or article with code excerpts. -We detect layout boundaries from tokens: (1) we note an column indent for each line[^This step is implemented using GHC API.],-then (2) mark a start of each operator (like `::`, `=`, or `>>=`) by column.+We detect layout boundaries from tokens:++(1) we note an column indent for each line^[This step is implemented using GHC API.],+then++(2) mark a start of each operator (like `::`, `=`, or `>>=`) by column. Our second step is that of transposing a list of `Line = [(Column, Token)]` into per-column list of indentations `[(Line, Token)]`.+ (3) If any token is present after more than a single space, we also mark its beginning as indentation boundary.+ (4) After sorting columns by line number, we mark these columns that have a consistent presence along consecutive lines as indentation anchors.+ (5) Additionally we mark the leftmost indent as a same indentation barrier as long as it follows the nesting order.+ (6) For postprocessing, we escape the text according to target text processing engine syntax, translate common operators to their ligatures (see appendix), and output the text with layout boundaries.+ We also plan to implement support for pointing to code fragments with TikZ target marks[@tikz] at this stage^[With syntax of `{->p1-}` standing for `\tikzmark{p1}`]. @@ -139,9 +153,9 @@ (\x |-> |x) ``` -## Generating \LaTeX or HTML output+## Generating \LaTeX{} or HTML output -When generating \LaTeX or HTML output, we simply+When generating \LaTeX{} or HTML output, we simply assign a list of columns to each span of code tex in line. This can be implemented using multicolumn marking in both output languages. Consider the above example with columns numbered at starting character:@@ -171,7 +185,7 @@ ``` General syntax of `\multicolumn`^[See [Overleaf tutorial](https://www.overleaf.com/learn/latex/tables#Combining_rows_and_columns)-if you do not know how \LaTeX tables work.] has three arguments, each enclosed with braces (`{}`):+if you do not know how \LaTeX{} tables work.] has three arguments, each enclosed with braces (`{}`): 1. Number of columns in the cell, for example `{2}` or `{6}`. 2. Alignment of text in the cell: - `{l}` for left,@@ -211,7 +225,7 @@ Main executable is a `pandoc` filter. You get `pandoc` input stream, and replace [`CodeBlock` blocks](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Block)-there with `Raw "latex"` LaTeX blocks. It is these block elements of ADT that should contain the \LaTeX code+there with `Raw "latex"` \LaTeX{} blocks. It is these block elements of ADT that should contain the \LaTeX{} code Pandoc will build the document for you, and do it better than you would. Below is a modified [example from `pandoc` documentation](https://texblog.org/2012/12/21/multi-column-and-multi-row-cells-in-latex-tables/) for making a `pandoc` filter executable:@@ -232,7 +246,7 @@ ``` In the executable above, you write `haskellCodeFormatter` function-that takes Haskell code, and returns \LaTeX code _fragment_ like this:+that takes Haskell code, and returns \LaTeX{} code _fragment_ like this: ```{.latex} \begin{array}@@ -244,7 +258,7 @@ We need to check Pandoc meta before issuing `walk` in order to check what is the output format.-In case it is \LaTeX or PDF, then we produce \LaTeX raw code fragments.+In case it is \LaTeX{} or PDF, then we produce \LaTeX{} raw code fragments. In case output format is any other, then we produce HTML table. ## Passing options to the filter@@ -272,9 +286,9 @@ - `spaces` -- space-only lexer: table column starts at the end of block of two or more whitespace characters - `python3` -- Python lexer (see below in @sec:python-lexer) * `debug` -- add option that shows table columns (for debugging layout)-* `underbar` -- do not escape `_`, but instead use \LaTeX/HTML subscript til the end of the token:- `method_agile` becomes `method_{agile}` in \LaTeX, `x_i_j` becomes `x_{i,j}` in \LaTeX-* table alignment (\LaTeX output only):+* `underbar` -- do not escape `_`, but instead use \LaTeX{}/HTML subscript til the end of the token:+ `method_agile` becomes `method_{agile}` in \LaTeX{}, `x_i_j` becomes `x_{i,j}` in \LaTeX{}+* table alignment (\LaTeX{} output only): - `array` -- use `array` environment, the default - `polytable` -- use `polytable` environment * code alignment:@@ -296,11 +310,11 @@ ``` -2. We also replace the common operators with \LaTeX symbols commonly used for this purpose:+2. We also replace the common operators with \LaTeX{} symbols commonly used for this purpose: -| Input token | \LaTeX code | Rendering |+| Input token | \LaTeX{} code | Rendering | |:-----------------|:--------------|:---------------:|-| `Operator "="` | `\longeq` | $\longeq{}$ |+| `Operator "="` | `\scalebox{1.7}[1]{=}` | $\scalebox{1.7}[1]{=}$ | | `Operator "<>"` | `\diamond` | $\diamond{}$ | | `Operator ">="` | `\geq` | $\geq{}$ | | `Operator "<="` | `\geq` | $\leq{}$ |@@ -372,13 +386,13 @@ * `b` to `\beta` shown as $\beta$ * etc. -2. [The Comprehensive LATEX Symbol List](http://tug.ctan.org/info/symbols/comprehensive/symbols-a4.pdf) is a good reference of LaTeX symbol names.+2. [The Comprehensive LATEX Symbol List](http://tug.ctan.org/info/symbols/comprehensive/symbols-a4.pdf) is a good reference of \LaTeX{} symbol names. See section _3. Mathematical symbols_. ## Formatting different token types -Since we want to use `\begin{array}` environment in LaTeX,-we should encompass the token characters with different LaTeX operators.+Since we want to use `\begin{array}` environment in \LaTeX{},+we should encompass the token characters with different \LaTeX{} operators. * variables with `\textrm{var}` * type variables with `\mathit{tyvar}`@@ -390,24 +404,11 @@ ## Safe escaping -In order to safely escape strings, it would be best to-make sure that conversion is done only on parts that are not escaped yet:-```{.haskell}-data Escapable x =- Literal x- | Escaped x--conversion :: Escapable Text -> Escapable LaTeX-```--Alternative is to perform the conversion as multistep process, with list of `Either` values:+In order to safely escape strings, we keep token type+next to original token text over entire pipeline+until rendering of code fragment as raw \LaTeX{} or HTML: ```{.haskell}-type Input = Either HaskellToken RawTeX-type RawTeX = String--type Intermediate = Either UnescapedString RawTeX--type Output = Either () RawTeX+type Token... = (MyTok, Text, ...) ``` ### Pandoc filter connection@@ -427,10 +428,10 @@ Then we make `ourPandocWalk` to be a function that: 1. Matches [`Meta`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Meta)-to check if we are targetting LaTeX or HTML.+to check if we are targetting \LaTeX{} or HTML. 2. Finds [`CodeBlock`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Block) and leaves everything else as-is. 3. Generates `\table` as [`RawBlock`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Block)- of LaTeX output.+ of \LaTeX{} output. Options **shall** be parsed **per `CodeBlock`**. First parameter to each `CodeBlock` are attributes ([`Attr`](https://hackage.haskell.org/package/pandoc-types-1.20/docs/Text-Pandoc-Definition.html#t:Attr)).@@ -446,24 +447,27 @@ Global `Meta` attributes to be handled: * output format:- - LaTeX or PDF -- produce LaTeX `RawBlock`+ - \LaTeX{} or PDF -- produce \LaTeX{} `RawBlock` - HTML -- produce HTML `RawBlock` - or all others -- produce `Table` ### TikZ marks -TikZ marks are useful for marking up things on generated code.+TikZ marks are useful for pointing to fragments of the generated code. You should just look for comments with syntax: `{->markName-}` and convert them to a raw-LaTeX string `\tikzmark{markName}`.+\LaTeX{} string `\tikzmark{markName}`. +For HTML, it generates `<span id="markName" />` which you+can then draw to with [a convenient JavaScript](https://stackoverflow.com/questions/554167/drawing-arrows-on-an-html-page-to-visualize-semantic-links-between-textual-spans#623770).+ ### Alternate lexers Please note that using alternate lexers disables token replacement! This is important, since the token replacement-to LaTeX special symbols is language-specific.+to \LaTeX{} special symbols is language-specific. #### Indent only @@ -509,33 +513,38 @@ ## Finding Haskell tokens -https://hackage.haskell.org/package/ghc-syntax-highlighter-0.0.6.0/docs/GHC-SyntaxHighlighter.html+We find Haskell tokens with GHC-lib that uses GHC parser itself.+For simplicity we use [`ghc-syntax-highlighter`](https://hackage.haskell.org/package/ghc-syntax-highlighter-0.0.6.0/docs/GHC-SyntaxHighlighter.html)+We readd locations after the fact, since it gives you pure tokens with text and spaces,+or pure locations (without text).+ ```-tokenizeHaskellLoc :: Text -> Maybe [(Token, Loc)]+tokenizeHaskell :: Text -> Maybe [(Token, Text)] ``` -## LaTeX output-1. For escaping text in TeX:-http://hackage.haskell.org/package/HaTeX-3.5/docs/Text-LaTeX-Base-Render.html#t:Render+## \LaTeX{} output -`Render Text` in particular+For escaping text in TeX we use [HaTeX](http://hackage.haskell.org/package/HaTeX-3.5/docs/Text-LaTeX-Base-Render.html#t:Render)[@hatex] -2. For tables:-* https://hackage.haskell.org/package/HaTeX-3.22.2.0/docs/Text-LaTeX-Packages-Multirow.html-* if not:- - add one to LaTeX:- * `multicol`- * `polytable`+We render tables inline tables with `multicolumn` (not `polytable` like lhs2TeX[@lhs2tex] does.) +We do not use [HaTeX table support](https://hackage.haskell.org/package/HaTeX-3.22.2.0/docs/Text-LaTeX-Packages-Multirow.html) yet.+ ## Debugging -The best debugging would be using the filter in different modes.-Pandoc can automatically detect output format:+Pandoc can automatically detect output format.+In order to get \LaTeX{} and HTML output just run:+ ```sh-pandoc input.md --filter=pandoc-filter-indent -o output.html pandoc input.md --filter=pandoc-filter-indent -o output.tex pandoc input.md --filter=pandoc-filter-indent -o output.pdf ``` +For debugging indentation use `text` output format:+```+pandoc input.md --filter=pandoc-filter-indent -o output.txt+```+In text mode, it renders indent boundaries as `|` and `^`.+ Note that the filter *does not* touch the text outside code blocks.-It can however add necessary LaTeX headers or HTML styles in meta `headers-include`.+It can add necessary \LaTeX{} headers or HTML styles in meta `headers-include`.
app/Main.hs view
@@ -7,7 +7,7 @@ import Data.String (fromString, IsString) import Data.Text (Text) import qualified Data.Text as T-import Debug.Trace(trace)+--import Debug.Trace(trace) import Token.Haskell import GHC.SyntaxHighlighter@@ -20,28 +20,16 @@ -- | Select the desired format output then process it. blockFormatter :: Maybe Format -> Block -> Block blockFormatter Nothing (CodeBlock attrs content) = -- debugging mode- trace ("No format given") $ codeFormatter "text" attrs content blockFormatter (Just (Format format)) (CodeBlock attrs content) | isHaskell attrs = codeFormatter format attrs content- | otherwise = CodeBlock attrs content -- Do not touch other blocks than 'CodeBlock' blockFormatter _ x = x -- | Select formatter codeFormatter :: Text -> Attr -> Text -> Block-codeFormatter format attrs content | True =- trace ("Output format: " <> show format) $- (render format attrs $ filterCodeBlock content)- --CodeBlock attrs $ T.pack $ show $ fmap findColumns $ tokenizer content- --CodeBlock attrs $ T.pack $ show $ tokenizer content- {-- | format == (fromString "latex") =- RawBlock (Format "latex") (renderLatex content)- | format == (fromString "html") =- RawBlock (Format "html") (renderHtmlO content)-}- -- Unknown formats gives the original elem.- | otherwise = CodeBlock attrs content+codeFormatter format attrs content =+ render format attrs (filterCodeBlock content) -- (Text, [Text], [(Text, Text)]) isHaskell :: (Foldable t, Eq a1, IsString a1) =>
pandoc-filter-indent.cabal view
@@ -4,10 +4,10 @@ -- -- see: https://github.com/sol/hpack ----- hash: 8fbe8b1245e7e2790a4df85621f625b1ec650eb3b6ade7b2c79f715880e4d926+-- hash: 3a5612ddfa28673f5b83117e2fde59d631d99c9de58e1b04aaddaf53c7559d58 name: pandoc-filter-indent-version: 0.1.0.0+version: 0.2.0.0 synopsis: Pandoc filter formatting Haskell code fragments using GHC lexer. description: Formats marked code fragments, and allows `pandoc` to safely process rest of your literate program: .
src/Alignment.hs view
@@ -1,17 +1,11 @@-{-# LANGUAGE OverloadedStrings #-}-{-# LANGUAGE PartialTypeSignatures #-}-{-# LANGUAGE ViewPatterns #-}-{-# LANGUAGE TypeSynonymInstances #-}-{-# LANGUAGE FlexibleInstances #-}-{-# LANGUAGE FlexibleContexts #-}+{-# LANGUAGE FlexibleContexts #-} module Alignment where import Data.Text (Text) import Data.Tuple.Optics import Optics.Lens -import Token-import Tuples+import Token ( MyLoc, MyTok ) -- | Datatype to present columns with alignment requirements. data Align =
src/Render/ColSpan.hs view
@@ -5,40 +5,26 @@ module Render.ColSpan where import Data.Function(on)-import Data.String (fromString, IsString) import Data.Text (Text)-import qualified Data.Text as T import Data.List(groupBy, sortBy)-import Data.Maybe(fromMaybe) import Prelude hiding(getLine)-import Optics.Core ( Field1(_1), Field2(_2), view, (%), lens )-import Data.Tuple.Optics ( Field1(_1), Field2(_2) )+import Optics.Core ( Field1(_1), Field2(_2), view, (%)) import Alignment-import FindColumns+ ( textContent, tokenType, Align(ALeft), Processed )+import FindColumns ( alignPos, getLine, getLineCol ) import Token(MyTok)-import Util+import Util ( maybeLens ) type TokensWithColSpan = ([(MyTok, Text)], Int, Align)---type TextWithColSpan = (Text, Int, [Processed]) ---extractTexts :: [[TextWithColSpan]] -> [[_]]-{-extractTexts ps = fmap extractText <$> ps- where- maxCol :: Int- maxCol = maximum $ map getAlignCol $ ps- extractText (tok:toks, colspan) = (T.concat (view textContent <$> (tok:toks))- ,colspan- ,view (alignPos % maybeLens (ALeft,-1) % _1) tok)- -}- -- | Find colspan parameters colspans :: [Processed] -> [[TokensWithColSpan]]-colspans ps = fmap ( fmap extractTokens- . addColSpans- . groupBy sameColSpan )- $ groupBy ((==) `on` getLine)- $ sortBy (compare `on` getLineCol) ps+colspans ps = fmap ( fmap extractTokens -- extract token and text content from each record+ . addColSpans -- add colspan length in alignment columns+ . groupBy sameColSpan ) -- group colspans+ $ groupBy ((==) `on` getLine) -- group lines+ $ sortBy (compare `on` getLineCol) ps where maxCol :: Int maxCol = maximum $ map getAlignCol $ ps
src/Render/Debug.hs view
@@ -1,23 +1,20 @@ {-# LANGUAGE AllowAmbiguousTypes #-} {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE PartialTypeSignatures #-}-{-# LANGUAGE TupleSections #-} {-# LANGUAGE FlexibleInstances #-} {-# LANGUAGE FlexibleContexts #-}+-- | Renders alignment to text with alignment markers,+-- for debugging purposes. module Render.Debug(render) where -import Data.Function(on)-import Data.String (fromString, IsString) import Data.Text (Text) import qualified Data.Text as T-import Data.List(groupBy, sortBy, sort, group)-import Data.Maybe(fromMaybe) import Prelude hiding(getLine)-import Optics.Core+import Optics.Core ( view ) -import FindColumns-import Alignment-import Util+import FindColumns ( alignPos, getCol, tableColumns )+import Alignment ( textContent, Align(..), Processed )+import Util ( safeTail ) insertAt :: Show a => Int -> a -> [a] -> [a] insertAt i e ls = case maybeInsertAt i e ls of@@ -55,10 +52,7 @@ T.pack $ insertMarkers unaccountedMarkers $ T.unpack $ view textContent tok where insertMarkers [] txt = txt- insertMarkers mrks txt = --(\result -> trace ("insertMarkers " <> show mrks <> " " <> show txt <> " => " <> result) result)- foldr insertMarker txt- $ -- trace (show unaccountedMarkers)- mrks+ insertMarkers mrks txt = foldr insertMarker txt mrks insertMarker index = insertAt index '.' unaccountedMarkers = fmap (-getCol tok+)
src/Render/HTML.hs view
@@ -2,22 +2,22 @@ {-# LANGUAGE ViewPatterns #-} module Render.HTML(htmlFromColSpans) where -import Prelude hiding(span)+import Prelude hiding(span, id) import Data.Text(Text)-import Data.Char(isSpace) import qualified Data.Text as T import qualified Data.Text.Lazy as LT import Text.Blaze.Html5 hiding(style)-import Text.Blaze.Html5.Attributes(colspan, style)+import Text.Blaze.Html5.Attributes(colspan, style, id) import Text.Blaze.Html.Renderer.Text(renderHtml) -import Alignment+import Alignment ( Align(..) ) import Render.Common(TokensWithColSpan)-import Token(MyTok(..))-import Debug.Trace(trace)-import Util+import Token (MyTok(..))+import Util ( preformatTokens, unbrace ) ---latexFromColSpans :: Int -> [[TokensWithColSpan]] -> Text+htmlFromColSpans :: p+ -> [[TokensWithColSpan]]+ -> Text htmlFromColSpans cols = LT.toStrict . renderHtml@@ -25,30 +25,31 @@ . tbody . mapM_ renderTr +renderTr :: [TokensWithColSpan] -> Html renderTr colspans = tr (mapM_ renderColSpan colspans) ---renderColSpan :: TokensWithColSpan -> Text+renderColSpan :: TokensWithColSpan -> Html renderColSpan ([(TBlank, txt)], colSpan, AIndent) = -- indentation- (td $ toHtml txt)- ! colspan (toValue colSpan)- ! style widthStyle+ td (toHtml txt)+ ! colspan (toValue colSpan)+ ! style widthStyle where widthStyle = toValue $ "min-width: " <> show (T.length txt) <> "ex" renderColSpan (toks, colSpan, alignment) =- (td $ formatTokens toks)- ! colspan (toValue colSpan)- ! style alignStyle+ td (formatTokens toks)+ ! colspan (toValue colSpan)+ ! style alignStyle where- alignStyle = "text-align: " <> alignMark alignment+ alignStyle = "text-align: " <> alignMark alignment alignMark ACenter = "center" alignMark ALeft = "left" --- Decrease column spacing: \\setlength{\\tabcolsep}{1ex} -- TODO: braced operators formatTokens :: [(MyTok, Text)] -> Html formatTokens = mapM_ formatToken . preformatTokens +-- | Format a single token as HTML fragment. formatToken :: (MyTok, Text) -> Html formatToken (TOperator,unbrace -> Just op) = do "(" formatToken (TOperator, op)@@ -57,7 +58,7 @@ formatToken (TOperator,"<>" ) = "⋄" formatToken (TOperator,"=>" ) = "⇒" formatToken (TOperator,"->" ) = "→"-formatToken (TOperator,"|->" ) = "↦"+formatToken (TOperator,"|->" ) = "↦" formatToken (TVar ,"undefined") = "⊥" formatToken (TVar ,"bot" ) = "⊥" formatToken (TVar ,"not" ) = "¬"@@ -77,18 +78,20 @@ formatToken (TOperator,"<<<" ) = "⋘" formatToken (TOperator,"||" ) = "∥" formatToken (TOperator,"<->" ) = "↔︎"-formatToken (TOperator,"<-" ) = "←"+formatToken (TOperator,"<-" ) = "←" formatToken (TOperator,"-<" ) = "≺" formatToken (TOperator,">-" ) = "≻" formatToken (TOperator,"!=" ) = "≠"-formatToken (TOperator,"\\/" ) = "⋁"-formatToken (TOperator,"/\\" ) = "⋀"-formatToken (TOperator,"~" ) = "∼"+formatToken (TOperator,"\\/" ) = "⋁"+formatToken (TOperator,"/\\" ) = "⋀"+formatToken (TOperator,"~" ) = "∼" formatToken (TOperator,"~=" ) = "≈"-formatToken (TVar,"top" ) = "⊤"-formatToken (TKeyword, kwd ) = b $ toHtml kwd-formatToken (TVar, v ) = i $ toHtml v-formatToken (TCons, v ) = span (toHtml v)+formatToken (TVar, "top" ) = "⊤"+formatToken (TKeyword, kwd ) = b $ toHtml kwd+formatToken (TVar, v ) = i $ toHtml v+formatToken (TCons, v ) = span (toHtml v) ! style ("font-variant: small-caps;")-formatToken (_, txt) = toHtml txt+formatToken (TTikz mark,_ ) = span ""+ ! id (toValue mark)+formatToken (_, txt ) = toHtml txt
src/Render/Latex.hs view
@@ -3,7 +3,6 @@ module Render.Latex(latexFromColSpans) where import Data.Text(Text)-import Data.Char(isSpace) import qualified Data.Text as T import Text.LaTeX.Base.Syntax(protectText) @@ -117,6 +116,7 @@ formatToken (TKeyword, kwd ) = "\\textbf{" <> protectText kwd <> "}" formatToken (TCons, cons ) = "\\textsc{" <> protectText cons <> "}" formatToken (TOperator,"\\" ) = mathop "lambda"+formatToken (TTikz mark,_ ) = mathop $ "tikzMark{" <> mark <> "}" formatToken (_, txt ) = "\\textit{" <> protectText txt <> "}" mathop code = "\\" <> code
src/Token.hs view
@@ -1,13 +1,16 @@+{-# LANGUAGE LambdaCase #-} {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE PartialTypeSignatures #-} {-# LANGUAGE TemplateHaskell #-} {-# LANGUAGE ViewPatterns #-} -- | Common token representation used.-module Token(MyTok(..), MyLoc(..), Tokenized, line, col) where+module Token(MyTok(..), MyLoc(..), Tokenized, line, col, unwrap, unTikzMark) where -import Data.Text(Text)-import Data.Tuple.Optics-import Optics.TH+import Data.Maybe(fromMaybe)+import Data.Text(Text)+import qualified Data.Text as T+import Data.Tuple.Optics+import Optics.TH -- * Common tokens and locations -- Location is just line and column@@ -25,8 +28,24 @@ | TVar | TNum | TOther+ | TTikz Text -- TikZmark in a comment deriving (Eq, Ord, Show) -- | Records tokenized and converted to common token format. type Tokenized = (MyTok, MyLoc, Text)++unTikzMark :: Text -> Maybe Text+unTikzMark txt =+ unwrap "{->" "-}" txt >>= \case+ "" -> Nothing+ mark -> Just mark++unwrap :: Text -> Text -> Text -> Maybe Text+unwrap starter trailer txt |+ starter `T.isPrefixOf` txt &&+ trailer `T.isPrefixOf` txt =+ Just+ $ T.dropEnd (T.length trailer)+ $ T.drop (T.length starter) txt+unwrap _ _ _ = Nothing
src/Token/Haskell.hs view
@@ -8,6 +8,7 @@ import Text.Pandoc.JSON import Text.Pandoc.Definition () import Data.Function(on)+import Data.Maybe(fromMaybe, isJust) import Data.String (fromString, IsString) import Data.Text (Text) import qualified Data.Text as T@@ -24,10 +25,15 @@ tokenizer = fmap ( joinEscapedOperators . splitTokens . restoreLocations- . fmap (first haskellTok) )+ . fmap recognizeToken) . tokenizeHaskell -haskellTok SymbolTok = TOperator++recognizeToken (CommentTok, tokText@(unTikzMark -> Just mark)) =+ (TTikz mark, tokText)+recognizeToken (tokType, tokText) =+ (haskellTok tokType, tokText)+ haskellTok SpaceTok = TBlank haskellTok CommentTok = TBlank haskellTok PragmaTok = TBlank
src/Util.hs view
@@ -8,21 +8,14 @@ {-# LANGUAGE FlexibleContexts #-} module Util where -import Text.Pandoc.JSON-import Text.Pandoc.Definition ()-import Data.Function(on)-import Data.String (fromString, IsString)-import Data.Text (Text)+import Data.Function(on)+import Data.List(groupBy, sortBy)+import Data.Maybe(fromMaybe) import qualified Data.Text as T-import Data.List(groupBy, sortBy, sort, group)-import Data.Maybe(fromMaybe)-import Prelude hiding(getLine)-import Optics.Core-import Data.Tuple.Optics-import Data.Tuple.Optics-import Data.Text.Lazy.Builder+import Optics.Core ( Lens, lens )+import Prelude hiding(getLine) -import Token(MyTok(..))+import Token(MyTok(..)) -- | Lens with default value. maybeLens :: a -> Lens (Maybe a) (Maybe a1) a a1@@ -36,6 +29,8 @@ grouping key = groupBy ((==) `on` key) . sortBy (compare `on` key) +-- | Sort and remove duplicates in a list.+-- Duplicates are detected as equals by default `Ord`ering. nubSorted :: Ord a => [a] -> [a] nubSorted = nubSortedBy compare