ghc-syntax-highlighter 0.0.1.0 → 0.0.2.0

raw patch · 4 files changed

+185/−29 lines, 4 filesPVP ok

version bump matches the API change (PVP)

API changes (from Hackage documentation)

Files

• CHANGELOG.md +5/−1
• GHC/SyntaxHighlighter.hs +41/−24
• README.md +136/−1
• ghc-syntax-highlighter.cabal +3/−3

@@ -1,3 +1,7 @@-## GHC syntax highligher 0.0.1.0+## GHC syntax highlighter 0.0.2.0++* Added `Loc` and `tokenizeHaskellLoc`.++## GHC syntax highlighter 0.0.1.0  * Initial release.

@@ -20,7 +20,9 @@  module GHC.SyntaxHighlighter   ( Token (..)-  , tokenizeHaskell )+  , Loc (..)+  , tokenizeHaskell+  , tokenizeHaskellLoc ) where  import Control.Monad@@ -57,11 +59,18 @@   | OtherTok           -- ^ Something else?   deriving (Eq, Ord, Enum, Bounded, Show) --- | Internal type containing line\/column combinations for start and end--- positions of a code span.+-- | Start and end positions of a span. The arguments of the data+-- constructor contain in order:+--+--     * Line number of start position of a span+--     * Column number of start position of a span+--     * Line number of end position of a span+--     * Column number of end position of a span+--+-- @since 0.0.2.0  data Loc = Loc !Int !Int !Int !Int-  deriving (Show)+  deriving (Eq, Ord, Show)  ---------------------------------------------------------------------------- -- High-level API@@ -74,10 +83,34 @@ -- time), it'll return something in 'Just'.  tokenizeHaskell :: Text -> Maybe [(Token, Text)]-tokenizeHaskell input =+tokenizeHaskell input = sliceInputStream input <$> tokenizeHaskellLoc input++-- | Replace 'Loc' locations with actual chunks of input 'Text'.++sliceInputStream :: Text -> [(Token, Loc)] -> [(Token, Text)]+sliceInputStream input toks = unfoldr sliceOnce (initText' input, toks)+  where+    sliceOnce (txt, []) = do+      (txt', chunk) <- tryFetchRest txt+      return ((SpaceTok, chunk), (txt', []))+    sliceOnce (txt, tss@((t, l):ts)) =+      case tryFetchSpace txt l of+        Nothing ->+          let (txt', chunk) = fetchSpan txt l+          in Just ((t, chunk), (txt', ts))+        Just (txt', chunk) ->+          Just ((SpaceTok, chunk), (txt', tss))++-- | Similar to 'tokenizeHaskell', but instead of 'Text' chunks provides+-- locations of corresponding spans in the given input stream.+--+-- @since 0.0.2.0++tokenizeHaskellLoc :: Text -> Maybe [(Token, Loc)]+tokenizeHaskellLoc input =   case L.unP pLexer parseState of     L.PFailed {} -> Nothing-    L.POk    _ x -> Just (sliceInputStream input x)+    L.POk    _ x -> Just x   where     location = mkRealSrcLoc (mkFastString "") 1 1     buffer = stringToStringBuffer (T.unpack input)@@ -103,22 +136,6 @@             Nothing -> go             Just  x -> (x:) <$> go --- | Replace 'Loc' locations with actual chunks of input 'Text'.--sliceInputStream :: Text -> [(Token, Loc)] -> [(Token, Text)]-sliceInputStream input toks = unfoldr sliceOnce (initText' input, toks)-  where-    sliceOnce (txt, []) = do-      (txt', chunk) <- tryFetchRest txt-      return ((SpaceTok, chunk), (txt', []))-    sliceOnce (txt, tss@((t, l):ts)) =-      case tryFetchSpace txt l of-        Nothing ->-          let (txt', chunk) = fetchSpan txt l-          in Just ((t, chunk), (txt', ts))-        Just (txt', chunk) ->-          Just ((SpaceTok, chunk), (txt', tss))- -- | Convert @'Located' 'L.Token'@ representation to a more convenient for -- us form. @@ -334,7 +351,7 @@   {-# UNPACK #-} !Text   deriving (Show) --- | Create 'Text' from 'Text''.+-- | Create 'Text'' from 'Text'.  initText' :: Text -> Text' initText' = Text' 1 1@@ -356,7 +373,7 @@     then Nothing     else Just (Text' l c "", txt) --- | Fetch span at 'Loc'.+-- | Fetch a span at 'Loc'.  fetchSpan :: Text' -> Loc -> (Text', Text) fetchSpan txt (Loc _ _ el ec) = reachLoc txt el ec

@@ -6,10 +6,145 @@ [![Stackage LTS](http://stackage.org/package/ghc-syntax-highlighter/badge/lts)](http://stackage.org/lts/package/ghc-syntax-highlighter) [![Build Status](https://travis-ci.org/mrkkrp/ghc-syntax-highlighter.svg?branch=master)](https://travis-ci.org/mrkkrp/ghc-syntax-highlighter) -Syntax highlighter for Haskell using GHC itself as a library.+This is a syntax highlighter library for Haskell using lexer of GHC itself. +Here is a blog post announcing the package, the readme is mostly derived+from it:++* https://markkarpov.com/post/announcing-ghc-syntax-highlighter.html++## Motivation++Parsing Haskell is hard, because Haskell is a complex language with+countless features. The only way to get it right 100% is to use parser of+GHC itself. Fortunately, now there is [`ghc`][ghc] package, which as of+version 8.4.1 exports enough of GHC's source code to allow us use its lexer.++Alternative approaches, even decent ones like [`highlight.js`][hljs] either+don't support cutting-edge features or do their work without sufficient+precision so that many tokens end up combined and the end result is+typically still hard to read.++## API++The API is extremely simple:++```haskell+-- | Token types that are used as tags to mark spans of source code.++data Token+  = KeywordTok         -- ^ Keyword+  | PragmaTok          -- ^ Pragmas+  | SymbolTok          -- ^ Symbols (punctuation that is not an operator)+  | VariableTok        -- ^ Variable name (term level)+  | ConstructorTok     -- ^ Data\/type constructor+  | OperatorTok        -- ^ Operator+  | CharTok            -- ^ Character+  | StringTok          -- ^ String+  | IntegerTok         -- ^ Integer+  | RationalTok        -- ^ Rational number+  | CommentTok         -- ^ Comment (including Haddocks)+  | SpaceTok           -- ^ Space filling+  | OtherTok           -- ^ Something else?+  deriving (Eq, Ord, Enum, Bounded, Show)++-- | Tokenize Haskell source code. If the code cannot be parsed, return+-- 'Nothing'. Otherwise return the original input tagged by 'Token's.+--+-- The parser does not require the input source code to form a valid Haskell+-- program, so as long as the lexer can decompose your input (most of the+-- time), it'll return something in 'Just'.++tokenizeHaskell :: Text -> Maybe [(Token, Text)]+```++So given a simple program:++```haskell+module Main (main) where++import Data.Bits++-- | Program's entry point.++main :: IO ()+main = return ()+```++It outputs something like this:++```haskell+basicModule :: [(Token, Text)]+basicModule =+  [ (KeywordTok,"module")+  , (SpaceTok," ")+  , (ConstructorTok,"Main")+  , (SpaceTok," ")+  , (SymbolTok,"(")+  , (VariableTok,"main")+  , (SymbolTok,")")+  , (SpaceTok," ")+  , (KeywordTok,"where")+  , (SpaceTok,"\n\n")+  , (KeywordTok,"import")+  , (SpaceTok," ")+  , (ConstructorTok,"Data.Bits")+  , (SpaceTok,"\n\n")+  , (CommentTok,"-- | Program's entry point.")+  , (SpaceTok,"\n\n")+  , (VariableTok,"main")+  , (SpaceTok," ")+  , (SymbolTok,"::")+  , (SpaceTok," ")+  , (ConstructorTok,"IO")+  , (SpaceTok," ")+  , (SymbolTok,"(")+  , (SymbolTok,")")+  , (SpaceTok,"\n")+  , (VariableTok,"main")+  , (SpaceTok," ")+  , (SymbolTok,"=")+  , (SpaceTok," ")+  , (VariableTok,"return")+  , (SpaceTok," ")+  , (SymbolTok,"(")+  , (SymbolTok,")")+  , (SpaceTok,"\n")+  ]+```++`Nothing` is rarely returned if ever, because it looks like the lexer is+capable of interpreting almost any text as some stream of GHC tokens.++## How to use it in your blog++Depends on your markdown processor. If you're an [`mmark`][mmark] user, good+news, since version 0.2.1.0 of [`mmark-ext`][mmark-ext] it includes the+`ghcSyntaxHighlighter` extension. Due to flexibility of MMark, it's possible+to use this highlighter for Haskell and [`skylighting`][skylighting] as a+fall-back for everything else. Consult [the docs][mmark-ext-docs] for more+information.++[skylighting][skylighting] is what Pandoc uses. And from what I can tell+it's hardcoded to use only that library for highlighting, so some creativity+may be necessary to get it work.++## Contribution++Issues, bugs, and questions may be reported in [the GitHub issue tracker for+this project](https://github.com/mrkkrp/ghc-syntax-highlighter/issues).++Pull requests are also welcome and will be reviewed quickly.+ ## License  Copyright © 2018 Mark Karpov  Distributed under BSD 3 clause license.++[ghc]: https://hackage.haskell.org/package/ghc+[hljs]: https://highlightjs.org/+[mmark]: https://hackage.haskell.org/package/mmark+[mmark-ext]: https://hackage.haskell.org/package/mmark-ext+[skylighting]: https://hackage.haskell.org/package/skylighting+[mmark-ext-docs]: https://hackage.haskell.org/package/mmark-ext/docs/Text-MMark-Extension-GhcSyntaxHighlighter.html

@@ -1,5 +1,5 @@ name:                 ghc-syntax-highlighter-version:              0.0.1.0+version:              0.0.2.0 cabal-version:        1.18 tested-with:          GHC==8.4.2 license:              BSD3@@ -9,9 +9,9 @@ homepage:             https://github.com/mrkkrp/ghc-syntax-highlighter bug-reports:          https://github.com/mrkkrp/ghc-syntax-highlighter/issues category:             Text-synopsis:             Correct syntax highlighter for Haskell using GHC itself+synopsis:             Syntax highlighter for Haskell using lexer of GHC itself build-type:           Simple-description:          Correct syntax highlighter for Haskell using GHC itself.+description:          Syntax highlighter for Haskell using lexer of GHC itself. extra-doc-files:      CHANGELOG.md                     , README.md data-files:           data/*.hs