diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,7 @@ -## GHC syntax highligher 0.0.1.0 +## GHC syntax highlighter 0.0.2.0 + +* Added `Loc` and `tokenizeHaskellLoc`. + +## GHC syntax highlighter 0.0.1.0 * Initial release. diff --git a/GHC/SyntaxHighlighter.hs b/GHC/SyntaxHighlighter.hs --- a/GHC/SyntaxHighlighter.hs +++ b/GHC/SyntaxHighlighter.hs @@ -20,7 +20,9 @@ module GHC.SyntaxHighlighter ( Token (..) - , tokenizeHaskell ) + , Loc (..) + , tokenizeHaskell + , tokenizeHaskellLoc ) where import Control.Monad @@ -57,11 +59,18 @@ | OtherTok -- ^ Something else? deriving (Eq, Ord, Enum, Bounded, Show) --- | Internal type containing line\/column combinations for start and end --- positions of a code span. +-- | Start and end positions of a span. The arguments of the data +-- constructor contain in order: +-- +-- * Line number of start position of a span +-- * Column number of start position of a span +-- * Line number of end position of a span +-- * Column number of end position of a span +-- +-- @since 0.0.2.0 data Loc = Loc !Int !Int !Int !Int - deriving (Show) + deriving (Eq, Ord, Show) ---------------------------------------------------------------------------- -- High-level API @@ -74,10 +83,34 @@ -- time), it'll return something in 'Just'. tokenizeHaskell :: Text -> Maybe [(Token, Text)] -tokenizeHaskell input = +tokenizeHaskell input = sliceInputStream input <$> tokenizeHaskellLoc input + +-- | Replace 'Loc' locations with actual chunks of input 'Text'. + +sliceInputStream :: Text -> [(Token, Loc)] -> [(Token, Text)] +sliceInputStream input toks = unfoldr sliceOnce (initText' input, toks) + where + sliceOnce (txt, []) = do + (txt', chunk) <- tryFetchRest txt + return ((SpaceTok, chunk), (txt', [])) + sliceOnce (txt, tss@((t, l):ts)) = + case tryFetchSpace txt l of + Nothing -> + let (txt', chunk) = fetchSpan txt l + in Just ((t, chunk), (txt', ts)) + Just (txt', chunk) -> + Just ((SpaceTok, chunk), (txt', tss)) + +-- | Similar to 'tokenizeHaskell', but instead of 'Text' chunks provides +-- locations of corresponding spans in the given input stream. +-- +-- @since 0.0.2.0 + +tokenizeHaskellLoc :: Text -> Maybe [(Token, Loc)] +tokenizeHaskellLoc input = case L.unP pLexer parseState of L.PFailed {} -> Nothing - L.POk _ x -> Just (sliceInputStream input x) + L.POk _ x -> Just x where location = mkRealSrcLoc (mkFastString "") 1 1 buffer = stringToStringBuffer (T.unpack input) @@ -103,22 +136,6 @@ Nothing -> go Just x -> (x:) <$> go --- | Replace 'Loc' locations with actual chunks of input 'Text'. - -sliceInputStream :: Text -> [(Token, Loc)] -> [(Token, Text)] -sliceInputStream input toks = unfoldr sliceOnce (initText' input, toks) - where - sliceOnce (txt, []) = do - (txt', chunk) <- tryFetchRest txt - return ((SpaceTok, chunk), (txt', [])) - sliceOnce (txt, tss@((t, l):ts)) = - case tryFetchSpace txt l of - Nothing -> - let (txt', chunk) = fetchSpan txt l - in Just ((t, chunk), (txt', ts)) - Just (txt', chunk) -> - Just ((SpaceTok, chunk), (txt', tss)) - -- | Convert @'Located' 'L.Token'@ representation to a more convenient for -- us form. @@ -334,7 +351,7 @@ {-# UNPACK #-} !Text deriving (Show) --- | Create 'Text' from 'Text''. +-- | Create 'Text'' from 'Text'. initText' :: Text -> Text' initText' = Text' 1 1 @@ -356,7 +373,7 @@ then Nothing else Just (Text' l c "", txt) --- | Fetch span at 'Loc'. +-- | Fetch a span at 'Loc'. fetchSpan :: Text' -> Loc -> (Text', Text) fetchSpan txt (Loc _ _ el ec) = reachLoc txt el ec diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -6,10 +6,145 @@ [![Stackage LTS](http://stackage.org/package/ghc-syntax-highlighter/badge/lts)](http://stackage.org/lts/package/ghc-syntax-highlighter) [![Build Status](https://travis-ci.org/mrkkrp/ghc-syntax-highlighter.svg?branch=master)](https://travis-ci.org/mrkkrp/ghc-syntax-highlighter) -Syntax highlighter for Haskell using GHC itself as a library. +This is a syntax highlighter library for Haskell using lexer of GHC itself. +Here is a blog post announcing the package, the readme is mostly derived +from it: + +* https://markkarpov.com/post/announcing-ghc-syntax-highlighter.html + +## Motivation + +Parsing Haskell is hard, because Haskell is a complex language with +countless features. The only way to get it right 100% is to use parser of +GHC itself. Fortunately, now there is [`ghc`][ghc] package, which as of +version 8.4.1 exports enough of GHC's source code to allow us use its lexer. + +Alternative approaches, even decent ones like [`highlight.js`][hljs] either +don't support cutting-edge features or do their work without sufficient +precision so that many tokens end up combined and the end result is +typically still hard to read. + +## API + +The API is extremely simple: + +```haskell +-- | Token types that are used as tags to mark spans of source code. + +data Token + = KeywordTok -- ^ Keyword + | PragmaTok -- ^ Pragmas + | SymbolTok -- ^ Symbols (punctuation that is not an operator) + | VariableTok -- ^ Variable name (term level) + | ConstructorTok -- ^ Data\/type constructor + | OperatorTok -- ^ Operator + | CharTok -- ^ Character + | StringTok -- ^ String + | IntegerTok -- ^ Integer + | RationalTok -- ^ Rational number + | CommentTok -- ^ Comment (including Haddocks) + | SpaceTok -- ^ Space filling + | OtherTok -- ^ Something else? + deriving (Eq, Ord, Enum, Bounded, Show) + +-- | Tokenize Haskell source code. If the code cannot be parsed, return +-- 'Nothing'. Otherwise return the original input tagged by 'Token's. +-- +-- The parser does not require the input source code to form a valid Haskell +-- program, so as long as the lexer can decompose your input (most of the +-- time), it'll return something in 'Just'. + +tokenizeHaskell :: Text -> Maybe [(Token, Text)] +``` + +So given a simple program: + +```haskell +module Main (main) where + +import Data.Bits + +-- | Program's entry point. + +main :: IO () +main = return () +``` + +It outputs something like this: + +```haskell +basicModule :: [(Token, Text)] +basicModule = + [ (KeywordTok,"module") + , (SpaceTok," ") + , (ConstructorTok,"Main") + , (SpaceTok," ") + , (SymbolTok,"(") + , (VariableTok,"main") + , (SymbolTok,")") + , (SpaceTok," ") + , (KeywordTok,"where") + , (SpaceTok,"\n\n") + , (KeywordTok,"import") + , (SpaceTok," ") + , (ConstructorTok,"Data.Bits") + , (SpaceTok,"\n\n") + , (CommentTok,"-- | Program's entry point.") + , (SpaceTok,"\n\n") + , (VariableTok,"main") + , (SpaceTok," ") + , (SymbolTok,"::") + , (SpaceTok," ") + , (ConstructorTok,"IO") + , (SpaceTok," ") + , (SymbolTok,"(") + , (SymbolTok,")") + , (SpaceTok,"\n") + , (VariableTok,"main") + , (SpaceTok," ") + , (SymbolTok,"=") + , (SpaceTok," ") + , (VariableTok,"return") + , (SpaceTok," ") + , (SymbolTok,"(") + , (SymbolTok,")") + , (SpaceTok,"\n") + ] +``` + +`Nothing` is rarely returned if ever, because it looks like the lexer is +capable of interpreting almost any text as some stream of GHC tokens. + +## How to use it in your blog + +Depends on your markdown processor. If you're an [`mmark`][mmark] user, good +news, since version 0.2.1.0 of [`mmark-ext`][mmark-ext] it includes the +`ghcSyntaxHighlighter` extension. Due to flexibility of MMark, it's possible +to use this highlighter for Haskell and [`skylighting`][skylighting] as a +fall-back for everything else. Consult [the docs][mmark-ext-docs] for more +information. + +[skylighting][skylighting] is what Pandoc uses. And from what I can tell +it's hardcoded to use only that library for highlighting, so some creativity +may be necessary to get it work. + +## Contribution + +Issues, bugs, and questions may be reported in [the GitHub issue tracker for +this project](https://github.com/mrkkrp/ghc-syntax-highlighter/issues). + +Pull requests are also welcome and will be reviewed quickly. + ## License Copyright © 2018 Mark Karpov Distributed under BSD 3 clause license. + +[ghc]: https://hackage.haskell.org/package/ghc +[hljs]: https://highlightjs.org/ +[mmark]: https://hackage.haskell.org/package/mmark +[mmark-ext]: https://hackage.haskell.org/package/mmark-ext +[skylighting]: https://hackage.haskell.org/package/skylighting +[mmark-ext-docs]: https://hackage.haskell.org/package/mmark-ext/docs/Text-MMark-Extension-GhcSyntaxHighlighter.html diff --git a/ghc-syntax-highlighter.cabal b/ghc-syntax-highlighter.cabal --- a/ghc-syntax-highlighter.cabal +++ b/ghc-syntax-highlighter.cabal @@ -1,5 +1,5 @@ name: ghc-syntax-highlighter -version: 0.0.1.0 +version: 0.0.2.0 cabal-version: 1.18 tested-with: GHC==8.4.2 license: BSD3 @@ -9,9 +9,9 @@ homepage: https://github.com/mrkkrp/ghc-syntax-highlighter bug-reports: https://github.com/mrkkrp/ghc-syntax-highlighter/issues category: Text -synopsis: Correct syntax highlighter for Haskell using GHC itself +synopsis: Syntax highlighter for Haskell using lexer of GHC itself build-type: Simple -description: Correct syntax highlighter for Haskell using GHC itself. +description: Syntax highlighter for Haskell using lexer of GHC itself. extra-doc-files: CHANGELOG.md , README.md data-files: data/*.hs