commonmark (empty) → 0.1.0.0
raw patch · 22 files changed
+16188/−0 lines, 22 filesdep +QuickCheckdep +basedep +bytestringsetup-changed
Dependencies added: QuickCheck, base, bytestring, commonmark, containers, criterion, parsec, semigroups, tasty, tasty-hunit, tasty-quickcheck, text, transformers
Files
- LICENSE +30/−0
- README.md +141/−0
- Setup.hs +2/−0
- benchmark/benchmark.hs +105/−0
- changelog.md +3/−0
- commonmark.cabal +114/−0
- src/Commonmark.hs +58/−0
- src/Commonmark/Blocks.hs +1219/−0
- src/Commonmark/Entity.hs +2352/−0
- src/Commonmark/Html.hs +247/−0
- src/Commonmark/Inlines.hs +945/−0
- src/Commonmark/Parser.hs +60/−0
- src/Commonmark/ReferenceMap.hs +57/−0
- src/Commonmark/SourceMap.hs +133/−0
- src/Commonmark/Syntax.hs +61/−0
- src/Commonmark/Tag.hs +221/−0
- src/Commonmark/TokParsers.hs +194/−0
- src/Commonmark/Tokens.hs +73/−0
- src/Commonmark/Types.hs +155/−0
- test/regression.md +156/−0
- test/spec.txt +9710/−0
- test/test-commonmark.hs +152/−0
+ LICENSE view
@@ -0,0 +1,30 @@+Copyright Author name here (c) 2018++All rights reserved.++Redistribution and use in source and binary forms, with or without+modification, are permitted provided that the following conditions are met:++ * Redistributions of source code must retain the above copyright+ notice, this list of conditions and the following disclaimer.++ * Redistributions in binary form must reproduce the above+ copyright notice, this list of conditions and the following+ disclaimer in the documentation and/or other materials provided+ with the distribution.++ * Neither the name of Author name here nor the names of other+ contributors may be used to endorse or promote products derived+ from this software without specific prior written permission.++THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ README.md view
@@ -0,0 +1,141 @@+# commonmark++This package provides the core parsing functionality+for commonmark, together with HTML renderers.++:construction: This library is still in an **experimental state**.+Comments on the API and implementation are very much welcome.+Further changes should be expected.++The library is **fully commonmark-compliant** and passes the+test suite for version 0.29 of the commonmark spec.+It is designed to be **customizable and easily+extensible.** To customize the output, create an+AST, or support a new output format, one need only define some+new typeclass instances. It is also easy to add new syntax+elements or modify existing ones.++**Accurate information about source positions** is available+for all block and inline elements. Thus the library can be+used to create an accurate syntax highlighter or+an editor with synced live preview.++Finally, the library has been designed for **robust performance+even in pathological cases**. The parser behaves well on+pathological cases that tend to cause stack overflows or+exponential slowdowns in other parsers, with parsing speed that+varies linearly with input length.++## Related libraries++- **[`commonmark-extensions`](https://github.com/jgm/commonmark-hs/tree/master/commonmark-extensions)**+ provides a set of useful extensions to core commonmark syntax,+ including all GitHub-flavored Markdown extensions and many+ pandoc extensions. For convenience, the package of extensions+ defining GitHub-flavored Markdown is exported as `gfmExtensions`.++- **[`commonmark-pandoc`](https://github.com/jgm/commonmark-hs/tree/master/commonmark-pandoc)** defines+ type instances for parsing commonmark as a Pandoc AST.++- **[`commonmark-cli`](https://github.com/jgm/commonmark-hs/tree/master/commonmark-cli)** is a+ command-line program that uses this library to convert+ and syntax-highlight commonmark documents.+++## Simple usage example++This program reads commonmark from stdin and renders HTML to stdout:++``` haskell+{-# LANGUAGE ScopedTypeVariables #-}+import Commonmark+import Data.Text.IO as TIO+import Data.Text.Lazy.IO as TLIO++main = do+ res <- commonmark "stdin" <$> TIO.getContents+ case res of+ Left e -> error (show e)+ Right (html :: Html ()) -> TLIO.putStr $ renderHtml html+```++## Notes on the design++The input is a token stream (`[Tok]`), which can be+be produced from a `Text` using `tokenize`. The `Tok`+elements record source positions, making these easier+to track.++Extensibility is emphasized throughout. There are two ways in+which one might want to extend a commonmark converter. First,+one might want to support an alternate output format, or to+change the output for a given format. Second, one might want+to add new syntactic elements (e.g., definition lists).++To support both kinds of extension, we export the function++```haskell+parseCommonmarkWith :: (Monad m, IsBlock il bl, IsInline il)+ => SyntaxSpec m il bl -- ^ Defines syntax+ -> [Tok] -- ^ Tokenized commonmark input+ -> m (Either ParseError bl) -- ^ Result or error+```++The parser function takes two arguments: a `SyntaxSpec` which+defines parsing for the various syntactic elements, and a list+of tokens. Output is polymorphic: you can+convert commonmark to any type that is an instance of the+`IsBlock` typeclass. This gives tremendous flexibility.+Want to produce HTML? You can use the `Html ()` type defined+in `Commonmark.Types` for basic HTML, or `Html SourceRange`+for HTML with source range attributes on every element.++```haskell+GHCI> :set -XOverloadedStrings+GHCI>+GHCI> parseCommonmarkWith defaultSyntaxSpec (tokenize "source" "Hi there") :: IO (Either ParseError (Html ()))+Right <p>Hi there</p>+> parseCommonmarkWith defaultSyntaxSpec (tokenize "source" "Hi there") :: IO (Either ParseError (Html SourceRange))+Right <p data-sourcepos="source@1:1-1:9">Hi there</p>+```++Want to produce a Pandoc AST? You can use the type+`Cm a Text.Pandoc.Builder.Blocks` defined in `commonmark-pandoc`.++```haskell+GHCI> parseCommonmarkWith defaultSyntaxSpec (tokenize "source" "Hi there") :: Maybe (Either ParseError (Cm () B.Blocks))+Just (Right (Cm {unCm = Many {unMany = fromList [Para [Str "Hi",Space,Str "there"]]}}))+GHCI> parseCommonmarkWith defaultSyntaxSpec (tokenize "source" "Hi there") :: Maybe (Either ParseError (Cm SourceRange B.Blocks))+Just (Right (Cm {unCm = Many {unMany = fromList [Div ("",[],[("data-pos","source@1:1-1:9")]) [Para [Span ("",[],[("data-pos","source@1:1-1:3")]) [Str "Hi"],Span ("",[],[("data-pos","source@1:3-1:4")]) [Space],Span ("",[],[("data-pos","source@1:4-1:9")]) [Str "there"]]]]}}))+```++If you want to support another format (for example, Haddock's `DocH`),+just define typeclass instances of `IsBlock` and `IsInline` for+your type.++Supporting a new syntactic element generally requires (a) adding+a `SyntaxSpec` for it and (b) defining relevant type class+instances for the element. See the examples in+`Commonmark.Extensions.*`. Note that `SyntaxSpec` is a Monoid,+so you can specify `myNewSyntaxSpec <> defaultSyntaxSpec`.++## Performance++Here are some benchmarks on real-world commonmark documents,+using `make benchmark`. To get `benchmark.md`, we concatenated+a number of real-world commonmark documents. The resulting file+was 355K. The+[`bench`](http://hackage.haskell.org/package/bench) tool was+used to run the benchmarks.++ | program | time (ms) |+ | ------- | ---------:|+ | cmark | 12 |+ | cheapskate | 105 |+ | commonmark.js | 217 |+ | **commonmark-hs** | 229 |+ | pandoc -f commonmark | 948 |++It would be good to improve performance. I'd welcome help+with this.+
+ Setup.hs view
@@ -0,0 +1,2 @@+import Distribution.Simple+main = defaultMain
+ benchmark/benchmark.hs view
@@ -0,0 +1,105 @@+{-# LANGUAGE CPP #-}+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE ScopedTypeVariables #-}+import Criterion.Main+import Data.Text (Text)+import Data.Functor.Identity -- base >= 4.8+import Commonmark+import qualified Data.Text as T+import qualified Data.Text.IO as TIO+#if !MIN_VERSION_base(4,11,0)+import Data.Monoid+#endif++main :: IO ()+main = do+ sample <- T.replicate 10 <$> TIO.readFile "benchmark/sample.md"+ defaultMainWith defaultConfig+ [ bgroup "tokenize"+ [ benchTokenize ("sample.md", sample) ]+ , bgroup "parse sample.md"+ [ benchCommonmark defaultSyntaxSpec ("commonmark default", sample)+ ]+ , bgroup "pathological"+ (map toPathBench pathtests)+ ]++toPathBench :: (String, Int -> T.Text) -> Benchmark+toPathBench (name, ptest) =+ bgroup name+ [ bgroup "commonmark"+ (map (\n -> benchCommonmark defaultSyntaxSpec (show n, ptest n))+ [1000, 2000, 3000, 4000])+ ]++pathtests :: [(String, Int -> T.Text)]+pathtests =+ [ ("nested strong emph", \n ->+ let num = n `div` 14 in+ T.replicate num "*a **a " <> "b" <>+ T.replicate num " a** a*")+ , ("many emph closers with no openers", \n ->+ let num = n `div` 3 in+ T.replicate num "a_ ")+ , ("many emph openers with no closers", \n ->+ let num = n `div` 3 in+ T.replicate num "_a ")+ , ("many link closers with no openers", \n ->+ let num = n `div` 2 in+ T.replicate num "a]")+ , ("many link openers with no closers", \n ->+ let num = n `div` 2 in+ T.replicate num "[a")+ , ("mismatched openers and closers", \n ->+ let num = n `div` 3 in+ T.replicate num "*a_ ")+ , ("openers and closers multiple of 3", \n ->+ let num = n `div` 7 in+ T.replicate num "a**b" <> T.replicate num "c* ")+ , ("link openers and emph closers", \n ->+ let num = n `div` 4 in+ mconcat (replicate num "[ a_"))+ , ("nested brackets", \n ->+ let num = n `div` 2 in+ T.replicate num "[" <> "a" <> T.replicate num "]")+ , ("inline link openers without closers", \n ->+ let num = n `div` 3 in+ T.replicate num "[](")+ , ("repeated pattern '[ (]('" , \n ->+ let num = n `div` 5 in+ T.replicate num "[ (](")+ , ("nested block quotes", \n ->+ let num = n `div` 2 in+ T.replicate num "> " <> "a")+ , ("nested list", \n ->+ let num = floor (sqrt (fromIntegral n :: Double)) in+ mconcat (map (\ind -> T.replicate ind " " <> "- a\n") [0..(num - 1)]))+ , ("nested list 2", \n ->+ let num = n `div` 2 in+ T.replicate num "* " <> "a\n")+ , ("backticks", \n ->+ let num = floor (sqrt (9 + (8 * (fromIntegral n :: Double))) / 2) in+ mconcat $ map (\x -> "e" <> T.replicate x "`") [1..num])+ , ("CDATA", \n ->+ let num = n `div` 11 in+ T.replicate num "a <![CDATA[")+ , ("<?", \n ->+ let num = n `div` 2 in+ ("a" <> T.replicate num "<?"))+ , ("<!A ", \n ->+ let num = n `div` 4 in+ ("a" <> T.replicate num "<!A "))+ ]++benchCommonmark :: SyntaxSpec Identity (Html ()) (Html ())+ -> (String, Text)+ -> Benchmark+benchCommonmark spec (name, contents) =+ bench name $+ nf (either (error . show) renderHtml+ . runIdentity . parseCommonmarkWith spec . tokenize name)+ contents++benchTokenize :: (String, Text) -> Benchmark+benchTokenize (name, contents) =+ bench ("tokenize " ++ name) $ nf (length . tokenize name) contents
+ changelog.md view
@@ -0,0 +1,3 @@+# Changelog for commonmark++## Unreleased changes
+ commonmark.cabal view
@@ -0,0 +1,114 @@+cabal-version: 2.2+name: commonmark+version: 0.1.0.0+synopsis: Pure Haskell commonmark parser.+description:+ This library provides the core data types and functions+ for parsing commonmark (<https://spec.commonmark.org>).+ The parser is fully commonmark-compliant and passes the test+ suite. It is designed to be customizable and easily extensible.+ To customize the output, create an AST, or support a new output+ format, one need only define some new typeclass instances.+ It is also easy to add new syntax elements or modify existing ones.+ .+ Accurate information about source positions is available+ for all block and inline elements. Thus the library can be+ used to create an accurate syntax highlighter or+ an editor with live preview.+ .+ The parser has been designed for robust performance+ even in pathological cases that tend to cause stack overflows or+ exponential slowdowns in other parsers, with parsing speed that+ varies linearly with input length.+ .+ Related packages:+ .+ - commonmark-extensions (which defines a number of syntax extensions)+ - commonmark-pandoc (which allows using this parser to create a Pandoc+ structure)+ - commonmark-cli (a command-line tool for converting and+ syntax-highlighting commonmark documents)++homepage: https://github.com/jgm/commonmark-hs+category: Text+bug-reports: https://github.com/jgm/commonmark-hs/issues+stability: experimental+author: John MacFarlane+maintainer: jgm@berkeley.edu+copyright: 2018-2020 John MacFarlane+license: BSD-3-Clause+license-file: LICENSE+build-type: Simple++extra-source-files:+ changelog.md+ README.md+ test/spec.txt+ test/regression.md++source-repository head+ type: git+ location: https://github.com/jgm/commonmark-hs++library+ hs-source-dirs:+ src+ build-depends:+ base >=4.7 && <5+ , text+ , bytestring+ , containers+ , transformers+ , parsec+ if !impl(ghc >= 8.0)+ build-depends: semigroups == 0.18.*+ exposed-modules:+ Commonmark+ Commonmark.Parser+ Commonmark.Types+ Commonmark.Html+ Commonmark.Syntax+ Commonmark.ReferenceMap+ Commonmark.Tokens+ Commonmark.Inlines+ Commonmark.Blocks+ Commonmark.TokParsers+ Commonmark.SourceMap+ Commonmark.Tag+ Commonmark.Entity+ if impl(ghc >= 8.8)+ ghc-options: -fwrite-ide-info -hiedir=.hie+ ghc-options: -Wall -fno-warn-unused-do-bind -funbox-small-strict-fields+ default-language: Haskell2010++test-suite test-commonmark+ type: exitcode-stdio-1.0+ main-is: test-commonmark.hs+ hs-source-dirs:+ test+ ghc-options: -threaded -rtsopts -with-rtsopts=-K40K+ build-depends:+ base >=4.7 && <5+ , commonmark+ , text+ , tasty+ , tasty-quickcheck+ , tasty-hunit+ , QuickCheck+ , parsec+ default-language: Haskell2010++benchmark benchmark-commonmark+ type: exitcode-stdio-1.0+ main-is: benchmark.hs+ hs-source-dirs: benchmark+ build-depends:+ commonmark+ , base >= 4.8 && < 5+ , text+ , transformers+ , containers+ , bytestring+ , criterion >= 1.0 && < 1.6+ ghc-options: -threaded -rtsopts -with-rtsopts=-K10K+ default-language: Haskell2010
+ src/Commonmark.hs view
@@ -0,0 +1,58 @@+{- |++The basic task of this library is to parse text as commonmark.+Usage example:++> {-# LANGUAGE ScopedTypeVariables #-}+> import Commonmark+> import Data.Text.IO as TIO+> import Data.Text.Lazy.IO as TLIO+>+> main = do+> res <- commonmark "stdin" <$> TIO.getContents+> case res of+> Left e -> error (show e)+> Right (html :: Html ()) -> TLIO.putStr $ renderHtml html++The parser is highly polymorphic: in this example, we use+the type annotation @'Html' ()@ to indicate that we want it+to produce basic HTML without source location attributes.+And we return a value in the IO monad. But we could have used a different+output format (e.g. @'Html' 'SourceRange'@ for HTML with source location+attributes). And we could have used the Identity monad to get a+pure value. (The default parsers work the same way in any monad, but it is+possible to define extensions that constrain the monad. For+example, an extension for include files might only work in IO,+or might have different behavior in IO and Identity.)++Extensibility is emphasized throughout. To change the output+for a given format, or support an alternate output format,+one has only to define instances of 'IsBlock' and 'IsInline'+for a new type. (For an example of this kind of extension,+see the @commonmark-pandoc@ package, which defines these+instances for pandoc's native types.)++Supporting a new syntactic element generally requires (a) adding+a 'SyntaxSpec' for it and (b) defining new type classes.+See the examples in the @commonmark-extensions@ package.+Note that 'SyntaxSpec' is a Monoid, so one can extend+'defaultSyntaxSpec' by specifying @myNewSyntaxSpec <> defaultSyntaxSpec@.++-}++module Commonmark+ ( module Commonmark.Tokens+ , module Commonmark.Types+ , module Commonmark.Syntax+ , module Commonmark.Parser+ , module Commonmark.SourceMap+ , module Commonmark.Html+ ) where++import Commonmark.Tokens+import Commonmark.Types+import Commonmark.Parser+import Commonmark.Syntax+import Commonmark.SourceMap+import Commonmark.Html+
+ src/Commonmark/Blocks.hs view
@@ -0,0 +1,1219 @@+{-# LANGUAGE CPP #-}+{-# LANGUAGE RankNTypes #-}+{-# LANGUAGE AllowAmbiguousTypes #-}+{-# LANGUAGE FlexibleInstances #-}+{-# LANGUAGE MultiParamTypeClasses #-}+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE ScopedTypeVariables #-}+{-# LANGUAGE TupleSections #-}+{-# LANGUAGE UndecidableInstances #-}+{-# LANGUAGE BangPatterns #-}+module Commonmark.Blocks+ ( mkBlockParser+ , defaultBlockSpecs+ , BlockStartResult(..)+ , BlockSpec(..)+ , BlockData(..)+ , defBlockData+ , BlockNode+ , BPState(..)+ , BlockParser+ , LinkInfo(..)+ , defaultFinalizer+ , runInlineParser+ , addNodeToStack+ , collapseNodeStack+ , getBlockText+ , removeIndent+ , bspec+ , endOfBlock+ , interruptsParagraph+ , linkReferenceDef+ , renderChildren+ -- * BlockSpecs+ , docSpec+ , indentedCodeSpec+ , fencedCodeSpec+ , blockQuoteSpec+ , atxHeadingSpec+ , setextHeadingSpec+ , thematicBreakSpec+ , listItemSpec+ , bulletListMarker+ , orderedListMarker+ , rawHtmlSpec+ , attributeSpec+ , paraSpec+ , plainSpec+ )+where++import Commonmark.Tag+import Commonmark.TokParsers+import Commonmark.ReferenceMap+import Commonmark.Inlines (pEscaped, pLinkDestination,+ pLinkLabel, pLinkTitle)+import Commonmark.Entity (unEntity)+import Commonmark.Tokens+import Commonmark.Types+import Control.Monad (foldM, guard, mzero, void, unless,+ when)+import Control.Monad.Trans.Class (lift)+import Data.Foldable (foldrM)+#if !MIN_VERSION_base(4,11,0)+import Data.Monoid+#endif+import Data.Char (isAsciiUpper, isDigit, isSpace)+import Data.Dynamic+import Data.List (sort)+import Data.Text (Text)+import qualified Data.Map.Strict as M+import qualified Data.Text as T+import qualified Data.Text.Read as TR+import Data.Tree+import Text.Parsec++mkBlockParser+ :: (Monad m, IsBlock il bl)+ => [BlockSpec m il bl] -- ^ Defines block syntax+ -> [BlockParser m il bl bl] -- ^ Parsers to run at end+ -> (ReferenceMap -> [Tok] -> m (Either ParseError il)) -- ^ Inline parser+ -> [BlockParser m il bl Attributes] -- ^ attribute parsers+ -> [Tok] -- ^ Tokenized commonmark input+ -> m (Either ParseError bl) -- ^ Result or error+mkBlockParser specs finalParsers ilParser attrParsers ts =+ runParserT (do case ts of+ (t:_) -> setPosition (tokPos t)+ [] -> return ()+ processLines specs finalParsers)+ BPState{ referenceMap = emptyReferenceMap+ , inlineParser = ilParser+ , nodeStack = [Node (defBlockData docSpec) []]+ , blockMatched = False+ , maybeLazy = True+ , maybeBlank = True+ , counters = M.empty+ , failurePositions = M.empty+ , attributeParsers = attrParsers+ , nextAttributes = mempty+ }+ "source" (length ts `seq` ts)+ -- we evaluate length ts to make sure the list is+ -- fully evaluated; this helps performance. note that+ -- we can't use deepseq because there's no instance for SourcePos.++processLines :: (Monad m, IsBlock il bl)+ => [BlockSpec m il bl]+ -> [BlockParser m il bl bl] -- ^ Parsers to run at end+ -> BlockParser m il bl bl+processLines specs finalParsers = do+ let go = eof <|> (processLine specs >> go) in go+ tree <- (nodeStack <$> getState) >>= collapseNodeStack+ updateState $ \st -> st{ nodeStack = [reverseSubforests tree] }+ endContent <- mconcat <$> sequence finalParsers+ tree':_ <- nodeStack <$> getState+ body <- blockConstructor (blockSpec (rootLabel tree')) tree'+ return $! body <> endContent+{-# SCC processLines #-}++reverseSubforests :: Tree a -> Tree a+reverseSubforests (Node x cs) = Node x $ map reverseSubforests $ reverse cs++processLine :: (Monad m, IsBlock il bl)+ => [BlockSpec m il bl] -> BlockParser m il bl ()+processLine specs = do+ -- check block continuations for each node in stack+ st' <- getState+ putState $ st'{ blockMatched = True+ , maybeLazy = True+ , maybeBlank = True+ , failurePositions = M.empty }+ (matched, unmatched) <- foldrM checkContinue ([],[]) (nodeStack st')++ -- if not everything matched, and last unmatched is paragraph,+ -- then we may have a lazy paragraph continuation+ updateState $ \st -> st{ maybeLazy = maybeLazy st &&+ case unmatched of+ m:_ -> blockParagraph (bspec m)+ _ -> False }++ -- close unmatched blocks+ if null unmatched+ then updateState $ \st -> st{ nodeStack = matched }+ -- this update is needed or we lose startpos information+ else case matched of+ [] -> error "no blocks matched"+ m:ms -> do+ m' <- collapseNodeStack (unmatched ++ [m])+ updateState $ \st -> st{ nodeStack = m':ms }++ restBlank <- option False $ True <$ lookAhead blankLine++ {-# SCC block_starts #-} unless restBlank $+ (do skipMany1 (doBlockStarts specs)+ optional (try (blockStart paraSpec)))+ <|>+ (do getState >>= guard . maybeLazy+ -- lazy line+ sp <- getPosition+ updateState $ \st -> st{ nodeStack =+ map (addStartPos sp) (unmatched ++ matched) })+ <|>+ void (try (blockStart paraSpec))+ <|>+ return ()++ (cur:rest) <- nodeStack <$> getState+ -- add line contents+ let curdata = rootLabel cur+ when (blockParagraph (bspec cur)) $ skipMany spaceTok+ pos <- getPosition+ toks <- {-# SCC restOfLine #-} restOfLine+ updateState $ \st -> st{+ nodeStack =+ cur{ rootLabel =+ if blockContainsLines (bspec cur)+ then curdata{ blockLines = toks : blockLines curdata }+ else+ if maybeBlank st && restBlank+ then curdata{ blockBlanks = sourceLine pos :+ blockBlanks curdata }+ else curdata+ } : rest+ }+ -- showNodeStack++addStartPos :: SourcePos -> BlockNode m il bl -> BlockNode m il bl+addStartPos sp (Node bd cs) = Node bd{ blockStartPos = sp : blockStartPos bd } cs++doBlockStarts :: Monad m => [BlockSpec m il bl] -> BlockParser m il bl ()+doBlockStarts specs = do+ st' <- getState+ initPos <- getPosition+ let failurePosMap = failurePositions st'+ let specs' = foldr (\spec sps ->+ case M.lookup (blockType spec) failurePosMap of+ Just pos' | initPos < pos' -> sps+ _ -> spec:sps) [] specs+ go initPos specs'+ where+ go _ [] = mzero+ go initPos (spec:otherSpecs) = try (do+ pst <- getParserState+ res <- blockStart spec+ case res of+ BlockStartMatch -> return ()+ BlockStartNoMatchBefore pos -> do+ setParserState pst+ unless (pos == initPos) $+ updateState $ \st ->+ st{ failurePositions =+ M.insert (blockType spec)+ pos (failurePositions st) }+ go initPos otherSpecs) <|> go initPos otherSpecs++checkContinue :: Monad m+ => BlockNode m il bl+ -> ([BlockNode m il bl],[BlockNode m il bl])+ -> BlockParser m il bl ([BlockNode m il bl],[BlockNode m il bl])+checkContinue nd (matched, unmatched) = do+ ismatched <- blockMatched <$> getState+ if ismatched+ then+ {-# SCC blockContinues #-}+ (do (startpos, Node bdata children) <- blockContinue (bspec nd) nd+ matched' <- blockMatched <$> getState+ -- if blockContinue set blockMatched to False, it's+ -- because of characters on the line closing the block,+ -- so it's not to be counted as blank:+ unless matched' $+ updateState $ \st -> st{ maybeBlank = False,+ maybeLazy = False }+ let new = Node bdata{ blockStartPos =+ startpos : blockStartPos bdata+ } children+ return $!+ if matched'+ then (new:matched, unmatched)+ else (matched, new:unmatched))+ <|> (matched, nd:unmatched) <$ updateState (\st -> st{+ blockMatched = False })+ else return $! (matched, nd:unmatched)+++{-+--- for debugging+showNodeStack :: Monad m => BlockParser m il bl a+showNodeStack = do+ ns <- nodeStack <$> getState+ trace (unlines ("NODESTACK:" : map showNode ns)) (return $! ())+ return undefined+ where+ showNode (Node bdata children) =+ unlines [ "-----"+ , show (blockSpec bdata)+ , show (blockStartPos bdata)+ , show (length children) ]+-}++data BlockStartResult =+ BlockStartMatch+ | BlockStartNoMatchBefore !SourcePos+ deriving (Show, Eq)++-- | Defines a block-level element type.+data BlockSpec m il bl = BlockSpec+ { blockType :: !Text -- ^ Descriptive name of block type+ , blockStart :: BlockParser m il bl BlockStartResult+ -- ^ Parses beginning+ -- of block. The parser should verify any+ -- preconditions, parse the opening of the block,+ -- and add the new block to the block stack using+ -- 'addNodeToStack', returning 'BlockStartMatch' on+ -- success. If the match fails, the parser can+ -- either fail or return 'BlockStartNoMatchBefore' and a+ -- 'SourcePos' before which the parser is known+ -- not to succeed (this will be stored in+ -- 'failurePositions' for the line, to ensure+ -- that future matches won't be attempted until+ -- after that position).+ , blockCanContain :: BlockSpec m il bl -> Bool -- ^ Returns True if+ -- this kind of block can contain the specified+ -- block type.+ , blockContainsLines :: !Bool -- ^ True if this kind of block+ -- can contain text lines.+ , blockParagraph :: !Bool -- ^ True if this kind of block+ -- is paragraph.+ , blockContinue :: BlockNode m il bl+ -> BlockParser m il bl (SourcePos, BlockNode m il bl)+ -- ^ Parser that checks to see if the current+ -- block (the 'BlockNode') can be kept open.+ -- If it fails, the block will be closed, unless+ -- we have a lazy paragraph continuation within+ -- the block.+ , blockConstructor :: BlockNode m il bl -> BlockParser m il bl bl+ -- ^ Renders the node into its target format,+ -- possibly after rendering inline content.+ , blockFinalize :: BlockNode m il bl -> BlockNode m il bl+ -> BlockParser m il bl (BlockNode m il bl)+ -- ^ Runs when the block is closed, but prior+ -- to rendering. The first parameter is the+ -- child, the second the parent.+ }++instance Show (BlockSpec m il bl) where+ show bs = "<BlockSpec " ++ T.unpack (blockType bs) ++ ">"++defaultBlockSpecs :: (Monad m, IsBlock il bl) => [BlockSpec m il bl]+defaultBlockSpecs =+ [ indentedCodeSpec+ , fencedCodeSpec+ , blockQuoteSpec+ , atxHeadingSpec+ , setextHeadingSpec+ , thematicBreakSpec+ , listItemSpec (bulletListMarker <|> orderedListMarker)+ , rawHtmlSpec+ , attributeSpec+ ]++defaultFinalizer :: Monad m+ => BlockNode m il bl+ -> BlockNode m il bl+ -> BlockParser m il bl (BlockNode m il bl)+defaultFinalizer !child !parent = do+ -- ensure that 'counters' carries information about all+ -- the block identifiers used, so that auto_identifiers works properly.+ case lookup "id" (blockAttributes (rootLabel child)) of+ Nothing -> return ()+ Just !ident -> updateState $ \st ->+ st{ counters = M.insert ("identifier:" <> ident)+ (toDyn (0 :: Int)) (counters st) }+ return $! parent{ subForest = child : subForest parent }++data BlockData m il bl = BlockData+ { blockSpec :: BlockSpec m il bl+ , blockLines :: [[Tok]] -- in reverse order+ , blockStartPos :: [SourcePos] -- in reverse order+ , blockData :: !Dynamic+ , blockBlanks :: [Int] -- non-content blank lines in block+ , blockAttributes :: !Attributes+ }+ deriving Show++defBlockData :: BlockSpec m il bl -> BlockData m il bl+defBlockData spec = BlockData+ { blockSpec = spec+ , blockLines = []+ , blockStartPos = []+ , blockData = toDyn ()+ , blockBlanks = []+ , blockAttributes = mempty+ }++type BlockNode m il bl = Tree (BlockData m il bl)++data BPState m il bl = BPState+ { referenceMap :: !ReferenceMap+ , inlineParser :: ReferenceMap -> [Tok] -> m (Either ParseError il)+ , nodeStack :: [BlockNode m il bl] -- reverse order, head is tip+ , blockMatched :: !Bool+ , maybeLazy :: !Bool+ , maybeBlank :: !Bool+ , counters :: M.Map Text Dynamic+ , failurePositions :: M.Map Text SourcePos -- record known positions+ -- where parsers fail to avoid repetition+ , attributeParsers :: [ParsecT [Tok] (BPState m il bl) m Attributes]+ , nextAttributes :: !Attributes+ }++type BlockParser m il bl = ParsecT [Tok] (BPState m il bl) m++data ListData = ListData+ { listType :: !ListType+ , listSpacing :: !ListSpacing+ } deriving (Show, Eq)++data ListItemData = ListItemData+ { listItemType :: !ListType+ , listItemIndent :: !Int+ , listItemBlanksInside :: !Bool+ , listItemBlanksAtEnd :: !Bool+ } deriving (Show, Eq)++runInlineParser :: Monad m+ => [Tok]+ -> BlockParser m il bl il+runInlineParser toks = do+ refmap <- referenceMap <$> getState+ ilParser <- inlineParser <$> getState+ res <- lift $ ilParser refmap toks+ case res of+ Right ils -> return $! ils+ Left err -> mkPT (\_ -> return (Empty (return (Error err))))+ -- pass up ParseError+{-# SCC runInlineParser #-}++addRange :: (Monad m, IsBlock il bl)+ => BlockNode m il bl -> bl -> bl+addRange (Node b _)+ = ranged (SourceRange+ (go . reverse $ map (\pos ->+ (pos, setSourceColumn+ (incSourceLine pos 1) 1))+ (blockStartPos b)))+ where+ go [] = []+ go ((!startpos1, !endpos1):(!startpos2, !endpos2):rest)+ | endpos1 == startpos2 = go ((startpos1, endpos2):rest)+ go (!x:xs) = x : go xs++-- Add a new node to the block stack. If current tip can contain+-- it, add it there; otherwise, close the tip and repeat til we get+-- to a block that can contain this node.+addNodeToStack :: Monad m => BlockNode m bl il -> BlockParser m bl il ()+addNodeToStack node = do+ (cur:rest) <- nodeStack <$> getState+ guard $ blockParagraph (bspec cur) || not (blockContainsLines (bspec cur))+ if blockCanContain (bspec cur) (bspec node)+ then do+ nextAttr <- nextAttributes <$> getState+ let node' = if null nextAttr+ then node+ else+ let rl = rootLabel node+ in node{ rootLabel = rl{+ blockAttributes = nextAttr+ }}+ updateState $ \st ->+ st{ nextAttributes = mempty+ , nodeStack = node' : cur : rest+ , maybeLazy = False }+ else case rest of+ (x:xs) -> do+ stack <- (:xs) <$> collapseNodeStack [cur,x]+ updateState $ \st -> st{ nodeStack = stack }+ addNodeToStack node+ _ -> mzero++interruptsParagraph :: Monad m => BlockParser m bl il Bool+interruptsParagraph = do+ (cur:_) <- nodeStack <$> getState+ return $! blockParagraph (bspec cur)++renderChildren :: (Monad m, IsBlock il bl)+ => BlockNode m il bl -> BlockParser m il bl [bl]+renderChildren node = mapM renderC $ subForest node+ where+ renderC n = do+ let attrs = blockAttributes (rootLabel n)+ (if null attrs+ then id+ else addAttributes attrs) .+ addRange n <$> blockConstructor (blockSpec (rootLabel n)) n++docSpec :: (Monad m, IsBlock il bl, Monoid bl) => BlockSpec m il bl+docSpec = BlockSpec+ { blockType = "Doc"+ , blockStart = mzero+ , blockCanContain = const True+ , blockContainsLines = False+ , blockParagraph = False+ , blockContinue = \n -> (,n) <$> getPosition+ , blockConstructor = fmap mconcat . renderChildren+ , blockFinalize = defaultFinalizer+ }++refLinkDefSpec :: (Monad m, IsBlock il bl)+ => BlockSpec m il bl+refLinkDefSpec = BlockSpec+ { blockType = "ReferenceLinkDefinition"+ , blockStart = mzero+ , blockCanContain = const False+ , blockContainsLines = False+ , blockParagraph = False+ , blockContinue = const mzero+ , blockConstructor = \node -> do+ let linkdefs = fromDyn (blockData (rootLabel node))+ undefined :: [((SourceRange, Text), LinkInfo)]+ return $! mconcat $ map (\((range, lab), linkinfo) ->+ (ranged range+ (addAttributes (linkAttributes linkinfo)+ (referenceLinkDefinition lab (linkDestination linkinfo,+ linkTitle linkinfo))))) linkdefs+ , blockFinalize = defaultFinalizer+ }++-- Parse reference links from beginning of block text;+-- update reference map and block text; return maybe altered node+-- (if it still contains lines) and maybe ref link node.+extractReferenceLinks :: (Monad m, IsBlock il bl)+ => BlockNode m il bl+ -> BlockParser m il bl (Maybe (BlockNode m il bl),+ Maybe (BlockNode m il bl))+extractReferenceLinks node = do+ st <- getState+ res <- lift $ runParserT ((,) <$> ((lookAhead anyTok >>= setPosition . tokPos) >>+ many1 (linkReferenceDef (choice $ attributeParsers st)))+ <*> getInput) st "" (getBlockText node)+ case res of+ Left _ -> return $! (Just node, Nothing)+ Right (linkdefs, toks') -> do+ mapM_+ (\((_,lab),linkinfo) ->+ updateState $ \s -> s{+ referenceMap = insertReference lab linkinfo+ (referenceMap s) }) linkdefs+ let isRefPos = case toks' of+ (t:_) -> (< tokPos t)+ _ -> const False+ let node' = if null toks'+ then Nothing+ else Just node{ rootLabel =+ (rootLabel node){+ blockLines = [toks'],+ blockStartPos = dropWhile isRefPos+ (blockStartPos (rootLabel node))+ }+ }+ let refnode = node{ rootLabel =+ (rootLabel node){+ blockLines = takeWhile (any (isRefPos . tokPos))+ (blockLines (rootLabel node))+ , blockStartPos = takeWhile isRefPos+ (blockStartPos (rootLabel node))+ , blockData = toDyn linkdefs+ , blockSpec = refLinkDefSpec+ }}+ return $! (node', Just refnode)++attributeSpec :: (Monad m, IsBlock il bl)+ => BlockSpec m il bl+attributeSpec = BlockSpec+ { blockType = "Attribute"+ , blockStart = do+ attrParsers <- attributeParsers <$> getState+ guard $ not (null attrParsers)+ interruptsParagraph >>= guard . not+ nonindentSpaces+ pos <- getPosition+ attrs <- choice attrParsers+ skipWhile (hasType Spaces)+ lookAhead (void lineEnd <|> eof)+ addNodeToStack $+ Node (defBlockData attributeSpec){+ blockData = toDyn attrs,+ blockStartPos = [pos] } []+ return BlockStartMatch+ , blockCanContain = const False+ , blockContainsLines = False+ , blockParagraph = False+ , blockContinue = \n -> do+ attrParsers <- attributeParsers <$> getState+ guard $ not (null attrParsers)+ nonindentSpaces+ pos <- getPosition+ attrs <- choice attrParsers+ skipWhile (hasType Spaces)+ lookAhead (void lineEnd <|> eof)+ let oldattrs = fromDyn (blockData (rootLabel n)) mempty :: Attributes+ let attrs' = oldattrs <> attrs+ return $! (pos, n{ rootLabel = (rootLabel n){+ blockData = toDyn attrs' }})+ , blockConstructor = \_ -> return $! mempty+ , blockFinalize = \node parent -> do+ let attrs = fromDyn (blockData (rootLabel node)) mempty :: Attributes+ updateState $ \st -> st{ nextAttributes = attrs }+ defaultFinalizer node parent+ }++paraSpec :: (Monad m, IsBlock il bl)+ => BlockSpec m il bl+paraSpec = BlockSpec+ { blockType = "Paragraph"+ , blockStart = do+ interruptsParagraph >>= guard . not+ skipWhile (hasType Spaces)+ pos <- getPosition+ notFollowedBy lineEnd+ addNodeToStack $+ Node (defBlockData paraSpec){+ blockStartPos = [pos] } []+ return BlockStartMatch+ , blockCanContain = const False+ , blockContainsLines = True+ , blockParagraph = True+ , blockContinue = \n -> lookAhead $ try $ do+ skipWhile (hasType Spaces)+ pos <- getPosition+ notFollowedBy lineEnd+ return $! (pos, n)+ , blockConstructor = \node ->+ paragraph <$> runInlineParser (getBlockText node)+ , blockFinalize = \child parent -> do+ (mbchild, mbrefdefs) <- extractReferenceLinks child+ case (mbchild, mbrefdefs) of+ (_, Nothing) -> defaultFinalizer child parent+ (Nothing, Just refnode)+ -> return $! parent{ subForest =+ refnode : subForest parent }+ (Just child', Just refnode)+ -> return $! parent{ subForest =+ child' : refnode : subForest parent }+ }++plainSpec :: (Monad m, IsBlock il bl)+ => BlockSpec m il bl+plainSpec = paraSpec{+ blockConstructor = \node ->+ plain <$> runInlineParser (getBlockText node)+ }+++linkReferenceDef :: Monad m+ => ParsecT [Tok] s m Attributes+ -> ParsecT [Tok] s m ((SourceRange, Text), LinkInfo)+linkReferenceDef attrParser = try $ do+ startpos <- getPosition+ lab <- pLinkLabel+ guard $ not $ T.all isSpace lab+ symbol ':'+ optional whitespace+ dest <- pLinkDestination+ (title, attrs) <- option (mempty, mempty) $ try $ do+ tit <- option mempty $ try (whitespace *> pLinkTitle)+ skipWhile (hasType Spaces)+ as <- option mempty attrParser+ skipWhile (hasType Spaces)+ lookAhead (void lineEnd <|> eof)+ return $! (tit, as)+ endpos <- getPosition+ void lineEnd <|> eof+ return $! ((SourceRange [(startpos, endpos)], lab),+ LinkInfo{ linkDestination = unEntity dest+ , linkTitle = unEntity title+ , linkAttributes = attrs })++atxHeadingSpec :: (Monad m, IsBlock il bl)+ => BlockSpec m il bl+atxHeadingSpec = BlockSpec+ { blockType = "ATXHeading"+ , blockStart = do+ nonindentSpaces+ pos <- getPosition+ hashes <- many1 (symbol '#')+ let level = length hashes+ guard $ level <= 6+ (spaceTok *> skipMany spaceTok)+ <|> void (lookAhead lineEnd)+ <|> lookAhead eof+ raw <- many (satisfyTok (not . hasType LineEnd))+ -- trim off closing ###+ let removeClosingHash (_ :: Int) [] = []+ removeClosingHash 0 (Tok Spaces _ _ : xs) =+ removeClosingHash 0 xs+ removeClosingHash _ (Tok (Symbol '#') _ _ :+ Tok (Symbol '\\') _ _ : _) =+ reverse raw+ removeClosingHash _ (Tok (Symbol '#') _ _ : xs) =+ removeClosingHash 1 xs+ removeClosingHash 1 (Tok Spaces _ _ : xs) = xs+ removeClosingHash 1 (x:_)+ | tokType x /= Symbol '#' = reverse raw+ removeClosingHash _ xs = xs+ let raw' = reverse . removeClosingHash 0 . reverse $ raw+ addNodeToStack $ Node (defBlockData atxHeadingSpec){+ blockLines = [raw'],+ blockData = toDyn level,+ blockStartPos = [pos] } []+ return BlockStartMatch+ , blockCanContain = const False+ , blockContainsLines = False+ , blockParagraph = False+ , blockContinue = const mzero+ , blockConstructor = \node -> do+ let level = fromDyn (blockData (rootLabel node)) 1+ ils <- runInlineParser (getBlockText node)+ return $! heading level ils+ , blockFinalize = \node@(Node cdata children) parent -> do+ let oldAttr = blockAttributes cdata+ let toks = getBlockText node+ (newtoks, attr) <- parseFinalAttributes True toks+ <|> (return $! (toks, mempty))+ defaultFinalizer (Node cdata{ blockAttributes = oldAttr <> attr+ , blockLines = [newtoks] }+ children) parent+ }++setextHeadingSpec :: (Monad m, IsBlock il bl)+ => BlockSpec m il bl+setextHeadingSpec = BlockSpec+ { blockType = "SetextHeading"+ , blockStart = do+ (cur:rest) <- nodeStack <$> getState+ guard $ blockParagraph (bspec cur)+ nonindentSpaces+ pos <- getPosition+ level <- (2 :: Int) <$ skipMany1 (symbol '-')+ <|> (1 :: Int) <$ skipMany1 (symbol '=')+ skipWhile (hasType Spaces)+ lookAhead (eof <|> void lineEnd)+ -- process any reference links, make sure there's some+ -- content left+ (mbcur, mbrefdefs) <- extractReferenceLinks cur+ updateState $ \st ->+ st{ nodeStack = case mbrefdefs of+ Nothing -> rest+ Just rd -> case rest of+ (x:xs) ->+ x{ subForest =+ rd : subForest x }:xs+ [] -> [rd] }+ case mbcur of+ Nothing -> mzero -- should not happen+ Just cur' -> do+ -- replace cur with new setext heading node+ addNodeToStack $+ Node (rootLabel cur'){+ blockSpec = setextHeadingSpec,+ blockData = toDyn level,+ blockStartPos =+ blockStartPos (rootLabel cur') ++ [pos] }+ []+ return BlockStartMatch+ , blockCanContain = const False+ , blockContainsLines = True+ , blockParagraph = False+ , blockContinue = const mzero+ , blockConstructor = \node -> do+ let level = fromDyn (blockData (rootLabel node)) 1+ ils <- runInlineParser (getBlockText node)+ return $! heading level ils+ , blockFinalize = \node@(Node cdata children) parent -> do+ let oldAttr = blockAttributes cdata+ let toks = getBlockText node+ (newtoks, attr) <- parseFinalAttributes True toks+ <|> (return $! (toks, mempty))+ defaultFinalizer (Node cdata{ blockAttributes = oldAttr <> attr+ , blockLines = [newtoks] }+ children) parent+ }++parseFinalAttributes :: Monad m+ => Bool -> [Tok] -> BlockParser m il bl ([Tok], Attributes)+parseFinalAttributes requireWhitespace ts = do+ attrParsers <- attributeParsers <$> getState+ let pAttr' = try $ (if requireWhitespace+ then () <$ whitespace+ else optional whitespace)+ *> choice attrParsers <* optional whitespace <* eof+ st <- getState+ res <- lift $ runParserT+ ((,) <$> many (notFollowedBy pAttr' >> anyTok)+ <*> option [] pAttr') st "heading contents" ts+ case res of+ Left _ -> mzero+ Right (xs, ys) -> return $! (xs, ys)++blockQuoteSpec :: (Monad m, IsBlock il bl) => BlockSpec m il bl+blockQuoteSpec = BlockSpec+ { blockType = "BlockQuote"+ , blockStart = do+ nonindentSpaces+ pos <- getPosition+ _ <- symbol '>'+ _ <- option 0 (gobbleSpaces 1)+ addNodeToStack $+ Node (defBlockData blockQuoteSpec){+ blockStartPos = [pos] } []+ return BlockStartMatch+ , blockCanContain = const True+ , blockContainsLines = False+ , blockParagraph = False+ , blockContinue = \n -> try $ do+ nonindentSpaces+ pos <- getPosition+ _ <- symbol '>'+ _ <- gobbleUpToSpaces 1+ return $! (pos, n)+ , blockConstructor = \node ->+ (blockQuote . mconcat) <$> renderChildren node+ , blockFinalize = defaultFinalizer+ }++listItemSpec :: (Monad m, IsBlock il bl)+ => BlockParser m il bl ListType+ -> BlockSpec m il bl+listItemSpec parseListMarker = BlockSpec+ { blockType = "ListItem"+ , blockStart = do+ (pos, lidata) <- itemStart parseListMarker+ let linode = Node (defBlockData $ listItemSpec parseListMarker){+ blockData = toDyn lidata,+ blockStartPos = [pos] } []+ let listdata = ListData{+ listType = listItemType lidata+ , listSpacing = TightList }+ -- spacing gets set in finalize+ let listnode = Node (defBlockData listSpec){+ blockData = toDyn listdata,+ blockStartPos = [pos] } []+ -- list can only interrupt paragraph if bullet+ -- list or ordered list w/ startnum == 1,+ -- and not followed by blank+ (cur:_) <- nodeStack <$> getState+ when (blockParagraph (bspec cur)) $ do+ guard $ case listType listdata of+ BulletList _ -> True+ OrderedList 1 Decimal _ -> True+ _ -> False+ notFollowedBy blankLine+ let curdata = fromDyn (blockData (rootLabel cur))+ (ListData (BulletList '*') TightList)+ let matchesList (BulletList c) (BulletList d) = c == d+ matchesList (OrderedList _ e1 d1)+ (OrderedList _ e2 d2) = e1 == e2 && d1 == d2+ matchesList _ _ = False+ case blockType (bspec cur) of+ "List" | listType curdata `matchesList`+ listItemType lidata+ -> addNodeToStack linode+ _ -> addNodeToStack listnode >> addNodeToStack linode+ return BlockStartMatch+ , blockCanContain = const True+ , blockContainsLines = False+ , blockParagraph = False+ , blockContinue = \node@(Node ndata children) -> do+ let lidata = fromDyn (blockData ndata)+ (ListItemData (BulletList '*') 0 False False)+ -- a marker followed by two blanks is just an empty item:+ guard $ null (blockBlanks ndata) ||+ not (null children)+ pos <- getPosition+ gobbleSpaces (listItemIndent lidata) <|> 0 <$ lookAhead blankLine+ return $! (pos, node)+ , blockConstructor = fmap mconcat . renderChildren+ , blockFinalize = \(Node cdata children) parent -> do+ let lidata = fromDyn (blockData cdata)+ (ListItemData (BulletList '*')+ 0 False False)+ let blanks = removeConsecutive $ sort $+ concat $ blockBlanks cdata :+ map (blockBlanks . rootLabel)+ (filter ((== "List") . blockType .+ blockSpec . rootLabel) children)+ curline <- sourceLine <$> getPosition+ let blanksAtEnd = case blanks of+ (l:_) -> l >= curline - 1+ _ -> False+ let blanksInside = case length blanks of+ n | n > 1 -> True+ | n == 1 -> not blanksAtEnd+ | otherwise -> False+ let lidata' = toDyn $ lidata{ listItemBlanksInside = blanksInside+ , listItemBlanksAtEnd = blanksAtEnd }+ defaultFinalizer (Node cdata{ blockData = lidata' } children)+ parent+ }++itemStart :: Monad m+ => BlockParser m il bl ListType+ -> BlockParser m il bl (SourcePos, ListItemData)+itemStart parseListMarker = do+ beforecol <- sourceColumn <$> getPosition+ gobbleUpToSpaces 3+ pos <- getPosition+ ty <- parseListMarker+ aftercol <- sourceColumn <$> getPosition+ lookAhead whitespace+ numspaces <- try (gobbleUpToSpaces 4 <* notFollowedBy whitespace)+ <|> gobbleSpaces 1+ <|> 1 <$ lookAhead lineEnd+ return $! (pos, ListItemData{+ listItemType = ty+ , listItemIndent = (aftercol - beforecol) + numspaces+ , listItemBlanksInside = False+ , listItemBlanksAtEnd = False+ })++bulletListMarker :: Monad m => BlockParser m il bl ListType+bulletListMarker = do+ Tok (Symbol c) _ _ <- symbol '-' <|> symbol '*' <|> symbol '+'+ return $! BulletList c++orderedListMarker :: Monad m => BlockParser m il bl ListType+orderedListMarker = do+ Tok WordChars _ ds <- satisfyWord (\t -> T.all isDigit t && T.length t < 10)+ (start :: Int) <- either fail (return . fst) (TR.decimal ds)+ delimtype <- Period <$ symbol '.' <|> OneParen <$ symbol ')'+ return $! OrderedList start Decimal delimtype++listSpec :: (Monad m, IsBlock il bl) => BlockSpec m il bl+listSpec = BlockSpec+ { blockType = "List"+ , blockStart = mzero+ , blockCanContain = \sp -> blockType sp == "ListItem"+ , blockContainsLines = False+ , blockParagraph = False+ , blockContinue = \n -> (,n) <$> getPosition+ , blockConstructor = \node -> do+ let ListData lt ls = fromDyn (blockData (rootLabel node))+ (ListData (BulletList '*') TightList)+ list lt ls <$> renderChildren node+ , blockFinalize = \(Node cdata children) parent -> do+ let ListData lt _ = fromDyn (blockData cdata)+ (ListData (BulletList '*') TightList)+ let getListItemData (Node d _) =+ fromDyn (blockData d)+ (ListItemData (BulletList '*') 0 False False)+ let childrenData = map getListItemData children+ let ls = case childrenData of+ c:cs | any listItemBlanksInside (c:cs) ||+ (not (null cs) &&+ any listItemBlanksAtEnd cs)+ -> LooseList+ _ -> TightList+ blockBlanks' <- case childrenData of+ c:_ | listItemBlanksAtEnd c -> do+ curline <- sourceLine <$> getPosition+ return $! curline - 1 : blockBlanks cdata+ _ -> return $! blockBlanks cdata+ let ldata' = toDyn (ListData lt ls)+ -- need to transform paragraphs on tight lists+ let totight (Node nd cs)+ | blockType (blockSpec nd) == "Paragraph"+ = Node nd{ blockSpec = plainSpec } cs+ | otherwise = Node nd cs+ let childrenToTight (Node nd cs) = Node nd (map totight cs)+ let children' =+ if ls == TightList+ then map childrenToTight children+ else children+ defaultFinalizer (Node cdata{ blockData = ldata'+ , blockBlanks = blockBlanks' } children')+ parent+ }++thematicBreakSpec :: (Monad m, IsBlock il bl)+ => BlockSpec m il bl+thematicBreakSpec = BlockSpec+ { blockType = "ThematicBreak"+ , blockStart = do+ nonindentSpaces+ pos <- getPosition+ Tok (Symbol c) _ _ <- symbol '-'+ <|> symbol '_'+ <|> symbol '*'+ skipWhile (hasType Spaces)+ let tbchar = symbol c <* skipWhile (hasType Spaces)+ count 2 tbchar+ skipMany tbchar+ (do lookAhead lineEnd+ addNodeToStack (Node (defBlockData thematicBreakSpec){+ blockStartPos = [pos] } [])+ return BlockStartMatch) <|>+ (BlockStartNoMatchBefore <$> getPosition)+ , blockCanContain = const False+ , blockContainsLines = False+ , blockParagraph = False+ , blockContinue = const mzero+ , blockConstructor = \_ -> return thematicBreak+ , blockFinalize = defaultFinalizer+ }++indentedCodeSpec :: (Monad m, IsBlock il bl)+ => BlockSpec m il bl+indentedCodeSpec = BlockSpec+ { blockType = "IndentedCode"+ , blockStart = do+ interruptsParagraph >>= guard . not+ getState >>= guard . not . maybeLazy+ _ <- gobbleSpaces 4+ pos <- getPosition+ notFollowedBy blankLine+ addNodeToStack $ Node (defBlockData indentedCodeSpec){+ blockStartPos = [pos] } []+ return BlockStartMatch+ , blockCanContain = const False+ , blockContainsLines = True+ , blockParagraph = False+ , blockContinue = \node -> do+ void (gobbleSpaces 4)+ <|> try (skipWhile (hasType Spaces) <* lookAhead lineEnd)+ pos <- getPosition+ return $! (pos, node)++ , blockConstructor = \node ->+ return $! codeBlock mempty (untokenize (getBlockText node))+ , blockFinalize = \(Node cdata children) parent -> do+ -- strip off blank lines at end:+ let blanks = takeWhile isblankLine $ blockLines cdata+ let numblanks = length blanks+ let cdata' = cdata{ blockLines =+ drop numblanks $ blockLines cdata+ , blockStartPos =+ drop numblanks $ blockStartPos cdata+ }+ defaultFinalizer (Node cdata' children) parent+ }++isblankLine :: [Tok] -> Bool+isblankLine [] = True+isblankLine [Tok LineEnd _ _] = True+isblankLine (Tok Spaces _ _ : xs) = isblankLine xs+isblankLine _ = False++fencedCodeSpec :: (Monad m, IsBlock il bl)+ => BlockSpec m il bl+fencedCodeSpec = BlockSpec+ { blockType = "FencedCode"+ , blockStart = do+ prepos <- getPosition+ nonindentSpaces+ pos <- getPosition+ let indentspaces = sourceColumn pos - sourceColumn prepos+ (c, ticks) <- (('`',) <$> many1 (symbol '`'))+ <|> (('~',) <$> many1 (symbol '~'))+ let fencelength = length ticks+ guard $ fencelength >= 3+ skipWhile (hasType Spaces)+ let infoTok = noneOfToks (LineEnd : [Symbol '`' | c == '`'])+ info <- T.strip . unEntity <$> many (pEscaped <|> infoTok)+ lookAhead $ void lineEnd <|> eof++ let infotoks = tokenize "info string" info+ (content, attrs) <- parseFinalAttributes False infotoks+ <|> (return $! (infotoks, mempty))+ addNodeToStack $+ Node (defBlockData fencedCodeSpec){+ blockData = toDyn+ (c, fencelength, indentspaces,+ untokenize content, attrs),+ blockStartPos = [pos] } []+ return BlockStartMatch+ , blockCanContain = const False+ , blockContainsLines = True+ , blockParagraph = False+ , blockContinue = \node -> try (do+ let ((c, fencelength, _, _, _)+ :: (Char, Int, Int, Text, Attributes)) = fromDyn+ (blockData (rootLabel node))+ ('`', 3, 0, mempty, mempty)+ nonindentSpaces+ pos <- getPosition+ ts <- many1 (symbol c)+ guard $ length ts >= fencelength+ skipWhile (hasType Spaces)+ lookAhead $ void lineEnd <|> eof+ endOfBlock+ return $! (pos, node))+ <|> (do let ((_, _, indentspaces, _, _)+ :: (Char, Int, Int, Text, Attributes)) = fromDyn+ (blockData (rootLabel node))+ ('`', 3, 0, mempty, mempty)+ pos <- getPosition+ _ <- gobbleUpToSpaces indentspaces+ return $! (pos, node))+ , blockConstructor = \node -> do+ let ((_, _, _, info, attrs) :: (Char, Int, Int, Text, Attributes)) =+ fromDyn (blockData (rootLabel node)) ('`', 3, 0, mempty, mempty)+ let codetext = untokenize $ drop 1 (getBlockText node)+ return $!+ if null attrs+ then codeBlock info codetext+ else addAttributes attrs $ codeBlock info codetext+ , blockFinalize = defaultFinalizer+ }++rawHtmlSpec :: (Monad m, IsBlock il bl)+ => BlockSpec m il bl+rawHtmlSpec = BlockSpec+ { blockType = "RawHTML"+ , blockStart = do+ pos <- getPosition+ (rawHtmlType, toks) <- withRaw $+ do nonindentSpaces+ symbol '<'+ ty <- choice $ map (\n -> n <$ startCond n) [1..7]+ -- some blocks can end on same line+ finished <- option False $ do+ guard (ty /= 6 && ty /= 7)+ endCond ty+ return True+ when (ty == 7) $ do+ -- type 7 blocks can't interrupt a paragraph+ (n:_) <- nodeStack <$> getState+ guard $ not $ blockParagraph (bspec n)+ skipWhile (not . hasType LineEnd)+ -- we use 0 as a code to indicate that the block is closed+ return $! if finished then 0 else ty+ addNodeToStack $ Node (defBlockData rawHtmlSpec){+ blockData = toDyn rawHtmlType,+ blockLines = [toks],+ blockStartPos = [pos] } []+ return BlockStartMatch+ , blockCanContain = const False+ , blockContainsLines = True+ , blockParagraph = False+ , blockContinue = \node@(Node ndata children) -> try $ do+ pos <- getPosition+ case fromDyn (blockData (rootLabel node)) (0 :: Int) of+ 0 -> mzero -- 0 means that the block start already closed+ 6 -> (pos, node) <$ notFollowedBy blankLine+ 7 -> (pos, node) <$ notFollowedBy blankLine+ n ->+ (do pos' <- getPosition+ lookAhead (endCond n)+ endOfBlock+ toks <- many (satisfyTok (not . hasType LineEnd))+ le <- option [] $ (:[]) <$> lookAhead lineEnd+ return $! (pos', Node ndata{+ blockData = toDyn (0 :: Int)+ , blockLines = (toks ++ le) : blockLines ndata+ } children)) <|> (return $! (pos, node))+ , blockConstructor = \node ->+ return $! rawBlock (Format "html")+ (untokenize (getBlockText node))+ , blockFinalize = defaultFinalizer+ }++---------------- for raw html:++startCond :: Monad m => Int -> BlockParser m il bl ()+startCond 1 = void $ try $ do+ satisfyWord (isOneOfCI ["script","pre","style"])+ spaceTok+ <|> symbol '>'+ <|> lookAhead lineEnd+startCond 2 = void $ try $ do+ symbol '!'+ symbol '-'+ symbol '-'+startCond 3 = void $ symbol '?'+startCond 4 = void $ try $ do+ symbol '!'+ satisfyWord (\t -> case T.uncons t of+ Just (c, _) -> isAsciiUpper c+ _ -> False)+startCond 5 = void $ try $ do+ symbol '!'+ symbol '['+ satisfyWord (== "CDATA")+ symbol '['+startCond 6 = void $ try $ do+ optional (symbol '/')+ satisfyWord (isOneOfCI ["address", "article", "aside", "base",+ "basefont", "blockquote", "body", "caption", "center", "col",+ "colgroup", "dd", "details", "dialog", "dir", "div", "dl",+ "dt", "fieldset", "figcaption", "figure", "footer", "form", "frame",+ "frameset", "h1", "h2", "h3", "h4", "h5", "h6", "head", "header",+ "hr", "html", "iframe", "legend", "li", "link", "main", "menu",+ "menuitem", "nav", "noframes", "ol", "optgroup", "option",+ "p", "param", "section", "source", "summary", "table", "tbody",+ "td", "tfoot", "th", "thead", "title", "tr", "track", "ul"])+ spaceTok+ <|> lookAhead lineEnd+ <|> symbol '>'+ <|> (symbol '/' >> symbol '>')+startCond 7 = void $ try $ do+ toks <- htmlOpenTag <|> htmlClosingTag+ guard $ not $ any (hasType LineEnd) toks+ skipWhile (hasType Spaces)+ lookAhead lineEnd+startCond n = fail $ "Unknown HTML block type " ++ show n++endCond :: Monad m => Int -> BlockParser m il bl ()+endCond 1 = try $ do+ let closer = try $ do+ symbol '<'+ symbol '/'+ satisfyWord (isOneOfCI ["script","pre","style"])+ symbol '>'+ skipManyTill (satisfyTok (not . hasType LineEnd)) closer+endCond 2 = try $ do+ let closer = try $ symbol '-' >> symbol '-' >> symbol '>'+ skipManyTill (satisfyTok (not . hasType LineEnd)) closer+endCond 3 = try $ do+ let closer = try $ symbol '?' >> symbol '>'+ skipManyTill (satisfyTok (not . hasType LineEnd)) closer+endCond 4 = try $+ skipManyTill (satisfyTok (not . hasType LineEnd)) (symbol '>')+endCond 5 = try $ do+ let closer = try $ symbol ']' >> symbol ']' >> symbol '>'+ skipManyTill (satisfyTok (not . hasType LineEnd)) closer+endCond 6 = void blankLine+endCond 7 = void blankLine+endCond n = fail $ "Unknown HTML block type " ++ show n++--------------------------------++getBlockText :: BlockNode m il bl -> [Tok]+getBlockText =+ concat . reverse . blockLines . rootLabel++removeIndent :: [Tok] -> [Tok]+removeIndent = dropWhile (hasType Spaces)++removeConsecutive :: [Int] -> [Int]+removeConsecutive (x:y:zs)+ | x == y + 1 = removeConsecutive (y:zs)+removeConsecutive xs = xs++-------------------------------------------------------------------------++collapseNodeStack :: [BlockNode m il bl] -> BlockParser m il bl (BlockNode m il bl)+collapseNodeStack [] = error "Empty node stack!" -- should not happen+collapseNodeStack (n:ns) = foldM go n ns+ where go child parent+ = if blockCanContain (bspec parent) (bspec child)+ then blockFinalize (bspec child) child parent+ else error $ "collapseNodeStack: " +++ T.unpack (blockType (bspec parent)) +++ " cannot contain " ++ T.unpack (blockType (bspec child))++bspec :: BlockNode m il bl -> BlockSpec m il bl+bspec = blockSpec . rootLabel++endOfBlock :: Monad m => BlockParser m il bl ()+endOfBlock = updateState $ \st -> st{ blockMatched = False }+
+ src/Commonmark/Entity.hs view
@@ -0,0 +1,2352 @@+{-# LANGUAGE CPP #-}+{-# LANGUAGE OverloadedStrings #-}+-- This code for lookupEntity is modified from Text.HTML.TagSoup.Entity+-- (C) 2006--2018 Neil Mitchell, released under the BSD-3 license++module Commonmark.Entity+ ( lookupEntity+ , charEntity+ , numEntity+ , unEntity+ )+where++import Data.Functor.Identity (Identity)+import qualified Data.Map.Strict as Map+import Commonmark.TokParsers+import Commonmark.Tokens+import Text.Parsec+import qualified Data.Text as T+import Data.Text (Text)+import qualified Data.Text.Read as TR+import Control.Monad (guard, mzero)+import Data.Char (isDigit, isHexDigit, chr)+import Data.Maybe (isJust)+#if !MIN_VERSION_base(4,11,0)+import Data.Semigroup (Semigroup(..))+#endif++-- | Lookup an entity, using 'lookupNumericEntity' if it starts with+-- @#@ and 'lookupNamedEntity' otherwise+lookupEntity :: Text -> Maybe Text+lookupEntity t =+ case T.uncons t of+ Just ('#', t') -> lookupNumericEntity t'+ _ -> lookupNamedEntity t+++-- | Lookup a numeric entity, the leading @\'#\'@ must have already been removed.+--+-- > lookupNumericEntity "65" == Just "A"+-- > lookupNumericEntity "x41" == Just "A"+-- > lookupNumericEntity "x4E" === Just "N"+-- > lookupNumericEntity "x4e" === Just "N"+-- > lookupNumericEntity "X4e" === Just "N"+-- > lookupNumericEntity "Haskell" == Nothing+-- > lookupNumericEntity "" == Nothing+-- > lookupNumericEntity "89439085908539082" == Nothing+lookupNumericEntity :: Text -> Maybe Text+lookupNumericEntity = f+ -- entity = '&#' [0-9]+ ';' | '&#x' [0-9a-fA-F]+ ';'+ where+ f t = case T.uncons t of+ Just ('x', t') -> g TR.hexadecimal t'+ Just ('X', t') -> g TR.hexadecimal t'+ _ -> g TR.decimal t++ g :: TR.Reader Integer -> Text -> Maybe Text+ g reader t =+ case reader t of+ Right (x,t')+ | T.null t' || t' == ";" ->+ if x < 1 || x > 0x10FFFF+ then Just "\xFFFD" -- illegal code point+ else Just $ T.singleton $ chr $ fromInteger x+ _ -> Nothing+++-- | Lookup a named entity, using 'htmlEntities'+--+-- > lookupNamedEntity "amp" == Just "&"+-- > lookupNamedEntity "haskell" == Nothing+lookupNamedEntity :: Text -> Maybe Text+lookupNamedEntity x = Map.lookup x htmlEntityMap++htmlEntityMap :: Map.Map Text Text+htmlEntityMap = Map.fromList htmlEntities++-- | A table mapping HTML entity names to resolved strings. Most resolved strings are a single character long,+-- but some (e.g. @"ngeqq"@) are two characters long. The list is taken from+-- <http://www.w3.org/TR/html5/syntax.html#named-character-references>.+htmlEntities :: [(Text, Text)]+htmlEntities =+ [("Aacute", "\x00C1")+ ,("aacute", "\x00E1")+ ,("Aacute;", "\x00C1")+ ,("aacute;", "\x00E1")+ ,("Abreve;", "\x0102")+ ,("abreve;", "\x0103")+ ,("ac;", "\x223E")+ ,("acd;", "\x223F")+ ,("acE;", "\x223E\x0333")+ ,("Acirc", "\x00C2")+ ,("acirc", "\x00E2")+ ,("Acirc;", "\x00C2")+ ,("acirc;", "\x00E2")+ ,("acute", "\x00B4")+ ,("acute;", "\x00B4")+ ,("Acy;", "\x0410")+ ,("acy;", "\x0430")+ ,("AElig", "\x00C6")+ ,("aelig", "\x00E6")+ ,("AElig;", "\x00C6")+ ,("aelig;", "\x00E6")+ ,("af;", "\x2061")+ ,("Afr;", "\x1D504")+ ,("afr;", "\x1D51E")+ ,("Agrave", "\x00C0")+ ,("agrave", "\x00E0")+ ,("Agrave;", "\x00C0")+ ,("agrave;", "\x00E0")+ ,("alefsym;", "\x2135")+ ,("aleph;", "\x2135")+ ,("Alpha;", "\x0391")+ ,("alpha;", "\x03B1")+ ,("Amacr;", "\x0100")+ ,("amacr;", "\x0101")+ ,("amalg;", "\x2A3F")+ ,("AMP", "\x0026")+ ,("amp", "\x0026")+ ,("AMP;", "\x0026")+ ,("amp;", "\x0026")+ ,("and;", "\x2227")+ ,("And;", "\x2A53")+ ,("andand;", "\x2A55")+ ,("andd;", "\x2A5C")+ ,("andslope;", "\x2A58")+ ,("andv;", "\x2A5A")+ ,("ang;", "\x2220")+ ,("ange;", "\x29A4")+ ,("angle;", "\x2220")+ ,("angmsd;", "\x2221")+ ,("angmsdaa;", "\x29A8")+ ,("angmsdab;", "\x29A9")+ ,("angmsdac;", "\x29AA")+ ,("angmsdad;", "\x29AB")+ ,("angmsdae;", "\x29AC")+ ,("angmsdaf;", "\x29AD")+ ,("angmsdag;", "\x29AE")+ ,("angmsdah;", "\x29AF")+ ,("angrt;", "\x221F")+ ,("angrtvb;", "\x22BE")+ ,("angrtvbd;", "\x299D")+ ,("angsph;", "\x2222")+ ,("angst;", "\x00C5")+ ,("angzarr;", "\x237C")+ ,("Aogon;", "\x0104")+ ,("aogon;", "\x0105")+ ,("Aopf;", "\x1D538")+ ,("aopf;", "\x1D552")+ ,("ap;", "\x2248")+ ,("apacir;", "\x2A6F")+ ,("ape;", "\x224A")+ ,("apE;", "\x2A70")+ ,("apid;", "\x224B")+ ,("apos;", "\x0027")+ ,("ApplyFunction;", "\x2061")+ ,("approx;", "\x2248")+ ,("approxeq;", "\x224A")+ ,("Aring", "\x00C5")+ ,("aring", "\x00E5")+ ,("Aring;", "\x00C5")+ ,("aring;", "\x00E5")+ ,("Ascr;", "\x1D49C")+ ,("ascr;", "\x1D4B6")+ ,("Assign;", "\x2254")+ ,("ast;", "\x002A")+ ,("asymp;", "\x2248")+ ,("asympeq;", "\x224D")+ ,("Atilde", "\x00C3")+ ,("atilde", "\x00E3")+ ,("Atilde;", "\x00C3")+ ,("atilde;", "\x00E3")+ ,("Auml", "\x00C4")+ ,("auml", "\x00E4")+ ,("Auml;", "\x00C4")+ ,("auml;", "\x00E4")+ ,("awconint;", "\x2233")+ ,("awint;", "\x2A11")+ ,("backcong;", "\x224C")+ ,("backepsilon;", "\x03F6")+ ,("backprime;", "\x2035")+ ,("backsim;", "\x223D")+ ,("backsimeq;", "\x22CD")+ ,("Backslash;", "\x2216")+ ,("Barv;", "\x2AE7")+ ,("barvee;", "\x22BD")+ ,("barwed;", "\x2305")+ ,("Barwed;", "\x2306")+ ,("barwedge;", "\x2305")+ ,("bbrk;", "\x23B5")+ ,("bbrktbrk;", "\x23B6")+ ,("bcong;", "\x224C")+ ,("Bcy;", "\x0411")+ ,("bcy;", "\x0431")+ ,("bdquo;", "\x201E")+ ,("becaus;", "\x2235")+ ,("Because;", "\x2235")+ ,("because;", "\x2235")+ ,("bemptyv;", "\x29B0")+ ,("bepsi;", "\x03F6")+ ,("bernou;", "\x212C")+ ,("Bernoullis;", "\x212C")+ ,("Beta;", "\x0392")+ ,("beta;", "\x03B2")+ ,("beth;", "\x2136")+ ,("between;", "\x226C")+ ,("Bfr;", "\x1D505")+ ,("bfr;", "\x1D51F")+ ,("bigcap;", "\x22C2")+ ,("bigcirc;", "\x25EF")+ ,("bigcup;", "\x22C3")+ ,("bigodot;", "\x2A00")+ ,("bigoplus;", "\x2A01")+ ,("bigotimes;", "\x2A02")+ ,("bigsqcup;", "\x2A06")+ ,("bigstar;", "\x2605")+ ,("bigtriangledown;", "\x25BD")+ ,("bigtriangleup;", "\x25B3")+ ,("biguplus;", "\x2A04")+ ,("bigvee;", "\x22C1")+ ,("bigwedge;", "\x22C0")+ ,("bkarow;", "\x290D")+ ,("blacklozenge;", "\x29EB")+ ,("blacksquare;", "\x25AA")+ ,("blacktriangle;", "\x25B4")+ ,("blacktriangledown;", "\x25BE")+ ,("blacktriangleleft;", "\x25C2")+ ,("blacktriangleright;", "\x25B8")+ ,("blank;", "\x2423")+ ,("blk12;", "\x2592")+ ,("blk14;", "\x2591")+ ,("blk34;", "\x2593")+ ,("block;", "\x2588")+ ,("bne;", "\x003D\x20E5")+ ,("bnequiv;", "\x2261\x20E5")+ ,("bnot;", "\x2310")+ ,("bNot;", "\x2AED")+ ,("Bopf;", "\x1D539")+ ,("bopf;", "\x1D553")+ ,("bot;", "\x22A5")+ ,("bottom;", "\x22A5")+ ,("bowtie;", "\x22C8")+ ,("boxbox;", "\x29C9")+ ,("boxdl;", "\x2510")+ ,("boxdL;", "\x2555")+ ,("boxDl;", "\x2556")+ ,("boxDL;", "\x2557")+ ,("boxdr;", "\x250C")+ ,("boxdR;", "\x2552")+ ,("boxDr;", "\x2553")+ ,("boxDR;", "\x2554")+ ,("boxh;", "\x2500")+ ,("boxH;", "\x2550")+ ,("boxhd;", "\x252C")+ ,("boxHd;", "\x2564")+ ,("boxhD;", "\x2565")+ ,("boxHD;", "\x2566")+ ,("boxhu;", "\x2534")+ ,("boxHu;", "\x2567")+ ,("boxhU;", "\x2568")+ ,("boxHU;", "\x2569")+ ,("boxminus;", "\x229F")+ ,("boxplus;", "\x229E")+ ,("boxtimes;", "\x22A0")+ ,("boxul;", "\x2518")+ ,("boxuL;", "\x255B")+ ,("boxUl;", "\x255C")+ ,("boxUL;", "\x255D")+ ,("boxur;", "\x2514")+ ,("boxuR;", "\x2558")+ ,("boxUr;", "\x2559")+ ,("boxUR;", "\x255A")+ ,("boxv;", "\x2502")+ ,("boxV;", "\x2551")+ ,("boxvh;", "\x253C")+ ,("boxvH;", "\x256A")+ ,("boxVh;", "\x256B")+ ,("boxVH;", "\x256C")+ ,("boxvl;", "\x2524")+ ,("boxvL;", "\x2561")+ ,("boxVl;", "\x2562")+ ,("boxVL;", "\x2563")+ ,("boxvr;", "\x251C")+ ,("boxvR;", "\x255E")+ ,("boxVr;", "\x255F")+ ,("boxVR;", "\x2560")+ ,("bprime;", "\x2035")+ ,("Breve;", "\x02D8")+ ,("breve;", "\x02D8")+ ,("brvbar", "\x00A6")+ ,("brvbar;", "\x00A6")+ ,("Bscr;", "\x212C")+ ,("bscr;", "\x1D4B7")+ ,("bsemi;", "\x204F")+ ,("bsim;", "\x223D")+ ,("bsime;", "\x22CD")+ ,("bsol;", "\x005C")+ ,("bsolb;", "\x29C5")+ ,("bsolhsub;", "\x27C8")+ ,("bull;", "\x2022")+ ,("bullet;", "\x2022")+ ,("bump;", "\x224E")+ ,("bumpe;", "\x224F")+ ,("bumpE;", "\x2AAE")+ ,("Bumpeq;", "\x224E")+ ,("bumpeq;", "\x224F")+ ,("Cacute;", "\x0106")+ ,("cacute;", "\x0107")+ ,("cap;", "\x2229")+ ,("Cap;", "\x22D2")+ ,("capand;", "\x2A44")+ ,("capbrcup;", "\x2A49")+ ,("capcap;", "\x2A4B")+ ,("capcup;", "\x2A47")+ ,("capdot;", "\x2A40")+ ,("CapitalDifferentialD;", "\x2145")+ ,("caps;", "\x2229\xFE00")+ ,("caret;", "\x2041")+ ,("caron;", "\x02C7")+ ,("Cayleys;", "\x212D")+ ,("ccaps;", "\x2A4D")+ ,("Ccaron;", "\x010C")+ ,("ccaron;", "\x010D")+ ,("Ccedil", "\x00C7")+ ,("ccedil", "\x00E7")+ ,("Ccedil;", "\x00C7")+ ,("ccedil;", "\x00E7")+ ,("Ccirc;", "\x0108")+ ,("ccirc;", "\x0109")+ ,("Cconint;", "\x2230")+ ,("ccups;", "\x2A4C")+ ,("ccupssm;", "\x2A50")+ ,("Cdot;", "\x010A")+ ,("cdot;", "\x010B")+ ,("cedil", "\x00B8")+ ,("cedil;", "\x00B8")+ ,("Cedilla;", "\x00B8")+ ,("cemptyv;", "\x29B2")+ ,("cent", "\x00A2")+ ,("cent;", "\x00A2")+ ,("CenterDot;", "\x00B7")+ ,("centerdot;", "\x00B7")+ ,("Cfr;", "\x212D")+ ,("cfr;", "\x1D520")+ ,("CHcy;", "\x0427")+ ,("chcy;", "\x0447")+ ,("check;", "\x2713")+ ,("checkmark;", "\x2713")+ ,("Chi;", "\x03A7")+ ,("chi;", "\x03C7")+ ,("cir;", "\x25CB")+ ,("circ;", "\x02C6")+ ,("circeq;", "\x2257")+ ,("circlearrowleft;", "\x21BA")+ ,("circlearrowright;", "\x21BB")+ ,("circledast;", "\x229B")+ ,("circledcirc;", "\x229A")+ ,("circleddash;", "\x229D")+ ,("CircleDot;", "\x2299")+ ,("circledR;", "\x00AE")+ ,("circledS;", "\x24C8")+ ,("CircleMinus;", "\x2296")+ ,("CirclePlus;", "\x2295")+ ,("CircleTimes;", "\x2297")+ ,("cire;", "\x2257")+ ,("cirE;", "\x29C3")+ ,("cirfnint;", "\x2A10")+ ,("cirmid;", "\x2AEF")+ ,("cirscir;", "\x29C2")+ ,("ClockwiseContourIntegral;", "\x2232")+ ,("CloseCurlyDoubleQuote;", "\x201D")+ ,("CloseCurlyQuote;", "\x2019")+ ,("clubs;", "\x2663")+ ,("clubsuit;", "\x2663")+ ,("colon;", "\x003A")+ ,("Colon;", "\x2237")+ ,("colone;", "\x2254")+ ,("Colone;", "\x2A74")+ ,("coloneq;", "\x2254")+ ,("comma;", "\x002C")+ ,("commat;", "\x0040")+ ,("comp;", "\x2201")+ ,("compfn;", "\x2218")+ ,("complement;", "\x2201")+ ,("complexes;", "\x2102")+ ,("cong;", "\x2245")+ ,("congdot;", "\x2A6D")+ ,("Congruent;", "\x2261")+ ,("conint;", "\x222E")+ ,("Conint;", "\x222F")+ ,("ContourIntegral;", "\x222E")+ ,("Copf;", "\x2102")+ ,("copf;", "\x1D554")+ ,("coprod;", "\x2210")+ ,("Coproduct;", "\x2210")+ ,("COPY", "\x00A9")+ ,("copy", "\x00A9")+ ,("COPY;", "\x00A9")+ ,("copy;", "\x00A9")+ ,("copysr;", "\x2117")+ ,("CounterClockwiseContourIntegral;", "\x2233")+ ,("crarr;", "\x21B5")+ ,("cross;", "\x2717")+ ,("Cross;", "\x2A2F")+ ,("Cscr;", "\x1D49E")+ ,("cscr;", "\x1D4B8")+ ,("csub;", "\x2ACF")+ ,("csube;", "\x2AD1")+ ,("csup;", "\x2AD0")+ ,("csupe;", "\x2AD2")+ ,("ctdot;", "\x22EF")+ ,("cudarrl;", "\x2938")+ ,("cudarrr;", "\x2935")+ ,("cuepr;", "\x22DE")+ ,("cuesc;", "\x22DF")+ ,("cularr;", "\x21B6")+ ,("cularrp;", "\x293D")+ ,("cup;", "\x222A")+ ,("Cup;", "\x22D3")+ ,("cupbrcap;", "\x2A48")+ ,("CupCap;", "\x224D")+ ,("cupcap;", "\x2A46")+ ,("cupcup;", "\x2A4A")+ ,("cupdot;", "\x228D")+ ,("cupor;", "\x2A45")+ ,("cups;", "\x222A\xFE00")+ ,("curarr;", "\x21B7")+ ,("curarrm;", "\x293C")+ ,("curlyeqprec;", "\x22DE")+ ,("curlyeqsucc;", "\x22DF")+ ,("curlyvee;", "\x22CE")+ ,("curlywedge;", "\x22CF")+ ,("curren", "\x00A4")+ ,("curren;", "\x00A4")+ ,("curvearrowleft;", "\x21B6")+ ,("curvearrowright;", "\x21B7")+ ,("cuvee;", "\x22CE")+ ,("cuwed;", "\x22CF")+ ,("cwconint;", "\x2232")+ ,("cwint;", "\x2231")+ ,("cylcty;", "\x232D")+ ,("dagger;", "\x2020")+ ,("Dagger;", "\x2021")+ ,("daleth;", "\x2138")+ ,("darr;", "\x2193")+ ,("Darr;", "\x21A1")+ ,("dArr;", "\x21D3")+ ,("dash;", "\x2010")+ ,("dashv;", "\x22A3")+ ,("Dashv;", "\x2AE4")+ ,("dbkarow;", "\x290F")+ ,("dblac;", "\x02DD")+ ,("Dcaron;", "\x010E")+ ,("dcaron;", "\x010F")+ ,("Dcy;", "\x0414")+ ,("dcy;", "\x0434")+ ,("DD;", "\x2145")+ ,("dd;", "\x2146")+ ,("ddagger;", "\x2021")+ ,("ddarr;", "\x21CA")+ ,("DDotrahd;", "\x2911")+ ,("ddotseq;", "\x2A77")+ ,("deg", "\x00B0")+ ,("deg;", "\x00B0")+ ,("Del;", "\x2207")+ ,("Delta;", "\x0394")+ ,("delta;", "\x03B4")+ ,("demptyv;", "\x29B1")+ ,("dfisht;", "\x297F")+ ,("Dfr;", "\x1D507")+ ,("dfr;", "\x1D521")+ ,("dHar;", "\x2965")+ ,("dharl;", "\x21C3")+ ,("dharr;", "\x21C2")+ ,("DiacriticalAcute;", "\x00B4")+ ,("DiacriticalDot;", "\x02D9")+ ,("DiacriticalDoubleAcute;", "\x02DD")+ ,("DiacriticalGrave;", "\x0060")+ ,("DiacriticalTilde;", "\x02DC")+ ,("diam;", "\x22C4")+ ,("Diamond;", "\x22C4")+ ,("diamond;", "\x22C4")+ ,("diamondsuit;", "\x2666")+ ,("diams;", "\x2666")+ ,("die;", "\x00A8")+ ,("DifferentialD;", "\x2146")+ ,("digamma;", "\x03DD")+ ,("disin;", "\x22F2")+ ,("div;", "\x00F7")+ ,("divide", "\x00F7")+ ,("divide;", "\x00F7")+ ,("divideontimes;", "\x22C7")+ ,("divonx;", "\x22C7")+ ,("DJcy;", "\x0402")+ ,("djcy;", "\x0452")+ ,("dlcorn;", "\x231E")+ ,("dlcrop;", "\x230D")+ ,("dollar;", "\x0024")+ ,("Dopf;", "\x1D53B")+ ,("dopf;", "\x1D555")+ ,("Dot;", "\x00A8")+ ,("dot;", "\x02D9")+ ,("DotDot;", "\x20DC")+ ,("doteq;", "\x2250")+ ,("doteqdot;", "\x2251")+ ,("DotEqual;", "\x2250")+ ,("dotminus;", "\x2238")+ ,("dotplus;", "\x2214")+ ,("dotsquare;", "\x22A1")+ ,("doublebarwedge;", "\x2306")+ ,("DoubleContourIntegral;", "\x222F")+ ,("DoubleDot;", "\x00A8")+ ,("DoubleDownArrow;", "\x21D3")+ ,("DoubleLeftArrow;", "\x21D0")+ ,("DoubleLeftRightArrow;", "\x21D4")+ ,("DoubleLeftTee;", "\x2AE4")+ ,("DoubleLongLeftArrow;", "\x27F8")+ ,("DoubleLongLeftRightArrow;", "\x27FA")+ ,("DoubleLongRightArrow;", "\x27F9")+ ,("DoubleRightArrow;", "\x21D2")+ ,("DoubleRightTee;", "\x22A8")+ ,("DoubleUpArrow;", "\x21D1")+ ,("DoubleUpDownArrow;", "\x21D5")+ ,("DoubleVerticalBar;", "\x2225")+ ,("DownArrow;", "\x2193")+ ,("downarrow;", "\x2193")+ ,("Downarrow;", "\x21D3")+ ,("DownArrowBar;", "\x2913")+ ,("DownArrowUpArrow;", "\x21F5")+ ,("DownBreve;", "\x0311")+ ,("downdownarrows;", "\x21CA")+ ,("downharpoonleft;", "\x21C3")+ ,("downharpoonright;", "\x21C2")+ ,("DownLeftRightVector;", "\x2950")+ ,("DownLeftTeeVector;", "\x295E")+ ,("DownLeftVector;", "\x21BD")+ ,("DownLeftVectorBar;", "\x2956")+ ,("DownRightTeeVector;", "\x295F")+ ,("DownRightVector;", "\x21C1")+ ,("DownRightVectorBar;", "\x2957")+ ,("DownTee;", "\x22A4")+ ,("DownTeeArrow;", "\x21A7")+ ,("drbkarow;", "\x2910")+ ,("drcorn;", "\x231F")+ ,("drcrop;", "\x230C")+ ,("Dscr;", "\x1D49F")+ ,("dscr;", "\x1D4B9")+ ,("DScy;", "\x0405")+ ,("dscy;", "\x0455")+ ,("dsol;", "\x29F6")+ ,("Dstrok;", "\x0110")+ ,("dstrok;", "\x0111")+ ,("dtdot;", "\x22F1")+ ,("dtri;", "\x25BF")+ ,("dtrif;", "\x25BE")+ ,("duarr;", "\x21F5")+ ,("duhar;", "\x296F")+ ,("dwangle;", "\x29A6")+ ,("DZcy;", "\x040F")+ ,("dzcy;", "\x045F")+ ,("dzigrarr;", "\x27FF")+ ,("Eacute", "\x00C9")+ ,("eacute", "\x00E9")+ ,("Eacute;", "\x00C9")+ ,("eacute;", "\x00E9")+ ,("easter;", "\x2A6E")+ ,("Ecaron;", "\x011A")+ ,("ecaron;", "\x011B")+ ,("ecir;", "\x2256")+ ,("Ecirc", "\x00CA")+ ,("ecirc", "\x00EA")+ ,("Ecirc;", "\x00CA")+ ,("ecirc;", "\x00EA")+ ,("ecolon;", "\x2255")+ ,("Ecy;", "\x042D")+ ,("ecy;", "\x044D")+ ,("eDDot;", "\x2A77")+ ,("Edot;", "\x0116")+ ,("edot;", "\x0117")+ ,("eDot;", "\x2251")+ ,("ee;", "\x2147")+ ,("efDot;", "\x2252")+ ,("Efr;", "\x1D508")+ ,("efr;", "\x1D522")+ ,("eg;", "\x2A9A")+ ,("Egrave", "\x00C8")+ ,("egrave", "\x00E8")+ ,("Egrave;", "\x00C8")+ ,("egrave;", "\x00E8")+ ,("egs;", "\x2A96")+ ,("egsdot;", "\x2A98")+ ,("el;", "\x2A99")+ ,("Element;", "\x2208")+ ,("elinters;", "\x23E7")+ ,("ell;", "\x2113")+ ,("els;", "\x2A95")+ ,("elsdot;", "\x2A97")+ ,("Emacr;", "\x0112")+ ,("emacr;", "\x0113")+ ,("empty;", "\x2205")+ ,("emptyset;", "\x2205")+ ,("EmptySmallSquare;", "\x25FB")+ ,("emptyv;", "\x2205")+ ,("EmptyVerySmallSquare;", "\x25AB")+ ,("emsp13;", "\x2004")+ ,("emsp14;", "\x2005")+ ,("emsp;", "\x2003")+ ,("ENG;", "\x014A")+ ,("eng;", "\x014B")+ ,("ensp;", "\x2002")+ ,("Eogon;", "\x0118")+ ,("eogon;", "\x0119")+ ,("Eopf;", "\x1D53C")+ ,("eopf;", "\x1D556")+ ,("epar;", "\x22D5")+ ,("eparsl;", "\x29E3")+ ,("eplus;", "\x2A71")+ ,("epsi;", "\x03B5")+ ,("Epsilon;", "\x0395")+ ,("epsilon;", "\x03B5")+ ,("epsiv;", "\x03F5")+ ,("eqcirc;", "\x2256")+ ,("eqcolon;", "\x2255")+ ,("eqsim;", "\x2242")+ ,("eqslantgtr;", "\x2A96")+ ,("eqslantless;", "\x2A95")+ ,("Equal;", "\x2A75")+ ,("equals;", "\x003D")+ ,("EqualTilde;", "\x2242")+ ,("equest;", "\x225F")+ ,("Equilibrium;", "\x21CC")+ ,("equiv;", "\x2261")+ ,("equivDD;", "\x2A78")+ ,("eqvparsl;", "\x29E5")+ ,("erarr;", "\x2971")+ ,("erDot;", "\x2253")+ ,("escr;", "\x212F")+ ,("Escr;", "\x2130")+ ,("esdot;", "\x2250")+ ,("esim;", "\x2242")+ ,("Esim;", "\x2A73")+ ,("Eta;", "\x0397")+ ,("eta;", "\x03B7")+ ,("ETH", "\x00D0")+ ,("eth", "\x00F0")+ ,("ETH;", "\x00D0")+ ,("eth;", "\x00F0")+ ,("Euml", "\x00CB")+ ,("euml", "\x00EB")+ ,("Euml;", "\x00CB")+ ,("euml;", "\x00EB")+ ,("euro;", "\x20AC")+ ,("excl;", "\x0021")+ ,("exist;", "\x2203")+ ,("Exists;", "\x2203")+ ,("expectation;", "\x2130")+ ,("ExponentialE;", "\x2147")+ ,("exponentiale;", "\x2147")+ ,("fallingdotseq;", "\x2252")+ ,("Fcy;", "\x0424")+ ,("fcy;", "\x0444")+ ,("female;", "\x2640")+ ,("ffilig;", "\xFB03")+ ,("fflig;", "\xFB00")+ ,("ffllig;", "\xFB04")+ ,("Ffr;", "\x1D509")+ ,("ffr;", "\x1D523")+ ,("filig;", "\xFB01")+ ,("FilledSmallSquare;", "\x25FC")+ ,("FilledVerySmallSquare;", "\x25AA")+ ,("fjlig;", "\x0066\x006A")+ ,("flat;", "\x266D")+ ,("fllig;", "\xFB02")+ ,("fltns;", "\x25B1")+ ,("fnof;", "\x0192")+ ,("Fopf;", "\x1D53D")+ ,("fopf;", "\x1D557")+ ,("ForAll;", "\x2200")+ ,("forall;", "\x2200")+ ,("fork;", "\x22D4")+ ,("forkv;", "\x2AD9")+ ,("Fouriertrf;", "\x2131")+ ,("fpartint;", "\x2A0D")+ ,("frac12", "\x00BD")+ ,("frac12;", "\x00BD")+ ,("frac13;", "\x2153")+ ,("frac14", "\x00BC")+ ,("frac14;", "\x00BC")+ ,("frac15;", "\x2155")+ ,("frac16;", "\x2159")+ ,("frac18;", "\x215B")+ ,("frac23;", "\x2154")+ ,("frac25;", "\x2156")+ ,("frac34", "\x00BE")+ ,("frac34;", "\x00BE")+ ,("frac35;", "\x2157")+ ,("frac38;", "\x215C")+ ,("frac45;", "\x2158")+ ,("frac56;", "\x215A")+ ,("frac58;", "\x215D")+ ,("frac78;", "\x215E")+ ,("frasl;", "\x2044")+ ,("frown;", "\x2322")+ ,("Fscr;", "\x2131")+ ,("fscr;", "\x1D4BB")+ ,("gacute;", "\x01F5")+ ,("Gamma;", "\x0393")+ ,("gamma;", "\x03B3")+ ,("Gammad;", "\x03DC")+ ,("gammad;", "\x03DD")+ ,("gap;", "\x2A86")+ ,("Gbreve;", "\x011E")+ ,("gbreve;", "\x011F")+ ,("Gcedil;", "\x0122")+ ,("Gcirc;", "\x011C")+ ,("gcirc;", "\x011D")+ ,("Gcy;", "\x0413")+ ,("gcy;", "\x0433")+ ,("Gdot;", "\x0120")+ ,("gdot;", "\x0121")+ ,("ge;", "\x2265")+ ,("gE;", "\x2267")+ ,("gel;", "\x22DB")+ ,("gEl;", "\x2A8C")+ ,("geq;", "\x2265")+ ,("geqq;", "\x2267")+ ,("geqslant;", "\x2A7E")+ ,("ges;", "\x2A7E")+ ,("gescc;", "\x2AA9")+ ,("gesdot;", "\x2A80")+ ,("gesdoto;", "\x2A82")+ ,("gesdotol;", "\x2A84")+ ,("gesl;", "\x22DB\xFE00")+ ,("gesles;", "\x2A94")+ ,("Gfr;", "\x1D50A")+ ,("gfr;", "\x1D524")+ ,("gg;", "\x226B")+ ,("Gg;", "\x22D9")+ ,("ggg;", "\x22D9")+ ,("gimel;", "\x2137")+ ,("GJcy;", "\x0403")+ ,("gjcy;", "\x0453")+ ,("gl;", "\x2277")+ ,("gla;", "\x2AA5")+ ,("glE;", "\x2A92")+ ,("glj;", "\x2AA4")+ ,("gnap;", "\x2A8A")+ ,("gnapprox;", "\x2A8A")+ ,("gnE;", "\x2269")+ ,("gne;", "\x2A88")+ ,("gneq;", "\x2A88")+ ,("gneqq;", "\x2269")+ ,("gnsim;", "\x22E7")+ ,("Gopf;", "\x1D53E")+ ,("gopf;", "\x1D558")+ ,("grave;", "\x0060")+ ,("GreaterEqual;", "\x2265")+ ,("GreaterEqualLess;", "\x22DB")+ ,("GreaterFullEqual;", "\x2267")+ ,("GreaterGreater;", "\x2AA2")+ ,("GreaterLess;", "\x2277")+ ,("GreaterSlantEqual;", "\x2A7E")+ ,("GreaterTilde;", "\x2273")+ ,("gscr;", "\x210A")+ ,("Gscr;", "\x1D4A2")+ ,("gsim;", "\x2273")+ ,("gsime;", "\x2A8E")+ ,("gsiml;", "\x2A90")+ ,("GT", "\x003E")+ ,("gt", "\x003E")+ ,("GT;", "\x003E")+ ,("gt;", "\x003E")+ ,("Gt;", "\x226B")+ ,("gtcc;", "\x2AA7")+ ,("gtcir;", "\x2A7A")+ ,("gtdot;", "\x22D7")+ ,("gtlPar;", "\x2995")+ ,("gtquest;", "\x2A7C")+ ,("gtrapprox;", "\x2A86")+ ,("gtrarr;", "\x2978")+ ,("gtrdot;", "\x22D7")+ ,("gtreqless;", "\x22DB")+ ,("gtreqqless;", "\x2A8C")+ ,("gtrless;", "\x2277")+ ,("gtrsim;", "\x2273")+ ,("gvertneqq;", "\x2269\xFE00")+ ,("gvnE;", "\x2269\xFE00")+ ,("Hacek;", "\x02C7")+ ,("hairsp;", "\x200A")+ ,("half;", "\x00BD")+ ,("hamilt;", "\x210B")+ ,("HARDcy;", "\x042A")+ ,("hardcy;", "\x044A")+ ,("harr;", "\x2194")+ ,("hArr;", "\x21D4")+ ,("harrcir;", "\x2948")+ ,("harrw;", "\x21AD")+ ,("Hat;", "\x005E")+ ,("hbar;", "\x210F")+ ,("Hcirc;", "\x0124")+ ,("hcirc;", "\x0125")+ ,("hearts;", "\x2665")+ ,("heartsuit;", "\x2665")+ ,("hellip;", "\x2026")+ ,("hercon;", "\x22B9")+ ,("Hfr;", "\x210C")+ ,("hfr;", "\x1D525")+ ,("HilbertSpace;", "\x210B")+ ,("hksearow;", "\x2925")+ ,("hkswarow;", "\x2926")+ ,("hoarr;", "\x21FF")+ ,("homtht;", "\x223B")+ ,("hookleftarrow;", "\x21A9")+ ,("hookrightarrow;", "\x21AA")+ ,("Hopf;", "\x210D")+ ,("hopf;", "\x1D559")+ ,("horbar;", "\x2015")+ ,("HorizontalLine;", "\x2500")+ ,("Hscr;", "\x210B")+ ,("hscr;", "\x1D4BD")+ ,("hslash;", "\x210F")+ ,("Hstrok;", "\x0126")+ ,("hstrok;", "\x0127")+ ,("HumpDownHump;", "\x224E")+ ,("HumpEqual;", "\x224F")+ ,("hybull;", "\x2043")+ ,("hyphen;", "\x2010")+ ,("Iacute", "\x00CD")+ ,("iacute", "\x00ED")+ ,("Iacute;", "\x00CD")+ ,("iacute;", "\x00ED")+ ,("ic;", "\x2063")+ ,("Icirc", "\x00CE")+ ,("icirc", "\x00EE")+ ,("Icirc;", "\x00CE")+ ,("icirc;", "\x00EE")+ ,("Icy;", "\x0418")+ ,("icy;", "\x0438")+ ,("Idot;", "\x0130")+ ,("IEcy;", "\x0415")+ ,("iecy;", "\x0435")+ ,("iexcl", "\x00A1")+ ,("iexcl;", "\x00A1")+ ,("iff;", "\x21D4")+ ,("Ifr;", "\x2111")+ ,("ifr;", "\x1D526")+ ,("Igrave", "\x00CC")+ ,("igrave", "\x00EC")+ ,("Igrave;", "\x00CC")+ ,("igrave;", "\x00EC")+ ,("ii;", "\x2148")+ ,("iiiint;", "\x2A0C")+ ,("iiint;", "\x222D")+ ,("iinfin;", "\x29DC")+ ,("iiota;", "\x2129")+ ,("IJlig;", "\x0132")+ ,("ijlig;", "\x0133")+ ,("Im;", "\x2111")+ ,("Imacr;", "\x012A")+ ,("imacr;", "\x012B")+ ,("image;", "\x2111")+ ,("ImaginaryI;", "\x2148")+ ,("imagline;", "\x2110")+ ,("imagpart;", "\x2111")+ ,("imath;", "\x0131")+ ,("imof;", "\x22B7")+ ,("imped;", "\x01B5")+ ,("Implies;", "\x21D2")+ ,("in;", "\x2208")+ ,("incare;", "\x2105")+ ,("infin;", "\x221E")+ ,("infintie;", "\x29DD")+ ,("inodot;", "\x0131")+ ,("int;", "\x222B")+ ,("Int;", "\x222C")+ ,("intcal;", "\x22BA")+ ,("integers;", "\x2124")+ ,("Integral;", "\x222B")+ ,("intercal;", "\x22BA")+ ,("Intersection;", "\x22C2")+ ,("intlarhk;", "\x2A17")+ ,("intprod;", "\x2A3C")+ ,("InvisibleComma;", "\x2063")+ ,("InvisibleTimes;", "\x2062")+ ,("IOcy;", "\x0401")+ ,("iocy;", "\x0451")+ ,("Iogon;", "\x012E")+ ,("iogon;", "\x012F")+ ,("Iopf;", "\x1D540")+ ,("iopf;", "\x1D55A")+ ,("Iota;", "\x0399")+ ,("iota;", "\x03B9")+ ,("iprod;", "\x2A3C")+ ,("iquest", "\x00BF")+ ,("iquest;", "\x00BF")+ ,("Iscr;", "\x2110")+ ,("iscr;", "\x1D4BE")+ ,("isin;", "\x2208")+ ,("isindot;", "\x22F5")+ ,("isinE;", "\x22F9")+ ,("isins;", "\x22F4")+ ,("isinsv;", "\x22F3")+ ,("isinv;", "\x2208")+ ,("it;", "\x2062")+ ,("Itilde;", "\x0128")+ ,("itilde;", "\x0129")+ ,("Iukcy;", "\x0406")+ ,("iukcy;", "\x0456")+ ,("Iuml", "\x00CF")+ ,("iuml", "\x00EF")+ ,("Iuml;", "\x00CF")+ ,("iuml;", "\x00EF")+ ,("Jcirc;", "\x0134")+ ,("jcirc;", "\x0135")+ ,("Jcy;", "\x0419")+ ,("jcy;", "\x0439")+ ,("Jfr;", "\x1D50D")+ ,("jfr;", "\x1D527")+ ,("jmath;", "\x0237")+ ,("Jopf;", "\x1D541")+ ,("jopf;", "\x1D55B")+ ,("Jscr;", "\x1D4A5")+ ,("jscr;", "\x1D4BF")+ ,("Jsercy;", "\x0408")+ ,("jsercy;", "\x0458")+ ,("Jukcy;", "\x0404")+ ,("jukcy;", "\x0454")+ ,("Kappa;", "\x039A")+ ,("kappa;", "\x03BA")+ ,("kappav;", "\x03F0")+ ,("Kcedil;", "\x0136")+ ,("kcedil;", "\x0137")+ ,("Kcy;", "\x041A")+ ,("kcy;", "\x043A")+ ,("Kfr;", "\x1D50E")+ ,("kfr;", "\x1D528")+ ,("kgreen;", "\x0138")+ ,("KHcy;", "\x0425")+ ,("khcy;", "\x0445")+ ,("KJcy;", "\x040C")+ ,("kjcy;", "\x045C")+ ,("Kopf;", "\x1D542")+ ,("kopf;", "\x1D55C")+ ,("Kscr;", "\x1D4A6")+ ,("kscr;", "\x1D4C0")+ ,("lAarr;", "\x21DA")+ ,("Lacute;", "\x0139")+ ,("lacute;", "\x013A")+ ,("laemptyv;", "\x29B4")+ ,("lagran;", "\x2112")+ ,("Lambda;", "\x039B")+ ,("lambda;", "\x03BB")+ ,("lang;", "\x27E8")+ ,("Lang;", "\x27EA")+ ,("langd;", "\x2991")+ ,("langle;", "\x27E8")+ ,("lap;", "\x2A85")+ ,("Laplacetrf;", "\x2112")+ ,("laquo", "\x00AB")+ ,("laquo;", "\x00AB")+ ,("larr;", "\x2190")+ ,("Larr;", "\x219E")+ ,("lArr;", "\x21D0")+ ,("larrb;", "\x21E4")+ ,("larrbfs;", "\x291F")+ ,("larrfs;", "\x291D")+ ,("larrhk;", "\x21A9")+ ,("larrlp;", "\x21AB")+ ,("larrpl;", "\x2939")+ ,("larrsim;", "\x2973")+ ,("larrtl;", "\x21A2")+ ,("lat;", "\x2AAB")+ ,("latail;", "\x2919")+ ,("lAtail;", "\x291B")+ ,("late;", "\x2AAD")+ ,("lates;", "\x2AAD\xFE00")+ ,("lbarr;", "\x290C")+ ,("lBarr;", "\x290E")+ ,("lbbrk;", "\x2772")+ ,("lbrace;", "\x007B")+ ,("lbrack;", "\x005B")+ ,("lbrke;", "\x298B")+ ,("lbrksld;", "\x298F")+ ,("lbrkslu;", "\x298D")+ ,("Lcaron;", "\x013D")+ ,("lcaron;", "\x013E")+ ,("Lcedil;", "\x013B")+ ,("lcedil;", "\x013C")+ ,("lceil;", "\x2308")+ ,("lcub;", "\x007B")+ ,("Lcy;", "\x041B")+ ,("lcy;", "\x043B")+ ,("ldca;", "\x2936")+ ,("ldquo;", "\x201C")+ ,("ldquor;", "\x201E")+ ,("ldrdhar;", "\x2967")+ ,("ldrushar;", "\x294B")+ ,("ldsh;", "\x21B2")+ ,("le;", "\x2264")+ ,("lE;", "\x2266")+ ,("LeftAngleBracket;", "\x27E8")+ ,("LeftArrow;", "\x2190")+ ,("leftarrow;", "\x2190")+ ,("Leftarrow;", "\x21D0")+ ,("LeftArrowBar;", "\x21E4")+ ,("LeftArrowRightArrow;", "\x21C6")+ ,("leftarrowtail;", "\x21A2")+ ,("LeftCeiling;", "\x2308")+ ,("LeftDoubleBracket;", "\x27E6")+ ,("LeftDownTeeVector;", "\x2961")+ ,("LeftDownVector;", "\x21C3")+ ,("LeftDownVectorBar;", "\x2959")+ ,("LeftFloor;", "\x230A")+ ,("leftharpoondown;", "\x21BD")+ ,("leftharpoonup;", "\x21BC")+ ,("leftleftarrows;", "\x21C7")+ ,("LeftRightArrow;", "\x2194")+ ,("leftrightarrow;", "\x2194")+ ,("Leftrightarrow;", "\x21D4")+ ,("leftrightarrows;", "\x21C6")+ ,("leftrightharpoons;", "\x21CB")+ ,("leftrightsquigarrow;", "\x21AD")+ ,("LeftRightVector;", "\x294E")+ ,("LeftTee;", "\x22A3")+ ,("LeftTeeArrow;", "\x21A4")+ ,("LeftTeeVector;", "\x295A")+ ,("leftthreetimes;", "\x22CB")+ ,("LeftTriangle;", "\x22B2")+ ,("LeftTriangleBar;", "\x29CF")+ ,("LeftTriangleEqual;", "\x22B4")+ ,("LeftUpDownVector;", "\x2951")+ ,("LeftUpTeeVector;", "\x2960")+ ,("LeftUpVector;", "\x21BF")+ ,("LeftUpVectorBar;", "\x2958")+ ,("LeftVector;", "\x21BC")+ ,("LeftVectorBar;", "\x2952")+ ,("leg;", "\x22DA")+ ,("lEg;", "\x2A8B")+ ,("leq;", "\x2264")+ ,("leqq;", "\x2266")+ ,("leqslant;", "\x2A7D")+ ,("les;", "\x2A7D")+ ,("lescc;", "\x2AA8")+ ,("lesdot;", "\x2A7F")+ ,("lesdoto;", "\x2A81")+ ,("lesdotor;", "\x2A83")+ ,("lesg;", "\x22DA\xFE00")+ ,("lesges;", "\x2A93")+ ,("lessapprox;", "\x2A85")+ ,("lessdot;", "\x22D6")+ ,("lesseqgtr;", "\x22DA")+ ,("lesseqqgtr;", "\x2A8B")+ ,("LessEqualGreater;", "\x22DA")+ ,("LessFullEqual;", "\x2266")+ ,("LessGreater;", "\x2276")+ ,("lessgtr;", "\x2276")+ ,("LessLess;", "\x2AA1")+ ,("lesssim;", "\x2272")+ ,("LessSlantEqual;", "\x2A7D")+ ,("LessTilde;", "\x2272")+ ,("lfisht;", "\x297C")+ ,("lfloor;", "\x230A")+ ,("Lfr;", "\x1D50F")+ ,("lfr;", "\x1D529")+ ,("lg;", "\x2276")+ ,("lgE;", "\x2A91")+ ,("lHar;", "\x2962")+ ,("lhard;", "\x21BD")+ ,("lharu;", "\x21BC")+ ,("lharul;", "\x296A")+ ,("lhblk;", "\x2584")+ ,("LJcy;", "\x0409")+ ,("ljcy;", "\x0459")+ ,("ll;", "\x226A")+ ,("Ll;", "\x22D8")+ ,("llarr;", "\x21C7")+ ,("llcorner;", "\x231E")+ ,("Lleftarrow;", "\x21DA")+ ,("llhard;", "\x296B")+ ,("lltri;", "\x25FA")+ ,("Lmidot;", "\x013F")+ ,("lmidot;", "\x0140")+ ,("lmoust;", "\x23B0")+ ,("lmoustache;", "\x23B0")+ ,("lnap;", "\x2A89")+ ,("lnapprox;", "\x2A89")+ ,("lnE;", "\x2268")+ ,("lne;", "\x2A87")+ ,("lneq;", "\x2A87")+ ,("lneqq;", "\x2268")+ ,("lnsim;", "\x22E6")+ ,("loang;", "\x27EC")+ ,("loarr;", "\x21FD")+ ,("lobrk;", "\x27E6")+ ,("LongLeftArrow;", "\x27F5")+ ,("longleftarrow;", "\x27F5")+ ,("Longleftarrow;", "\x27F8")+ ,("LongLeftRightArrow;", "\x27F7")+ ,("longleftrightarrow;", "\x27F7")+ ,("Longleftrightarrow;", "\x27FA")+ ,("longmapsto;", "\x27FC")+ ,("LongRightArrow;", "\x27F6")+ ,("longrightarrow;", "\x27F6")+ ,("Longrightarrow;", "\x27F9")+ ,("looparrowleft;", "\x21AB")+ ,("looparrowright;", "\x21AC")+ ,("lopar;", "\x2985")+ ,("Lopf;", "\x1D543")+ ,("lopf;", "\x1D55D")+ ,("loplus;", "\x2A2D")+ ,("lotimes;", "\x2A34")+ ,("lowast;", "\x2217")+ ,("lowbar;", "\x005F")+ ,("LowerLeftArrow;", "\x2199")+ ,("LowerRightArrow;", "\x2198")+ ,("loz;", "\x25CA")+ ,("lozenge;", "\x25CA")+ ,("lozf;", "\x29EB")+ ,("lpar;", "\x0028")+ ,("lparlt;", "\x2993")+ ,("lrarr;", "\x21C6")+ ,("lrcorner;", "\x231F")+ ,("lrhar;", "\x21CB")+ ,("lrhard;", "\x296D")+ ,("lrm;", "\x200E")+ ,("lrtri;", "\x22BF")+ ,("lsaquo;", "\x2039")+ ,("Lscr;", "\x2112")+ ,("lscr;", "\x1D4C1")+ ,("Lsh;", "\x21B0")+ ,("lsh;", "\x21B0")+ ,("lsim;", "\x2272")+ ,("lsime;", "\x2A8D")+ ,("lsimg;", "\x2A8F")+ ,("lsqb;", "\x005B")+ ,("lsquo;", "\x2018")+ ,("lsquor;", "\x201A")+ ,("Lstrok;", "\x0141")+ ,("lstrok;", "\x0142")+ ,("LT", "\x003C")+ ,("lt", "\x003C")+ ,("LT;", "\x003C")+ ,("lt;", "\x003C")+ ,("Lt;", "\x226A")+ ,("ltcc;", "\x2AA6")+ ,("ltcir;", "\x2A79")+ ,("ltdot;", "\x22D6")+ ,("lthree;", "\x22CB")+ ,("ltimes;", "\x22C9")+ ,("ltlarr;", "\x2976")+ ,("ltquest;", "\x2A7B")+ ,("ltri;", "\x25C3")+ ,("ltrie;", "\x22B4")+ ,("ltrif;", "\x25C2")+ ,("ltrPar;", "\x2996")+ ,("lurdshar;", "\x294A")+ ,("luruhar;", "\x2966")+ ,("lvertneqq;", "\x2268\xFE00")+ ,("lvnE;", "\x2268\xFE00")+ ,("macr", "\x00AF")+ ,("macr;", "\x00AF")+ ,("male;", "\x2642")+ ,("malt;", "\x2720")+ ,("maltese;", "\x2720")+ ,("map;", "\x21A6")+ ,("Map;", "\x2905")+ ,("mapsto;", "\x21A6")+ ,("mapstodown;", "\x21A7")+ ,("mapstoleft;", "\x21A4")+ ,("mapstoup;", "\x21A5")+ ,("marker;", "\x25AE")+ ,("mcomma;", "\x2A29")+ ,("Mcy;", "\x041C")+ ,("mcy;", "\x043C")+ ,("mdash;", "\x2014")+ ,("mDDot;", "\x223A")+ ,("measuredangle;", "\x2221")+ ,("MediumSpace;", "\x205F")+ ,("Mellintrf;", "\x2133")+ ,("Mfr;", "\x1D510")+ ,("mfr;", "\x1D52A")+ ,("mho;", "\x2127")+ ,("micro", "\x00B5")+ ,("micro;", "\x00B5")+ ,("mid;", "\x2223")+ ,("midast;", "\x002A")+ ,("midcir;", "\x2AF0")+ ,("middot", "\x00B7")+ ,("middot;", "\x00B7")+ ,("minus;", "\x2212")+ ,("minusb;", "\x229F")+ ,("minusd;", "\x2238")+ ,("minusdu;", "\x2A2A")+ ,("MinusPlus;", "\x2213")+ ,("mlcp;", "\x2ADB")+ ,("mldr;", "\x2026")+ ,("mnplus;", "\x2213")+ ,("models;", "\x22A7")+ ,("Mopf;", "\x1D544")+ ,("mopf;", "\x1D55E")+ ,("mp;", "\x2213")+ ,("Mscr;", "\x2133")+ ,("mscr;", "\x1D4C2")+ ,("mstpos;", "\x223E")+ ,("Mu;", "\x039C")+ ,("mu;", "\x03BC")+ ,("multimap;", "\x22B8")+ ,("mumap;", "\x22B8")+ ,("nabla;", "\x2207")+ ,("Nacute;", "\x0143")+ ,("nacute;", "\x0144")+ ,("nang;", "\x2220\x20D2")+ ,("nap;", "\x2249")+ ,("napE;", "\x2A70\x0338")+ ,("napid;", "\x224B\x0338")+ ,("napos;", "\x0149")+ ,("napprox;", "\x2249")+ ,("natur;", "\x266E")+ ,("natural;", "\x266E")+ ,("naturals;", "\x2115")+ ,("nbsp", "\x00A0")+ ,("nbsp;", "\x00A0")+ ,("nbump;", "\x224E\x0338")+ ,("nbumpe;", "\x224F\x0338")+ ,("ncap;", "\x2A43")+ ,("Ncaron;", "\x0147")+ ,("ncaron;", "\x0148")+ ,("Ncedil;", "\x0145")+ ,("ncedil;", "\x0146")+ ,("ncong;", "\x2247")+ ,("ncongdot;", "\x2A6D\x0338")+ ,("ncup;", "\x2A42")+ ,("Ncy;", "\x041D")+ ,("ncy;", "\x043D")+ ,("ndash;", "\x2013")+ ,("ne;", "\x2260")+ ,("nearhk;", "\x2924")+ ,("nearr;", "\x2197")+ ,("neArr;", "\x21D7")+ ,("nearrow;", "\x2197")+ ,("nedot;", "\x2250\x0338")+ ,("NegativeMediumSpace;", "\x200B")+ ,("NegativeThickSpace;", "\x200B")+ ,("NegativeThinSpace;", "\x200B")+ ,("NegativeVeryThinSpace;", "\x200B")+ ,("nequiv;", "\x2262")+ ,("nesear;", "\x2928")+ ,("nesim;", "\x2242\x0338")+ ,("NestedGreaterGreater;", "\x226B")+ ,("NestedLessLess;", "\x226A")+ ,("NewLine;", "\x000A")+ ,("nexist;", "\x2204")+ ,("nexists;", "\x2204")+ ,("Nfr;", "\x1D511")+ ,("nfr;", "\x1D52B")+ ,("ngE;", "\x2267\x0338")+ ,("nge;", "\x2271")+ ,("ngeq;", "\x2271")+ ,("ngeqq;", "\x2267\x0338")+ ,("ngeqslant;", "\x2A7E\x0338")+ ,("nges;", "\x2A7E\x0338")+ ,("nGg;", "\x22D9\x0338")+ ,("ngsim;", "\x2275")+ ,("nGt;", "\x226B\x20D2")+ ,("ngt;", "\x226F")+ ,("ngtr;", "\x226F")+ ,("nGtv;", "\x226B\x0338")+ ,("nharr;", "\x21AE")+ ,("nhArr;", "\x21CE")+ ,("nhpar;", "\x2AF2")+ ,("ni;", "\x220B")+ ,("nis;", "\x22FC")+ ,("nisd;", "\x22FA")+ ,("niv;", "\x220B")+ ,("NJcy;", "\x040A")+ ,("njcy;", "\x045A")+ ,("nlarr;", "\x219A")+ ,("nlArr;", "\x21CD")+ ,("nldr;", "\x2025")+ ,("nlE;", "\x2266\x0338")+ ,("nle;", "\x2270")+ ,("nleftarrow;", "\x219A")+ ,("nLeftarrow;", "\x21CD")+ ,("nleftrightarrow;", "\x21AE")+ ,("nLeftrightarrow;", "\x21CE")+ ,("nleq;", "\x2270")+ ,("nleqq;", "\x2266\x0338")+ ,("nleqslant;", "\x2A7D\x0338")+ ,("nles;", "\x2A7D\x0338")+ ,("nless;", "\x226E")+ ,("nLl;", "\x22D8\x0338")+ ,("nlsim;", "\x2274")+ ,("nLt;", "\x226A\x20D2")+ ,("nlt;", "\x226E")+ ,("nltri;", "\x22EA")+ ,("nltrie;", "\x22EC")+ ,("nLtv;", "\x226A\x0338")+ ,("nmid;", "\x2224")+ ,("NoBreak;", "\x2060")+ ,("NonBreakingSpace;", "\x00A0")+ ,("Nopf;", "\x2115")+ ,("nopf;", "\x1D55F")+ ,("not", "\x00AC")+ ,("not;", "\x00AC")+ ,("Not;", "\x2AEC")+ ,("NotCongruent;", "\x2262")+ ,("NotCupCap;", "\x226D")+ ,("NotDoubleVerticalBar;", "\x2226")+ ,("NotElement;", "\x2209")+ ,("NotEqual;", "\x2260")+ ,("NotEqualTilde;", "\x2242\x0338")+ ,("NotExists;", "\x2204")+ ,("NotGreater;", "\x226F")+ ,("NotGreaterEqual;", "\x2271")+ ,("NotGreaterFullEqual;", "\x2267\x0338")+ ,("NotGreaterGreater;", "\x226B\x0338")+ ,("NotGreaterLess;", "\x2279")+ ,("NotGreaterSlantEqual;", "\x2A7E\x0338")+ ,("NotGreaterTilde;", "\x2275")+ ,("NotHumpDownHump;", "\x224E\x0338")+ ,("NotHumpEqual;", "\x224F\x0338")+ ,("notin;", "\x2209")+ ,("notindot;", "\x22F5\x0338")+ ,("notinE;", "\x22F9\x0338")+ ,("notinva;", "\x2209")+ ,("notinvb;", "\x22F7")+ ,("notinvc;", "\x22F6")+ ,("NotLeftTriangle;", "\x22EA")+ ,("NotLeftTriangleBar;", "\x29CF\x0338")+ ,("NotLeftTriangleEqual;", "\x22EC")+ ,("NotLess;", "\x226E")+ ,("NotLessEqual;", "\x2270")+ ,("NotLessGreater;", "\x2278")+ ,("NotLessLess;", "\x226A\x0338")+ ,("NotLessSlantEqual;", "\x2A7D\x0338")+ ,("NotLessTilde;", "\x2274")+ ,("NotNestedGreaterGreater;", "\x2AA2\x0338")+ ,("NotNestedLessLess;", "\x2AA1\x0338")+ ,("notni;", "\x220C")+ ,("notniva;", "\x220C")+ ,("notnivb;", "\x22FE")+ ,("notnivc;", "\x22FD")+ ,("NotPrecedes;", "\x2280")+ ,("NotPrecedesEqual;", "\x2AAF\x0338")+ ,("NotPrecedesSlantEqual;", "\x22E0")+ ,("NotReverseElement;", "\x220C")+ ,("NotRightTriangle;", "\x22EB")+ ,("NotRightTriangleBar;", "\x29D0\x0338")+ ,("NotRightTriangleEqual;", "\x22ED")+ ,("NotSquareSubset;", "\x228F\x0338")+ ,("NotSquareSubsetEqual;", "\x22E2")+ ,("NotSquareSuperset;", "\x2290\x0338")+ ,("NotSquareSupersetEqual;", "\x22E3")+ ,("NotSubset;", "\x2282\x20D2")+ ,("NotSubsetEqual;", "\x2288")+ ,("NotSucceeds;", "\x2281")+ ,("NotSucceedsEqual;", "\x2AB0\x0338")+ ,("NotSucceedsSlantEqual;", "\x22E1")+ ,("NotSucceedsTilde;", "\x227F\x0338")+ ,("NotSuperset;", "\x2283\x20D2")+ ,("NotSupersetEqual;", "\x2289")+ ,("NotTilde;", "\x2241")+ ,("NotTildeEqual;", "\x2244")+ ,("NotTildeFullEqual;", "\x2247")+ ,("NotTildeTilde;", "\x2249")+ ,("NotVerticalBar;", "\x2224")+ ,("npar;", "\x2226")+ ,("nparallel;", "\x2226")+ ,("nparsl;", "\x2AFD\x20E5")+ ,("npart;", "\x2202\x0338")+ ,("npolint;", "\x2A14")+ ,("npr;", "\x2280")+ ,("nprcue;", "\x22E0")+ ,("npre;", "\x2AAF\x0338")+ ,("nprec;", "\x2280")+ ,("npreceq;", "\x2AAF\x0338")+ ,("nrarr;", "\x219B")+ ,("nrArr;", "\x21CF")+ ,("nrarrc;", "\x2933\x0338")+ ,("nrarrw;", "\x219D\x0338")+ ,("nrightarrow;", "\x219B")+ ,("nRightarrow;", "\x21CF")+ ,("nrtri;", "\x22EB")+ ,("nrtrie;", "\x22ED")+ ,("nsc;", "\x2281")+ ,("nsccue;", "\x22E1")+ ,("nsce;", "\x2AB0\x0338")+ ,("Nscr;", "\x1D4A9")+ ,("nscr;", "\x1D4C3")+ ,("nshortmid;", "\x2224")+ ,("nshortparallel;", "\x2226")+ ,("nsim;", "\x2241")+ ,("nsime;", "\x2244")+ ,("nsimeq;", "\x2244")+ ,("nsmid;", "\x2224")+ ,("nspar;", "\x2226")+ ,("nsqsube;", "\x22E2")+ ,("nsqsupe;", "\x22E3")+ ,("nsub;", "\x2284")+ ,("nsube;", "\x2288")+ ,("nsubE;", "\x2AC5\x0338")+ ,("nsubset;", "\x2282\x20D2")+ ,("nsubseteq;", "\x2288")+ ,("nsubseteqq;", "\x2AC5\x0338")+ ,("nsucc;", "\x2281")+ ,("nsucceq;", "\x2AB0\x0338")+ ,("nsup;", "\x2285")+ ,("nsupe;", "\x2289")+ ,("nsupE;", "\x2AC6\x0338")+ ,("nsupset;", "\x2283\x20D2")+ ,("nsupseteq;", "\x2289")+ ,("nsupseteqq;", "\x2AC6\x0338")+ ,("ntgl;", "\x2279")+ ,("Ntilde", "\x00D1")+ ,("ntilde", "\x00F1")+ ,("Ntilde;", "\x00D1")+ ,("ntilde;", "\x00F1")+ ,("ntlg;", "\x2278")+ ,("ntriangleleft;", "\x22EA")+ ,("ntrianglelefteq;", "\x22EC")+ ,("ntriangleright;", "\x22EB")+ ,("ntrianglerighteq;", "\x22ED")+ ,("Nu;", "\x039D")+ ,("nu;", "\x03BD")+ ,("num;", "\x0023")+ ,("numero;", "\x2116")+ ,("numsp;", "\x2007")+ ,("nvap;", "\x224D\x20D2")+ ,("nvdash;", "\x22AC")+ ,("nvDash;", "\x22AD")+ ,("nVdash;", "\x22AE")+ ,("nVDash;", "\x22AF")+ ,("nvge;", "\x2265\x20D2")+ ,("nvgt;", "\x003E\x20D2")+ ,("nvHarr;", "\x2904")+ ,("nvinfin;", "\x29DE")+ ,("nvlArr;", "\x2902")+ ,("nvle;", "\x2264\x20D2")+ ,("nvlt;", "\x003C\x20D2")+ ,("nvltrie;", "\x22B4\x20D2")+ ,("nvrArr;", "\x2903")+ ,("nvrtrie;", "\x22B5\x20D2")+ ,("nvsim;", "\x223C\x20D2")+ ,("nwarhk;", "\x2923")+ ,("nwarr;", "\x2196")+ ,("nwArr;", "\x21D6")+ ,("nwarrow;", "\x2196")+ ,("nwnear;", "\x2927")+ ,("Oacute", "\x00D3")+ ,("oacute", "\x00F3")+ ,("Oacute;", "\x00D3")+ ,("oacute;", "\x00F3")+ ,("oast;", "\x229B")+ ,("ocir;", "\x229A")+ ,("Ocirc", "\x00D4")+ ,("ocirc", "\x00F4")+ ,("Ocirc;", "\x00D4")+ ,("ocirc;", "\x00F4")+ ,("Ocy;", "\x041E")+ ,("ocy;", "\x043E")+ ,("odash;", "\x229D")+ ,("Odblac;", "\x0150")+ ,("odblac;", "\x0151")+ ,("odiv;", "\x2A38")+ ,("odot;", "\x2299")+ ,("odsold;", "\x29BC")+ ,("OElig;", "\x0152")+ ,("oelig;", "\x0153")+ ,("ofcir;", "\x29BF")+ ,("Ofr;", "\x1D512")+ ,("ofr;", "\x1D52C")+ ,("ogon;", "\x02DB")+ ,("Ograve", "\x00D2")+ ,("ograve", "\x00F2")+ ,("Ograve;", "\x00D2")+ ,("ograve;", "\x00F2")+ ,("ogt;", "\x29C1")+ ,("ohbar;", "\x29B5")+ ,("ohm;", "\x03A9")+ ,("oint;", "\x222E")+ ,("olarr;", "\x21BA")+ ,("olcir;", "\x29BE")+ ,("olcross;", "\x29BB")+ ,("oline;", "\x203E")+ ,("olt;", "\x29C0")+ ,("Omacr;", "\x014C")+ ,("omacr;", "\x014D")+ ,("Omega;", "\x03A9")+ ,("omega;", "\x03C9")+ ,("Omicron;", "\x039F")+ ,("omicron;", "\x03BF")+ ,("omid;", "\x29B6")+ ,("ominus;", "\x2296")+ ,("Oopf;", "\x1D546")+ ,("oopf;", "\x1D560")+ ,("opar;", "\x29B7")+ ,("OpenCurlyDoubleQuote;", "\x201C")+ ,("OpenCurlyQuote;", "\x2018")+ ,("operp;", "\x29B9")+ ,("oplus;", "\x2295")+ ,("or;", "\x2228")+ ,("Or;", "\x2A54")+ ,("orarr;", "\x21BB")+ ,("ord;", "\x2A5D")+ ,("order;", "\x2134")+ ,("orderof;", "\x2134")+ ,("ordf", "\x00AA")+ ,("ordf;", "\x00AA")+ ,("ordm", "\x00BA")+ ,("ordm;", "\x00BA")+ ,("origof;", "\x22B6")+ ,("oror;", "\x2A56")+ ,("orslope;", "\x2A57")+ ,("orv;", "\x2A5B")+ ,("oS;", "\x24C8")+ ,("oscr;", "\x2134")+ ,("Oscr;", "\x1D4AA")+ ,("Oslash", "\x00D8")+ ,("oslash", "\x00F8")+ ,("Oslash;", "\x00D8")+ ,("oslash;", "\x00F8")+ ,("osol;", "\x2298")+ ,("Otilde", "\x00D5")+ ,("otilde", "\x00F5")+ ,("Otilde;", "\x00D5")+ ,("otilde;", "\x00F5")+ ,("otimes;", "\x2297")+ ,("Otimes;", "\x2A37")+ ,("otimesas;", "\x2A36")+ ,("Ouml", "\x00D6")+ ,("ouml", "\x00F6")+ ,("Ouml;", "\x00D6")+ ,("ouml;", "\x00F6")+ ,("ovbar;", "\x233D")+ ,("OverBar;", "\x203E")+ ,("OverBrace;", "\x23DE")+ ,("OverBracket;", "\x23B4")+ ,("OverParenthesis;", "\x23DC")+ ,("par;", "\x2225")+ ,("para", "\x00B6")+ ,("para;", "\x00B6")+ ,("parallel;", "\x2225")+ ,("parsim;", "\x2AF3")+ ,("parsl;", "\x2AFD")+ ,("part;", "\x2202")+ ,("PartialD;", "\x2202")+ ,("Pcy;", "\x041F")+ ,("pcy;", "\x043F")+ ,("percnt;", "\x0025")+ ,("period;", "\x002E")+ ,("permil;", "\x2030")+ ,("perp;", "\x22A5")+ ,("pertenk;", "\x2031")+ ,("Pfr;", "\x1D513")+ ,("pfr;", "\x1D52D")+ ,("Phi;", "\x03A6")+ ,("phi;", "\x03C6")+ ,("phiv;", "\x03D5")+ ,("phmmat;", "\x2133")+ ,("phone;", "\x260E")+ ,("Pi;", "\x03A0")+ ,("pi;", "\x03C0")+ ,("pitchfork;", "\x22D4")+ ,("piv;", "\x03D6")+ ,("planck;", "\x210F")+ ,("planckh;", "\x210E")+ ,("plankv;", "\x210F")+ ,("plus;", "\x002B")+ ,("plusacir;", "\x2A23")+ ,("plusb;", "\x229E")+ ,("pluscir;", "\x2A22")+ ,("plusdo;", "\x2214")+ ,("plusdu;", "\x2A25")+ ,("pluse;", "\x2A72")+ ,("PlusMinus;", "\x00B1")+ ,("plusmn", "\x00B1")+ ,("plusmn;", "\x00B1")+ ,("plussim;", "\x2A26")+ ,("plustwo;", "\x2A27")+ ,("pm;", "\x00B1")+ ,("Poincareplane;", "\x210C")+ ,("pointint;", "\x2A15")+ ,("Popf;", "\x2119")+ ,("popf;", "\x1D561")+ ,("pound", "\x00A3")+ ,("pound;", "\x00A3")+ ,("pr;", "\x227A")+ ,("Pr;", "\x2ABB")+ ,("prap;", "\x2AB7")+ ,("prcue;", "\x227C")+ ,("pre;", "\x2AAF")+ ,("prE;", "\x2AB3")+ ,("prec;", "\x227A")+ ,("precapprox;", "\x2AB7")+ ,("preccurlyeq;", "\x227C")+ ,("Precedes;", "\x227A")+ ,("PrecedesEqual;", "\x2AAF")+ ,("PrecedesSlantEqual;", "\x227C")+ ,("PrecedesTilde;", "\x227E")+ ,("preceq;", "\x2AAF")+ ,("precnapprox;", "\x2AB9")+ ,("precneqq;", "\x2AB5")+ ,("precnsim;", "\x22E8")+ ,("precsim;", "\x227E")+ ,("prime;", "\x2032")+ ,("Prime;", "\x2033")+ ,("primes;", "\x2119")+ ,("prnap;", "\x2AB9")+ ,("prnE;", "\x2AB5")+ ,("prnsim;", "\x22E8")+ ,("prod;", "\x220F")+ ,("Product;", "\x220F")+ ,("profalar;", "\x232E")+ ,("profline;", "\x2312")+ ,("profsurf;", "\x2313")+ ,("prop;", "\x221D")+ ,("Proportion;", "\x2237")+ ,("Proportional;", "\x221D")+ ,("propto;", "\x221D")+ ,("prsim;", "\x227E")+ ,("prurel;", "\x22B0")+ ,("Pscr;", "\x1D4AB")+ ,("pscr;", "\x1D4C5")+ ,("Psi;", "\x03A8")+ ,("psi;", "\x03C8")+ ,("puncsp;", "\x2008")+ ,("Qfr;", "\x1D514")+ ,("qfr;", "\x1D52E")+ ,("qint;", "\x2A0C")+ ,("Qopf;", "\x211A")+ ,("qopf;", "\x1D562")+ ,("qprime;", "\x2057")+ ,("Qscr;", "\x1D4AC")+ ,("qscr;", "\x1D4C6")+ ,("quaternions;", "\x210D")+ ,("quatint;", "\x2A16")+ ,("quest;", "\x003F")+ ,("questeq;", "\x225F")+ ,("QUOT", "\x0022")+ ,("quot", "\x0022")+ ,("QUOT;", "\x0022")+ ,("quot;", "\x0022")+ ,("rAarr;", "\x21DB")+ ,("race;", "\x223D\x0331")+ ,("Racute;", "\x0154")+ ,("racute;", "\x0155")+ ,("radic;", "\x221A")+ ,("raemptyv;", "\x29B3")+ ,("rang;", "\x27E9")+ ,("Rang;", "\x27EB")+ ,("rangd;", "\x2992")+ ,("range;", "\x29A5")+ ,("rangle;", "\x27E9")+ ,("raquo", "\x00BB")+ ,("raquo;", "\x00BB")+ ,("rarr;", "\x2192")+ ,("Rarr;", "\x21A0")+ ,("rArr;", "\x21D2")+ ,("rarrap;", "\x2975")+ ,("rarrb;", "\x21E5")+ ,("rarrbfs;", "\x2920")+ ,("rarrc;", "\x2933")+ ,("rarrfs;", "\x291E")+ ,("rarrhk;", "\x21AA")+ ,("rarrlp;", "\x21AC")+ ,("rarrpl;", "\x2945")+ ,("rarrsim;", "\x2974")+ ,("rarrtl;", "\x21A3")+ ,("Rarrtl;", "\x2916")+ ,("rarrw;", "\x219D")+ ,("ratail;", "\x291A")+ ,("rAtail;", "\x291C")+ ,("ratio;", "\x2236")+ ,("rationals;", "\x211A")+ ,("rbarr;", "\x290D")+ ,("rBarr;", "\x290F")+ ,("RBarr;", "\x2910")+ ,("rbbrk;", "\x2773")+ ,("rbrace;", "\x007D")+ ,("rbrack;", "\x005D")+ ,("rbrke;", "\x298C")+ ,("rbrksld;", "\x298E")+ ,("rbrkslu;", "\x2990")+ ,("Rcaron;", "\x0158")+ ,("rcaron;", "\x0159")+ ,("Rcedil;", "\x0156")+ ,("rcedil;", "\x0157")+ ,("rceil;", "\x2309")+ ,("rcub;", "\x007D")+ ,("Rcy;", "\x0420")+ ,("rcy;", "\x0440")+ ,("rdca;", "\x2937")+ ,("rdldhar;", "\x2969")+ ,("rdquo;", "\x201D")+ ,("rdquor;", "\x201D")+ ,("rdsh;", "\x21B3")+ ,("Re;", "\x211C")+ ,("real;", "\x211C")+ ,("realine;", "\x211B")+ ,("realpart;", "\x211C")+ ,("reals;", "\x211D")+ ,("rect;", "\x25AD")+ ,("REG", "\x00AE")+ ,("reg", "\x00AE")+ ,("REG;", "\x00AE")+ ,("reg;", "\x00AE")+ ,("ReverseElement;", "\x220B")+ ,("ReverseEquilibrium;", "\x21CB")+ ,("ReverseUpEquilibrium;", "\x296F")+ ,("rfisht;", "\x297D")+ ,("rfloor;", "\x230B")+ ,("Rfr;", "\x211C")+ ,("rfr;", "\x1D52F")+ ,("rHar;", "\x2964")+ ,("rhard;", "\x21C1")+ ,("rharu;", "\x21C0")+ ,("rharul;", "\x296C")+ ,("Rho;", "\x03A1")+ ,("rho;", "\x03C1")+ ,("rhov;", "\x03F1")+ ,("RightAngleBracket;", "\x27E9")+ ,("RightArrow;", "\x2192")+ ,("rightarrow;", "\x2192")+ ,("Rightarrow;", "\x21D2")+ ,("RightArrowBar;", "\x21E5")+ ,("RightArrowLeftArrow;", "\x21C4")+ ,("rightarrowtail;", "\x21A3")+ ,("RightCeiling;", "\x2309")+ ,("RightDoubleBracket;", "\x27E7")+ ,("RightDownTeeVector;", "\x295D")+ ,("RightDownVector;", "\x21C2")+ ,("RightDownVectorBar;", "\x2955")+ ,("RightFloor;", "\x230B")+ ,("rightharpoondown;", "\x21C1")+ ,("rightharpoonup;", "\x21C0")+ ,("rightleftarrows;", "\x21C4")+ ,("rightleftharpoons;", "\x21CC")+ ,("rightrightarrows;", "\x21C9")+ ,("rightsquigarrow;", "\x219D")+ ,("RightTee;", "\x22A2")+ ,("RightTeeArrow;", "\x21A6")+ ,("RightTeeVector;", "\x295B")+ ,("rightthreetimes;", "\x22CC")+ ,("RightTriangle;", "\x22B3")+ ,("RightTriangleBar;", "\x29D0")+ ,("RightTriangleEqual;", "\x22B5")+ ,("RightUpDownVector;", "\x294F")+ ,("RightUpTeeVector;", "\x295C")+ ,("RightUpVector;", "\x21BE")+ ,("RightUpVectorBar;", "\x2954")+ ,("RightVector;", "\x21C0")+ ,("RightVectorBar;", "\x2953")+ ,("ring;", "\x02DA")+ ,("risingdotseq;", "\x2253")+ ,("rlarr;", "\x21C4")+ ,("rlhar;", "\x21CC")+ ,("rlm;", "\x200F")+ ,("rmoust;", "\x23B1")+ ,("rmoustache;", "\x23B1")+ ,("rnmid;", "\x2AEE")+ ,("roang;", "\x27ED")+ ,("roarr;", "\x21FE")+ ,("robrk;", "\x27E7")+ ,("ropar;", "\x2986")+ ,("Ropf;", "\x211D")+ ,("ropf;", "\x1D563")+ ,("roplus;", "\x2A2E")+ ,("rotimes;", "\x2A35")+ ,("RoundImplies;", "\x2970")+ ,("rpar;", "\x0029")+ ,("rpargt;", "\x2994")+ ,("rppolint;", "\x2A12")+ ,("rrarr;", "\x21C9")+ ,("Rrightarrow;", "\x21DB")+ ,("rsaquo;", "\x203A")+ ,("Rscr;", "\x211B")+ ,("rscr;", "\x1D4C7")+ ,("Rsh;", "\x21B1")+ ,("rsh;", "\x21B1")+ ,("rsqb;", "\x005D")+ ,("rsquo;", "\x2019")+ ,("rsquor;", "\x2019")+ ,("rthree;", "\x22CC")+ ,("rtimes;", "\x22CA")+ ,("rtri;", "\x25B9")+ ,("rtrie;", "\x22B5")+ ,("rtrif;", "\x25B8")+ ,("rtriltri;", "\x29CE")+ ,("RuleDelayed;", "\x29F4")+ ,("ruluhar;", "\x2968")+ ,("rx;", "\x211E")+ ,("Sacute;", "\x015A")+ ,("sacute;", "\x015B")+ ,("sbquo;", "\x201A")+ ,("sc;", "\x227B")+ ,("Sc;", "\x2ABC")+ ,("scap;", "\x2AB8")+ ,("Scaron;", "\x0160")+ ,("scaron;", "\x0161")+ ,("sccue;", "\x227D")+ ,("sce;", "\x2AB0")+ ,("scE;", "\x2AB4")+ ,("Scedil;", "\x015E")+ ,("scedil;", "\x015F")+ ,("Scirc;", "\x015C")+ ,("scirc;", "\x015D")+ ,("scnap;", "\x2ABA")+ ,("scnE;", "\x2AB6")+ ,("scnsim;", "\x22E9")+ ,("scpolint;", "\x2A13")+ ,("scsim;", "\x227F")+ ,("Scy;", "\x0421")+ ,("scy;", "\x0441")+ ,("sdot;", "\x22C5")+ ,("sdotb;", "\x22A1")+ ,("sdote;", "\x2A66")+ ,("searhk;", "\x2925")+ ,("searr;", "\x2198")+ ,("seArr;", "\x21D8")+ ,("searrow;", "\x2198")+ ,("sect", "\x00A7")+ ,("sect;", "\x00A7")+ ,("semi;", "\x003B")+ ,("seswar;", "\x2929")+ ,("setminus;", "\x2216")+ ,("setmn;", "\x2216")+ ,("sext;", "\x2736")+ ,("Sfr;", "\x1D516")+ ,("sfr;", "\x1D530")+ ,("sfrown;", "\x2322")+ ,("sharp;", "\x266F")+ ,("SHCHcy;", "\x0429")+ ,("shchcy;", "\x0449")+ ,("SHcy;", "\x0428")+ ,("shcy;", "\x0448")+ ,("ShortDownArrow;", "\x2193")+ ,("ShortLeftArrow;", "\x2190")+ ,("shortmid;", "\x2223")+ ,("shortparallel;", "\x2225")+ ,("ShortRightArrow;", "\x2192")+ ,("ShortUpArrow;", "\x2191")+ ,("shy", "\x00AD")+ ,("shy;", "\x00AD")+ ,("Sigma;", "\x03A3")+ ,("sigma;", "\x03C3")+ ,("sigmaf;", "\x03C2")+ ,("sigmav;", "\x03C2")+ ,("sim;", "\x223C")+ ,("simdot;", "\x2A6A")+ ,("sime;", "\x2243")+ ,("simeq;", "\x2243")+ ,("simg;", "\x2A9E")+ ,("simgE;", "\x2AA0")+ ,("siml;", "\x2A9D")+ ,("simlE;", "\x2A9F")+ ,("simne;", "\x2246")+ ,("simplus;", "\x2A24")+ ,("simrarr;", "\x2972")+ ,("slarr;", "\x2190")+ ,("SmallCircle;", "\x2218")+ ,("smallsetminus;", "\x2216")+ ,("smashp;", "\x2A33")+ ,("smeparsl;", "\x29E4")+ ,("smid;", "\x2223")+ ,("smile;", "\x2323")+ ,("smt;", "\x2AAA")+ ,("smte;", "\x2AAC")+ ,("smtes;", "\x2AAC\xFE00")+ ,("SOFTcy;", "\x042C")+ ,("softcy;", "\x044C")+ ,("sol;", "\x002F")+ ,("solb;", "\x29C4")+ ,("solbar;", "\x233F")+ ,("Sopf;", "\x1D54A")+ ,("sopf;", "\x1D564")+ ,("spades;", "\x2660")+ ,("spadesuit;", "\x2660")+ ,("spar;", "\x2225")+ ,("sqcap;", "\x2293")+ ,("sqcaps;", "\x2293\xFE00")+ ,("sqcup;", "\x2294")+ ,("sqcups;", "\x2294\xFE00")+ ,("Sqrt;", "\x221A")+ ,("sqsub;", "\x228F")+ ,("sqsube;", "\x2291")+ ,("sqsubset;", "\x228F")+ ,("sqsubseteq;", "\x2291")+ ,("sqsup;", "\x2290")+ ,("sqsupe;", "\x2292")+ ,("sqsupset;", "\x2290")+ ,("sqsupseteq;", "\x2292")+ ,("squ;", "\x25A1")+ ,("Square;", "\x25A1")+ ,("square;", "\x25A1")+ ,("SquareIntersection;", "\x2293")+ ,("SquareSubset;", "\x228F")+ ,("SquareSubsetEqual;", "\x2291")+ ,("SquareSuperset;", "\x2290")+ ,("SquareSupersetEqual;", "\x2292")+ ,("SquareUnion;", "\x2294")+ ,("squarf;", "\x25AA")+ ,("squf;", "\x25AA")+ ,("srarr;", "\x2192")+ ,("Sscr;", "\x1D4AE")+ ,("sscr;", "\x1D4C8")+ ,("ssetmn;", "\x2216")+ ,("ssmile;", "\x2323")+ ,("sstarf;", "\x22C6")+ ,("Star;", "\x22C6")+ ,("star;", "\x2606")+ ,("starf;", "\x2605")+ ,("straightepsilon;", "\x03F5")+ ,("straightphi;", "\x03D5")+ ,("strns;", "\x00AF")+ ,("sub;", "\x2282")+ ,("Sub;", "\x22D0")+ ,("subdot;", "\x2ABD")+ ,("sube;", "\x2286")+ ,("subE;", "\x2AC5")+ ,("subedot;", "\x2AC3")+ ,("submult;", "\x2AC1")+ ,("subne;", "\x228A")+ ,("subnE;", "\x2ACB")+ ,("subplus;", "\x2ABF")+ ,("subrarr;", "\x2979")+ ,("subset;", "\x2282")+ ,("Subset;", "\x22D0")+ ,("subseteq;", "\x2286")+ ,("subseteqq;", "\x2AC5")+ ,("SubsetEqual;", "\x2286")+ ,("subsetneq;", "\x228A")+ ,("subsetneqq;", "\x2ACB")+ ,("subsim;", "\x2AC7")+ ,("subsub;", "\x2AD5")+ ,("subsup;", "\x2AD3")+ ,("succ;", "\x227B")+ ,("succapprox;", "\x2AB8")+ ,("succcurlyeq;", "\x227D")+ ,("Succeeds;", "\x227B")+ ,("SucceedsEqual;", "\x2AB0")+ ,("SucceedsSlantEqual;", "\x227D")+ ,("SucceedsTilde;", "\x227F")+ ,("succeq;", "\x2AB0")+ ,("succnapprox;", "\x2ABA")+ ,("succneqq;", "\x2AB6")+ ,("succnsim;", "\x22E9")+ ,("succsim;", "\x227F")+ ,("SuchThat;", "\x220B")+ ,("Sum;", "\x2211")+ ,("sum;", "\x2211")+ ,("sung;", "\x266A")+ ,("sup1", "\x00B9")+ ,("sup1;", "\x00B9")+ ,("sup2", "\x00B2")+ ,("sup2;", "\x00B2")+ ,("sup3", "\x00B3")+ ,("sup3;", "\x00B3")+ ,("sup;", "\x2283")+ ,("Sup;", "\x22D1")+ ,("supdot;", "\x2ABE")+ ,("supdsub;", "\x2AD8")+ ,("supe;", "\x2287")+ ,("supE;", "\x2AC6")+ ,("supedot;", "\x2AC4")+ ,("Superset;", "\x2283")+ ,("SupersetEqual;", "\x2287")+ ,("suphsol;", "\x27C9")+ ,("suphsub;", "\x2AD7")+ ,("suplarr;", "\x297B")+ ,("supmult;", "\x2AC2")+ ,("supne;", "\x228B")+ ,("supnE;", "\x2ACC")+ ,("supplus;", "\x2AC0")+ ,("supset;", "\x2283")+ ,("Supset;", "\x22D1")+ ,("supseteq;", "\x2287")+ ,("supseteqq;", "\x2AC6")+ ,("supsetneq;", "\x228B")+ ,("supsetneqq;", "\x2ACC")+ ,("supsim;", "\x2AC8")+ ,("supsub;", "\x2AD4")+ ,("supsup;", "\x2AD6")+ ,("swarhk;", "\x2926")+ ,("swarr;", "\x2199")+ ,("swArr;", "\x21D9")+ ,("swarrow;", "\x2199")+ ,("swnwar;", "\x292A")+ ,("szlig", "\x00DF")+ ,("szlig;", "\x00DF")+ ,("Tab;", "\x0009")+ ,("target;", "\x2316")+ ,("Tau;", "\x03A4")+ ,("tau;", "\x03C4")+ ,("tbrk;", "\x23B4")+ ,("Tcaron;", "\x0164")+ ,("tcaron;", "\x0165")+ ,("Tcedil;", "\x0162")+ ,("tcedil;", "\x0163")+ ,("Tcy;", "\x0422")+ ,("tcy;", "\x0442")+ ,("tdot;", "\x20DB")+ ,("telrec;", "\x2315")+ ,("Tfr;", "\x1D517")+ ,("tfr;", "\x1D531")+ ,("there4;", "\x2234")+ ,("Therefore;", "\x2234")+ ,("therefore;", "\x2234")+ ,("Theta;", "\x0398")+ ,("theta;", "\x03B8")+ ,("thetasym;", "\x03D1")+ ,("thetav;", "\x03D1")+ ,("thickapprox;", "\x2248")+ ,("thicksim;", "\x223C")+ ,("ThickSpace;", "\x205F\x200A")+ ,("thinsp;", "\x2009")+ ,("ThinSpace;", "\x2009")+ ,("thkap;", "\x2248")+ ,("thksim;", "\x223C")+ ,("THORN", "\x00DE")+ ,("thorn", "\x00FE")+ ,("THORN;", "\x00DE")+ ,("thorn;", "\x00FE")+ ,("tilde;", "\x02DC")+ ,("Tilde;", "\x223C")+ ,("TildeEqual;", "\x2243")+ ,("TildeFullEqual;", "\x2245")+ ,("TildeTilde;", "\x2248")+ ,("times", "\x00D7")+ ,("times;", "\x00D7")+ ,("timesb;", "\x22A0")+ ,("timesbar;", "\x2A31")+ ,("timesd;", "\x2A30")+ ,("tint;", "\x222D")+ ,("toea;", "\x2928")+ ,("top;", "\x22A4")+ ,("topbot;", "\x2336")+ ,("topcir;", "\x2AF1")+ ,("Topf;", "\x1D54B")+ ,("topf;", "\x1D565")+ ,("topfork;", "\x2ADA")+ ,("tosa;", "\x2929")+ ,("tprime;", "\x2034")+ ,("TRADE;", "\x2122")+ ,("trade;", "\x2122")+ ,("triangle;", "\x25B5")+ ,("triangledown;", "\x25BF")+ ,("triangleleft;", "\x25C3")+ ,("trianglelefteq;", "\x22B4")+ ,("triangleq;", "\x225C")+ ,("triangleright;", "\x25B9")+ ,("trianglerighteq;", "\x22B5")+ ,("tridot;", "\x25EC")+ ,("trie;", "\x225C")+ ,("triminus;", "\x2A3A")+ ,("TripleDot;", "\x20DB")+ ,("triplus;", "\x2A39")+ ,("trisb;", "\x29CD")+ ,("tritime;", "\x2A3B")+ ,("trpezium;", "\x23E2")+ ,("Tscr;", "\x1D4AF")+ ,("tscr;", "\x1D4C9")+ ,("TScy;", "\x0426")+ ,("tscy;", "\x0446")+ ,("TSHcy;", "\x040B")+ ,("tshcy;", "\x045B")+ ,("Tstrok;", "\x0166")+ ,("tstrok;", "\x0167")+ ,("twixt;", "\x226C")+ ,("twoheadleftarrow;", "\x219E")+ ,("twoheadrightarrow;", "\x21A0")+ ,("Uacute", "\x00DA")+ ,("uacute", "\x00FA")+ ,("Uacute;", "\x00DA")+ ,("uacute;", "\x00FA")+ ,("uarr;", "\x2191")+ ,("Uarr;", "\x219F")+ ,("uArr;", "\x21D1")+ ,("Uarrocir;", "\x2949")+ ,("Ubrcy;", "\x040E")+ ,("ubrcy;", "\x045E")+ ,("Ubreve;", "\x016C")+ ,("ubreve;", "\x016D")+ ,("Ucirc", "\x00DB")+ ,("ucirc", "\x00FB")+ ,("Ucirc;", "\x00DB")+ ,("ucirc;", "\x00FB")+ ,("Ucy;", "\x0423")+ ,("ucy;", "\x0443")+ ,("udarr;", "\x21C5")+ ,("Udblac;", "\x0170")+ ,("udblac;", "\x0171")+ ,("udhar;", "\x296E")+ ,("ufisht;", "\x297E")+ ,("Ufr;", "\x1D518")+ ,("ufr;", "\x1D532")+ ,("Ugrave", "\x00D9")+ ,("ugrave", "\x00F9")+ ,("Ugrave;", "\x00D9")+ ,("ugrave;", "\x00F9")+ ,("uHar;", "\x2963")+ ,("uharl;", "\x21BF")+ ,("uharr;", "\x21BE")+ ,("uhblk;", "\x2580")+ ,("ulcorn;", "\x231C")+ ,("ulcorner;", "\x231C")+ ,("ulcrop;", "\x230F")+ ,("ultri;", "\x25F8")+ ,("Umacr;", "\x016A")+ ,("umacr;", "\x016B")+ ,("uml", "\x00A8")+ ,("uml;", "\x00A8")+ ,("UnderBar;", "\x005F")+ ,("UnderBrace;", "\x23DF")+ ,("UnderBracket;", "\x23B5")+ ,("UnderParenthesis;", "\x23DD")+ ,("Union;", "\x22C3")+ ,("UnionPlus;", "\x228E")+ ,("Uogon;", "\x0172")+ ,("uogon;", "\x0173")+ ,("Uopf;", "\x1D54C")+ ,("uopf;", "\x1D566")+ ,("UpArrow;", "\x2191")+ ,("uparrow;", "\x2191")+ ,("Uparrow;", "\x21D1")+ ,("UpArrowBar;", "\x2912")+ ,("UpArrowDownArrow;", "\x21C5")+ ,("UpDownArrow;", "\x2195")+ ,("updownarrow;", "\x2195")+ ,("Updownarrow;", "\x21D5")+ ,("UpEquilibrium;", "\x296E")+ ,("upharpoonleft;", "\x21BF")+ ,("upharpoonright;", "\x21BE")+ ,("uplus;", "\x228E")+ ,("UpperLeftArrow;", "\x2196")+ ,("UpperRightArrow;", "\x2197")+ ,("upsi;", "\x03C5")+ ,("Upsi;", "\x03D2")+ ,("upsih;", "\x03D2")+ ,("Upsilon;", "\x03A5")+ ,("upsilon;", "\x03C5")+ ,("UpTee;", "\x22A5")+ ,("UpTeeArrow;", "\x21A5")+ ,("upuparrows;", "\x21C8")+ ,("urcorn;", "\x231D")+ ,("urcorner;", "\x231D")+ ,("urcrop;", "\x230E")+ ,("Uring;", "\x016E")+ ,("uring;", "\x016F")+ ,("urtri;", "\x25F9")+ ,("Uscr;", "\x1D4B0")+ ,("uscr;", "\x1D4CA")+ ,("utdot;", "\x22F0")+ ,("Utilde;", "\x0168")+ ,("utilde;", "\x0169")+ ,("utri;", "\x25B5")+ ,("utrif;", "\x25B4")+ ,("uuarr;", "\x21C8")+ ,("Uuml", "\x00DC")+ ,("uuml", "\x00FC")+ ,("Uuml;", "\x00DC")+ ,("uuml;", "\x00FC")+ ,("uwangle;", "\x29A7")+ ,("vangrt;", "\x299C")+ ,("varepsilon;", "\x03F5")+ ,("varkappa;", "\x03F0")+ ,("varnothing;", "\x2205")+ ,("varphi;", "\x03D5")+ ,("varpi;", "\x03D6")+ ,("varpropto;", "\x221D")+ ,("varr;", "\x2195")+ ,("vArr;", "\x21D5")+ ,("varrho;", "\x03F1")+ ,("varsigma;", "\x03C2")+ ,("varsubsetneq;", "\x228A\xFE00")+ ,("varsubsetneqq;", "\x2ACB\xFE00")+ ,("varsupsetneq;", "\x228B\xFE00")+ ,("varsupsetneqq;", "\x2ACC\xFE00")+ ,("vartheta;", "\x03D1")+ ,("vartriangleleft;", "\x22B2")+ ,("vartriangleright;", "\x22B3")+ ,("vBar;", "\x2AE8")+ ,("Vbar;", "\x2AEB")+ ,("vBarv;", "\x2AE9")+ ,("Vcy;", "\x0412")+ ,("vcy;", "\x0432")+ ,("vdash;", "\x22A2")+ ,("vDash;", "\x22A8")+ ,("Vdash;", "\x22A9")+ ,("VDash;", "\x22AB")+ ,("Vdashl;", "\x2AE6")+ ,("vee;", "\x2228")+ ,("Vee;", "\x22C1")+ ,("veebar;", "\x22BB")+ ,("veeeq;", "\x225A")+ ,("vellip;", "\x22EE")+ ,("verbar;", "\x007C")+ ,("Verbar;", "\x2016")+ ,("vert;", "\x007C")+ ,("Vert;", "\x2016")+ ,("VerticalBar;", "\x2223")+ ,("VerticalLine;", "\x007C")+ ,("VerticalSeparator;", "\x2758")+ ,("VerticalTilde;", "\x2240")+ ,("VeryThinSpace;", "\x200A")+ ,("Vfr;", "\x1D519")+ ,("vfr;", "\x1D533")+ ,("vltri;", "\x22B2")+ ,("vnsub;", "\x2282\x20D2")+ ,("vnsup;", "\x2283\x20D2")+ ,("Vopf;", "\x1D54D")+ ,("vopf;", "\x1D567")+ ,("vprop;", "\x221D")+ ,("vrtri;", "\x22B3")+ ,("Vscr;", "\x1D4B1")+ ,("vscr;", "\x1D4CB")+ ,("vsubne;", "\x228A\xFE00")+ ,("vsubnE;", "\x2ACB\xFE00")+ ,("vsupne;", "\x228B\xFE00")+ ,("vsupnE;", "\x2ACC\xFE00")+ ,("Vvdash;", "\x22AA")+ ,("vzigzag;", "\x299A")+ ,("Wcirc;", "\x0174")+ ,("wcirc;", "\x0175")+ ,("wedbar;", "\x2A5F")+ ,("wedge;", "\x2227")+ ,("Wedge;", "\x22C0")+ ,("wedgeq;", "\x2259")+ ,("weierp;", "\x2118")+ ,("Wfr;", "\x1D51A")+ ,("wfr;", "\x1D534")+ ,("Wopf;", "\x1D54E")+ ,("wopf;", "\x1D568")+ ,("wp;", "\x2118")+ ,("wr;", "\x2240")+ ,("wreath;", "\x2240")+ ,("Wscr;", "\x1D4B2")+ ,("wscr;", "\x1D4CC")+ ,("xcap;", "\x22C2")+ ,("xcirc;", "\x25EF")+ ,("xcup;", "\x22C3")+ ,("xdtri;", "\x25BD")+ ,("Xfr;", "\x1D51B")+ ,("xfr;", "\x1D535")+ ,("xharr;", "\x27F7")+ ,("xhArr;", "\x27FA")+ ,("Xi;", "\x039E")+ ,("xi;", "\x03BE")+ ,("xlarr;", "\x27F5")+ ,("xlArr;", "\x27F8")+ ,("xmap;", "\x27FC")+ ,("xnis;", "\x22FB")+ ,("xodot;", "\x2A00")+ ,("Xopf;", "\x1D54F")+ ,("xopf;", "\x1D569")+ ,("xoplus;", "\x2A01")+ ,("xotime;", "\x2A02")+ ,("xrarr;", "\x27F6")+ ,("xrArr;", "\x27F9")+ ,("Xscr;", "\x1D4B3")+ ,("xscr;", "\x1D4CD")+ ,("xsqcup;", "\x2A06")+ ,("xuplus;", "\x2A04")+ ,("xutri;", "\x25B3")+ ,("xvee;", "\x22C1")+ ,("xwedge;", "\x22C0")+ ,("Yacute", "\x00DD")+ ,("yacute", "\x00FD")+ ,("Yacute;", "\x00DD")+ ,("yacute;", "\x00FD")+ ,("YAcy;", "\x042F")+ ,("yacy;", "\x044F")+ ,("Ycirc;", "\x0176")+ ,("ycirc;", "\x0177")+ ,("Ycy;", "\x042B")+ ,("ycy;", "\x044B")+ ,("yen", "\x00A5")+ ,("yen;", "\x00A5")+ ,("Yfr;", "\x1D51C")+ ,("yfr;", "\x1D536")+ ,("YIcy;", "\x0407")+ ,("yicy;", "\x0457")+ ,("Yopf;", "\x1D550")+ ,("yopf;", "\x1D56A")+ ,("Yscr;", "\x1D4B4")+ ,("yscr;", "\x1D4CE")+ ,("YUcy;", "\x042E")+ ,("yucy;", "\x044E")+ ,("yuml", "\x00FF")+ ,("yuml;", "\x00FF")+ ,("Yuml;", "\x0178")+ ,("Zacute;", "\x0179")+ ,("zacute;", "\x017A")+ ,("Zcaron;", "\x017D")+ ,("zcaron;", "\x017E")+ ,("Zcy;", "\x0417")+ ,("zcy;", "\x0437")+ ,("Zdot;", "\x017B")+ ,("zdot;", "\x017C")+ ,("zeetrf;", "\x2128")+ ,("ZeroWidthSpace;", "\x200B")+ ,("Zeta;", "\x0396")+ ,("zeta;", "\x03B6")+ ,("Zfr;", "\x2128")+ ,("zfr;", "\x1D537")+ ,("ZHcy;", "\x0416")+ ,("zhcy;", "\x0436")+ ,("zigrarr;", "\x21DD")+ ,("Zopf;", "\x2124")+ ,("zopf;", "\x1D56B")+ ,("Zscr;", "\x1D4B5")+ ,("zscr;", "\x1D4CF")+ ,("zwj;", "\x200D")+ ,("zwnj;", "\x200C")+ ]++charEntity :: Monad m => ParsecT [Tok] s m [Tok]+charEntity = do+ wc@(Tok WordChars _ ts) <- satisfyTok (hasType WordChars)+ semi <- symbol ';'+ guard $ isJust $ lookupEntity (ts <> ";")+ return [wc, semi]++numEntity :: Monad m => ParsecT [Tok] s m [Tok]+numEntity = do+ octo <- symbol '#'+ wc@(Tok WordChars _ t) <- satisfyTok (hasType WordChars)+ guard $+ case T.uncons t of+ Just (x, rest)+ | x == 'x' || x == 'X' ->+ T.all isHexDigit rest &&+ not (T.null rest) &&+ T.length rest <= 6+ | otherwise -> T.all isDigit t &&+ T.length t <= 7+ _ -> False+ semi <- symbol ';'+ return [octo, wc, semi]++unEntity :: [Tok] -> Text+unEntity ts = untokenize $+ case parse (many (pEntity' <|> anyTok)) "" ts of+ Left _ -> ts+ Right ts' -> ts'+ where pEntity' :: ParsecT [Tok] () Identity Tok+ pEntity' = try $ do+ pos <- getPosition+ symbol '&'+ ent <- untokenize <$> (numEntity <|> charEntity)+ case lookupEntity ent of+ Just s -> return $ Tok WordChars pos s+ Nothing -> mzero
+ src/Commonmark/Html.hs view
@@ -0,0 +1,247 @@+{-# LANGUAGE CPP #-}+{-# LANGUAGE BangPatterns #-}+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE MultiParamTypeClasses #-}+{-# LANGUAGE FlexibleInstances #-}+{-# LANGUAGE FlexibleContexts #-}+{-# LANGUAGE UndecidableInstances #-}+module Commonmark.Html+ ( Html+ , htmlInline+ , htmlBlock+ , htmlText+ , htmlRaw+ , addAttribute+ , renderHtml+ , escapeURI+ , escapeHtml+ )+where++import Commonmark.Types+import Commonmark.Entity (lookupEntity)+import Data.Text (Text)+import qualified Data.Text as T+import qualified Data.Text.Lazy as TL+import Data.Text.Lazy.Builder (Builder, fromText, toLazyText,+ singleton)+import Data.Text.Encoding (encodeUtf8)+import qualified Data.ByteString.Char8 as B+import Text.Printf (printf)+import Data.Char (ord, isAlphaNum, isAscii, isSpace)+import Data.Maybe (fromMaybe)+#if !MIN_VERSION_base(4,11,0)+import Data.Semigroup+#endif++data ElementType =+ InlineElement+ | BlockElement++data Html a =+ HtmlElement !ElementType {-# UNPACK #-} !Text [Attribute] (Maybe (Html a))+ | HtmlText {-# UNPACK #-} !Text+ | HtmlRaw {-# UNPACK #-} !Text+ | HtmlNull+ | HtmlConcat !(Html a) !(Html a)++instance Show (Html a) where+ show = TL.unpack . renderHtml++instance Semigroup (Html a) where+ x <> HtmlNull = x+ HtmlNull <> x = x+ HtmlText t1 <> HtmlText t2 = HtmlText (t1 <> t2)+ HtmlRaw t1 <> HtmlRaw t2 = HtmlRaw (t1 <> t2)+ x <> y = HtmlConcat x y++instance Monoid (Html a) where+ mempty = HtmlNull+ mappend = (<>)++instance HasAttributes (Html a) where+ addAttributes attrs x = foldr addAttribute x attrs++instance ToPlainText (Html a) where+ toPlainText h =+ case h of+ HtmlElement InlineElement "span" attr (Just x)+ -> case lookup "data-emoji" attr of+ Just alias -> ":" <> alias <> ":"+ Nothing -> toPlainText x+ HtmlElement _ _ _ (Just x) -> toPlainText x+ HtmlElement _ _ attrs Nothing+ -> fromMaybe mempty $ lookup "alt" attrs+ HtmlText t -> t+ HtmlConcat x y -> toPlainText x <> toPlainText y+ _ -> mempty+++-- This instance mirrors what is expected in the spec tests.+instance Rangeable (Html a) => IsInline (Html a) where+ lineBreak = htmlInline "br" Nothing <> nl+ softBreak = nl+ str t = htmlText t+ entity t = case lookupEntity (T.drop 1 t) of+ Just t' -> htmlText t'+ Nothing -> htmlRaw t+ escapedChar c = htmlText (T.singleton c)+ emph ils = htmlInline "em" (Just ils)+ strong ils = htmlInline "strong" (Just ils)+ link target title ils =+ addAttribute ("href", escapeURI target) .+ (if T.null title+ then id+ else addAttribute ("title", title)) $+ htmlInline "a" (Just ils)+ image target title ils =+ addAttribute ("src", escapeURI target) .+ addAttribute ("alt", toPlainText ils) .+ (if T.null title+ then id+ else addAttribute ("title", title)) $+ htmlInline "img" Nothing+ code t = htmlInline "code" (Just (htmlText t))+ rawInline f t+ | f == Format "html" = htmlRaw t+ | otherwise = mempty++instance IsInline (Html a) => IsBlock (Html a) (Html a) where+ paragraph ils = htmlBlock "p" (Just ils)+ plain ils = ils <> nl+ thematicBreak = htmlBlock "hr" Nothing+ blockQuote bs = htmlBlock "blockquote" $ Just (nl <> bs)+ codeBlock info t =+ htmlBlock "pre" $ Just $+ (if T.null lang+ then id+ else addAttribute ("class", "language-" <> lang)) $+ htmlInline "code" $ Just (htmlText t)+ where lang = T.takeWhile (not . isSpace) info+ heading level ils = htmlBlock h (Just ils)+ where h = case level of+ 1 -> "h1"+ 2 -> "h2"+ 3 -> "h3"+ 4 -> "h4"+ 5 -> "h5"+ 6 -> "h6"+ _ -> "p"+ rawBlock f t+ | f == Format "html" = htmlRaw t+ | otherwise = mempty+ referenceLinkDefinition _ _ = mempty+ list (BulletList _) lSpacing items =+ htmlBlock "ul" $ Just (nl <> mconcat (map li items))+ where li x = htmlBlock "li" $+ Just ((if lSpacing == TightList+ then mempty+ else nl) <> x)+ list (OrderedList startnum enumtype _delimtype) lSpacing items =+ (if startnum /= 1+ then addAttribute ("start", T.pack (show startnum))+ else id) .+ (case enumtype of+ Decimal -> id+ UpperAlpha -> addAttribute ("type", "A")+ LowerAlpha -> addAttribute ("type", "a")+ UpperRoman -> addAttribute ("type", "I")+ LowerRoman -> addAttribute ("type", "i"))+ $ htmlBlock "ol" $+ Just (nl <> mconcat (map li items))+ where li x = htmlBlock "li" $+ Just ((if lSpacing == TightList+ then mempty+ else nl) <> x)++nl :: Html a+nl = htmlRaw "\n"++instance Rangeable (Html ()) where+ ranged _ x = x++instance Rangeable (Html SourceRange) where+ ranged sr x = addAttribute ("data-sourcepos", T.pack (show sr)) x++++htmlInline :: Text -> Maybe (Html a) -> Html a+htmlInline tagname mbcontents = HtmlElement InlineElement tagname [] mbcontents++htmlBlock :: Text -> Maybe (Html a) -> Html a+htmlBlock tagname mbcontents = HtmlElement BlockElement tagname [] mbcontents++htmlText :: Text -> Html a+htmlText = HtmlText++htmlRaw :: Text -> Html a+htmlRaw = HtmlRaw++addAttribute :: Attribute -> Html a -> Html a+addAttribute attr (HtmlElement eltType tagname attrs mbcontents) =+ HtmlElement eltType tagname (incorporateAttribute attr attrs) mbcontents+addAttribute attr (HtmlText t)+ = HtmlElement InlineElement "span" [attr] $ Just (HtmlText t)+addAttribute _ elt = elt++incorporateAttribute :: Attribute -> [Attribute] -> [Attribute]+incorporateAttribute (k, v) as =+ case lookup k as of+ Nothing -> (k, v) : as+ Just v' -> (if k == "class"+ then ("class", v <> " " <> v')+ else (k, v')) :+ filter (\(x, _) -> x /= k) as++renderHtml :: Html a -> TL.Text+renderHtml = {-# SCC renderHtml #-} toLazyText . toBuilder++toBuilder :: Html a -> Builder+toBuilder (HtmlNull) = mempty+toBuilder (HtmlConcat x y) = toBuilder x <> toBuilder y+toBuilder (HtmlRaw t) = fromText t+toBuilder (HtmlText t) = escapeHtml t+toBuilder (HtmlElement eltType tagname attrs mbcontents) =+ "<" <> fromText tagname <> mconcat (map toAttr attrs) <> filling <> nl'+ where+ toAttr (x,y) = " " <> fromText x <> "=\"" <> escapeHtml y <> "\""+ nl' = case eltType of+ BlockElement -> "\n"+ _ -> mempty+ filling = case mbcontents of+ Nothing -> " />"+ Just cont -> ">" <> toBuilder cont <> "</" <>+ fromText tagname <> ">"++escapeHtml :: Text -> Builder+escapeHtml t =+ case T.uncons post of+ Just (c, rest) -> fromText pre <> escapeHtmlChar c <> escapeHtml rest+ Nothing -> fromText pre+ where+ (pre,post) = T.break needsEscaping t+ needsEscaping '<' = True+ needsEscaping '>' = True+ needsEscaping '&' = True+ needsEscaping '"' = True+ needsEscaping _ = False++escapeHtmlChar :: Char -> Builder+escapeHtmlChar '<' = "<"+escapeHtmlChar '>' = ">"+escapeHtmlChar '&' = "&"+escapeHtmlChar '"' = """+escapeHtmlChar c = singleton c++escapeURI :: Text -> Text+escapeURI = mconcat . map escapeURIChar . B.unpack . encodeUtf8++escapeURIChar :: Char -> Text+escapeURIChar c+ | isEscapable c = T.singleton '%' <> T.pack (printf "%02X" (ord c))+ | otherwise = T.singleton c+ where isEscapable d = not (isAscii d && isAlphaNum d)+ && d `notElem` ['%','/','?',':','@','-','.','_','~','&',+ '#','!','$','\'','(',')','*','+',',',+ ';','=']+
+ src/Commonmark/Inlines.hs view
@@ -0,0 +1,945 @@+{-# LANGUAGE RankNTypes #-}+{-# LANGUAGE CPP #-}+{-# LANGUAGE LambdaCase #-}+{-# LANGUAGE BangPatterns #-}+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE TupleSections #-}+{-# LANGUAGE ScopedTypeVariables #-}+{-# LANGUAGE StrictData #-}++module Commonmark.Inlines+ ( mkInlineParser+ , defaultInlineParser+ , IPState+ , InlineParser+ , getReferenceMap+ , FormattingSpec(..)+ , defaultFormattingSpecs+ , BracketedSpec(..)+ , defaultBracketedSpecs+ , imageSpec+ , linkSpec+ , pLinkLabel+ , pLinkDestination+ , pLinkTitle+ , pEscaped+ , processEmphasis+ , processBrackets+ , pBacktickSpan+ , normalizeCodeSpan+ , withAttributes+ )+where++import Commonmark.Tag (htmlTag, Enders, defaultEnders)+import Commonmark.Tokens+import Commonmark.TokParsers+import Commonmark.ReferenceMap+import Commonmark.Types+import Control.Monad (guard, mzero)+import Control.Monad.Trans.State.Strict+import Data.List (foldl')+import Data.Char (isAscii, isLetter)+import qualified Data.IntMap.Strict as IntMap+import qualified Data.Map.Strict as M+import Data.Maybe (isJust, mapMaybe)+import qualified Data.Set as Set+#if !MIN_VERSION_base(4,11,0)+import Data.Monoid ((<>))+#endif+import Data.Text (Text)+import qualified Data.Text as T+import Commonmark.Entity (unEntity, charEntity, numEntity)+import Text.Parsec hiding (State, space)+import Text.Parsec.Pos++mkInlineParser :: (Monad m, IsInline a)+ => [BracketedSpec a]+ -> [FormattingSpec a]+ -> [InlineParser m a]+ -> [InlineParser m Attributes]+ -> ReferenceMap+ -> [Tok]+ -> m (Either ParseError a)+mkInlineParser bracketedSpecs formattingSpecs ilParsers attrParsers rm toks = do+ let iswhite t = hasType Spaces t || hasType LineEnd t+ let attrParser = choice attrParsers+ let toks' = dropWhile iswhite . reverse . dropWhile iswhite . reverse $ toks+ res <- {-# SCC parseChunks #-} evalStateT+ (parseChunks bracketedSpecs formattingSpecs ilParsers+ attrParser rm toks') defaultEnders+ return $!+ case res of+ Left err -> Left err+ Right chunks ->+ (Right .+ unChunks .+ processEmphasis .+ processBrackets bracketedSpecs rm) chunks++defaultInlineParser :: (Monad m, IsInline a) => InlineParser m a+defaultInlineParser =+ {-# SCC defaultInlineParser #-} try $ do+ tok@(Tok toktype _ t) <- anyTok+ case toktype of+ WordChars -> return $ str t+ LineEnd -> return softBreak+ Spaces -> doBreak (T.length t) <|> return (str t)+ UnicodeSpace -> return $ str t+ Symbol '\\' -> option (str "\\") doEscape+ Symbol '`' -> doCodeSpan tok+ Symbol '&' -> option (str "&") doEntity+ Symbol '<' -> option (str "<") (doAutolink <|> doHtml tok)+ _ -> mzero+ where+ doBreak len+ | len >= 2 = lineBreak <$ satisfyTok (hasType LineEnd)+ | otherwise = mempty <$ lookAhead (satisfyTok (hasType LineEnd))+ doEscape = do+ tok <- satisfyTok+ (\case+ Tok (Symbol c) _ _ -> isAscii c+ Tok LineEnd _ _ -> True+ _ -> False)+ case tok of+ Tok (Symbol c) _ _ -> return $ escapedChar c+ Tok LineEnd _ _ -> return lineBreak+ _ -> fail "Should not happen"+ doEntity = do+ ent <- numEntity <|> charEntity+ return (entity ("&" <> untokenize ent))+ doAutolink = try $ do+ (target, lab) <- pUri <|> pEmail+ symbol '>'+ return $ link target "" (str lab)+ doHtml tok = rawInline (Format "html") . untokenize . (tok:) <$>+ try htmlTag+ doCodeSpan tok = pBacktickSpan tok >>=+ \case+ Left ticks -> return $ str (untokenize ticks)+ Right codetoks -> return $ code . normalizeCodeSpan . untokenize $+ codetoks++unChunks :: IsInline a => [Chunk a] -> a+unChunks = {-# SCC unChunks #-} foldl' mappend mempty . go+ where+ go [] = []+ go (c:cs) =+ let (f, rest) =+ case cs of+ (Chunk (AddAttributes attrs) _pos _ts : ds) ->+ (addAttributes attrs, ds)+ _ -> (id, cs) in+ case chunkType c of+ AddAttributes _ -> go rest+ Delim{ delimType = ch, delimSpec = mbspec } -> x : go rest+ where !x = f (ranged range (str txt))+ txt = untokenize $ alterToks $ chunkToks c+ alterToks =+ case formattingWhenUnmatched <$> mbspec of+ Just ch' | ch' /= ch ->+ map (\t -> t{ tokContents =+ T.map (const ch') (tokContents t) })+ _ -> id+ range = SourceRange+ [(chunkPos c,+ incSourceColumn (chunkPos c) (T.length txt))]+ Parsed ils -> x : go rest+ where !x = f ils++parseChunks :: (Monad m, IsInline a)+ => [BracketedSpec a]+ -> [FormattingSpec a]+ -> [InlineParser m a]+ -> InlineParser m Attributes+ -> ReferenceMap+ -> [Tok]+ -> StateT Enders m (Either ParseError [Chunk a])+parseChunks bspecs specs ilParsers attrParser rm ts =+ runParserT+ (do case ts of+ t:_ -> setPosition (tokPos t)+ [] -> return ()+ many (pChunk specmap attrParser ilParsers isDelimChar) <* eof)+ IPState{ backtickSpans = getBacktickSpans ts,+ ipReferenceMap = rm,+ precedingTokTypes = precedingTokTypeMap,+ attributeParser = attrParser }+ "source" ts+ where+ isDelimChar = (`Set.member` delimcharset)+ !delimcharset = Set.fromList delimchars+ delimchars = '[' : ']' : suffixchars +++ prefixchars ++ M.keys specmap+ specmap = mkFormattingSpecMap specs+ prefixchars = mapMaybe bracketedPrefix bspecs+ suffixchars = mapMaybe bracketedSuffixEnd bspecs+ precedingTokTypeMap = {-# SCC precedingTokTypeMap #-}fst $! foldl' go (mempty, LineEnd) ts+ go (!m, !prevTy) (Tok !ty !pos _) =+ case ty of+ Symbol c | isDelimChar c -> (M.insert pos prevTy m, ty)+ _ -> (m, ty)++data Chunk a = Chunk+ { chunkType :: ChunkType a+ , chunkPos :: !SourcePos+ , chunkToks :: [Tok]+ } deriving Show++data ChunkType a =+ Delim{ delimType :: !Char+ , delimCanOpen :: !Bool+ , delimCanClose :: !Bool+ , delimLength :: !Int+ , delimSpec :: Maybe (FormattingSpec a)+ }+ | Parsed a+ | AddAttributes Attributes+ deriving Show++data IPState m = IPState+ { backtickSpans :: IntMap.IntMap [SourcePos]+ -- record of lengths of+ -- backtick spans so we don't scan in vain+ , ipReferenceMap :: !ReferenceMap+ , precedingTokTypes :: M.Map SourcePos TokType+ , attributeParser :: InlineParser m Attributes+ }++type InlineParser m = ParsecT [Tok] (IPState m) (StateT Enders m)++--- Formatting specs:++-- ^ Specifies delimiters for formatting, e.g. strong emphasis.+data FormattingSpec il = FormattingSpec+ { formattingDelimChar :: !Char+ -- ^ Character that triggers formatting+ , formattingIntraWord :: !Bool+ -- ^ True if formatting can start/end in a word+ , formattingIgnorePunctuation :: !Bool+ -- ^ Treat punctuation like letters for+ -- purposes of computing can open/can close+ , formattingSingleMatch :: Maybe (il -> il)+ -- ^ Constructor to use for text between+ -- single delimiters.+ , formattingDoubleMatch :: Maybe (il -> il)+ -- ^ Constructor to use for text between+ -- double delimiters.+ , formattingWhenUnmatched :: !Char -- ^ Fallback when not matched.+ }++instance Show (FormattingSpec il) where+ show _ = "<FormattingSpec>"++type FormattingSpecMap il = M.Map Char (FormattingSpec il)++defaultFormattingSpecs :: IsInline il => [FormattingSpec il]+defaultFormattingSpecs =+ [ FormattingSpec '*' True False (Just emph) (Just strong) '*'+ , FormattingSpec '_' False False (Just emph) (Just strong) '_'+ ]++mkFormattingSpecMap :: [FormattingSpec il] -> FormattingSpecMap il+mkFormattingSpecMap fs = M.fromList [(formattingDelimChar s, s) | s <- fs]++--- Bracketed specs:++-- ^ Defines an inline element between square brackets.+data BracketedSpec il = BracketedSpec+ { bracketedName :: !Text -- ^ Name of bracketed text type.+ , bracketedNests :: !Bool -- ^ True if this can be nested.+ , bracketedPrefix :: Maybe Char -- ^ Prefix character.+ , bracketedSuffixEnd :: Maybe Char -- ^ Suffix character.+ , bracketedSuffix :: ReferenceMap+ -> Text+ -> Parsec [Tok] () (il -> il)+ -- ^ Parser for suffix after+ -- brackets. Returns a constructor.+ -- Second parameter is the raw key.+ }++instance Show (BracketedSpec il) where+ show s = "<BracketedSpec " ++ show (bracketedName s) ++ ">"++-- It's important that specs with prefix chars come first:+defaultBracketedSpecs :: IsInline il+ => [BracketedSpec il]+defaultBracketedSpecs =+ [ imageSpec+ , linkSpec+ ]++linkSpec :: IsInline il => BracketedSpec il+linkSpec = BracketedSpec+ { bracketedName = "Link"+ , bracketedNests = False -- links don't nest inside links+ , bracketedPrefix = Nothing+ , bracketedSuffixEnd = Just ')'+ , bracketedSuffix = pLinkSuffix+ }++imageSpec :: IsInline il => BracketedSpec il+imageSpec = BracketedSpec+ { bracketedName = "Image"+ , bracketedNests = True+ , bracketedPrefix = Just '!'+ , bracketedSuffixEnd = Just ')'+ , bracketedSuffix = pImageSuffix+ }++pLinkSuffix :: IsInline il+ => ReferenceMap -> Text -> Parsec [Tok] s (il -> il)+pLinkSuffix rm key = do+ LinkInfo target title attrs <- pLink rm key+ return $! addAttributes attrs . link target title++pImageSuffix :: IsInline il+ => ReferenceMap -> Text -> Parsec [Tok] s (il -> il)+pImageSuffix rm key = do+ LinkInfo target title attrs <- pLink rm key+ return $! addAttributes attrs . image target title++---++-- Construct a map of n-length backtick spans, with source positions,+-- so we can avoid scanning forward when it will be fruitless.+getBacktickSpans :: [Tok] -> IntMap.IntMap [SourcePos]+getBacktickSpans = go 0 (initialPos "")+ where+ go :: Int -> SourcePos -> [Tok] -> IntMap.IntMap [SourcePos]+ go n pos []+ | n > 0 = IntMap.singleton n [pos]+ | otherwise = IntMap.empty+ go n pos (t:ts) =+ case tokType t of+ Symbol '`'+ | n > 0 -> go (n+1) pos ts+ | otherwise -> go (n+1) (tokPos t) ts+ _ | n > 0 -> IntMap.alter (\x ->+ case x of+ Nothing -> Just [pos]+ Just ps -> Just (pos:ps)) n $ go 0 pos ts+ | otherwise -> go 0 pos ts++pChunk :: (IsInline a, Monad m)+ => FormattingSpecMap a+ -> InlineParser m Attributes+ -> [InlineParser m a]+ -> (Char -> Bool)+ -> InlineParser m (Chunk a)+pChunk specmap attrParser ilParsers isDelimChar =+ do pos <- getPosition+ (res, ts) <- withRaw $+ {-# SCC attrParser #-} (AddAttributes <$> attrParser)+ <|>+ {-# SCC pInline #-} (Parsed <$> pInline ilParsers)+ return $! Chunk res pos ts+ <|> {-# SCC pDelimChunk #-} pDelimChunk specmap isDelimChar+ <|> (do t <- anyTok+ endpos <- getPosition+ return $! Chunk+ (Parsed $ ranged (SourceRange [(tokPos t,endpos)])+ (str $ tokContents t))+ (tokPos t) [t])++pDelimChunk :: (IsInline a, Monad m)+ => FormattingSpecMap a+ -> (Char -> Bool)+ -> InlineParser m (Chunk a)+pDelimChunk specmap isDelimChar = do+ tok@(Tok (Symbol !c) !pos _) <-+ satisfyTok (\case+ Tok (Symbol c) _ _ -> isDelimChar c+ _ -> False)+ let !mbspec = M.lookup c specmap+ more <- if isJust mbspec+ then many $ symbol c+ else return []+ let toks = tok:more+ st <- getState+ next <- option LineEnd (tokType <$> lookAhead anyTok)+ let precedingTokType = M.lookup pos (precedingTokTypes st)+ let precededByWhitespace = case precedingTokType of+ Just Spaces -> True+ Just UnicodeSpace -> True+ Just LineEnd -> True+ _ -> False+ let precededByPunctuation =+ case formattingIgnorePunctuation <$> mbspec of+ Just True -> False+ _ -> case precedingTokType of+ Just (Symbol _) -> True+ _ -> False+ let followedByWhitespace = next == Spaces ||+ next == LineEnd ||+ next == UnicodeSpace+ let followedByPunctuation =+ case formattingIgnorePunctuation <$> mbspec of+ Just True -> False+ _ -> not followedByWhitespace && next /= WordChars+ let leftFlanking = not followedByWhitespace &&+ (not followedByPunctuation ||+ precededByWhitespace ||+ precededByPunctuation)+ let rightFlanking = not precededByWhitespace &&+ (not precededByPunctuation ||+ followedByWhitespace ||+ followedByPunctuation)+ let !canOpen =+ leftFlanking &&+ (maybe True formattingIntraWord mbspec ||+ not rightFlanking ||+ precededByPunctuation)+ let !canClose =+ rightFlanking &&+ (maybe True formattingIntraWord mbspec ||+ not leftFlanking ||+ followedByPunctuation)++ let !len = length toks+ return $! Chunk Delim{ delimType = c+ , delimCanOpen = canOpen+ , delimCanClose = canClose+ , delimSpec = mbspec+ , delimLength = len+ } pos toks++withAttributes :: (IsInline a, Monad m) => InlineParser m a -> InlineParser m a+withAttributes p = do+ x <- p+ attrParser <- attributeParser <$> getState+ option x $ (\attr -> addAttributes attr x) <$> attrParser++pInline :: (IsInline a, Monad m)+ => [InlineParser m a]+ -> InlineParser m a+pInline ilParsers =+ mconcat <$> many1 oneInline+ where+ oneInline = withAttributes $ do+ toks <- getInput+ res <- choice ilParsers+ endpos <- getPosition+ let range = rangeFromToks+ (takeWhile ((< endpos) . tokPos) toks) endpos+ return $! ranged range res++rangeFromToks :: [Tok] -> SourcePos -> SourceRange+rangeFromToks [] _ = SourceRange mempty+rangeFromToks (!z:zs) !endpos+ | sourceLine (tokPos z) == sourceLine endpos+ = SourceRange [(tokPos z, endpos)]+ | otherwise+ = SourceRange $ go (z:zs)+ where+ go ts =+ case break (hasType LineEnd) ts of+ ([], []) -> []+ ([], _:ys) -> go ys+ (!x:_, []) -> [(tokPos x, endpos)]+ (!x:_, !y:ys) ->+ case ys of+ (Tok _ !pos _ : _) | sourceColumn pos == 1 -> go (x:ys)+ _ -> (tokPos x, tokPos y) : go ys++getReferenceMap :: Monad m => InlineParser m ReferenceMap+getReferenceMap = ipReferenceMap <$> getState++pBacktickSpan :: Monad m+ => Tok -> InlineParser m (Either [Tok] [Tok])+pBacktickSpan tok = do+ ts <- (tok:) <$> many (symbol '`')+ let numticks = length ts+ st' <- getState+ case dropWhile (<= tokPos tok) <$> IntMap.lookup numticks (backtickSpans st') of+ Just (pos'':ps) -> do+ codetoks <- many $ satisfyTok (\tok' -> tokPos tok' < pos'')+ backticks <- many $ satisfyTok (hasType (Symbol '`'))+ guard $ length backticks == numticks+ updateState $ \st ->+ st{ backtickSpans = IntMap.insert numticks ps (backtickSpans st) }+ return $ Right codetoks+ _ -> return $ Left ts++normalizeCodeSpan :: Text -> Text+normalizeCodeSpan = removeSurroundingSpace . T.map nltosp+ where+ nltosp '\n' = ' '+ nltosp c = c+ removeSurroundingSpace s+ | not (T.null s)+ , not (T.all (== ' ') s)+ , T.head s == ' '+ , T.last s == ' ' = T.drop 1 $ T.dropEnd 1 s+ | otherwise = s++pUri :: Monad m => InlineParser m (Text, Text)+pUri = try $ do+ s <- pScheme+ _ <- symbol ':'+ let isURITok t =+ case tokType t of+ Spaces -> False+ LineEnd -> False+ (Symbol c) -> c > ' ' && c /= '<' && c /= '>'+ _ -> True+ ts <- many $ satisfyTok isURITok+ let uri = s <> ":" <> untokenize ts+ return (uri, uri)++pScheme :: Monad m => InlineParser m Text+pScheme = do+ t <- satisfyWord (\t -> case T.uncons t of+ Nothing -> False+ Just (c,rest) -> isAscii c && isLetter c &&+ T.all isAscii rest)+ ts <- many $ oneOfToks [WordChars, Symbol '+', Symbol '.', Symbol '-']+ let s = untokenize (t:ts)+ let len = T.length s+ guard $ len >= 2 && len <= 32+ return s++pEmail :: Monad m => InlineParser m (Text, Text)+pEmail = do+ let isEmailSymbolTok (Tok (Symbol c) _ _) =+ c == '.' || c == '!' || c == '#' || c == '$' || c == '%' ||+ c == '&' || c == '\'' || c == '*' || c == '+' || c == '/' ||+ c == '=' || c == '?' || c == '^' || c == '_' || c == '`' ||+ c == '{' || c == '|' || c == '}' || c == '~' || c == '-' ||+ c == ']'+ isEmailSymbolTok _ = False+ name <- many1 $ satisfyWord (T.all isAscii)+ <|> satisfyTok isEmailSymbolTok+ _ <- symbol '@'+ let domainPart = do+ x <- satisfyWord (T.all isAscii)+ xs <- many $ (symbol '-' <* notFollowedBy eof <* notFollowedBy (symbol '.'))+ <|> satisfyWord (T.all isAscii)+ return $! (x:xs)+ d <- domainPart+ ds <- many (symbol '.' >> domainPart)+ let addr = untokenize name <> "@" <> T.intercalate "." (map untokenize (d:ds))+ return ("mailto:" <> addr, addr)++data DState a = DState+ { leftCursor :: Cursor (Chunk a)+ , rightCursor :: Cursor (Chunk a)+ , refmap :: ReferenceMap+ , stackBottoms :: M.Map Text SourcePos+ , absoluteBottom :: SourcePos+ }+++processEmphasis :: IsInline a => [Chunk a] -> [Chunk a]+processEmphasis xs =+ case break (\case+ (Chunk Delim{ delimCanOpen = True } _ _) -> True+ _ -> False) xs of+ (_,[]) -> xs+ (ys,z:zs) ->+ let startcursor = Cursor (Just z) (reverse ys) zs+ in processEm DState{ leftCursor = startcursor+ , rightCursor = startcursor+ , refmap = emptyReferenceMap+ , stackBottoms = mempty+ , absoluteBottom = chunkPos z }++{- for debugging:+prettyCursors :: (IsInline a) => Cursor (Chunk a) -> Cursor (Chunk a) -> String+prettyCursors left right =+ toS (reverse $ befores left) <> (maybe "" (inBrs . toS . (:[])) (center left)) <>+ if (chunkPos <$> center left) == (chunkPos <$> center right)+ then toS (afters right)+ else toS (middles) <> (maybe "" (inBrs . toS . (:[])) (center right)) <>+ toS (afters right)+ where middles = take (length (afters left) - length (afters right) -+ maybe 0 (const 1) (center right)) (afters left)+ toS = show . unChunks+ inBrs x = "{" ++ x ++ "}"+-}++processEm :: IsInline a => DState a -> [Chunk a]+processEm st =+ let left = leftCursor st+ right = rightCursor st+ bottoms = stackBottoms st+ in case -- trace (prettyCursors left right)+ (center left, center right) of+ (_, Nothing) -> reverse $+ case center (rightCursor st) of+ Nothing -> befores (rightCursor st)+ Just c -> c : befores (rightCursor st)++ (Nothing, Just (Chunk Delim{ delimType = c+ , delimCanClose = True } pos ts)) ->+ processEm+ st{ leftCursor = right+ , rightCursor = moveRight right+ , stackBottoms = M.insert+ (T.pack (c : show (length ts `mod` 3))) pos+ $ stackBottoms st+ }++ (Nothing, Just _) -> processEm+ st{ leftCursor = right+ , rightCursor = moveRight right+ }++ (Just chunk, Just closedelim@(Chunk Delim{ delimType = c,+ delimCanClose = True,+ delimSpec = Just spec}+ closePos ts))+ | delimsMatch chunk closedelim ->+ let closelen = length ts+ opendelim = chunk+ contents = takeWhile (\ch -> chunkPos ch /= closePos)+ (afters left)+ openlen = length (chunkToks opendelim)+ fallbackConstructor x = str (T.singleton c) <> x <>+ str (T.singleton c)+ (constructor, numtoks) =+ case (formattingSingleMatch spec, formattingDoubleMatch spec) of+ (_, Just c2)+ | min openlen closelen >= 2 -> (c2, 2)+ (Just c1, _) -> (c1, 1)+ _ -> (fallbackConstructor, 1)+ (openrest, opentoks) =+ splitAt (openlen - numtoks) (chunkToks opendelim)+ (closetoks, closerest) =+ splitAt numtoks (chunkToks closedelim)+ addnewopen = if null openrest+ then id+ else (opendelim{ chunkToks = openrest } :)+ addnewclose = if null closerest+ then id+ else (closedelim{ chunkToks = closerest } :)+ emphtoks = opentoks ++ concatMap chunkToks contents ++ closetoks+ newelt = Chunk+ (Parsed $+ ranged (rangeFromToks emphtoks+ (incSourceColumn (chunkPos closedelim)+ numtoks)) $+ constructor $ unChunks contents)+ (chunkPos chunk)+ emphtoks+ newcursor = Cursor (Just newelt)+ (addnewopen (befores left))+ (addnewclose (afters right))+ in processEm+ st{ rightCursor = moveRight newcursor+ , leftCursor = newcursor+ }++ | Just (chunkPos chunk) <=+ M.lookup (T.pack (c: show (length ts `mod` 3))) bottoms ->+ processEm+ st{ leftCursor = right+ , rightCursor = moveRight right+ , stackBottoms = M.insert+ (T.pack (c : show (length ts `mod` 3)))+ (chunkPos closedelim)+ $ stackBottoms st+ }++ | otherwise -> processEm st{ leftCursor = moveLeft left }++ _ -> processEm+ st{ rightCursor = moveRight right+ , leftCursor = moveRight left }+{-# SCC processEm #-}++-- This only applies to emph delims, not []:+delimsMatch :: IsInline a+ => Chunk a -> Chunk a -> Bool+delimsMatch (Chunk open@Delim{} _ opents) (Chunk close@Delim{} _ closets) =+ delimCanOpen open && delimCanClose close &&+ (delimType open == delimType close &&+ if (delimCanOpen open && delimCanClose open) ||+ (delimCanOpen close && delimCanClose close)+ then delimLength close `mod` 3 == 0 ||+ (delimLength open + delimLength close) `mod` 3 /= 0+ else True) &&+ opents /= closets+delimsMatch _ _ = False++processBrackets :: IsInline a+ => [BracketedSpec a] -> ReferenceMap -> [Chunk a] -> [Chunk a]+processBrackets bracketedSpecs rm xs =+ case break (\case+ (Chunk Delim{ delimType = '[' } _ _) -> True+ _ -> False) xs of+ (_,[]) -> xs+ (ys,z:zs) ->+ let startcursor = Cursor (Just z) (reverse ys) zs+ in processBs bracketedSpecs+ DState{ leftCursor = startcursor+ , rightCursor = startcursor+ , refmap = rm+ , stackBottoms = mempty+ , absoluteBottom = chunkPos z+ }++data Cursor a = Cursor+ { center :: Maybe a+ , befores :: [a]+ , afters :: [a]+ }+ deriving Show++moveLeft :: Cursor a -> Cursor a+moveLeft (Cursor Nothing [] zs) = Cursor Nothing [] zs+moveLeft (Cursor Nothing (x:xs) zs) = Cursor (Just x) xs zs+moveLeft (Cursor (Just x) [] zs) = Cursor Nothing [] (x:zs)+moveLeft (Cursor (Just x) (y:ys) zs) = Cursor (Just y) ys (x:zs)+{-# INLINE moveLeft #-}++moveRight :: Cursor a -> Cursor a+moveRight (Cursor Nothing zs []) = Cursor Nothing zs []+moveRight (Cursor Nothing zs (x:xs)) = Cursor (Just x) zs xs+moveRight (Cursor (Just x) zs []) = Cursor Nothing (x:zs) []+moveRight (Cursor (Just x) zs (y:ys)) = Cursor (Just y) (x:zs) ys+{-# INLINE moveRight #-}++processBs :: IsInline a+ => [BracketedSpec a] -> DState a -> [Chunk a]+processBs bracketedSpecs st =+ let left = leftCursor st+ right = rightCursor st+ bottoms = stackBottoms st+ bottom = absoluteBottom st+ -- trace (prettyCursors left right) $ return $! ()+ in case (center left, center right) of+ (_, Nothing) -> reverse $+ case center (rightCursor st) of+ Nothing -> befores (rightCursor st)+ Just c -> c : befores (rightCursor st)++ (Nothing, Just chunk) ->+ processBs bracketedSpecs+ st{ leftCursor = moveRight right+ , rightCursor = moveRight right+ , absoluteBottom = chunkPos chunk+ }++ (Just chunk, Just chunk')+ | chunkPos chunk < bottom ->+ processBs bracketedSpecs+ st { leftCursor = moveRight right+ , rightCursor = moveRight right+ , absoluteBottom = chunkPos chunk'+ }++ (Just opener@(Chunk Delim{ delimCanOpen = True, delimType = '[' } _ _),+ Just closer@(Chunk Delim{ delimCanClose = True, delimType = ']'} closePos _)) ->+ let chunksinside = takeWhile (\ch -> chunkPos ch /= closePos)+ (afters left)+ isBracket (Chunk Delim{ delimType = c' } _ _) =+ c' == '[' || c' == ']'+ isBracket _ = False+ key = if any isBracket chunksinside+ then ""+ else+ case untokenize (concatMap chunkToks chunksinside) of+ ks | T.length ks <= 999 -> ks+ _ -> ""+ prefixChar = case befores left of+ Chunk Delim{delimType = c} _ [_] : _+ -> Just c+ _ -> Nothing+ rm = refmap st++ specs = [s | s <- bracketedSpecs+ , case bracketedPrefix s of+ Just c -> Just c == prefixChar+ Nothing -> True+ , maybe True (< chunkPos opener)+ (M.lookup (bracketedName s) bottoms) ]++ suffixToks = mconcat (map chunkToks (afters right))++ suffixPos = incSourceColumn closePos 1++ in case parse+ (withRaw+ (do setPosition suffixPos+ (spec, constructor) <- choice $+ map (\s -> (s,) <$> bracketedSuffix s rm key)+ specs+ pos <- getPosition+ return (spec, constructor, pos)))+ "" suffixToks of+ Left _ -> -- match but no link/image+ processBs bracketedSpecs+ st{ leftCursor = moveLeft (leftCursor st)+ , rightCursor = fixSingleQuote $+ moveRight (rightCursor st) }+ Right ((spec, constructor, newpos), desttoks) ->+ let left' = case bracketedPrefix spec of+ Just _ -> moveLeft left+ Nothing -> left+ openers = case bracketedPrefix spec of+ Just _ -> maybe id (:) (center left')+ [opener]+ Nothing -> [opener]+ openerPos = case openers of+ (x:_) -> chunkPos x+ _ -> chunkPos opener+ elttoks = concatMap chunkToks+ (openers ++ chunksinside ++ [closer])+ ++ desttoks+ elt = ranged (rangeFromToks elttoks newpos)+ $ constructor $ unChunks $+ processEmphasis chunksinside+ eltchunk = Chunk (Parsed elt) openerPos elttoks+ afterchunks = dropWhile ((< newpos) . chunkPos)+ (afters right)+ afterchunkpos = case afterchunks of+ [] -> newpos+ (ch:_) -> chunkPos ch+ -- in the event that newpos is not at the+ -- beginning of a chunk, we need to add+ -- some tokens from that chunk...+ missingtoks =+ [t | t <- suffixToks+ , tokPos t < afterchunkpos+ , tokPos t >= newpos]+ addMissing =+ if null missingtoks+ then id+ else (Chunk (Parsed (ranged+ (rangeFromToks missingtoks newpos)+ (str (untokenize missingtoks))))+ newpos missingtoks :)++ in case addMissing afterchunks of+ [] -> processBs bracketedSpecs+ st{ rightCursor = Cursor Nothing+ (eltchunk : befores left') [] }+ (y:ys) ->+ let lbs = befores left'+ in processBs bracketedSpecs st{+ leftCursor =+ Cursor (Just eltchunk) lbs (y:ys)+ , rightCursor = fixSingleQuote $+ Cursor (Just y) (eltchunk:lbs) ys+ , stackBottoms =+ -- if a link, we need to ensure that+ -- nothing matches as link containing it+ if bracketedNests spec+ then stackBottoms st+ else M.insert (bracketedName spec)+ (chunkPos opener)+ $ stackBottoms st+ }+++ (_, Just (Chunk Delim{ delimCanClose = True, delimType = ']' } _ _))+ -> processBs bracketedSpecs st{ leftCursor = moveLeft left }++ (Just _, Just (Chunk Delim{ delimCanOpen = True, delimType = '[' } _ _))+ -> processBs bracketedSpecs+ st{ leftCursor = right+ , rightCursor = moveRight right }++ (_, _) -> processBs bracketedSpecs+ st{ rightCursor = moveRight right }+{-# SCC processBs #-}+++-- This just changes a single quote Delim that occurs+-- after ) or ] so that canOpen = False. This is an ad hoc+-- way to prevent "[a]'s dog'" from being parsed wrong.+-- Ideally there'd be a way to put this restriction in+-- the FormattingSpec for smart ', but currently there+-- isn't.+fixSingleQuote :: Cursor (Chunk a) -> Cursor (Chunk a)+fixSingleQuote+ (Cursor (Just (Chunk d@(Delim{ delimType = '\'' }) pos toks)) xs ys) =+ Cursor (Just (Chunk d{ delimCanOpen = False } pos toks)) xs ys+fixSingleQuote cursor = cursor++pLink :: ReferenceMap -> Text -> Parsec [Tok] s LinkInfo+pLink rm key = do+ pInlineLink <|> pReferenceLink rm key++pInlineLink :: Monad m => ParsecT [Tok] s m LinkInfo+pInlineLink = try $ do+ _ <- symbol '('+ optional whitespace+ target <- unEntity <$> pLinkDestination+ optional whitespace+ title <- option "" $+ unEntity <$> (pLinkTitle <* optional whitespace)+ _ <- symbol ')'+ return $! LinkInfo { linkDestination = target+ , linkTitle = title+ , linkAttributes = mempty }++pLinkDestination :: Monad m => ParsecT [Tok] s m [Tok]+pLinkDestination = pAngleDest <|> pNormalDest 0+ where+ pAngleDest = do+ _ <- symbol '<'+ res <- many (noneOfToks [Symbol '<', Symbol '>', Symbol '\\',+ LineEnd] <|> pEscaped)+ _ <- symbol '>'+ return res++ pNormalDest (numparens :: Int) = do+ res <- pNormalDest' numparens+ if null res+ then res <$ lookAhead (symbol ')')+ else return res++ pNormalDest' numparens+ | numparens > 32 = mzero+ | otherwise = (do+ t <- satisfyTok (\case+ Tok (Symbol '\\') _ _ -> True+ Tok (Symbol ')') _ _ -> numparens >= 1+ Tok Spaces _ _ -> False+ Tok LineEnd _ _ -> False+ _ -> True)+ case t of+ Tok (Symbol '\\') _ _ -> do+ t' <- option t $ satisfyTok asciiSymbol+ (t':) <$> pNormalDest' numparens+ Tok (Symbol '(') _ _ -> (t:) <$> pNormalDest' (numparens + 1)+ Tok (Symbol ')') _ _ -> (t:) <$> pNormalDest' (numparens - 1)+ _ -> (t:) <$> pNormalDest' numparens)+ <|> ([] <$ guard (numparens == 0))++-- parses backslash + escapable character, or just backslash+pEscaped :: Monad m => ParsecT [Tok] s m Tok+pEscaped = do+ bs <- symbol '\\'+ option bs $ satisfyTok asciiSymbol <|> lineEnd++asciiSymbol :: Tok -> Bool+asciiSymbol (Tok (Symbol c) _ _) = isAscii c+asciiSymbol _ = False++pLinkTitle :: Monad m => ParsecT [Tok] s m [Tok]+pLinkTitle = inbetween '"' '"' <|> inbetween '\'' '\'' <|> inbetween '(' ')'++inbetween :: Monad m => Char -> Char -> ParsecT [Tok] s m [Tok]+inbetween op cl =+ try $ between (symbol op) (symbol cl)+ (many (pEscaped <|> noneOfToks [Symbol op, Symbol cl]))++pLinkLabel :: Monad m => ParsecT [Tok] s m Text+pLinkLabel = try $ do+ lab <- untokenize+ <$> try (between (symbol '[') (symbol ']')+ (snd <$> withRaw (many+ (pEscaped <|> noneOfToks [Symbol ']', Symbol '[']))))+ guard $ T.length lab <= 999+ return lab++pReferenceLink :: ReferenceMap -> Text -> Parsec [Tok] s LinkInfo+pReferenceLink rm key = do+ lab <- option key pLinkLabel+ let key' = if T.null lab+ then key+ else lab+ maybe mzero return $! lookupReference key' rm
+ src/Commonmark/Parser.hs view
@@ -0,0 +1,60 @@+module Commonmark.Parser+ ( commonmark+ , commonmarkWith+ , parseCommonmarkWith+ -- * Exported from "Text.Parsec.Error"+ , ParseError+ ) where++import Commonmark.Blocks+import Commonmark.Inlines+import Commonmark.Tokens+import Commonmark.Types+import Commonmark.Syntax (SyntaxSpec(..), defaultSyntaxSpec)+import Text.Parsec.Error (ParseError)+import Data.Functor.Identity (runIdentity)+import Data.Text (Text)++-- | Parse a commonmark document using the core syntax+-- elements.+-- To produce HTML, instantiate 'bl' with @'Html' ()@ (see+-- 'Commonmark.Html'.+-- If you want to add syntax extensions or run the parser in a+-- monadic context, use 'commonmarkWith'.+-- If you want to operate on tokenized input, use 'parseCommonmarkWith'.+commonmark :: IsBlock il bl+ => String -- ^ Name or path of input+ -> Text -- ^ Commonmark text input+ -> Either ParseError bl -- ^ Result or error+commonmark sourcename =+ runIdentity .+ parseCommonmarkWith defaultSyntaxSpec .+ tokenize sourcename++-- | Like 'commonmark', but allows specifying a custom syntax+-- and a monadic context (since some syntax extensions may+-- only be defined in certain monads, e.g. an extension for+-- include files may require IO).+commonmarkWith :: (Monad m, IsBlock il bl, IsInline il)+ => SyntaxSpec m il bl -- ^ Defines syntax+ -> String -- ^ Name or path of input+ -> Text -- ^ Commonmark text input+ -> m (Either ParseError bl) -- ^ Result or error+commonmarkWith syntax sourcename =+ parseCommonmarkWith syntax .+ tokenize sourcename++-- | Parse a tokenized commonmark document using specified+-- syntax elements. Use 'tokenize' to convert 'Text' into ['Tok'].+parseCommonmarkWith :: (Monad m, IsBlock il bl, IsInline il)+ => SyntaxSpec m il bl -- ^ Defines syntax+ -> [Tok] -- ^ Tokenized commonmark input+ -> m (Either ParseError bl) -- ^ Result or error+parseCommonmarkWith syntax =+ mkBlockParser (syntaxBlockSpecs syntax)+ (syntaxFinalParsers syntax)+ (mkInlineParser (syntaxBracketedSpecs syntax)+ (syntaxFormattingSpecs syntax)+ (syntaxInlineParsers syntax)+ (syntaxAttributeParsers syntax))+ (syntaxAttributeParsers syntax)
+ src/Commonmark/ReferenceMap.hs view
@@ -0,0 +1,57 @@+{-# LANGUAGE CPP #-}+{-# LANGUAGE BangPatterns #-}+module Commonmark.ReferenceMap+ ( ReferenceMap(..)+ , LinkInfo(..)+ , emptyReferenceMap+ , insertReference+ , lookupReference+ ) where+import Data.Text (Text)+import qualified Data.Text as T+import qualified Data.Map.Strict as M+import Data.Dynamic+import Commonmark.Types+#if !MIN_VERSION_base(4,13,0)+import Data.Typeable (Typeable)+#endif++-- | Lookup table for link references.+newtype ReferenceMap = ReferenceMap { unReferenceMap :: M.Map Text [Dynamic] }+ deriving (Show)++data LinkInfo = LinkInfo{ linkDestination :: !Text+ , linkTitle :: !Text+ , linkAttributes :: !Attributes }+ deriving (Show, Typeable)++emptyReferenceMap :: ReferenceMap+emptyReferenceMap = ReferenceMap M.empty++-- | Insert a link reference into a reference map.+insertReference :: Typeable a+ => Text -- ^ Reference label+ -> a+ -> ReferenceMap+ -> ReferenceMap+insertReference label x (ReferenceMap m) =+ ReferenceMap (M.insertWith (\new old -> old ++ new)+ (T.toCaseFold $! normalizeSpaces label) [toDyn x] m)++-- | Lookup a reference in a reference map. If there are several+-- values at this key, we return the first one in the list that+-- can be converted to an 'a'.+lookupReference :: Typeable a+ => Text -- ^ Reference label+ -> ReferenceMap+ -> Maybe a+lookupReference label (ReferenceMap m) =+ getFirst $! M.lookup (T.toCaseFold $! normalizeSpaces label) m+ where getFirst Nothing = Nothing+ getFirst (Just []) = Nothing+ getFirst (Just (x:xs)) = case fromDynamic x of+ Just !v -> Just v+ Nothing -> getFirst (Just xs)++normalizeSpaces :: Text -> Text+normalizeSpaces = T.unwords . T.words
+ src/Commonmark/SourceMap.hs view
@@ -0,0 +1,133 @@+{-# LANGUAGE CPP #-}+{-# LANGUAGE FlexibleContexts #-}+{-# LANGUAGE UndecidableInstances #-}+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE FlexibleInstances #-}+{-# LANGUAGE MultiParamTypeClasses #-}+{-# LANGUAGE GeneralizedNewtypeDeriving #-}+{-# LANGUAGE LambdaCase #-}+module Commonmark.SourceMap+ ( SourceMap(..)+ , WithSourceMap(..)+ , runWithSourceMap+ , addName+ )+where+import Data.Text (Text)+import qualified Data.Text as T+import qualified Data.Map.Strict as M+import qualified Data.Sequence as Seq+import Commonmark.Types+import Control.Monad.Trans.State+#if !MIN_VERSION_base(4,11,0)+import Data.Semigroup (Semigroup, (<>))+#endif++-- | A map from source positions to a pair of sequences:+-- first, elements that start at that position; then, elements+-- that end at that position.+newtype SourceMap =+ SourceMap { unSourceMap :: M.Map SourcePos (Seq.Seq Text, Seq.Seq Text) }+ deriving (Show)++instance Semigroup SourceMap where+ (SourceMap m1) <> (SourceMap m2) =+ SourceMap (M.unionWith combine m1 m2)++instance Monoid SourceMap where+ mempty = SourceMap mempty+ mappend = (<>)++instance HasAttributes (WithSourceMap a) where+ addAttributes _attrs x = x+++combine :: (Seq.Seq Text, Seq.Seq Text)+ -> (Seq.Seq Text, Seq.Seq Text)+ -> (Seq.Seq Text, Seq.Seq Text)+combine (s1,e1) (s2,e2) = (s1 <> s2, e1 <> e2)++-- | Use this when you want to extract a source map as well+-- as the parsed content.+newtype WithSourceMap a =+ WithSourceMap { unWithSourceMap :: State (Maybe Text, SourceMap) a }+ deriving (Functor, Applicative, Monad)++instance (Show a, Semigroup a) => Semigroup (WithSourceMap a) where+ (WithSourceMap x1) <> (WithSourceMap x2) =+ WithSourceMap ((<>) <$> x1 <*> x2)++instance (Show a, Semigroup a, Monoid a) => Monoid (WithSourceMap a) where+ mempty = WithSourceMap (return mempty)+ mappend = (<>)++instance (Show a, Monoid a) => Show (WithSourceMap a) where+ show (WithSourceMap x) = show $ evalState x mempty++-- | Extract a parsed value and a source map from a+-- 'WithSourceMap'.+runWithSourceMap :: (Show a, Monoid a)+ => WithSourceMap a -> (a, SourceMap)+runWithSourceMap (WithSourceMap x) = (v, sm)+ where (v, (_,sm)) = runState x (mempty, mempty)++addName :: Text -> WithSourceMap ()+addName name =+ WithSourceMap $ modify (\(_,sm) -> (Just name,sm))++instance (IsInline a, Semigroup a) => IsInline (WithSourceMap a) where+ lineBreak = lineBreak <$ addName "lineBreak"+ softBreak = softBreak <$ addName "softBreak"+ str t = str t <$ addName "str"+ entity t = entity t <$ addName "str"+ escapedChar c = escapedChar c <$ addName "escapedChar"+ emph x = (emph <$> x) <* addName "emph"+ strong x = (strong <$> x) <* addName "strong"+ link dest tit x = (link dest tit <$> x) <* addName "link"+ image dest tit x = (image dest tit <$> x) <* addName "image"+ code t = code t <$ addName "code"+ rawInline f t = rawInline f t <$ addName "rawInline"++instance (IsBlock b a, IsInline b, IsInline (WithSourceMap b), Semigroup a)+ => IsBlock (WithSourceMap b) (WithSourceMap a) where+ paragraph x = (paragraph <$> x) <* addName "paragraph"+ plain x = (plain <$> x) <* addName "plain"+ thematicBreak = thematicBreak <$ addName "thematicBreak"+ blockQuote x = (blockQuote <$> x) <* addName "blockQuote"+ codeBlock i t = codeBlock i t <$ addName "codeBlock"+ heading lev x = (heading lev <$> x) <*+ addName ("heading" <> T.pack (show lev))+ rawBlock f t = rawBlock f t <$ addName "rawBlock"+ referenceLinkDefinition k x = referenceLinkDefinition k x <$+ addName "referenceLinkDefinition"+ list lt ls items = (list lt ls <$> sequence items) <* addName "list"++instance (Rangeable a, Monoid a, Show a)+ => Rangeable (WithSourceMap a) where+ ranged (SourceRange rs) (WithSourceMap x) =+ WithSourceMap $+ do res <- x+ (mbt, SourceMap sm) <- get+ case mbt of+ Just t -> do+ let (starts, ends) = unzip rs+ let addStart = M.alter (\case+ Nothing ->+ Just (Seq.singleton t, mempty)+ Just (s,e) ->+ Just (t Seq.<| s, e))+ let addEnd = M.alter (\case+ Nothing ->+ Just (mempty, Seq.singleton t)+ Just (s,e) ->+ Just (s, e Seq.|> t))+ let sm' = foldr addStart sm starts+ let sm'' = foldr addEnd sm' ends+ put (mempty, SourceMap sm'')+ return $! res+ Nothing -> return $! res++instance ToPlainText a => ToPlainText (WithSourceMap a) where+ toPlainText (WithSourceMap x) =+ let v = evalState x (mempty, mempty)+ in toPlainText v
+ src/Commonmark/Syntax.hs view
@@ -0,0 +1,61 @@+{-# LANGUAGE CPP #-}+{-# LANGUAGE RankNTypes #-}+module Commonmark.Syntax+ ( SyntaxSpec(..)+ , defaultSyntaxSpec+ )+where++import Text.Parsec (ParsecT)+import Commonmark.Tokens (Tok)+import Commonmark.Types+import Commonmark.Blocks+import Commonmark.Inlines+#if !MIN_VERSION_base(4,11,0)+import Data.Semigroup+#endif++-- | A 'SyntaxSpec' defines a basic collection of syntax+-- elements or an extension. 'SyntaxSpec's can be composed+-- using monoidal 'mappend'.+data SyntaxSpec m il bl = SyntaxSpec+ { syntaxBlockSpecs :: [BlockSpec m il bl]+ -- ^ Defines block structure+ , syntaxBracketedSpecs :: [BracketedSpec il]+ -- ^ Defines bracketed inline containers (inli, image)+ , syntaxFormattingSpecs :: [FormattingSpec il]+ -- ^ Defines formatted inline containers (strong, emph)+ , syntaxInlineParsers :: [InlineParser m il]+ -- ^ Defines inline elements that don't contain inlines+ , syntaxFinalParsers :: [BlockParser m il bl bl]+ -- ^ Run at the end of document, e.g. to collect footnotes+ , syntaxAttributeParsers+ :: forall u m1 . Monad m1 => [ParsecT [Tok] u m1 Attributes]+ -- ^ Parse attributes+ }++instance Semigroup (SyntaxSpec m il bl) where+ SyntaxSpec bl1 br1 fo1 il1 fp1 ap1 <> SyntaxSpec bl2 br2 fo2 il2 fp2 ap2+ = SyntaxSpec (removeDuplicateBlockSpecs $ bl1 <> bl2)+ (br1 <> br2) (fo1 <> fo2) (il1 <> il2)+ (fp1 <> fp2) (ap1 <> ap2)+instance Monoid (SyntaxSpec m il bl) where+ mempty = SyntaxSpec mempty mempty mempty mempty mempty mempty+ mappend = (<>)++removeDuplicateBlockSpecs :: [BlockSpec m il bl] -> [BlockSpec m il bl]+removeDuplicateBlockSpecs [] = []+removeDuplicateBlockSpecs (b:bs) =+ b : removeDuplicateBlockSpecs (filter ((/= blockType b) . blockType) bs)++-- | Standard commonmark syntax.+defaultSyntaxSpec :: (Monad m, IsBlock il bl, IsInline il)+ => SyntaxSpec m il bl+defaultSyntaxSpec = SyntaxSpec+ { syntaxBlockSpecs = defaultBlockSpecs+ , syntaxBracketedSpecs = defaultBracketedSpecs+ , syntaxFormattingSpecs = defaultFormattingSpecs+ , syntaxInlineParsers = [defaultInlineParser]+ , syntaxFinalParsers = []+ , syntaxAttributeParsers = []+ }
+ src/Commonmark/Tag.hs view
@@ -0,0 +1,221 @@+{-# LANGUAGE OverloadedStrings #-}+module Commonmark.Tag+ ( htmlTag+ , htmlOpenTag+ , htmlClosingTag+ , htmlAttributeName+ , htmlAttributeValue+ , htmlDoubleQuotedAttributeValue+ , Enders+ , defaultEnders )+where+import Commonmark.Tokens+import Commonmark.TokParsers+import Control.Monad (liftM2, guard)+import Control.Monad.Trans.State.Strict+import Control.Monad.Trans.Class (lift)+import Data.Char (isAscii, isLetter)+import qualified Data.Text as T+import Text.Parsec hiding (State)++data Enders =+ Enders+ { scannedForCDATA :: !Bool+ , scannedForProcessingInstruction :: !Bool+ , scannedForDeclaration :: !Bool+ } deriving Show++defaultEnders :: Enders+defaultEnders = Enders { scannedForCDATA = False+ , scannedForProcessingInstruction = False+ , scannedForDeclaration = False }++(.&&.) :: (a -> Bool) -> (a -> Bool) -> (a -> Bool)+(.&&.) = liftM2 (&&)++-- A tag name consists of an ASCII letter followed by zero or more ASCII+-- letters, digits, or hyphens (-).+htmlTagName :: Monad m => ParsecT [Tok] s m [Tok]+htmlTagName = try $ do+ let isTagText = T.all isAscii+ let startsWithLetter t' = not (T.null t') && isLetter (T.head t')+ t <- satisfyWord (isTagText .&&. startsWithLetter)+ rest <- many (symbol '-' <|> satisfyWord isTagText)+ return (t:rest)++-- An attribute name consists of an ASCII letter, _, or :, followed by+-- zero or more ASCII letters, digits, _, ., :, or -. (Note: This is+-- the XML specification restricted to ASCII. HTML5 is laxer.)+htmlAttributeName :: Monad m => ParsecT [Tok] s m [Tok]+htmlAttributeName = try $ do+ let isTagText t' = T.all isAscii t'+ let startsWithLetter t' = not (T.null t') && isLetter (T.head t')+ t <- satisfyWord (startsWithLetter .&&. isTagText) <|>+ symbol '_' <|>+ symbol ':'+ rest <- many $ satisfyWord isTagText+ <|> symbol '_'+ <|> symbol '-'+ <|> symbol '.'+ <|> symbol ':'+ return (t:rest)++-- An attribute value specification consists of optional whitespace,+-- a = character, optional whitespace, and an attribute value.+htmlAttributeValueSpec :: Monad m => ParsecT [Tok] s m [Tok]+htmlAttributeValueSpec = try $ do+ sps1 <- option [] whitespace+ eq <- symbol '='+ sps2 <- option [] whitespace+ val <- htmlAttributeValue+ return $ sps1 ++ [eq] ++ sps2 ++ val++-- An attribute value consists of an unquoted attribute value,+-- a single-quoted attribute value, or a double-quoted attribute value.+htmlAttributeValue :: Monad m => ParsecT [Tok] s m [Tok]+htmlAttributeValue =+ htmlUnquotedAttributeValue <|>+ htmlSingleQuotedAttributeValue <|>+ htmlDoubleQuotedAttributeValue++-- An attribute consists of whitespace, an attribute name, and an optional+-- attribute value specification.+htmlAttribute :: Monad m => ParsecT [Tok] s m [Tok]+htmlAttribute = try $ do+ sps <- whitespace+ n <- htmlAttributeName+ val <- option [] htmlAttributeValueSpec+ return $ sps ++ n ++ val++-- An unquoted attribute value is a nonempty string of characters not+-- including spaces, ", ', =, <, >, or `.+htmlUnquotedAttributeValue :: Monad m => ParsecT [Tok] s m [Tok]+htmlUnquotedAttributeValue =+ many1 $ noneOfToks [Spaces, LineEnd, Symbol '<', Symbol '>',+ Symbol '=', Symbol '`', Symbol '\'', Symbol '"']++-- A single-quoted attribute value consists of ', zero or more characters+-- not including ', and a final '.+htmlSingleQuotedAttributeValue :: Monad m => ParsecT [Tok] s m [Tok]+htmlSingleQuotedAttributeValue = try $ do+ op <- symbol '\''+ contents <- many (satisfyTok (not . hasType (Symbol '\'')))+ cl <- symbol '\''+ return $ op : contents ++ [cl]++-- A double-quoted attribute value consists of ", zero or more characters+-- not including ", and a final ".+htmlDoubleQuotedAttributeValue :: Monad m => ParsecT [Tok] s m [Tok]+htmlDoubleQuotedAttributeValue = try $ do+ op <- symbol '"'+ contents <- many (satisfyTok (not . hasType (Symbol '"')))+ cl <- symbol '"'+ return $ op : contents ++ [cl]++-- | An open tag consists of a @<@ character, a tag name, zero or more+-- attributes, optional whitespace, an optional @/@ character, and a+-- @>@ character. This parses assumes that the opening @<@ has already+-- been parsed.+htmlOpenTag :: Monad m => ParsecT [Tok] s m [Tok]+htmlOpenTag = try $ do+ -- assume < has already been parsed+ n <- htmlTagName+ attrs <- concat <$> many htmlAttribute+ sps <- option [] whitespace+ sl <- option [] $ (:[]) <$> symbol '/'+ cl <- symbol '>'+ return $ n ++ attrs ++ sps ++ sl ++ [cl]++-- | A closing tag consists of the string @</@, a tag name, optional+-- whitespace, and the character @>@. This parser assumes that the+-- opening @<@ has already been parsed.+htmlClosingTag :: Monad m => ParsecT [Tok] s m [Tok]+htmlClosingTag = try $ do+ -- assume < has already been parsed+ op <- symbol '/'+ n <- htmlTagName+ sps <- option [] whitespace+ cl <- symbol '>'+ return $ op : n ++ sps ++ [cl]++-- An HTML comment consists of <!-- + text + -->, where text does not+-- start with > or ->, does not end with -, and does not contain --.+-- (See the HTML5 spec.)+htmlComment :: Monad m => ParsecT [Tok] s m [Tok]+htmlComment = try $ do+ -- assume < has already been parsed+ op <- sequence [ symbol '!'+ , symbol '-'+ , symbol '-' ]+ notFollowedBy $ do+ optional $ symbol '-'+ symbol '>'+ contents <- many $ satisfyTok (not . hasType (Symbol '-'))+ <|> try (symbol '-' <* notFollowedBy (symbol '-'))+ cl <- sequence [ symbol '-'+ , symbol '-'+ , symbol '>' ]+ return $ op ++ contents ++ cl++-- A processing instruction consists of the string <?, a string of+-- characters not including the string ?>, and the string ?>.+htmlProcessingInstruction :: Monad m+ => ParsecT [Tok] s (StateT Enders m) [Tok]+htmlProcessingInstruction = try $ do+ -- assume < has already been parsed+ let questionmark = symbol '?'+ op <- questionmark+ alreadyScanned <- lift $ gets scannedForProcessingInstruction+ guard $ not alreadyScanned+ contents <- many $ satisfyTok (not . hasType (Symbol '?'))+ <|> try (questionmark <*+ notFollowedBy (symbol '>'))+ lift $ modify $ \st -> st{ scannedForProcessingInstruction = True }+ cl <- sequence [ questionmark+ , symbol '>' ]+ return $ op : contents ++ cl++-- A declaration consists of the string <!, a name consisting of one or+-- more uppercase ASCII letters, whitespace, a string of characters not+-- including the character >, and the character >.+htmlDeclaration :: Monad m => ParsecT [Tok] s (StateT Enders m) [Tok]+htmlDeclaration = try $ do+ -- assume < has already been parsed+ op <- symbol '!'+ alreadyScanned <- lift $ gets scannedForDeclaration+ guard $ not alreadyScanned+ let isDeclName t = not (T.null t) && T.all (isAscii .&&. isLetter) t+ name <- satisfyWord isDeclName+ ws <- whitespace+ contents <- many (satisfyTok (not . hasType (Symbol '>')))+ lift $ modify $ \st -> st{ scannedForDeclaration = True }+ cl <- symbol '>'+ return $ op : name : ws ++ contents ++ [cl]++-- A CDATA section consists of the string <![CDATA[, a string of characters+-- not including the string ]]>, and the string ]]>.+htmlCDATASection :: Monad m => ParsecT [Tok] s (StateT Enders m) [Tok]+htmlCDATASection = try $ do+ -- assume < has already been parsed+ op <- sequence [ symbol '!'+ , symbol '['+ , satisfyWord (== "CDATA")+ , symbol '[' ]+ alreadyScanned <- lift $ gets scannedForCDATA+ guard $ not alreadyScanned+ let ender = try $ sequence [ symbol ']'+ , symbol ']'+ , symbol '>' ]+ contents <- many $ do+ notFollowedBy ender+ anyTok+ lift $ modify $ \st -> st{ scannedForCDATA = True }+ cl <- ender+ return $ op ++ contents ++ cl++-- An HTML tag consists of an open tag, a closing tag, an HTML comment,+-- a processing instruction, a declaration, or a CDATA section.+-- Assumes @<@ has already been parsed.+htmlTag :: Monad m => ParsecT [Tok] s (StateT Enders m) [Tok]+htmlTag = htmlOpenTag <|> htmlClosingTag <|> htmlComment <|>+ htmlProcessingInstruction <|> htmlDeclaration <|> htmlCDATASection
+ src/Commonmark/TokParsers.hs view
@@ -0,0 +1,194 @@+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE BangPatterns #-}+module Commonmark.TokParsers+ ( satisfyTok+ , satisfyWord+ , anyTok+ , anySymbol+ , symbol+ , whitespace+ , lineEnd+ , spaceTok+ , oneOfToks+ , noneOfToks+ , gobbleSpaces+ , gobbleUpToSpaces+ , withRaw+ , hasType+ , textIs+ , blankLine+ , restOfLine+ , isOneOfCI+ , nonindentSpaces+ , skipManyTill+ , skipWhile+ )+ where+import Control.Monad (mzero, void)+import Data.Text (Text)+import qualified Data.Text as T+import Text.Parsec+import Text.Parsec.Pos (updatePosString)+import Commonmark.Tokens++-- | Parses a single 'Tok' satisfying a predicate.+satisfyTok :: Monad m => (Tok -> Bool) -> ParsecT [Tok] s m Tok+satisfyTok f = tokenPrim (T.unpack . tokContents) updatePos matcher+ where matcher t | f t = Just t+ | otherwise = Nothing+ updatePos :: SourcePos -> Tok -> [Tok] -> SourcePos+ updatePos _spos _ (Tok _ !pos _ : _) = pos+ updatePos !spos (Tok _ _pos !t) [] =+ updatePosString spos (T.unpack t)+{-# INLINE satisfyTok #-}++-- | Parses any 'Tok'.+anyTok :: Monad m => ParsecT [Tok] s m Tok+anyTok = satisfyTok (const True)+{-# INLINE anyTok #-}++-- | Parses any 'Symbol' 'Tok'.+anySymbol :: Monad m => ParsecT [Tok] s m Tok+anySymbol = satisfyTok (\t -> case tokType t of+ Symbol _ -> True+ _ -> False)+{-# INLINE anySymbol #-}++-- | Parses a 'Symbol' with character @c@.+symbol :: Monad m => Char -> ParsecT [Tok] s m Tok+symbol c = satisfyTok (hasType (Symbol c))+{-# INLINE symbol #-}++-- | Parses a 'Tok' with one of the listed types.+oneOfToks :: Monad m => [TokType] -> ParsecT [Tok] s m Tok+oneOfToks toktypes = satisfyTok (hasTypeIn toktypes)+{-# INLINE oneOfToks #-}++-- | Parses a 'Tok' with none of the listed types.+noneOfToks :: Monad m => [TokType] -> ParsecT [Tok] s m Tok+noneOfToks toktypes = satisfyTok (not . hasTypeIn toktypes)+{-# INLINE noneOfToks #-}++-- | Parses one or more whitespace 'Tok's.+whitespace :: Monad m => ParsecT [Tok] s m [Tok]+whitespace = many1 $ satisfyTok (\t -> case tokType t of+ Spaces -> True+ LineEnd -> True+ _ -> False)+{-# INLINE whitespace #-}++-- | Parses a 'LineEnd' token.+lineEnd :: Monad m => ParsecT [Tok] s m Tok+lineEnd = satisfyTok (hasType LineEnd)+{-# INLINE lineEnd #-}++-- | Parses a 'Spaces' token.+spaceTok :: Monad m => ParsecT [Tok] s m Tok+spaceTok = satisfyTok (hasType Spaces)+{-# INLINE spaceTok #-}++-- | Parses a 'WordChars' token matching a predicate.+satisfyWord :: Monad m => (Text -> Bool) -> ParsecT [Tok] s m Tok+satisfyWord f = satisfyTok (\t -> hasType WordChars t && textIs f t)+{-# INLINE satisfyWord #-}++-- | Parses exactly @n@ spaces. If tabs are encountered,+-- they are split into spaces before being consumed; so+-- a tab may be partially consumed by this parser.+gobbleSpaces :: Monad m => Int -> ParsecT [Tok] u m Int+gobbleSpaces 0 = return 0+gobbleSpaces n = try $ gobble' True n+{-# INLINE gobbleSpaces #-}++-- | Parses up to @n@ spaces.+gobbleUpToSpaces :: Monad m => Int -> ParsecT [Tok] u m Int+gobbleUpToSpaces 0 = return 0+gobbleUpToSpaces n = gobble' False n+{-# INLINE gobbleUpToSpaces #-}++gobble' :: Monad m => Bool -> Int -> ParsecT [Tok] u m Int+gobble' requireAll numspaces+ | numspaces >= 1 = (do+ Tok Spaces pos _ <- satisfyTok (hasType Spaces)+ pos' <- getPosition+ case sourceColumn pos' - sourceColumn pos of+ n | n < numspaces -> (+ n) <$> gobble' requireAll (numspaces - n)+ | n == numspaces -> return $! n+ | otherwise -> do+ let newtok = Tok Spaces+ (incSourceColumn pos numspaces)+ (T.replicate (n - numspaces) " ")+ getInput >>= setInput . (newtok:)+ return $! numspaces)+ <|> if requireAll+ then mzero+ else return 0+ | otherwise = return 0+{-# INLINE gobble' #-}++-- | Applies a parser and returns its value (if successful)+-- plus a list of the raw tokens parsed.+withRaw :: Monad m => ParsecT [Tok] s m a -> ParsecT [Tok] s m (a, [Tok])+withRaw parser = do+ toks <- getInput+ res <- parser+ newpos <- getPosition+ let getrawtoks (t:ts)+ | tokPos t < newpos = t : getrawtoks ts+ getrawtoks _ = []+ let rawtoks = getrawtoks toks+ return (res, rawtoks)+{-# INLINE withRaw #-}++-- | Filters tokens of a certain type.+hasType :: TokType -> Tok -> Bool+hasType ty (Tok ty' _ _) = ty == ty'+{-# INLINE hasType #-}++hasTypeIn :: [TokType] -> Tok -> Bool+hasTypeIn tys (Tok ty' _ _) = ty' `elem` tys++-- | Filters tokens with certain contents.+textIs :: (Text -> Bool) -> Tok -> Bool+textIs f (Tok _ _ t) = f t+{-# INLINE textIs #-}++-- | Gobble up to 3 spaces (may be part of a tab).+nonindentSpaces :: Monad m => ParsecT [Tok] u m ()+nonindentSpaces = void $ gobbleUpToSpaces 3+{-# INLINE nonindentSpaces #-}++-- | Case-insensitive membership in a list of 'Text's.+isOneOfCI :: [Text] -> Text -> Bool+isOneOfCI ts t = T.toLower t `elem` ts+{-# INLINE isOneOfCI #-}++-- | Apply @p@ many times until @stop@ succeeds, discarding results.+skipManyTill :: ParsecT s u m a -> ParsecT s u m b -> ParsecT s u m ()+skipManyTill p stop = scan+ where scan = (() <$ stop) <|> (p >> scan)+{-# INLINE skipManyTill #-}++-- | Efficiently skip 'Tok's satisfying a certain condition.+skipWhile :: Monad m => (Tok -> Bool) -> ParsecT [Tok] u m ()+skipWhile f = skipMany (satisfyTok f)+{-# INLINE skipWhile #-}++-- | Parse optional spaces and an endline.+blankLine :: Monad m => ParsecT [Tok] s m ()+blankLine = try $ do+ skipWhile (hasType Spaces)+ void lineEnd+{-# INLINE blankLine #-}++-- | Efficiently parse the remaining tokens on a line,+-- including the LineEnd (if any).+restOfLine :: Monad m => ParsecT [Tok] s m [Tok]+restOfLine = go+ where+ go = option [] $ do+ !tok <- anyTok+ case tokType tok of+ LineEnd -> return [tok]+ _ -> (tok:) <$> go+{-# INLINE restOfLine #-}
+ src/Commonmark/Tokens.hs view
@@ -0,0 +1,73 @@+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE DeriveDataTypeable #-}+{-# LANGUAGE MultiWayIf #-}+{-# LANGUAGE BangPatterns #-}++module Commonmark.Tokens+ ( Tok(..)+ , TokType(..)+ , SourcePos+ , tokenize+ , untokenize+ ) where++import Data.Char (isAlphaNum, isSpace)+import Data.Text (Text)+import qualified Data.Text as T+import Data.Data (Data, Typeable)+import Text.Parsec.Pos++data Tok = Tok { tokType :: !TokType+ , tokPos :: !SourcePos+ , tokContents :: {-# UNPACK #-} !Text+ }+ deriving (Show, Eq, Data, Typeable)++data TokType =+ Spaces+ | UnicodeSpace+ | LineEnd+ | WordChars+ | Symbol {-# UNPACK #-} !Char+ deriving (Show, Eq, Ord, Data, Typeable)++-- | Convert a 'Text' into a list of 'Tok'. The first parameter+-- species the source name.+tokenize :: String -> Text -> [Tok]+tokenize name = go (initialPos name) . T.groupBy f+ where+ -- We group \r\n, consecutive spaces, and consecutive alphanums;+ -- everything else gets in a token by itself.+ f '\r' '\n' = True+ f ' ' ' ' = True+ f x y = isAlphaNum x && isAlphaNum y++ go _pos [] = []+ go !pos (t:ts) = -- note that t:ts are guaranteed to be nonempty+ case T.head t of+ ' ' -> Tok Spaces pos t :+ go (incSourceColumn pos (T.length t)) ts+ '\t' -> Tok Spaces pos t :+ go (incSourceColumn pos+ (4 - (sourceColumn pos - 1) `mod` 4)) ts+ '\r' -> Tok LineEnd pos t :+ go (incSourceLine (setSourceColumn pos 1) 1) ts+ '\n' -> Tok LineEnd pos t :+ go (incSourceLine (setSourceColumn pos 1) 1) ts+ thead+ | isAlphaNum thead ->+ Tok WordChars pos t :+ go (incSourceColumn pos (T.length t)) ts+ | isSpace thead ->+ Tok UnicodeSpace pos t :+ go (incSourceColumn pos 1) ts+ | otherwise ->+ Tok (Symbol thead) pos t :+ go (incSourceColumn pos 1) ts+{-# SCC tokenize #-}++-- | Reverses 'tokenize'. @untokenize . tokenize@ should be+-- the identity.+untokenize :: [Tok] -> Text+untokenize = mconcat . map tokContents+{-# SCC untokenize #-}
+ src/Commonmark/Types.hs view
@@ -0,0 +1,155 @@+{-# LANGUAGE CPP #-}+{-# LANGUAGE DeriveDataTypeable #-}+{-# LANGUAGE FunctionalDependencies #-}+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE MonoLocalBinds #-}++module Commonmark.Types+ ( Format(..)+ , ListSpacing(..)+ , ListType(..)+ , DelimiterType(..)+ , EnumeratorType(..)+ , IsInline(..)+ , IsBlock(..)+ , SourceRange(..)+ , SourcePos+ , Rangeable(..)+ , Attribute+ , Attributes+ , HasAttributes(..)+ , ToPlainText(..)+ )+where+import Data.Data (Data)+import Data.Text (Text)+import qualified Data.Text as T+import Data.Typeable (Typeable)+import Text.Parsec.Pos (SourcePos, sourceColumn, sourceLine,+ sourceName)+#if !MIN_VERSION_base(4,11,0)+import Data.Semigroup (Semigroup, (<>))+#endif++newtype Format = Format Text+ deriving (Show, Data, Typeable)++instance Eq Format where+ (Format t1) == (Format t2) = T.toCaseFold t1 == T.toCaseFold t2++data ListSpacing =+ TightList+ | LooseList+ deriving (Show, Ord, Eq, Data, Typeable)++data EnumeratorType =+ Decimal+ | UpperAlpha+ | LowerAlpha+ | UpperRoman+ | LowerRoman+ deriving (Show, Ord, Eq, Data, Typeable)++data DelimiterType =+ Period+ | OneParen+ | TwoParens+ deriving (Show, Ord, Eq, Data, Typeable)++data ListType =+ BulletList !Char+ | OrderedList !Int !EnumeratorType !DelimiterType+ -- first Text is before, second Text is after enumerator+ deriving (Show, Ord, Eq, Data, Typeable)++class (Monoid a, Show a, Rangeable a, HasAttributes a) => IsInline a where+ lineBreak :: a+ softBreak :: a+ str :: Text -> a+ entity :: Text -> a+ escapedChar :: Char -> a+ emph :: a -> a+ strong :: a -> a+ link :: Text -- ^ Destination+ -> Text -- ^ Title+ -> a -- ^ Link description+ -> a+ image :: Text -- ^ Source+ -> Text -- ^ Title+ -> a -- ^ Description+ -> a+ code :: Text -> a+ rawInline :: Format -> Text -> a++class (Monoid b, Show b, Rangeable b, IsInline il, HasAttributes b)+ => IsBlock il b | b -> il where+ paragraph :: il -> b+ plain :: il -> b+ thematicBreak :: b+ blockQuote :: b -> b+ codeBlock :: Text -> Text -> b+ heading :: Int -- ^ Level+ -> il -- ^ text+ -> b+ rawBlock :: Format -> Text -> b+ referenceLinkDefinition :: Text -- ^ Label+ -> (Text, Text) -- ^ Destination, title+ -> b+ list :: ListType -> ListSpacing -> [b] -> b++newtype SourceRange = SourceRange+ { unSourceRange :: [(SourcePos, SourcePos)] }+ deriving (Eq, Ord, Data, Typeable)++instance Semigroup SourceRange where+ (SourceRange xs) <> (SourceRange ys) =+ SourceRange (consolidateRanges xs ys)++instance Monoid SourceRange where+ mempty = SourceRange mempty+ mappend = (<>)++consolidateRanges :: Eq a => [(a,a)] -> [(a,a)] -> [(a,a)]+consolidateRanges [] xs = xs+consolidateRanges xs [] = xs+consolidateRanges xs@(_:_) ((s2,e2):ys) =+ if e1 == s2+ then init xs ++ (s1,e2):ys+ else xs ++ (s2,e2):ys+ where (s1,e1) = last xs++instance Show SourceRange where+ show = prettyRange++class Rangeable a where+ ranged :: SourceRange -> a -> a++prettyRange :: SourceRange -> String+prettyRange (SourceRange []) = ""+prettyRange (SourceRange xs@((p,_):_)) =+ sourceName p ++ "@" ++ go (sourceName p) xs+ where+ go _ [] = ""+ go curname ((p1,p2):rest)+ | sourceName p1 /= curname =+ sourceName p1 ++ "@" ++ go (sourceName p) ((p1,p2):rest)+ | otherwise =+ show (sourceLine p1) ++ ":" +++ show (sourceColumn p1) ++ "-" +++ (if sourceName p2 /= curname+ then sourceName p2 ++ "@"+ else "") ++ show (sourceLine p2) +++ ":" ++ show (sourceColumn p2) +++ if null rest+ then ""+ else ";" ++ go (sourceName p2) rest++type Attribute = (Text, Text)++type Attributes = [Attribute]++class HasAttributes a where+ addAttributes :: Attributes -> a -> a++class ToPlainText a where+ toPlainText :: a -> Text
+ test/regression.md view
@@ -0,0 +1,156 @@+### Regression tests++Issue #113: EOL character weirdness on Windows+(Important: first line ends with CR + CR + LF)++```````````````````````````````` example+line1 +line2+.+<p>line1</p>+<p>line2</p>+````````````````````````````````++Issue #114: cmark skipping first character in line+(Important: the blank lines around "Repeatedly" contain a tab.)++```````````````````````````````` example+By taking it apart++- alternative solutions+→+Repeatedly solving+→+- how techniques+.+<p>By taking it apart</p>+<ul>+<li>alternative solutions</li>+</ul>+<p>Repeatedly solving</p>+<ul>+<li>how techniques</li>+</ul>+````````````````````````````````++Issue jgm/CommonMark#430: h2..h6 not recognized as block tags.++```````````````````````````````` example+<h1>lorem</h1>++<h2>lorem</h2>++<h3>lorem</h3>++<h4>lorem</h4>++<h5>lorem</h5>++<h6>lorem</h6>+.+<h1>lorem</h1>+<h2>lorem</h2>+<h3>lorem</h3>+<h4>lorem</h4>+<h5>lorem</h5>+<h6>lorem</h6>+````````````````````````````````++Issue jgm/commonmark.js#109 - tabs after setext header line+++```````````````````````````````` example+hi+--→+.+<h2>hi</h2>+````````````````````````````````++Issue #177 - incorrect emphasis parsing++```````````````````````````````` example+a***b* c*+.+<p>a*<em><em>b</em> c</em></p>+````````````````````````````````++Issue #193 - unescaped left angle brackets in link destination++```````````````````````````````` example+[a]++[a]: <te<st>+.+<p>[a]</p>+<p>[a]: <te<st></p>+````````````````````````````````++Issue #192 - escaped spaces in link destination+++```````````````````````````````` example+[a](te\ st)+.+<p>[a](te\ st)</p>+````````````````````````````````++Issue #527 - meta tags in inline contexts++```````````````````````````````` example+City:+<span itemprop="contentLocation" itemscope itemtype="https://schema.org/City">+ <meta itemprop="name" content="Springfield">+</span>+.+<p>City:+<span itemprop="contentLocation" itemscope itemtype="https://schema.org/City">+<meta itemprop="name" content="Springfield">+</span></p>+````````````````````````````````++Issue #530 - link parsing corner cases++```````````````````````````````` example+[a](\ b)++[a](<<b)++[a](<b+)+.+<p>[a](\ b)</p>+<p>[a](<<b)</p>+<p>[a](<b+)</p>+````````````````````````````````++Issue commonmark#526 - unescaped ( in link title++```````````````````````````````` example+[link](url ((title))+.+<p>[link](url ((title))</p>+````````````````````````````````++Issue commonamrk#517 - script, pre, style close tag without+opener.++```````````````````````````````` example+</script>++</pre>++</style>+.+</script>+</pre>+</style>+````````````````````````````````++Issue #289.++```````````````````````````````` example+[a](<b) c>+.+<p>[a](<b) c></p>+````````````````````````````````
+ test/spec.txt view
@@ -0,0 +1,9710 @@+---+title: CommonMark Spec+author: John MacFarlane+version: 0.29+date: '2019-04-06'+license: '[CC-BY-SA 4.0](http://creativecommons.org/licenses/by-sa/4.0/)'+...++# Introduction++## What is Markdown?++Markdown is a plain text format for writing structured documents,+based on conventions for indicating formatting in email+and usenet posts. It was developed by John Gruber (with+help from Aaron Swartz) and released in 2004 in the form of a+[syntax description](http://daringfireball.net/projects/markdown/syntax)+and a Perl script (`Markdown.pl`) for converting Markdown to+HTML. In the next decade, dozens of implementations were+developed in many languages. Some extended the original+Markdown syntax with conventions for footnotes, tables, and+other document elements. Some allowed Markdown documents to be+rendered in formats other than HTML. Websites like Reddit,+StackOverflow, and GitHub had millions of people using Markdown.+And Markdown started to be used beyond the web, to author books,+articles, slide shows, letters, and lecture notes.++What distinguishes Markdown from many other lightweight markup+syntaxes, which are often easier to write, is its readability.+As Gruber writes:++> The overriding design goal for Markdown's formatting syntax is+> to make it as readable as possible. The idea is that a+> Markdown-formatted document should be publishable as-is, as+> plain text, without looking like it's been marked up with tags+> or formatting instructions.+> (<http://daringfireball.net/projects/markdown/>)++The point can be illustrated by comparing a sample of+[AsciiDoc](http://www.methods.co.nz/asciidoc/) with+an equivalent sample of Markdown. Here is a sample of+AsciiDoc from the AsciiDoc manual:++```+1. List item one.+++List item one continued with a second paragraph followed by an+Indented block.+++.................+$ ls *.sh+$ mv *.sh ~/tmp+.................+++List item continued with a third paragraph.++2. List item two continued with an open block.+++--+This paragraph is part of the preceding list item.++a. This list is nested and does not require explicit item+continuation.+++This paragraph is part of the preceding list item.++b. List item b.++This paragraph belongs to item two of the outer list.+--+```++And here is the equivalent in Markdown:+```+1. List item one.++ List item one continued with a second paragraph followed by an+ Indented block.++ $ ls *.sh+ $ mv *.sh ~/tmp++ List item continued with a third paragraph.++2. List item two continued with an open block.++ This paragraph is part of the preceding list item.++ 1. This list is nested and does not require explicit item continuation.++ This paragraph is part of the preceding list item.++ 2. List item b.++ This paragraph belongs to item two of the outer list.+```++The AsciiDoc version is, arguably, easier to write. You don't need+to worry about indentation. But the Markdown version is much easier+to read. The nesting of list items is apparent to the eye in the+source, not just in the processed document.++## Why is a spec needed?++John Gruber's [canonical description of Markdown's+syntax](http://daringfireball.net/projects/markdown/syntax)+does not specify the syntax unambiguously. Here are some examples of+questions it does not answer:++1. How much indentation is needed for a sublist? The spec says that+ continuation paragraphs need to be indented four spaces, but is+ not fully explicit about sublists. It is natural to think that+ they, too, must be indented four spaces, but `Markdown.pl` does+ not require that. This is hardly a "corner case," and divergences+ between implementations on this issue often lead to surprises for+ users in real documents. (See [this comment by John+ Gruber](http://article.gmane.org/gmane.text.markdown.general/1997).)++2. Is a blank line needed before a block quote or heading?+ Most implementations do not require the blank line. However,+ this can lead to unexpected results in hard-wrapped text, and+ also to ambiguities in parsing (note that some implementations+ put the heading inside the blockquote, while others do not).+ (John Gruber has also spoken [in favor of requiring the blank+ lines](http://article.gmane.org/gmane.text.markdown.general/2146).)++3. Is a blank line needed before an indented code block?+ (`Markdown.pl` requires it, but this is not mentioned in the+ documentation, and some implementations do not require it.)++ ``` markdown+ paragraph+ code?+ ```++4. What is the exact rule for determining when list items get+ wrapped in `<p>` tags? Can a list be partially "loose" and partially+ "tight"? What should we do with a list like this?++ ``` markdown+ 1. one++ 2. two+ 3. three+ ```++ Or this?++ ``` markdown+ 1. one+ - a++ - b+ 2. two+ ```++ (There are some relevant comments by John Gruber+ [here](http://article.gmane.org/gmane.text.markdown.general/2554).)++5. Can list markers be indented? Can ordered list markers be right-aligned?++ ``` markdown+ 8. item 1+ 9. item 2+ 10. item 2a+ ```++6. Is this one list with a thematic break in its second item,+ or two lists separated by a thematic break?++ ``` markdown+ * a+ * * * * *+ * b+ ```++7. When list markers change from numbers to bullets, do we have+ two lists or one? (The Markdown syntax description suggests two,+ but the perl scripts and many other implementations produce one.)++ ``` markdown+ 1. fee+ 2. fie+ - foe+ - fum+ ```++8. What are the precedence rules for the markers of inline structure?+ For example, is the following a valid link, or does the code span+ take precedence ?++ ``` markdown+ [a backtick (`)](/url) and [another backtick (`)](/url).+ ```++9. What are the precedence rules for markers of emphasis and strong+ emphasis? For example, how should the following be parsed?++ ``` markdown+ *foo *bar* baz*+ ```++10. What are the precedence rules between block-level and inline-level+ structure? For example, how should the following be parsed?++ ``` markdown+ - `a long code span can contain a hyphen like this+ - and it can screw things up`+ ```++11. Can list items include section headings? (`Markdown.pl` does not+ allow this, but does allow blockquotes to include headings.)++ ``` markdown+ - # Heading+ ```++12. Can list items be empty?++ ``` markdown+ * a+ *+ * b+ ```++13. Can link references be defined inside block quotes or list items?++ ``` markdown+ > Blockquote [foo].+ >+ > [foo]: /url+ ```++14. If there are multiple definitions for the same reference, which takes+ precedence?++ ``` markdown+ [foo]: /url1+ [foo]: /url2++ [foo][]+ ```++In the absence of a spec, early implementers consulted `Markdown.pl`+to resolve these ambiguities. But `Markdown.pl` was quite buggy, and+gave manifestly bad results in many cases, so it was not a+satisfactory replacement for a spec.++Because there is no unambiguous spec, implementations have diverged+considerably. As a result, users are often surprised to find that+a document that renders one way on one system (say, a GitHub wiki)+renders differently on another (say, converting to docbook using+pandoc). To make matters worse, because nothing in Markdown counts+as a "syntax error," the divergence often isn't discovered right away.++## About this document++This document attempts to specify Markdown syntax unambiguously.+It contains many examples with side-by-side Markdown and+HTML. These are intended to double as conformance tests. An+accompanying script `spec_tests.py` can be used to run the tests+against any Markdown program:++ python test/spec_tests.py --spec spec.txt --program PROGRAM++Since this document describes how Markdown is to be parsed into+an abstract syntax tree, it would have made sense to use an abstract+representation of the syntax tree instead of HTML. But HTML is capable+of representing the structural distinctions we need to make, and the+choice of HTML for the tests makes it possible to run the tests against+an implementation without writing an abstract syntax tree renderer.++This document is generated from a text file, `spec.txt`, written+in Markdown with a small extension for the side-by-side tests.+The script `tools/makespec.py` can be used to convert `spec.txt` into+HTML or CommonMark (which can then be converted into other formats).++In the examples, the `→` character is used to represent tabs.++# Preliminaries++## Characters and lines++Any sequence of [characters] is a valid CommonMark+document.++A [character](@) is a Unicode code point. Although some+code points (for example, combining accents) do not correspond to+characters in an intuitive sense, all code points count as characters+for purposes of this spec.++This spec does not specify an encoding; it thinks of lines as composed+of [characters] rather than bytes. A conforming parser may be limited+to a certain encoding.++A [line](@) is a sequence of zero or more [characters]+other than newline (`U+000A`) or carriage return (`U+000D`),+followed by a [line ending] or by the end of file.++A [line ending](@) is a newline (`U+000A`), a carriage return+(`U+000D`) not followed by a newline, or a carriage return and a+following newline.++A line containing no characters, or a line containing only spaces+(`U+0020`) or tabs (`U+0009`), is called a [blank line](@).++The following definitions of character classes will be used in this spec:++A [whitespace character](@) is a space+(`U+0020`), tab (`U+0009`), newline (`U+000A`), line tabulation (`U+000B`),+form feed (`U+000C`), or carriage return (`U+000D`).++[Whitespace](@) is a sequence of one or more [whitespace+characters].++A [Unicode whitespace character](@) is+any code point in the Unicode `Zs` general category, or a tab (`U+0009`),+carriage return (`U+000D`), newline (`U+000A`), or form feed+(`U+000C`).++[Unicode whitespace](@) is a sequence of one+or more [Unicode whitespace characters].++A [space](@) is `U+0020`.++A [non-whitespace character](@) is any character+that is not a [whitespace character].++An [ASCII punctuation character](@)+is `!`, `"`, `#`, `$`, `%`, `&`, `'`, `(`, `)`,+`*`, `+`, `,`, `-`, `.`, `/` (U+0021–2F), +`:`, `;`, `<`, `=`, `>`, `?`, `@` (U+003A–0040),+`[`, `\`, `]`, `^`, `_`, `` ` `` (U+005B–0060), +`{`, `|`, `}`, or `~` (U+007B–007E).++A [punctuation character](@) is an [ASCII+punctuation character] or anything in+the general Unicode categories `Pc`, `Pd`, `Pe`, `Pf`, `Pi`, `Po`, or `Ps`.++## Tabs++Tabs in lines are not expanded to [spaces]. However,+in contexts where whitespace helps to define block structure,+tabs behave as if they were replaced by spaces with a tab stop+of 4 characters.++Thus, for example, a tab can be used instead of four spaces+in an indented code block. (Note, however, that internal+tabs are passed through as literal tabs, not expanded to+spaces.)++```````````````````````````````` example+→foo→baz→→bim+.+<pre><code>foo→baz→→bim+</code></pre>+````````````````````````````````++```````````````````````````````` example+ →foo→baz→→bim+.+<pre><code>foo→baz→→bim+</code></pre>+````````````````````````````````++```````````````````````````````` example+ a→a+ ὐ→a+.+<pre><code>a→a+ὐ→a+</code></pre>+````````````````````````````````++In the following example, a continuation paragraph of a list+item is indented with a tab; this has exactly the same effect+as indentation with four spaces would:++```````````````````````````````` example+ - foo++→bar+.+<ul>+<li>+<p>foo</p>+<p>bar</p>+</li>+</ul>+````````````````````````````````++```````````````````````````````` example+- foo++→→bar+.+<ul>+<li>+<p>foo</p>+<pre><code> bar+</code></pre>+</li>+</ul>+````````````````````````````````++Normally the `>` that begins a block quote may be followed+optionally by a space, which is not considered part of the+content. In the following case `>` is followed by a tab,+which is treated as if it were expanded into three spaces.+Since one of these spaces is considered part of the+delimiter, `foo` is considered to be indented six spaces+inside the block quote context, so we get an indented+code block starting with two spaces.++```````````````````````````````` example+>→→foo+.+<blockquote>+<pre><code> foo+</code></pre>+</blockquote>+````````````````````````````````++```````````````````````````````` example+-→→foo+.+<ul>+<li>+<pre><code> foo+</code></pre>+</li>+</ul>+````````````````````````````````+++```````````````````````````````` example+ foo+→bar+.+<pre><code>foo+bar+</code></pre>+````````````````````````````````++```````````````````````````````` example+ - foo+ - bar+→ - baz+.+<ul>+<li>foo+<ul>+<li>bar+<ul>+<li>baz</li>+</ul>+</li>+</ul>+</li>+</ul>+````````````````````````````````++```````````````````````````````` example+#→Foo+.+<h1>Foo</h1>+````````````````````````````````++```````````````````````````````` example+*→*→*→+.+<hr />+````````````````````````````````+++## Insecure characters++For security reasons, the Unicode character `U+0000` must be replaced+with the REPLACEMENT CHARACTER (`U+FFFD`).++# Blocks and inlines++We can think of a document as a sequence of+[blocks](@)---structural elements like paragraphs, block+quotations, lists, headings, rules, and code blocks. Some blocks (like+block quotes and list items) contain other blocks; others (like+headings and paragraphs) contain [inline](@) content---text,+links, emphasized text, images, code spans, and so on.++## Precedence++Indicators of block structure always take precedence over indicators+of inline structure. So, for example, the following is a list with+two items, not a list with one item containing a code span:++```````````````````````````````` example+- `one+- two`+.+<ul>+<li>`one</li>+<li>two`</li>+</ul>+````````````````````````````````+++This means that parsing can proceed in two steps: first, the block+structure of the document can be discerned; second, text lines inside+paragraphs, headings, and other block constructs can be parsed for inline+structure. The second step requires information about link reference+definitions that will be available only at the end of the first+step. Note that the first step requires processing lines in sequence,+but the second can be parallelized, since the inline parsing of+one block element does not affect the inline parsing of any other.++## Container blocks and leaf blocks++We can divide blocks into two types:+[container blocks](@),+which can contain other blocks, and [leaf blocks](@),+which cannot.++# Leaf blocks++This section describes the different kinds of leaf block that make up a+Markdown document.++## Thematic breaks++A line consisting of 0-3 spaces of indentation, followed by a sequence+of three or more matching `-`, `_`, or `*` characters, each followed+optionally by any number of spaces or tabs, forms a+[thematic break](@).++```````````````````````````````` example+***+---+___+.+<hr />+<hr />+<hr />+````````````````````````````````+++Wrong characters:++```````````````````````````````` example+++++.+<p>+++</p>+````````````````````````````````+++```````````````````````````````` example+===+.+<p>===</p>+````````````````````````````````+++Not enough characters:++```````````````````````````````` example+--+**+__+.+<p>--+**+__</p>+````````````````````````````````+++One to three spaces indent are allowed:++```````````````````````````````` example+ ***+ ***+ ***+.+<hr />+<hr />+<hr />+````````````````````````````````+++Four spaces is too many:++```````````````````````````````` example+ ***+.+<pre><code>***+</code></pre>+````````````````````````````````+++```````````````````````````````` example+Foo+ ***+.+<p>Foo+***</p>+````````````````````````````````+++More than three characters may be used:++```````````````````````````````` example+_____________________________________+.+<hr />+````````````````````````````````+++Spaces are allowed between the characters:++```````````````````````````````` example+ - - -+.+<hr />+````````````````````````````````+++```````````````````````````````` example+ ** * ** * ** * **+.+<hr />+````````````````````````````````+++```````````````````````````````` example+- - - -+.+<hr />+````````````````````````````````+++Spaces are allowed at the end:++```````````````````````````````` example+- - - - +.+<hr />+````````````````````````````````+++However, no other characters may occur in the line:++```````````````````````````````` example+_ _ _ _ a++a------++---a---+.+<p>_ _ _ _ a</p>+<p>a------</p>+<p>---a---</p>+````````````````````````````````+++It is required that all of the [non-whitespace characters] be the same.+So, this is not a thematic break:++```````````````````````````````` example+ *-*+.+<p><em>-</em></p>+````````````````````````````````+++Thematic breaks do not need blank lines before or after:++```````````````````````````````` example+- foo+***+- bar+.+<ul>+<li>foo</li>+</ul>+<hr />+<ul>+<li>bar</li>+</ul>+````````````````````````````````+++Thematic breaks can interrupt a paragraph:++```````````````````````````````` example+Foo+***+bar+.+<p>Foo</p>+<hr />+<p>bar</p>+````````````````````````````````+++If a line of dashes that meets the above conditions for being a+thematic break could also be interpreted as the underline of a [setext+heading], the interpretation as a+[setext heading] takes precedence. Thus, for example,+this is a setext heading, not a paragraph followed by a thematic break:++```````````````````````````````` example+Foo+---+bar+.+<h2>Foo</h2>+<p>bar</p>+````````````````````````````````+++When both a thematic break and a list item are possible+interpretations of a line, the thematic break takes precedence:++```````````````````````````````` example+* Foo+* * *+* Bar+.+<ul>+<li>Foo</li>+</ul>+<hr />+<ul>+<li>Bar</li>+</ul>+````````````````````````````````+++If you want a thematic break in a list item, use a different bullet:++```````````````````````````````` example+- Foo+- * * *+.+<ul>+<li>Foo</li>+<li>+<hr />+</li>+</ul>+````````````````````````````````+++## ATX headings++An [ATX heading](@)+consists of a string of characters, parsed as inline content, between an+opening sequence of 1--6 unescaped `#` characters and an optional+closing sequence of any number of unescaped `#` characters.+The opening sequence of `#` characters must be followed by a+[space] or by the end of line. The optional closing sequence of `#`s must be+preceded by a [space] and may be followed by spaces only. The opening+`#` character may be indented 0-3 spaces. The raw contents of the+heading are stripped of leading and trailing spaces before being parsed+as inline content. The heading level is equal to the number of `#`+characters in the opening sequence.++Simple headings:++```````````````````````````````` example+# foo+## foo+### foo+#### foo+##### foo+###### foo+.+<h1>foo</h1>+<h2>foo</h2>+<h3>foo</h3>+<h4>foo</h4>+<h5>foo</h5>+<h6>foo</h6>+````````````````````````````````+++More than six `#` characters is not a heading:++```````````````````````````````` example+####### foo+.+<p>####### foo</p>+````````````````````````````````+++At least one space is required between the `#` characters and the+heading's contents, unless the heading is empty. Note that many+implementations currently do not require the space. However, the+space was required by the+[original ATX implementation](http://www.aaronsw.com/2002/atx/atx.py),+and it helps prevent things like the following from being parsed as+headings:++```````````````````````````````` example+#5 bolt++#hashtag+.+<p>#5 bolt</p>+<p>#hashtag</p>+````````````````````````````````+++This is not a heading, because the first `#` is escaped:++```````````````````````````````` example+\## foo+.+<p>## foo</p>+````````````````````````````````+++Contents are parsed as inlines:++```````````````````````````````` example+# foo *bar* \*baz\*+.+<h1>foo <em>bar</em> *baz*</h1>+````````````````````````````````+++Leading and trailing [whitespace] is ignored in parsing inline content:++```````````````````````````````` example+# foo +.+<h1>foo</h1>+````````````````````````````````+++One to three spaces indentation are allowed:++```````````````````````````````` example+ ### foo+ ## foo+ # foo+.+<h3>foo</h3>+<h2>foo</h2>+<h1>foo</h1>+````````````````````````````````+++Four spaces are too much:++```````````````````````````````` example+ # foo+.+<pre><code># foo+</code></pre>+````````````````````````````````+++```````````````````````````````` example+foo+ # bar+.+<p>foo+# bar</p>+````````````````````````````````+++A closing sequence of `#` characters is optional:++```````````````````````````````` example+## foo ##+ ### bar ###+.+<h2>foo</h2>+<h3>bar</h3>+````````````````````````````````+++It need not be the same length as the opening sequence:++```````````````````````````````` example+# foo ##################################+##### foo ##+.+<h1>foo</h1>+<h5>foo</h5>+````````````````````````````````+++Spaces are allowed after the closing sequence:++```````````````````````````````` example+### foo ### +.+<h3>foo</h3>+````````````````````````````````+++A sequence of `#` characters with anything but [spaces] following it+is not a closing sequence, but counts as part of the contents of the+heading:++```````````````````````````````` example+### foo ### b+.+<h3>foo ### b</h3>+````````````````````````````````+++The closing sequence must be preceded by a space:++```````````````````````````````` example+# foo#+.+<h1>foo#</h1>+````````````````````````````````+++Backslash-escaped `#` characters do not count as part+of the closing sequence:++```````````````````````````````` example+### foo \###+## foo #\##+# foo \#+.+<h3>foo ###</h3>+<h2>foo ###</h2>+<h1>foo #</h1>+````````````````````````````````+++ATX headings need not be separated from surrounding content by blank+lines, and they can interrupt paragraphs:++```````````````````````````````` example+****+## foo+****+.+<hr />+<h2>foo</h2>+<hr />+````````````````````````````````+++```````````````````````````````` example+Foo bar+# baz+Bar foo+.+<p>Foo bar</p>+<h1>baz</h1>+<p>Bar foo</p>+````````````````````````````````+++ATX headings can be empty:++```````````````````````````````` example+## +#+### ###+.+<h2></h2>+<h1></h1>+<h3></h3>+````````````````````````````````+++## Setext headings++A [setext heading](@) consists of one or more+lines of text, each containing at least one [non-whitespace+character], with no more than 3 spaces indentation, followed by+a [setext heading underline]. The lines of text must be such+that, were they not followed by the setext heading underline,+they would be interpreted as a paragraph: they cannot be+interpretable as a [code fence], [ATX heading][ATX headings],+[block quote][block quotes], [thematic break][thematic breaks],+[list item][list items], or [HTML block][HTML blocks].++A [setext heading underline](@) is a sequence of+`=` characters or a sequence of `-` characters, with no more than 3+spaces indentation and any number of trailing spaces. If a line+containing a single `-` can be interpreted as an+empty [list items], it should be interpreted this way+and not as a [setext heading underline].++The heading is a level 1 heading if `=` characters are used in+the [setext heading underline], and a level 2 heading if `-`+characters are used. The contents of the heading are the result+of parsing the preceding lines of text as CommonMark inline+content.++In general, a setext heading need not be preceded or followed by a+blank line. However, it cannot interrupt a paragraph, so when a+setext heading comes after a paragraph, a blank line is needed between+them.++Simple examples:++```````````````````````````````` example+Foo *bar*+=========++Foo *bar*+---------+.+<h1>Foo <em>bar</em></h1>+<h2>Foo <em>bar</em></h2>+````````````````````````````````+++The content of the header may span more than one line:++```````````````````````````````` example+Foo *bar+baz*+====+.+<h1>Foo <em>bar+baz</em></h1>+````````````````````````````````++The contents are the result of parsing the headings's raw+content as inlines. The heading's raw content is formed by+concatenating the lines and removing initial and final+[whitespace].++```````````````````````````````` example+ Foo *bar+baz*→+====+.+<h1>Foo <em>bar+baz</em></h1>+````````````````````````````````+++The underlining can be any length:++```````````````````````````````` example+Foo+-------------------------++Foo+=+.+<h2>Foo</h2>+<h1>Foo</h1>+````````````````````````````````+++The heading content can be indented up to three spaces, and need+not line up with the underlining:++```````````````````````````````` example+ Foo+---++ Foo+-----++ Foo+ ===+.+<h2>Foo</h2>+<h2>Foo</h2>+<h1>Foo</h1>+````````````````````````````````+++Four spaces indent is too much:++```````````````````````````````` example+ Foo+ ---++ Foo+---+.+<pre><code>Foo+---++Foo+</code></pre>+<hr />+````````````````````````````````+++The setext heading underline can be indented up to three spaces, and+may have trailing spaces:++```````````````````````````````` example+Foo+ ---- +.+<h2>Foo</h2>+````````````````````````````````+++Four spaces is too much:++```````````````````````````````` example+Foo+ ---+.+<p>Foo+---</p>+````````````````````````````````+++The setext heading underline cannot contain internal spaces:++```````````````````````````````` example+Foo+= =++Foo+--- -+.+<p>Foo+= =</p>+<p>Foo</p>+<hr />+````````````````````````````````+++Trailing spaces in the content line do not cause a line break:++```````````````````````````````` example+Foo +-----+.+<h2>Foo</h2>+````````````````````````````````+++Nor does a backslash at the end:++```````````````````````````````` example+Foo\+----+.+<h2>Foo\</h2>+````````````````````````````````+++Since indicators of block structure take precedence over+indicators of inline structure, the following are setext headings:++```````````````````````````````` example+`Foo+----+`++<a title="a lot+---+of dashes"/>+.+<h2>`Foo</h2>+<p>`</p>+<h2><a title="a lot</h2>+<p>of dashes"/></p>+````````````````````````````````+++The setext heading underline cannot be a [lazy continuation+line] in a list item or block quote:++```````````````````````````````` example+> Foo+---+.+<blockquote>+<p>Foo</p>+</blockquote>+<hr />+````````````````````````````````+++```````````````````````````````` example+> foo+bar+===+.+<blockquote>+<p>foo+bar+===</p>+</blockquote>+````````````````````````````````+++```````````````````````````````` example+- Foo+---+.+<ul>+<li>Foo</li>+</ul>+<hr />+````````````````````````````````+++A blank line is needed between a paragraph and a following+setext heading, since otherwise the paragraph becomes part+of the heading's content:++```````````````````````````````` example+Foo+Bar+---+.+<h2>Foo+Bar</h2>+````````````````````````````````+++But in general a blank line is not required before or after+setext headings:++```````````````````````````````` example+---+Foo+---+Bar+---+Baz+.+<hr />+<h2>Foo</h2>+<h2>Bar</h2>+<p>Baz</p>+````````````````````````````````+++Setext headings cannot be empty:++```````````````````````````````` example++====+.+<p>====</p>+````````````````````````````````+++Setext heading text lines must not be interpretable as block+constructs other than paragraphs. So, the line of dashes+in these examples gets interpreted as a thematic break:++```````````````````````````````` example+---+---+.+<hr />+<hr />+````````````````````````````````+++```````````````````````````````` example+- foo+-----+.+<ul>+<li>foo</li>+</ul>+<hr />+````````````````````````````````+++```````````````````````````````` example+ foo+---+.+<pre><code>foo+</code></pre>+<hr />+````````````````````````````````+++```````````````````````````````` example+> foo+-----+.+<blockquote>+<p>foo</p>+</blockquote>+<hr />+````````````````````````````````+++If you want a heading with `> foo` as its literal text, you can+use backslash escapes:++```````````````````````````````` example+\> foo+------+.+<h2>> foo</h2>+````````````````````````````````+++**Compatibility note:** Most existing Markdown implementations+do not allow the text of setext headings to span multiple lines.+But there is no consensus about how to interpret++``` markdown+Foo+bar+---+baz+```++One can find four different interpretations:++1. paragraph "Foo", heading "bar", paragraph "baz"+2. paragraph "Foo bar", thematic break, paragraph "baz"+3. paragraph "Foo bar --- baz"+4. heading "Foo bar", paragraph "baz"++We find interpretation 4 most natural, and interpretation 4+increases the expressive power of CommonMark, by allowing+multiline headings. Authors who want interpretation 1 can+put a blank line after the first paragraph:++```````````````````````````````` example+Foo++bar+---+baz+.+<p>Foo</p>+<h2>bar</h2>+<p>baz</p>+````````````````````````````````+++Authors who want interpretation 2 can put blank lines around+the thematic break,++```````````````````````````````` example+Foo+bar++---++baz+.+<p>Foo+bar</p>+<hr />+<p>baz</p>+````````````````````````````````+++or use a thematic break that cannot count as a [setext heading+underline], such as++```````````````````````````````` example+Foo+bar+* * *+baz+.+<p>Foo+bar</p>+<hr />+<p>baz</p>+````````````````````````````````+++Authors who want interpretation 3 can use backslash escapes:++```````````````````````````````` example+Foo+bar+\---+baz+.+<p>Foo+bar+---+baz</p>+````````````````````````````````+++## Indented code blocks++An [indented code block](@) is composed of one or more+[indented chunks] separated by blank lines.+An [indented chunk](@) is a sequence of non-blank lines,+each indented four or more spaces. The contents of the code block are+the literal contents of the lines, including trailing+[line endings], minus four spaces of indentation.+An indented code block has no [info string].++An indented code block cannot interrupt a paragraph, so there must be+a blank line between a paragraph and a following indented code block.+(A blank line is not needed, however, between a code block and a following+paragraph.)++```````````````````````````````` example+ a simple+ indented code block+.+<pre><code>a simple+ indented code block+</code></pre>+````````````````````````````````+++If there is any ambiguity between an interpretation of indentation+as a code block and as indicating that material belongs to a [list+item][list items], the list item interpretation takes precedence:++```````````````````````````````` example+ - foo++ bar+.+<ul>+<li>+<p>foo</p>+<p>bar</p>+</li>+</ul>+````````````````````````````````+++```````````````````````````````` example+1. foo++ - bar+.+<ol>+<li>+<p>foo</p>+<ul>+<li>bar</li>+</ul>+</li>+</ol>+````````````````````````````````++++The contents of a code block are literal text, and do not get parsed+as Markdown:++```````````````````````````````` example+ <a/>+ *hi*++ - one+.+<pre><code><a/>+*hi*++- one+</code></pre>+````````````````````````````````+++Here we have three chunks separated by blank lines:++```````````````````````````````` example+ chunk1++ chunk2+ + + + chunk3+.+<pre><code>chunk1++chunk2++++chunk3+</code></pre>+````````````````````````````````+++Any initial spaces beyond four will be included in the content, even+in interior blank lines:++```````````````````````````````` example+ chunk1+ + chunk2+.+<pre><code>chunk1+ + chunk2+</code></pre>+````````````````````````````````+++An indented code block cannot interrupt a paragraph. (This+allows hanging indents and the like.)++```````````````````````````````` example+Foo+ bar++.+<p>Foo+bar</p>+````````````````````````````````+++However, any non-blank line with fewer than four leading spaces ends+the code block immediately. So a paragraph may occur immediately+after indented code:++```````````````````````````````` example+ foo+bar+.+<pre><code>foo+</code></pre>+<p>bar</p>+````````````````````````````````+++And indented code can occur immediately before and after other kinds of+blocks:++```````````````````````````````` example+# Heading+ foo+Heading+------+ foo+----+.+<h1>Heading</h1>+<pre><code>foo+</code></pre>+<h2>Heading</h2>+<pre><code>foo+</code></pre>+<hr />+````````````````````````````````+++The first line can be indented more than four spaces:++```````````````````````````````` example+ foo+ bar+.+<pre><code> foo+bar+</code></pre>+````````````````````````````````+++Blank lines preceding or following an indented code block+are not included in it:++```````````````````````````````` example++ + foo+ ++.+<pre><code>foo+</code></pre>+````````````````````````````````+++Trailing spaces are included in the code block's content:++```````````````````````````````` example+ foo +.+<pre><code>foo +</code></pre>+````````````````````````````````++++## Fenced code blocks++A [code fence](@) is a sequence+of at least three consecutive backtick characters (`` ` ``) or+tildes (`~`). (Tildes and backticks cannot be mixed.)+A [fenced code block](@)+begins with a code fence, indented no more than three spaces.++The line with the opening code fence may optionally contain some text+following the code fence; this is trimmed of leading and trailing+whitespace and called the [info string](@). If the [info string] comes+after a backtick fence, it may not contain any backtick+characters. (The reason for this restriction is that otherwise+some inline code would be incorrectly interpreted as the+beginning of a fenced code block.)++The content of the code block consists of all subsequent lines, until+a closing [code fence] of the same type as the code block+began with (backticks or tildes), and with at least as many backticks+or tildes as the opening code fence. If the leading code fence is+indented N spaces, then up to N spaces of indentation are removed from+each line of the content (if present). (If a content line is not+indented, it is preserved unchanged. If it is indented less than N+spaces, all of the indentation is removed.)++The closing code fence may be indented up to three spaces, and may be+followed only by spaces, which are ignored. If the end of the+containing block (or document) is reached and no closing code fence+has been found, the code block contains all of the lines after the+opening code fence until the end of the containing block (or+document). (An alternative spec would require backtracking in the+event that a closing code fence is not found. But this makes parsing+much less efficient, and there seems to be no real down side to the+behavior described here.)++A fenced code block may interrupt a paragraph, and does not require+a blank line either before or after.++The content of a code fence is treated as literal text, not parsed+as inlines. The first word of the [info string] is typically used to+specify the language of the code sample, and rendered in the `class`+attribute of the `code` tag. However, this spec does not mandate any+particular treatment of the [info string].++Here is a simple example with backticks:++```````````````````````````````` example+```+<+ >+```+.+<pre><code><+ >+</code></pre>+````````````````````````````````+++With tildes:++```````````````````````````````` example+~~~+<+ >+~~~+.+<pre><code><+ >+</code></pre>+````````````````````````````````++Fewer than three backticks is not enough:++```````````````````````````````` example+``+foo+``+.+<p><code>foo</code></p>+````````````````````````````````++The closing code fence must use the same character as the opening+fence:++```````````````````````````````` example+```+aaa+~~~+```+.+<pre><code>aaa+~~~+</code></pre>+````````````````````````````````+++```````````````````````````````` example+~~~+aaa+```+~~~+.+<pre><code>aaa+```+</code></pre>+````````````````````````````````+++The closing code fence must be at least as long as the opening fence:++```````````````````````````````` example+````+aaa+```+``````+.+<pre><code>aaa+```+</code></pre>+````````````````````````````````+++```````````````````````````````` example+~~~~+aaa+~~~+~~~~+.+<pre><code>aaa+~~~+</code></pre>+````````````````````````````````+++Unclosed code blocks are closed by the end of the document+(or the enclosing [block quote][block quotes] or [list item][list items]):++```````````````````````````````` example+```+.+<pre><code></code></pre>+````````````````````````````````+++```````````````````````````````` example+`````++```+aaa+.+<pre><code>+```+aaa+</code></pre>+````````````````````````````````+++```````````````````````````````` example+> ```+> aaa++bbb+.+<blockquote>+<pre><code>aaa+</code></pre>+</blockquote>+<p>bbb</p>+````````````````````````````````+++A code block can have all empty lines as its content:++```````````````````````````````` example+```++ +```+.+<pre><code>+ +</code></pre>+````````````````````````````````+++A code block can be empty:++```````````````````````````````` example+```+```+.+<pre><code></code></pre>+````````````````````````````````+++Fences can be indented. If the opening fence is indented,+content lines will have equivalent opening indentation removed,+if present:++```````````````````````````````` example+ ```+ aaa+aaa+```+.+<pre><code>aaa+aaa+</code></pre>+````````````````````````````````+++```````````````````````````````` example+ ```+aaa+ aaa+aaa+ ```+.+<pre><code>aaa+aaa+aaa+</code></pre>+````````````````````````````````+++```````````````````````````````` example+ ```+ aaa+ aaa+ aaa+ ```+.+<pre><code>aaa+ aaa+aaa+</code></pre>+````````````````````````````````+++Four spaces indentation produces an indented code block:++```````````````````````````````` example+ ```+ aaa+ ```+.+<pre><code>```+aaa+```+</code></pre>+````````````````````````````````+++Closing fences may be indented by 0-3 spaces, and their indentation+need not match that of the opening fence:++```````````````````````````````` example+```+aaa+ ```+.+<pre><code>aaa+</code></pre>+````````````````````````````````+++```````````````````````````````` example+ ```+aaa+ ```+.+<pre><code>aaa+</code></pre>+````````````````````````````````+++This is not a closing fence, because it is indented 4 spaces:++```````````````````````````````` example+```+aaa+ ```+.+<pre><code>aaa+ ```+</code></pre>+````````````````````````````````++++Code fences (opening and closing) cannot contain internal spaces:++```````````````````````````````` example+``` ```+aaa+.+<p><code> </code>+aaa</p>+````````````````````````````````+++```````````````````````````````` example+~~~~~~+aaa+~~~ ~~+.+<pre><code>aaa+~~~ ~~+</code></pre>+````````````````````````````````+++Fenced code blocks can interrupt paragraphs, and can be followed+directly by paragraphs, without a blank line between:++```````````````````````````````` example+foo+```+bar+```+baz+.+<p>foo</p>+<pre><code>bar+</code></pre>+<p>baz</p>+````````````````````````````````+++Other blocks can also occur before and after fenced code blocks+without an intervening blank line:++```````````````````````````````` example+foo+---+~~~+bar+~~~+# baz+.+<h2>foo</h2>+<pre><code>bar+</code></pre>+<h1>baz</h1>+````````````````````````````````+++An [info string] can be provided after the opening code fence.+Although this spec doesn't mandate any particular treatment of+the info string, the first word is typically used to specify+the language of the code block. In HTML output, the language is+normally indicated by adding a class to the `code` element consisting+of `language-` followed by the language name.++```````````````````````````````` example+```ruby+def foo(x)+ return 3+end+```+.+<pre><code class="language-ruby">def foo(x)+ return 3+end+</code></pre>+````````````````````````````````+++```````````````````````````````` example+~~~~ ruby startline=3 $%@#$+def foo(x)+ return 3+end+~~~~~~~+.+<pre><code class="language-ruby">def foo(x)+ return 3+end+</code></pre>+````````````````````````````````+++```````````````````````````````` example+````;+````+.+<pre><code class="language-;"></code></pre>+````````````````````````````````+++[Info strings] for backtick code blocks cannot contain backticks:++```````````````````````````````` example+``` aa ```+foo+.+<p><code>aa</code>+foo</p>+````````````````````````````````+++[Info strings] for tilde code blocks can contain backticks and tildes:++```````````````````````````````` example+~~~ aa ``` ~~~+foo+~~~+.+<pre><code class="language-aa">foo+</code></pre>+````````````````````````````````+++Closing code fences cannot have [info strings]:++```````````````````````````````` example+```+``` aaa+```+.+<pre><code>``` aaa+</code></pre>+````````````````````````````````++++## HTML blocks++An [HTML block](@) is a group of lines that is treated+as raw HTML (and will not be escaped in HTML output).++There are seven kinds of [HTML block], which can be defined by their+start and end conditions. The block begins with a line that meets a+[start condition](@) (after up to three spaces optional indentation).+It ends with the first subsequent line that meets a matching [end+condition](@), or the last line of the document, or the last line of+the [container block](#container-blocks) containing the current HTML+block, if no line is encountered that meets the [end condition]. If+the first line meets both the [start condition] and the [end+condition], the block will contain just that line.++1. **Start condition:** line begins with the string `<script`,+`<pre`, or `<style` (case-insensitive), followed by whitespace,+the string `>`, or the end of the line.\+**End condition:** line contains an end tag+`</script>`, `</pre>`, or `</style>` (case-insensitive; it+need not match the start tag).++2. **Start condition:** line begins with the string `<!--`.\+**End condition:** line contains the string `-->`.++3. **Start condition:** line begins with the string `<?`.\+**End condition:** line contains the string `?>`.++4. **Start condition:** line begins with the string `<!`+followed by an uppercase ASCII letter.\+**End condition:** line contains the character `>`.++5. **Start condition:** line begins with the string+`<![CDATA[`.\+**End condition:** line contains the string `]]>`.++6. **Start condition:** line begins the string `<` or `</`+followed by one of the strings (case-insensitive) `address`,+`article`, `aside`, `base`, `basefont`, `blockquote`, `body`,+`caption`, `center`, `col`, `colgroup`, `dd`, `details`, `dialog`,+`dir`, `div`, `dl`, `dt`, `fieldset`, `figcaption`, `figure`,+`footer`, `form`, `frame`, `frameset`,+`h1`, `h2`, `h3`, `h4`, `h5`, `h6`, `head`, `header`, `hr`,+`html`, `iframe`, `legend`, `li`, `link`, `main`, `menu`, `menuitem`,+`nav`, `noframes`, `ol`, `optgroup`, `option`, `p`, `param`,+`section`, `source`, `summary`, `table`, `tbody`, `td`,+`tfoot`, `th`, `thead`, `title`, `tr`, `track`, `ul`, followed+by [whitespace], the end of the line, the string `>`, or+the string `/>`.\+**End condition:** line is followed by a [blank line].++7. **Start condition:** line begins with a complete [open tag]+(with any [tag name] other than `script`,+`style`, or `pre`) or a complete [closing tag],+followed only by [whitespace] or the end of the line.\+**End condition:** line is followed by a [blank line].++HTML blocks continue until they are closed by their appropriate+[end condition], or the last line of the document or other [container+block](#container-blocks). This means any HTML **within an HTML+block** that might otherwise be recognised as a start condition will+be ignored by the parser and passed through as-is, without changing+the parser's state.++For instance, `<pre>` within a HTML block started by `<table>` will not affect+the parser state; as the HTML block was started in by start condition 6, it+will end at any blank line. This can be surprising:++```````````````````````````````` example+<table><tr><td>+<pre>+**Hello**,++_world_.+</pre>+</td></tr></table>+.+<table><tr><td>+<pre>+**Hello**,+<p><em>world</em>.+</pre></p>+</td></tr></table>+````````````````````````````````++In this case, the HTML block is terminated by the newline — the `**Hello**`+text remains verbatim — and regular parsing resumes, with a paragraph,+emphasised `world` and inline and block HTML following.++All types of [HTML blocks] except type 7 may interrupt+a paragraph. Blocks of type 7 may not interrupt a paragraph.+(This restriction is intended to prevent unwanted interpretation+of long tags inside a wrapped paragraph as starting HTML blocks.)++Some simple examples follow. Here are some basic HTML blocks+of type 6:++```````````````````````````````` example+<table>+ <tr>+ <td>+ hi+ </td>+ </tr>+</table>++okay.+.+<table>+ <tr>+ <td>+ hi+ </td>+ </tr>+</table>+<p>okay.</p>+````````````````````````````````+++```````````````````````````````` example+ <div>+ *hello*+ <foo><a>+.+ <div>+ *hello*+ <foo><a>+````````````````````````````````+++A block can also start with a closing tag:++```````````````````````````````` example+</div>+*foo*+.+</div>+*foo*+````````````````````````````````+++Here we have two HTML blocks with a Markdown paragraph between them:++```````````````````````````````` example+<DIV CLASS="foo">++*Markdown*++</DIV>+.+<DIV CLASS="foo">+<p><em>Markdown</em></p>+</DIV>+````````````````````````````````+++The tag on the first line can be partial, as long+as it is split where there would be whitespace:++```````````````````````````````` example+<div id="foo"+ class="bar">+</div>+.+<div id="foo"+ class="bar">+</div>+````````````````````````````````+++```````````````````````````````` example+<div id="foo" class="bar+ baz">+</div>+.+<div id="foo" class="bar+ baz">+</div>+````````````````````````````````+++An open tag need not be closed:+```````````````````````````````` example+<div>+*foo*++*bar*+.+<div>+*foo*+<p><em>bar</em></p>+````````````````````````````````++++A partial tag need not even be completed (garbage+in, garbage out):++```````````````````````````````` example+<div id="foo"+*hi*+.+<div id="foo"+*hi*+````````````````````````````````+++```````````````````````````````` example+<div class+foo+.+<div class+foo+````````````````````````````````+++The initial tag doesn't even need to be a valid+tag, as long as it starts like one:++```````````````````````````````` example+<div *???-&&&-<---+*foo*+.+<div *???-&&&-<---+*foo*+````````````````````````````````+++In type 6 blocks, the initial tag need not be on a line by+itself:++```````````````````````````````` example+<div><a href="bar">*foo*</a></div>+.+<div><a href="bar">*foo*</a></div>+````````````````````````````````+++```````````````````````````````` example+<table><tr><td>+foo+</td></tr></table>+.+<table><tr><td>+foo+</td></tr></table>+````````````````````````````````+++Everything until the next blank line or end of document+gets included in the HTML block. So, in the following+example, what looks like a Markdown code block+is actually part of the HTML block, which continues until a blank+line or the end of the document is reached:++```````````````````````````````` example+<div></div>+``` c+int x = 33;+```+.+<div></div>+``` c+int x = 33;+```+````````````````````````````````+++To start an [HTML block] with a tag that is *not* in the+list of block-level tags in (6), you must put the tag by+itself on the first line (and it must be complete):++```````````````````````````````` example+<a href="foo">+*bar*+</a>+.+<a href="foo">+*bar*+</a>+````````````````````````````````+++In type 7 blocks, the [tag name] can be anything:++```````````````````````````````` example+<Warning>+*bar*+</Warning>+.+<Warning>+*bar*+</Warning>+````````````````````````````````+++```````````````````````````````` example+<i class="foo">+*bar*+</i>+.+<i class="foo">+*bar*+</i>+````````````````````````````````+++```````````````````````````````` example+</ins>+*bar*+.+</ins>+*bar*+````````````````````````````````+++These rules are designed to allow us to work with tags that+can function as either block-level or inline-level tags.+The `<del>` tag is a nice example. We can surround content with+`<del>` tags in three different ways. In this case, we get a raw+HTML block, because the `<del>` tag is on a line by itself:++```````````````````````````````` example+<del>+*foo*+</del>+.+<del>+*foo*+</del>+````````````````````````````````+++In this case, we get a raw HTML block that just includes+the `<del>` tag (because it ends with the following blank+line). So the contents get interpreted as CommonMark:++```````````````````````````````` example+<del>++*foo*++</del>+.+<del>+<p><em>foo</em></p>+</del>+````````````````````````````````+++Finally, in this case, the `<del>` tags are interpreted+as [raw HTML] *inside* the CommonMark paragraph. (Because+the tag is not on a line by itself, we get inline HTML+rather than an [HTML block].)++```````````````````````````````` example+<del>*foo*</del>+.+<p><del><em>foo</em></del></p>+````````````````````````````````+++HTML tags designed to contain literal content+(`script`, `style`, `pre`), comments, processing instructions,+and declarations are treated somewhat differently.+Instead of ending at the first blank line, these blocks+end at the first line containing a corresponding end tag.+As a result, these blocks can contain blank lines:++A pre tag (type 1):++```````````````````````````````` example+<pre language="haskell"><code>+import Text.HTML.TagSoup++main :: IO ()+main = print $ parseTags tags+</code></pre>+okay+.+<pre language="haskell"><code>+import Text.HTML.TagSoup++main :: IO ()+main = print $ parseTags tags+</code></pre>+<p>okay</p>+````````````````````````````````+++A script tag (type 1):++```````````````````````````````` example+<script type="text/javascript">+// JavaScript example++document.getElementById("demo").innerHTML = "Hello JavaScript!";+</script>+okay+.+<script type="text/javascript">+// JavaScript example++document.getElementById("demo").innerHTML = "Hello JavaScript!";+</script>+<p>okay</p>+````````````````````````````````+++A style tag (type 1):++```````````````````````````````` example+<style+ type="text/css">+h1 {color:red;}++p {color:blue;}+</style>+okay+.+<style+ type="text/css">+h1 {color:red;}++p {color:blue;}+</style>+<p>okay</p>+````````````````````````````````+++If there is no matching end tag, the block will end at the+end of the document (or the enclosing [block quote][block quotes]+or [list item][list items]):++```````````````````````````````` example+<style+ type="text/css">++foo+.+<style+ type="text/css">++foo+````````````````````````````````+++```````````````````````````````` example+> <div>+> foo++bar+.+<blockquote>+<div>+foo+</blockquote>+<p>bar</p>+````````````````````````````````+++```````````````````````````````` example+- <div>+- foo+.+<ul>+<li>+<div>+</li>+<li>foo</li>+</ul>+````````````````````````````````+++The end tag can occur on the same line as the start tag:++```````````````````````````````` example+<style>p{color:red;}</style>+*foo*+.+<style>p{color:red;}</style>+<p><em>foo</em></p>+````````````````````````````````+++```````````````````````````````` example+<!-- foo -->*bar*+*baz*+.+<!-- foo -->*bar*+<p><em>baz</em></p>+````````````````````````````````+++Note that anything on the last line after the+end tag will be included in the [HTML block]:++```````````````````````````````` example+<script>+foo+</script>1. *bar*+.+<script>+foo+</script>1. *bar*+````````````````````````````````+++A comment (type 2):++```````````````````````````````` example+<!-- Foo++bar+ baz -->+okay+.+<!-- Foo++bar+ baz -->+<p>okay</p>+````````````````````````````````++++A processing instruction (type 3):++```````````````````````````````` example+<?php++ echo '>';++?>+okay+.+<?php++ echo '>';++?>+<p>okay</p>+````````````````````````````````+++A declaration (type 4):++```````````````````````````````` example+<!DOCTYPE html>+.+<!DOCTYPE html>+````````````````````````````````+++CDATA (type 5):++```````````````````````````````` example+<![CDATA[+function matchwo(a,b)+{+ if (a < b && a < 0) then {+ return 1;++ } else {++ return 0;+ }+}+]]>+okay+.+<![CDATA[+function matchwo(a,b)+{+ if (a < b && a < 0) then {+ return 1;++ } else {++ return 0;+ }+}+]]>+<p>okay</p>+````````````````````````````````+++The opening tag can be indented 1-3 spaces, but not 4:++```````````````````````````````` example+ <!-- foo -->++ <!-- foo -->+.+ <!-- foo -->+<pre><code><!-- foo -->+</code></pre>+````````````````````````````````+++```````````````````````````````` example+ <div>++ <div>+.+ <div>+<pre><code><div>+</code></pre>+````````````````````````````````+++An HTML block of types 1--6 can interrupt a paragraph, and need not be+preceded by a blank line.++```````````````````````````````` example+Foo+<div>+bar+</div>+.+<p>Foo</p>+<div>+bar+</div>+````````````````````````````````+++However, a following blank line is needed, except at the end of+a document, and except for blocks of types 1--5, [above][HTML+block]:++```````````````````````````````` example+<div>+bar+</div>+*foo*+.+<div>+bar+</div>+*foo*+````````````````````````````````+++HTML blocks of type 7 cannot interrupt a paragraph:++```````````````````````````````` example+Foo+<a href="bar">+baz+.+<p>Foo+<a href="bar">+baz</p>+````````````````````````````````+++This rule differs from John Gruber's original Markdown syntax+specification, which says:++> The only restrictions are that block-level HTML elements —+> e.g. `<div>`, `<table>`, `<pre>`, `<p>`, etc. — must be separated from+> surrounding content by blank lines, and the start and end tags of the+> block should not be indented with tabs or spaces.++In some ways Gruber's rule is more restrictive than the one given+here:++- It requires that an HTML block be preceded by a blank line.+- It does not allow the start tag to be indented.+- It requires a matching end tag, which it also does not allow to+ be indented.++Most Markdown implementations (including some of Gruber's own) do not+respect all of these restrictions.++There is one respect, however, in which Gruber's rule is more liberal+than the one given here, since it allows blank lines to occur inside+an HTML block. There are two reasons for disallowing them here.+First, it removes the need to parse balanced tags, which is+expensive and can require backtracking from the end of the document+if no matching end tag is found. Second, it provides a very simple+and flexible way of including Markdown content inside HTML tags:+simply separate the Markdown from the HTML using blank lines:++Compare:++```````````````````````````````` example+<div>++*Emphasized* text.++</div>+.+<div>+<p><em>Emphasized</em> text.</p>+</div>+````````````````````````````````+++```````````````````````````````` example+<div>+*Emphasized* text.+</div>+.+<div>+*Emphasized* text.+</div>+````````````````````````````````+++Some Markdown implementations have adopted a convention of+interpreting content inside tags as text if the open tag has+the attribute `markdown=1`. The rule given above seems a simpler and+more elegant way of achieving the same expressive power, which is also+much simpler to parse.++The main potential drawback is that one can no longer paste HTML+blocks into Markdown documents with 100% reliability. However,+*in most cases* this will work fine, because the blank lines in+HTML are usually followed by HTML block tags. For example:++```````````````````````````````` example+<table>++<tr>++<td>+Hi+</td>++</tr>++</table>+.+<table>+<tr>+<td>+Hi+</td>+</tr>+</table>+````````````````````````````````+++There are problems, however, if the inner tags are indented+*and* separated by spaces, as then they will be interpreted as+an indented code block:++```````````````````````````````` example+<table>++ <tr>++ <td>+ Hi+ </td>++ </tr>++</table>+.+<table>+ <tr>+<pre><code><td>+ Hi+</td>+</code></pre>+ </tr>+</table>+````````````````````````````````+++Fortunately, blank lines are usually not necessary and can be+deleted. The exception is inside `<pre>` tags, but as described+[above][HTML blocks], raw HTML blocks starting with `<pre>`+*can* contain blank lines.++## Link reference definitions++A [link reference definition](@)+consists of a [link label], indented up to three spaces, followed+by a colon (`:`), optional [whitespace] (including up to one+[line ending]), a [link destination],+optional [whitespace] (including up to one+[line ending]), and an optional [link+title], which if it is present must be separated+from the [link destination] by [whitespace].+No further [non-whitespace characters] may occur on the line.++A [link reference definition]+does not correspond to a structural element of a document. Instead, it+defines a label which can be used in [reference links]+and reference-style [images] elsewhere in the document. [Link+reference definitions] can come either before or after the links that use+them.++```````````````````````````````` example+[foo]: /url "title"++[foo]+.+<p><a href="/url" title="title">foo</a></p>+````````````````````````````````+++```````````````````````````````` example+ [foo]: + /url + 'the title' ++[foo]+.+<p><a href="/url" title="the title">foo</a></p>+````````````````````````````````+++```````````````````````````````` example+[Foo*bar\]]:my_(url) 'title (with parens)'++[Foo*bar\]]+.+<p><a href="my_(url)" title="title (with parens)">Foo*bar]</a></p>+````````````````````````````````+++```````````````````````````````` example+[Foo bar]:+<my url>+'title'++[Foo bar]+.+<p><a href="my%20url" title="title">Foo bar</a></p>+````````````````````````````````+++The title may extend over multiple lines:++```````````````````````````````` example+[foo]: /url '+title+line1+line2+'++[foo]+.+<p><a href="/url" title="+title+line1+line2+">foo</a></p>+````````````````````````````````+++However, it may not contain a [blank line]:++```````````````````````````````` example+[foo]: /url 'title++with blank line'++[foo]+.+<p>[foo]: /url 'title</p>+<p>with blank line'</p>+<p>[foo]</p>+````````````````````````````````+++The title may be omitted:++```````````````````````````````` example+[foo]:+/url++[foo]+.+<p><a href="/url">foo</a></p>+````````````````````````````````+++The link destination may not be omitted:++```````````````````````````````` example+[foo]:++[foo]+.+<p>[foo]:</p>+<p>[foo]</p>+````````````````````````````````++ However, an empty link destination may be specified using+ angle brackets:++```````````````````````````````` example+[foo]: <>++[foo]+.+<p><a href="">foo</a></p>+````````````````````````````````++The title must be separated from the link destination by+whitespace:++```````````````````````````````` example+[foo]: <bar>(baz)++[foo]+.+<p>[foo]: <bar>(baz)</p>+<p>[foo]</p>+````````````````````````````````+++Both title and destination can contain backslash escapes+and literal backslashes:++```````````````````````````````` example+[foo]: /url\bar\*baz "foo\"bar\baz"++[foo]+.+<p><a href="/url%5Cbar*baz" title="foo"bar\baz">foo</a></p>+````````````````````````````````+++A link can come before its corresponding definition:++```````````````````````````````` example+[foo]++[foo]: url+.+<p><a href="url">foo</a></p>+````````````````````````````````+++If there are several matching definitions, the first one takes+precedence:++```````````````````````````````` example+[foo]++[foo]: first+[foo]: second+.+<p><a href="first">foo</a></p>+````````````````````````````````+++As noted in the section on [Links], matching of labels is+case-insensitive (see [matches]).++```````````````````````````````` example+[FOO]: /url++[Foo]+.+<p><a href="/url">Foo</a></p>+````````````````````````````````+++```````````````````````````````` example+[ΑΓΩ]: /φου++[αγω]+.+<p><a href="/%CF%86%CE%BF%CF%85">αγω</a></p>+````````````````````````````````+++Here is a link reference definition with no corresponding link.+It contributes nothing to the document.++```````````````````````````````` example+[foo]: /url+.+````````````````````````````````+++Here is another one:++```````````````````````````````` example+[+foo+]: /url+bar+.+<p>bar</p>+````````````````````````````````+++This is not a link reference definition, because there are+[non-whitespace characters] after the title:++```````````````````````````````` example+[foo]: /url "title" ok+.+<p>[foo]: /url "title" ok</p>+````````````````````````````````+++This is a link reference definition, but it has no title:++```````````````````````````````` example+[foo]: /url+"title" ok+.+<p>"title" ok</p>+````````````````````````````````+++This is not a link reference definition, because it is indented+four spaces:++```````````````````````````````` example+ [foo]: /url "title"++[foo]+.+<pre><code>[foo]: /url "title"+</code></pre>+<p>[foo]</p>+````````````````````````````````+++This is not a link reference definition, because it occurs inside+a code block:++```````````````````````````````` example+```+[foo]: /url+```++[foo]+.+<pre><code>[foo]: /url+</code></pre>+<p>[foo]</p>+````````````````````````````````+++A [link reference definition] cannot interrupt a paragraph.++```````````````````````````````` example+Foo+[bar]: /baz++[bar]+.+<p>Foo+[bar]: /baz</p>+<p>[bar]</p>+````````````````````````````````+++However, it can directly follow other block elements, such as headings+and thematic breaks, and it need not be followed by a blank line.++```````````````````````````````` example+# [Foo]+[foo]: /url+> bar+.+<h1><a href="/url">Foo</a></h1>+<blockquote>+<p>bar</p>+</blockquote>+````````````````````````````````++```````````````````````````````` example+[foo]: /url+bar+===+[foo]+.+<h1>bar</h1>+<p><a href="/url">foo</a></p>+````````````````````````````````++```````````````````````````````` example+[foo]: /url+===+[foo]+.+<p>===+<a href="/url">foo</a></p>+````````````````````````````````+++Several [link reference definitions]+can occur one after another, without intervening blank lines.++```````````````````````````````` example+[foo]: /foo-url "foo"+[bar]: /bar-url+ "bar"+[baz]: /baz-url++[foo],+[bar],+[baz]+.+<p><a href="/foo-url" title="foo">foo</a>,+<a href="/bar-url" title="bar">bar</a>,+<a href="/baz-url">baz</a></p>+````````````````````````````````+++[Link reference definitions] can occur+inside block containers, like lists and block quotations. They+affect the entire document, not just the container in which they+are defined:++```````````````````````````````` example+[foo]++> [foo]: /url+.+<p><a href="/url">foo</a></p>+<blockquote>+</blockquote>+````````````````````````````````+++Whether something is a [link reference definition] is+independent of whether the link reference it defines is+used in the document. Thus, for example, the following+document contains just a link reference definition, and+no visible content:++```````````````````````````````` example+[foo]: /url+.+````````````````````````````````+++## Paragraphs++A sequence of non-blank lines that cannot be interpreted as other+kinds of blocks forms a [paragraph](@).+The contents of the paragraph are the result of parsing the+paragraph's raw content as inlines. The paragraph's raw content+is formed by concatenating the lines and removing initial and final+[whitespace].++A simple example with two paragraphs:++```````````````````````````````` example+aaa++bbb+.+<p>aaa</p>+<p>bbb</p>+````````````````````````````````+++Paragraphs can contain multiple lines, but no blank lines:++```````````````````````````````` example+aaa+bbb++ccc+ddd+.+<p>aaa+bbb</p>+<p>ccc+ddd</p>+````````````````````````````````+++Multiple blank lines between paragraph have no effect:++```````````````````````````````` example+aaa+++bbb+.+<p>aaa</p>+<p>bbb</p>+````````````````````````````````+++Leading spaces are skipped:++```````````````````````````````` example+ aaa+ bbb+.+<p>aaa+bbb</p>+````````````````````````````````+++Lines after the first may be indented any amount, since indented+code blocks cannot interrupt paragraphs.++```````````````````````````````` example+aaa+ bbb+ ccc+.+<p>aaa+bbb+ccc</p>+````````````````````````````````+++However, the first line may be indented at most three spaces,+or an indented code block will be triggered:++```````````````````````````````` example+ aaa+bbb+.+<p>aaa+bbb</p>+````````````````````````````````+++```````````````````````````````` example+ aaa+bbb+.+<pre><code>aaa+</code></pre>+<p>bbb</p>+````````````````````````````````+++Final spaces are stripped before inline parsing, so a paragraph+that ends with two or more spaces will not end with a [hard line+break]:++```````````````````````````````` example+aaa +bbb +.+<p>aaa<br />+bbb</p>+````````````````````````````````+++## Blank lines++[Blank lines] between block-level elements are ignored,+except for the role they play in determining whether a [list]+is [tight] or [loose].++Blank lines at the beginning and end of the document are also ignored.++```````````````````````````````` example+ ++aaa+ ++# aaa++ +.+<p>aaa</p>+<h1>aaa</h1>+````````````````````````````````++++# Container blocks++A [container block](#container-blocks) is a block that has other+blocks as its contents. There are two basic kinds of container blocks:+[block quotes] and [list items].+[Lists] are meta-containers for [list items].++We define the syntax for container blocks recursively. The general+form of the definition is:++> If X is a sequence of blocks, then the result of+> transforming X in such-and-such a way is a container of type Y+> with these blocks as its content.++So, we explain what counts as a block quote or list item by explaining+how these can be *generated* from their contents. This should suffice+to define the syntax, although it does not give a recipe for *parsing*+these constructions. (A recipe is provided below in the section entitled+[A parsing strategy](#appendix-a-parsing-strategy).)++## Block quotes++A [block quote marker](@)+consists of 0-3 spaces of initial indent, plus (a) the character `>` together+with a following space, or (b) a single character `>` not followed by a space.++The following rules define [block quotes]:++1. **Basic case.** If a string of lines *Ls* constitute a sequence+ of blocks *Bs*, then the result of prepending a [block quote+ marker] to the beginning of each line in *Ls*+ is a [block quote](#block-quotes) containing *Bs*.++2. **Laziness.** If a string of lines *Ls* constitute a [block+ quote](#block-quotes) with contents *Bs*, then the result of deleting+ the initial [block quote marker] from one or+ more lines in which the next [non-whitespace character] after the [block+ quote marker] is [paragraph continuation+ text] is a block quote with *Bs* as its content.+ [Paragraph continuation text](@) is text+ that will be parsed as part of the content of a paragraph, but does+ not occur at the beginning of the paragraph.++3. **Consecutiveness.** A document cannot contain two [block+ quotes] in a row unless there is a [blank line] between them.++Nothing else counts as a [block quote](#block-quotes).++Here is a simple example:++```````````````````````````````` example+> # Foo+> bar+> baz+.+<blockquote>+<h1>Foo</h1>+<p>bar+baz</p>+</blockquote>+````````````````````````````````+++The spaces after the `>` characters can be omitted:++```````````````````````````````` example+># Foo+>bar+> baz+.+<blockquote>+<h1>Foo</h1>+<p>bar+baz</p>+</blockquote>+````````````````````````````````+++The `>` characters can be indented 1-3 spaces:++```````````````````````````````` example+ > # Foo+ > bar+ > baz+.+<blockquote>+<h1>Foo</h1>+<p>bar+baz</p>+</blockquote>+````````````````````````````````+++Four spaces gives us a code block:++```````````````````````````````` example+ > # Foo+ > bar+ > baz+.+<pre><code>> # Foo+> bar+> baz+</code></pre>+````````````````````````````````+++The Laziness clause allows us to omit the `>` before+[paragraph continuation text]:++```````````````````````````````` example+> # Foo+> bar+baz+.+<blockquote>+<h1>Foo</h1>+<p>bar+baz</p>+</blockquote>+````````````````````````````````+++A block quote can contain some lazy and some non-lazy+continuation lines:++```````````````````````````````` example+> bar+baz+> foo+.+<blockquote>+<p>bar+baz+foo</p>+</blockquote>+````````````````````````````````+++Laziness only applies to lines that would have been continuations of+paragraphs had they been prepended with [block quote markers].+For example, the `> ` cannot be omitted in the second line of++``` markdown+> foo+> ---+```++without changing the meaning:++```````````````````````````````` example+> foo+---+.+<blockquote>+<p>foo</p>+</blockquote>+<hr />+````````````````````````````````+++Similarly, if we omit the `> ` in the second line of++``` markdown+> - foo+> - bar+```++then the block quote ends after the first line:++```````````````````````````````` example+> - foo+- bar+.+<blockquote>+<ul>+<li>foo</li>+</ul>+</blockquote>+<ul>+<li>bar</li>+</ul>+````````````````````````````````+++For the same reason, we can't omit the `> ` in front of+subsequent lines of an indented or fenced code block:++```````````````````````````````` example+> foo+ bar+.+<blockquote>+<pre><code>foo+</code></pre>+</blockquote>+<pre><code>bar+</code></pre>+````````````````````````````````+++```````````````````````````````` example+> ```+foo+```+.+<blockquote>+<pre><code></code></pre>+</blockquote>+<p>foo</p>+<pre><code></code></pre>+````````````````````````````````+++Note that in the following case, we have a [lazy+continuation line]:++```````````````````````````````` example+> foo+ - bar+.+<blockquote>+<p>foo+- bar</p>+</blockquote>+````````````````````````````````+++To see why, note that in++```markdown+> foo+> - bar+```++the `- bar` is indented too far to start a list, and can't+be an indented code block because indented code blocks cannot+interrupt paragraphs, so it is [paragraph continuation text].++A block quote can be empty:++```````````````````````````````` example+>+.+<blockquote>+</blockquote>+````````````````````````````````+++```````````````````````````````` example+>+> +> +.+<blockquote>+</blockquote>+````````````````````````````````+++A block quote can have initial or final blank lines:++```````````````````````````````` example+>+> foo+> +.+<blockquote>+<p>foo</p>+</blockquote>+````````````````````````````````+++A blank line always separates block quotes:++```````````````````````````````` example+> foo++> bar+.+<blockquote>+<p>foo</p>+</blockquote>+<blockquote>+<p>bar</p>+</blockquote>+````````````````````````````````+++(Most current Markdown implementations, including John Gruber's+original `Markdown.pl`, will parse this example as a single block quote+with two paragraphs. But it seems better to allow the author to decide+whether two block quotes or one are wanted.)++Consecutiveness means that if we put these block quotes together,+we get a single block quote:++```````````````````````````````` example+> foo+> bar+.+<blockquote>+<p>foo+bar</p>+</blockquote>+````````````````````````````````+++To get a block quote with two paragraphs, use:++```````````````````````````````` example+> foo+>+> bar+.+<blockquote>+<p>foo</p>+<p>bar</p>+</blockquote>+````````````````````````````````+++Block quotes can interrupt paragraphs:++```````````````````````````````` example+foo+> bar+.+<p>foo</p>+<blockquote>+<p>bar</p>+</blockquote>+````````````````````````````````+++In general, blank lines are not needed before or after block+quotes:++```````````````````````````````` example+> aaa+***+> bbb+.+<blockquote>+<p>aaa</p>+</blockquote>+<hr />+<blockquote>+<p>bbb</p>+</blockquote>+````````````````````````````````+++However, because of laziness, a blank line is needed between+a block quote and a following paragraph:++```````````````````````````````` example+> bar+baz+.+<blockquote>+<p>bar+baz</p>+</blockquote>+````````````````````````````````+++```````````````````````````````` example+> bar++baz+.+<blockquote>+<p>bar</p>+</blockquote>+<p>baz</p>+````````````````````````````````+++```````````````````````````````` example+> bar+>+baz+.+<blockquote>+<p>bar</p>+</blockquote>+<p>baz</p>+````````````````````````````````+++It is a consequence of the Laziness rule that any number+of initial `>`s may be omitted on a continuation line of a+nested block quote:++```````````````````````````````` example+> > > foo+bar+.+<blockquote>+<blockquote>+<blockquote>+<p>foo+bar</p>+</blockquote>+</blockquote>+</blockquote>+````````````````````````````````+++```````````````````````````````` example+>>> foo+> bar+>>baz+.+<blockquote>+<blockquote>+<blockquote>+<p>foo+bar+baz</p>+</blockquote>+</blockquote>+</blockquote>+````````````````````````````````+++When including an indented code block in a block quote,+remember that the [block quote marker] includes+both the `>` and a following space. So *five spaces* are needed after+the `>`:++```````````````````````````````` example+> code++> not code+.+<blockquote>+<pre><code>code+</code></pre>+</blockquote>+<blockquote>+<p>not code</p>+</blockquote>+````````````````````````````````++++## List items++A [list marker](@) is a+[bullet list marker] or an [ordered list marker].++A [bullet list marker](@)+is a `-`, `+`, or `*` character.++An [ordered list marker](@)+is a sequence of 1--9 arabic digits (`0-9`), followed by either a+`.` character or a `)` character. (The reason for the length+limit is that with 10 digits we start seeing integer overflows+in some browsers.)++The following rules define [list items]:++1. **Basic case.** If a sequence of lines *Ls* constitute a sequence of+ blocks *Bs* starting with a [non-whitespace character], and *M* is a+ list marker of width *W* followed by 1 ≤ *N* ≤ 4 spaces, then the result+ of prepending *M* and the following spaces to the first line of+ *Ls*, and indenting subsequent lines of *Ls* by *W + N* spaces, is a+ list item with *Bs* as its contents. The type of the list item+ (bullet or ordered) is determined by the type of its list marker.+ If the list item is ordered, then it is also assigned a start+ number, based on the ordered list marker.++ Exceptions:++ 1. When the first list item in a [list] interrupts+ a paragraph---that is, when it starts on a line that would+ otherwise count as [paragraph continuation text]---then (a)+ the lines *Ls* must not begin with a blank line, and (b) if+ the list item is ordered, the start number must be 1.+ 2. If any line is a [thematic break][thematic breaks] then+ that line is not a list item.++For example, let *Ls* be the lines++```````````````````````````````` example+A paragraph+with two lines.++ indented code++> A block quote.+.+<p>A paragraph+with two lines.</p>+<pre><code>indented code+</code></pre>+<blockquote>+<p>A block quote.</p>+</blockquote>+````````````````````````````````+++And let *M* be the marker `1.`, and *N* = 2. Then rule #1 says+that the following is an ordered list item with start number 1,+and the same contents as *Ls*:++```````````````````````````````` example+1. A paragraph+ with two lines.++ indented code++ > A block quote.+.+<ol>+<li>+<p>A paragraph+with two lines.</p>+<pre><code>indented code+</code></pre>+<blockquote>+<p>A block quote.</p>+</blockquote>+</li>+</ol>+````````````````````````````````+++The most important thing to notice is that the position of+the text after the list marker determines how much indentation+is needed in subsequent blocks in the list item. If the list+marker takes up two spaces, and there are three spaces between+the list marker and the next [non-whitespace character], then blocks+must be indented five spaces in order to fall under the list+item.++Here are some examples showing how far content must be indented to be+put under the list item:++```````````````````````````````` example+- one++ two+.+<ul>+<li>one</li>+</ul>+<p>two</p>+````````````````````````````````+++```````````````````````````````` example+- one++ two+.+<ul>+<li>+<p>one</p>+<p>two</p>+</li>+</ul>+````````````````````````````````+++```````````````````````````````` example+ - one++ two+.+<ul>+<li>one</li>+</ul>+<pre><code> two+</code></pre>+````````````````````````````````+++```````````````````````````````` example+ - one++ two+.+<ul>+<li>+<p>one</p>+<p>two</p>+</li>+</ul>+````````````````````````````````+++It is tempting to think of this in terms of columns: the continuation+blocks must be indented at least to the column of the first+[non-whitespace character] after the list marker. However, that is not quite right.+The spaces after the list marker determine how much relative indentation+is needed. Which column this indentation reaches will depend on+how the list item is embedded in other constructions, as shown by+this example:++```````````````````````````````` example+ > > 1. one+>>+>> two+.+<blockquote>+<blockquote>+<ol>+<li>+<p>one</p>+<p>two</p>+</li>+</ol>+</blockquote>+</blockquote>+````````````````````````````````+++Here `two` occurs in the same column as the list marker `1.`,+but is actually contained in the list item, because there is+sufficient indentation after the last containing blockquote marker.++The converse is also possible. In the following example, the word `two`+occurs far to the right of the initial text of the list item, `one`, but+it is not considered part of the list item, because it is not indented+far enough past the blockquote marker:++```````````````````````````````` example+>>- one+>>+ > > two+.+<blockquote>+<blockquote>+<ul>+<li>one</li>+</ul>+<p>two</p>+</blockquote>+</blockquote>+````````````````````````````````+++Note that at least one space is needed between the list marker and+any following content, so these are not list items:++```````````````````````````````` example+-one++2.two+.+<p>-one</p>+<p>2.two</p>+````````````````````````````````+++A list item may contain blocks that are separated by more than+one blank line.++```````````````````````````````` example+- foo+++ bar+.+<ul>+<li>+<p>foo</p>+<p>bar</p>+</li>+</ul>+````````````````````````````````+++A list item may contain any kind of block:++```````````````````````````````` example+1. foo++ ```+ bar+ ```++ baz++ > bam+.+<ol>+<li>+<p>foo</p>+<pre><code>bar+</code></pre>+<p>baz</p>+<blockquote>+<p>bam</p>+</blockquote>+</li>+</ol>+````````````````````````````````+++A list item that contains an indented code block will preserve+empty lines within the code block verbatim.++```````````````````````````````` example+- Foo++ bar+++ baz+.+<ul>+<li>+<p>Foo</p>+<pre><code>bar+++baz+</code></pre>+</li>+</ul>+````````````````````````````````++Note that ordered list start numbers must be nine digits or less:++```````````````````````````````` example+123456789. ok+.+<ol start="123456789">+<li>ok</li>+</ol>+````````````````````````````````+++```````````````````````````````` example+1234567890. not ok+.+<p>1234567890. not ok</p>+````````````````````````````````+++A start number may begin with 0s:++```````````````````````````````` example+0. ok+.+<ol start="0">+<li>ok</li>+</ol>+````````````````````````````````+++```````````````````````````````` example+003. ok+.+<ol start="3">+<li>ok</li>+</ol>+````````````````````````````````+++A start number may not be negative:++```````````````````````````````` example+-1. not ok+.+<p>-1. not ok</p>+````````````````````````````````++++2. **Item starting with indented code.** If a sequence of lines *Ls*+ constitute a sequence of blocks *Bs* starting with an indented code+ block, and *M* is a list marker of width *W* followed by+ one space, then the result of prepending *M* and the following+ space to the first line of *Ls*, and indenting subsequent lines of+ *Ls* by *W + 1* spaces, is a list item with *Bs* as its contents.+ If a line is empty, then it need not be indented. The type of the+ list item (bullet or ordered) is determined by the type of its list+ marker. If the list item is ordered, then it is also assigned a+ start number, based on the ordered list marker.++An indented code block will have to be indented four spaces beyond+the edge of the region where text will be included in the list item.+In the following case that is 6 spaces:++```````````````````````````````` example+- foo++ bar+.+<ul>+<li>+<p>foo</p>+<pre><code>bar+</code></pre>+</li>+</ul>+````````````````````````````````+++And in this case it is 11 spaces:++```````````````````````````````` example+ 10. foo++ bar+.+<ol start="10">+<li>+<p>foo</p>+<pre><code>bar+</code></pre>+</li>+</ol>+````````````````````````````````+++If the *first* block in the list item is an indented code block,+then by rule #2, the contents must be indented *one* space after the+list marker:++```````````````````````````````` example+ indented code++paragraph++ more code+.+<pre><code>indented code+</code></pre>+<p>paragraph</p>+<pre><code>more code+</code></pre>+````````````````````````````````+++```````````````````````````````` example+1. indented code++ paragraph++ more code+.+<ol>+<li>+<pre><code>indented code+</code></pre>+<p>paragraph</p>+<pre><code>more code+</code></pre>+</li>+</ol>+````````````````````````````````+++Note that an additional space indent is interpreted as space+inside the code block:++```````````````````````````````` example+1. indented code++ paragraph++ more code+.+<ol>+<li>+<pre><code> indented code+</code></pre>+<p>paragraph</p>+<pre><code>more code+</code></pre>+</li>+</ol>+````````````````````````````````+++Note that rules #1 and #2 only apply to two cases: (a) cases+in which the lines to be included in a list item begin with a+[non-whitespace character], and (b) cases in which+they begin with an indented code+block. In a case like the following, where the first block begins with+a three-space indent, the rules do not allow us to form a list item by+indenting the whole thing and prepending a list marker:++```````````````````````````````` example+ foo++bar+.+<p>foo</p>+<p>bar</p>+````````````````````````````````+++```````````````````````````````` example+- foo++ bar+.+<ul>+<li>foo</li>+</ul>+<p>bar</p>+````````````````````````````````+++This is not a significant restriction, because when a block begins+with 1-3 spaces indent, the indentation can always be removed without+a change in interpretation, allowing rule #1 to be applied. So, in+the above case:++```````````````````````````````` example+- foo++ bar+.+<ul>+<li>+<p>foo</p>+<p>bar</p>+</li>+</ul>+````````````````````````````````+++3. **Item starting with a blank line.** If a sequence of lines *Ls*+ starting with a single [blank line] constitute a (possibly empty)+ sequence of blocks *Bs*, not separated from each other by more than+ one blank line, and *M* is a list marker of width *W*,+ then the result of prepending *M* to the first line of *Ls*, and+ indenting subsequent lines of *Ls* by *W + 1* spaces, is a list+ item with *Bs* as its contents.+ If a line is empty, then it need not be indented. The type of the+ list item (bullet or ordered) is determined by the type of its list+ marker. If the list item is ordered, then it is also assigned a+ start number, based on the ordered list marker.++Here are some list items that start with a blank line but are not empty:++```````````````````````````````` example+-+ foo+-+ ```+ bar+ ```+-+ baz+.+<ul>+<li>foo</li>+<li>+<pre><code>bar+</code></pre>+</li>+<li>+<pre><code>baz+</code></pre>+</li>+</ul>+````````````````````````````````++When the list item starts with a blank line, the number of spaces+following the list marker doesn't change the required indentation:++```````````````````````````````` example+- + foo+.+<ul>+<li>foo</li>+</ul>+````````````````````````````````+++A list item can begin with at most one blank line.+In the following example, `foo` is not part of the list+item:++```````````````````````````````` example+-++ foo+.+<ul>+<li></li>+</ul>+<p>foo</p>+````````````````````````````````+++Here is an empty bullet list item:++```````````````````````````````` example+- foo+-+- bar+.+<ul>+<li>foo</li>+<li></li>+<li>bar</li>+</ul>+````````````````````````````````+++It does not matter whether there are spaces following the [list marker]:++```````````````````````````````` example+- foo+- +- bar+.+<ul>+<li>foo</li>+<li></li>+<li>bar</li>+</ul>+````````````````````````````````+++Here is an empty ordered list item:++```````````````````````````````` example+1. foo+2.+3. bar+.+<ol>+<li>foo</li>+<li></li>+<li>bar</li>+</ol>+````````````````````````````````+++A list may start or end with an empty list item:++```````````````````````````````` example+*+.+<ul>+<li></li>+</ul>+````````````````````````````````++However, an empty list item cannot interrupt a paragraph:++```````````````````````````````` example+foo+*++foo+1.+.+<p>foo+*</p>+<p>foo+1.</p>+````````````````````````````````+++4. **Indentation.** If a sequence of lines *Ls* constitutes a list item+ according to rule #1, #2, or #3, then the result of indenting each line+ of *Ls* by 1-3 spaces (the same for each line) also constitutes a+ list item with the same contents and attributes. If a line is+ empty, then it need not be indented.++Indented one space:++```````````````````````````````` example+ 1. A paragraph+ with two lines.++ indented code++ > A block quote.+.+<ol>+<li>+<p>A paragraph+with two lines.</p>+<pre><code>indented code+</code></pre>+<blockquote>+<p>A block quote.</p>+</blockquote>+</li>+</ol>+````````````````````````````````+++Indented two spaces:++```````````````````````````````` example+ 1. A paragraph+ with two lines.++ indented code++ > A block quote.+.+<ol>+<li>+<p>A paragraph+with two lines.</p>+<pre><code>indented code+</code></pre>+<blockquote>+<p>A block quote.</p>+</blockquote>+</li>+</ol>+````````````````````````````````+++Indented three spaces:++```````````````````````````````` example+ 1. A paragraph+ with two lines.++ indented code++ > A block quote.+.+<ol>+<li>+<p>A paragraph+with two lines.</p>+<pre><code>indented code+</code></pre>+<blockquote>+<p>A block quote.</p>+</blockquote>+</li>+</ol>+````````````````````````````````+++Four spaces indent gives a code block:++```````````````````````````````` example+ 1. A paragraph+ with two lines.++ indented code++ > A block quote.+.+<pre><code>1. A paragraph+ with two lines.++ indented code++ > A block quote.+</code></pre>+````````````````````````````````++++5. **Laziness.** If a string of lines *Ls* constitute a [list+ item](#list-items) with contents *Bs*, then the result of deleting+ some or all of the indentation from one or more lines in which the+ next [non-whitespace character] after the indentation is+ [paragraph continuation text] is a+ list item with the same contents and attributes. The unindented+ lines are called+ [lazy continuation line](@)s.++Here is an example with [lazy continuation lines]:++```````````````````````````````` example+ 1. A paragraph+with two lines.++ indented code++ > A block quote.+.+<ol>+<li>+<p>A paragraph+with two lines.</p>+<pre><code>indented code+</code></pre>+<blockquote>+<p>A block quote.</p>+</blockquote>+</li>+</ol>+````````````````````````````````+++Indentation can be partially deleted:++```````````````````````````````` example+ 1. A paragraph+ with two lines.+.+<ol>+<li>A paragraph+with two lines.</li>+</ol>+````````````````````````````````+++These examples show how laziness can work in nested structures:++```````````````````````````````` example+> 1. > Blockquote+continued here.+.+<blockquote>+<ol>+<li>+<blockquote>+<p>Blockquote+continued here.</p>+</blockquote>+</li>+</ol>+</blockquote>+````````````````````````````````+++```````````````````````````````` example+> 1. > Blockquote+> continued here.+.+<blockquote>+<ol>+<li>+<blockquote>+<p>Blockquote+continued here.</p>+</blockquote>+</li>+</ol>+</blockquote>+````````````````````````````````++++6. **That's all.** Nothing that is not counted as a list item by rules+ #1--5 counts as a [list item](#list-items).++The rules for sublists follow from the general rules+[above][List items]. A sublist must be indented the same number+of spaces a paragraph would need to be in order to be included+in the list item.++So, in this case we need two spaces indent:++```````````````````````````````` example+- foo+ - bar+ - baz+ - boo+.+<ul>+<li>foo+<ul>+<li>bar+<ul>+<li>baz+<ul>+<li>boo</li>+</ul>+</li>+</ul>+</li>+</ul>+</li>+</ul>+````````````````````````````````+++One is not enough:++```````````````````````````````` example+- foo+ - bar+ - baz+ - boo+.+<ul>+<li>foo</li>+<li>bar</li>+<li>baz</li>+<li>boo</li>+</ul>+````````````````````````````````+++Here we need four, because the list marker is wider:++```````````````````````````````` example+10) foo+ - bar+.+<ol start="10">+<li>foo+<ul>+<li>bar</li>+</ul>+</li>+</ol>+````````````````````````````````+++Three is not enough:++```````````````````````````````` example+10) foo+ - bar+.+<ol start="10">+<li>foo</li>+</ol>+<ul>+<li>bar</li>+</ul>+````````````````````````````````+++A list may be the first block in a list item:++```````````````````````````````` example+- - foo+.+<ul>+<li>+<ul>+<li>foo</li>+</ul>+</li>+</ul>+````````````````````````````````+++```````````````````````````````` example+1. - 2. foo+.+<ol>+<li>+<ul>+<li>+<ol start="2">+<li>foo</li>+</ol>+</li>+</ul>+</li>+</ol>+````````````````````````````````+++A list item can contain a heading:++```````````````````````````````` example+- # Foo+- Bar+ ---+ baz+.+<ul>+<li>+<h1>Foo</h1>+</li>+<li>+<h2>Bar</h2>+baz</li>+</ul>+````````````````````````````````+++### Motivation++John Gruber's Markdown spec says the following about list items:++1. "List markers typically start at the left margin, but may be indented+ by up to three spaces. List markers must be followed by one or more+ spaces or a tab."++2. "To make lists look nice, you can wrap items with hanging indents....+ But if you don't want to, you don't have to."++3. "List items may consist of multiple paragraphs. Each subsequent+ paragraph in a list item must be indented by either 4 spaces or one+ tab."++4. "It looks nice if you indent every line of the subsequent paragraphs,+ but here again, Markdown will allow you to be lazy."++5. "To put a blockquote within a list item, the blockquote's `>`+ delimiters need to be indented."++6. "To put a code block within a list item, the code block needs to be+ indented twice — 8 spaces or two tabs."++These rules specify that a paragraph under a list item must be indented+four spaces (presumably, from the left margin, rather than the start of+the list marker, but this is not said), and that code under a list item+must be indented eight spaces instead of the usual four. They also say+that a block quote must be indented, but not by how much; however, the+example given has four spaces indentation. Although nothing is said+about other kinds of block-level content, it is certainly reasonable to+infer that *all* block elements under a list item, including other+lists, must be indented four spaces. This principle has been called the+*four-space rule*.++The four-space rule is clear and principled, and if the reference+implementation `Markdown.pl` had followed it, it probably would have+become the standard. However, `Markdown.pl` allowed paragraphs and+sublists to start with only two spaces indentation, at least on the+outer level. Worse, its behavior was inconsistent: a sublist of an+outer-level list needed two spaces indentation, but a sublist of this+sublist needed three spaces. It is not surprising, then, that different+implementations of Markdown have developed very different rules for+determining what comes under a list item. (Pandoc and python-Markdown,+for example, stuck with Gruber's syntax description and the four-space+rule, while discount, redcarpet, marked, PHP Markdown, and others+followed `Markdown.pl`'s behavior more closely.)++Unfortunately, given the divergences between implementations, there+is no way to give a spec for list items that will be guaranteed not+to break any existing documents. However, the spec given here should+correctly handle lists formatted with either the four-space rule or+the more forgiving `Markdown.pl` behavior, provided they are laid out+in a way that is natural for a human to read.++The strategy here is to let the width and indentation of the list marker+determine the indentation necessary for blocks to fall under the list+item, rather than having a fixed and arbitrary number. The writer can+think of the body of the list item as a unit which gets indented to the+right enough to fit the list marker (and any indentation on the list+marker). (The laziness rule, #5, then allows continuation lines to be+unindented if needed.)++This rule is superior, we claim, to any rule requiring a fixed level of+indentation from the margin. The four-space rule is clear but+unnatural. It is quite unintuitive that++``` markdown+- foo++ bar++ - baz+```++should be parsed as two lists with an intervening paragraph,++``` html+<ul>+<li>foo</li>+</ul>+<p>bar</p>+<ul>+<li>baz</li>+</ul>+```++as the four-space rule demands, rather than a single list,++``` html+<ul>+<li>+<p>foo</p>+<p>bar</p>+<ul>+<li>baz</li>+</ul>+</li>+</ul>+```++The choice of four spaces is arbitrary. It can be learned, but it is+not likely to be guessed, and it trips up beginners regularly.++Would it help to adopt a two-space rule? The problem is that such+a rule, together with the rule allowing 1--3 spaces indentation of the+initial list marker, allows text that is indented *less than* the+original list marker to be included in the list item. For example,+`Markdown.pl` parses++``` markdown+ - one++ two+```++as a single list item, with `two` a continuation paragraph:++``` html+<ul>+<li>+<p>one</p>+<p>two</p>+</li>+</ul>+```++and similarly++``` markdown+> - one+>+> two+```++as++``` html+<blockquote>+<ul>+<li>+<p>one</p>+<p>two</p>+</li>+</ul>+</blockquote>+```++This is extremely unintuitive.++Rather than requiring a fixed indent from the margin, we could require+a fixed indent (say, two spaces, or even one space) from the list marker (which+may itself be indented). This proposal would remove the last anomaly+discussed. Unlike the spec presented above, it would count the following+as a list item with a subparagraph, even though the paragraph `bar`+is not indented as far as the first paragraph `foo`:++``` markdown+ 10. foo++ bar +```++Arguably this text does read like a list item with `bar` as a subparagraph,+which may count in favor of the proposal. However, on this proposal indented+code would have to be indented six spaces after the list marker. And this+would break a lot of existing Markdown, which has the pattern:++``` markdown+1. foo++ indented code+```++where the code is indented eight spaces. The spec above, by contrast, will+parse this text as expected, since the code block's indentation is measured+from the beginning of `foo`.++The one case that needs special treatment is a list item that *starts*+with indented code. How much indentation is required in that case, since+we don't have a "first paragraph" to measure from? Rule #2 simply stipulates+that in such cases, we require one space indentation from the list marker+(and then the normal four spaces for the indented code). This will match the+four-space rule in cases where the list marker plus its initial indentation+takes four spaces (a common case), but diverge in other cases.++## Lists++A [list](@) is a sequence of one or more+list items [of the same type]. The list items+may be separated by any number of blank lines.++Two list items are [of the same type](@)+if they begin with a [list marker] of the same type.+Two list markers are of the+same type if (a) they are bullet list markers using the same character+(`-`, `+`, or `*`) or (b) they are ordered list numbers with the same+delimiter (either `.` or `)`).++A list is an [ordered list](@)+if its constituent list items begin with+[ordered list markers], and a+[bullet list](@) if its constituent list+items begin with [bullet list markers].++The [start number](@)+of an [ordered list] is determined by the list number of+its initial list item. The numbers of subsequent list items are+disregarded.++A list is [loose](@) if any of its constituent+list items are separated by blank lines, or if any of its constituent+list items directly contain two block-level elements with a blank line+between them. Otherwise a list is [tight](@).+(The difference in HTML output is that paragraphs in a loose list are+wrapped in `<p>` tags, while paragraphs in a tight list are not.)++Changing the bullet or ordered list delimiter starts a new list:++```````````````````````````````` example+- foo+- bar++ baz+.+<ul>+<li>foo</li>+<li>bar</li>+</ul>+<ul>+<li>baz</li>+</ul>+````````````````````````````````+++```````````````````````````````` example+1. foo+2. bar+3) baz+.+<ol>+<li>foo</li>+<li>bar</li>+</ol>+<ol start="3">+<li>baz</li>+</ol>+````````````````````````````````+++In CommonMark, a list can interrupt a paragraph. That is,+no blank line is needed to separate a paragraph from a following+list:++```````````````````````````````` example+Foo+- bar+- baz+.+<p>Foo</p>+<ul>+<li>bar</li>+<li>baz</li>+</ul>+````````````````````````````````++`Markdown.pl` does not allow this, through fear of triggering a list+via a numeral in a hard-wrapped line:++``` markdown+The number of windows in my house is+14. The number of doors is 6.+```++Oddly, though, `Markdown.pl` *does* allow a blockquote to+interrupt a paragraph, even though the same considerations might+apply.++In CommonMark, we do allow lists to interrupt paragraphs, for+two reasons. First, it is natural and not uncommon for people+to start lists without blank lines:++``` markdown+I need to buy+- new shoes+- a coat+- a plane ticket+```++Second, we are attracted to a++> [principle of uniformity](@):+> if a chunk of text has a certain+> meaning, it will continue to have the same meaning when put into a+> container block (such as a list item or blockquote).++(Indeed, the spec for [list items] and [block quotes] presupposes+this principle.) This principle implies that if++``` markdown+ * I need to buy+ - new shoes+ - a coat+ - a plane ticket+```++is a list item containing a paragraph followed by a nested sublist,+as all Markdown implementations agree it is (though the paragraph+may be rendered without `<p>` tags, since the list is "tight"),+then++``` markdown+I need to buy+- new shoes+- a coat+- a plane ticket+```++by itself should be a paragraph followed by a nested sublist.++Since it is well established Markdown practice to allow lists to+interrupt paragraphs inside list items, the [principle of+uniformity] requires us to allow this outside list items as+well. ([reStructuredText](http://docutils.sourceforge.net/rst.html)+takes a different approach, requiring blank lines before lists+even inside other list items.)++In order to solve of unwanted lists in paragraphs with+hard-wrapped numerals, we allow only lists starting with `1` to+interrupt paragraphs. Thus,++```````````````````````````````` example+The number of windows in my house is+14. The number of doors is 6.+.+<p>The number of windows in my house is+14. The number of doors is 6.</p>+````````````````````````````````++We may still get an unintended result in cases like++```````````````````````````````` example+The number of windows in my house is+1. The number of doors is 6.+.+<p>The number of windows in my house is</p>+<ol>+<li>The number of doors is 6.</li>+</ol>+````````````````````````````````++but this rule should prevent most spurious list captures.++There can be any number of blank lines between items:++```````````````````````````````` example+- foo++- bar+++- baz+.+<ul>+<li>+<p>foo</p>+</li>+<li>+<p>bar</p>+</li>+<li>+<p>baz</p>+</li>+</ul>+````````````````````````````````++```````````````````````````````` example+- foo+ - bar+ - baz+++ bim+.+<ul>+<li>foo+<ul>+<li>bar+<ul>+<li>+<p>baz</p>+<p>bim</p>+</li>+</ul>+</li>+</ul>+</li>+</ul>+````````````````````````````````+++To separate consecutive lists of the same type, or to separate a+list from an indented code block that would otherwise be parsed+as a subparagraph of the final list item, you can insert a blank HTML+comment:++```````````````````````````````` example+- foo+- bar++<!-- -->++- baz+- bim+.+<ul>+<li>foo</li>+<li>bar</li>+</ul>+<!-- -->+<ul>+<li>baz</li>+<li>bim</li>+</ul>+````````````````````````````````+++```````````````````````````````` example+- foo++ notcode++- foo++<!-- -->++ code+.+<ul>+<li>+<p>foo</p>+<p>notcode</p>+</li>+<li>+<p>foo</p>+</li>+</ul>+<!-- -->+<pre><code>code+</code></pre>+````````````````````````````````+++List items need not be indented to the same level. The following+list items will be treated as items at the same list level,+since none is indented enough to belong to the previous list+item:++```````````````````````````````` example+- a+ - b+ - c+ - d+ - e+ - f+- g+.+<ul>+<li>a</li>+<li>b</li>+<li>c</li>+<li>d</li>+<li>e</li>+<li>f</li>+<li>g</li>+</ul>+````````````````````````````````+++```````````````````````````````` example+1. a++ 2. b++ 3. c+.+<ol>+<li>+<p>a</p>+</li>+<li>+<p>b</p>+</li>+<li>+<p>c</p>+</li>+</ol>+````````````````````````````````++Note, however, that list items may not be indented more than+three spaces. Here `- e` is treated as a paragraph continuation+line, because it is indented more than three spaces:++```````````````````````````````` example+- a+ - b+ - c+ - d+ - e+.+<ul>+<li>a</li>+<li>b</li>+<li>c</li>+<li>d+- e</li>+</ul>+````````````````````````````````++And here, `3. c` is treated as in indented code block,+because it is indented four spaces and preceded by a+blank line.++```````````````````````````````` example+1. a++ 2. b++ 3. c+.+<ol>+<li>+<p>a</p>+</li>+<li>+<p>b</p>+</li>+</ol>+<pre><code>3. c+</code></pre>+````````````````````````````````+++This is a loose list, because there is a blank line between+two of the list items:++```````````````````````````````` example+- a+- b++- c+.+<ul>+<li>+<p>a</p>+</li>+<li>+<p>b</p>+</li>+<li>+<p>c</p>+</li>+</ul>+````````````````````````````````+++So is this, with a empty second item:++```````````````````````````````` example+* a+*++* c+.+<ul>+<li>+<p>a</p>+</li>+<li></li>+<li>+<p>c</p>+</li>+</ul>+````````````````````````````````+++These are loose lists, even though there is no space between the items,+because one of the items directly contains two block-level elements+with a blank line between them:++```````````````````````````````` example+- a+- b++ c+- d+.+<ul>+<li>+<p>a</p>+</li>+<li>+<p>b</p>+<p>c</p>+</li>+<li>+<p>d</p>+</li>+</ul>+````````````````````````````````+++```````````````````````````````` example+- a+- b++ [ref]: /url+- d+.+<ul>+<li>+<p>a</p>+</li>+<li>+<p>b</p>+</li>+<li>+<p>d</p>+</li>+</ul>+````````````````````````````````+++This is a tight list, because the blank lines are in a code block:++```````````````````````````````` example+- a+- ```+ b+++ ```+- c+.+<ul>+<li>a</li>+<li>+<pre><code>b+++</code></pre>+</li>+<li>c</li>+</ul>+````````````````````````````````+++This is a tight list, because the blank line is between two+paragraphs of a sublist. So the sublist is loose while+the outer list is tight:++```````````````````````````````` example+- a+ - b++ c+- d+.+<ul>+<li>a+<ul>+<li>+<p>b</p>+<p>c</p>+</li>+</ul>+</li>+<li>d</li>+</ul>+````````````````````````````````+++This is a tight list, because the blank line is inside the+block quote:++```````````````````````````````` example+* a+ > b+ >+* c+.+<ul>+<li>a+<blockquote>+<p>b</p>+</blockquote>+</li>+<li>c</li>+</ul>+````````````````````````````````+++This list is tight, because the consecutive block elements+are not separated by blank lines:++```````````````````````````````` example+- a+ > b+ ```+ c+ ```+- d+.+<ul>+<li>a+<blockquote>+<p>b</p>+</blockquote>+<pre><code>c+</code></pre>+</li>+<li>d</li>+</ul>+````````````````````````````````+++A single-paragraph list is tight:++```````````````````````````````` example+- a+.+<ul>+<li>a</li>+</ul>+````````````````````````````````+++```````````````````````````````` example+- a+ - b+.+<ul>+<li>a+<ul>+<li>b</li>+</ul>+</li>+</ul>+````````````````````````````````+++This list is loose, because of the blank line between the+two block elements in the list item:++```````````````````````````````` example+1. ```+ foo+ ```++ bar+.+<ol>+<li>+<pre><code>foo+</code></pre>+<p>bar</p>+</li>+</ol>+````````````````````````````````+++Here the outer list is loose, the inner list tight:++```````````````````````````````` example+* foo+ * bar++ baz+.+<ul>+<li>+<p>foo</p>+<ul>+<li>bar</li>+</ul>+<p>baz</p>+</li>+</ul>+````````````````````````````````+++```````````````````````````````` example+- a+ - b+ - c++- d+ - e+ - f+.+<ul>+<li>+<p>a</p>+<ul>+<li>b</li>+<li>c</li>+</ul>+</li>+<li>+<p>d</p>+<ul>+<li>e</li>+<li>f</li>+</ul>+</li>+</ul>+````````````````````````````````+++# Inlines++Inlines are parsed sequentially from the beginning of the character+stream to the end (left to right, in left-to-right languages).+Thus, for example, in++```````````````````````````````` example+`hi`lo`+.+<p><code>hi</code>lo`</p>+````````````````````````````````++`hi` is parsed as code, leaving the backtick at the end as a literal+backtick.+++## Backslash escapes++Any ASCII punctuation character may be backslash-escaped:++```````````````````````````````` example+\!\"\#\$\%\&\'\(\)\*\+\,\-\.\/\:\;\<\=\>\?\@\[\\\]\^\_\`\{\|\}\~+.+<p>!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~</p>+````````````````````````````````+++Backslashes before other characters are treated as literal+backslashes:++```````````````````````````````` example+\→\A\a\ \3\φ\«+.+<p>\→\A\a\ \3\φ\«</p>+````````````````````````````````+++Escaped characters are treated as regular characters and do+not have their usual Markdown meanings:++```````````````````````````````` example+\*not emphasized*+\<br/> not a tag+\[not a link](/foo)+\`not code`+1\. not a list+\* not a list+\# not a heading+\[foo]: /url "not a reference"+\ö not a character entity+.+<p>*not emphasized*+<br/> not a tag+[not a link](/foo)+`not code`+1. not a list+* not a list+# not a heading+[foo]: /url "not a reference"+&ouml; not a character entity</p>+````````````````````````````````+++If a backslash is itself escaped, the following character is not:++```````````````````````````````` example+\\*emphasis*+.+<p>\<em>emphasis</em></p>+````````````````````````````````+++A backslash at the end of the line is a [hard line break]:++```````````````````````````````` example+foo\+bar+.+<p>foo<br />+bar</p>+````````````````````````````````+++Backslash escapes do not work in code blocks, code spans, autolinks, or+raw HTML:++```````````````````````````````` example+`` \[\` ``+.+<p><code>\[\`</code></p>+````````````````````````````````+++```````````````````````````````` example+ \[\]+.+<pre><code>\[\]+</code></pre>+````````````````````````````````+++```````````````````````````````` example+~~~+\[\]+~~~+.+<pre><code>\[\]+</code></pre>+````````````````````````````````+++```````````````````````````````` example+<http://example.com?find=\*>+.+<p><a href="http://example.com?find=%5C*">http://example.com?find=\*</a></p>+````````````````````````````````+++```````````````````````````````` example+<a href="/bar\/)">+.+<a href="/bar\/)">+````````````````````````````````+++But they work in all other contexts, including URLs and link titles,+link references, and [info strings] in [fenced code blocks]:++```````````````````````````````` example+[foo](/bar\* "ti\*tle")+.+<p><a href="/bar*" title="ti*tle">foo</a></p>+````````````````````````````````+++```````````````````````````````` example+[foo]++[foo]: /bar\* "ti\*tle"+.+<p><a href="/bar*" title="ti*tle">foo</a></p>+````````````````````````````````+++```````````````````````````````` example+``` foo\+bar+foo+```+.+<pre><code class="language-foo+bar">foo+</code></pre>+````````````````````````````````++++## Entity and numeric character references++Valid HTML entity references and numeric character references+can be used in place of the corresponding Unicode character,+with the following exceptions:++- Entity and character references are not recognized in code+ blocks and code spans.++- Entity and character references cannot stand in place of+ special characters that define structural elements in+ CommonMark. For example, although `*` can be used+ in place of a literal `*` character, `*` cannot replace+ `*` in emphasis delimiters, bullet list markers, or thematic+ breaks.++Conforming CommonMark parsers need not store information about+whether a particular character was represented in the source+using a Unicode character or an entity reference.++[Entity references](@) consist of `&` + any of the valid+HTML5 entity names + `;`. The+document <https://html.spec.whatwg.org/multipage/entities.json>+is used as an authoritative source for the valid entity+references and their corresponding code points.++```````````````````````````````` example+ & © Æ Ď+¾ ℋ ⅆ+∲ ≧̸+.+<p> & © Æ Ď+¾ ℋ ⅆ+∲ ≧̸</p>+````````````````````````````````+++[Decimal numeric character+references](@)+consist of `&#` + a string of 1--7 arabic digits + `;`. A+numeric character reference is parsed as the corresponding+Unicode character. Invalid Unicode code points will be replaced by+the REPLACEMENT CHARACTER (`U+FFFD`). For security reasons,+the code point `U+0000` will also be replaced by `U+FFFD`.++```````````````````````````````` example+# Ӓ Ϡ �+.+<p># Ӓ Ϡ �</p>+````````````````````````````````+++[Hexadecimal numeric character+references](@) consist of `&#` ++either `X` or `x` + a string of 1-6 hexadecimal digits + `;`.+They too are parsed as the corresponding Unicode character (this+time specified with a hexadecimal numeral instead of decimal).++```````````````````````````````` example+" ആ ಫ+.+<p>" ആ ಫ</p>+````````````````````````````````+++Here are some nonentities:++```````````````````````````````` example+  &x; &#; &#x;+�+&#abcdef0;+&ThisIsNotDefined; &hi?;+.+<p>&nbsp &x; &#; &#x;+&#987654321;+&#abcdef0;+&ThisIsNotDefined; &hi?;</p>+````````````````````````````````+++Although HTML5 does accept some entity references+without a trailing semicolon (such as `©`), these are not+recognized here, because it makes the grammar too ambiguous:++```````````````````````````````` example+©+.+<p>&copy</p>+````````````````````````````````+++Strings that are not on the list of HTML5 named entities are not+recognized as entity references either:++```````````````````````````````` example+&MadeUpEntity;+.+<p>&MadeUpEntity;</p>+````````````````````````````````+++Entity and numeric character references are recognized in any+context besides code spans or code blocks, including+URLs, [link titles], and [fenced code block][] [info strings]:++```````````````````````````````` example+<a href="öö.html">+.+<a href="öö.html">+````````````````````````````````+++```````````````````````````````` example+[foo](/föö "föö")+.+<p><a href="/f%C3%B6%C3%B6" title="föö">foo</a></p>+````````````````````````````````+++```````````````````````````````` example+[foo]++[foo]: /föö "föö"+.+<p><a href="/f%C3%B6%C3%B6" title="föö">foo</a></p>+````````````````````````````````+++```````````````````````````````` example+``` föö+foo+```+.+<pre><code class="language-föö">foo+</code></pre>+````````````````````````````````+++Entity and numeric character references are treated as literal+text in code spans and code blocks:++```````````````````````````````` example+`föö`+.+<p><code>f&ouml;&ouml;</code></p>+````````````````````````````````+++```````````````````````````````` example+ föfö+.+<pre><code>f&ouml;f&ouml;+</code></pre>+````````````````````````````````+++Entity and numeric character references cannot be used+in place of symbols indicating structure in CommonMark+documents.++```````````````````````````````` example+*foo*+*foo*+.+<p>*foo*+<em>foo</em></p>+````````````````````````````````++```````````````````````````````` example+* foo++* foo+.+<p>* foo</p>+<ul>+<li>foo</li>+</ul>+````````````````````````````````++```````````````````````````````` example+foo bar+.+<p>foo++bar</p>+````````````````````````````````++```````````````````````````````` example+	foo+.+<p>→foo</p>+````````````````````````````````+++```````````````````````````````` example+[a](url "tit")+.+<p>[a](url "tit")</p>+````````````````````````````````+++## Code spans++A [backtick string](@)+is a string of one or more backtick characters (`` ` ``) that is neither+preceded nor followed by a backtick.++A [code span](@) begins with a backtick string and ends with+a backtick string of equal length. The contents of the code span are+the characters between the two backtick strings, normalized in the+following ways:++- First, [line endings] are converted to [spaces].+- If the resulting string both begins *and* ends with a [space]+ character, but does not consist entirely of [space]+ characters, a single [space] character is removed from the+ front and back. This allows you to include code that begins+ or ends with backtick characters, which must be separated by+ whitespace from the opening or closing backtick strings.++This is a simple code span:++```````````````````````````````` example+`foo`+.+<p><code>foo</code></p>+````````````````````````````````+++Here two backticks are used, because the code contains a backtick.+This example also illustrates stripping of a single leading and+trailing space:++```````````````````````````````` example+`` foo ` bar ``+.+<p><code>foo ` bar</code></p>+````````````````````````````````+++This example shows the motivation for stripping leading and trailing+spaces:++```````````````````````````````` example+` `` `+.+<p><code>``</code></p>+````````````````````````````````++Note that only *one* space is stripped:++```````````````````````````````` example+` `` `+.+<p><code> `` </code></p>+````````````````````````````````++The stripping only happens if the space is on both+sides of the string:++```````````````````````````````` example+` a`+.+<p><code> a</code></p>+````````````````````````````````++Only [spaces], and not [unicode whitespace] in general, are+stripped in this way:++```````````````````````````````` example+` b `+.+<p><code> b </code></p>+````````````````````````````````++No stripping occurs if the code span contains only spaces:++```````````````````````````````` example+` `+` `+.+<p><code> </code>+<code> </code></p>+````````````````````````````````+++[Line endings] are treated like spaces:++```````````````````````````````` example+``+foo+bar +baz+``+.+<p><code>foo bar baz</code></p>+````````````````````````````````++```````````````````````````````` example+``+foo +``+.+<p><code>foo </code></p>+````````````````````````````````+++Interior spaces are not collapsed:++```````````````````````````````` example+`foo bar +baz`+.+<p><code>foo bar baz</code></p>+````````````````````````````````++Note that browsers will typically collapse consecutive spaces+when rendering `<code>` elements, so it is recommended that+the following CSS be used:++ code{white-space: pre-wrap;}+++Note that backslash escapes do not work in code spans. All backslashes+are treated literally:++```````````````````````````````` example+`foo\`bar`+.+<p><code>foo\</code>bar`</p>+````````````````````````````````+++Backslash escapes are never needed, because one can always choose a+string of *n* backtick characters as delimiters, where the code does+not contain any strings of exactly *n* backtick characters.++```````````````````````````````` example+``foo`bar``+.+<p><code>foo`bar</code></p>+````````````````````````````````++```````````````````````````````` example+` foo `` bar `+.+<p><code>foo `` bar</code></p>+````````````````````````````````+++Code span backticks have higher precedence than any other inline+constructs except HTML tags and autolinks. Thus, for example, this is+not parsed as emphasized text, since the second `*` is part of a code+span:++```````````````````````````````` example+*foo`*`+.+<p>*foo<code>*</code></p>+````````````````````````````````+++And this is not parsed as a link:++```````````````````````````````` example+[not a `link](/foo`)+.+<p>[not a <code>link](/foo</code>)</p>+````````````````````````````````+++Code spans, HTML tags, and autolinks have the same precedence.+Thus, this is code:++```````````````````````````````` example+`<a href="`">`+.+<p><code><a href="</code>">`</p>+````````````````````````````````+++But this is an HTML tag:++```````````````````````````````` example+<a href="`">`+.+<p><a href="`">`</p>+````````````````````````````````+++And this is code:++```````````````````````````````` example+`<http://foo.bar.`baz>`+.+<p><code><http://foo.bar.</code>baz>`</p>+````````````````````````````````+++But this is an autolink:++```````````````````````````````` example+<http://foo.bar.`baz>`+.+<p><a href="http://foo.bar.%60baz">http://foo.bar.`baz</a>`</p>+````````````````````````````````+++When a backtick string is not closed by a matching backtick string,+we just have literal backticks:++```````````````````````````````` example+```foo``+.+<p>```foo``</p>+````````````````````````````````+++```````````````````````````````` example+`foo+.+<p>`foo</p>+````````````````````````````````++The following case also illustrates the need for opening and+closing backtick strings to be equal in length:++```````````````````````````````` example+`foo``bar``+.+<p>`foo<code>bar</code></p>+````````````````````````````````+++## Emphasis and strong emphasis++John Gruber's original [Markdown syntax+description](http://daringfireball.net/projects/markdown/syntax#em) says:++> Markdown treats asterisks (`*`) and underscores (`_`) as indicators of+> emphasis. Text wrapped with one `*` or `_` will be wrapped with an HTML+> `<em>` tag; double `*`'s or `_`'s will be wrapped with an HTML `<strong>`+> tag.++This is enough for most users, but these rules leave much undecided,+especially when it comes to nested emphasis. The original+`Markdown.pl` test suite makes it clear that triple `***` and+`___` delimiters can be used for strong emphasis, and most+implementations have also allowed the following patterns:++``` markdown+***strong emph***+***strong** in emph*+***emph* in strong**+**in strong *emph***+*in emph **strong***+```++The following patterns are less widely supported, but the intent+is clear and they are useful (especially in contexts like bibliography+entries):++``` markdown+*emph *with emph* in it*+**strong **with strong** in it**+```++Many implementations have also restricted intraword emphasis to+the `*` forms, to avoid unwanted emphasis in words containing+internal underscores. (It is best practice to put these in code+spans, but users often do not.)++``` markdown+internal emphasis: foo*bar*baz+no emphasis: foo_bar_baz+```++The rules given below capture all of these patterns, while allowing+for efficient parsing strategies that do not backtrack.++First, some definitions. A [delimiter run](@) is either+a sequence of one or more `*` characters that is not preceded or+followed by a non-backslash-escaped `*` character, or a sequence+of one or more `_` characters that is not preceded or followed by+a non-backslash-escaped `_` character.++A [left-flanking delimiter run](@) is+a [delimiter run] that is (1) not followed by [Unicode whitespace],+and either (2a) not followed by a [punctuation character], or+(2b) followed by a [punctuation character] and+preceded by [Unicode whitespace] or a [punctuation character].+For purposes of this definition, the beginning and the end of+the line count as Unicode whitespace.++A [right-flanking delimiter run](@) is+a [delimiter run] that is (1) not preceded by [Unicode whitespace],+and either (2a) not preceded by a [punctuation character], or+(2b) preceded by a [punctuation character] and+followed by [Unicode whitespace] or a [punctuation character].+For purposes of this definition, the beginning and the end of+the line count as Unicode whitespace.++Here are some examples of delimiter runs.++ - left-flanking but not right-flanking:++ ```+ ***abc+ _abc+ **"abc"+ _"abc"+ ```++ - right-flanking but not left-flanking:++ ```+ abc***+ abc_+ "abc"**+ "abc"_+ ```++ - Both left and right-flanking:++ ```+ abc***def+ "abc"_"def"+ ```++ - Neither left nor right-flanking:++ ```+ abc *** def+ a _ b+ ```++(The idea of distinguishing left-flanking and right-flanking+delimiter runs based on the character before and the character+after comes from Roopesh Chander's+[vfmd](http://www.vfmd.org/vfmd-spec/specification/#procedure-for-identifying-emphasis-tags).+vfmd uses the terminology "emphasis indicator string" instead of "delimiter+run," and its rules for distinguishing left- and right-flanking runs+are a bit more complex than the ones given here.)++The following rules define emphasis and strong emphasis:++1. A single `*` character [can open emphasis](@)+ iff (if and only if) it is part of a [left-flanking delimiter run].++2. A single `_` character [can open emphasis] iff+ it is part of a [left-flanking delimiter run]+ and either (a) not part of a [right-flanking delimiter run]+ or (b) part of a [right-flanking delimiter run]+ preceded by punctuation.++3. A single `*` character [can close emphasis](@)+ iff it is part of a [right-flanking delimiter run].++4. A single `_` character [can close emphasis] iff+ it is part of a [right-flanking delimiter run]+ and either (a) not part of a [left-flanking delimiter run]+ or (b) part of a [left-flanking delimiter run]+ followed by punctuation.++5. A double `**` [can open strong emphasis](@)+ iff it is part of a [left-flanking delimiter run].++6. A double `__` [can open strong emphasis] iff+ it is part of a [left-flanking delimiter run]+ and either (a) not part of a [right-flanking delimiter run]+ or (b) part of a [right-flanking delimiter run]+ preceded by punctuation.++7. A double `**` [can close strong emphasis](@)+ iff it is part of a [right-flanking delimiter run].++8. A double `__` [can close strong emphasis] iff+ it is part of a [right-flanking delimiter run]+ and either (a) not part of a [left-flanking delimiter run]+ or (b) part of a [left-flanking delimiter run]+ followed by punctuation.++9. Emphasis begins with a delimiter that [can open emphasis] and ends+ with a delimiter that [can close emphasis], and that uses the same+ character (`_` or `*`) as the opening delimiter. The+ opening and closing delimiters must belong to separate+ [delimiter runs]. If one of the delimiters can both+ open and close emphasis, then the sum of the lengths of the+ delimiter runs containing the opening and closing delimiters+ must not be a multiple of 3 unless both lengths are+ multiples of 3.++10. Strong emphasis begins with a delimiter that+ [can open strong emphasis] and ends with a delimiter that+ [can close strong emphasis], and that uses the same character+ (`_` or `*`) as the opening delimiter. The+ opening and closing delimiters must belong to separate+ [delimiter runs]. If one of the delimiters can both open+ and close strong emphasis, then the sum of the lengths of+ the delimiter runs containing the opening and closing+ delimiters must not be a multiple of 3 unless both lengths+ are multiples of 3.++11. A literal `*` character cannot occur at the beginning or end of+ `*`-delimited emphasis or `**`-delimited strong emphasis, unless it+ is backslash-escaped.++12. A literal `_` character cannot occur at the beginning or end of+ `_`-delimited emphasis or `__`-delimited strong emphasis, unless it+ is backslash-escaped.++Where rules 1--12 above are compatible with multiple parsings,+the following principles resolve ambiguity:++13. The number of nestings should be minimized. Thus, for example,+ an interpretation `<strong>...</strong>` is always preferred to+ `<em><em>...</em></em>`.++14. An interpretation `<em><strong>...</strong></em>` is always+ preferred to `<strong><em>...</em></strong>`.++15. When two potential emphasis or strong emphasis spans overlap,+ so that the second begins before the first ends and ends after+ the first ends, the first takes precedence. Thus, for example,+ `*foo _bar* baz_` is parsed as `<em>foo _bar</em> baz_` rather+ than `*foo <em>bar* baz</em>`.++16. When there are two potential emphasis or strong emphasis spans+ with the same closing delimiter, the shorter one (the one that+ opens later) takes precedence. Thus, for example,+ `**foo **bar baz**` is parsed as `**foo <strong>bar baz</strong>`+ rather than `<strong>foo **bar baz</strong>`.++17. Inline code spans, links, images, and HTML tags group more tightly+ than emphasis. So, when there is a choice between an interpretation+ that contains one of these elements and one that does not, the+ former always wins. Thus, for example, `*[foo*](bar)` is+ parsed as `*<a href="bar">foo*</a>` rather than as+ `<em>[foo</em>](bar)`.++These rules can be illustrated through a series of examples.++Rule 1:++```````````````````````````````` example+*foo bar*+.+<p><em>foo bar</em></p>+````````````````````````````````+++This is not emphasis, because the opening `*` is followed by+whitespace, and hence not part of a [left-flanking delimiter run]:++```````````````````````````````` example+a * foo bar*+.+<p>a * foo bar*</p>+````````````````````````````````+++This is not emphasis, because the opening `*` is preceded+by an alphanumeric and followed by punctuation, and hence+not part of a [left-flanking delimiter run]:++```````````````````````````````` example+a*"foo"*+.+<p>a*"foo"*</p>+````````````````````````````````+++Unicode nonbreaking spaces count as whitespace, too:++```````````````````````````````` example+* a *+.+<p>* a *</p>+````````````````````````````````+++Intraword emphasis with `*` is permitted:++```````````````````````````````` example+foo*bar*+.+<p>foo<em>bar</em></p>+````````````````````````````````+++```````````````````````````````` example+5*6*78+.+<p>5<em>6</em>78</p>+````````````````````````````````+++Rule 2:++```````````````````````````````` example+_foo bar_+.+<p><em>foo bar</em></p>+````````````````````````````````+++This is not emphasis, because the opening `_` is followed by+whitespace:++```````````````````````````````` example+_ foo bar_+.+<p>_ foo bar_</p>+````````````````````````````````+++This is not emphasis, because the opening `_` is preceded+by an alphanumeric and followed by punctuation:++```````````````````````````````` example+a_"foo"_+.+<p>a_"foo"_</p>+````````````````````````````````+++Emphasis with `_` is not allowed inside words:++```````````````````````````````` example+foo_bar_+.+<p>foo_bar_</p>+````````````````````````````````+++```````````````````````````````` example+5_6_78+.+<p>5_6_78</p>+````````````````````````````````+++```````````````````````````````` example+пристаням_стремятся_+.+<p>пристаням_стремятся_</p>+````````````````````````````````+++Here `_` does not generate emphasis, because the first delimiter run+is right-flanking and the second left-flanking:++```````````````````````````````` example+aa_"bb"_cc+.+<p>aa_"bb"_cc</p>+````````````````````````````````+++This is emphasis, even though the opening delimiter is+both left- and right-flanking, because it is preceded by+punctuation:++```````````````````````````````` example+foo-_(bar)_+.+<p>foo-<em>(bar)</em></p>+````````````````````````````````+++Rule 3:++This is not emphasis, because the closing delimiter does+not match the opening delimiter:++```````````````````````````````` example+_foo*+.+<p>_foo*</p>+````````````````````````````````+++This is not emphasis, because the closing `*` is preceded by+whitespace:++```````````````````````````````` example+*foo bar *+.+<p>*foo bar *</p>+````````````````````````````````+++A newline also counts as whitespace:++```````````````````````````````` example+*foo bar+*+.+<p>*foo bar+*</p>+````````````````````````````````+++This is not emphasis, because the second `*` is+preceded by punctuation and followed by an alphanumeric+(hence it is not part of a [right-flanking delimiter run]:++```````````````````````````````` example+*(*foo)+.+<p>*(*foo)</p>+````````````````````````````````+++The point of this restriction is more easily appreciated+with this example:++```````````````````````````````` example+*(*foo*)*+.+<p><em>(<em>foo</em>)</em></p>+````````````````````````````````+++Intraword emphasis with `*` is allowed:++```````````````````````````````` example+*foo*bar+.+<p><em>foo</em>bar</p>+````````````````````````````````++++Rule 4:++This is not emphasis, because the closing `_` is preceded by+whitespace:++```````````````````````````````` example+_foo bar _+.+<p>_foo bar _</p>+````````````````````````````````+++This is not emphasis, because the second `_` is+preceded by punctuation and followed by an alphanumeric:++```````````````````````````````` example+_(_foo)+.+<p>_(_foo)</p>+````````````````````````````````+++This is emphasis within emphasis:++```````````````````````````````` example+_(_foo_)_+.+<p><em>(<em>foo</em>)</em></p>+````````````````````````````````+++Intraword emphasis is disallowed for `_`:++```````````````````````````````` example+_foo_bar+.+<p>_foo_bar</p>+````````````````````````````````+++```````````````````````````````` example+_пристаням_стремятся+.+<p>_пристаням_стремятся</p>+````````````````````````````````+++```````````````````````````````` example+_foo_bar_baz_+.+<p><em>foo_bar_baz</em></p>+````````````````````````````````+++This is emphasis, even though the closing delimiter is+both left- and right-flanking, because it is followed by+punctuation:++```````````````````````````````` example+_(bar)_.+.+<p><em>(bar)</em>.</p>+````````````````````````````````+++Rule 5:++```````````````````````````````` example+**foo bar**+.+<p><strong>foo bar</strong></p>+````````````````````````````````+++This is not strong emphasis, because the opening delimiter is+followed by whitespace:++```````````````````````````````` example+** foo bar**+.+<p>** foo bar**</p>+````````````````````````````````+++This is not strong emphasis, because the opening `**` is preceded+by an alphanumeric and followed by punctuation, and hence+not part of a [left-flanking delimiter run]:++```````````````````````````````` example+a**"foo"**+.+<p>a**"foo"**</p>+````````````````````````````````+++Intraword strong emphasis with `**` is permitted:++```````````````````````````````` example+foo**bar**+.+<p>foo<strong>bar</strong></p>+````````````````````````````````+++Rule 6:++```````````````````````````````` example+__foo bar__+.+<p><strong>foo bar</strong></p>+````````````````````````````````+++This is not strong emphasis, because the opening delimiter is+followed by whitespace:++```````````````````````````````` example+__ foo bar__+.+<p>__ foo bar__</p>+````````````````````````````````+++A newline counts as whitespace:+```````````````````````````````` example+__+foo bar__+.+<p>__+foo bar__</p>+````````````````````````````````+++This is not strong emphasis, because the opening `__` is preceded+by an alphanumeric and followed by punctuation:++```````````````````````````````` example+a__"foo"__+.+<p>a__"foo"__</p>+````````````````````````````````+++Intraword strong emphasis is forbidden with `__`:++```````````````````````````````` example+foo__bar__+.+<p>foo__bar__</p>+````````````````````````````````+++```````````````````````````````` example+5__6__78+.+<p>5__6__78</p>+````````````````````````````````+++```````````````````````````````` example+пристаням__стремятся__+.+<p>пристаням__стремятся__</p>+````````````````````````````````+++```````````````````````````````` example+__foo, __bar__, baz__+.+<p><strong>foo, <strong>bar</strong>, baz</strong></p>+````````````````````````````````+++This is strong emphasis, even though the opening delimiter is+both left- and right-flanking, because it is preceded by+punctuation:++```````````````````````````````` example+foo-__(bar)__+.+<p>foo-<strong>(bar)</strong></p>+````````````````````````````````++++Rule 7:++This is not strong emphasis, because the closing delimiter is preceded+by whitespace:++```````````````````````````````` example+**foo bar **+.+<p>**foo bar **</p>+````````````````````````````````+++(Nor can it be interpreted as an emphasized `*foo bar *`, because of+Rule 11.)++This is not strong emphasis, because the second `**` is+preceded by punctuation and followed by an alphanumeric:++```````````````````````````````` example+**(**foo)+.+<p>**(**foo)</p>+````````````````````````````````+++The point of this restriction is more easily appreciated+with these examples:++```````````````````````````````` example+*(**foo**)*+.+<p><em>(<strong>foo</strong>)</em></p>+````````````````````````````````+++```````````````````````````````` example+**Gomphocarpus (*Gomphocarpus physocarpus*, syn.+*Asclepias physocarpa*)**+.+<p><strong>Gomphocarpus (<em>Gomphocarpus physocarpus</em>, syn.+<em>Asclepias physocarpa</em>)</strong></p>+````````````````````````````````+++```````````````````````````````` example+**foo "*bar*" foo**+.+<p><strong>foo "<em>bar</em>" foo</strong></p>+````````````````````````````````+++Intraword emphasis:++```````````````````````````````` example+**foo**bar+.+<p><strong>foo</strong>bar</p>+````````````````````````````````+++Rule 8:++This is not strong emphasis, because the closing delimiter is+preceded by whitespace:++```````````````````````````````` example+__foo bar __+.+<p>__foo bar __</p>+````````````````````````````````+++This is not strong emphasis, because the second `__` is+preceded by punctuation and followed by an alphanumeric:++```````````````````````````````` example+__(__foo)+.+<p>__(__foo)</p>+````````````````````````````````+++The point of this restriction is more easily appreciated+with this example:++```````````````````````````````` example+_(__foo__)_+.+<p><em>(<strong>foo</strong>)</em></p>+````````````````````````````````+++Intraword strong emphasis is forbidden with `__`:++```````````````````````````````` example+__foo__bar+.+<p>__foo__bar</p>+````````````````````````````````+++```````````````````````````````` example+__пристаням__стремятся+.+<p>__пристаням__стремятся</p>+````````````````````````````````+++```````````````````````````````` example+__foo__bar__baz__+.+<p><strong>foo__bar__baz</strong></p>+````````````````````````````````+++This is strong emphasis, even though the closing delimiter is+both left- and right-flanking, because it is followed by+punctuation:++```````````````````````````````` example+__(bar)__.+.+<p><strong>(bar)</strong>.</p>+````````````````````````````````+++Rule 9:++Any nonempty sequence of inline elements can be the contents of an+emphasized span.++```````````````````````````````` example+*foo [bar](/url)*+.+<p><em>foo <a href="/url">bar</a></em></p>+````````````````````````````````+++```````````````````````````````` example+*foo+bar*+.+<p><em>foo+bar</em></p>+````````````````````````````````+++In particular, emphasis and strong emphasis can be nested+inside emphasis:++```````````````````````````````` example+_foo __bar__ baz_+.+<p><em>foo <strong>bar</strong> baz</em></p>+````````````````````````````````+++```````````````````````````````` example+_foo _bar_ baz_+.+<p><em>foo <em>bar</em> baz</em></p>+````````````````````````````````+++```````````````````````````````` example+__foo_ bar_+.+<p><em><em>foo</em> bar</em></p>+````````````````````````````````+++```````````````````````````````` example+*foo *bar**+.+<p><em>foo <em>bar</em></em></p>+````````````````````````````````+++```````````````````````````````` example+*foo **bar** baz*+.+<p><em>foo <strong>bar</strong> baz</em></p>+````````````````````````````````++```````````````````````````````` example+*foo**bar**baz*+.+<p><em>foo<strong>bar</strong>baz</em></p>+````````````````````````````````++Note that in the preceding case, the interpretation++``` markdown+<p><em>foo</em><em>bar<em></em>baz</em></p>+```+++is precluded by the condition that a delimiter that+can both open and close (like the `*` after `foo`)+cannot form emphasis if the sum of the lengths of+the delimiter runs containing the opening and+closing delimiters is a multiple of 3 unless+both lengths are multiples of 3.+++For the same reason, we don't get two consecutive+emphasis sections in this example:++```````````````````````````````` example+*foo**bar*+.+<p><em>foo**bar</em></p>+````````````````````````````````+++The same condition ensures that the following+cases are all strong emphasis nested inside+emphasis, even when the interior spaces are+omitted:+++```````````````````````````````` example+***foo** bar*+.+<p><em><strong>foo</strong> bar</em></p>+````````````````````````````````+++```````````````````````````````` example+*foo **bar***+.+<p><em>foo <strong>bar</strong></em></p>+````````````````````````````````+++```````````````````````````````` example+*foo**bar***+.+<p><em>foo<strong>bar</strong></em></p>+````````````````````````````````+++When the lengths of the interior closing and opening+delimiter runs are *both* multiples of 3, though,+they can match to create emphasis:++```````````````````````````````` example+foo***bar***baz+.+<p>foo<em><strong>bar</strong></em>baz</p>+````````````````````````````````++```````````````````````````````` example+foo******bar*********baz+.+<p>foo<strong><strong><strong>bar</strong></strong></strong>***baz</p>+````````````````````````````````+++Indefinite levels of nesting are possible:++```````````````````````````````` example+*foo **bar *baz* bim** bop*+.+<p><em>foo <strong>bar <em>baz</em> bim</strong> bop</em></p>+````````````````````````````````+++```````````````````````````````` example+*foo [*bar*](/url)*+.+<p><em>foo <a href="/url"><em>bar</em></a></em></p>+````````````````````````````````+++There can be no empty emphasis or strong emphasis:++```````````````````````````````` example+** is not an empty emphasis+.+<p>** is not an empty emphasis</p>+````````````````````````````````+++```````````````````````````````` example+**** is not an empty strong emphasis+.+<p>**** is not an empty strong emphasis</p>+````````````````````````````````++++Rule 10:++Any nonempty sequence of inline elements can be the contents of an+strongly emphasized span.++```````````````````````````````` example+**foo [bar](/url)**+.+<p><strong>foo <a href="/url">bar</a></strong></p>+````````````````````````````````+++```````````````````````````````` example+**foo+bar**+.+<p><strong>foo+bar</strong></p>+````````````````````````````````+++In particular, emphasis and strong emphasis can be nested+inside strong emphasis:++```````````````````````````````` example+__foo _bar_ baz__+.+<p><strong>foo <em>bar</em> baz</strong></p>+````````````````````````````````+++```````````````````````````````` example+__foo __bar__ baz__+.+<p><strong>foo <strong>bar</strong> baz</strong></p>+````````````````````````````````+++```````````````````````````````` example+____foo__ bar__+.+<p><strong><strong>foo</strong> bar</strong></p>+````````````````````````````````+++```````````````````````````````` example+**foo **bar****+.+<p><strong>foo <strong>bar</strong></strong></p>+````````````````````````````````+++```````````````````````````````` example+**foo *bar* baz**+.+<p><strong>foo <em>bar</em> baz</strong></p>+````````````````````````````````+++```````````````````````````````` example+**foo*bar*baz**+.+<p><strong>foo<em>bar</em>baz</strong></p>+````````````````````````````````+++```````````````````````````````` example+***foo* bar**+.+<p><strong><em>foo</em> bar</strong></p>+````````````````````````````````+++```````````````````````````````` example+**foo *bar***+.+<p><strong>foo <em>bar</em></strong></p>+````````````````````````````````+++Indefinite levels of nesting are possible:++```````````````````````````````` example+**foo *bar **baz**+bim* bop**+.+<p><strong>foo <em>bar <strong>baz</strong>+bim</em> bop</strong></p>+````````````````````````````````+++```````````````````````````````` example+**foo [*bar*](/url)**+.+<p><strong>foo <a href="/url"><em>bar</em></a></strong></p>+````````````````````````````````+++There can be no empty emphasis or strong emphasis:++```````````````````````````````` example+__ is not an empty emphasis+.+<p>__ is not an empty emphasis</p>+````````````````````````````````+++```````````````````````````````` example+____ is not an empty strong emphasis+.+<p>____ is not an empty strong emphasis</p>+````````````````````````````````++++Rule 11:++```````````````````````````````` example+foo ***+.+<p>foo ***</p>+````````````````````````````````+++```````````````````````````````` example+foo *\**+.+<p>foo <em>*</em></p>+````````````````````````````````+++```````````````````````````````` example+foo *_*+.+<p>foo <em>_</em></p>+````````````````````````````````+++```````````````````````````````` example+foo *****+.+<p>foo *****</p>+````````````````````````````````+++```````````````````````````````` example+foo **\***+.+<p>foo <strong>*</strong></p>+````````````````````````````````+++```````````````````````````````` example+foo **_**+.+<p>foo <strong>_</strong></p>+````````````````````````````````+++Note that when delimiters do not match evenly, Rule 11 determines+that the excess literal `*` characters will appear outside of the+emphasis, rather than inside it:++```````````````````````````````` example+**foo*+.+<p>*<em>foo</em></p>+````````````````````````````````+++```````````````````````````````` example+*foo**+.+<p><em>foo</em>*</p>+````````````````````````````````+++```````````````````````````````` example+***foo**+.+<p>*<strong>foo</strong></p>+````````````````````````````````+++```````````````````````````````` example+****foo*+.+<p>***<em>foo</em></p>+````````````````````````````````+++```````````````````````````````` example+**foo***+.+<p><strong>foo</strong>*</p>+````````````````````````````````+++```````````````````````````````` example+*foo****+.+<p><em>foo</em>***</p>+````````````````````````````````++++Rule 12:++```````````````````````````````` example+foo ___+.+<p>foo ___</p>+````````````````````````````````+++```````````````````````````````` example+foo _\__+.+<p>foo <em>_</em></p>+````````````````````````````````+++```````````````````````````````` example+foo _*_+.+<p>foo <em>*</em></p>+````````````````````````````````+++```````````````````````````````` example+foo _____+.+<p>foo _____</p>+````````````````````````````````+++```````````````````````````````` example+foo __\___+.+<p>foo <strong>_</strong></p>+````````````````````````````````+++```````````````````````````````` example+foo __*__+.+<p>foo <strong>*</strong></p>+````````````````````````````````+++```````````````````````````````` example+__foo_+.+<p>_<em>foo</em></p>+````````````````````````````````+++Note that when delimiters do not match evenly, Rule 12 determines+that the excess literal `_` characters will appear outside of the+emphasis, rather than inside it:++```````````````````````````````` example+_foo__+.+<p><em>foo</em>_</p>+````````````````````````````````+++```````````````````````````````` example+___foo__+.+<p>_<strong>foo</strong></p>+````````````````````````````````+++```````````````````````````````` example+____foo_+.+<p>___<em>foo</em></p>+````````````````````````````````+++```````````````````````````````` example+__foo___+.+<p><strong>foo</strong>_</p>+````````````````````````````````+++```````````````````````````````` example+_foo____+.+<p><em>foo</em>___</p>+````````````````````````````````+++Rule 13 implies that if you want emphasis nested directly inside+emphasis, you must use different delimiters:++```````````````````````````````` example+**foo**+.+<p><strong>foo</strong></p>+````````````````````````````````+++```````````````````````````````` example+*_foo_*+.+<p><em><em>foo</em></em></p>+````````````````````````````````+++```````````````````````````````` example+__foo__+.+<p><strong>foo</strong></p>+````````````````````````````````+++```````````````````````````````` example+_*foo*_+.+<p><em><em>foo</em></em></p>+````````````````````````````````+++However, strong emphasis within strong emphasis is possible without+switching delimiters:++```````````````````````````````` example+****foo****+.+<p><strong><strong>foo</strong></strong></p>+````````````````````````````````+++```````````````````````````````` example+____foo____+.+<p><strong><strong>foo</strong></strong></p>+````````````````````````````````++++Rule 13 can be applied to arbitrarily long sequences of+delimiters:++```````````````````````````````` example+******foo******+.+<p><strong><strong><strong>foo</strong></strong></strong></p>+````````````````````````````````+++Rule 14:++```````````````````````````````` example+***foo***+.+<p><em><strong>foo</strong></em></p>+````````````````````````````````+++```````````````````````````````` example+_____foo_____+.+<p><em><strong><strong>foo</strong></strong></em></p>+````````````````````````````````+++Rule 15:++```````````````````````````````` example+*foo _bar* baz_+.+<p><em>foo _bar</em> baz_</p>+````````````````````````````````+++```````````````````````````````` example+*foo __bar *baz bim__ bam*+.+<p><em>foo <strong>bar *baz bim</strong> bam</em></p>+````````````````````````````````+++Rule 16:++```````````````````````````````` example+**foo **bar baz**+.+<p>**foo <strong>bar baz</strong></p>+````````````````````````````````+++```````````````````````````````` example+*foo *bar baz*+.+<p>*foo <em>bar baz</em></p>+````````````````````````````````+++Rule 17:++```````````````````````````````` example+*[bar*](/url)+.+<p>*<a href="/url">bar*</a></p>+````````````````````````````````+++```````````````````````````````` example+_foo [bar_](/url)+.+<p>_foo <a href="/url">bar_</a></p>+````````````````````````````````+++```````````````````````````````` example+*<img src="foo" title="*"/>+.+<p>*<img src="foo" title="*"/></p>+````````````````````````````````+++```````````````````````````````` example+**<a href="**">+.+<p>**<a href="**"></p>+````````````````````````````````+++```````````````````````````````` example+__<a href="__">+.+<p>__<a href="__"></p>+````````````````````````````````+++```````````````````````````````` example+*a `*`*+.+<p><em>a <code>*</code></em></p>+````````````````````````````````+++```````````````````````````````` example+_a `_`_+.+<p><em>a <code>_</code></em></p>+````````````````````````````````+++```````````````````````````````` example+**a<http://foo.bar/?q=**>+.+<p>**a<a href="http://foo.bar/?q=**">http://foo.bar/?q=**</a></p>+````````````````````````````````+++```````````````````````````````` example+__a<http://foo.bar/?q=__>+.+<p>__a<a href="http://foo.bar/?q=__">http://foo.bar/?q=__</a></p>+````````````````````````````````++++## Links++A link contains [link text] (the visible text), a [link destination]+(the URI that is the link destination), and optionally a [link title].+There are two basic kinds of links in Markdown. In [inline links] the+destination and title are given immediately after the link text. In+[reference links] the destination and title are defined elsewhere in+the document.++A [link text](@) consists of a sequence of zero or more+inline elements enclosed by square brackets (`[` and `]`). The+following rules apply:++- Links may not contain other links, at any level of nesting. If+ multiple otherwise valid link definitions appear nested inside each+ other, the inner-most definition is used.++- Brackets are allowed in the [link text] only if (a) they+ are backslash-escaped or (b) they appear as a matched pair of brackets,+ with an open bracket `[`, a sequence of zero or more inlines, and+ a close bracket `]`.++- Backtick [code spans], [autolinks], and raw [HTML tags] bind more tightly+ than the brackets in link text. Thus, for example,+ `` [foo`]` `` could not be a link text, since the second `]`+ is part of a code span.++- The brackets in link text bind more tightly than markers for+ [emphasis and strong emphasis]. Thus, for example, `*[foo*](url)` is a link.++A [link destination](@) consists of either++- a sequence of zero or more characters between an opening `<` and a+ closing `>` that contains no line breaks or unescaped+ `<` or `>` characters, or++- a nonempty sequence of characters that does not start with+ `<`, does not include ASCII space or control characters, and+ includes parentheses only if (a) they are backslash-escaped or+ (b) they are part of a balanced pair of unescaped parentheses.+ (Implementations may impose limits on parentheses nesting to+ avoid performance issues, but at least three levels of nesting+ should be supported.)++A [link title](@) consists of either++- a sequence of zero or more characters between straight double-quote+ characters (`"`), including a `"` character only if it is+ backslash-escaped, or++- a sequence of zero or more characters between straight single-quote+ characters (`'`), including a `'` character only if it is+ backslash-escaped, or++- a sequence of zero or more characters between matching parentheses+ (`(...)`), including a `(` or `)` character only if it is+ backslash-escaped.++Although [link titles] may span multiple lines, they may not contain+a [blank line].++An [inline link](@) consists of a [link text] followed immediately+by a left parenthesis `(`, optional [whitespace], an optional+[link destination], an optional [link title] separated from the link+destination by [whitespace], optional [whitespace], and a right+parenthesis `)`. The link's text consists of the inlines contained+in the [link text] (excluding the enclosing square brackets).+The link's URI consists of the link destination, excluding enclosing+`<...>` if present, with backslash-escapes in effect as described+above. The link's title consists of the link title, excluding its+enclosing delimiters, with backslash-escapes in effect as described+above.++Here is a simple inline link:++```````````````````````````````` example+[link](/uri "title")+.+<p><a href="/uri" title="title">link</a></p>+````````````````````````````````+++The title may be omitted:++```````````````````````````````` example+[link](/uri)+.+<p><a href="/uri">link</a></p>+````````````````````````````````+++Both the title and the destination may be omitted:++```````````````````````````````` example+[link]()+.+<p><a href="">link</a></p>+````````````````````````````````+++```````````````````````````````` example+[link](<>)+.+<p><a href="">link</a></p>+````````````````````````````````++The destination can only contain spaces if it is+enclosed in pointy brackets:++```````````````````````````````` example+[link](/my uri)+.+<p>[link](/my uri)</p>+````````````````````````````````++```````````````````````````````` example+[link](</my uri>)+.+<p><a href="/my%20uri">link</a></p>+````````````````````````````````++The destination cannot contain line breaks,+even if enclosed in pointy brackets:++```````````````````````````````` example+[link](foo+bar)+.+<p>[link](foo+bar)</p>+````````````````````````````````++```````````````````````````````` example+[link](<foo+bar>)+.+<p>[link](<foo+bar>)</p>+````````````````````````````````++The destination can contain `)` if it is enclosed+in pointy brackets:++```````````````````````````````` example+[a](<b)c>)+.+<p><a href="b)c">a</a></p>+````````````````````````````````++Pointy brackets that enclose links must be unescaped:++```````````````````````````````` example+[link](<foo\>)+.+<p>[link](<foo>)</p>+````````````````````````````````++These are not links, because the opening pointy bracket+is not matched properly:++```````````````````````````````` example+[a](<b)c+[a](<b)c>+[a](<b>c)+.+<p>[a](<b)c+[a](<b)c>+[a](<b>c)</p>+````````````````````````````````++Parentheses inside the link destination may be escaped:++```````````````````````````````` example+[link](\(foo\))+.+<p><a href="(foo)">link</a></p>+````````````````````````````````++Any number of parentheses are allowed without escaping, as long as they are+balanced:++```````````````````````````````` example+[link](foo(and(bar)))+.+<p><a href="foo(and(bar))">link</a></p>+````````````````````````````````++However, if you have unbalanced parentheses, you need to escape or use the+`<...>` form:++```````````````````````````````` example+[link](foo\(and\(bar\))+.+<p><a href="foo(and(bar)">link</a></p>+````````````````````````````````+++```````````````````````````````` example+[link](<foo(and(bar)>)+.+<p><a href="foo(and(bar)">link</a></p>+````````````````````````````````+++Parentheses and other symbols can also be escaped, as usual+in Markdown:++```````````````````````````````` example+[link](foo\)\:)+.+<p><a href="foo):">link</a></p>+````````````````````````````````+++A link can contain fragment identifiers and queries:++```````````````````````````````` example+[link](#fragment)++[link](http://example.com#fragment)++[link](http://example.com?foo=3#frag)+.+<p><a href="#fragment">link</a></p>+<p><a href="http://example.com#fragment">link</a></p>+<p><a href="http://example.com?foo=3#frag">link</a></p>+````````````````````````````````+++Note that a backslash before a non-escapable character is+just a backslash:++```````````````````````````````` example+[link](foo\bar)+.+<p><a href="foo%5Cbar">link</a></p>+````````````````````````````````+++URL-escaping should be left alone inside the destination, as all+URL-escaped characters are also valid URL characters. Entity and+numerical character references in the destination will be parsed+into the corresponding Unicode code points, as usual. These may+be optionally URL-escaped when written as HTML, but this spec+does not enforce any particular policy for rendering URLs in+HTML or other formats. Renderers may make different decisions+about how to escape or normalize URLs in the output.++```````````````````````````````` example+[link](foo%20bä)+.+<p><a href="foo%20b%C3%A4">link</a></p>+````````````````````````````````+++Note that, because titles can often be parsed as destinations,+if you try to omit the destination and keep the title, you'll+get unexpected results:++```````````````````````````````` example+[link]("title")+.+<p><a href="%22title%22">link</a></p>+````````````````````````````````+++Titles may be in single quotes, double quotes, or parentheses:++```````````````````````````````` example+[link](/url "title")+[link](/url 'title')+[link](/url (title))+.+<p><a href="/url" title="title">link</a>+<a href="/url" title="title">link</a>+<a href="/url" title="title">link</a></p>+````````````````````````````````+++Backslash escapes and entity and numeric character references+may be used in titles:++```````````````````````````````` example+[link](/url "title \""")+.+<p><a href="/url" title="title """>link</a></p>+````````````````````````````````+++Titles must be separated from the link using a [whitespace].+Other [Unicode whitespace] like non-breaking space doesn't work.++```````````````````````````````` example+[link](/url "title")+.+<p><a href="/url%C2%A0%22title%22">link</a></p>+````````````````````````````````+++Nested balanced quotes are not allowed without escaping:++```````````````````````````````` example+[link](/url "title "and" title")+.+<p>[link](/url "title "and" title")</p>+````````````````````````````````+++But it is easy to work around this by using a different quote type:++```````````````````````````````` example+[link](/url 'title "and" title')+.+<p><a href="/url" title="title "and" title">link</a></p>+````````````````````````````````+++(Note: `Markdown.pl` did allow double quotes inside a double-quoted+title, and its test suite included a test demonstrating this.+But it is hard to see a good rationale for the extra complexity this+brings, since there are already many ways---backslash escaping,+entity and numeric character references, or using a different+quote type for the enclosing title---to write titles containing+double quotes. `Markdown.pl`'s handling of titles has a number+of other strange features. For example, it allows single-quoted+titles in inline links, but not reference links. And, in+reference links but not inline links, it allows a title to begin+with `"` and end with `)`. `Markdown.pl` 1.0.1 even allows+titles with no closing quotation mark, though 1.0.2b8 does not.+It seems preferable to adopt a simple, rational rule that works+the same way in inline links and link reference definitions.)++[Whitespace] is allowed around the destination and title:++```````````````````````````````` example+[link]( /uri+ "title" )+.+<p><a href="/uri" title="title">link</a></p>+````````````````````````````````+++But it is not allowed between the link text and the+following parenthesis:++```````````````````````````````` example+[link] (/uri)+.+<p>[link] (/uri)</p>+````````````````````````````````+++The link text may contain balanced brackets, but not unbalanced ones,+unless they are escaped:++```````````````````````````````` example+[link [foo [bar]]](/uri)+.+<p><a href="/uri">link [foo [bar]]</a></p>+````````````````````````````````+++```````````````````````````````` example+[link] bar](/uri)+.+<p>[link] bar](/uri)</p>+````````````````````````````````+++```````````````````````````````` example+[link [bar](/uri)+.+<p>[link <a href="/uri">bar</a></p>+````````````````````````````````+++```````````````````````````````` example+[link \[bar](/uri)+.+<p><a href="/uri">link [bar</a></p>+````````````````````````````````+++The link text may contain inline content:++```````````````````````````````` example+[link *foo **bar** `#`*](/uri)+.+<p><a href="/uri">link <em>foo <strong>bar</strong> <code>#</code></em></a></p>+````````````````````````````````+++```````````````````````````````` example+[](/uri)+.+<p><a href="/uri"><img src="moon.jpg" alt="moon" /></a></p>+````````````````````````````````+++However, links may not contain other links, at any level of nesting.++```````````````````````````````` example+[foo [bar](/uri)](/uri)+.+<p>[foo <a href="/uri">bar</a>](/uri)</p>+````````````````````````````````+++```````````````````````````````` example+[foo *[bar [baz](/uri)](/uri)*](/uri)+.+<p>[foo <em>[bar <a href="/uri">baz</a>](/uri)</em>](/uri)</p>+````````````````````````````````+++```````````````````````````````` example+](uri2)](uri3)+.+<p><img src="uri3" alt="[foo](uri2)" /></p>+````````````````````````````````+++These cases illustrate the precedence of link text grouping over+emphasis grouping:++```````````````````````````````` example+*[foo*](/uri)+.+<p>*<a href="/uri">foo*</a></p>+````````````````````````````````+++```````````````````````````````` example+[foo *bar](baz*)+.+<p><a href="baz*">foo *bar</a></p>+````````````````````````````````+++Note that brackets that *aren't* part of links do not take+precedence:++```````````````````````````````` example+*foo [bar* baz]+.+<p><em>foo [bar</em> baz]</p>+````````````````````````````````+++These cases illustrate the precedence of HTML tags, code spans,+and autolinks over link grouping:++```````````````````````````````` example+[foo <bar attr="](baz)">+.+<p>[foo <bar attr="](baz)"></p>+````````````````````````````````+++```````````````````````````````` example+[foo`](/uri)`+.+<p>[foo<code>](/uri)</code></p>+````````````````````````````````+++```````````````````````````````` example+[foo<http://example.com/?search=](uri)>+.+<p>[foo<a href="http://example.com/?search=%5D(uri)">http://example.com/?search=](uri)</a></p>+````````````````````````````````+++There are three kinds of [reference link](@)s:+[full](#full-reference-link), [collapsed](#collapsed-reference-link),+and [shortcut](#shortcut-reference-link).++A [full reference link](@)+consists of a [link text] immediately followed by a [link label]+that [matches] a [link reference definition] elsewhere in the document.++A [link label](@) begins with a left bracket (`[`) and ends+with the first right bracket (`]`) that is not backslash-escaped.+Between these brackets there must be at least one [non-whitespace character].+Unescaped square bracket characters are not allowed inside the+opening and closing square brackets of [link labels]. A link+label can have at most 999 characters inside the square+brackets.++One label [matches](@)+another just in case their normalized forms are equal. To normalize a+label, strip off the opening and closing brackets,+perform the *Unicode case fold*, strip leading and trailing+[whitespace] and collapse consecutive internal+[whitespace] to a single space. If there are multiple+matching reference link definitions, the one that comes first in the+document is used. (It is desirable in such cases to emit a warning.)++The contents of the first link label are parsed as inlines, which are+used as the link's text. The link's URI and title are provided by the+matching [link reference definition].++Here is a simple example:++```````````````````````````````` example+[foo][bar]++[bar]: /url "title"+.+<p><a href="/url" title="title">foo</a></p>+````````````````````````````````+++The rules for the [link text] are the same as with+[inline links]. Thus:++The link text may contain balanced brackets, but not unbalanced ones,+unless they are escaped:++```````````````````````````````` example+[link [foo [bar]]][ref]++[ref]: /uri+.+<p><a href="/uri">link [foo [bar]]</a></p>+````````````````````````````````+++```````````````````````````````` example+[link \[bar][ref]++[ref]: /uri+.+<p><a href="/uri">link [bar</a></p>+````````````````````````````````+++The link text may contain inline content:++```````````````````````````````` example+[link *foo **bar** `#`*][ref]++[ref]: /uri+.+<p><a href="/uri">link <em>foo <strong>bar</strong> <code>#</code></em></a></p>+````````````````````````````````+++```````````````````````````````` example+[][ref]++[ref]: /uri+.+<p><a href="/uri"><img src="moon.jpg" alt="moon" /></a></p>+````````````````````````````````+++However, links may not contain other links, at any level of nesting.++```````````````````````````````` example+[foo [bar](/uri)][ref]++[ref]: /uri+.+<p>[foo <a href="/uri">bar</a>]<a href="/uri">ref</a></p>+````````````````````````````````+++```````````````````````````````` example+[foo *bar [baz][ref]*][ref]++[ref]: /uri+.+<p>[foo <em>bar <a href="/uri">baz</a></em>]<a href="/uri">ref</a></p>+````````````````````````````````+++(In the examples above, we have two [shortcut reference links]+instead of one [full reference link].)++The following cases illustrate the precedence of link text grouping over+emphasis grouping:++```````````````````````````````` example+*[foo*][ref]++[ref]: /uri+.+<p>*<a href="/uri">foo*</a></p>+````````````````````````````````+++```````````````````````````````` example+[foo *bar][ref]++[ref]: /uri+.+<p><a href="/uri">foo *bar</a></p>+````````````````````````````````+++These cases illustrate the precedence of HTML tags, code spans,+and autolinks over link grouping:++```````````````````````````````` example+[foo <bar attr="][ref]">++[ref]: /uri+.+<p>[foo <bar attr="][ref]"></p>+````````````````````````````````+++```````````````````````````````` example+[foo`][ref]`++[ref]: /uri+.+<p>[foo<code>][ref]</code></p>+````````````````````````````````+++```````````````````````````````` example+[foo<http://example.com/?search=][ref]>++[ref]: /uri+.+<p>[foo<a href="http://example.com/?search=%5D%5Bref%5D">http://example.com/?search=][ref]</a></p>+````````````````````````````````+++Matching is case-insensitive:++```````````````````````````````` example+[foo][BaR]++[bar]: /url "title"+.+<p><a href="/url" title="title">foo</a></p>+````````````````````````````````+++Unicode case fold is used:++```````````````````````````````` example+[Толпой][Толпой] is a Russian word.++[ТОЛПОЙ]: /url+.+<p><a href="/url">Толпой</a> is a Russian word.</p>+````````````````````````````````+++Consecutive internal [whitespace] is treated as one space for+purposes of determining matching:++```````````````````````````````` example+[Foo+ bar]: /url++[Baz][Foo bar]+.+<p><a href="/url">Baz</a></p>+````````````````````````````````+++No [whitespace] is allowed between the [link text] and the+[link label]:++```````````````````````````````` example+[foo] [bar]++[bar]: /url "title"+.+<p>[foo] <a href="/url" title="title">bar</a></p>+````````````````````````````````+++```````````````````````````````` example+[foo]+[bar]++[bar]: /url "title"+.+<p>[foo]+<a href="/url" title="title">bar</a></p>+````````````````````````````````+++This is a departure from John Gruber's original Markdown syntax+description, which explicitly allows whitespace between the link+text and the link label. It brings reference links in line with+[inline links], which (according to both original Markdown and+this spec) cannot have whitespace after the link text. More+importantly, it prevents inadvertent capture of consecutive+[shortcut reference links]. If whitespace is allowed between the+link text and the link label, then in the following we will have+a single reference link, not two shortcut reference links, as+intended:++``` markdown+[foo]+[bar]++[foo]: /url1+[bar]: /url2+```++(Note that [shortcut reference links] were introduced by Gruber+himself in a beta version of `Markdown.pl`, but never included+in the official syntax description. Without shortcut reference+links, it is harmless to allow space between the link text and+link label; but once shortcut references are introduced, it is+too dangerous to allow this, as it frequently leads to+unintended results.)++When there are multiple matching [link reference definitions],+the first is used:++```````````````````````````````` example+[foo]: /url1++[foo]: /url2++[bar][foo]+.+<p><a href="/url1">bar</a></p>+````````````````````````````````+++Note that matching is performed on normalized strings, not parsed+inline content. So the following does not match, even though the+labels define equivalent inline content:++```````````````````````````````` example+[bar][foo\!]++[foo!]: /url+.+<p>[bar][foo!]</p>+````````````````````````````````+++[Link labels] cannot contain brackets, unless they are+backslash-escaped:++```````````````````````````````` example+[foo][ref[]++[ref[]: /uri+.+<p>[foo][ref[]</p>+<p>[ref[]: /uri</p>+````````````````````````````````+++```````````````````````````````` example+[foo][ref[bar]]++[ref[bar]]: /uri+.+<p>[foo][ref[bar]]</p>+<p>[ref[bar]]: /uri</p>+````````````````````````````````+++```````````````````````````````` example+[[[foo]]]++[[[foo]]]: /url+.+<p>[[[foo]]]</p>+<p>[[[foo]]]: /url</p>+````````````````````````````````+++```````````````````````````````` example+[foo][ref\[]++[ref\[]: /uri+.+<p><a href="/uri">foo</a></p>+````````````````````````````````+++Note that in this example `]` is not backslash-escaped:++```````````````````````````````` example+[bar\\]: /uri++[bar\\]+.+<p><a href="/uri">bar\</a></p>+````````````````````````````````+++A [link label] must contain at least one [non-whitespace character]:++```````````````````````````````` example+[]++[]: /uri+.+<p>[]</p>+<p>[]: /uri</p>+````````````````````````````````+++```````````````````````````````` example+[+ ]++[+ ]: /uri+.+<p>[+]</p>+<p>[+]: /uri</p>+````````````````````````````````+++A [collapsed reference link](@)+consists of a [link label] that [matches] a+[link reference definition] elsewhere in the+document, followed by the string `[]`.+The contents of the first link label are parsed as inlines,+which are used as the link's text. The link's URI and title are+provided by the matching reference link definition. Thus,+`[foo][]` is equivalent to `[foo][foo]`.++```````````````````````````````` example+[foo][]++[foo]: /url "title"+.+<p><a href="/url" title="title">foo</a></p>+````````````````````````````````+++```````````````````````````````` example+[*foo* bar][]++[*foo* bar]: /url "title"+.+<p><a href="/url" title="title"><em>foo</em> bar</a></p>+````````````````````````````````+++The link labels are case-insensitive:++```````````````````````````````` example+[Foo][]++[foo]: /url "title"+.+<p><a href="/url" title="title">Foo</a></p>+````````````````````````````````++++As with full reference links, [whitespace] is not+allowed between the two sets of brackets:++```````````````````````````````` example+[foo] +[]++[foo]: /url "title"+.+<p><a href="/url" title="title">foo</a>+[]</p>+````````````````````````````````+++A [shortcut reference link](@)+consists of a [link label] that [matches] a+[link reference definition] elsewhere in the+document and is not followed by `[]` or a link label.+The contents of the first link label are parsed as inlines,+which are used as the link's text. The link's URI and title+are provided by the matching link reference definition.+Thus, `[foo]` is equivalent to `[foo][]`.++```````````````````````````````` example+[foo]++[foo]: /url "title"+.+<p><a href="/url" title="title">foo</a></p>+````````````````````````````````+++```````````````````````````````` example+[*foo* bar]++[*foo* bar]: /url "title"+.+<p><a href="/url" title="title"><em>foo</em> bar</a></p>+````````````````````````````````+++```````````````````````````````` example+[[*foo* bar]]++[*foo* bar]: /url "title"+.+<p>[<a href="/url" title="title"><em>foo</em> bar</a>]</p>+````````````````````````````````+++```````````````````````````````` example+[[bar [foo]++[foo]: /url+.+<p>[[bar <a href="/url">foo</a></p>+````````````````````````````````+++The link labels are case-insensitive:++```````````````````````````````` example+[Foo]++[foo]: /url "title"+.+<p><a href="/url" title="title">Foo</a></p>+````````````````````````````````+++A space after the link text should be preserved:++```````````````````````````````` example+[foo] bar++[foo]: /url+.+<p><a href="/url">foo</a> bar</p>+````````````````````````````````+++If you just want bracketed text, you can backslash-escape the+opening bracket to avoid links:++```````````````````````````````` example+\[foo]++[foo]: /url "title"+.+<p>[foo]</p>+````````````````````````````````+++Note that this is a link, because a link label ends with the first+following closing bracket:++```````````````````````````````` example+[foo*]: /url++*[foo*]+.+<p>*<a href="/url">foo*</a></p>+````````````````````````````````+++Full and compact references take precedence over shortcut+references:++```````````````````````````````` example+[foo][bar]++[foo]: /url1+[bar]: /url2+.+<p><a href="/url2">foo</a></p>+````````````````````````````````++```````````````````````````````` example+[foo][]++[foo]: /url1+.+<p><a href="/url1">foo</a></p>+````````````````````````````````++Inline links also take precedence:++```````````````````````````````` example+[foo]()++[foo]: /url1+.+<p><a href="">foo</a></p>+````````````````````````````````++```````````````````````````````` example+[foo](not a link)++[foo]: /url1+.+<p><a href="/url1">foo</a>(not a link)</p>+````````````````````````````````++In the following case `[bar][baz]` is parsed as a reference,+`[foo]` as normal text:++```````````````````````````````` example+[foo][bar][baz]++[baz]: /url+.+<p>[foo]<a href="/url">bar</a></p>+````````````````````````````````+++Here, though, `[foo][bar]` is parsed as a reference, since+`[bar]` is defined:++```````````````````````````````` example+[foo][bar][baz]++[baz]: /url1+[bar]: /url2+.+<p><a href="/url2">foo</a><a href="/url1">baz</a></p>+````````````````````````````````+++Here `[foo]` is not parsed as a shortcut reference, because it+is followed by a link label (even though `[bar]` is not defined):++```````````````````````````````` example+[foo][bar][baz]++[baz]: /url1+[foo]: /url2+.+<p>[foo]<a href="/url1">bar</a></p>+````````````````````````````````++++## Images++Syntax for images is like the syntax for links, with one+difference. Instead of [link text], we have an+[image description](@). The rules for this are the+same as for [link text], except that (a) an+image description starts with `+.+<p><img src="/url" alt="foo" title="title" /></p>+````````````````````````````````+++```````````````````````````````` example+![foo *bar*]++[foo *bar*]: train.jpg "train & tracks"+.+<p><img src="train.jpg" alt="foo bar" title="train & tracks" /></p>+````````````````````````````````+++```````````````````````````````` example+](/url2)+.+<p><img src="/url2" alt="foo bar" /></p>+````````````````````````````````+++```````````````````````````````` example+](/url2)+.+<p><img src="/url2" alt="foo bar" /></p>+````````````````````````````````+++Though this spec is concerned with parsing, not rendering, it is+recommended that in rendering to HTML, only the plain string content+of the [image description] be used. Note that in+the above example, the alt attribute's value is `foo bar`, not `foo+[bar](/url)` or `foo <a href="/url">bar</a>`. Only the plain string+content is rendered, without formatting.++```````````````````````````````` example+![foo *bar*][]++[foo *bar*]: train.jpg "train & tracks"+.+<p><img src="train.jpg" alt="foo bar" title="train & tracks" /></p>+````````````````````````````````+++```````````````````````````````` example+![foo *bar*][foobar]++[FOOBAR]: train.jpg "train & tracks"+.+<p><img src="train.jpg" alt="foo bar" title="train & tracks" /></p>+````````````````````````````````+++```````````````````````````````` example++.+<p><img src="train.jpg" alt="foo" /></p>+````````````````````````````````+++```````````````````````````````` example+My +.+<p>My <img src="/path/to/train.jpg" alt="foo bar" title="title" /></p>+````````````````````````````````+++```````````````````````````````` example++.+<p><img src="url" alt="foo" /></p>+````````````````````````````````+++```````````````````````````````` example++.+<p><img src="/url" alt="" /></p>+````````````````````````````````+++Reference-style:++```````````````````````````````` example+![foo][bar]++[bar]: /url+.+<p><img src="/url" alt="foo" /></p>+````````````````````````````````+++```````````````````````````````` example+![foo][bar]++[BAR]: /url+.+<p><img src="/url" alt="foo" /></p>+````````````````````````````````+++Collapsed:++```````````````````````````````` example+![foo][]++[foo]: /url "title"+.+<p><img src="/url" alt="foo" title="title" /></p>+````````````````````````````````+++```````````````````````````````` example+![*foo* bar][]++[*foo* bar]: /url "title"+.+<p><img src="/url" alt="foo bar" title="title" /></p>+````````````````````````````````+++The labels are case-insensitive:++```````````````````````````````` example+![Foo][]++[foo]: /url "title"+.+<p><img src="/url" alt="Foo" title="title" /></p>+````````````````````````````````+++As with reference links, [whitespace] is not allowed+between the two sets of brackets:++```````````````````````````````` example+![foo] +[]++[foo]: /url "title"+.+<p><img src="/url" alt="foo" title="title" />+[]</p>+````````````````````````````````+++Shortcut:++```````````````````````````````` example+![foo]++[foo]: /url "title"+.+<p><img src="/url" alt="foo" title="title" /></p>+````````````````````````````````+++```````````````````````````````` example+![*foo* bar]++[*foo* bar]: /url "title"+.+<p><img src="/url" alt="foo bar" title="title" /></p>+````````````````````````````````+++Note that link labels cannot contain unescaped brackets:++```````````````````````````````` example+![[foo]]++[[foo]]: /url "title"+.+<p>![[foo]]</p>+<p>[[foo]]: /url "title"</p>+````````````````````````````````+++The link labels are case-insensitive:++```````````````````````````````` example+![Foo]++[foo]: /url "title"+.+<p><img src="/url" alt="Foo" title="title" /></p>+````````````````````````````````+++If you just want a literal `!` followed by bracketed text, you can+backslash-escape the opening `[`:++```````````````````````````````` example+!\[foo]++[foo]: /url "title"+.+<p>![foo]</p>+````````````````````````````````+++If you want a link after a literal `!`, backslash-escape the+`!`:++```````````````````````````````` example+\![foo]++[foo]: /url "title"+.+<p>!<a href="/url" title="title">foo</a></p>+````````````````````````````````+++## Autolinks++[Autolink](@)s are absolute URIs and email addresses inside+`<` and `>`. They are parsed as links, with the URL or email address+as the link label.++A [URI autolink](@) consists of `<`, followed by an+[absolute URI] followed by `>`. It is parsed as+a link to the URI, with the URI as the link's label.++An [absolute URI](@),+for these purposes, consists of a [scheme] followed by a colon (`:`)+followed by zero or more characters other than ASCII+[whitespace] and control characters, `<`, and `>`. If+the URI includes these characters, they must be percent-encoded+(e.g. `%20` for a space).++For purposes of this spec, a [scheme](@) is any sequence+of 2--32 characters beginning with an ASCII letter and followed+by any combination of ASCII letters, digits, or the symbols plus+("+"), period ("."), or hyphen ("-").++Here are some valid autolinks:++```````````````````````````````` example+<http://foo.bar.baz>+.+<p><a href="http://foo.bar.baz">http://foo.bar.baz</a></p>+````````````````````````````````+++```````````````````````````````` example+<http://foo.bar.baz/test?q=hello&id=22&boolean>+.+<p><a href="http://foo.bar.baz/test?q=hello&id=22&boolean">http://foo.bar.baz/test?q=hello&id=22&boolean</a></p>+````````````````````````````````+++```````````````````````````````` example+<irc://foo.bar:2233/baz>+.+<p><a href="irc://foo.bar:2233/baz">irc://foo.bar:2233/baz</a></p>+````````````````````````````````+++Uppercase is also fine:++```````````````````````````````` example+<MAILTO:FOO@BAR.BAZ>+.+<p><a href="MAILTO:FOO@BAR.BAZ">MAILTO:FOO@BAR.BAZ</a></p>+````````````````````````````````+++Note that many strings that count as [absolute URIs] for+purposes of this spec are not valid URIs, because their+schemes are not registered or because of other problems+with their syntax:++```````````````````````````````` example+<a+b+c:d>+.+<p><a href="a+b+c:d">a+b+c:d</a></p>+````````````````````````````````+++```````````````````````````````` example+<made-up-scheme://foo,bar>+.+<p><a href="made-up-scheme://foo,bar">made-up-scheme://foo,bar</a></p>+````````````````````````````````+++```````````````````````````````` example+<http://../>+.+<p><a href="http://../">http://../</a></p>+````````````````````````````````+++```````````````````````````````` example+<localhost:5001/foo>+.+<p><a href="localhost:5001/foo">localhost:5001/foo</a></p>+````````````````````````````````+++Spaces are not allowed in autolinks:++```````````````````````````````` example+<http://foo.bar/baz bim>+.+<p><http://foo.bar/baz bim></p>+````````````````````````````````+++Backslash-escapes do not work inside autolinks:++```````````````````````````````` example+<http://example.com/\[\>+.+<p><a href="http://example.com/%5C%5B%5C">http://example.com/\[\</a></p>+````````````````````````````````+++An [email autolink](@)+consists of `<`, followed by an [email address],+followed by `>`. The link's label is the email address,+and the URL is `mailto:` followed by the email address.++An [email address](@),+for these purposes, is anything that matches+the [non-normative regex from the HTML5+spec](https://html.spec.whatwg.org/multipage/forms.html#e-mail-state-(type=email)):++ /^[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?+ (?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$/++Examples of email autolinks:++```````````````````````````````` example+<foo@bar.example.com>+.+<p><a href="mailto:foo@bar.example.com">foo@bar.example.com</a></p>+````````````````````````````````+++```````````````````````````````` example+<foo+special@Bar.baz-bar0.com>+.+<p><a href="mailto:foo+special@Bar.baz-bar0.com">foo+special@Bar.baz-bar0.com</a></p>+````````````````````````````````+++Backslash-escapes do not work inside email autolinks:++```````````````````````````````` example+<foo\+@bar.example.com>+.+<p><foo+@bar.example.com></p>+````````````````````````````````+++These are not autolinks:++```````````````````````````````` example+<>+.+<p><></p>+````````````````````````````````+++```````````````````````````````` example+< http://foo.bar >+.+<p>< http://foo.bar ></p>+````````````````````````````````+++```````````````````````````````` example+<m:abc>+.+<p><m:abc></p>+````````````````````````````````+++```````````````````````````````` example+<foo.bar.baz>+.+<p><foo.bar.baz></p>+````````````````````````````````+++```````````````````````````````` example+http://example.com+.+<p>http://example.com</p>+````````````````````````````````+++```````````````````````````````` example+foo@bar.example.com+.+<p>foo@bar.example.com</p>+````````````````````````````````+++## Raw HTML++Text between `<` and `>` that looks like an HTML tag is parsed as a+raw HTML tag and will be rendered in HTML without escaping.+Tag and attribute names are not limited to current HTML tags,+so custom tags (and even, say, DocBook tags) may be used.++Here is the grammar for tags:++A [tag name](@) consists of an ASCII letter+followed by zero or more ASCII letters, digits, or+hyphens (`-`).++An [attribute](@) consists of [whitespace],+an [attribute name], and an optional+[attribute value specification].++An [attribute name](@)+consists of an ASCII letter, `_`, or `:`, followed by zero or more ASCII+letters, digits, `_`, `.`, `:`, or `-`. (Note: This is the XML+specification restricted to ASCII. HTML5 is laxer.)++An [attribute value specification](@)+consists of optional [whitespace],+a `=` character, optional [whitespace], and an [attribute+value].++An [attribute value](@)+consists of an [unquoted attribute value],+a [single-quoted attribute value], or a [double-quoted attribute value].++An [unquoted attribute value](@)+is a nonempty string of characters not+including [whitespace], `"`, `'`, `=`, `<`, `>`, or `` ` ``.++A [single-quoted attribute value](@)+consists of `'`, zero or more+characters not including `'`, and a final `'`.++A [double-quoted attribute value](@)+consists of `"`, zero or more+characters not including `"`, and a final `"`.++An [open tag](@) consists of a `<` character, a [tag name],+zero or more [attributes], optional [whitespace], an optional `/`+character, and a `>` character.++A [closing tag](@) consists of the string `</`, a+[tag name], optional [whitespace], and the character `>`.++An [HTML comment](@) consists of `<!--` + *text* + `-->`,+where *text* does not start with `>` or `->`, does not end with `-`,+and does not contain `--`. (See the+[HTML5 spec](http://www.w3.org/TR/html5/syntax.html#comments).)++A [processing instruction](@)+consists of the string `<?`, a string+of characters not including the string `?>`, and the string+`?>`.++A [declaration](@) consists of the+string `<!`, a name consisting of one or more uppercase ASCII letters,+[whitespace], a string of characters not including the+character `>`, and the character `>`.++A [CDATA section](@) consists of+the string `<![CDATA[`, a string of characters not including the string+`]]>`, and the string `]]>`.++An [HTML tag](@) consists of an [open tag], a [closing tag],+an [HTML comment], a [processing instruction], a [declaration],+or a [CDATA section].++Here are some simple open tags:++```````````````````````````````` example+<a><bab><c2c>+.+<p><a><bab><c2c></p>+````````````````````````````````+++Empty elements:++```````````````````````````````` example+<a/><b2/>+.+<p><a/><b2/></p>+````````````````````````````````+++[Whitespace] is allowed:++```````````````````````````````` example+<a /><b2+data="foo" >+.+<p><a /><b2+data="foo" ></p>+````````````````````````````````+++With attributes:++```````````````````````````````` example+<a foo="bar" bam = 'baz <em>"</em>'+_boolean zoop:33=zoop:33 />+.+<p><a foo="bar" bam = 'baz <em>"</em>'+_boolean zoop:33=zoop:33 /></p>+````````````````````````````````+++Custom tag names can be used:++```````````````````````````````` example+Foo <responsive-image src="foo.jpg" />+.+<p>Foo <responsive-image src="foo.jpg" /></p>+````````````````````````````````+++Illegal tag names, not parsed as HTML:++```````````````````````````````` example+<33> <__>+.+<p><33> <__></p>+````````````````````````````````+++Illegal attribute names:++```````````````````````````````` example+<a h*#ref="hi">+.+<p><a h*#ref="hi"></p>+````````````````````````````````+++Illegal attribute values:++```````````````````````````````` example+<a href="hi'> <a href=hi'>+.+<p><a href="hi'> <a href=hi'></p>+````````````````````````````````+++Illegal [whitespace]:++```````````````````````````````` example+< a><+foo><bar/ >+<foo bar=baz+bim!bop />+.+<p>< a><+foo><bar/ >+<foo bar=baz+bim!bop /></p>+````````````````````````````````+++Missing [whitespace]:++```````````````````````````````` example+<a href='bar'title=title>+.+<p><a href='bar'title=title></p>+````````````````````````````````+++Closing tags:++```````````````````````````````` example+</a></foo >+.+<p></a></foo ></p>+````````````````````````````````+++Illegal attributes in closing tag:++```````````````````````````````` example+</a href="foo">+.+<p></a href="foo"></p>+````````````````````````````````+++Comments:++```````````````````````````````` example+foo <!-- this is a+comment - with hyphen -->+.+<p>foo <!-- this is a+comment - with hyphen --></p>+````````````````````````````````+++```````````````````````````````` example+foo <!-- not a comment -- two hyphens -->+.+<p>foo <!-- not a comment -- two hyphens --></p>+````````````````````````````````+++Not comments:++```````````````````````````````` example+foo <!--> foo -->++foo <!-- foo--->+.+<p>foo <!--> foo --></p>+<p>foo <!-- foo---></p>+````````````````````````````````+++Processing instructions:++```````````````````````````````` example+foo <?php echo $a; ?>+.+<p>foo <?php echo $a; ?></p>+````````````````````````````````+++Declarations:++```````````````````````````````` example+foo <!ELEMENT br EMPTY>+.+<p>foo <!ELEMENT br EMPTY></p>+````````````````````````````````+++CDATA sections:++```````````````````````````````` example+foo <![CDATA[>&<]]>+.+<p>foo <![CDATA[>&<]]></p>+````````````````````````````````+++Entity and numeric character references are preserved in HTML+attributes:++```````````````````````````````` example+foo <a href="ö">+.+<p>foo <a href="ö"></p>+````````````````````````````````+++Backslash escapes do not work in HTML attributes:++```````````````````````````````` example+foo <a href="\*">+.+<p>foo <a href="\*"></p>+````````````````````````````````+++```````````````````````````````` example+<a href="\"">+.+<p><a href="""></p>+````````````````````````````````+++## Hard line breaks++A line break (not in a code span or HTML tag) that is preceded+by two or more spaces and does not occur at the end of a block+is parsed as a [hard line break](@) (rendered+in HTML as a `<br />` tag):++```````````````````````````````` example+foo +baz+.+<p>foo<br />+baz</p>+````````````````````````````````+++For a more visible alternative, a backslash before the+[line ending] may be used instead of two spaces:++```````````````````````````````` example+foo\+baz+.+<p>foo<br />+baz</p>+````````````````````````````````+++More than two spaces can be used:++```````````````````````````````` example+foo +baz+.+<p>foo<br />+baz</p>+````````````````````````````````+++Leading spaces at the beginning of the next line are ignored:++```````````````````````````````` example+foo + bar+.+<p>foo<br />+bar</p>+````````````````````````````````+++```````````````````````````````` example+foo\+ bar+.+<p>foo<br />+bar</p>+````````````````````````````````+++Line breaks can occur inside emphasis, links, and other constructs+that allow inline content:++```````````````````````````````` example+*foo +bar*+.+<p><em>foo<br />+bar</em></p>+````````````````````````````````+++```````````````````````````````` example+*foo\+bar*+.+<p><em>foo<br />+bar</em></p>+````````````````````````````````+++Line breaks do not occur inside code spans++```````````````````````````````` example+`code +span`+.+<p><code>code span</code></p>+````````````````````````````````+++```````````````````````````````` example+`code\+span`+.+<p><code>code\ span</code></p>+````````````````````````````````+++or HTML tags:++```````````````````````````````` example+<a href="foo +bar">+.+<p><a href="foo +bar"></p>+````````````````````````````````+++```````````````````````````````` example+<a href="foo\+bar">+.+<p><a href="foo\+bar"></p>+````````````````````````````````+++Hard line breaks are for separating inline content within a block.+Neither syntax for hard line breaks works at the end of a paragraph or+other block element:++```````````````````````````````` example+foo\+.+<p>foo\</p>+````````````````````````````````+++```````````````````````````````` example+foo +.+<p>foo</p>+````````````````````````````````+++```````````````````````````````` example+### foo\+.+<h3>foo\</h3>+````````````````````````````````+++```````````````````````````````` example+### foo +.+<h3>foo</h3>+````````````````````````````````+++## Soft line breaks++A regular line break (not in a code span or HTML tag) that is not+preceded by two or more spaces or a backslash is parsed as a+[softbreak](@). (A softbreak may be rendered in HTML either as a+[line ending] or as a space. The result will be the same in+browsers. In the examples here, a [line ending] will be used.)++```````````````````````````````` example+foo+baz+.+<p>foo+baz</p>+````````````````````````````````+++Spaces at the end of the line and beginning of the next line are+removed:++```````````````````````````````` example+foo + baz+.+<p>foo+baz</p>+````````````````````````````````+++A conforming parser may render a soft line break in HTML either as a+line break or as a space.++A renderer may also provide an option to render soft line breaks+as hard line breaks.++## Textual content++Any characters not given an interpretation by the above rules will+be parsed as plain textual content.++```````````````````````````````` example+hello $.;'there+.+<p>hello $.;'there</p>+````````````````````````````````+++```````````````````````````````` example+Foo χρῆν+.+<p>Foo χρῆν</p>+````````````````````````````````+++Internal spaces are preserved verbatim:++```````````````````````````````` example+Multiple spaces+.+<p>Multiple spaces</p>+````````````````````````````````+++<!-- END TESTS -->++# Appendix: A parsing strategy++In this appendix we describe some features of the parsing strategy+used in the CommonMark reference implementations.++## Overview++Parsing has two phases:++1. In the first phase, lines of input are consumed and the block+structure of the document---its division into paragraphs, block quotes,+list items, and so on---is constructed. Text is assigned to these+blocks but not parsed. Link reference definitions are parsed and a+map of links is constructed.++2. In the second phase, the raw text contents of paragraphs and headings+are parsed into sequences of Markdown inline elements (strings,+code spans, links, emphasis, and so on), using the map of link+references constructed in phase 1.++At each point in processing, the document is represented as a tree of+**blocks**. The root of the tree is a `document` block. The `document`+may have any number of other blocks as **children**. These children+may, in turn, have other blocks as children. The last child of a block+is normally considered **open**, meaning that subsequent lines of input+can alter its contents. (Blocks that are not open are **closed**.)+Here, for example, is a possible document tree, with the open blocks+marked by arrows:++``` tree+-> document+ -> block_quote+ paragraph+ "Lorem ipsum dolor\nsit amet."+ -> list (type=bullet tight=true bullet_char=-)+ list_item+ paragraph+ "Qui *quodsi iracundia*"+ -> list_item+ -> paragraph+ "aliquando id"+```++## Phase 1: block structure++Each line that is processed has an effect on this tree. The line is+analyzed and, depending on its contents, the document may be altered+in one or more of the following ways:++1. One or more open blocks may be closed.+2. One or more new blocks may be created as children of the+ last open block.+3. Text may be added to the last (deepest) open block remaining+ on the tree.++Once a line has been incorporated into the tree in this way,+it can be discarded, so input can be read in a stream.++For each line, we follow this procedure:++1. First we iterate through the open blocks, starting with the+root document, and descending through last children down to the last+open block. Each block imposes a condition that the line must satisfy+if the block is to remain open. For example, a block quote requires a+`>` character. A paragraph requires a non-blank line.+In this phase we may match all or just some of the open+blocks. But we cannot close unmatched blocks yet, because we may have a+[lazy continuation line].++2. Next, after consuming the continuation markers for existing+blocks, we look for new block starts (e.g. `>` for a block quote).+If we encounter a new block start, we close any blocks unmatched+in step 1 before creating the new block as a child of the last+matched block.++3. Finally, we look at the remainder of the line (after block+markers like `>`, list markers, and indentation have been consumed).+This is text that can be incorporated into the last open+block (a paragraph, code block, heading, or raw HTML).++Setext headings are formed when we see a line of a paragraph+that is a [setext heading underline].++Reference link definitions are detected when a paragraph is closed;+the accumulated text lines are parsed to see if they begin with+one or more reference link definitions. Any remainder becomes a+normal paragraph.++We can see how this works by considering how the tree above is+generated by four lines of Markdown:++``` markdown+> Lorem ipsum dolor+sit amet.+> - Qui *quodsi iracundia*+> - aliquando id+```++At the outset, our document model is just++``` tree+-> document+```++The first line of our text,++``` markdown+> Lorem ipsum dolor+```++causes a `block_quote` block to be created as a child of our+open `document` block, and a `paragraph` block as a child of+the `block_quote`. Then the text is added to the last open+block, the `paragraph`:++``` tree+-> document+ -> block_quote+ -> paragraph+ "Lorem ipsum dolor"+```++The next line,++``` markdown+sit amet.+```++is a "lazy continuation" of the open `paragraph`, so it gets added+to the paragraph's text:++``` tree+-> document+ -> block_quote+ -> paragraph+ "Lorem ipsum dolor\nsit amet."+```++The third line,++``` markdown+> - Qui *quodsi iracundia*+```++causes the `paragraph` block to be closed, and a new `list` block+opened as a child of the `block_quote`. A `list_item` is also+added as a child of the `list`, and a `paragraph` as a child of+the `list_item`. The text is then added to the new `paragraph`:++``` tree+-> document+ -> block_quote+ paragraph+ "Lorem ipsum dolor\nsit amet."+ -> list (type=bullet tight=true bullet_char=-)+ -> list_item+ -> paragraph+ "Qui *quodsi iracundia*"+```++The fourth line,++``` markdown+> - aliquando id+```++causes the `list_item` (and its child the `paragraph`) to be closed,+and a new `list_item` opened up as child of the `list`. A `paragraph`+is added as a child of the new `list_item`, to contain the text.+We thus obtain the final tree:++``` tree+-> document+ -> block_quote+ paragraph+ "Lorem ipsum dolor\nsit amet."+ -> list (type=bullet tight=true bullet_char=-)+ list_item+ paragraph+ "Qui *quodsi iracundia*"+ -> list_item+ -> paragraph+ "aliquando id"+```++## Phase 2: inline structure++Once all of the input has been parsed, all open blocks are closed.++We then "walk the tree," visiting every node, and parse raw+string contents of paragraphs and headings as inlines. At this+point we have seen all the link reference definitions, so we can+resolve reference links as we go.++``` tree+document+ block_quote+ paragraph+ str "Lorem ipsum dolor"+ softbreak+ str "sit amet."+ list (type=bullet tight=true bullet_char=-)+ list_item+ paragraph+ str "Qui "+ emph+ str "quodsi iracundia"+ list_item+ paragraph+ str "aliquando id"+```++Notice how the [line ending] in the first paragraph has+been parsed as a `softbreak`, and the asterisks in the first list item+have become an `emph`.++### An algorithm for parsing nested emphasis and links++By far the trickiest part of inline parsing is handling emphasis,+strong emphasis, links, and images. This is done using the following+algorithm.++When we're parsing inlines and we hit either++- a run of `*` or `_` characters, or+- a `[` or `.++The [delimiter stack] is a doubly linked list. Each+element contains a pointer to a text node, plus information about++- the type of delimiter (`[`, `![`, `*`, `_`)+- the number of delimiters,+- whether the delimiter is "active" (all are active to start), and+- whether the delimiter is a potential opener, a potential closer,+ or both (which depends on what sort of characters precede+ and follow the delimiters).++When we hit a `]` character, we call the *look for link or image*+procedure (see below).++When we hit the end of the input, we call the *process emphasis*+procedure (see below), with `stack_bottom` = NULL.++#### *look for link or image*++Starting at the top of the delimiter stack, we look backwards+through the stack for an opening `[` or `![` delimiter.++- If we don't find one, we return a literal text node `]`.++- If we do find one, but it's not *active*, we remove the inactive+ delimiter from the stack, and return a literal text node `]`.++- If we find one and it's active, then we parse ahead to see if+ we have an inline link/image, reference link/image, compact reference+ link/image, or shortcut reference link/image.++ + If we don't, then we remove the opening delimiter from the+ delimiter stack and return a literal text node `]`.++ + If we do, then++ * We return a link or image node whose children are the inlines+ after the text node pointed to by the opening delimiter.++ * We run *process emphasis* on these inlines, with the `[` opener+ as `stack_bottom`.++ * We remove the opening delimiter.++ * If we have a link (and not an image), we also set all+ `[` delimiters before the opening delimiter to *inactive*. (This+ will prevent us from getting links within links.)++#### *process emphasis*++Parameter `stack_bottom` sets a lower bound to how far we+descend in the [delimiter stack]. If it is NULL, we can+go all the way to the bottom. Otherwise, we stop before+visiting `stack_bottom`.++Let `current_position` point to the element on the [delimiter stack]+just above `stack_bottom` (or the first element if `stack_bottom`+is NULL).++We keep track of the `openers_bottom` for each delimiter+type (`*`, `_`) and each length of the closing delimiter run+(modulo 3). Initialize this to `stack_bottom`.++Then we repeat the following until we run out of potential+closers:++- Move `current_position` forward in the delimiter stack (if needed)+ until we find the first potential closer with delimiter `*` or `_`.+ (This will be the potential closer closest+ to the beginning of the input -- the first one in parse order.)++- Now, look back in the stack (staying above `stack_bottom` and+ the `openers_bottom` for this delimiter type) for the+ first matching potential opener ("matching" means same delimiter).++- If one is found:++ + Figure out whether we have emphasis or strong emphasis:+ if both closer and opener spans have length >= 2, we have+ strong, otherwise regular.++ + Insert an emph or strong emph node accordingly, after+ the text node corresponding to the opener.++ + Remove any delimiters between the opener and closer from+ the delimiter stack.++ + Remove 1 (for regular emph) or 2 (for strong emph) delimiters+ from the opening and closing text nodes. If they become empty+ as a result, remove them and remove the corresponding element+ of the delimiter stack. If the closing node is removed, reset+ `current_position` to the next element in the stack.++- If none is found:++ + Set `openers_bottom` to the element before `current_position`.+ (We know that there are no openers for this kind of closer up to and+ including this point, so this puts a lower bound on future searches.)++ + If the closer at `current_position` is not a potential opener,+ remove it from the delimiter stack (since we know it can't+ be a closer either).++ + Advance `current_position` to the next element in the stack.++After we're done, we remove all delimiters above `stack_bottom` from the+delimiter stack.+
+ test/test-commonmark.hs view
@@ -0,0 +1,152 @@+{-# LANGUAGE CPP #-}+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE ScopedTypeVariables #-}++import Commonmark+import Control.Monad (when)+import Data.Functor.Identity+import Data.List (groupBy)+import Data.Text (Text)+import qualified Data.Text as T+import qualified Data.Text.IO as T+import System.IO (hSetEncoding, utf8, openFile,+ IOMode(..))+import qualified Data.Text.Lazy as TL+import Test.Tasty+import Test.Tasty.HUnit+import Test.Tasty.QuickCheck+import Text.Parsec+import Text.Parsec.Pos+#if !MIN_VERSION_base(4,11,0)+import Data.Semigroup+#endif++readTextFile :: FilePath -> IO Text+readTextFile fp = do+ h <- openFile fp ReadMode+ hSetEncoding h utf8+ T.hGetContents h++main :: IO ()+main = do+ let defaultParser = runIdentity . parseCommonmarkWith defaultSyntaxSpec+ tests <- mapM (uncurry getSpecTestTree)+ [ ("test/spec.txt", mempty)+ , ("test/regression.md", mempty)+ ]+ defaultMain $ testGroup "Tests"+ (testProperty "tokenize/untokenize roundtrip" tokenize_roundtrip+ : toSpecTest defaultParser+ SpecTest+ { section = "Issue #24 (eof after HTML block)"+ , example = 1+ , markdown = "<? a ?>"+ , end_line = 1+ , start_line = 1+ , html = "<? a ?>" }+ : toSpecTest defaultParser+ SpecTest+ { section = "Issue #24 (eof after HTML block)"+ , example = 2+ , markdown = "<!-- a -->"+ , end_line = 2+ , start_line = 2+ , html = "<!-- a -->" }+ : tests)++getSpecTestTree :: FilePath+ -> SyntaxSpec Identity (Html ()) (Html ())+ -> IO TestTree+getSpecTestTree fp syntaxspec = do+ spectests <- getSpecTests fp+ let spectestgroups = groupBy (\t1 t2 -> section t1 == section t2)+ spectests+ let spectestsecs = [(section (head xs), xs) | xs <- spectestgroups]+ let parser = runIdentity . parseCommonmarkWith+ (syntaxspec <> defaultSyntaxSpec)+ return $ testGroup fp $+ map (\(secname, tests) ->+ testGroup (T.unpack secname) $+ map (toSpecTest parser) tests)+ spectestsecs++getSpecTests :: FilePath -> IO [SpecTest]+getSpecTests fp = do+ speclines <- zip [1..] . T.lines . T.replace "→" "\t"+ <$> readTextFile fp+ return $ either (error . show) id $ runParser+ (many (try (skipMany normalLine *> parseSpecTest))+ <* skipMany normalLine <* eof) ("",1) fp+ speclines++data SpecTest = SpecTest+ { section :: Text+ , example :: Int+ , markdown :: Text+ , end_line :: Int+ , start_line :: Int+ , html :: Text }+ deriving (Show)++toSpecTest :: ([Tok] -> Either ParseError (Html ()))+ -> SpecTest -> TestTree+toSpecTest parser st =+ testCase name (actual @?= expected)+ where name = T.unpack (section st) ++ " example " ++ show (example st) +++ " (" ++ show (start_line st) ++ "-" +++ show (end_line st) ++ ")"+ expected = normalizeHtml $ html st+ actual = normalizeHtml . TL.toStrict . renderHtml .+ fromRight mempty $+ (parser (tokenize "" (markdown st))+ :: Either ParseError (Html ()))++normalizeHtml :: Text -> Text+normalizeHtml = T.replace "\n</li>" "</li>" .+ T.replace "<li>\n" "<li>"++fromRight :: b -> Either a b -> b+fromRight fallback (Left _) = fallback+fromRight _ (Right x) = x++tokenize_roundtrip :: String -> Bool+tokenize_roundtrip s = untokenize (tokenize "source" t) == t+ where t = T.pack s++--- parser for spec test cases++satisfyLine :: (Text -> Bool)+ -> Parsec [(Int, Text)] (Text, Int) Text+satisfyLine f = token showTok posFromTok testTok+ where+ showTok (_,t) = T.unpack t+ posFromTok (pos,_) = newPos "" pos 1+ testTok (_,t) = if f t then Just t else Nothing++parseSpecTest :: Parsec [(Int, Text)] (Text, Int) SpecTest+parseSpecTest = do+ startpos <- getPosition+ () <$ satisfyLine (== "```````````````````````````````` example")+ markdownTxt <- T.unlines <$> manyTill (satisfyLine (const True))+ (satisfyLine (=="."))+ htmlTxt <- T.unlines <$> manyTill (satisfyLine (const True))+ (satisfyLine (== "````````````````````````````````"))+ endline <- (\x -> x - 1) . sourceLine <$> getPosition+ (sectionName, exampleNumber) <- getState+ putState (sectionName, exampleNumber + 1)+ return SpecTest{+ section = sectionName+ , example = exampleNumber+ , markdown = markdownTxt+ , end_line = endline+ , start_line = sourceLine startpos+ , html = htmlTxt+ }++normalLine :: Parsec [(Int, Text)] (Text, Int) ()+normalLine = do+ t <- satisfyLine (/= "```````````````````````````````` example")+ when ("#" `T.isPrefixOf` t) $ updateState $ \(_secname, exampnum) ->+ (T.strip $ T.dropWhile (=='#') t, exampnum)++---