xyz
+xyz
+xyz
+xyz
+hi
hi
hi
bar
+```````````````````````````````` + +```````````````````````````````` example +{#foo .special} +# Hi +. +hi
+```````````````````````````````` + +**Consecutive attribute specifiers may be used, +either for blocks or for inlines.** + +```````````````````````````````` example +*hi*{.underline}{#foo} +. +hi
+```````````````````````````````` + +```````````````````````````````` example +{.special} +{#foo} +* * * * +. +Hi
+```````````````````````````````` + +Numerical suffixes will be added to avoid +duplicate identifiers: + +```````````````````````````````` example +# Hi +# Hi +# Hi +. +Visit www.commonmark.org/help for more information.
+```````````````````````````````` + +We then apply [extended autolink path validation](@) as follows: + +Trailing punctuation (specifically, `?`, `!`, `.`, `,`, `:`, `*`, `_`, and `~`) +will not be considered part of the autolink, though they may be included in the +interior of the link: + +```````````````````````````````` example +Visit www.commonmark.org. + +Visit www.commonmark.org/a.b. +. +Visit www.commonmark.org.
+Visit www.commonmark.org/a.b.
+```````````````````````````````` + +When an autolink ends in `)`, we scan the entire autolink for the total number +of parentheses. If there is a greater number of closing parentheses than +opening ones, we don't consider the last character part of the autolink, in +order to facilitate including an autolink inside a parenthesis: + +```````````````````````````````` example +www.google.com/search?q=Markup+(business) + +(www.google.com/search?q=Markup+(business)) +. +www.google.com/search?q=Markup+(business)
+(www.google.com/search?q=Markup+(business))
+```````````````````````````````` + +This check is only done when the link ends in a closing parentheses `)`, so if +the only parentheses are in the interior of the autolink, no special rules are +applied: + +```````````````````````````````` example +www.google.com/search?q=(business))+ok +. +www.google.com/search?q=(business))+ok
+```````````````````````````````` + +If an autolink ends in a semicolon (`;`), we check to see if it appears to +resemble an [entity reference][entity references]; if the preceding text is `&` +followed by one or more alphanumeric characters. If so, it is excluded from +the autolink: + +```````````````````````````````` example +www.google.com/search?q=commonmark&hl=en + +www.google.com/search?q=commonmark&hl; +. +www.google.com/search?q=commonmark&hl=en
+www.google.com/search?q=commonmark&hl;
+```````````````````````````````` + +`<` immediately ends an autolink. + +```````````````````````````````` example +www.commonmark.org/he(Visit https://encrypted.google.com/search?q=Markup+(business))
+Anonymous FTP is available at ftp://foo.bar.baz.
+```````````````````````````````` + + +An [extended email autolink](@) will be recognised when an email address is +recognised within any text node. Email addresses are recognised according to +the following rules: + +* One ore more characters which are alphanumeric, or `.`, `-`, `_`, or `+`. +* An `@` symbol. +* One or more characters which are alphanumeric, or `.`, `-`, or `_`. At least + one of the characters here must be a period (`.`). The last character must + not be one of `-` or `_`. If the last character is a period (`.`), it will + be excluded from the autolink. + +The scheme `mailto:` will automatically be added to the generated link: + +```````````````````````````````` example +foo@bar.baz +. + +```````````````````````````````` + +`+` can occur before the `@`, but not after. + +```````````````````````````````` example +hello@mail+xyz.example isn't valid, but hello+xyz@mail.example is. +. +hello@mail+xyz.example isn't valid, but hello+xyz@mail.example is.
+```````````````````````````````` + +`.`, `-`, and `_` can occur on both sides of the `@`, but only `.` may occur at +the end of the email address, in which case it will not be considered part of +the address: + +```````````````````````````````` example +a.b-c_d@a.b + +a.b-c_d@a.b. + +a.b-c_d@a.b- + +a.b-c_d@a.b_ +. + + +a.b-c_d@a.b-
+a.b-c_d@a.b_
+```````````````````````````````` diff --git a/test/bracketed_spans.md b/test/bracketed_spans.md new file mode 100644 --- /dev/null +++ b/test/bracketed_spans.md @@ -0,0 +1,25 @@ +Bracketed spans work like links with attributes, but +without the link destination. + +```````````````````````````````` example +[foo]{#ident .class key="value value" key2=value2} +. +foo
+```````````````````````````````` + +```````````````````````````````` example +[foo +*bar*]{#ident +.class} +. +foo +bar
+```````````````````````````````` + +An attribute is required: + +```````````````````````````````` example +[foo] +. +[foo]
+```````````````````````````````` diff --git a/test/definition_lists.md b/test/definition_lists.md new file mode 100644 --- /dev/null +++ b/test/definition_lists.md @@ -0,0 +1,283 @@ +## Definition lists + +The term is given on a line by itself, followed by +one or more definitions. Each definition must begin +with `:` (after 0-2 spaces); subsequent lines must +be indented unless they are lazy paragraph +continuations. + +The list is tight if there is no blank line between +the term and the first definition, otherwise loose. + +```````````````````````````````` example +apple +: red fruit + +orange +: orange fruit +. +red fruit
+orange fruit
+red fruit
+orange fruit
+red fruit
+contains seeds, +crisp, pleasant to taste
+orange fruit
+{ orange code block }
+++orange block quote
+
red fruit
+computer company
+orange fruit
+telecom company
+red fruit
+computer +company
+orange +fruit
+telecom company
+foo
+Foo
+😊 +A💔 +😺
+```````````````````````````````` + +```````````````````````````````` example +:nothing_really: +:broken_heart +. +:nothing_really: +:broken_heart
+```````````````````````````````` diff --git a/test/fancy_lists.md b/test/fancy_lists.md new file mode 100644 --- /dev/null +++ b/test/fancy_lists.md @@ -0,0 +1,223 @@ +The `fancy_lists` extension allows various styles of ordered lists: + +With period: + +```````````````````````````````` example +1. decimal +2. decimal +. +B. Russell
+I. J. Good
+```````````````````````````````` + +A new list starts with any style change: + +```````````````````````````````` example +1. one +2) one +. +Hi
+++A block quote.
+
Hi
+++A block quote.
+
Hi
+++A block quote.
+
Paragraph text
+Hi
+::: +Hi +:::
+```````````````````````````````` + +The closing fence must be at leats as long as the opening +fence. + +```````````````````````````````` example +::::: {.foo} +Hi +::: +:::::: +. +Hi +:::
+++```````````````````````````````` + diff --git a/test/footnotes.md b/test/footnotes.md new file mode 100644 --- /dev/null +++ b/test/footnotes.md @@ -0,0 +1,90 @@ +## Footnotes + +TODO: proper spec. +This test is based on the one from gfm. + +```````````````````````````````` example +This is some text![^1]. Other text.[^footnote]. + +Here's a thing[^other-note]. + +And another thing[^codeblock-note]. + +This doesn't have a referent[^nope]. + + +[^1]: Some *bolded* footnote definition. + +[^footnote]: + > Blockquotes can be in a footnote. + + as well as code blocks + + or, naturally, simple paragraphs. + +[^other-note]: no code block here (spaces are stripped away) + +[^codeblock-note]: + this is now a code block (8 spaces indentation) + +Hi! + +[^unused]: This is unused. +. +++Hi
+
This is some text!1. Other text.2.
+Here's a thing3.
+And another thing4.
+This doesn't have a referent[^nope].
+Hi!
+Some bolded footnote definition.
+++Blockquotes can be in a footnote.
+
as well as code blocks
+or, naturally, simple paragraphs.
+ no code block here (spaces are stripped away)
+this is now a code block (8 spaces indentation)
+This is unused.
+Hello
+there
Hello
+there
See the Heading above.
+```````````````````````````````` + +```````````````````````````````` example +{#foo} +# Heading + +See the [Heading] above. +. +See the Heading above.
+```````````````````````````````` + +Explicitly defined references take precedence: + +```````````````````````````````` example +# Heading + +See the [Heading] above. + +[Heading]: foo +. +See the Heading above.
+```````````````````````````````` + +When there are two headings with the same text, +the first takes precedence: + +```````````````````````````````` example +# Heading + +# Heading + +See the [Heading] above. +. +See the Heading above.
+```````````````````````````````` + +Empty headings don't create implicit references: + +```````````````````````````````` example +# + +## + +See [] and [ ]. +. + + +See [] and [ ].
+```````````````````````````````` diff --git a/test/math.md b/test/math.md new file mode 100644 --- /dev/null +++ b/test/math.md @@ -0,0 +1,55 @@ +# TeX Math + +Inline math goes between `$` characters, and display math +goes between `$$`: + +```````````````````````````````` example +Let $x$ and $y$ be integers such that +$$x=y + 2$$ +. +Let \(x\) and \(y\) be integers such that +\[x=y + 2\]
+```````````````````````````````` + +In inline math, the opening `$` must not be followed by +a whitespace, and the closing `$` must not be +preceeded by whitespace. + +```````````````````````````````` example +This is not math: 2000$. +And neither is this $ 4 $. +. +This is not math: 2000$. +And neither is this $ 4 $.
+```````````````````````````````` + +Display math delimiters can be surrounded by whitespace: + +```````````````````````````````` example +This is display math: +$$ +e=mc^2 +$$ +. +This is display math: +\[ +e=mc^2 +\]
+```````````````````````````````` + +To avoid treating currency signs as math delimiters, +one may occasionally have to backslash-escape them: + +```````````````````````````````` example +The cost is between \$10 and 30$. +. +The cost is between $10 and 30$.
+```````````````````````````````` + +Everthing inside the math construction is treated +as math, and not given its normal commonmark meaning. +```````````````````````````````` example +$bc$ +. +\(b<a>c\)
+```````````````````````````````` diff --git a/test/pipe_tables.md b/test/pipe_tables.md new file mode 100644 --- /dev/null +++ b/test/pipe_tables.md @@ -0,0 +1,266 @@ +## Tables (extension) + +GFM enables the `table` extension, where an additional leaf block type is +available. + +A [table](@) is an arrangement of data with rows and columns, consisting of a +single header row, a [delimiter row] separating the header from the data, and +zero or more data rows. + +Each row consists of cells containing arbitrary text, in which [inlines] are +parsed, separated by pipes (`|`). A leading and trailing pipe is also +recommended for clarity of reading, and if there's otherwise parsing ambiguity. +Spaces between pipes and cell content are trimmed. Block-level elements cannot +be inserted in a table. + +The [delimiter row](@) consists of cells whose only content are hyphens (`-`), +and optionally, a leading or trailing colon (`:`), or both, to indicate left, +right, or center alignment respectively. + +```````````````````````````````` example +| foo | bar | +| --- | --- | +| baz | bim | +. +| foo | +bar | +
|---|---|
| baz | +bim | +
| abc | +defghi | +
|---|---|
| bar | +baz | +
| f|oo | +
|---|
b | az |
+
| b | im | +
| abc | +def | +
|---|---|
| bar | +baz | +
++```````````````````````````````` + +```````````````````````````````` example +| abc | def | +| --- | --- | +| bar | baz | + +bar +. +bar
+
| abc | +def | +
|---|---|
| bar | +baz | +
bar
+```````````````````````````````` + +```````````````````````````````` example +| abc | def | +| --- | --- | +| bar | baz | +bar +. +| abc | +def | +
|---|---|
| bar | +baz | +
bar
+```````````````````````````````` + +The header row must match the [delimiter row] in the number of cells. If not, +a table will not be recognized: + +```````````````````````````````` example +| abc | def | +| --- | +| bar | +. +| abc | def | +| --- | +| bar |
+```````````````````````````````` + +The remainder of the table's rows may vary in the number of cells. If there +are a number of cells fewer than the number of cells in the header row, empty +cells are inserted. If there are greater, the excess is ignored: + +```````````````````````````````` example +| abc | def | +| --- | --- | +| bar | +| bar | baz | boo | +. +| abc | +def | +
|---|---|
| bar | ++ |
| bar | +baz | +
| abc | +def | +
|---|
| Not enough table | to be considered table |
+| Not enough table | to be considered table | +| Not enough table | to be considered table |
+| ---- | --- |
+```````````````````````````````` + +A table may be indented up to three spaces: + +```````````````````````````````` example + a | b | c + - | - | - +. +| a | +b | +c | +
|---|
a | b | c
+- | - | -
+foo
+```````````````````````````````` + +You can't mix regular and raw attributes: + +```````````````````````````````` example +``` {=html #id} +foo +. +<b>foo</b>
+<b>foo</b>{=html .bar}
“Hello,” said the spider. +“‘Shelob’ is my name.”
+```````````````````````````````` + +```````````````````````````````` example +'A', 'B', and 'C' are letters. +. +‘A’, ‘B’, and ‘C’ are letters.
+```````````````````````````````` + +```````````````````````````````` example +'Oak,' 'elm,' and 'beech' are names of trees. +So is 'pine.' +. +‘Oak,’ ‘elm,’ and ‘beech’ are names of trees. +So is ‘pine.’
+```````````````````````````````` + +```````````````````````````````` example +'He said, "I want to go."' +. +‘He said, “I want to go.”’
+```````````````````````````````` + +A single quote that isn't an open quote matched +with a close quote will be treated as an +apostrophe: + +```````````````````````````````` example +Were you alive in the 70's? +. +Were you alive in the 70’s?
+```````````````````````````````` + +```````````````````````````````` example +Here is some quoted '`code`' and a "[quoted link](url)". +. +Here is some quoted ‘code’ and a “quoted link”.
’tis the season to be ‘jolly’
+```````````````````````````````` + +Multiple apostrophes should not be marked as open/closing quotes. + +```````````````````````````````` example +'We'll use Jane's boat and John's truck,' Jenna said. +. +‘We’ll use Jane’s boat and John’s truck,’ Jenna said.
+```````````````````````````````` + +An unmatched double quote will be interpreted as a +left double quote, to facilitate this style: + +```````````````````````````````` example +"A paragraph with no closing quote. + +"Second paragraph by same speaker, in fiction." +. +“A paragraph with no closing quote.
+“Second paragraph by same speaker, in fiction.”
+```````````````````````````````` + +A quote following a `]` or `)` character cannot +be an open quote: + +```````````````````````````````` example +[a]'s b' +. +[a]’s b’
+```````````````````````````````` + +Quotes that are escaped come out as literal straight +quotes: + +```````````````````````````````` example +\"This is not smart.\" +This isn\'t either. +5\'8\" +. +"This is not smart." +This isn't either. +5'8"
+```````````````````````````````` + +Doubled quotes are treated as nested: + +```````````````````````````````` example +''hi'' +. +‘‘hi’’
+```````````````````````````````` + +Two hyphens form an en-dash, three an em-dash. + +```````````````````````````````` example +Some dashes: em---em +en--en +em --- em +en -- en +2--3 +. +Some dashes: em—em +en–en +em — em +en – en +2–3
+```````````````````````````````` + +A sequence of more than three hyphens is +parsed as a sequence of em and/or en dashes, +with no hyphens. If possible, a homogeneous +sequence of dashes is used (so, 10 hyphens += 5 en dashes, and 9 hyphens = 3 em dashes). +When a heterogeneous sequence must be used, +the em dashes come first, followed by the en +dashes, and as few en dashes as possible are +used (so, 7 hyphens = 2 em dashes an 1 en +dash). + +```````````````````````````````` example +one- +two-- +three--- +four---- +five----- +six------ +seven------- +eight-------- +nine--------- +thirteen-------------. +. +one- +two– +three— +four–– +five—– +six—— +seven—–– +eight–––– +nine——— +thirteen———––.
+```````````````````````````````` + +Hyphens can be escaped: + +```````````````````````````````` example +Escaped hyphens: \-- \-\-\-. +. +Escaped hyphens: -- ---.
+```````````````````````````````` + +Three periods form an ellipsis: + +```````````````````````````````` example +Ellipses...and...and.... +. +Ellipses…and…and….
+```````````````````````````````` + +Periods can be escaped if ellipsis-formation +is not wanted: + +```````````````````````````````` example +No ellipses\.\.\. +. +No ellipses...
+```````````````````````````````` diff --git a/test/strikethrough.md b/test/strikethrough.md new file mode 100644 --- /dev/null +++ b/test/strikethrough.md @@ -0,0 +1,50 @@ +## Strikethrough + +Basic strikethrough is between two tildes: + +```````````````````````````````` example +~~This is *stricken out*~~ +. +This is stricken out
~This is nothing~
+```````````````````````````````` + +Backslash escapes: + +```````````````````````````````` example +~~This is \~\~stricken~~ +. +This is ~~stricken
Thisisstricken
Thisisstricken
Here I strike out an exclamation point!.
E=mc2.
+```````````````````````````````` + +Backslash escapes: + +```````````````````````````````` example +E=mc\~2\~. +. +E=mc~2~.
+```````````````````````````````` + +Spaces and formatting are allowed: + +```````````````````````````````` example +E=mc~2 or *3*~. +. +E=mc2 or 3.
+```````````````````````````````` + +Punctuation is ignored for purposes of determining +flankingness: + +```````````````````````````````` example +E=mc~!~. +. +E=mc!.
+```````````````````````````````` + +Subscript can't begin or end with a space: + +```````````````````````````````` example +E=mc~ 2~. + +E=mc~2 ~. + +~ ~ +. +E=mc~ 2~.
+E=mc~2 ~.
+~ ~
+```````````````````````````````` + diff --git a/test/superscript.md b/test/superscript.md new file mode 100644 --- /dev/null +++ b/test/superscript.md @@ -0,0 +1,48 @@ +## Superscript + +Basic superscript is between `^`s: + +```````````````````````````````` example +E=mc^2^. +. +E=mc2.
+```````````````````````````````` + +Backslash escapes: + +```````````````````````````````` example +E=mc\^2\^. +. +E=mc^2^.
+```````````````````````````````` + +Spaces and formatting are allowed: + +```````````````````````````````` example +E=mc^2 or *3*^. +. +E=mc2 or 3.
+```````````````````````````````` + +Punctuation is ignored for purposes of determining +flankingness: + +```````````````````````````````` example +E=mc^!^. +. +E=mc!.
+```````````````````````````````` + +Superscript can't begin or end with a space: + +```````````````````````````````` example +E=mc^ 2^. + +E=mc^2 ^. + +^ ^ +. +E=mc^ 2^.
+E=mc^2 ^.
+^ ^
+```````````````````````````````` diff --git a/test/test-commonmark-extensions.hs b/test/test-commonmark-extensions.hs new file mode 100644 --- /dev/null +++ b/test/test-commonmark-extensions.hs @@ -0,0 +1,147 @@ +{-# LANGUAGE CPP #-} +{-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE ScopedTypeVariables #-} + +import Commonmark +import Commonmark.Extensions +import Control.Monad (when) +import Data.Functor.Identity +import Data.List (groupBy) +import Data.Text (Text) +import qualified Data.Text as T +import qualified Data.Text.IO as T +import System.IO (hSetEncoding, utf8, openFile, + IOMode(..)) +import qualified Data.Text.Lazy as TL +import Test.Tasty +import Test.Tasty.HUnit +import Text.Parsec +import Text.Parsec.Pos +#if !MIN_VERSION_base(4,11,0) +import Data.Semigroup +#endif + +readTextFile :: FilePath -> IO Text +readTextFile fp = do + h <- openFile fp ReadMode + hSetEncoding h utf8 + T.hGetContents h + +main :: IO () +main = do + tests <- mapM (uncurry getSpecTestTree) + [ ("test/smart.md", smartPunctuationSpec) + , ("test/hard_line_breaks.md", hardLineBreaksSpec) + , ("test/strikethrough.md", strikethroughSpec) + , ("test/superscript.md", superscriptSpec) + , ("test/subscript.md", subscriptSpec) + , ("test/pipe_tables.md", pipeTableSpec) + , ("test/footnotes.md", footnoteSpec) + , ("test/math.md", mathSpec) + , ("test/emoji.md", emojiSpec) + , ("test/autolinks.md", autolinkSpec) + , ("test/definition_lists.md", definitionListSpec) + , ("test/fancy_lists.md", fancyListSpec) + , ("test/task_lists.md", taskListSpec) + , ("test/attributes.md", attributesSpec) + , ("test/raw_attribute.md", rawAttributeSpec) + , ("test/bracketed_spans.md", bracketedSpanSpec) + , ("test/fenced_divs.md", fencedDivSpec) + , ("test/auto_identifiers.md", autoIdentifiersSpec <> attributesSpec) + , ("test/implicit_heading_references.md", + autoIdentifiersSpec <> attributesSpec <> implicitHeadingReferencesSpec) + ] + defaultMain $ testGroup "Tests" tests + +getSpecTestTree :: FilePath + -> SyntaxSpec Identity (Html ()) (Html ()) + -> IO TestTree +getSpecTestTree fp syntaxspec = do + spectests <- getSpecTests fp + let spectestgroups = groupBy (\t1 t2 -> section t1 == section t2) + spectests + let spectestsecs = [(section (head xs), xs) | xs <- spectestgroups] + let parser = runIdentity . parseCommonmarkWith + (syntaxspec <> defaultSyntaxSpec) + return $ testGroup fp $ + map (\(secname, tests) -> + testGroup (T.unpack secname) $ + map (toSpecTest parser) tests) + spectestsecs + +getSpecTests :: FilePath -> IO [SpecTest] +getSpecTests fp = do + speclines <- zip [1..] . T.lines . T.replace "→" "\t" + <$> readTextFile fp + return $ either (error . show) id $ runParser + (many (try (skipMany normalLine *> parseSpecTest)) + <* skipMany normalLine <* eof) ("",1) fp + speclines + +data SpecTest = SpecTest + { section :: Text + , example :: Int + , markdown :: Text + , end_line :: Int + , start_line :: Int + , html :: Text } + deriving (Show) + +toSpecTest :: ([Tok] -> Either ParseError (Html ())) + -> SpecTest -> TestTree +toSpecTest parser st = + testCase name (actual @?= expected) + where name = T.unpack (section st) ++ " example " ++ show (example st) ++ + " (" ++ show (start_line st) ++ "-" ++ + show (end_line st) ++ ")" + expected = normalizeHtml $ html st + actual = normalizeHtml . TL.toStrict . renderHtml . + fromRight mempty $ + (parser (tokenize "" (markdown st)) + :: Either ParseError (Html ())) + +normalizeHtml :: Text -> Text +normalizeHtml = T.replace "\n" "" . + T.replace "