cmark 0.1.0.1 → 0.2
raw patch · 7 files changed
+1187/−101 lines, 7 filesdep +blaze-htmldep +cheapskatedep +cmarkdep ~basedep ~textPVP ok
version bump matches the API change (PVP)
Dependencies added: blaze-html, cheapskate, cmark, criterion, discount, markdown, pandoc, sundown
Dependency ranges changed: base, text
API changes (from Hackage documentation)
- CMark: markdownToHtml :: [CMarkOption] -> Text -> Text
- CMark: parseDocument :: [CMarkOption] -> Text -> Node
+ CMark: commonmarkToHtml :: [CMarkOption] -> Text -> Text
+ CMark: commonmarkToMan :: [CMarkOption] -> Text -> Text
+ CMark: commonmarkToNode :: [CMarkOption] -> Text -> Node
+ CMark: commonmarkToXml :: [CMarkOption] -> Text -> Text
Files
- CMark.hsc +37/−11
- README.md +17/−35
- bench/Bench.hs +40/−0
- bench/full-sample.md +726/−0
- bench/sample.md +284/−0
- cmark.cabal +83/−54
- cmark/cmark.h +0/−1
CMark.hsc view
@@ -2,7 +2,15 @@ DeriveGeneric, DeriveDataTypeable, FlexibleContexts #-} module CMark (- Node(..)+ commonmarkToHtml+ , commonmarkToXml+ , commonmarkToMan+ , commonmarkToNode+ , optSourcePos+ , optNormalize+ , optHardBreaks+ , optSmart+ , Node(..) , NodeType(..) , PosInfo(..) , DelimType(..)@@ -13,12 +21,6 @@ , Level , Info , CMarkOption- , markdownToHtml- , parseDocument- , optSourcePos- , optNormalize- , optHardBreaks- , optSmart ) where import Foreign@@ -197,6 +199,12 @@ foreign import ccall "cmark.h cmark_markdown_to_html" c_cmark_markdown_to_html :: CString -> Int -> CInt -> CString +foreign import ccall "cmark.h cmark_render_xml"+ c_cmark_render_xml :: NodePtr -> CInt -> CString++foreign import ccall "cmark.h cmark_render_man"+ c_cmark_render_man :: NodePtr -> CInt -> CString+ foreign import ccall "cmark.h cmark_parse_document" c_cmark_parse_document :: CString -> Int -> CInt -> NodePtr @@ -247,15 +255,33 @@ -- | Convert CommonMark formatted text to Html, using cmark's -- built-in renderer.-markdownToHtml :: [CMarkOption] -> Text -> Text-markdownToHtml opts s = io $+commonmarkToHtml :: [CMarkOption] -> Text -> Text+commonmarkToHtml opts s = io $ TF.withCStringLen s $ \(ptr, len) -> return (peekCString $ c_cmark_markdown_to_html ptr len (combineOptions opts)) +-- | Convert CommonMark formatted text to CommonMark XML, using cmark's+-- built-in renderer.+commonmarkToXml :: [CMarkOption] -> Text -> Text+commonmarkToXml opts s = io $+ TF.withCStringLen s $ \(ptr, len) -> do+ let opts' = combineOptions opts+ let doc = c_cmark_parse_document ptr len opts'+ return (peekCString $ c_cmark_render_xml doc opts')++-- | Convert CommonMark formatted text to groff man, using cmark's+-- built-in renderer.+commonmarkToMan :: [CMarkOption] -> Text -> Text+commonmarkToMan opts s = io $+ TF.withCStringLen s $ \(ptr, len) -> do+ let opts' = combineOptions opts+ let doc = c_cmark_parse_document ptr len opts'+ return (peekCString $ c_cmark_render_man doc opts')+ -- | Convert CommonMark formatted text to a structured 'Node' tree, -- which can be transformed or rendered using Haskell code.-parseDocument :: [CMarkOption] -> Text -> Node-parseDocument opts s = io $+commonmarkToNode :: [CMarkOption] -> Text -> Node+commonmarkToNode opts s = io $ TF.withCStringLen s $ \(ptr, len) -> return $ toNode $ c_cmark_parse_document ptr len (combineOptions opts)
README.md view
@@ -2,16 +2,19 @@ ======== This package provides Haskell bindings for [libcmark], the reference-parser for [CommonMark]. It includes sources for [libcmark], and-does not require prior installation of the C library.+parser for [CommonMark], a fully specified variant of Markdown.+It includes sources for [libcmark] (version 0.18.1) and does not+require prior installation of the C library. cmark provides the following advantages over existing Markdown libraries for Haskell: - - **Speed:** Conversion speed is on par with the [sundown] library:- about 30 times faster than [pandoc], 24 times- faster than the Haskell [markdown] package, 8 times faster than- [cheapskate].+ - **Speed:** Conversion speed is on par with the [sundown] library.+ We were unable to measure precisely against [sundown], which+ raised a malloc error when compiled into our benchmark suite.+ Relative to other implementations: cmark was 82 times faster than+ [cheapskate], 59 times faster than [markdown], 105 times faster+ than [pandoc], and 2.8 times faster than [discount]. - **Memory footprint:** Memory footprint is on par with [sundown]. On one sample, the library uses a fourth the memory that [markdown]@@ -19,12 +22,8 @@ - **Robustness:** cmark can handle whatever is thrown at it, without the exponential blowups in parsing time one can sometimes- get with other libraries. For example, the input produced by- this command will tie [markdown] and [pandoc] in knots:-- python -c "print ((500 * '[') + 'hi' + (500 * ']') + '(url)')"-- cmark handles it easily, with no slowdown.+ get with other libraries. (The input `bench/full-sample.md`,+ for example, causes both [pandoc] and [markdown] to grind to a halt.) - **Accuracy:** cmark passes the CommonMark spec's suite of over 500 conformance tests.@@ -35,32 +34,14 @@ example, one could use this library for server-side rendering and [commonmark.js] for client-side previewing. - - **Ease of installation:** cmark has minimal dependencies.+ - **Ease of installation:** cmark is portable and has minimal+ dependencies. cmark does not provide Haskell versions of the whole [libcmark] API, which is built around mutable `cmark_node` objects. Instead, it-provides two functions:-- - `markdownToHtml` uses [libcmark]'s parser and renderer for a- maximally efficient conversion of CommonMark to HTML (as a Text).- ("Smart punctuation" and a few other options can be enabled.)-- ``` haskell- Prelude CMark Data.Text> markdownToHtml [optSmart] (pack "dog's *breakfast*")- "<p>dog\8217s <em>breakfast</em></p>\n"- ```-- - `parseDocument` uses [libcmark]'s parser to produce a `Node` tree- that can be processed further using Haskell. One can transform- the tree using generics, convert it to another kind of- structure (such as a Pandoc object that can be rendered using- pandoc's renderers) or render it using a custom rendering- function.-- ``` haskell- Prelude CMark Data.Text> parseDocument [optSmart] (pack "dog's *breakfast*")- Node (Just (PosInfo {startLine = 1, startColumn = 1, endLine = 1, endColumn = 17})) DOCUMENT [Node (Just (PosInfo {startLine = 1, startColumn = 1, endLine = 1, endColumn = 17})) PARAGRAPH [Node Nothing (TEXT "dog") [],Node Nothing (TEXT "\8217") [],Node Nothing (TEXT "s ") [],Node Nothing EMPH [Node Nothing (TEXT "breakfast") []]]]- ```+provides functions for converting CommonMark to HTML (and other+formats), and a function for converting CommonMark to a `Node`+tree that can be processed further using Haskell. **A note on security:** This library does not attempt to sanitize HTML output. We recommend using [xss-sanitize] to filter the output.@@ -77,3 +58,4 @@ [markdown]: https://hackage.haskell.org/package/markdown [commonmark.js]: http://github.com/jgm/commonmark.js [xss-sanitize]: https://hackage.haskell.org/package/xss-sanitize+[discount]: https://hackage.haskell.org/package/discount
+ bench/Bench.hs view
@@ -0,0 +1,40 @@+import qualified Text.Pandoc as Pandoc+import qualified Cheapskate as Cheapskate+import qualified Cheapskate.Html as CheapskateHtml+import qualified CMark as CMark+import qualified Text.Sundown.Html.Text as Sundown+import qualified Text.Discount as Discount+import qualified Text.Blaze.Html.Renderer.Text as Blaze+import qualified Text.Markdown as Markdown+import Data.Text (Text)+import Data.Text as T+import Data.Text.Lazy (fromChunks, toChunks)+import Data.Text.IO as T+import Criterion.Main+import Criterion.Config+import System.Environment (getArgs)++main :: IO ()+main = do+ args <- getArgs+ (conf,_) <- parseArgs defaultConfig defaultOptions args+ sample <- T.readFile "bench/sample.md"+ defaultMainWith conf (return ()) [+ mkBench "cmark" (CMark.commonmarkToHtml []) sample+ , mkBench "cheapskate" (T.concat . toChunks . Blaze.renderHtml . CheapskateHtml.renderDoc . Cheapskate.markdown Cheapskate.def) sample+ , mkBench "discount" (Discount.parseMarkdownUtf8 []) sample+ , mkBench "pandoc" (T.concat . toChunks . Blaze.renderHtml . Pandoc.writeHtml Pandoc.def . Pandoc.readMarkdown Pandoc.def . T.unpack) sample+ , mkBench "markdown" (T.concat . toChunks . Blaze.renderHtml . Markdown.markdown Markdown.def . fromChunks . (:[])) sample+ ]++-- Note: when full-sample.md rather than sample.md is used, pandoc and markdown+-- hang (> 1 minute).++-- even with sample.md, sundown gives this error+-- , mkBench "sundown" (Sundown.renderHtml Sundown.noExtensions Sundown.noHtmlModes False Nothing) sample+-- bench-cmark(50437,0x7fff7bfbe310) malloc: *** error for object 0x7ffde3d00928: incorrect checksum for freed object - object was probably modified after being freed.+-- *** set a breakpoint in malloc_error_break to debug++mkBench :: String -> (Text -> Text) -> Text -> Benchmark+mkBench name converter inp =+ bench name $ nf converter inp
+ bench/full-sample.md view
@@ -0,0 +1,726 @@+CommonMark+==========++CommonMark is a rationalized version of Markdown syntax,+with a [spec][the spec] and BSD3-licensed reference+implementations in C and JavaScript.++[Try it now!](http://spec.commonmark.org/dingus.html)++The implementations+-------------------++The C implementation provides both a shared library (`libcmark`) and a+standalone program `cmark` that converts CommonMark to HTML. It is+written in standard C99 and has no library dependencies. The parser is+very fast (see [benchmarks](benchmarks.md)).++It is easy to use `libcmark` in python, lua, ruby, and other dynamic+languages: see the `wrappers/` subdirectory for some simple examples.++The JavaScript implementation provides both an NPM package and a+single JavaScript file, with no dependencies, that can be linked into+an HTML page. For further information, see the+[README in the js directory](js/README.md).++**A note on security:**+Neither implementation attempts to sanitize link attributes or+raw HTML. If you use these libraries in applications that accept+untrusted user input, you must run the output through an HTML+sanitizer to protect against+[XSS attacks](http://en.wikipedia.org/wiki/Cross-site_scripting).++Installing (C)+--------------++Building the C program (`cmark`) and shared library (`libcmark`)+requires [cmake]. If you modify `scanners.re`, then you will also+need [re2c], which is used to generate `scanners.c` from+`scanners.re`. We have included a pre-generated `scanners.c` in+the repository to reduce build dependencies.++If you have GNU make, you can simply `make`, `make test`, and `make+install`. This calls [cmake] to create a `Makefile` in the `build`+directory, then uses that `Makefile` to create the executable and+library. The binaries can be found in `build/src`.++For a more portable method, you can use [cmake] manually. [cmake] knows+how to create build environments for many build systems. For example,+on FreeBSD:++ mkdir build+ cd build+ cmake .. # optionally: -DCMAKE_INSTALL_PREFIX=path+ make # executable will be created as build/src/cmark+ make test+ make install++Or, to create Xcode project files on OSX:++ mkdir build+ cd build+ cmake -G Xcode ..+ make+ make test+ make install++The GNU Makefile also provides a few other targets for developers.+To run a benchmark:++ make bench++To run a "fuzz test" against ten long randomly generated inputs:++ make fuzztest++To run a test for memory leaks using `valgrind`:++ make leakcheck++To reformat source code using `astyle`:++ make astyle++To make a release tarball and zip archive:++ make archive+++Compiling for Windows+---------------------++To compile with MSVC and NMAKE:++ nmake++You can cross-compile a Windows binary and dll on linux if you have the+`mingw32` compiler:++ make mingw++The binaries will be in `build-mingw/windows/bin`.++Installing (JavaScript)+-----------------------++The JavaScript library can be installed through `npm`:++ npm install commonmark++This includes a command-line converter called `commonmark`.++If you want to use it in a client application, you can fetch+a pre-built copy of `commonmark.js` from+<http://spec.commonmark.org/js/commonmark.js>.++For further information, see the+[README in the js directory](js/README.md).++The spec+--------++[The spec] contains over 500 embedded examples which serve as conformance+tests. To run the tests using an executable `$PROG`:++ python3 test/spec_tests.py --program $PROG++If you want to extract the raw test data from the spec without+actually running the tests, you can do:++ python3 test/spec_tests.py --dump-tests++and you'll get all the tests in JSON format.++[The spec]: http://spec.commonmark.org/0.13/++The source of [the spec] is `spec.txt`. This is basically a Markdown+file, with code examples written in a shorthand form:++ .+ Markdown source+ .+ expected HTML output+ .++To build an HTML version of the spec, do `make spec.html`. To build a+PDF version, do `make spec.pdf`. (Creating a PDF requires [pandoc]+and a LaTeX installation. Creating the HTML version requires only+`libcmark` and `python3`.)++The spec is written from the point of view of the human writer, not+the computer reader. It is not an algorithm---an English translation of+a computer program---but a declarative description of what counts as a block+quote, a code block, and each of the other structural elements that can+make up a Markdown document.++Because John Gruber's [canonical syntax+description](http://daringfireball.net/projects/markdown/syntax) leaves+many aspects of the syntax undetermined, writing a precise spec requires+making a large number of decisions, many of them somewhat arbitrary.+In making them, we have appealed to existing conventions and+considerations of simplicity, readability, expressive power, and+consistency. We have tried to ensure that "normal" documents in the many+incompatible existing implementations of Markdown will render, as far as+possible, as their authors intended. And we have tried to make the rules+for different elements work together harmoniously. In places where+different decisions could have been made (for example, the rules+governing list indentation), we have explained the rationale for+my choices. In a few cases, we have departed slightly from the canonical+syntax description, in ways that we think further the goals of Markdown+as stated in that description.++For the most part, we have limited ourselves to the basic elements+described in Gruber's canonical syntax description, eschewing extensions+like footnotes and definition lists. It is important to get the core+right before considering such things. However, we have included a visible+syntax for line breaks and fenced code blocks.++Differences from original Markdown+----------------------------------++There are only a few places where this spec says things that contradict+the canonical syntax description:++- It allows all punctuation symbols to be backslash-escaped,+ not just the symbols with special meanings in Markdown. We found+ that it was just too hard to remember which symbols could be+ escaped.++- It introduces an alternative syntax for hard line+ breaks, a backslash at the end of the line, supplementing the+ two-spaces-at-the-end-of-line rule. This is motivated by persistent+ complaints about the “invisible” nature of the two-space rule.++- Link syntax has been made a bit more predictable (in a+ backwards-compatible way). For example, `Markdown.pl` allows single+ quotes around a title in inline links, but not in reference links.+ This kind of difference is really hard for users to remember, so the+ spec allows single quotes in both contexts.++- The rule for HTML blocks differs, though in most real cases it+ shouldn't make a difference. (See the section on HTML Blocks+ for details.) The spec's proposal makes it easy to include Markdown+ inside HTML block-level tags, if you want to, but also allows you to+ exclude this. It is also makes parsing much easier, avoiding+ expensive backtracking.++- It does not collapse adjacent bird-track blocks into a single+ blockquote:++ > this is two++ > blockquotes++ > this is a single+ >+ > blockquote with two paragraphs++- Rules for content in lists differ in a few respects, though (as with+ HTML blocks), most lists in existing documents should render as+ intended. There is some discussion of the choice points and+ differences in the subsection of List Items entitled Motivation.+ We think that the spec's proposal does better than any existing+ implementation in rendering lists the way a human writer or reader+ would intuitively understand them. (We could give numerous examples+ of perfectly natural looking lists that nearly every existing+ implementation flubs up.)++- The spec stipulates that two blank lines break out of all list+ contexts. This is an attempt to deal with issues that often come up+ when someone wants to have two adjacent lists, or a list followed by+ an indented code block.++- Changing bullet characters, or changing from bullets to numbers or+ vice versa, starts a new list. We think that is almost always going+ to be the writer's intent.++- The number that begins an ordered list item may be followed by+ either `.` or `)`. Changing the delimiter style starts a new+ list.++- The start number of an ordered list is significant.++- Fenced code blocks are supported, delimited by either+ backticks (```` ``` ```` or tildes (` ~~~ `).++Contributing+------------++There is a [forum for discussing+CommonMark](http://talk.commonmark.org); you should use it instead of+github issues for questions and possibly open-ended discussions.+Use the [github issue tracker](http://github.com/jgm/CommonMark/issues)+only for simple, clear, actionable issues.++Authors+-------++The spec was written by John MacFarlane, drawing on++- his experience writing and maintaining Markdown implementations in several+ languages, including the first Markdown parser not based on regular+ expression substitutions ([pandoc](http://github.com/jgm/pandoc)) and+ the first markdown parsers based on PEG grammars+ ([peg-markdown](http://github.com/jgm/peg-markdown),+ [lunamark](http://github.com/jgm/lunamark))+- a detailed examination of the differences between existing Markdown+ implementations using [BabelMark 2](http://johnmacfarlane.net/babelmark2/),+ and+- extensive discussions with David Greenspan, Jeff Atwood, Vicent+ Marti, Neil Williams, and Benjamin Dumke-von der Ehe.++John MacFarlane was also responsible for the original versions of the+C and JavaScript implementations. The block parsing algorithm was+worked out together with David Greenspan. Vicent Marti+optimized the C implementation for performance, increasing its speed+tenfold. Kārlis Gaņģis helped work out a better parsing algorithm+for links and emphasis, eliminating several worst-case performance+issues. Nick Wellnhofer contributed many improvements, including+most of the C library's API and its test harness. Vitaly Puzrin+has offered much good advice about the JavaScript implementation.++[cmake]: http://www.cmake.org/download/+[pandoc]: http://johnmacfarlane.net/pandoc/+[re2c]: http://re2c.org++> the simple example of a blockquote +> the simple example of a blockquote+> the simple example of a blockquote+> the simple example of a blockquote+... continuation+... continuation+... continuation+... continuation++empty blockquote:++>+>+>+>++>>>>>> deeply nested blockquote+>>>>> deeply nested blockquote+>>>> deeply nested blockquote+>>> deeply nested blockquote+>> deeply nested blockquote+> deeply nested blockquote++> deeply nested blockquote+>> deeply nested blockquote+>>> deeply nested blockquote+>>>> deeply nested blockquote+>>>>> deeply nested blockquote+>>>>>> deeply nested blockquote++ an+ example++ of++++ a code+ block+++``````````text+an+example+```+of+++a fenced+```+code+block+``````````++# heading+### heading+##### heading++# heading #+### heading ###+##### heading \#\#\#\#\######++############ not a heading++ * * * * *++ - - - - -++ ________+++ ************************* text++<div class="this is an html block">++blah blah++</div>++<table>+ <tr>+ <td>+ **test**+ </td>+ </tr>+</table>++<table>++ <tr>++ <td>++ test++ </td>++ </tr>++</table>++<![CDATA[+ [[[[[[[[[[[... *cdata section - this should not be parsed* ...]]]]]]]]]]]+]]>++heading+---++heading+===================================++not a heading+----------------------------------- text+ - tidy+ - bullet+ - list+++ - loose++ - bullet++ - list+++ 0. ordered+ 1. list+ 2. example+++ -+ -+ -+ -+++ 1.+ 2.+ 3.+++ - an example+of a list item+ with a continuation++ this part is inside the list++ this part is just a paragraph +++ 1. test+ - test+ 1. test+ - test+++111111111111111111111111111111111111111111. is this a valid bullet?++ - _________________________++ - this+ - is++ a++ long+ - loose+ - list++ - with+ - some++ tidy++ - list+ - items+ - in++ - between+ - _________________________++ - this+ - is+ - a+ - deeply+ - nested+ - bullet+ - list+ ++ 1. this+ 2. is+ 3. a+ 4. deeply+ 5. nested+ 6. unordered+ 7. list+++ - 1+ - 2+ - 3+ - 4+ - 5+ - 6+ - 7+ - 6+ - 5+ - 4+ - 3+ - 2+ - 1+++ - - - - - - - - - deeply-nested one-element item++[1] [2] [3] [1] [2] [3]++[looooooooooooooooooooooooooooooooooooooooooooooooooong label]++ [1]: <http://something.example.com/foo/bar>+ [2]: http://something.example.com/foo/bar 'test'+ [3]:+ http://foo/bar+ [ looooooooooooooooooooooooooooooooooooooooooooooooooong label ]:+ 111+ 'test'+ [[[[[[[[[[[[[[[[[[[[ this should not slow down anything ]]]]]]]]]]]]]]]]]]]]: q+ (as long as it is not referenced anywhere)++ [[[[[[[[[[[[[[[[[[[[]: this is not a valid reference+[[[[[[[foo]]]]]]]++[[[[[[[foo]]]]]]]: bar+[[[[[[foo]]]]]]: bar+[[[[[foo]]]]]: bar+[[[[foo]]]]: bar+[[[foo]]]: bar+[[foo]]: bar+[foo]: bar++[*[*[*[*[foo]*]*]*]*]++[*[*[*[*[foo]*]*]*]*]: bar+[*[*[*[foo]*]*]*]: bar+[*[*[foo]*]*]: bar+[*[foo]*]: bar+[foo]: bar+closed (valid) autolinks:++ <ftp://1.2.3.4:21/path/foo>+ <http://foo.bar.baz?q=hello&id=22&boolean>+ <http://veeeeeeeeeeeeeeeeeeery.loooooooooooooooooooooooooooooooong.autolink/>+ <teeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeest@gmail.com>++these are not autolinks:++ <ftp://1.2.3.4:21/path/foo+ <http://foo.bar.baz?q=hello&id=22&boolean+ <http://veeeeeeeeeeeeeeeeeeery.loooooooooooooooooooooooooooooooong.autolink+ <teeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeest@gmail.com+ < http://foo.bar.baz?q=hello&id=22&boolean >+`lots`of`backticks`++``i``wonder``how``this``will``be``parsed``+*this* *is* *your* *basic* *boring* *emphasis*++_this_ _is_ _your_ _basic_ _boring_ _emphasis_++**this** **is** **your** **basic** **boring** **emphasis**+*this *is *a *bunch* of* nested* emphases* ++__this __is __a __bunch__ of__ nested__ emphases__ ++***this ***is ***a ***bunch*** of*** nested*** emphases*** +*this *is *a *worst *case *for *em *backtracking++__this __is __a __worst __case __for __em __backtracking++***this ***is ***a ***worst ***case ***for ***em ***backtracking+entities:++ & © Æ Ď ¾ ℋ ⅆ ∲++# Ӓ Ϡ �++non-entities:++&18900987654321234567890; &1234567890098765432123456789009876543212345678987654;++&qwertyuioppoiuytrewqwer; &oiuytrewqwertyuioiuytrewqwertyuioytrewqwertyuiiuytri;++\t\e\s\t\i\n\g \e\s\c\a\p\e \s\e\q\u\e\n\c\e\s++\!\\\"\#\$\%\&\'\(\)\*\+\,\.\/\:\;\<\=\>\?++\@ \[ \] \^ \_ \` \{ \| \} \~ \- \'++\+\\+\\\+\\\\+\\\\\++\<this\> \<is\> \<not\> \<html\>++Taking commonmark tests from the spec for benchmarking here:++<a><bab><c2c>++<a/><b2/>++<a /><b2+data="foo" >++<a foo="bar" bam = 'baz <em>"</em>'+_boolean zoop:33=zoop:33 />++<33> <__>++<a h*#ref="hi">++<a href="hi'> <a href=hi'>++< a><+foo><bar/ >++<a href='bar'title=title>++</a>+</foo >++</a href="foo">++foo <!-- this is a+comment - with hyphen -->++foo <!-- not a comment -- two hyphens -->++foo <?php echo $a; ?>++foo <!ELEMENT br EMPTY>++foo <![CDATA[>&<]]>++<a href="ö">++<a href="\*">++<a href="\"">+Valid links:++ [this is a link]()+ [this is a link](<http://something.example.com/foo/bar>)+ [this is a link](http://something.example.com/foo/bar 'test')+ ![this is an image]()+ + + + [escape test](<\>\>\>\>\>\>\>\>\>\>\>\>\>\>> '\'\'\'\'\'\'\'\'\'\'\'\'\'\'')+ [escape test \]\]\]\]\]\]\]\]\]\]\]\]\]\]\]\]](\)\)\)\)\)\)\)\)\)\)\)\)\)\))++Invalid links:++ [this is not a link++ [this is not a link](++ [this is not a link](http://something.example.com/foo/bar 'test'+ + [this is not a link](((((((((((((((((((((((((((((((((((((((((((((((+ + [this is not a link]((((((((((()))))))))) (((((((((()))))))))))+Valid links:++[[[[[[[[](test)](test)](test)](test)](test)](test)](test)]++[ [[[[[[[[[[[[[[[[[[ [](test) ]]]]]]]]]]]]]]]]]] ](test)++Invalid links:++[[[[[[[[[++[ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [++ lobortis, sapien arcu mattis erat, vel aliquet sem urna et risus. Ut feugiat sapien vitae mi elementum laoreet. Suspendisse potenti. Aliquam erat nisl, aliquam pretium libero aliquet, sagittis eleifend nunc. In hac habitasse platea dictumst. Integer turpis augue, tincidunt dignissim mauris id, rhoncus dapibus purus. Maecenas et enim odio. Nullam massa metus, varius quis vehicula sed, pharetra mollis erat. In quis viverra velit. Vivamus placerat, est nec hendrerit varius, enim dui hendrerit magna, ut pulvinar nibh lorem vel lacus. Mauris a orci iaculis, hendrerit eros sed, gravida leo. In dictum mauris vel augue varius, ac ullamcorper nisl ornare. In eu posuere velit, ac fermentum arcu. Interdum et malesuada fames ac ante ipsum primis in faucibus. Nullam sed malesuada leo, at interdum elit.++Nullam ut tincidunt nunc. [Pellentesque][1] metus lacus, commodo eget justo ut, rutrum varius nunc. Sed non rhoncus risus. Morbi sodales gravida pulvinar. Duis malesuada, odio volutpat elementum vulputate, massa magna scelerisque ante, et accumsan tellus nunc in sem. Donec mattis arcu et velit aliquet, non sagittis justo vestibulum. Suspendisse volutpat felis lectus, nec consequat ipsum mattis id. Donec dapibus vehicula facilisis. In tincidunt mi nisi, nec faucibus tortor euismod nec. Suspendisse ante ligula, aliquet vitae libero eu, vulputate dapibus libero. Sed bibendum, sapien at posuere interdum, libero est sollicitudin magna, ac gravida tellus purus eu ipsum. Proin ut quam arcu.++Suspendisse potenti. Donec ante velit, ornare at augue quis, tristique laoreet sem. Etiam in ipsum elit. Nullam cursus dolor sit amet nulla feugiat tristique. Phasellus ac tellus tincidunt, imperdiet purus eget, ullamcorper ipsum. Cras eu tincidunt sem. Nullam sed dapibus magna. Lorem ipsum dolor sit amet, consectetur adipiscing elit. In id venenatis tortor. In consectetur sollicitudin pharetra. Etiam convallis nisi nunc, et aliquam turpis viverra sit amet. Maecenas faucibus sodales tortor. Suspendisse lobortis mi eu leo viverra volutpat. Pellentesque velit ante, vehicula sodales congue ut, elementum a urna. Cras tempor, ipsum eget luctus rhoncus, arcu ligula fermentum urna, vulputate pharetra enim enim non libero.++Proin diam quam, elementum in eleifend id, elementum et metus. Cras in justo consequat justo semper ultrices. Sed dignissim lectus a ante mollis, nec vulputate ante molestie. Proin in porta nunc. Etiam pulvinar turpis sed velit porttitor, vel adipiscing velit fringilla. Cras ac tellus vitae purus pharetra tincidunt. Sed cursus aliquet aliquet. Cras eleifend commodo malesuada. In turpis turpis, ullamcorper ut tincidunt a, ullamcorper a nunc. Etiam luctus tellus ac dapibus gravida. Ut nec lacus laoreet neque ullamcorper volutpat.++Nunc et leo erat. Aenean mattis ultrices lorem, eget adipiscing dolor ultricies eu. In hac habitasse platea dictumst. Vivamus cursus feugiat sapien quis aliquam. Mauris quam libero, porta vel volutpat ut, blandit a purus. Vivamus vestibulum dui vel tortor molestie, sit amet feugiat sem commodo. Nulla facilisi. Sed molestie arcu eget tellus vestibulum tristique.++[1]: https://github.com/markdown-it++this is a test for tab expansion, be careful not to replace them with spaces++1 4444+22 333+333 22+4444 1+++ tab-indented line+ space-indented line+ tab-indented line+++a lot of spaces in between here++a lot of tabs in between here+
+ bench/sample.md view
@@ -0,0 +1,284 @@+CommonMark+==========++CommonMark is a rationalized version of Markdown syntax,+with a [spec][the spec] and BSD3-licensed reference+implementations in C and JavaScript.++[Try it now!](http://spec.commonmark.org/dingus.html)++The implementations+-------------------++The C implementation provides both a shared library (`libcmark`) and a+standalone program `cmark` that converts CommonMark to HTML. It is+written in standard C99 and has no library dependencies. The parser is+very fast (see [benchmarks](benchmarks.md)).++It is easy to use `libcmark` in python, lua, ruby, and other dynamic+languages: see the `wrappers/` subdirectory for some simple examples.++The JavaScript implementation provides both an NPM package and a+single JavaScript file, with no dependencies, that can be linked into+an HTML page. For further information, see the+[README in the js directory](js/README.md).++**A note on security:**+Neither implementation attempts to sanitize link attributes or+raw HTML. If you use these libraries in applications that accept+untrusted user input, you must run the output through an HTML+sanitizer to protect against+[XSS attacks](http://en.wikipedia.org/wiki/Cross-site_scripting).++Installing (C)+--------------++Building the C program (`cmark`) and shared library (`libcmark`)+requires [cmake]. If you modify `scanners.re`, then you will also+need [re2c], which is used to generate `scanners.c` from+`scanners.re`. We have included a pre-generated `scanners.c` in+the repository to reduce build dependencies.++If you have GNU make, you can simply `make`, `make test`, and `make+install`. This calls [cmake] to create a `Makefile` in the `build`+directory, then uses that `Makefile` to create the executable and+library. The binaries can be found in `build/src`.++For a more portable method, you can use [cmake] manually. [cmake] knows+how to create build environments for many build systems. For example,+on FreeBSD:++ mkdir build+ cd build+ cmake .. # optionally: -DCMAKE_INSTALL_PREFIX=path+ make # executable will be created as build/src/cmark+ make test+ make install++Or, to create Xcode project files on OSX:++ mkdir build+ cd build+ cmake -G Xcode ..+ make+ make test+ make install++The GNU Makefile also provides a few other targets for developers.+To run a benchmark:++ make bench++To run a "fuzz test" against ten long randomly generated inputs:++ make fuzztest++To run a test for memory leaks using `valgrind`:++ make leakcheck++To reformat source code using `astyle`:++ make astyle++To make a release tarball and zip archive:++ make archive+++Compiling for Windows+---------------------++To compile with MSVC and NMAKE:++ nmake++You can cross-compile a Windows binary and dll on linux if you have the+`mingw32` compiler:++ make mingw++The binaries will be in `build-mingw/windows/bin`.++Installing (JavaScript)+-----------------------++The JavaScript library can be installed through `npm`:++ npm install commonmark++This includes a command-line converter called `commonmark`.++If you want to use it in a client application, you can fetch+a pre-built copy of `commonmark.js` from+<http://spec.commonmark.org/js/commonmark.js>.++For further information, see the+[README in the js directory](js/README.md).++The spec+--------++[The spec] contains over 500 embedded examples which serve as conformance+tests. To run the tests using an executable `$PROG`:++ python3 test/spec_tests.py --program $PROG++If you want to extract the raw test data from the spec without+actually running the tests, you can do:++ python3 test/spec_tests.py --dump-tests++and you'll get all the tests in JSON format.++[The spec]: http://spec.commonmark.org/0.13/++The source of [the spec] is `spec.txt`. This is basically a Markdown+file, with code examples written in a shorthand form:++ .+ Markdown source+ .+ expected HTML output+ .++To build an HTML version of the spec, do `make spec.html`. To build a+PDF version, do `make spec.pdf`. (Creating a PDF requires [pandoc]+and a LaTeX installation. Creating the HTML version requires only+`libcmark` and `python3`.)++The spec is written from the point of view of the human writer, not+the computer reader. It is not an algorithm---an English translation of+a computer program---but a declarative description of what counts as a block+quote, a code block, and each of the other structural elements that can+make up a Markdown document.++Because John Gruber's [canonical syntax+description](http://daringfireball.net/projects/markdown/syntax) leaves+many aspects of the syntax undetermined, writing a precise spec requires+making a large number of decisions, many of them somewhat arbitrary.+In making them, we have appealed to existing conventions and+considerations of simplicity, readability, expressive power, and+consistency. We have tried to ensure that "normal" documents in the many+incompatible existing implementations of Markdown will render, as far as+possible, as their authors intended. And we have tried to make the rules+for different elements work together harmoniously. In places where+different decisions could have been made (for example, the rules+governing list indentation), we have explained the rationale for+my choices. In a few cases, we have departed slightly from the canonical+syntax description, in ways that we think further the goals of Markdown+as stated in that description.++For the most part, we have limited ourselves to the basic elements+described in Gruber's canonical syntax description, eschewing extensions+like footnotes and definition lists. It is important to get the core+right before considering such things. However, we have included a visible+syntax for line breaks and fenced code blocks.++Differences from original Markdown+----------------------------------++There are only a few places where this spec says things that contradict+the canonical syntax description:++- It allows all punctuation symbols to be backslash-escaped,+ not just the symbols with special meanings in Markdown. We found+ that it was just too hard to remember which symbols could be+ escaped.++- It introduces an alternative syntax for hard line+ breaks, a backslash at the end of the line, supplementing the+ two-spaces-at-the-end-of-line rule. This is motivated by persistent+ complaints about the “invisible” nature of the two-space rule.++- Link syntax has been made a bit more predictable (in a+ backwards-compatible way). For example, `Markdown.pl` allows single+ quotes around a title in inline links, but not in reference links.+ This kind of difference is really hard for users to remember, so the+ spec allows single quotes in both contexts.++- The rule for HTML blocks differs, though in most real cases it+ shouldn't make a difference. (See the section on HTML Blocks+ for details.) The spec's proposal makes it easy to include Markdown+ inside HTML block-level tags, if you want to, but also allows you to+ exclude this. It is also makes parsing much easier, avoiding+ expensive backtracking.++- It does not collapse adjacent bird-track blocks into a single+ blockquote:++ > this is two++ > blockquotes++ > this is a single+ >+ > blockquote with two paragraphs++- Rules for content in lists differ in a few respects, though (as with+ HTML blocks), most lists in existing documents should render as+ intended. There is some discussion of the choice points and+ differences in the subsection of List Items entitled Motivation.+ We think that the spec's proposal does better than any existing+ implementation in rendering lists the way a human writer or reader+ would intuitively understand them. (We could give numerous examples+ of perfectly natural looking lists that nearly every existing+ implementation flubs up.)++- The spec stipulates that two blank lines break out of all list+ contexts. This is an attempt to deal with issues that often come up+ when someone wants to have two adjacent lists, or a list followed by+ an indented code block.++- Changing bullet characters, or changing from bullets to numbers or+ vice versa, starts a new list. We think that is almost always going+ to be the writer's intent.++- The number that begins an ordered list item may be followed by+ either `.` or `)`. Changing the delimiter style starts a new+ list.++- The start number of an ordered list is significant.++- Fenced code blocks are supported, delimited by either+ backticks (```` ``` ```` or tildes (` ~~~ `).++Contributing+------------++There is a [forum for discussing+CommonMark](http://talk.commonmark.org); you should use it instead of+github issues for questions and possibly open-ended discussions.+Use the [github issue tracker](http://github.com/jgm/CommonMark/issues)+only for simple, clear, actionable issues.++Authors+-------++The spec was written by John MacFarlane, drawing on++- his experience writing and maintaining Markdown implementations in several+ languages, including the first Markdown parser not based on regular+ expression substitutions ([pandoc](http://github.com/jgm/pandoc)) and+ the first markdown parsers based on PEG grammars+ ([peg-markdown](http://github.com/jgm/peg-markdown),+ [lunamark](http://github.com/jgm/lunamark))+- a detailed examination of the differences between existing Markdown+ implementations using [BabelMark 2](http://johnmacfarlane.net/babelmark2/),+ and+- extensive discussions with David Greenspan, Jeff Atwood, Vicent+ Marti, Neil Williams, and Benjamin Dumke-von der Ehe.++John MacFarlane was also responsible for the original versions of the+C and JavaScript implementations. The block parsing algorithm was+worked out together with David Greenspan. Vicent Marti+optimized the C implementation for performance, increasing its speed+tenfold. Kārlis Gaņģis helped work out a better parsing algorithm+for links and emphasis, eliminating several worst-case performance+issues. Nick Wellnhofer contributed many improvements, including+most of the C library's API and its test harness. Vitaly Puzrin+has offered much good advice about the JavaScript implementation.++[cmake]: http://www.cmake.org/download/+[pandoc]: http://johnmacfarlane.net/pandoc/+[re2c]: http://re2c.org
cmark.cabal view
@@ -1,71 +1,84 @@ name: cmark-version: 0.1.0.1-synopsis: Haskell bindings to libcmark, CommonMark parser and renderer+version: 0.2+synopsis: Fast, accurate CommonMark (Markdown) parser and renderer description: This package provides Haskell bindings for- <http://github.com/jgm/cmark libcmark>, the reference parser for- <http://commonmark.org CommonMark>. It includes sources for- <http://github.com/jgm/cmark libcmark>, and does not require prior- installation of the C library.+ <http://github.com/jgm/cmark libcmark>, the reference+ parser for <http://commonmark.org CommonMark>, a fully+ specified variant of Markdown. It includes sources for+ <http://github.com/jgm/cmark libcmark> (version 0.18.1)+ and does not require prior installation of the C+ library. .- cmark provides the following advantages over existing Markdown libraries- for Haskell:+ cmark provides the following advantages over existing+ Markdown libraries for Haskell: . - __Speed:__ Conversion speed is on par with the- <https://hackage.haskell.org/package/sundown sundown> library: about- 30 times faster than- <https://hackage.haskell.org/package/pandoc pandoc>, 24 times faster- than the Haskell- <https://hackage.haskell.org/package/markdown markdown> package, 8- times faster than- <https://hackage.haskell.org/package/cheapskate cheapskate>.+ <https://hackage.haskell.org/package/sundown sundown>+ library. We were unable to measure precisely+ against+ <https://hackage.haskell.org/package/sundown sundown>,+ which raised a malloc error when compiled into our+ benchmark suite. Relative to other implementations:+ cmark was 82 times faster than+ <https://hackage.haskell.org/package/cheapskate cheapskate>,+ 59 times faster than+ <https://hackage.haskell.org/package/markdown markdown>,+ 105 times faster than+ <https://hackage.haskell.org/package/pandoc pandoc>,+ and 2.8 times faster than+ <https://hackage.haskell.org/package/discount discount>. .- - __Memory footprint:__ Memory footprint is on par with- <https://hackage.haskell.org/package/sundown sundown>. On one- sample, the library uses a fourth the memory that- <https://hackage.haskell.org/package/markdown markdown> uses, and- less than a tenth the memory that- <https://hackage.haskell.org/package/pandoc pandoc> uses.+ - __Memory footprint:__ Memory footprint is on par+ with+ <https://hackage.haskell.org/package/sundown sundown>.+ On one sample, the library uses a fourth the memory+ that+ <https://hackage.haskell.org/package/markdown markdown>+ uses, and less than a tenth the memory that+ <https://hackage.haskell.org/package/pandoc pandoc>+ uses. .- - __Robustness:__ cmark can handle whatever is thrown at it, without- the exponential blowups in parsing time one can sometimes get with- other libraries.+ - __Robustness:__ cmark can handle whatever is thrown+ at it, without the exponential blowups in parsing+ time one can sometimes get with other libraries.+ (The input @bench\/full-sample.md@, for example,+ causes both+ <https://hackage.haskell.org/package/pandoc pandoc>+ and+ <https://hackage.haskell.org/package/markdown markdown>+ to grind to a halt.) .- - __Accuracy:__ cmark passes the CommonMark spec\'s suite of over 500- conformance tests.+ - __Accuracy:__ cmark passes the CommonMark spec\'s+ suite of over 500 conformance tests. .- - __Standardization:__ Since there is a spec and a comprehensive suite- of tests, we can have a high degree of confidence that any two- CommonMark implementations will behave the same. Thus, for example,- one could use this library for server-side rendering and- <http://github.com/jgm/commonmark.js commonmark.js> for client-side- previewing.+ - __Standardization:__ Since there is a spec and a+ comprehensive suite of tests, we can have a high+ degree of confidence that any two CommonMark+ implementations will behave the same. Thus, for+ example, one could use this library for server-side+ rendering and+ <http://github.com/jgm/commonmark.js commonmark.js>+ for client-side previewing. .- - __Ease of installation:__ cmark has minimal dependencies.+ - __Ease of installation:__ cmark is portable and has+ minimal dependencies. . cmark does not provide Haskell versions of the whole- <http://github.com/jgm/cmark libcmark> API, which is built around- mutable @cmark_node@ objects. Instead, it provides two functions:- .- - @markdownToHtml@ uses <http://github.com/jgm/cmark libcmark>\'s- parser and renderer for a maximally efficient conversion of- CommonMark to HTML (as a Text). (\"Smart punctuation\" and a few- other options can be enabled.)- .- - @parseDocument@ uses <http://github.com/jgm/cmark libcmark>\'s- parser to produce a @Node@ tree that can be processed further using- Haskell. One can transform the tree using generics, convert it to- another kind of structure (such as a Pandoc object that can be- rendered using pandoc\'s renderers) or render it using a custom- rendering function.+ <http://github.com/jgm/cmark libcmark> API, which is+ built around mutable @cmark_node@ objects. Instead, it+ provides functions for converting CommonMark to HTML+ (and other formats), and a function for converting+ CommonMark to a @Node@ tree that can be processed+ further using Haskell. .- __A note on security:__ This library does not attempt to sanitize HTML- output. We recommend using- <https://hackage.haskell.org/package/xss-sanitize xss-sanitize> to- filter the output.+ __A note on security:__ This library does not attempt+ to sanitize HTML output. We recommend using+ <https://hackage.haskell.org/package/xss-sanitize xss-sanitize>+ to filter the output. .- __A note on stability:__ There is a good chance the API will change- significantly after this early release.+ __A note on stability:__ There is a good chance the API+ will change significantly after this early release. homepage: https://github.com/jgm/commonmark-hs license: BSD3@@ -96,7 +109,9 @@ cmark/config.h cmark/scanners.h cmark/case_fold_switch.inc-cabal-version: >=1.10+ bench/sample.md+ bench/full-sample.md+cabal-version: >=1.14 Source-repository head type: git@@ -128,3 +143,17 @@ cmark/houdini_html_e.c cmark/inlines.c cmark/main.c++benchmark bench-cmark+ type: exitcode-stdio-1.0+ hs-source-dirs: bench+ main-is: Bench.hs+ build-depends: base, text, criterion, cmark,+ sundown >= 0.6 && < 0.7,+ pandoc >= 1.13.2 && < 1.14,+ cheapskate >= 0.1 && < 0.2,+ markdown >= 0.1 && < 0.2,+ discount >= 0.1 && < 0.2,+ blaze-html >= 0.7 && < 0.8+ ghc-options: -O2+ default-language: Haskell2010
cmark/cmark.h view
@@ -547,7 +547,6 @@ #define NODE_STRONG CMARK_NODE_STRONG #define NODE_LINK CMARK_NODE_LINK #define NODE_IMAGE CMARK_NODE_IMAGE-#define NODE_LINK_LABEL CMARK_NODE_LINK_LABEL #define BULLET_LIST CMARK_BULLET_LIST #define ORDERED_LIST CMARK_ORDERED_LIST #define PERIOD_DELIM CMARK_PERIOD_DELIM