pandoc 2.0.2 → 2.0.3

raw patch · 30 files changed

+546/−137 lines, 30 filesdep +hslua-module-textdep ~http-typesPVP: major bump suggested

API removals or changes: PVP suggests a major version bump

Dependencies added: hslua-module-text

Dependency ranges changed: http-types

API changes (from Hackage documentation)

Files

• MANUAL.txt +6/−2
• README.md +2/−2
• changelog +93/−0
• data/pandoc.lua +92/−1
• data/templates/default.latex +10/−10
• man/manfilter.lua +6/−3
• man/pandoc.1 +7/−1
• pandoc.cabal +2/−1
• src/Text/Pandoc/Extensions.hs +4/−0
• src/Text/Pandoc/Lua.hs +2/−0
• src/Text/Pandoc/Options.hs +11/−2
• src/Text/Pandoc/Parsing.hs +3/−3
• src/Text/Pandoc/Readers/CommonMark.hs +9/−13
• src/Text/Pandoc/Readers/Creole.hs +1/−1
• src/Text/Pandoc/Readers/HTML.hs +5/−5
• src/Text/Pandoc/Readers/LaTeX.hs +16/−6
• src/Text/Pandoc/Readers/Muse.hs +61/−27
• src/Text/Pandoc/Readers/RST.hs +3/−1
• src/Text/Pandoc/Writers/EPUB.hs +3/−3
• src/Text/Pandoc/Writers/Markdown.hs +22/−12
• stack.yaml +1/−0
• test/Tests/Lua.hs +17/−3
• test/Tests/Readers/Muse.hs +99/−40
• test/Tests/Writers/Muse.hs +3/−1
• test/command/4056.md +24/−0
• test/command/4061.md +14/−0
• test/command/4062.md +6/−0
• test/command/4068.md +9/−0
• test/lua/attr-test.lua +6/−0
• test/lua/uppercase-header.lua +9/−0

@@ -1,6 +1,6 @@ % Pandoc User's Guide % John MacFarlane-% November 11, 2017+% November 20, 2017  Synopsis ========@@ -2930,7 +2930,7 @@     <span class="smallcaps">Small caps</span>  For compatibility with other Markdown flavors, CSS is also supported:-    +     <span style="font-variant:small-caps;">Small caps</span>  This will work in all output formats that support small caps.@@ -2998,6 +2998,8 @@     command-line options selected. Therefore see [Math rendering in HTML]     above. +This extension can be used with both `markdown` and `html` input.+ [interpreted text role `:math:`]: http://docutils.sourceforge.net/docs/ref/rst/roles.html#math  Raw HTML@@ -3707,6 +3709,8 @@ TeX math, and anything between `\[` and `\]` to be interpreted as display TeX math.  Note: a drawback of this extension is that it precludes escaping `(` and `[`.++This extension can be used with both `markdown` and `html` input.  #### Extension: `tex_math_double_backslash` #### 

@@ -38,8 +38,8 @@ [tables], flexible [ordered lists], [definition lists], [fenced code blocks], [superscripts and subscripts], [strikeout], [metadata blocks], automatic tables of contents, embedded LaTeX [math], [citations], and-[Markdown inside HTML block elements][Extension:-`markdown_in_html_blocks`]. (These enhancements, described further under+[Markdown inside HTML block elements](#extension-markdown_in_html_blocks).+(These enhancements, described further under [Pandoc's Markdown], can be disabled using the `markdown_strict` input or output format.) 

@@ -1,3 +1,96 @@+pandoc (2.0.3)++  * Lua filters: preload text module (Albert Krewinkel, #4077).+    The `text` module is preloaded in lua. The module contains some UTF-8+    aware string functions, implemented in Haskell.  The module is loaded on+    request only, e.g.:++        text = require 'text'+        function Str (s)+          s.text = text.upper(s.text)+          return s+        end++  * Allow table-like access to attributes in lua filters (Albert Krewinkel,+    #4071).  Attribute lists are represented as associative lists in Lua. Pure+    associative lists are awkward to work with. A metatable is attached to+    attribute lists, allowing to access and use the associative list as if+    the attributes were stored in as normal key-value pair in table.+    Note that this changes the way `pairs` works on attribute lists. Instead+    of producing integer keys and two-element tables, the resulting iterator+    function now returns the key and value of those pairs.  Use `ipairs` to+    get the old behavior.  Warning: the new iteration mechanism only works if+    pandoc has been compiled with Lua 5.2 or later (current default: 5.3).++  * Text.Pandoc.Parsing.uri:  allow `&` and `=` as word characters (#4068).+    This fixes a bug where pandoc would stop parsing a URI with an+    empty attribute:  for example, `&a=&b=` wolud stop at `a`.+    (The uri parser tries to guess which punctuation characters+    are part of the URI and which might be punctuation after it.)++  * Introduce `HasSyntaxExtensions` typeclass (Alexander Krotov, #4074).++    + Added new `HasSyntaxExtensions` typeclass for `ReaderOptions` and+      `WriterOptions`.+    + Reimplemented `isEnabled` function from `Options.hs` to accept both+      `ReaderOptions` and `WriterOptions`.+    + Replaced `enabled` from `CommonMark.hs` with new `isEnabled`.++  * Add `amuse` extension (Alexander Krotov) to enable Amuse wiki+    behavior for `muse`.  New `Ext_amuse` constructor for+    `Extension`. Note: this is switched on by default; for+    Emacs behavior, use `muse-amuse`.++  * Muse reader (Alexander Krotov):++    + Count only one space as part of list item marker.+    + Produce SoftBreaks on newlines. Now wrapping can be preserved+      with `--wrap=preserve`.+    + Add Text::Amuse footnote extensions.  Footnote end is indicated by+      indentation, so footnotes can be placed anywhere in the text,+      not just at the end of it.+    + Accept Emacs Muse definition lists when `-amuse`.+      Emacs Muse does not require indentation.++  * HTML reader:++    + Ensure we don't produce level 0 headers (#4076), even for chapter+      sections in epubs.  This causes problems because writers aren't set+      up to expect these.+    + Allow spaces after `\(` and before `\)` with `tex_math_single_backslash`.+      Previously `\( \frac{1}{a} < \frac{1}{b} \)` was not parsed as math in+      `markdown` or `html` `+tex_math_single_backslash`.++  * MANUAL: clarify that math extensions work with HTML.+    Clarify that `tex_math_dollars` and `tex_math_single_backslash`+    will work with HTML as well as Markdown.++  * Creole reader: Fix performance issue for longer lists (Sascha Wilde,+    #4067).++  * RST reader: better support for 'container' directive (#4066).+    Create a div, incorporate name attribute and classes.++  * LaTeX reader:++    + Support column specs like `*{2}{r}` (#4056).  This is equivalent to+      `rr`.  We now expand it like a macro.+    + Allow optional args for parbox (#4056).+    + Allow optional arguments on `\footnote` (#4062).++  * EPUB writer: Fixed path for cover image (#4069).  It was previously+    `media/media/imagename`, and should have been `media/imagename`.++  * Markdown writer: fix bug with doubled footnotes in grid tables+    (#4061).++  * LaTeX template:  include natbib/biblatex after polyglossia (#4073).+    Otherwise we seem to get an error; biblatex wants polyglossia+    language to be defined.++  * Added examples to lua filters documentation.++ pandoc (2.0.2)    * Deprecated ancient HTML math methods: `--latexmathml`, `--gladtex`,

@@ -627,6 +627,97 @@ -- Helpers -- @section helpers +-- Find a value pair in a list.+-- @function find+-- @tparam table list to be searched+-- @param needle element to search for+-- @param[opt] key when non-nil, compare on this field of each list element+local function find (alist, needle, key)+  local test+  if key then+    test = function(x) return x[key] == needle end+  else+    test = function(x) return x == needle end+  end+  for i, k in ipairs(alist) do+    if test(k) then+      return i, k+    end+  end+  return nil+end++-- Lookup a value in an associative list+-- @function lookup+-- @tparam {{key, value},...} alist associative list+-- @param key key for which the associated value is to be looked up+local function lookup(alist, key)+  return (select(2, find(alist, key, 1)) or {})[2]+end++--- Return an iterator which returns key-value pairs of an associative list.+-- @function apairs+-- @tparam {{key, value},...} alist associative list+local apairs = function (alist)+  local i = 1+  local cur+  function nxt ()+    cur = rawget(alist, i)+    if cur then+      i = i + 1+      return cur[1], cur[2]+    end+    return nil+  end+  return nxt, nil, nil+end++-- AttributeList, a metatable to allow table-like access to attribute lists+-- represented by associative lists.+local AttributeList = {+  __index = function (t, k)+    if type(k) == "number" then+      return rawget(t, k)+    else+      return lookup(t, k)+    end+  end,++  __newindex = function (t, k, v)+    local idx, cur = find(t, k, 1)+    if v == nil then+      table.remove(t, idx)+    elseif cur then+      cur[2] = v+    elseif type(k) == "number" then+      rawset(t, k, v)+    else+      rawset(t, #t + 1, {k, v})+    end+  end,++  __pairs = apairs+}++-- convert a table to an associative list. The order of key-value pairs in the+-- alist is undefined. The table should either contain no numeric keys or+-- already be an associative list.+-- @tparam table associative list or table without numeric keys.+-- @treturn table associative list+local to_alist = function (tbl)+  if #tbl ~= 0 or next(tbl) == nil then+    -- probably already an alist+    return tbl+  end+  local alist = {}+  local i = 1+  for k, v in pairs(tbl) do+    alist[i] = {k, v}+    i = i + 1+  end+  return alist+end+ -- Attr M.Attr = {} M.Attr._field_names = {identifier = 1, classes = 2, attributes = 3}@@ -639,7 +730,7 @@ M.Attr.__call = function(t, identifier, classes, attributes)   identifier = identifier or ''   classes = classes or {}-  attributes = attributes or {}+  attributes = setmetatable(to_alist(attributes or {}), AttributeList)   local attr = {identifier, classes, attributes}   setmetatable(attr, t)   return attr

@@ -154,16 +154,6 @@ $if(beamer)$ \newif\ifbibliography $endif$-$if(natbib)$-\usepackage[$natbiboptions$]{natbib}-\bibliographystyle{$if(biblio-style)$$biblio-style$$else$plainnat$endif$}-$endif$-$if(biblatex)$-\usepackage[$if(biblio-style)$style=$biblio-style$,$endif$$for(biblatexoptions)$$biblatexoptions$$sep$,$endfor$]{biblatex}-$for(bibliography)$-\addbibresource{$bibliography$}-$endfor$-$endif$ $if(listings)$ \usepackage{listings} \newcommand{\passthrough}[1]{#1}@@ -295,6 +285,16 @@   \newenvironment{RTL}{\beginR}{\endR}   \newenvironment{LTR}{\beginL}{\endL} \fi+$endif$+$if(natbib)$+\usepackage[$natbiboptions$]{natbib}+\bibliographystyle{$if(biblio-style)$$biblio-style$$else$plainnat$endif$}+$endif$+$if(biblatex)$+\usepackage[$if(biblio-style)$style=$biblio-style$,$endif$$for(biblatexoptions)$$biblatexoptions$$sep$,$endfor$]{biblatex}+$for(bibliography)$+\addbibresource{$bibliography$}+$endfor$ $endif$  $if(title)$

@@ -1,19 +1,22 @@--- filters to create the pandoc man page from MANUAL.txt+-- we use preloaded text to get a UTF-8 aware 'upper' function+local text = require('text') --- capitalize headers+-- capitalize level 1 headers function Header(el)     if el.level == 1 then       return pandoc.walk_block(el, {         Str = function(el)-            return pandoc.Str(el.text:upper())+            return pandoc.Str(text.upper(el.text))         end })     end end +-- replace links with link text function Link(el)     return el.content end +-- remove notes function Note(el)     return {} end

@@ -1,5 +1,5 @@ .\"t-.TH PANDOC 1 "November 11, 2017" "pandoc 2.0.1.1"+.TH PANDOC 1 "November 20, 2017" "pandoc 2.0.3" .SH NAME pandoc - general markup converter .SH SYNOPSIS@@ -3728,6 +3728,9 @@ Therefore see Math rendering in HTML above. .RS .RE+.PP+This extension can be used with both \f[C]markdown\f[] and \f[C]html\f[]+input. .SS Raw HTML .SS Extension: \f[C]raw_html\f[] .PP@@ -4644,6 +4647,9 @@ to be interpreted as display TeX math. Note: a drawback of this extension is that it precludes escaping \f[C](\f[] and \f[C][\f[].+.PP+This extension can be used with both \f[C]markdown\f[] and \f[C]html\f[]+input. .SS Extension: \f[C]tex_math_double_backslash\f[] .PP Causes anything between \f[C]\\\\(\f[] and \f[C]\\\\)\f[] to be

@@ -1,5 +1,5 @@ name:            pandoc-version:         2.0.2+version:         2.0.3 cabal-version:   >= 1.10 build-type:      Custom license:         GPL@@ -327,6 +327,7 @@                  scientific >= 0.2 && < 0.4,                  vector >= 0.10 && < 0.13,                  hslua >= 0.9 && < 0.10,+                 hslua-module-text >= 0.1.2 && < 0.2,                  binary >= 0.5 && < 0.9,                  SHA >= 1.6 && < 1.7,                  haddock-library >= 1.1 && < 1.5,

@@ -153,6 +153,7 @@     | Ext_smart               -- ^ "Smart" quotes, apostrophes, ellipses, dashes     | Ext_old_dashes          -- ^ -- = em, - before number = en     | Ext_spaced_reference_links -- ^ Allow space between two parts of ref link+    | Ext_amuse -- ^ Enable Text::Amuse extensions to Emacs Muse markup     deriving (Show, Read, Enum, Eq, Ord, Bounded, Data, Typeable, Generic)  instance ToJSON Extension where@@ -314,6 +315,9 @@ getDefaultExtensions "markdown_mmd" = multimarkdownExtensions getDefaultExtensions "markdown_github" = githubMarkdownExtensions getDefaultExtensions "markdown"        = pandocExtensions+getDefaultExtensions "muse"            = extensionsFromList+                                           [Ext_amuse,+                                            Ext_auto_identifiers] getDefaultExtensions "plain"           = plainExtensions getDefaultExtensions "gfm"             = githubMarkdownExtensions getDefaultExtensions "org"             = extensionsFromList

@@ -46,6 +46,7 @@ import Text.Pandoc.Lua.Filter (LuaFilter, walkMWithLuaFilter) import Text.Pandoc.MediaBag (MediaBag) import qualified Foreign.Lua as Lua+import qualified Foreign.Lua.Module.Text as Lua  runLuaFilter :: Maybe FilePath -> FilePath -> String              -> Pandoc -> PandocIO (Either LuaException Pandoc)@@ -64,6 +65,7 @@               -> Pandoc -> Lua Pandoc runLuaFilter' commonState datadir filterPath format mbRef pd = do   Lua.openlibs+  Lua.preloadTextModule "text"   -- store module in global "pandoc"   pushPandocModule datadir   Lua.setglobal "pandoc"

@@ -56,6 +56,9 @@ import Text.Pandoc.Extensions import Text.Pandoc.Highlighting (Style, pygments) +class HasSyntaxExtensions a where+  getExtensions :: a -> Extensions+ data ReaderOptions = ReaderOptions{          readerExtensions            :: Extensions  -- ^ Syntax extensions        , readerStandalone            :: Bool -- ^ Standalone document with header@@ -69,6 +72,9 @@        , readerStripComments         :: Bool -- ^ Strip HTML comments instead of parsing as raw HTML } deriving (Show, Read, Data, Typeable, Generic) +instance HasSyntaxExtensions ReaderOptions where+  getExtensions opts = readerExtensions opts+ instance ToJSON ReaderOptions where   toEncoding = genericToEncoding defaultOptions instance FromJSON ReaderOptions@@ -259,6 +265,9 @@                       , writerSyntaxMap        = defaultSyntaxMap                       } +instance HasSyntaxExtensions WriterOptions where+  getExtensions opts = writerExtensions opts+ -- | Returns True if the given extension is enabled.-isEnabled :: Extension -> WriterOptions -> Bool-isEnabled ext opts = ext `extensionEnabled` writerExtensions opts+isEnabled :: HasSyntaxExtensions a => Extension -> a -> Bool+isEnabled ext opts = ext `extensionEnabled` getExtensions opts

@@ -563,7 +563,7 @@   -- http://en.wikipedia.org/wiki/State_of_emergency_(disambiguation)   -- as a URL, while NOT picking up the closing paren in   -- (http://wikipedia.org). So we include balanced parens in the URL.-  let isWordChar c = isAlphaNum c || c `elem` "#$%*+/@\\_-"+  let isWordChar c = isAlphaNum c || c `elem` "#$%*+/@\\_-&="   let wordChar = satisfy isWordChar   let percentEscaped = try $ char '%' >> skipMany1 (satisfy isHexDigit)   let entity = () <$ characterReference@@ -586,7 +586,7 @@ mathInlineWith :: Stream s m Char  => String -> String -> ParserT s st m String mathInlineWith op cl = try $ do   string op-  notFollowedBy space+  when (op == "$") $ notFollowedBy space   words' <- many1Till (count 1 (noneOf " \t\n\\")                    <|> (char '\\' >>                            -- This next clause is needed because \text{..} can@@ -600,7 +600,7 @@                           return " "                     ) (try $ string cl)   notFollowedBy digit  -- to prevent capture of $5-  return $ concat words'+  return $ trim $ concat words'  where   inBalancedBraces :: Stream s m Char => Int -> String -> ParserT s st m String   inBalancedBraces 0 "" = do

@@ -48,18 +48,14 @@ -- | Parse a CommonMark formatted string into a 'Pandoc' structure. readCommonMark :: PandocMonad m => ReaderOptions -> Text -> m Pandoc readCommonMark opts s = return $-  (if enabled Ext_gfm_auto_identifiers opts+  (if isEnabled Ext_gfm_auto_identifiers opts       then addHeaderIdentifiers       else id) $   nodeToPandoc opts $ commonmarkToNode opts' exts s-  where opts' = [ optSmart | enabled Ext_smart opts ]-        exts = [ extStrikethrough | enabled Ext_strikeout opts ] ++-               [ extTable | enabled Ext_pipe_tables opts ] ++-               [ extAutolink | enabled Ext_autolink_bare_uris opts ]---- | Returns True if the given extension is enabled.-enabled :: Extension -> ReaderOptions -> Bool-enabled ext opts = ext `extensionEnabled` readerExtensions opts+  where opts' = [ optSmart | isEnabled Ext_smart opts ]+        exts = [ extStrikethrough | isEnabled Ext_strikeout opts ] +++               [ extTable | isEnabled Ext_pipe_tables opts ] +++               [ extAutolink | isEnabled Ext_autolink_bare_uris opts ]  convertEmojis :: String -> String convertEmojis (':':xs) =@@ -112,7 +108,7 @@ addBlock opts (Node _ BLOCK_QUOTE nodes) =   (BlockQuote (addBlocks opts nodes) :) addBlock opts (Node _ (HTML_BLOCK t) _)-  | enabled Ext_raw_html opts = (RawBlock (Format "html") (unpack t) :)+  | isEnabled Ext_raw_html opts = (RawBlock (Format "html") (unpack t) :)   | otherwise                 = id -- Note:  the cmark parser will never generate CUSTOM_BLOCK, -- so we don't need to handle it:@@ -210,15 +206,15 @@         samekind _   ' ' = False         samekind _  _    = True         toinl (' ':_) = Space-        toinl xs      = Str $ if enabled Ext_emoji opts+        toinl xs      = Str $ if isEnabled Ext_emoji opts                                  then convertEmojis xs                                  else xs addInline _ (Node _ LINEBREAK _) = (LineBreak :) addInline opts (Node _ SOFTBREAK _)-  | enabled Ext_hard_line_breaks opts = (LineBreak :)+  | isEnabled Ext_hard_line_breaks opts = (LineBreak :)   | otherwise                           = (SoftBreak :) addInline opts (Node _ (HTML_INLINE t) _)-  | enabled Ext_raw_html opts = (RawInline (Format "html") (unpack t) :)+  | isEnabled Ext_raw_html opts = (RawInline (Format "html") (unpack t) :)   | otherwise                 = id -- Note:  the cmark parser will never generate CUSTOM_BLOCK, -- so we don't need to handle it:

@@ -194,7 +194,7 @@    endOfPara     = try $ blankline >> skipMany1 blankline    startOf      :: PandocMonad m => CRLParser m a -> CRLParser m ()    startOf p     = try $ blankline >> p >> return mempty-   startOfList   = startOf $ anyList 1+   startOfList   = startOf $ anyListItem 1    startOfTable  = startOf table    startOfHeader = startOf header    startOfNowiki = startOf nowiki

@@ -429,11 +429,11 @@ headerLevel tagtype =   case safeRead (T.unpack (T.drop 1 tagtype)) of         Just level ->-          try (do-            guardEnabled Ext_epub_html_exts-            asks inChapter >>= guard-            return (level - 1))-            <|>+--          try (do+--            guardEnabled Ext_epub_html_exts+--            asks inChapter >>= guard+--            return (level - 1))+--            <|>               return level         Nothing -> fail "Could not retrieve header level" 

@@ -1337,8 +1337,8 @@   , ("bar", lit "|")   , ("textless", lit "<")   , ("textgreater", lit ">")-  , ("thanks", note <$> grouped block)-  , ("footnote", note <$> grouped block)+  , ("thanks", skipopts >> note <$> grouped block)+  , ("footnote", skipopts >> note <$> grouped block)   , ("verb", doverb)   , ("lstinline", dolstinline)   , ("Verb", doverb)@@ -2003,7 +2003,7 @@ blockCommands :: PandocMonad m => M.Map Text (LP m Blocks) blockCommands = M.fromList $    [ ("par", mempty <$ skipopts)-   , ("parbox",  braced >> grouped blocks)+   , ("parbox",  skipopts >> braced >> grouped blocks)    , ("title", mempty <$ (skipopts *>                              (grouped inline >>= addMeta "title")                          <|> (grouped block >>= addMeta "title")))@@ -2406,8 +2406,7 @@         case safeRead ds of               Just w  -> return w               Nothing -> return 0.0-  let alignSpec = try $ do-        spaces+  let alignSpec = do         pref <- option [] alignPrefix         spaces         al <- alignChar@@ -2418,10 +2417,21 @@         spaces         suff <- option [] alignSuffix         return (al, width, (pref, suff))+  let starAlign = do -- '*{2}{r}' == 'rr', we just expand like a macro+        symbol '*'+        spaces+        ds <- trim . toksToString <$> braced+        spaces+        spec <- braced+        case safeRead ds of+             Just n  -> do+               getInput >>= setInput . (mconcat (replicate n spec) ++)+             Nothing -> fail $ "Could not parse " ++ ds ++ " as number"   bgroup   spaces   maybeBar-  aligns' <- many (alignSpec <* maybeBar)+  aligns' <- many $ try $ spaces >> optional starAlign >>+                            (alignSpec <* maybeBar)   spaces   egroup   spaces

@@ -189,7 +189,8 @@                        , definitionList                        , table                        , commentTag-                       , noteBlock+                       , amuseNoteBlock+                       , emacsNoteBlock                        ]  comment :: PandocMonad m => MuseParser m (F Blocks)@@ -308,10 +309,28 @@   char '['   many1Till digit $ char ']' -noteBlock :: PandocMonad m => MuseParser m (F Blocks)-noteBlock = try $ do+-- Amusewiki version of note+-- Parsing is similar to list item, except that note marker is used instead of list marker+amuseNoteBlock :: PandocMonad m => MuseParser m (F Blocks)+amuseNoteBlock = try $ do+  guardEnabled Ext_amuse   pos <- getPosition   ref <- noteMarker <* skipSpaces+  content <- listItemContents $ 2 + length ref+  oldnotes <- stateNotes' <$> getState+  case M.lookup ref oldnotes of+    Just _  -> logMessage $ DuplicateNoteReference ref pos+    Nothing -> return ()+  updateState $ \s -> s{ stateNotes' = M.insert ref (pos, content) oldnotes }+  return mempty++-- Emacs version of note+-- Notes are allowed only at the end of text, no indentation is required.+emacsNoteBlock :: PandocMonad m => MuseParser m (F Blocks)+emacsNoteBlock = try $ do+  guardDisabled Ext_amuse+  pos <- getPosition+  ref <- noteMarker <* skipSpaces   content <- mconcat <$> blocksTillNote   oldnotes <- stateNotes' <$> getState   case M.lookup ref oldnotes of@@ -373,12 +392,11 @@   st <- stateParserContext <$> getState   getPosition >>= \pos -> guard (st == ListItemState || sourceColumn pos /= 1)   markerLength <- marker-  postWhitespace <- length <$> many1 spaceChar-  return $ preWhitespace + markerLength + postWhitespace+  many1 spaceChar+  return $ preWhitespace + markerLength + 1 -listItem :: PandocMonad m => MuseParser m Int -> MuseParser m (F Blocks)-listItem start = try $ do-  markerLength <- start+listItemContents :: PandocMonad m => Int -> MuseParser m (F Blocks)+listItemContents markerLength = do   firstLine <- anyLineNewline   restLines <- many $ listLine markerLength   blank <- option "" ("\n" <$ blankline)@@ -386,6 +404,11 @@   rest <- many $ listContinuation markerLength   parseFromString (withListContext parseBlocks) $ concat (first:rest) ++ "\n" +listItem :: PandocMonad m => MuseParser m Int -> MuseParser m (F Blocks)+listItem start = try $ do+  markerLength <- start+  listItemContents markerLength+ bulletListItems :: PandocMonad m => MuseParser m (F [Blocks]) bulletListItems = sequence <$> many1 (listItem bulletListStart) @@ -423,7 +446,8 @@   pure $ do lineContent' <- lineContent             pure (B.text term, [lineContent'])   where-    termParser = many1 spaceChar >> -- Initial space as required by Amusewiki, but not Emacs Muse+    termParser = (guardDisabled Ext_amuse <|> void spaceChar) >> -- Initial space is required by Amusewiki, but not Emacs Muse+                 many spaceChar >>                  many1Till anyChar (lookAhead (void (try (spaceChar >> string "::")) <|> void newline))     endOfInput = try $ skipMany blankline >> skipSpaces >> eof     twoBlankLines = try $ blankline >> skipMany1 blankline@@ -532,25 +556,35 @@ -- inline parsers -- +inlineList :: PandocMonad m => [MuseParser m (F Inlines)]+inlineList = [ endline+             , br+             , anchor+             , footnote+             , strong+             , strongTag+             , emph+             , emphTag+             , superscriptTag+             , subscriptTag+             , strikeoutTag+             , verbatimTag+             , link+             , code+             , codeTag+             , whitespace+             , str+             , symbol+             ]+ inline :: PandocMonad m => MuseParser m (F Inlines)-inline = choice [ br-                , anchor-                , footnote-                , strong-                , strongTag-                , emph-                , emphTag-                , superscriptTag-                , subscriptTag-                , strikeoutTag-                , verbatimTag-                , link-                , code-                , codeTag-                , whitespace-                , str-                , symbol-                ] <?> "inline"+inline = (choice inlineList) <?> "inline"++endline :: PandocMonad m => MuseParser m (F Inlines)+endline = try $ do+  newline+  notFollowedBy blankline+  returnF B.softbreak  anchor :: PandocMonad m => MuseParser m (F Inlines) anchor = try $ do

@@ -658,6 +658,7 @@   body <- option "" $ try $ blanklines >> indentedBlock   optional blanklines   let body' = body ++ "\n\n"+      name = trim $ fromMaybe "" (lookup "name" fields)       imgAttr cl = ("", classes, widthAttr ++ heightAttr)         where           classes = words $ maybe "" trim (lookup cl fields) ++@@ -691,7 +692,8 @@         "line-block" -> lineBlockDirective body'         "raw" -> return $ B.rawBlock (trim top) (stripTrailingNewlines body)         "role" -> addNewRole top $ map (second trim) fields-        "container" -> parseFromString' parseBlocks body'+        "container" -> B.divWith (name, "container" : words top, []) <$>+                            parseFromString' parseBlocks body'         "replace" -> B.para <$>  -- consumed by substKey                    parseInlineFromString (trim top)         "unicode" -> B.para <$>  -- consumed by substKey

@@ -434,7 +434,7 @@                 case epubCoverImage metadata of                      Nothing   -> return ([],[])                      Just img  -> do-                       let coverImage = "media/" ++ takeFileName img+                       let coverImage = takeFileName img                        cpContent <- lift $ writeHtml                             opts'{ writerVariables =                                     ("coverpage","true"):@@ -667,7 +667,7 @@              [ unode "reference" !                    [("type","cover")                    ,("title","Cover")-                   ,("href","cover.xhtml")] $ ()+                   ,("href","text/cover.xhtml")] $ ()                | isJust (epubCoverImage metadata)              ]           ]@@ -770,7 +770,7 @@                                           ,("hidden","hidden")] $                             [ unode "ol" $                               [ unode "li"-                                [ unode "a" ! [("href", "cover.xhtml")+                                [ unode "a" ! [("href", "text/cover.xhtml")                                               ,("epub:type", "cover")] $                                   "Cover"] |                                   epubCoverImage metadata /= Nothing

@@ -575,8 +575,6 @@   let padRow r = case numcols - length r of                        x | x > 0 -> r ++ replicate x empty                          | otherwise -> r-  rawHeaders <- padRow <$> mapM (blockListToMarkdown opts) headers-  rawRows <- mapM (fmap padRow . mapM (blockListToMarkdown opts)) rows   let aligns' = case numcols - length aligns of                      x | x > 0 -> aligns ++ replicate x AlignDefault                        | otherwise -> aligns@@ -586,16 +584,25 @@   (nst,tbl) <-      case True of           _ | isSimple &&-              isEnabled Ext_simple_tables opts -> fmap (nest 2,) $-                   pandocTable opts False (all null headers) aligns' widths'-                       rawHeaders rawRows+              isEnabled Ext_simple_tables opts -> do+                rawHeaders <- padRow <$> mapM (blockListToMarkdown opts) headers+                rawRows <- mapM (fmap padRow . mapM (blockListToMarkdown opts))+                           rows+                (nest 2,) <$> pandocTable opts False (all null headers)+                                aligns' widths' rawHeaders rawRows             | isSimple &&-              isEnabled Ext_pipe_tables opts -> fmap (id,) $-                   pipeTable (all null headers) aligns' rawHeaders rawRows+              isEnabled Ext_pipe_tables opts -> do+                rawHeaders <- padRow <$> mapM (blockListToMarkdown opts) headers+                rawRows <- mapM (fmap padRow . mapM (blockListToMarkdown opts))+                           rows+                (id,) <$> pipeTable (all null headers) aligns' rawHeaders rawRows             | not hasBlocks &&-              isEnabled Ext_multiline_tables opts -> fmap (nest 2,) $-                   pandocTable opts True (all null headers) aligns' widths'-                       rawHeaders rawRows+              isEnabled Ext_multiline_tables opts -> do+                rawHeaders <- padRow <$> mapM (blockListToMarkdown opts) headers+                rawRows <- mapM (fmap padRow . mapM (blockListToMarkdown opts))+                           rows+                (nest 2,) <$> pandocTable opts True (all null headers)+                                aligns' widths' rawHeaders rawRows             | isEnabled Ext_grid_tables opts &&                writerColumns opts >= 8 * numcols -> (id,) <$>                 gridTable opts blockListToMarkdown@@ -604,8 +611,11 @@                    (text . T.unpack) <$>                    (writeHtml5String def $ Pandoc nullMeta [t])             | hasSimpleCells &&-              isEnabled Ext_pipe_tables opts -> fmap (id,) $-                   pipeTable (all null headers) aligns' rawHeaders rawRows+              isEnabled Ext_pipe_tables opts -> do+                rawHeaders <- padRow <$> mapM (blockListToMarkdown opts) headers+                rawRows <- mapM (fmap padRow . mapM (blockListToMarkdown opts))+                           rows+                (id,) <$> pipeTable (all null headers) aligns' rawHeaders rawRows             | otherwise -> return $ (id, text "[TABLE]")   return $ nst $ tbl $$ caption'' $$ blankline blockToMarkdown' opts (BulletList items) = do

@@ -9,6 +9,7 @@ extra-deps: - pandoc-types-1.17.3 - hslua-0.9.2+- hslua-module-text-0.1.2 - skylighting-0.4.3.2 - texmath-0.10 - cmark-gfm-0.1.1

@@ -7,9 +7,9 @@ import Test.Tasty.HUnit (Assertion, assertEqual, testCase) import Test.Tasty.QuickCheck (QuickCheckTests (..), ioProperty, testProperty) import Text.Pandoc.Arbitrary ()-import Text.Pandoc.Builder (bulletList, doc, doubleQuoted, emph, linebreak,-                            para, plain, rawBlock, singleQuoted, space, str,-                            strong, (<>))+import Text.Pandoc.Builder (bulletList, divWith, doc, doubleQuoted, emph,+                            header, linebreak, para, plain, rawBlock,+                            singleQuoted, space, str, strong, (<>)) import Text.Pandoc.Class (runIOorExplode) import Text.Pandoc.Definition (Block, Inline, Meta, Pandoc) import Text.Pandoc.Lua@@ -77,6 +77,20 @@       "block-count.lua"       (doc $ para "one" <> para "two")       (doc $ para "2")++  , testCase "Convert header upper case" $+    assertFilterConversion "converting header to upper case failed"+      "uppercase-header.lua"+      (doc $ header 1 "les états-unis" <> para "text")+      (doc $ header 1 "LES ÉTATS-UNIS" <> para "text")++  , testCase "Attribute lists are convenient to use" $+    let kv_before = [("one", "1"), ("two", "2"), ("three", "3")]+        kv_after  = [("one", "eins"), ("three", "3"), ("five", "5")]+    in assertFilterConversion "Attr doesn't behave as expected"+      "attr-test.lua"+      (doc $ divWith ("", [], kv_before) (para "nil"))+      (doc $ divWith ("", [], kv_after) (para "nil"))   ]  assertFilterConversion :: String -> FilePath -> Pandoc -> Pandoc -> Assertion

@@ -10,16 +10,16 @@ import Text.Pandoc.Arbitrary () import Text.Pandoc.Builder -muse :: Text -> Pandoc-muse = purely $ \s -> do-  setInputFiles ["in"]-  setOutputFile (Just "out")-  readMuse def s+amuse :: Text -> Pandoc+amuse = purely $ readMuse def { readerExtensions = extensionsFromList [Ext_amuse]} +emacsMuse :: Text -> Pandoc+emacsMuse = purely $ readMuse def { readerExtensions = emptyExtensions }+ infix 4 =: (=:) :: ToString c      => String -> (Text, c) -> TestTree-(=:) = test muse+(=:) = test amuse  spcSep :: [Inlines] -> Inlines spcSep = mconcat . intersperse space@@ -76,7 +76,7 @@                   , ""                   , "Fourth line</em>"                   ] =?>-        para "First line <em>Second line" <>+        para "First line\n<em>Second line" <>         para "Fourth line</em>"        , "Linebreak" =: "Line <br>  break" =?> para ("Line" <> linebreak <> "break")@@ -168,12 +168,12 @@           T.unlines [ "First line"                     , "second line."                     ] =?>-          para "First line second line."+          para "First line\nsecond line."         , "Indented paragraph" =:           T.unlines [ " First line"                     , "second line."                     ] =?>-          para "First line second line."+          para "First line\nsecond line."         -- Emacs Muse starts a blockquote on the second line.         -- We copy Amusewiki behavior and require a blank line to start a blockquote.         , "Indentation in the middle of paragraph" =:@@ -181,7 +181,7 @@                      , "  second line"                      , "third line"                      ] =?>-           para "First line second line third line"+           para "First line\nsecond line\nthird line"         , "Quote" =:           "  This is a quotation\n" =?>           blockQuote (para "This is a quotation")@@ -189,7 +189,7 @@           T.unlines [ "  This is a quotation"                     , "  with a continuation"                     ] =?>-          blockQuote (para "This is a quotation with a continuation")+          blockQuote (para "This is a quotation\nwith a continuation")         , testGroup "Div"           [ "Div without id" =:             "<div>Foo bar</div>" =?>@@ -359,7 +359,7 @@           T.unlines [ "Paragraph starts here"                     , "#anchor and ends here."                     ] =?>-          para ("Paragraph starts here " <> spanWith ("anchor", [], []) mempty <> "and ends here.")+          para ("Paragraph starts here\n" <> spanWith ("anchor", [], []) mempty <> "and ends here.")         ]       , testGroup "Footnotes"         [ "Simple footnote" =:@@ -377,6 +377,43 @@                     ] =?>           para (text "Start recursion here" <>                 note (para "Recursion continues here[1]"))+        , testGroup "Multiparagraph footnotes"+          [ "Amusewiki multiparagraph footnotes" =:+            T.unlines [ "Multiparagraph[1] footnotes[2]"+                      , ""+                      , "[1] First footnote paragraph"+                      , ""+                      , "    Second footnote paragraph"+                      , "Not a note"+                      , "[2] Second footnote"+                      ] =?>+            para (text "Multiparagraph" <>+                  note (para "First footnote paragraph" <>+                        para "Second footnote paragraph") <>+                  text " footnotes" <>+                  note (para "Second footnote")) <>+            para (text "Not a note")+          , test emacsMuse "Emacs multiparagraph footnotes"+            (T.unlines+              [ "First footnote reference[1] and second footnote reference[2]."+              , ""+              , "[1] First footnote paragraph"+              , ""+              , "Second footnote"+              , "paragraph"+              , ""+              , "[2] Third footnote paragraph"+              , ""+              , "Fourth footnote paragraph"+              ] =?>+            para (text "First footnote reference" <>+                  note (para "First footnote paragraph" <>+                        para "Second footnote\nparagraph") <>+                  text " and second footnote reference" <>+                  note (para "Third footnote paragraph" <>+                        para "Fourth footnote paragraph") <>+                  text "."))+          ]         ]       ]     , testGroup "Tables"@@ -490,28 +527,42 @@          orderedListWith (1, Decimal, Period) [ para "Item1"                                               , para "Item2"                                               ]-      , "Nested list" =:-         T.unlines-           [ " - Item1"-           , "   - Item2"-           , "   - Item3"-           , " - Item4"-           , "   1. Nested"-           , "   2. Ordered"-           , "   3. List"-           ] =?>-         bulletList [ mconcat [ para "Item1"-                              , bulletList [ para "Item2"-                                           , para "Item3"-                                           ]-                              ]-                    , mconcat [ para "Item4"-                              , orderedListWith (1, Decimal, Period) [ para "Nested"-                                                                     , para "Ordered"-                                                                     , para "List"-                                                                     ]-                              ]-                    ]+      , testGroup "Nested lists"+        [ "Nested list" =:+          T.unlines+            [ " - Item1"+            , "   - Item2"+            , "   - Item3"+            , " - Item4"+            , "   1. Nested"+            , "   2. Ordered"+            , "   3. List"+            ] =?>+          bulletList [ mconcat [ para "Item1"+                               , bulletList [ para "Item2"+                                            , para "Item3"+                                            ]+                               ]+                     , mconcat [ para "Item4"+                               , orderedListWith (1, Decimal, Period) [ para "Nested"+                                                                      , para "Ordered"+                                                                      , para "List"+                                                                      ]+                               ]+                     ]+        , "Incorrectly indented Text::Amuse nested list" =:+          T.unlines+            [ " - First item"+            , "  - Not nested item"+            ] =?>+          bulletList [ para "First item", para "Not nested item"]+        , "Text::Amuse includes only one space in list marker" =:+          T.unlines+            [ " -    First item"+            , "   - Nested item"+            ] =?>+          bulletList [ para "First item" <> bulletList [ para "Nested item"]]+        ]       , "List continuation" =:          T.unlines            [ " - a"@@ -560,7 +611,7 @@             , "   bar"             , " - Baz"             ] =?>-          bulletList [ para "Foo bar"+          bulletList [ para "Foo\nbar"                      , para "Baz"                      ]         , "One blank line after multiline first item" =:@@ -570,7 +621,7 @@             , ""             , " - Baz"             ] =?>-          bulletList [ para "Foo bar"+          bulletList [ para "Foo\nbar"                      , para "Baz"                      ]         , "Two blank lines after multiline first item" =:@@ -581,7 +632,7 @@             , ""             , " - Baz"             ] =?>-          bulletList [ para "Foo bar" ] <> bulletList [ para "Baz" ]+          bulletList [ para "Foo\nbar" ] <> bulletList [ para "Baz" ]         , "No blank line after list continuation" =:           T.unlines             [ " - Foo"@@ -621,7 +672,15 @@           [ "First :: second"           , "Foo :: bar"           ] =?>-        para "First :: second Foo :: bar"+        para "First :: second\nFoo :: bar"+      , test emacsMuse "Emacs Muse definition list"+        (T.unlines+          [ "First :: second"+          , "Foo :: bar"+          ] =?>+        definitionList [ ("First", [ para "second" ])+                       , ("Foo", [ para "bar" ])+                       ])       , "Definition list" =:         T.unlines           [ " First :: second"@@ -643,7 +702,7 @@           , "and its continuation."           , " Second term :: Definition of second term."           ] =?>-        definitionList [ ("First term", [ para "Definition of first term and its continuation." ])+        definitionList [ ("First term", [ para "Definition of first term\nand its continuation." ])                        , ("Second term", [ para "Definition of second term." ])                        ]       -- Emacs Muse creates two separate lists when indentation of items is different.@@ -654,7 +713,7 @@           , "and its continuation."           , "   Second term :: Definition of second term."           ] =?>-        definitionList [ ("First term", [ para "Definition of first term and its continuation." ])+        definitionList [ ("First term", [ para "Definition of first term\nand its continuation." ])                        , ("Second term", [ para "Definition of second term." ])                        ]       , "Two blank lines separate definition lists" =:

@@ -8,7 +8,9 @@ import Text.Pandoc.Builder  muse :: (ToPandoc a) => a -> String-muse = museWithOpts def{ writerWrapText = WrapNone }+muse = museWithOpts def{ writerWrapText = WrapNone,+                         writerExtensions = extensionsFromList [Ext_amuse,+                                                                Ext_auto_identifiers] }  museWithOpts :: (ToPandoc a) => WriterOptions -> a -> String museWithOpts opts = unpack . purely (writeMuse opts) . toPandoc

@@ -0,0 +1,24 @@+```+% pandoc -f markdown -t native+\parbox[t]{0.4\textwidth}{+\begin{shaded}+\end{shaded}+}+^D+[RawBlock (Format "latex") "\\parbox[t]{0.4\\textwidth}{\n\\begin{shaded}\n\\end{shaded}\n}"]+```++```+% pandoc -f latex -t native+\begin{tabular}{l*{2}{r}}+Blah & Foo & Bar \\+\end{tabular}+^D+[Table [] [AlignLeft,AlignRight,AlignRight] [0.0,0.0,0.0]+ [[]+ ,[]+ ,[]]+ [[[Plain [Str "Blah"]]+  ,[Plain [Str "Foo"]]+  ,[Plain [Str "Bar"]]]]]+```

@@ -0,0 +1,14 @@+```+% pandoc -t markdown-simple_tables-multiline_tables-pipe_tables++-----------------------------------++| Text [^1]                         |++-----------------------------------+++[^1]: Footnote.+^D++-----------------------------------++| Text [^1]                         |++-----------------------------------+++[^1]: Footnote.+```

@@ -0,0 +1,6 @@+```+% pandoc -t latex+Sentence blah.\footnote[][-.5in]{I'm a footnote}+^D+Sentence blah.\footnote[][-.5in]{I'm a footnote}+```

@@ -0,0 +1,9 @@+```+pandoc -f mediawiki -t native+[https://domain.com/script.php?a=1&b=2&c=&d=4 open productname bugs]++[http://domain.com?a=. open productname bugs]+^D+[Para [Link ("",[],[]) [Str "open",Space,Str "productname",Space,Str "bugs"] ("https://domain.com/script.php?a=1&b=2&c=&d=4","")]+,Para [Str "[",Link ("",[],[]) [Str "http://domain.com?a="] ("http://domain.com?a=",""),Str ".",Space,Str "open",Space,Str "productname",Space,Str "bugs]"]]+```

@@ -0,0 +1,6 @@+function Div (div)+  div.attributes.five = ("%d"):format(div.attributes.two + div.attributes.three)+  div.attributes.two = nil+  div.attributes.one = "eins"+  return div+end

@@ -0,0 +1,9 @@+local text = require 'text'++local function str_to_uppercase (s)+  return pandoc.Str(text.upper(s.text))+end++function Header (el)+  return pandoc.walk_block(el, {Str = str_to_uppercase})+end