diff --git a/AUTHORS.md b/AUTHORS.md --- a/AUTHORS.md +++ b/AUTHORS.md @@ -78,6 +78,7 @@ - Christoffer Ackelman - Christoffer Sawicki - Christophe Dervieux +- Christopher Kenny - Clare Macrae - Clint Adams - Conal Elliott @@ -113,6 +114,7 @@ - Eric Kow - Eric Schrijver - Eric Seidel +- Erik Post - Erik Rask - Ethan Riley - Étienne Bersac @@ -342,11 +344,14 @@ - Raniere Silva - Raymond Ehlers - Recai Oktaş +- Repetitive +- Reuben Thomas - Rowan Rodrik van der Molen - Roland Hieber - Roman Beránek - Ruqi - RyanGlScott +- Ryan Gibb - S.P.H - Salim B - Sam S. Almahri @@ -356,6 +361,7 @@ - Santiago Zarate - Sascha Wilde - Scott Morrison +- Sean Soon - Sebastian Talmon - Sebbones - Sen-wen Deng @@ -465,6 +471,7 @@ - lux-lth - luz paz - lwolfsonkin +- massifrg - mbracke - mbrackeantidot - mh4ckt3mh4ckt1c4s diff --git a/MANUAL.txt b/MANUAL.txt --- a/MANUAL.txt +++ b/MANUAL.txt @@ -1,7 +1,7 @@ --- title: Pandoc User's Guide author: John MacFarlane -date: 2025-05-28 +date: 2025-09-06 --- # Synopsis @@ -156,7 +156,9 @@ for [typography] if the `csquotes` variable or metadata field is set to a true value. The [`natbib`], [`biblatex`], [`bibtex`], and [`biber`] packages can optionally be used for [citation -rendering]. The following packages will be used to improve +rendering]. If math with `\cancel`, `\bcancel`, or `\xcancel` +is used, the [`cancel`] package is needed. +The following packages will be used to improve output quality if present, but pandoc does not require them to be present: [`upquote`] (for straight quotes in verbatim environments), [`microtype`] (for better spacing adjustments), @@ -182,6 +184,7 @@ [`framed`]: https://ctan.org/pkg/framed [`geometry`]: https://ctan.org/pkg/geometry [`graphicx`]: https://ctan.org/pkg/graphicx +[`cancel`]: https://ctan.org/pkg/cancel [`hyperref`]: https://ctan.org/pkg/hyperref [`iftex`]: https://ctan.org/pkg/iftex [`listings`]: https://ctan.org/pkg/listings @@ -279,6 +282,7 @@ - `twiki` ([TWiki markup]) - `typst` ([typst]) - `vimwiki` ([Vimwiki]) + - `xml` (XML version of native AST) - the path of a custom Lua reader, see [Custom readers and writers] below ::: @@ -357,6 +361,7 @@ - `s5` ([S5] HTML and JavaScript slide show) - `tei` ([TEI Simple]) - `typst` ([typst]) + - `xml` (XML version of native AST) - `xwiki` ([XWiki markup]) - `zimwiki` ([ZimWiki markup]) - the path of a custom Lua writer, see [Custom readers and writers] below @@ -461,7 +466,7 @@ `--list-highlight-styles` : List supported styles for syntax highlighting, one per line. - See `--highlight-style`. + See `--syntax-highlighting`. `-v`, `--version` @@ -922,18 +927,32 @@ inside raw HTML blocks when the `markdown_in_html_blocks` extension is not set. -`--no-highlight` +`--syntax-highlighting="default"|"none"|"idiomatic"|`*STYLE*`|`*FILE* -: Disables syntax highlighting for code blocks and inlines, even when - a language attribute is given. +: The method to use for code syntax highlighting. Setting a + specific *STYLE* causes highlighting to be performed with the + internal highlighting engine, using KDE syntax definitions and + styles. The `"idiomatic"` method uses a format-specific + highlighter if one is available, or the default style if the + target format has no idiomatic highlighting method. Setting this + option to `none` disables all syntax highlighting. The + `"default"` method uses a format-specific default. -`--highlight-style=`*STYLE*|*FILE* + The default for HTML, EPUB, Docx, Ms, Man, and LaTeX output is + to use the internal highlighter with the default style; Typst + output relies on Typst's own syntax highlighting system by + default. -: Specifies the coloring style to be used in highlighted source code. - Options are `pygments` (the default), `kate`, `monochrome`, - `breezeDark`, `espresso`, `zenburn`, `haddock`, and `tango`. - For more information on syntax highlighting in pandoc, see - [Syntax highlighting], below. See also + The [`listings`][] LaTeX package is used for idiomatic + highlighting in LaTeX. The package does not support multi-byte + encoding for source code. To handle UTF-8 you would need to + use a custom template. This issue is fully documented here: + [Encoding issue with the listings package][]. + + Style options are `pygments` (the default), `kate`, + `monochrome`, `breezeDark`, `espresso`, `zenburn`, `haddock`, + and `tango`. For more information on syntax highlighting in + pandoc, see [Syntax highlighting], below. See also `--list-highlight-styles`. Instead of a *STYLE* name, a JSON file with extension @@ -944,11 +963,24 @@ To generate the JSON version of an existing style, use `--print-highlight-style`. +`--no-highlight` + +: *Deprecated, use `--syntax-highlighting=none` instead.* + + Disables syntax highlighting for code blocks and inlines, even when + a language attribute is given. + +`--highlight-style=`*STYLE*|*FILE* + +: _Deprecated, use `--syntax-highlighting=`*STYLE*|*FILE* instead._ + + Specifies the coloring style to be used in highlighted source code. + `--print-highlight-style=`*STYLE*|*FILE* : Prints a JSON version of a highlighting style, which can be modified, saved with a `.theme` extension, and used - with `--highlight-style`. This option may be used with + with `--syntax-highlighting`. This option may be used with `-o`/`--output` to redirect output to a file, but `-o`/`--output` must come before `--print-highlight-style` on the command line. @@ -1171,7 +1203,10 @@ `--listings[=true|false]` -: Use the [`listings`] package for LaTeX code blocks. The package +: *Deprecated, use `--syntax-highlighting=idiomatic` or + `--syntax-highlighting=default` instead. + + Use the [`listings`] package for LaTeX code blocks. The package does not support multi-byte encoding for source code. To handle UTF-8 you would need to use a custom template. This issue is fully documented here: [Encoding issue with the listings package]. @@ -1700,6 +1735,7 @@ 92 PandocUTF8DecodingError 93 PandocIpynbDecodingError 94 PandocUnsupportedCharsetError + 95 PandocInputNotTextError 97 PandocCouldNotFindDataFileError 98 PandocCouldNotFindMetadataFileError 99 PandocResourceNotFound @@ -1975,11 +2011,11 @@ | ``` | ``` | +----------------------------------+-----------------------------------+ | ``` | ``` yaml | -| --no-highlight | highlight-style: null | +| --no-highlight | syntax-highlighting: 'none' | | ``` | ``` | +----------------------------------+-----------------------------------+ | ``` | ``` yaml | -| --highlight-style kate | highlight-style: kate | +| --syntax-highlighting kate | syntax-highlighting: kate | | ``` | ``` | +----------------------------------+-----------------------------------+ | ``` | ``` yaml | @@ -2728,7 +2764,7 @@ documents `abstract-title` -: title of abstract, currently used only in HTML, EPUB, and docx. +: title of abstract, currently used only in HTML, EPUB, docx, and Typst. This will be set automatically to a localized value, depending on `lang`, but can be manually overridden. @@ -3207,6 +3243,16 @@ `natbiboptions` : list of options for natbib +#### Other + +`pdf-trailer-id` +: the PDF trailer ID; must be two PDF byte strings if set, + conventionally with 16 bytes each. E.g., + `<00112233445566778899aabbccddeeff> + <00112233445566778899aabbccddeeff>`. + + See the section on [reproducible builds]. + [KOMA-Script]: https://ctan.org/pkg/koma-script [LaTeX Font Catalogue]: https://tug.org/FontCatalogue/ [LaTeX font encodings guide]: https://ctan.org/pkg/encguide @@ -3313,14 +3359,14 @@ `includesource` : include all source documents as file attachments in the PDF file -[ConTeXt Paper Setup]: https://wiki.contextgarden.net/PaperSetup -[ConTeXt Layout]: https://wiki.contextgarden.net/Layout -[ConTeXt Font Switching]: https://wiki.contextgarden.net/Font_Switching +[ConTeXt Paper Setup]: https://wiki.contextgarden.net/Document_layout_and_layers/Paper_setup +[ConTeXt Layout]: https://wiki.contextgarden.net/Document_layout_and_layers/Tutorials +[ConTeXt Font Switching]: https://wiki.contextgarden.net/Characters_words_and_fonts/Tutorials [ConTeXt Color]: https://wiki.contextgarden.net/Color -[ConTeXt Headers and Footers]: https://wiki.contextgarden.net/Headers_and_Footers -[ConTeXt Indentation]: https://wiki.contextgarden.net/Indentation -[ConTeXt ICC Profiles]: https://wiki.contextgarden.net/PDFX#ICC_profiles -[ConTeXt PDFA]: https://wiki.contextgarden.net/PDF/A +[ConTeXt Headers and Footers]: https://wiki.contextgarden.net/Document_layout_and_layers/Headers_and_footers +[ConTeXt Indentation]: https://wiki.contextgarden.net/Text_blocks/Typography/Indentation +[ConTeXt PDFA]: https://wiki.contextgarden.net/Input_and_compilation/PDF/PDFA +[ConTeXt ICC Profiles]: https://wiki.contextgarden.net/Input_and_compilation/PDF/PDFX#ICC_profiles [`setupwhitespace`]: https://wiki.contextgarden.net/Command/setupwhitespace [`setupinterlinespace`]: https://wiki.contextgarden.net/Command/setupinterlinespace [`setuppagenumbering`]: https://wiki.contextgarden.net/Command/setuppagenumbering @@ -3391,6 +3437,19 @@ `columns` : Number of columns for body text. +`thanks` +: contents of acknowledgments footnote after document title + +`mathfont`, `codefont` +: Name of system font to use for math and code, respectively. + +`linestretch` +: adjusts line spacing, e.g. `1.25`, `1.5` + +`linkcolor`, `filecolor`, `citecolor` +: color for external links, internal links, and citation links, + respectively: expects a hexadecimal color code + ### Variables for ms `fontfamily` @@ -3528,7 +3587,7 @@ `html` output formats -: `markdown`, `latex`, `context`, `rst` +: `markdown`, `latex`, `context`, `org`, `rst` enabled by default in : `markdown`, `latex`, `context` (both input and output) @@ -3878,6 +3937,21 @@ Natural tables allow more fine-grained global customization but come at a performance penalty compared to extreme tables. +### Extension: `smart_quotes` (org) ### + +Interpret straight quotes as curly quotes during parsing. When +*writing* Org, then the `smart_quotes` extension has the reverse +effect: what would have been curly quotes comes out straight. + +This extension is implied if `smart` is enabled. + +### Extension: `special_strings` (org) ### + +Interpret `---` as em-dashes, `--` as en-dashes, `\-` as shy +hyphen, and `...` as ellipses. + +This extension is implied if `smart` is enabled. + ### Extension: `tagging` ### {#extension--tagging} Enabling this extension with `context` output will produce markup @@ -4233,10 +4307,10 @@ class attribute will be printed after the opening fence as a bare word. -To prevent all highlighting, use the `--no-highlight` flag. -To set the highlighting style, use `--highlight-style`. -For more information on highlighting, see [Syntax highlighting], -below. +To prevent all highlighting, use the `--syntax-highlighting=none` +option. To set the highlighting style or method, use +`--syntax-highlighting`. For more information on highlighting, see +[Syntax highlighting], below. ## Line blocks @@ -4502,9 +4576,6 @@ ~ Definition 2b Note that space between items in a definition list is required. -(A variant that loosens this requirement, but disallows "lazy" -hard wrapping, can be activated with the [`compact_definition_lists` -extension][Extension: `compact_definition_lists`].) [^3]: I have been influenced by the suggestions of [David Wheeler](https://justatheory.com/2009/02/modest-markdown-proposal/). @@ -5615,10 +5686,14 @@ ![This is the caption](/url/of/image.png) How this is rendered depends on the output format. Some output -formats (e.g. RTF) do not yet support figures. In those +formats (e.g. RTF) do not yet support figures. In those formats, you'll just get an image in a paragraph by itself, with -no caption. +no caption. For LaTeX output, you can specify a [figure's +positioning](https://www.overleaf.com/learn/latex/Positioning_images_and_tables#The_figure_environment) +by adding the `latex-placement` attribute. + ![This is the caption](/url/of/image.png){latex-placement="ht"} + If you just want a regular inline image, just make sure it is not the only thing in the paragraph. One way to do this is to insert a nonbreaking space after the image: @@ -6109,33 +6184,6 @@ Parses MultiMarkdown-style heading identifiers (in square brackets, after the heading but before any trailing `#`s in an ATX heading). -### Extension: `compact_definition_lists` ### - -Activates the definition list syntax of pandoc 1.12.x and earlier. -This syntax differs from the one described above under [Definition lists] -in several respects: - - - No blank line is required between consecutive items of the - definition list. - - To get a "tight" or "compact" list, omit space between consecutive - items; the space between a term and its definition does not affect - anything. - - Lazy wrapping of paragraphs is not allowed: the entire definition must - be indented four spaces.[^6] - -[^6]: To see why laziness is incompatible with relaxing the requirement - of a blank line between items, consider the following example: - - bar - : definition - foo - : definition - - Is this a single list item with two definitions of "bar," the first of - which is lazily wrapped, or two list items? To remove the ambiguity - we must either disallow lazy wrapping or require a blank line between - list items. - ### Extension: `gutenberg` ### Use [Project Gutenberg] conventions for `plain` output: @@ -6439,7 +6487,7 @@ ## Placement of the bibliography If the style calls for a list of works cited, it will be placed -in a div with id `refs`, if one exists: +in a div with id `refs`, if one exists:[^note-on-refs] ::: {#refs} ::: @@ -6448,6 +6496,13 @@ Generation of the bibliography can be suppressed by setting `suppress-bibliography: true` in the YAML metadata. +[^note-on-refs]: + Note that if `--file-scope` is used, a div written this way will be + given an identifier of the form `FILE__refs`, to avoid duplicate + identifiers (see `--file-scope`). In view of this possibility, + pandoc will place the bibliography in any div whose identifier is + `refs` *or* ends with `__refs`. + If you wish the bibliography to have a section heading, you can set `reference-section-title` in the metadata, or put the heading at the beginning of the div with id `refs` (if you are using it) @@ -7370,22 +7425,23 @@ of language names that pandoc will recognize, type `pandoc --list-highlight-languages`. -The color scheme can be selected using the `--highlight-style` option. -The default color scheme is `pygments`, which imitates the default color -scheme used by the Python library pygments (though pygments is not actually -used to do the highlighting). To see a list of highlight styles, -type `pandoc --list-highlight-styles`. +The color scheme can be selected using the `--syntax-highlighting` +option. The default color scheme is `pygments`, which imitates the +default color scheme used by the Python library pygments (though +pygments is not actually used to do the highlighting). To see a +list of highlight styles, type `pandoc --list-highlight-styles`. -If you are not satisfied with the predefined styles, you can -use `--print-highlight-style` to generate a JSON `.theme` file which -can be modified and used as the argument to `--highlight-style`. To -get a JSON version of the `pygments` style, for example: +If you are not satisfied with the predefined styles, you can use +`--print-highlight-style` to generate a JSON `.theme` file which +can be modified and used as the argument to +`--syntax-highlighting`. To get a JSON version of the `pygments` +style, for example: pandoc -o my.theme --print-highlight-style pygments Then edit `my.theme` and use it like this: - pandoc --highlight-style my.theme + pandoc --syntax-highlighting my.theme If you are not satisfied with the built-in highlighting, or you want to highlight a language that isn't supported, you can use the @@ -7397,7 +7453,7 @@ If you receive an error that pandoc "Could not read highlighting theme", check that the JSON file is encoded with UTF-8 and has no Byte-Order Mark (BOM). -To disable highlighting, use the `--no-highlight` option. +To disable highlighting, use `--syntax-highlighting=none`. [skylighting]: https://github.com/jgm/skylighting @@ -7556,6 +7612,11 @@ `SOURCE_DATE_EPOCH` should contain an integer unix timestamp (specifying the number of seconds since midnight UTC January 1, 1970). +For reproducible builds with LaTeX, you can either specify the +`pdf-trailer-id` in the metadata or leave it undefined, in which +case pandoc will create a trailer-id based on a hash of the +`SOURCE_DATE_EPOCH` and the document's contents. + Some document formats also include a unique identifier. For EPUB, this can be set explicitly by setting the `identifier` metadata field (see [EPUB Metadata], above). @@ -7610,7 +7671,7 @@ ## Prince XML -The non-free HTML-to-PDf converter `prince` has extensive support +The non-free HTML-to-PDF converter `prince` has extensive support for various PDF standards as well as tagging. E.g.: pandoc --pdf-engine=prince \ diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -105,6 +105,7 @@ markup](https://twiki.org/cgi-bin/view/TWiki/TextFormattingRules)) - `typst` ([typst](https://typst.app)) - `vimwiki` ([Vimwiki](https://vimwiki.github.io)) +- `xml` (XML version of native AST) - the path of a custom Lua reader, see [Custom readers and writers](https://pandoc.org/MANUAL.html#custom-readers-and-writers) below @@ -216,6 +217,7 @@ slide show) - `tei` ([TEI Simple](https://github.com/TEIC/TEI-Simple)) - `typst` ([typst](https://typst.app)) +- `xml` (XML version of native AST) - `xwiki` ([XWiki markup](https://www.xwiki.org/xwiki/bin/view/Documentation/UserGuide/Features/XWikiSyntax/)) - `zimwiki` ([ZimWiki diff --git a/changelog.md b/changelog.md --- a/changelog.md +++ b/changelog.md @@ -1,5 +1,412 @@ # Revision history for pandoc +## pandoc 3.8 (2025-09-06) + + * Add a new input and output format `xml`, exactly representing a Pandoc + AST and isomorphic to the existing `native` and `json` formats (massifrg). + XML schemas for validation can be found in `tools/pandoc-xml.*`. + The format is documented in `doc/xml.md`. Pandoc now defaults to this + reader and writer when the `.xml` extension is used. + + Two new exported modules are added [API change]: + Text.Pandoc.Readers.XML, exporting `readXML`, and + Text.Pandoc.Writers.XML, exporting `writeXML`. + A new unexported module Text.Pandoc.XMLFormat is also added. + + * Add a new command line option `--syntax-highlighting`; this takes + the values `none`, `default`, `idiomatic`, a style name, or a path to a + theme file. It replaces the `--no-highlighting`, `--highlighting-style`, + and `--listings` options, which will still work but with a deprecation + warning. (Albert Krewinkel) + + * Create directory of output file if it doesn't exist (#11040). + + * Update `--version` copyright dates (#10961), and use a hardcoded + string "pandoc" for the program name in `--version`, per GNU + guidelines. + + * Add `smart_quotes` and `special_strings` extensions (Albert Krewinkel). + Currently these only affect `org`. Org mode makes a distinction between + smart parsing of quotes, and smart parsing of special strings like `...`. + The finer grained control over these features is necessary to truthfully + reproduce Emacs Org mode behavior. Special strings are enabled by default, + while smart quotes are disabled. + + * Remove the old `compact_definition_lists` extension. This was + neded to preserve backwards compatibility after pandoc 1.12 + was released, but at this point we can get rid of it. + + * Make `-t chunkedhtml -o -` output to stdout (as documented), rather + than creating a directory called `-` (#11068). + + * RST reader: Support multiple header rows (#10338, TuongNM). + + * LaTeX reader: + + + Support soft hyphens (Albert Krewinkel). + + Parse `\minisec` as unlisted level 6 headings (#10635, Albert Krewinkel). + + Support `\ifmmode` (#10915). + + Change handling of math environments (#9711, #9296). + Certain environments in LaTeX will trigger math mode and can't + occur within math mode: e.g., `align` or `equation`. Previously + we "downshifted" these, parsing an `align` environment as a + Math element with `aligned`, and an `equation` environment as a + regular display math element. With this shift, we put these in + Math inlines but retain the original environments. + texmath and MathJax both handle these environments well. + + * Typst reader: + + + Fix addition of image path prefix to use posix separator. + + Properly resolve image paths in included files (#11090). + + Handle inline-level show rules on block content (#11017). + Typst allows things like `smallcaps` to be applied to block-level + content like headings. This produces a type mismatch in pandoc, + so before processing the output of typst-hs, we transform it, + pulling the block-level elements outside of the inline-level + elements. + + * Org reader: + + + Improve sub- and superscript parsing (Albert Krewinkel). + Sub- and superscript must be preceded by a string in Org mode. Some text + preceded by space or at the start of a paragraph was previously parsed + incorrectly as sub- or superscript. + + Allow "greater block" names to contain any non-space char (#4287, + Albert Krewinkel). + + Accept quoted values as argument values (#8869, Albert Krewinkel). + + Recognize "fast access" characters in TODO state definitions + (#10990, Ryan Gibb). + + Improve org-cite parsing: Handle global prefix and suffix properly. + Use all and only the styles mentioned in oc-basic.el. + Allow space after `;`. + + * HTML reader: + + + Don't drop the initial newline in a `pre` element (#11064). + + * DocBook reader: + + + Add rowspan support (#10981, Sean Soon). + + Be sensitive to startingnumber attribute on ordered lists (#10912). + + * POD reader: + + + Fix named entity lookup (#11015, Evan Silberman). + + * Man reader: + + + Support header and footer reader (Sean Soon). + + * Markdown reader: + + + Don't confuse a span after an author-in-text citation with a + locator. E.g. `@foo [test]{.bar}`. + See https://github.com/jgm/pandoc/issues/9080#issuecomment-3221689892. + + Make definition lists behave like other lists (#10889). + If the `four_space_rule` extension is not enabled, + figure out the indentation needed for child blocks dynamically, + by looking at the first nonspace content after the `:` marker. + Previously the four-space rule was always obeyed. + + Fix tight/loose detection for definition lists, to conform to + the documentation. + + * ODT reader: + + + Support `table-header-rows` (Tuong Nguyen Manh). + + * Docx reader: + + + Don't add highlighting if highlight color is "none" (#10900). + + Handle strict OpenXML as well as transitional (#7691). + + Fix `stringToInteger` (#9184). It previously converted things like + `11ccc` to an integer; now it requires that the whole string be parsable + as an integer. + + Improve handling of AlternateContent. This fixes handling of + one representation of emojis in Word (#11113). + + * LaTeX writer: + + + Control figure placement with attribute (#10369, Sean Soon). + If a `latex-placement` attribute is present on a figure, it will be used + as the optional positioning hint in LaTeX (e.g. `ht`). With implicit + figures, `latex-placement` will be added to the figure (and removed from + the image) if it is present on the image. + + Include cancel package only if there is math that contains `\cancel`, + `\bcancel`, or `\xcancel`. + + Add braces around comments in `title-meta` (#10501). This is needed to + prevent PDFs from interpreting this as a sequence of titles. + + Set `pdf-trailer-id` if `SOURCE_DATE_EPOCH` envvar is set (#6539, Albert + Krewinkel). The `SOURCE_DATE_EPOCH` environment variable is used to trigger + reproducible PDF compilation, i.e., PDFs that are identical down to the + byte level for repeated runs. + + Be more conservative about using `\url` (#8802). We only use it when the + URL is all ASCII, since the `\url` macro causes problems when used with + some non-ASCII characters. + + Support soft hyphens (Albert Krewinkel). + + Change handling of math environments (#9711, #9296). + When certain math environments (e.g. `align`) are found in Math + elements, we emit them "raw" instead of putting them in `$..$`. + + * Typst writer: + + + Check `XID_Continue` in identifiers (Tuong Nguyen Manh). + + Add escapes to prevent inadvertent lists due to automatic wrapping + (#10047). Also simplify existing code that was meant to do this. + + Add parentheses around typst-native year-only citations (#11044). + + Add native Typst support for `nocite` (#10680, Albert Krewinkel). + The `nocite` metadata field can now be used to supply + additional citations that don't appear in the text, just as with citeproc + and LaTeX's bibtex and natbib. + + Set `lang` attribute in Divs (#10965). + + Rename `numbering` variable to `section-numbering` (Albert Krewinkel). + This is the name expected by the default template. + + Add support for custom and/or translated "Abstract" titles (Albert + Krewinkel, #9724). + + * Org writer: + + + Don't wrap link descriptions (#9000). Org doesn't reliable display + these as links if they have hard breaks. + + Disable smart quotes by default (Albert Krewinkel). + + * Markdown writer: + + + Better handling of pandoc-generated code blocks (#10926). + Omit the wrapper sourceCode divs added by pandoc around code blocks. + More intelligently identify which class to use for the one class + allowed in GFM code blocks. If there is a class of form `language-X`, + use `X`; otherwise use the first class other than `sourceCode`. + + Use fenced divs even with empty attributes (#10955, Carlos Scheidegger). + Previously fenced divs were not used in this case, causing the writer to + fall back to raw HTML. + + Match indents in definition items (#10890, Albert Krewinkel). + Previously, the first line of a definition details item always used a + colon and three spaces instead of respecting the tab-stop setting, which + could lead to round-tripping issues. Likewise, the indentation of + continuation paragraphs in definition lists now matches the + two-characters leader of the first line for Markua output. + + * DocBook writer: + + + Use `startingnumber` instead of `override` for start numbers on ordered + lists (#10912). + + * ANSI writer: + + + Make `--wrap=none` work properly (#10898). + + * Djot writer: + + + Fix duplicate attributes before section headings (#10984). + + * Docx writer: + + + Ensure that documents don't start with a section separator + (#10578, Albert Krewinkel). Any leading section separator is removed from + the result. + + * HTML writer: + + + Unwrap "wrapper" divs (#11014). Some of the readers (e.g. djot) add + "wrapper" divs to hold attributes for elements that have no slot for + attributes in the pandoc AST. The HTML reader now "unwraps" + these wrappers so that the attributes go on the intended elements. + + * Asciidoc writer: + + + Handle lists with sublists following continuations (#11006). + These require an additional blank line in some cases. + + * HTML styles template: prefix default styles with informative CSS comment + (Albert Krewinkel, #8819). + + * Org template: add `#+options` lines if necessary (Albert Krewinkel). + The default template now adds `#+options` lines if non-default settings + are used for the `smart_quotes` and `special_strings` extensions. + + * LaTeX template: + + + Don't emit empty `linkcolor=` in hypersetup (#11098). + + Add RTL support for LuaTeX engine (Reuben Thomas). + + * Typst template: + + + Add several new variables (Christopher T. Kenny, #9956): + `thanks`, `abstract-title`, `linestretch`, `mathfont`, `codefont`, + `linkcolor`, `filecolor`, `citecolor`. + + * `reference.docx`: + + + Don't left-align table header row (R. N. West, #11019). + + Update East Asia font theme in `styles.xml` to `minorEastAsia` (TomBen). + + Update language settings in `styles.xml` for East Asia to Simplified + Chinese (TomBen). + + * Text.Pandoc.PDF: + + + `makePDF`: automatically embed resources from media bag in HTML + before trying to convert it with weasyprint, etc. (#11099). + This will give better results when converting from formats like docx. + + Use `utf8ToText` for LaTeX log messages. + + Make images from MediaBag available in tmp dir for every PDF engine, + not just LaTeX/ConTeXt (#10911). + + Improve error readability when pdf-engine is not supported (Albert + Krewinkel). Each supported engine is now printed on a line of its own. + + Allow `pdflatex-dev` and `lualatex-dev` as PDF engines (#10991, Albert + Krewinkel). These are the development versions of the LaTeX binaries; + installable, e.g., with `tlmgr install latex-base-dev`. + + Clean up `makePDF` (Albert Krewinkel). + + Avoid encoding errors when reading LaTeX logs (#10954). + + * Text.Pandoc.Readers: + + + Raise unknown reader error for `ods`, `odp`, `odf`, `xls`, + `xslx`, `zip` extensions. + + * Text.Pandoc.App: + + + Recognize binary signatures and fail early (Repetitive). Fail early when + receiving binary input with recognized signature: zip[-based], including + OpenDocument and Microsoft formats, PDF, CFBF-based (old Microsoft formats + including .doc and .xls), DjVu. + + Remove code duplication around version info. + Text.Pandoc.App.CommandLineOptions and `pandoc-cli/src/pandoc.hs` + had similar code for generating version information. + To avoid duplication, we now export `versionInfo` from + Text.Pandoc.App [API change]. This function has three parameters that can + be filled in when it is called by `pandoc-cli`. + + * Text.Pandoc.Parsing: + + + `tableWith` and `tableWith'` now return a list of lists of Blocks, rather + than a list of Blocks, for the header rows, allowing for multiple header + rows [API change] (#10338, TuongNM). + + * Text.Pandoc.Citeproc: + + + Don't move footnotes around em-dashes (#11046). + + Allow `--citeproc` to put the bibliography in a Div with id `refs` + even when `--file-scope` is used (#11072). When `--file-scope` + is used, a prefix will be added based on the filename, so the Div + will end up having an identifier like `myfile.md__refs`. + Previously, this prevented the bibliography from being added to + the marked Div. Now pandoc will add the bibliography to any Div + with the id `refs` or any id ending in `__refs`. + + * Text.Pandoc.Citeproc.BibTeX: Protect case in periodical titles (#11048). + Thus, for example, `{npj} Quantum Information` should translate as + `[npj]{.nocase} Quantum Information`. + + * Text.Pandoc.ImageSize: + + + Detect more JPEG file signatures (R. N. West and John MacFarlane, #11049). + + Unpack compressed object streams in PDFs and look inside for MediaBox + information (#10902). + + Add Point and Pica as constructors of ImageSize [API change] (#8957). + This will prevent unnecessary conversion of units. + + Add Avif constructor on ImageType [API change] and support avif images + (#10979). + + * Text.Pandoc.Writers.Shared: + + + Amend docs of `lookupMeta...` functions (#10634, Albert Krewinkel). + + * Text.Pandoc.Options: + + + Add and export `defaultWebTeXURL` WebTeX URL [API change] (#11029, + Sean Soon). This fixes the `webtex` option when used without + parameter in a defaults file. + + Add type `HighlightMethod` and patterns [API Change] (Albert Krewinkel). + + The `writerListings` and `writerHighlightStyle` fields of the + `WriterOptions` type are replaced with `writerHighlightStyle` + [API change] (Albert Krewinkel, #10525). + + * Text.Pandoc.Extensions: + + + Remove `Ext_compact_definition_lists` constructor for + `Extension` [API change]. + + * Text.Pandoc.SelfContained: + + + Try fetching relative resources without query or fragment if the original + fetch fails. This provides a fix for #1477 in a way that doesn't raise + the problems mentioned in #11021. + + * Text.Pandoc.Highlighting: + + + Export `defaultStyle` [API Change] (Albert Krewinkel). + This allows to be more explicit about using a default style, and + providing a single point of truth for its value. The variable is + an alias for `pygments`. + + * Text.Pandoc.Class: + + + `downloadOrRead`: do not drop fragment/hash for local file paths (#11021). + With the previous behavior it was impossible to have an image + file containing `#` or `?`. + + Export function `runSilently` [API Change] (Albert Krewinkel). + The function runs an action in the PandocMonad, but returns all log + messages reported by that action instead of adding them to the main log. + + Make CommonState opaque. Text.Pandoc.Class now exports CommonState as + an opaque object, without its fields. [API change] + The internal module Text.Pandoc.Class.CommonState still exports + the fields. + + Text.Pandoc.Class now exports the following new functions: + `getRequestHeaders`, `setRequestHeaders`, `getSourceURL`, + `getTrace`. [API change] + + CommonState now has a `stManager` field. This allows us to cache the HTTP + client manager and reuse it for many requests, instead of creating it again + (an expensive operation) for each request. This fixes a memory leak and + performance issue in files with a large number of remote images (#10997). + + * Lua subsystem (Albert Krewinkel): + + + Add function `pandoc.structure.unique_identifier`. + + Add functions `pandoc.text.superscript` and `subscript`. + + Use proper interface functions to access the CommonState. + The `PANDOC_STATE` is no longer a userdata object, but a table that + behaves like the old object. Log messages in `PANDOC_STATE.log` are now + in temporal order. + + Add function `pandoc.path.exists`. + + Add `normalize` function to *Pandoc* objects (#10356). + This function performs a normalization of Pandoc documents. E.g., + multiple successive spaces are collapsed, and tables are normalized such + that all rows and columns contain the same number of cells. + + Add more UTF-8-aware file operations to `pandoc.system`. + Functions that expect UTF-8-encoded filenames should make it easier to + write platform-independent scripts, as the encoding of the actual + filename depends on the system. In addition, there is a new + generalized method to run commands, and functions to retrieve XDG + directory names. The new functions are `command`, `copy`, `read_file`, + `remove`, `rename`, `times`, `write_file`, `xdg`. + + Allow hslua-2.4. + + Require lua-module-system 1.2.3. This provides List methods to the value + returned by `pandoc.system.list_directory` (#11032). + + * MANUAL.txt: + + + Fix broken ConTeXt links (R. N. West, #11055). + + Add `xml` as input/output format. + + Fix minor capitalization typo (#11052, Albert Krewinkel). + + * `doc/lua-filters`: + + + Fix docs for `pandoc.Cite` (Albert Krewinkel). + + Don't encourage returning tables of filters from Lua filters + (R. N. West, #10995). Use the `Pandoc:walk` method instead. + + * doc/extras.md: Fix link to pandoc-mode (Erik Post). + + * doc/lua-filters.md: Add example on using pandoc.Table constructor (#10956, + Sean Soon). + + * Update `default.csl` from new chicago-author-date.csl, which is now + for the 18th edition. + + * Use latest releases of citeproc, typst-hs, texmath, doclayout, + skylighting-core, skylighting. + ## pandoc 3.7.0.2 (2025-05-28) * RST writer: diff --git a/data/default.csl b/data/default.csl --- a/data/default.csl +++ b/data/default.csl @@ -1,679 +1,3933 @@ - diff --git a/data/docx/word/styles.xml b/data/docx/word/styles.xml --- a/data/docx/word/styles.xml +++ b/data/docx/word/styles.xml @@ -3,10 +3,10 @@ - + - + @@ -604,13 +604,6 @@ - - - - - - - diff --git a/data/templates/common.latex b/data/templates/common.latex --- a/data/templates/common.latex +++ b/data/templates/common.latex @@ -234,6 +234,12 @@ \newenvironment{RTL}{\beginR}{\endR} \newenvironment{LTR}{\beginL}{\endL} \fi +\ifluatex + \newcommand{\RL}[1]{\bgroup\textdir TRT#1\egroup} + \newcommand{\LR}[1]{\bgroup\textdir TLT#1\egroup} + \newenvironment{RTL}{\textdir TRT\pardir TRT\bodydir TRT}{} + \newenvironment{LTR}{\textdir TLT\pardir TLT\bodydir TLT}{} +\fi $endif$ $-- $-- bibliography support support for natbib and biblatex diff --git a/data/templates/default.latex b/data/templates/default.latex --- a/data/templates/default.latex +++ b/data/templates/default.latex @@ -24,6 +24,9 @@ \usepackage[$for(geometry)$$geometry$$sep$,$endfor$]{geometry} $endif$ \usepackage{amsmath,amssymb} +$if(cancel)$ +\usepackage{cancel} +$endif$ $-- $-- section numbering $-- @@ -40,7 +43,20 @@ $endfor$ $after-header-includes.latex()$ $hypersetup.latex()$ +$if(pdf-trailer-id)$ +\ifXeTeX +\special{pdf:trailerid [ $pdf-trailer-id$ ]} +\fi +\ifPDFTeX +\pdftrailerid{} +\pdftrailer{/ID [ $pdf-trailer-id$ ]} +\fi +\ifLuaTeX +\pdfvariable trailerid {[ $pdf-trailer-id$ ]} +\fi +$endif$ + $if(title)$ \title{$title$$if(thanks)$\thanks{$thanks$}$endif$} $endif$ @@ -79,7 +95,9 @@ $endif$ { $if(colorlinks)$ -\hypersetup{linkcolor=$if(toccolor)$$toccolor$$else$$endif$} +$if(toccolor)$ +\hypersetup{linkcolor=$toccolor$} +$endif$ $endif$ \setcounter{tocdepth}{$toc-depth$} \tableofcontents diff --git a/data/templates/default.org b/data/templates/default.org --- a/data/templates/default.org +++ b/data/templates/default.org @@ -9,6 +9,12 @@ #+date: $date$ $endif$ +$if(options/pairs)$ +$for(options/pairs)$ +#+options: ${it.key}:${it.value} +$endfor$ + +$endif$ $for(header-includes)$ $header-includes$ diff --git a/data/templates/default.typst b/data/templates/default.typst --- a/data/templates/default.typst +++ b/data/templates/default.typst @@ -71,9 +71,15 @@ $if(region)$ region: "$region$", $endif$ +$if(abstract-title)$ + abstract-title: [$abstract-title$], +$endif$ $if(abstract)$ abstract: [$abstract$], $endif$ +$if(thanks)$ + thanks: [$thanks$], +$endif$ $if(margin)$ margin: ($for(margin/pairs)$$margin.key$: $margin.value$,$endfor$), $endif$ @@ -86,10 +92,28 @@ $if(fontsize)$ fontsize: $fontsize$, $endif$ +$if(mathfont)$ + mathfont: ($for(mathfont)$"$mathfont$",$endfor$), +$endif$ +$if(codefont)$ + codefont: ($for(codefont)$"$codefont$",$endfor$), +$endif$ +$if(linestretch)$ + linestretch: $linestretch$, +$endif$ $if(section-numbering)$ sectionnumbering: "$section-numbering$", $endif$ pagenumbering: $if(page-numbering)$"$page-numbering$"$else$none$endif$, +$if(linkcolor)$ + linkcolor: [$linkcolor$], +$endif$ +$if(citecolor)$ + citecolor: [$citecolor$], +$endif$ +$if(filecolor)$ + filecolor: [$filecolor$], +$endif$ cols: $if(columns)$$columns$$else$1$endif$, doc, ) @@ -108,6 +132,9 @@ $body$ $if(citations)$ +$for(nocite-ids)$ +#cite(label("${it}"), form: none) +$endfor$ $if(csl)$ #set bibliography(style: "$csl$") @@ -117,7 +144,7 @@ $endif$ $if(bibliography)$ -#bibliography($for(bibliography)$"$bibliography$"$sep$,$endfor$) +#bibliography($for(bibliography)$"$bibliography$"$sep$,$endfor$$if(full-bibliography)$, full: true$endif$) $endif$ $endif$ $for(include-after)$ diff --git a/data/templates/styles.html b/data/templates/styles.html --- a/data/templates/styles.html +++ b/data/templates/styles.html @@ -1,3 +1,6 @@ +/* Default styles provided by pandoc. +** See https://pandoc.org/MANUAL.html#variables-for-html for config info. +*/ $if(document-css)$ html { $if(mainfont)$ diff --git a/data/templates/template.typst b/data/templates/template.typst --- a/data/templates/template.typst +++ b/data/templates/template.typst @@ -15,7 +15,9 @@ authors: (), keywords: (), date: none, + abstract-title: none, abstract: none, + thanks: none, cols: 1, margin: (x: 1.25in, y: 1.25in), paper: "us-letter", @@ -23,66 +25,92 @@ region: "US", font: (), fontsize: 11pt, + mathfont: none, + codefont: none, + linestretch: 1, sectionnumbering: none, + linkcolor: none, + citecolor: none, + filecolor: none, pagenumbering: "1", doc, ) = { set document( title: title, - author: authors.map(author => content-to-string(author.name)), keywords: keywords, ) + set document( + author: authors.map(author => content-to-string(author.name)).join(", ", last: " & "), + ) if authors != none and authors != () set page( paper: paper, margin: margin, numbering: pagenumbering, - columns: cols, - ) - set par(justify: true) + ) + + set par( + justify: true, + leading: linestretch * 0.65em + ) set text(lang: lang, region: region, font: font, size: fontsize) + + show math.equation: set text(font: mathfont) if mathfont != none + show raw: set text(font: codefont) if codefont != none + set heading(numbering: sectionnumbering) - place(top, float: true, scope: "parent", clearance: 4mm)[ - #if title != none { - align(center)[#block(inset: 2em)[ - #text(weight: "bold", size: 1.5em)[#title] - #(if subtitle != none { - parbreak() - text(weight: "bold", size: 1.25em)[#subtitle] - }) - ]] + show link: set text(fill: rgb(content-to-string(linkcolor))) if linkcolor != none + show ref: set text(fill: rgb(content-to-string(citecolor))) if citecolor != none + show link: this => { + if filecolor != none and type(this.dest) == label { + text(this, fill: rgb(content-to-string(filecolor))) + } } - #if authors != none and authors != [] { - let count = authors.len() - let ncols = calc.min(count, 3) - grid( - columns: (1fr,) * ncols, - row-gutter: 1.5em, - ..authors.map(author => - align(center)[ - #author.name \ - #author.affiliation \ - #author.email - ] + block(below: 4mm)[ + #if title != none { + align(center)[#block(inset: 2em)[ + #text(weight: "bold", size: 1.5em)[#title #if thanks != none { + footnote(thanks, numbering: "*") + counter(footnote).update(n => n - 1) + }] + #( + if subtitle != none { + parbreak() + text(weight: "bold", size: 1.25em)[#subtitle] + } + ) + ]] + } + + #if authors != none and authors != [] { + let count = authors.len() + let ncols = calc.min(count, 3) + grid( + columns: (1fr,) * ncols, + row-gutter: 1.5em, + ..authors.map(author => align(center)[ + #author.name \ + #author.affiliation \ + #author.email + ]) ) - ) - } + } - #if date != none { - align(center)[#block(inset: 1em)[ - #date - ]] - } + #if date != none { + align(center)[#block(inset: 1em)[ + #date + ]] + } - #if abstract != none { - block(inset: 2em)[ - #text(weight: "semibold")[Abstract] #h(1em) #abstract - ] - } + #if abstract != none { + block(inset: 2em)[ + #text(weight: "semibold")[#abstract-title] #h(1em) #abstract + ] + } ] doc diff --git a/pandoc.cabal b/pandoc.cabal --- a/pandoc.cabal +++ b/pandoc.cabal @@ -1,6 +1,6 @@ cabal-version: 2.4 name: pandoc -version: 3.7.0.2 +version: 3.8 build-type: Simple license: GPL-2.0-or-later license-file: COPYING.md @@ -218,10 +218,12 @@ test/command/*.md test/command/*.csl test/command/*.svg + test/command/7691.docx test/command/9391.docx test/command/9358.docx test/command/9002.docx test/command/9603.docx + test/command/11113.docx test/command/biblio.bib test/command/averroes.bib test/command/A.txt @@ -231,6 +233,7 @@ test/command/file1.txt test/command/file2.txt test/command/three.txt + test/command/11090/ch1.typ test/command/01.csv test/command/chap1/spider.png test/command/chap2/spider.png @@ -450,7 +453,7 @@ -Wpartial-fields -Wmissing-signatures -fhide-source-paths - -- -Wmissing-export-lists + -Wmissing-export-lists if impl(ghc >= 8.10) ghc-options: -Wunused-packages @@ -495,7 +498,7 @@ blaze-markup >= 0.8 && < 0.9, bytestring >= 0.9 && < 0.13, case-insensitive >= 1.2 && < 1.3, - citeproc >= 0.9.0.1 && < 0.10, + citeproc >= 0.10 && < 0.11, commonmark >= 0.2.6.1 && < 0.3, commonmark-extensions >= 0.2.6 && < 0.3, commonmark-pandoc >= 0.2.3 && < 0.3, @@ -505,7 +508,7 @@ data-default >= 0.4 && < 0.9, deepseq >= 1.3 && < 1.6, directory >= 1.2.3 && < 1.4, - doclayout >= 0.5 && < 0.6, + doclayout >= 0.5.0.1 && < 0.6, doctemplates >= 0.11 && < 0.12, emojis >= 0.1.4.1 && < 0.2, exceptions >= 0.8 && < 0.11, @@ -530,24 +533,25 @@ random >= 1.2 && < 1.4, safe >= 0.3.18 && < 0.4, scientific >= 0.3 && < 0.4, - skylighting >= 0.14.5 && < 0.15, - skylighting-core >= 0.14.5 && < 0.15, + skylighting >= 0.14.7 && < 0.15, + skylighting-core >= 0.14.7 && < 0.15, split >= 0.2 && < 0.3, syb >= 0.1 && < 0.8, tagsoup >= 0.14.6 && < 0.15, temporary >= 1.1 && < 1.4, - texmath >= 0.12.10.3 && < 0.13, + texmath >= 0.13 && < 0.14, text >= 1.1.1.0 && < 2.2, text-conversions >= 0.3 && < 0.4, - time >= 1.5 && < 1.15, + time >= 1.5 && < 1.16, unicode-collation >= 0.1.1 && < 0.2, + unicode-data >= 0.6 && < 0.7, unicode-transforms >= 0.3 && < 0.5, yaml >= 0.11 && < 0.12, libyaml >= 0.1.4 && < 0.2, zip-archive >= 0.4.3.1 && < 0.5, zlib >= 0.5 && < 0.8, xml >= 1.3.12 && < 1.4, - typst >= 0.8.0.1 && < 0.9, + typst >= 0.8.0.2 && < 0.9, vector >= 0.12 && < 0.14, djot >= 0.1.2.2 && < 0.2, tls >= 2.0.1 && < 2.2, @@ -613,6 +617,7 @@ Text.Pandoc.Readers.Pod, Text.Pandoc.Writers, Text.Pandoc.Writers.Native, + Text.Pandoc.Writers.XML, Text.Pandoc.Writers.DocBook, Text.Pandoc.Writers.JATS, Text.Pandoc.Writers.OPML, @@ -743,6 +748,7 @@ Text.Pandoc.Readers.Metadata, Text.Pandoc.Readers.Roff, Text.Pandoc.Readers.Roff.Escape, + Text.Pandoc.Readers.XML, Text.Pandoc.Writers.Docx.OpenXML, Text.Pandoc.Writers.Docx.StyleMap, Text.Pandoc.Writers.Docx.Table, @@ -768,6 +774,7 @@ Text.Pandoc.Char, Text.Pandoc.TeX, Text.Pandoc.URI, + Text.Pandoc.XMLFormat, Text.Pandoc.CSS, Text.Pandoc.CSV, Text.Pandoc.RoffChar, @@ -806,7 +813,7 @@ tasty-quickcheck >= 0.8 && < 0.12, text >= 1.1.1.0 && < 2.2, temporary >= 1.1 && < 1.4, - time >= 1.5 && < 1.15, + time >= 1.5 && < 1.16, xml >= 1.3.12 && < 1.4, zip-archive >= 0.4.3 && < 0.5 other-modules: Tests.Old @@ -814,6 +821,7 @@ Tests.Helpers Tests.Shared Tests.MediaBag + Tests.XML Tests.Readers.LaTeX Tests.Readers.HTML Tests.Readers.JATS diff --git a/src/Text/Pandoc/App.hs b/src/Text/Pandoc/App.hs --- a/src/Text/Pandoc/App.hs +++ b/src/Text/Pandoc/App.hs @@ -26,6 +26,7 @@ , parseOptionsFromArgs , options , applyFilters + , versionInfo ) where import qualified Control.Exception as E import Control.Monad ( (>=>), when, forM, forM_ ) @@ -41,11 +42,12 @@ import qualified Data.Text.Lazy.Encoding as TE import qualified Data.Text.Encoding.Error as TE import Data.Char (toLower) -import System.Directory (doesDirectoryExist, createDirectory) +import System.Directory (doesDirectoryExist, createDirectory, + createDirectoryIfMissing) import Codec.Archive.Zip (toArchiveOrFail, extractFilesFromArchive, ZipOption(..)) import System.Exit (exitSuccess) -import System.FilePath ( takeBaseName, takeExtension) +import System.FilePath ( takeBaseName, takeExtension, takeDirectory) import System.IO (nativeNewline, stdout) import qualified System.IO as IO (Newline (..)) import Text.Pandoc @@ -55,7 +57,7 @@ import Text.Pandoc.App.Opt (Opt (..), LineEnding (..), defaultOpts, IpynbOutput (..), OptInfo(..)) import Text.Pandoc.App.CommandLineOptions (parseOptions, parseOptionsFromArgs, - options, handleOptInfo) + options, handleOptInfo, versionInfo) import Text.Pandoc.App.Input (InputParameters (..), readInput) import Text.Pandoc.App.OutputSettings (OutputSettings (..), optToOutputSettings, sandbox') @@ -115,11 +117,14 @@ CRLF -> IO.CRLF LF -> IO.LF Native -> nativeNewline + let outputFileDir = takeDirectory outputFile + createDirectoryIfMissing True outputFileDir case output of TextOutput t -> writerFn eol outputFile t BinaryOutput bs -> writeFnBinary outputFile bs ZipOutput bs - | null (takeExtension outputFile) -> do + | null (takeExtension outputFile) + , outputFile /= "-" -> do -- create directory and unzip createDirectory outputFile -- will fail if directory exists let zipopts = [OptRecursive, OptDestination outputFile] ++ diff --git a/src/Text/Pandoc/App/CommandLineOptions.hs b/src/Text/Pandoc/App/CommandLineOptions.hs --- a/src/Text/Pandoc/App/CommandLineOptions.hs +++ b/src/Text/Pandoc/App/CommandLineOptions.hs @@ -21,6 +21,7 @@ , options , engines , setVariable + , versionInfo ) where import Control.Monad.Trans import Control.Monad.State.Strict @@ -46,6 +47,7 @@ import Text.DocTemplates (Context (..), ToContext (toVal), Val (..)) import Text.Pandoc import Text.Pandoc.Builder (setMeta) +import Data.Version (showVersion) import Text.Pandoc.App.Opt (Opt (..), LineEnding (..), IpynbOutput (..), DefaultsState (..), applyDefaults, fullDefaultsPath, OptInfo(..)) @@ -193,14 +195,7 @@ ,"text-styles"]) ,confNumFormat = Generic ,confTrailingNewline = True} sty - VersionInfo -> do - prg <- getProgName - defaultDatadir <- defaultUserDataDir - UTF8.hPutStrLn stdout - $ T.pack - $ prg ++ " " ++ T.unpack pandocVersionText ++ - "\nUser data directory: " ++ defaultDatadir ++ - ('\n':copyrightMessage) + VersionInfo -> versionInfo [] Nothing "" Help -> do prg <- getProgName UTF8.hPutStr stdout (T.pack $ usageMessage prg options) @@ -210,7 +205,8 @@ -- | Supported LaTeX engines; the first item is used as default engine -- when going through LaTeX. latexEngines :: [String] -latexEngines = ["pdflatex", "lualatex", "xelatex", "latexmk", "tectonic"] +latexEngines = [ "pdflatex", "lualatex", "xelatex", "latexmk", "tectonic" + , "pdflatex-dev", "lualatex-dev" ] -- | Supported HTML PDF engines; the first item is used as default -- engine when going through HTML. @@ -524,13 +520,18 @@ , Option "" ["no-highlight"] (NoArg - (\opt -> return opt { optHighlightStyle = Nothing })) + (\opt -> do + deprecatedOption "--no-highlight" + "Use --syntax-highlighting=none instead." + return opt { optSyntaxHighlighting = NoHighlightingString })) "" -- "Don't highlight source code" , Option "" ["highlight-style"] (ReqArg - (\arg opt -> - return opt{ optHighlightStyle = Just $ + (\arg opt -> do + deprecatedOption "--highlight-style" + "Use --syntax-highlighting instead." + return opt{ optSyntaxHighlighting = T.pack $ normalizePath arg }) "STYLE|FILE") "" -- "Style for highlighted code" @@ -543,6 +544,14 @@ "FILE") "" -- "Syntax definition (xml) file" + , Option "" ["syntax-highlighting"] + (ReqArg + (\arg opt -> return opt{ optSyntaxHighlighting = + T.pack $ normalizePath arg }) + "none|default|idiomatic||") + "" -- "syntax highlighting method for code" + + , Option "" ["dpi"] (ReqArg (\arg opt -> @@ -602,8 +611,8 @@ then return opt { optPdfEngine = Just arg } else optError $ PandocOptionError $ T.pack $ - "Argument of --pdf-engine must be one of " - ++ intercalate ", " pdfEngines) + "Argument of --pdf-engine must be one of\n" + ++ concatMap (\e -> "\t" <> e <> "\n") pdfEngines) "PROGRAM") "" -- "Name of program to use in generating PDF" @@ -813,8 +822,14 @@ , Option "" ["listings"] (OptArg (\arg opt -> do - boolValue <- readBoolFromOptArg "--listings" arg - return opt { optListings = boolValue }) + deprecatedOption "--listings" + "Use --syntax-highlighting=idiomatic instead." + boolValue <- readBoolFromOptArg "--listings" arg + return $ + if boolValue + then opt { optSyntaxHighlighting = + IdiomaticHighlightingString } + else opt) "true|false") "" -- "Use listings package for LaTeX code blocks" @@ -1027,8 +1042,8 @@ , Option "" ["webtex"] (OptArg (\arg opt -> do - let url' = fromMaybe "https://latex.codecogs.com/png.latex?" arg - return opt { optHTMLMathMethod = WebTeX $ T.pack url' }) + let url' = maybe defaultWebTeXURL T.pack arg + return opt { optHTMLMathMethod = WebTeX url' }) "URL") "" -- "Use web service for HTML math" @@ -1172,7 +1187,7 @@ copyrightMessage :: String copyrightMessage = intercalate "\n" [ - "Copyright (C) 2006-2024 John MacFarlane. Web: https://pandoc.org", + "Copyright (C) 2006-2025 John MacFarlane. Web: https://pandoc.org", "This is free software; see the source for copying conditions. There is no", "warranty, not even for merchantability or fitness for a particular purpose." ] @@ -1270,3 +1285,21 @@ #else normalizePath = id #endif + +-- | Print version information with customizable features and scripting engine +versionInfo :: [String] -> Maybe String -> String -> IO () +versionInfo features mbScriptingEngineName suffix = do + defaultDatadir <- defaultUserDataDir + let featuresLine = if null features + then [] + else ["Features: " ++ unwords features] + let scriptingLine = case mbScriptingEngineName of + Nothing -> [] + Just name -> ["Scripting engine: " ++ name] + UTF8.putStr $ T.unlines $ map T.pack $ + ["pandoc " ++ showVersion pandocVersion ++ suffix] ++ + featuresLine ++ + scriptingLine ++ + ["User data directory: " ++ defaultDatadir, + copyrightMessage] + exitSuccess diff --git a/src/Text/Pandoc/App/Input.hs b/src/Text/Pandoc/App/Input.hs --- a/src/Text/Pandoc/App/Input.hs +++ b/src/Text/Pandoc/App/Input.hs @@ -14,7 +14,7 @@ , readInput ) where -import Control.Monad ((>=>)) +import Control.Monad ((>=>), when) import Control.Monad.Except (throwError, catchError) import Data.Text (Text) import Network.URI (URI (..), parseURI) @@ -109,12 +109,29 @@ (toTextM fp bs) (\case PandocUTF8DecodingError{} -> do + when (hasKnownSignature bs) $ + throwError $ + PandocInputNotTextError (T.pack fp) report $ NotUTF8Encoded (if null fp then "input" else fp) return $ T.pack $ B8.unpack bs e -> throwError e) + where + -- "50 4B 03 04" is zip file signature + isZip bs' = "\x50\x4B\x03\x04" `BS.isPrefixOf` bs' + -- "25 50 44 46 2D" is PDF file signature + isPDF bs' = "\x25\x50\x44\x46\x2D" `BS.isPrefixOf` bs' + -- "D0 CF 11 E0 A1 B1 1A E1" is Compound File Binary Format signature used in + -- variety of old Microsoft formats (.doc and .xls among others) + isCFBF bs' = "\xD0\xCF\x11\xE0\xA1\xB1\x1A\xE1" `BS.isPrefixOf` bs' + -- "41 54 26 54 46 4F 52 4D ?? ?? ?? ?? 44 4A 56" is DjVu signature + isDjVu bs' = case BS.stripPrefix "\x41\x54\x26\x54\x46\x4F\x52\x4D" bs' of + Nothing -> False + Just x -> BS.isPrefixOf "\x44\x4A\x56" $ BS.drop 4 x + + hasKnownSignature bs' = any ($ bs') [isZip, isPDF, isCFBF, isDjVu] inputToLazyByteString :: (FilePath, (BS.ByteString, Maybe MimeType)) -> BL.ByteString diff --git a/src/Text/Pandoc/App/Opt.hs b/src/Text/Pandoc/App/Opt.hs --- a/src/Text/Pandoc/App/Opt.hs +++ b/src/Text/Pandoc/App/Opt.hs @@ -2,6 +2,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE CPP #-} {-# LANGUAGE DeriveGeneric #-} +{-# LANGUAGE PatternSynonyms #-} {-# LANGUAGE TemplateHaskell #-} {-# LANGUAGE FlexibleInstances #-} {-# LANGUAGE FlexibleContexts #-} @@ -43,7 +44,8 @@ ReferenceLocation (EndOfDocument), CaptionPosition (..), ObfuscationMethod (NoObfuscation), - CiteMethod (Citeproc)) + CiteMethod (Citeproc), + pattern DefaultHighlightingString) import Text.Pandoc.Class (readFileStrict, fileExists, setVerbosity, report, PandocMonad(lookupEnv), getUserDataDir) import Text.Pandoc.Error (PandocError (PandocParseError, PandocSomeError)) @@ -124,8 +126,8 @@ , optEmbedResources :: Bool -- ^ Make HTML accessible offline , optLinkImages :: Bool -- ^ Link ODT images rather than embedding , optHtmlQTags :: Bool -- ^ Use tags in HTML - , optHighlightStyle :: Maybe Text -- ^ Style to use for highlighted code , optSyntaxDefinitions :: [FilePath] -- ^ xml syntax defs to load + , optSyntaxHighlighting :: Text -- ^ Syntax highlighting method for code , optTopLevelDivision :: TopLevelDivision -- ^ Type of the top-level divisions , optHTMLMathMethod :: HTMLMathMethod -- ^ Method to print HTML math , optAbbreviations :: Maybe FilePath -- ^ Path to abbrevs file @@ -157,7 +159,6 @@ , optIndentedCodeClasses :: [Text] -- ^ Default classes for indented code blocks , optDataDir :: Maybe FilePath , optCiteMethod :: CiteMethod -- ^ Method to output cites - , optListings :: Bool -- ^ Use listings package for code blocks , optPdfEngine :: Maybe String -- ^ Program to use for latex/html -> pdf , optPdfEngineOpts :: [String] -- ^ Flags to pass to the engine , optSlideLevel :: Maybe Int -- ^ Header level that creates slides @@ -211,8 +212,8 @@ <*> o .:? "embed-resources" .!= optEmbedResources defaultOpts <*> o .:? "link-images" .!= optLinkImages defaultOpts <*> o .:? "html-q-tags" .!= optHtmlQTags defaultOpts - <*> o .:? "highlight-style" <*> o .:? "syntax-definitions" .!= optSyntaxDefinitions defaultOpts + <*> o .:? "syntax-highlighting" .!= optSyntaxHighlighting defaultOpts <*> o .:? "top-level-division" .!= optTopLevelDivision defaultOpts <*> o .:? "html-math-method" .!= optHTMLMathMethod defaultOpts <*> o .:? "abbreviations" @@ -245,7 +246,6 @@ <*> o .:? "indented-code-classes" .!= optIndentedCodeClasses defaultOpts <*> o .:? "data-dir" <*> o .:? "cite-method" .!= optCiteMethod defaultOpts - <*> o .:? "listings" .!= optListings defaultOpts <*> o .:? "pdf-engine" <*> o .:? "pdf-engine-opts" .!= optPdfEngineOpts defaultOpts <*> o .:? "slide-level" @@ -319,6 +319,7 @@ , optOutputFile = oOutputFile , optInputFiles = oInputFiles , optSyntaxDefinitions = oSyntaxDefinitions + , optSyntaxHighlighting = oSyntaxHighlighting , optAbbreviations = oAbbreviations , optReferenceDoc = oReferenceDoc , optEpubMetadata = oEpubMetadata @@ -337,7 +338,6 @@ , optBibliography = oBibliography , optCitationAbbreviations = oCitationAbbreviations , optPdfEngine = oPdfEngine - , optHighlightStyle = oHighlightStyle } = do oTo' <- mapM (fmap T.pack . resolveVars . T.unpack) oTo @@ -365,7 +365,7 @@ oBibliography' <- mapM resolveVars oBibliography oCitationAbbreviations' <- mapM resolveVars oCitationAbbreviations oPdfEngine' <- mapM resolveVars oPdfEngine - oHighlightStyle' <- mapM (fmap T.pack . resolveVars . T.unpack) oHighlightStyle + oSyntaxHighlighting' <- T.pack <$> resolveVars (T.unpack oSyntaxHighlighting) return opt{ optTo = oTo' , optFrom = oFrom' , optTemplate = oTemplate' @@ -373,6 +373,7 @@ , optOutputFile = oOutputFile' , optInputFiles = oInputFiles' , optSyntaxDefinitions = oSyntaxDefinitions' + , optSyntaxHighlighting = oSyntaxHighlighting' , optAbbreviations = oAbbreviations' , optReferenceDoc = oReferenceDoc' , optEpubMetadata = oEpubMetadata' @@ -391,7 +392,6 @@ , optBibliography = oBibliography' , optCitationAbbreviations = oCitationAbbreviations' , optPdfEngine = oPdfEngine' - , optHighlightStyle = oHighlightStyle' } where @@ -555,8 +555,9 @@ parseJSON v >>= \x -> return (\o -> o{ optLinkImages = x }) "html-q-tags" -> parseJSON v >>= \x -> return (\o -> o{ optHtmlQTags = x }) + -- Deprecated "highlight-style" -> - parseJSON v >>= \x -> return (\o -> o{ optHighlightStyle = x }) + parseJSON v >>= \x -> return (\o -> o{ optSyntaxHighlighting = x }) "syntax-definition" -> (parseJSON v >>= \x -> return (\o -> o{ optSyntaxDefinitions = @@ -569,6 +570,8 @@ parseJSON v >>= \x -> return (\o -> o{ optSyntaxDefinitions = optSyntaxDefinitions o <> map unpack x }) + "syntax-highlighting" -> + parseJSON v >>= \x -> return (\o -> o{ optSyntaxHighlighting = x }) "top-level-division" -> parseJSON v >>= \x -> return (\o -> o{ optTopLevelDivision = x }) "html-math-method" -> @@ -647,7 +650,10 @@ "cite-method" -> parseJSON v >>= \x -> return (\o -> o{ optCiteMethod = x }) "listings" -> - parseJSON v >>= \x -> return (\o -> o{ optListings = x }) + parseJSON v >>= \x -> + if x + then return (\o -> o{ optSyntaxHighlighting = "idiomatic" }) + else return id "pdf-engine" -> parseJSON v >>= \x -> return (\o -> o{ optPdfEngine = unpack <$> x }) "pdf-engine-opts" -> @@ -773,8 +779,8 @@ , optEmbedResources = False , optLinkImages = False , optHtmlQTags = False - , optHighlightStyle = Just "pygments" , optSyntaxDefinitions = [] + , optSyntaxHighlighting = DefaultHighlightingString , optTopLevelDivision = TopLevelDefault , optHTMLMathMethod = PlainMath , optAbbreviations = Nothing @@ -806,7 +812,6 @@ , optIndentedCodeClasses = [] , optDataDir = Nothing , optCiteMethod = Citeproc - , optListings = False , optPdfEngine = Nothing , optPdfEngineOpts = [] , optSlideLevel = Nothing diff --git a/src/Text/Pandoc/App/OutputSettings.hs b/src/Text/Pandoc/App/OutputSettings.hs --- a/src/Text/Pandoc/App/OutputSettings.hs +++ b/src/Text/Pandoc/App/OutputSettings.hs @@ -2,6 +2,7 @@ {-# LANGUAGE FlexibleContexts #-} {-# LANGUAGE LambdaCase #-} {-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE PatternSynonyms #-} {-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TupleSections #-} {- | @@ -165,8 +166,11 @@ syntaxMap <- foldM addSyntaxMap defaultSyntaxMap (optSyntaxDefinitions opts) - hlStyle <- traverse (lookupHighlightingStyle . T.unpack) $ - optHighlightStyle opts + hlStyle <- case optSyntaxHighlighting opts of + NoHighlightingString -> pure NoHighlighting + DefaultHighlightingString -> pure DefaultHighlighting + IdiomaticHighlightingString -> pure IdiomaticHighlighting + style -> Skylighting <$> lookupHighlightingStyle (T.unpack style) let setListVariableM _ [] ctx = return ctx setListVariableM k vs ctx = do @@ -251,9 +255,8 @@ , writerIdentifierPrefix = optIdentifierPrefix opts , writerHtmlQTags = optHtmlQTags opts , writerTopLevelDivision = optTopLevelDivision opts - , writerListings = optListings opts , writerSlideLevel = optSlideLevel opts - , writerHighlightStyle = hlStyle + , writerHighlightMethod = hlStyle , writerSetextHeaders = optSetextHeaders opts , writerListTables = optListTables opts , writerEpubSubdirectory = T.pack $ optEpubSubdirectory opts diff --git a/src/Text/Pandoc/Citeproc.hs b/src/Text/Pandoc/Citeproc.hs --- a/src/Text/Pandoc/Citeproc.hs +++ b/src/Text/Pandoc/Citeproc.hs @@ -299,6 +299,8 @@ where getCitation (Cite cs _fallback) = Seq.singleton $ Citeproc.Citation { Citeproc.citationId = Nothing + , Citeproc.citationPrefix = Nothing + , Citeproc.citationSuffix = Nothing , Citeproc.citationNoteNumber = case cs of [] -> Nothing @@ -318,7 +320,7 @@ where locmap = toLocatorMap locale go c = - let (mblocinfo, suffix) = parseLocator locmap (citationSuffix c) + let (mblocinfo, suffix) = parseLocator locmap (Pandoc.citationSuffix c) cit = CitationItem { citationItemId = fromMaybe (ItemId $ Pandoc.citationId c) @@ -326,7 +328,7 @@ , citationItemLabel = locatorLabel <$> mblocinfo , citationItemLocator = locatorLoc <$> mblocinfo , citationItemType = NormalCite - , citationItemPrefix = case citationPrefix c of + , citationItemPrefix = case Pandoc.citationPrefix c of [] -> Nothing ils -> Just $ B.fromList ils <> B.space @@ -408,14 +410,14 @@ mvPunct moveNotes locale (q : s : x@(Cite _ [il]) : ys) | isSpacy s , isNote il - = let spunct = T.takeWhile isPunctuation $ stringify ys + = let spunct = T.takeWhile isPunct $ stringify ys in if moveNotes then if T.null spunct then q : x : mvPunct moveNotes locale ys else movePunctInsideQuotes locale [q , Str spunct , x] ++ mvPunct moveNotes locale (B.toList - (dropTextWhile isPunctuation (B.fromList ys))) + (dropTextWhile isPunct (B.fromList ys))) else q : x : mvPunct moveNotes locale ys -- 'x[^1],' -> 'x,[^1]' mvPunct moveNotes locale (Cite cs ils@(_:_) : ys) @@ -423,13 +425,13 @@ , startWithPunct ys , moveNotes = let s = stringify ys - spunct = T.takeWhile isPunctuation s + spunct = T.takeWhile isPunct s in Cite cs (movePunctInsideQuotes locale $ init ils ++ [Str spunct | not (endWithPunct False (init ils))] ++ [last ils]) : mvPunct moveNotes locale - (B.toList (dropTextWhile isPunctuation (B.fromList ys))) + (B.toList (dropTextWhile isPunct (B.fromList ys))) mvPunct moveNotes locale (s : x@(Cite _ [il]) : ys) | isSpacy s , isNote il @@ -442,13 +444,18 @@ mvPunct moveNotes locale (x:xs) = x : mvPunct moveNotes locale xs mvPunct _ _ [] = [] +-- We don't treat an em-dash or en-dash as punctuation here, because we don't +-- want notes and quotes to move around them. +isPunct :: Char -> Bool +isPunct c = isPunctuation c && c /= '\x2014' && c /= '\x2013' + endWithPunct :: Bool -> [Inline] -> Bool endWithPunct _ [] = False endWithPunct onlyFinal xs@(_:_) = case reverse (T.unpack $ stringify xs) of [] -> True -- covers .), .", etc.: - (d:c:_) | isPunctuation d + (d:c:_) | isPunct d && not onlyFinal && isEndPunct c -> True (c:_) | isEndPunct c -> True @@ -478,7 +485,10 @@ -- if document contains a Div with id="refs", insert -- references as its contents. Otherwise, insert references --- at the end of the document in a Div with id="refs" +-- at the end of the document in a Div with id="refs" or +-- id=".*__refs" (where .* stands for any string -- this is +-- because --file-scope will add such a prefix based on the filename, +-- see #11072.) insertRefs :: [(Text,Text)] -> [Text] -> [Block] -> Pandoc -> Pandoc insertRefs _ _ [] d = d insertRefs refkvs refclasses refs (Pandoc meta bs) = @@ -503,12 +513,13 @@ refDiv = Div ("refs", refclasses, refkvs) refs addUnNumbered cs = "unnumbered" : [c | c <- cs, c /= "unnumbered"] go :: Block -> State Bool Block - go (Div ("refs",cs,kvs) xs) = do - put True - -- refHeader isn't used if you have an explicit references div - let cs' = nubOrd $ cs ++ refclasses - let kvs' = nubOrd $ kvs ++ refkvs - return $ Div ("refs",cs',kvs') (xs ++ refs) + go (Div (ident,cs,kvs) xs) + | ident == "refs" || "__refs" `T.isSuffixOf` ident = do + put True + -- refHeader isn't used if you have an explicit references div + let cs' = nubOrd $ cs ++ refclasses + let kvs' = nubOrd $ kvs ++ refkvs + return $ Div (ident,cs',kvs') (xs ++ refs) go x = return x refTitle :: Meta -> Maybe [Inline] diff --git a/src/Text/Pandoc/Citeproc/BibTeX.hs b/src/Text/Pandoc/Citeproc/BibTeX.hs --- a/src/Text/Pandoc/Citeproc/BibTeX.hs +++ b/src/Text/Pandoc/Citeproc/BibTeX.hs @@ -57,6 +57,7 @@ import Text.Printf (printf) import Text.DocLayout (literal, hsep, nest, hang, Doc(..), braces, ($$), cr) + data Variant = Bibtex | Biblatex deriving (Show, Eq, Ord) @@ -967,7 +968,7 @@ getPeriodicalTitle :: Text -> Bib Inlines getPeriodicalTitle f = do ils <- getField f - return ils + return $ protectCase id ils protectCase :: (Inlines -> Inlines) -> (Inlines -> Inlines) protectCase f = Walk.walk unprotect . f . Walk.walk protect diff --git a/src/Text/Pandoc/Class.hs b/src/Text/Pandoc/Class.hs --- a/src/Text/Pandoc/Class.hs +++ b/src/Text/Pandoc/Class.hs @@ -23,7 +23,9 @@ , Translations ) where -import Text.Pandoc.Class.CommonState (CommonState (..)) +-- We export CommonState as an opaque object. Accessors for +-- its fields are provided in Text.Pandoc.Class.PandocMonad. +import Text.Pandoc.Class.CommonState (CommonState) import Text.Pandoc.Class.PandocMonad import Text.Pandoc.Class.PandocIO import Text.Pandoc.Class.PandocPure diff --git a/src/Text/Pandoc/Class/CommonState.hs b/src/Text/Pandoc/Class/CommonState.hs --- a/src/Text/Pandoc/Class/CommonState.hs +++ b/src/Text/Pandoc/Class/CommonState.hs @@ -23,6 +23,7 @@ import Text.Pandoc.MediaBag (MediaBag) import Text.Pandoc.Logging (LogMessage, Verbosity (WARNING)) import Text.Pandoc.Translations.Types (Translations) +import Network.HTTP.Client (Manager) -- | 'CommonState' represents state that is used by all -- instances of 'PandocMonad'. Normally users should not @@ -50,6 +51,9 @@ , stResourcePath :: [FilePath] -- ^ Path to search for resources like -- included images + , stManager :: Maybe Manager + -- ^ Manager for HTTP client; this needs to persist across many requests + -- for efficiency. , stVerbosity :: Verbosity -- ^ Verbosity level , stTrace :: Bool @@ -75,6 +79,7 @@ , stInputFiles = [] , stOutputFile = Nothing , stResourcePath = ["."] + , stManager = Nothing , stVerbosity = WARNING , stTrace = False } diff --git a/src/Text/Pandoc/Class/IO.hs b/src/Text/Pandoc/Class/IO.hs --- a/src/Text/Pandoc/Class/IO.hs +++ b/src/Text/Pandoc/Class/IO.hs @@ -45,7 +45,7 @@ import qualified Network.TLS as TLS import qualified Network.TLS.Extra as TLS import Network.HTTP.Client - (httpLbs, responseBody, responseHeaders, + (httpLbs, Manager, responseBody, responseHeaders, Request(port, host, requestHeaders), parseRequest, newManager) import Network.HTTP.Client.Internal (addProxy) import Network.HTTP.Client.TLS (mkManagerSettings) @@ -61,7 +61,8 @@ import System.Random (StdGen) import Text.Pandoc.Class.CommonState (CommonState (..)) import Text.Pandoc.Class.PandocMonad - (PandocMonad, getsCommonState, getMediaBag, report, extractURIData) + (PandocMonad, getsCommonState, modifyCommonState, + getMediaBag, report, extractURIData) import Text.Pandoc.Definition (Pandoc, Inline (Image)) import Text.Pandoc.Error (PandocError (..)) import Text.Pandoc.Logging (LogMessage (..), messageVerbosity, showLogMessage) @@ -70,7 +71,6 @@ import Text.Pandoc.Walk (walk) import qualified Control.Exception as E import qualified Data.ByteString as B -import qualified Data.ByteString.Char8 as B8 import qualified Data.ByteString.Lazy as BL import qualified Data.CaseInsensitive as CI import qualified Data.Text as T @@ -124,6 +124,36 @@ newUniqueHash :: MonadIO m => m Int newUniqueHash = hashUnique <$> liftIO Data.Unique.newUnique +getManager :: (PandocMonad m, MonadIO m) => m Manager +getManager = do + mbManager <- getsCommonState stManager + disableCertificateValidation <- getsCommonState stNoCheckCertificate + case mbManager of + Just manager -> pure manager + Nothing -> do + manager <- liftIO $ do + certificateStore <- getSystemCertificateStore + let tlsSettings = TLSSettings $ + (TLS.defaultParamsClient "localhost.localdomain" "80") + { TLS.clientSupported = def{ TLS.supportedCiphers = + TLS.ciphersuite_default + , TLS.supportedExtendedMainSecret = + TLS.AllowEMS } + , TLS.clientShared = def + { TLS.sharedCAStore = certificateStore + , TLS.sharedValidationCache = + if disableCertificateValidation + then TLS.ValidationCache + (\_ _ _ -> return TLS.ValidationCachePass) + (\_ _ _ -> return ()) + else def + } + } + let tlsManagerSettings = mkManagerSettings tlsSettings Nothing + newManager tlsManagerSettings + modifyCommonState $ \st -> st{ stManager = Just manager } + pure manager + openURL :: (PandocMonad m, MonadIO m) => Text -> m (B.ByteString, Maybe MimeType) openURL u | Just (URI{ uriScheme = "data:", @@ -132,8 +162,8 @@ | otherwise = do let toReqHeader (n, v) = (CI.mk (UTF8.fromText n), UTF8.fromText v) customHeaders <- map toReqHeader <$> getsCommonState stRequestHeaders - disableCertificateValidation <- getsCommonState stNoCheckCertificate report $ Fetching u + manager <- getManager res <- liftIO $ E.try $ withSocketsDo $ do proxy <- tryIOError (getEnv "http_proxy") let addProxy' x = case proxy of @@ -142,26 +172,7 @@ return (addProxy (host r) (port r) x) req <- parseRequest (unpack u) >>= addProxy' let req' = req{requestHeaders = customHeaders ++ requestHeaders req} - certificateStore <- getSystemCertificateStore - let tlsSettings = TLSSettings $ - (TLS.defaultParamsClient (show $ host req') - (B8.pack $ show $ port req')) - { TLS.clientSupported = def{ TLS.supportedCiphers = - TLS.ciphersuite_default - , TLS.supportedExtendedMainSecret = - TLS.AllowEMS } - , TLS.clientShared = def - { TLS.sharedCAStore = certificateStore - , TLS.sharedValidationCache = - if disableCertificateValidation - then TLS.ValidationCache - (\_ _ _ -> return TLS.ValidationCachePass) - (\_ _ _ -> return ()) - else def - } - } - let tlsManagerSettings = mkManagerSettings tlsSettings Nothing - resp <- newManager tlsManagerSettings >>= httpLbs req' + resp <- httpLbs req' manager return (B.concat $ toChunks $ responseBody resp, UTF8.toText `fmap` lookup hContentType (responseHeaders resp)) diff --git a/src/Text/Pandoc/Class/PandocMonad.hs b/src/Text/Pandoc/Class/PandocMonad.hs --- a/src/Text/Pandoc/Class/PandocMonad.hs +++ b/src/Text/Pandoc/Class/PandocMonad.hs @@ -28,12 +28,14 @@ , getZonedTime , readFileFromDirs , report - , setTrace + , runSilently , setRequestHeader , setNoCheckCertificate , getLog , setVerbosity , getVerbosity + , setTrace + , getTrace , getMediaBag , setMediaBag , insertMedia @@ -47,6 +49,9 @@ , setOutputFile , setResourcePath , getResourcePath + , setRequestHeaders + , getRequestHeaders + , getSourceURL , readMetadataFile , toTextM , fillMediaBag @@ -164,6 +169,15 @@ getVerbosity :: PandocMonad m => m Verbosity getVerbosity = getsCommonState stVerbosity +-- | Set tracing. This affects the behavior of 'trace'. If tracing +-- is not enabled, 'trace' does nothing. +setTrace :: PandocMonad m => Bool -> m () +setTrace enabled = modifyCommonState $ \st -> st{ stTrace = enabled } + +-- | Get tracing status. +getTrace :: PandocMonad m => m Bool +getTrace = getsCommonState stTrace + -- | Get the accumulated log messages (in temporal order). getLog :: PandocMonad m => m [LogMessage] getLog = reverse <$> getsCommonState stLog @@ -179,12 +193,23 @@ when (level <= verbosity) $ logOutput msg modifyCommonState $ \st -> st{ stLog = msg : stLog st } --- | Determine whether tracing is enabled. This affects --- the behavior of 'trace'. If tracing is not enabled, --- 'trace' does nothing. -setTrace :: PandocMonad m => Bool -> m () -setTrace useTracing = modifyCommonState $ \st -> st{stTrace = useTracing} +-- | Run an action, but suppress the output of any log messages; +-- instead, all messages reported by @action@ are returned separately +-- and not added to the main log. +runSilently :: PandocMonad m => m a -> m (a, [LogMessage]) +runSilently action = do + -- get current settings + origLog <- getsCommonState stLog + origVerbosity <- getVerbosity + -- reset log level and set verbosity to the minimum + modifyCommonState (\st -> st { stVerbosity = ERROR, stLog = []}) + result <- action + -- get log messages reported while running `action` + newLog <- getsCommonState stLog + modifyCommonState (\st -> st { stVerbosity = origVerbosity, stLog = origLog}) + return (result, newLog) + -- | Set request header to use in HTTP requests. setRequestHeader :: PandocMonad m => T.Text -- ^ Header name @@ -248,6 +273,18 @@ setResourcePath :: PandocMonad m => [FilePath] -> m () setResourcePath ps = modifyCommonState $ \st -> st{stResourcePath = ps} +-- | Retrieve the request headers to add for HTTP requests. +getRequestHeaders :: PandocMonad m => m [(T.Text, T.Text)] +getRequestHeaders = getsCommonState stRequestHeaders + +-- | Set the request headers to add for HTTP requests. +setRequestHeaders :: PandocMonad m => [(T.Text, T.Text)] -> m () +setRequestHeaders hs = modifyCommonState $ \st -> st{ stRequestHeaders = hs } + +-- | Get the absolute UL or directory of first source file. +getSourceURL :: PandocMonad m => m (Maybe T.Text) +getSourceURL = getsCommonState stSourceURL + -- | Get the current UTC time. If the @SOURCE_DATE_EPOCH@ environment -- variable is set to a unix time (number of seconds since midnight -- Jan 01 1970 UTC), it is used instead of the current time, to support @@ -371,8 +408,7 @@ uriPath = "", uriQuery = "", uriFragment = "" } - dropFragmentAndQuery = T.takeWhile (\c -> c /= '?' && c /= '#') - fp = unEscapeString $ T.unpack $ dropFragmentAndQuery s + fp = unEscapeString $ T.unpack s mime = getMimeType $ case takeExtension fp of ".gz" -> dropExtension fp ".svgz" -> dropExtension fp ++ ".svg" diff --git a/src/Text/Pandoc/Error.hs b/src/Text/Pandoc/Error.hs --- a/src/Text/Pandoc/Error.hs +++ b/src/Text/Pandoc/Error.hs @@ -65,6 +65,7 @@ | PandocUnsupportedExtensionError Text Text | PandocCiteprocError CiteprocError | PandocBibliographyError Text Text + | PandocInputNotTextError Text deriving (Show, Typeable, Generic) instance Exception PandocError @@ -143,6 +144,13 @@ prettyCiteprocError e' PandocBibliographyError fp msg -> "Error reading bibliography file " <> fp <> ":\n" <> msg + PandocInputNotTextError fp -> + "Expected text as an input, but received binary data from " <> + (if T.null fp + then "stdin" + else "file " <> fp) <> + ".\nIf you intended to convert from binary format, verify that it's " <> + "supported and use\nexplicit -f FORMAT." -- | Handle PandocError by exiting with an error message. @@ -184,6 +192,7 @@ PandocUTF8DecodingError{} -> 92 PandocIpynbDecodingError{} -> 93 PandocUnsupportedCharsetError{} -> 94 + PandocInputNotTextError{} -> 95 PandocCouldNotFindDataFileError{} -> 97 PandocCouldNotFindMetadataFileError{} -> 98 PandocResourceNotFound{} -> 99 diff --git a/src/Text/Pandoc/Extensions.hs b/src/Text/Pandoc/Extensions.hs --- a/src/Text/Pandoc/Extensions.hs +++ b/src/Text/Pandoc/Extensions.hs @@ -59,8 +59,6 @@ | Ext_blank_before_header -- ^ Require blank line before a header | Ext_bracketed_spans -- ^ Bracketed spans with attributes | Ext_citations -- ^ Pandoc/citeproc citations - | Ext_compact_definition_lists -- ^ Definition lists without space between items, - -- and disallow laziness | Ext_definition_lists -- ^ Definition lists as in pandoc, mmd, php | Ext_east_asian_line_breaks -- ^ Newlines in paragraphs are ignored between -- East Asian wide characters. Note: this extension @@ -122,6 +120,8 @@ | Ext_shortcut_reference_links -- ^ Shortcut reference links | Ext_simple_tables -- ^ Pandoc-style simple tables | Ext_smart -- ^ "Smart" quotes, apostrophes, ellipses, dashes + | Ext_smart_quotes -- ^ "Smart" quotes + | Ext_special_strings -- ^ Treat certain strings like special characters | Ext_sourcepos -- ^ Include source position attributes | Ext_space_in_atx_header -- ^ Require space between # and header text | Ext_spaced_reference_links -- ^ Allow space between two parts of ref link @@ -431,6 +431,7 @@ ] getDefaultExtensions "org" = extensionsFromList [Ext_citations, + Ext_special_strings, Ext_task_lists, Ext_auto_identifiers] getDefaultExtensions "html" = extensionsFromList @@ -511,7 +512,6 @@ , Ext_mark , Ext_mmd_link_attributes , Ext_mmd_header_identifiers - , Ext_compact_definition_lists , Ext_gutenberg , Ext_smart , Ext_literate_haskell @@ -583,6 +583,8 @@ extensionsFromList [ Ext_citations , Ext_smart + , Ext_smart_quotes + , Ext_special_strings , Ext_fancy_lists , Ext_task_lists ] diff --git a/src/Text/Pandoc/Format.hs b/src/Text/Pandoc/Format.hs --- a/src/Text/Pandoc/Format.hs +++ b/src/Text/Pandoc/Format.hs @@ -181,6 +181,7 @@ ".ctx" -> defFlavor "context" ".db" -> defFlavor "docbook" ".dj" -> defFlavor "djot" + ".djvu" -> defFlavor "djvu" -- so we get an "unknown reader" error ".doc" -> defFlavor "doc" -- so we get an "unknown reader" error ".docx" -> defFlavor "docx" ".dokuwiki" -> defFlavor "dokuwiki" @@ -204,6 +205,9 @@ ".ms" -> defFlavor "ms" ".muse" -> defFlavor "muse" ".native" -> defFlavor "native" + ".ods" -> defFlavor "ods" -- so we get an "unknown reader" error + ".odp" -> defFlavor "odp" -- so we get an "unknown reader" error + ".odf" -> defFlavor "odf" -- so we get an "unknown reader" error ".odt" -> defFlavor "odt" ".opml" -> defFlavor "opml" ".org" -> defFlavor "org" @@ -229,6 +233,10 @@ ".typ" -> defFlavor "typst" ".wiki" -> defFlavor "mediawiki" ".xhtml" -> defFlavor "html" + ".xls" -> defFlavor "xls" -- so we get an "unknown reader" error + ".xlsx" -> defFlavor "xlsx" -- so we get an "unknown reader" error + ".xml" -> defFlavor "xml" + ".zip" -> defFlavor "zip" -- so we get an "unknown reader" error ['.',y] | y `elem` ['1'..'9'] -> defFlavor "man" _ -> Nothing where diff --git a/src/Text/Pandoc/Highlighting.hs b/src/Text/Pandoc/Highlighting.hs --- a/src/Text/Pandoc/Highlighting.hs +++ b/src/Text/Pandoc/Highlighting.hs @@ -1,7 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} {- | Module : Text.Pandoc.Highlighting - Copyright : Copyright (C) 2008-2024 John MacFarlane + Copyright : Copyright (C) 2008-2025 John MacFarlane License : GNU GPL, version 2 or above Maintainer : John MacFarlane @@ -31,6 +31,7 @@ , styleToConTeXt , formatANSI -- * Styles + , defaultStyle , pygments , espresso , zenburn @@ -55,6 +56,10 @@ import Control.Monad.Except (throwError) import System.FilePath (takeExtension) import Text.Pandoc.Shared (safeRead) + +-- | The default highlighting style used by pandoc (pygments). +defaultStyle :: Style +defaultStyle = pygments highlightingStyles :: [(T.Text, Style)] highlightingStyles = diff --git a/src/Text/Pandoc/ImageSize.hs b/src/Text/Pandoc/ImageSize.hs --- a/src/Text/Pandoc/ImageSize.hs +++ b/src/Text/Pandoc/ImageSize.hs @@ -38,7 +38,7 @@ import qualified Data.ByteString.Lazy as BL import Data.Binary.Get import Data.Bits ((.&.), shiftR, shiftL) -import Data.Word (bitReverse32) +import Data.Word (bitReverse32, Word32) import Data.Maybe (isJust, fromJust) import Data.Char (isDigit) import Control.Monad @@ -57,11 +57,12 @@ import qualified Data.Attoparsec.ByteString.Char8 as A import qualified Codec.Picture.Metadata as Metadata import Codec.Picture (decodeImageWithMetadata) +import Codec.Compression.Zlib (decompress) +-- import Debug.Trace -- quick and dirty functions to get image sizes --- algorithms borrowed from wwwis.pl -data ImageType = Png | Gif | Jpeg | Svg | Pdf | Eps | Emf | Tiff | Webp +data ImageType = Png | Gif | Jpeg | Svg | Pdf | Eps | Emf | Tiff | Webp | Avif deriving Show data Direction = Width | Height instance Show Direction where @@ -72,6 +73,8 @@ | Centimeter Double | Millimeter Double | Inch Double + | Point Double + | Pica Double | Percent Double | Em Double deriving Eq @@ -81,6 +84,8 @@ show (Centimeter a) = T.unpack (showFl a) ++ "cm" show (Millimeter a) = T.unpack (showFl a) ++ "mm" show (Inch a) = T.unpack (showFl a) ++ "in" + show (Point a) = T.unpack (showFl a) ++ "pt" + show (Pica a) = T.unpack (showFl a) ++ "pc" show (Percent a) = show a ++ "%" show (Em a) = T.unpack (showFl a) ++ "em" @@ -113,8 +118,11 @@ "\x47\x49\x46\x38" -> return Gif "\x49\x49\x2a\x00" -> return Tiff "\x4D\x4D\x00\x2a" -> return Tiff - "\xff\xd8\xff\xe0" -> return Jpeg -- JFIF - "\xff\xd8\xff\xe1" -> return Jpeg -- Exif + "\xff\xd8\xff\xbd" -> return Jpeg -- JPEG without application segment -- see p.32 in https://www.w3.org/Graphics/JPEG/itu-t81.pdf (and https://gist.github.com/leommoore/f9e57ba2aa4bf197ebc5?permalink_comment_id=3863054#gistcomment-3863054) + _ | B.take 3 img == "\xff\xd8\xff" + && (let byte4 = B.take 1 (B.drop 3 img) + in byte4 >= "\xe0" && byte4 <= "\xef") -- JPEG with application segment + -> return Jpeg "%PDF" -> return Pdf " return Svg " return Webp + _ | B.take 4 (B.drop 4 img) == "ftyp" -> return Avif _ -> mzero findSvgTag :: ByteString -> Bool @@ -148,6 +157,7 @@ Just Pdf -> mbToEither "could not determine PDF size" $ pdfSize img Just Emf -> mbToEither "could not determine EMF size" $ emfSize img Just Webp -> mbToEither "could not determine WebP size" $ webpSize opts img + Just Avif -> mbToEither "could not determine AVIF size" $ avifSize opts img Nothing -> Left "could not determine image type" where mbToEither msg Nothing = Left msg mbToEither _ (Just x) = Right x @@ -201,6 +211,8 @@ (Centimeter a) -> a * 0.3937007874 (Millimeter a) -> a * 0.03937007874 (Inch a) -> a + (Point a) -> (a / 72) + (Pica a) -> (a / 6) (Percent _) -> 0 (Em a) -> a * (11/64) @@ -208,11 +220,7 @@ inPixel opts dim = case dim of (Pixel a) -> a - (Centimeter a) -> floor $ dpi * a * 0.3937007874 :: Integer - (Millimeter a) -> floor $ dpi * a * 0.03937007874 :: Integer - (Inch a) -> floor $ dpi * a :: Integer - (Percent _) -> 0 - (Em a) -> floor $ dpi * a * (11/64) :: Integer + _ -> floor (dpi * inInch opts dim) where dpi = fromIntegral $ writerDpi opts @@ -242,6 +250,8 @@ Centimeter x -> Centimeter (factor * x) Millimeter x -> Millimeter (factor * x) Inch x -> Inch (factor * x) + Point x -> Point (factor * x) + Pica x -> Pica (factor * x) Percent x -> Percent (factor * x) Em x -> Em (factor * x) @@ -265,8 +275,8 @@ toDim a "%" = Just $ Percent a toDim a "px" = Just $ Pixel (floor a::Integer) toDim a "" = Just $ Pixel (floor a::Integer) - toDim a "pt" = Just $ Inch (a / 72) - toDim a "pc" = Just $ Inch (a / 6) + toDim a "pt" = Just $ Point a + toDim a "pc" = Just $ Pica a toDim a "em" = Just $ Em a toDim _ _ = Nothing @@ -294,10 +304,10 @@ Right sz -> Just sz pPdfSize :: A.Parser ImageSize -pPdfSize = do - A.skipWhile (/='/') - A.char8 '/' - (do A.string "MediaBox" +pPdfSize = + (A.takeWhile1 (/= '/') *> pPdfSize) + <|> + (do A.string "/MediaBox" A.skipSpace A.char8 '[' A.skipSpace @@ -314,7 +324,27 @@ , pxY = y2 - y1 , dpiX = 72 , dpiY = 72 } - ) <|> pPdfSize + ) + <|> -- if we encounter a compressed object stream, uncompress it (#10902) + (do A.string "/Type" + A.skipSpace + A.string "/ObjStm" + _ <- A.manyTill pLine (A.string "stream" *> pEol) + stream <- BL.pack <$> A.manyTill + (AW.satisfy (const True)) + (pEol *> A.string "endstream" *> pEol) + let contents = BL.toStrict (decompress stream) + case A.parseOnly pPdfSize contents of + Left _ -> pPdfSize + Right is -> pure is) + <|> + (A.char '/' *> pPdfSize) + where + iseol '\r' = True + iseol '\n' = True + iseol _ = False + pEol = A.satisfy iseol *> A.skipMany (A.satisfy iseol) + pLine = A.takeWhile (not . iseol) <* pEol getSize :: ByteString -> Either T.Text ImageSize getSize img = @@ -451,3 +481,173 @@ case AW.parseOnly pWebpSize img of Left _ -> Nothing Right sz -> Just sz { dpiX = fromIntegral $ writerDpi opts, dpiY = fromIntegral $ writerDpi opts} + +avifSize :: WriterOptions -> ByteString -> Maybe ImageSize +avifSize _opts img = + case runGetOrFail (verifyFtyp >> findAvifDimensions) (BL.fromStrict img) of + Left (_, _, _err) -> Nothing + Right (_, _, (width, height)) -> + Just $ ImageSize { pxX = fromIntegral width + , pxY = fromIntegral height + , dpiX = 72 + , dpiY = 72 } + +---- AVIF parsing: + +verifyFtyp :: Get () +verifyFtyp = do + ftypSize <- getWord32be + when (ftypSize < 16) $ fail "Invalid ftyp size" + + ftyp <- getByteString 4 + unless (ftyp == "ftyp") $ fail "ftyp signature not found" + + brand <- getByteString 4 + unless (brand == "avif" || brand == "avis") $ fail "Not an AVIF file" + + -- Skip minor version and compatible brands + -- (we've read 12 bytes: size+type+brand) + let remaining_ftyp = fromIntegral ftypSize - 12 + when (remaining_ftyp > 0) $ skip remaining_ftyp + +findAvifDimensions :: Get (Word32, Word32) +findAvifDimensions = searchAvifBoxes [] + +searchAvifBoxes :: [B.ByteString] -> Get (Word32, Word32) +searchAvifBoxes path = do + isempty <- isEmpty + if isempty + then fail $ "No dimensions found. Searched: " ++ show (reverse path) + else do + boxSize <- getWord32be + boxType <- getByteString 4 + + let contentSize = fromIntegral boxSize - 8 + let newPath = boxType : path + + -- If it's a container box, search inside it + if isContainerBox boxType + then searchInsideBox contentSize newPath + else do + -- Try to parse dimensions from this box + result <- tryParseDimensions boxType contentSize + case result of + Just dims -> return dims + Nothing -> do + -- Skip this box and continue + when (contentSize > 0 && contentSize < 10000000) $ + skip contentSize + searchAvifBoxes path + +tryParseDimensions :: B.ByteString -> Int -> Get (Maybe (Word32, Word32)) +tryParseDimensions boxType size = do + pos <- bytesRead + result <- case boxType of + "ispe" -> parseIspeBox + "tkhd" -> parseTkhdBox + "stsd" -> parseStsdBox + "av01" -> parseAv01Box + _ -> return Nothing + + -- Reset position if we didn't find dimensions + case result of + Nothing -> do + newPos <- bytesRead + let consumed = fromIntegral (newPos - pos) + case size - consumed of + n | n > 0 -> skip n + _ -> return () + Just _ -> return () + + return result + +parseIspeBox :: Get (Maybe (Word32, Word32)) +parseIspeBox = do + skip 4 -- version/flags + width <- getWord32be + height <- getWord32be + return $ Just (width, height) + +parseTkhdBox :: Get (Maybe (Word32, Word32)) +parseTkhdBox = do + version <- getWord8 + skip 3 -- flags + + -- Skip to width/height based on version + let skipBytes = if version == 1 then 76 else 64 + skip skipBytes + + width <- getWord32be + height <- getWord32be + -- Convert from 16.16 fixed point + return $ Just (width `shiftR` 16, height `shiftR` 16) + +parseStsdBox :: Get (Maybe (Word32, Word32)) +parseStsdBox = do + skip 8 -- version, flags, entry count + findAv01Entry + +findAv01Entry :: Get (Maybe (Word32, Word32)) +findAv01Entry = do + entrySize <- getWord32be + codec <- getByteString 4 + + if codec == "av01" + then do + skip 6 -- reserved + skip 2 -- data reference index + skip 16 -- pre-defined + reserved + width <- getWord16be + height <- getWord16be + return $ Just (fromIntegral width, fromIntegral height) + else do + let skipSize = fromIntegral entrySize - 8 + when (skipSize > 0) $ skip skipSize + findAv01Entry + +parseAv01Box :: Get (Maybe (Word32, Word32)) +parseAv01Box = do + skip 6 -- reserved + skip 2 -- data reference index + skip 16 -- predefined/reserved + width <- getWord16be + height <- getWord16be + return $ Just (fromIntegral width, fromIntegral height) + +searchInsideBox :: Int -> [B.ByteString] -> Get (Word32, Word32) +searchInsideBox size path = do + -- For meta boxes, skip version/flags + let isMeta = case path of + "meta":_ -> True + _ -> False + when isMeta $ skip 4 + + let searchSize = if isMeta then size - 4 else size + searchAvifBoxesInRange searchSize path + +searchAvifBoxesInRange :: Int -> [B.ByteString] -> Get (Word32, Word32) +searchAvifBoxesInRange remaining' path + | remaining' < 8 = searchAvifBoxes path + | otherwise = do + boxSize <- getWord32be + boxType <- getByteString 4 + + let contentSize = fromIntegral boxSize - 8 + let newPath = boxType : path + + when (contentSize < 0 || fromIntegral boxSize > remaining') $ do + fail $ "Malformed box at path: " ++ show (reverse newPath) + + if isContainerBox boxType + then searchInsideBox contentSize newPath + else do + result <- tryParseDimensions boxType contentSize + case result of + Just dims -> return dims + Nothing -> do + -- Don't skip here - tryParseDimensions already handled it + searchAvifBoxesInRange (remaining' - fromIntegral boxSize) path + +isContainerBox :: B.ByteString -> Bool +isContainerBox boxType = boxType `elem` + ["moov", "trak", "mdia", "minf", "stbl", "meta", "dinf", "ipco", "iprp"] diff --git a/src/Text/Pandoc/Options.hs b/src/Text/Pandoc/Options.hs --- a/src/Text/Pandoc/Options.hs +++ b/src/Text/Pandoc/Options.hs @@ -3,6 +3,7 @@ {-# LANGUAGE DeriveGeneric #-} {-# LANGUAGE LambdaCase #-} {-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE PatternSynonyms #-} {-# LANGUAGE TemplateHaskell #-} {- | Module : Text.Pandoc.Options @@ -21,6 +22,10 @@ , HTMLMathMethod (..) , CiteMethod (..) , ObfuscationMethod (..) + , HighlightMethod (..) + , pattern NoHighlightingString + , pattern DefaultHighlightingString + , pattern IdiomaticHighlightingString , HTMLSlideVariant (..) , EPUBVersion (..) , WrapOption (..) @@ -32,6 +37,7 @@ , def , isEnabled , defaultMathJaxURL + , defaultWebTeXURL , defaultKaTeXURL ) where import Control.Applicative ((<|>)) @@ -41,13 +47,14 @@ import Data.Char (toLower) import Data.Text (Text) import qualified Data.Set as Set +import qualified Data.Text as T import Data.Typeable (Typeable) import GHC.Generics (Generic) import Skylighting (SyntaxMap, defaultSyntaxMap) import Text.DocTemplates (Context(..), Template) import Text.Pandoc.Extensions import Text.Pandoc.Chunks (PathTemplate) -import Text.Pandoc.Highlighting (Style, pygments) +import Text.Pandoc.Highlighting (Style) import Text.Pandoc.UTF8 (toStringLazy) import Data.Aeson.TH (deriveJSON) import Data.Aeson @@ -114,7 +121,8 @@ mburl <- m .:? "url" case method :: Text of "plain" -> return PlainMath - "webtex" -> return $ WebTeX $ fromMaybe "" mburl + "webtex" -> return $ WebTeX $ + fromMaybe defaultWebTeXURL mburl "gladtex" -> return GladTeX "mathml" -> return MathML "mathjax" -> return $ MathJax $ @@ -124,7 +132,7 @@ _ -> fail $ "Unknown HTML math method " ++ show method) node <|> (case node of String "plain" -> return PlainMath - String "webtex" -> return $ WebTeX "" + String "webtex" -> return $ WebTeX defaultWebTeXURL String "gladtex" -> return GladTeX String "mathml" -> return MathML String "mathjax" -> return $ MathJax defaultMathJaxURL @@ -184,6 +192,43 @@ toJSON ReferenceObfuscation = String "references" toJSON JavascriptObfuscation = String "javascript" +-- | Method to provide code highlighting. +data HighlightMethod = + Skylighting Style + | IdiomaticHighlighting + | DefaultHighlighting + | NoHighlighting + deriving (Show, Read, Eq, Data, Typeable, Generic) + +-- | String representation of the idiomatic highlighting option. +pattern IdiomaticHighlightingString :: Text +pattern IdiomaticHighlightingString = "idiomatic" + +-- | String representation of the default highlighting option. +pattern DefaultHighlightingString :: Text +pattern DefaultHighlightingString = "default" + +-- | String representation of the no highlighting option +pattern NoHighlightingString :: Text +pattern NoHighlightingString = "none" + +instance ToJSON HighlightMethod where + toJSON NoHighlighting = String NoHighlightingString + toJSON IdiomaticHighlighting = String IdiomaticHighlightingString + toJSON DefaultHighlighting = String DefaultHighlightingString + toJSON (Skylighting style) = toJSON style + +instance FromJSON HighlightMethod where + parseJSON = \case + String NoHighlightingString -> pure NoHighlighting + String IdiomaticHighlightingString -> pure IdiomaticHighlighting + String DefaultHighlightingString -> pure DefaultHighlighting + String x -> fail $ "Unknown highlighting method " <> T.unpack x + Bool True -> pure DefaultHighlighting + Bool False -> pure NoHighlighting + Null -> pure NoHighlighting + v -> Skylighting <$> parseJSON v + -- | Varieties of HTML slide shows. data HTMLSlideVariant = S5Slides | SlidySlides @@ -328,9 +373,7 @@ , writerHtmlQTags :: Bool -- ^ Use @@ tags for quotes in HTML , writerSlideLevel :: Maybe Int -- ^ Force header level of slides , writerTopLevelDivision :: TopLevelDivision -- ^ Type of top-level divisions - , writerListings :: Bool -- ^ Use listings package for code - , writerHighlightStyle :: Maybe Style -- ^ Style to use for highlighting - -- (Nothing = no highlighting) + , writerHighlightMethod :: HighlightMethod -- ^ Style to use for highlighting , writerSetextHeaders :: Bool -- ^ Use setext headers for levels 1-2 in markdown , writerListTables :: Bool -- ^ Use list tables for RST tables , writerEpubSubdirectory :: Text -- ^ Subdir for epub in OCF @@ -372,8 +415,7 @@ , writerHtmlQTags = False , writerSlideLevel = Nothing , writerTopLevelDivision = TopLevelDefault - , writerListings = False - , writerHighlightStyle = Just pygments + , writerHighlightMethod = DefaultHighlighting , writerSetextHeaders = False , writerListTables = False , writerEpubSubdirectory = "EPUB" @@ -401,6 +443,9 @@ defaultMathJaxURL :: Text defaultMathJaxURL = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-chtml-full.js" + +defaultWebTeXURL :: Text +defaultWebTeXURL = "https://latex.codecogs.com/png.latex?" defaultKaTeXURL :: Text defaultKaTeXURL = "https://cdn.jsdelivr.net/npm/katex@latest/dist/" diff --git a/src/Text/Pandoc/PDF.hs b/src/Text/Pandoc/PDF.hs --- a/src/Text/Pandoc/PDF.hs +++ b/src/Text/Pandoc/PDF.hs @@ -43,6 +43,7 @@ import Text.DocLayout (literal, render, hsep) import Text.Pandoc.Definition import Text.Pandoc.Error (PandocError (PandocPDFProgramNotFoundError)) +import Text.Pandoc.SelfContained (makeSelfContained) import Text.Pandoc.MIME (getMimeType) import Text.Pandoc.Options (HTMLMathMethod (..), WriterOptions (..)) import Text.Pandoc.Extensions (disableExtension, Extension(Ext_smart)) @@ -57,7 +58,8 @@ import Data.List (intercalate) #endif import Data.List (isPrefixOf, find) -import Text.Pandoc.Class (fillMediaBag, getVerbosity, setVerbosity, +import Text.Pandoc.MediaBag (mediaItems) +import Text.Pandoc.Class (fillMediaBag, getMediaBag, getVerbosity, setVerbosity, readFileStrict, fileExists, report, extractMedia, PandocMonad, runIOorExplode) import Text.Pandoc.Logging @@ -81,24 +83,36 @@ -> WriterOptions -- ^ options -> Pandoc -- ^ document -> m (Either ByteString ByteString) -makePDF program pdfargs writer opts doc = +makePDF program pdfargs writer opts doc = withTempDir (program == "typst") "media" $ \mediaDir -> do +#ifdef _WINDOWS + -- note: we want / even on Windows, for TexLive + let tmpdir = changePathSeparators mediaDir +#else + let tmpdir = mediaDir +#endif + doc' <- handleImages opts tmpdir doc + source <- writer opts{ writerExtensions = -- disable use of quote + -- ligatures to avoid bad ligatures like ?` + disableExtension Ext_smart + (writerExtensions opts) } doc' + verbosity <- getVerbosity + let compileHTML mkOutArgs = do + -- check to see if there is anything in mediabag, and if so, + -- make the HTML self contained + mediabag <- getMediaBag + source' <- case mediaItems mediabag of + [] -> pure source + _ -> makeSelfContained source + liftIO $ + toPdfViaTempFile verbosity program pdfargs mkOutArgs ".html" source' case takeBaseName program of "wkhtmltopdf" -> makeWithWkhtmltopdf program pdfargs writer opts doc - prog | prog `elem` ["pagedjs-cli" ,"weasyprint", "prince"] -> do - let mkOutArgs f = - if program `elem` ["pagedjs-cli", "prince"] - then ["-o", f] - else [f] - source <- writer opts doc - verbosity <- getVerbosity - liftIO $ toPdfViaTempFile verbosity program pdfargs mkOutArgs ".html" source - "typst" -> do - source <- writer opts doc - verbosity <- getVerbosity - liftIO $ + "pagedjs-cli" -> compileHTML (\f -> ["-o", f]) + "prince" -> compileHTML (\f -> ["-o", f]) + "weasyprint" -> compileHTML (:[]) + "typst" -> liftIO $ toPdfViaTempFile verbosity program ("compile":pdfargs) (:[]) ".typ" source "pdfroff" -> do - source <- writer opts doc let paperargs = case lookupContext "papersize" (writerVariables opts) of Just s @@ -112,7 +126,6 @@ paperargs ++ pdfargs generic2pdf program args source "groff" -> do - source <- writer opts doc let paperargs = case lookupContext "papersize" (writerVariables opts) of Just s @@ -125,33 +138,22 @@ ["-U" | ".PDFPIC" `T.isInfixOf` source] ++ paperargs ++ pdfargs generic2pdf program args source - baseProg -> do - withTempDir "tex2pdf." $ \tmpdir' -> do -#ifdef _WINDOWS - -- note: we want / even on Windows, for TexLive - let tmpdir = changePathSeparators tmpdir' -#else - let tmpdir = tmpdir' -#endif - doc' <- handleImages opts tmpdir doc - source <- writer opts{ writerExtensions = -- disable use of quote - -- ligatures to avoid bad ligatures like ?` - disableExtension Ext_smart - (writerExtensions opts) } doc' - case baseProg of - "context" -> context2pdf program pdfargs tmpdir source - "tectonic" -> tectonic2pdf program pdfargs tmpdir source - prog | prog `elem` ["pdflatex", "lualatex", "xelatex", "latexmk"] - -> tex2pdf program pdfargs tmpdir source - _ -> return $ Left $ UTF8.fromStringLazy - $ "Unknown program " ++ program + "context" -> context2pdf program pdfargs tmpdir source + "tectonic" -> tectonic2pdf program pdfargs tmpdir source + "latexmk" -> tex2pdf program pdfargs tmpdir source + "lualatex" -> tex2pdf program pdfargs tmpdir source + "lualatex-dev" -> tex2pdf program pdfargs tmpdir source + "pdflatex" -> tex2pdf program pdfargs tmpdir source + "pdflatex-dev" -> tex2pdf program pdfargs tmpdir source + "xelatex" -> tex2pdf program pdfargs tmpdir source + _ -> return $ Left $ UTF8.fromStringLazy $ "Unknown program " ++ program -- latex has trouble with tildes in paths, which -- you find in Windows temp dir paths with longer -- user names (see #777) withTempDir :: (PandocMonad m, MonadMask m, MonadIO m) - => FilePath -> (FilePath -> m a) -> m a -withTempDir templ action = do + => Bool -> FilePath -> (FilePath -> m a) -> m a +withTempDir useWorkingDirectory templ action = do tmp <- liftIO getTemporaryDirectory uname <- liftIO $ E.catch (do (ec, sout, _) <- readProcessWithExitCode "uname" ["-o"] "" @@ -159,9 +161,9 @@ then return $ Just $ filter (not . isSpace) sout else return Nothing) (\(_ :: E.SomeException) -> return Nothing) - if '~' `elem` tmp || uname == Just "Cygwin" -- see #5451 - then withTempDirectory "." templ action - else withSystemTempDirectory templ action + if useWorkingDirectory || '~' `elem` tmp || uname == Just "Cygwin" -- see #5451 + then withTempDirectory "." templ action + else withSystemTempDirectory templ action makeWithWkhtmltopdf :: (PandocMonad m, MonadIO m) => String -- ^ wkhtmltopdf or path @@ -328,7 +330,7 @@ go (Just msg) ln | ln == "" = do -- emit report and reset accumulator report $ MakePDFWarning $ render (Just 60) $ - hsep $ map literal $ T.words $ UTF8.toText $ BC.toStrict msg + hsep $ map literal $ T.words $ utf8ToText msg pure Nothing | otherwise = pure $ Just (msg <> ln) @@ -372,7 +374,7 @@ (exit, out) <- liftIO $ E.catch (pipeProcess (Just env) program programArgs sourceBL) (handlePDFProgramNotFound program) - report $ MakePDFInfo "tectonic output" (UTF8.toText $ BL.toStrict out) + report $ MakePDFInfo "tectonic output" (utf8ToText out) let pdfFile = tmpDir ++ "/texput.pdf" (_, pdf) <- getResultingPDF Nothing pdfFile return (exit, out, pdf) @@ -435,7 +437,7 @@ (exit, out) <- liftIO $ E.catch (pipeProcess (Just env'') program programArgs BL.empty) (handlePDFProgramNotFound program) - report $ MakePDFInfo "LaTeX output" (UTF8.toText $ BL.toStrict out) + report $ MakePDFInfo "LaTeX output" (utf8ToText out) -- parse log to see if we need to rerun LaTeX let logFile = replaceExtension outfile ".log" logExists <- fileExists logFile @@ -449,8 +451,7 @@ if not (null rerunWarnings') && runNumber < maxruns then do report $ MakePDFInfo "Rerun needed" - (T.intercalate "\n" - (map (UTF8.toText . BC.toStrict) rerunWarnings')) + (T.intercalate "\n" (map utf8ToText rerunWarnings')) go env'' programArgs (runNumber + 1) else do (log', pdf) <- getResultingPDF (Just logFile) outfile @@ -503,7 +504,7 @@ (handlePDFProgramNotFound program) runIOorExplode $ do setVerbosity verbosity - report $ MakePDFInfo "pdf-engine output" (UTF8.toText $ BL.toStrict out) + report $ MakePDFInfo "pdf-engine output" (utf8ToText out) pdfExists <- doesFileExist pdfFile mbPdf <- if pdfExists -- We read PDF as a strict bytestring to make sure that the @@ -536,7 +537,7 @@ (handlePDFProgramNotFound program) runIOorExplode $ do setVerbosity verbosity - report $ MakePDFInfo "ConTeXt run output" (UTF8.toText $ BL.toStrict out) + report $ MakePDFInfo "ConTeXt run output" (utf8ToText out) let pdfFile = replaceExtension file ".pdf" pdfExists <- doesFileExist pdfFile mbPdf <- if pdfExists diff --git a/src/Text/Pandoc/Parsing/GridTable.hs b/src/Text/Pandoc/Parsing/GridTable.hs --- a/src/Text/Pandoc/Parsing/GridTable.hs +++ b/src/Text/Pandoc/Parsing/GridTable.hs @@ -34,6 +34,7 @@ import Text.Pandoc.Sources import Text.Parsec (Stream (..), ParsecT, optional, sepEndBy1, try) +import Data.Maybe (mapMaybe) import qualified Data.Text as T import qualified Text.GridTable as GT import qualified Text.Pandoc.Builder as B @@ -54,17 +55,17 @@ B.tableWith attr capt colspecs th tb tf -- | Bundles basic table components into a single value. -toTableComponents :: [Alignment] -> [Double] -> [Blocks] -> [[Blocks]] +toTableComponents :: [Alignment] -> [Double] -> [[Blocks]] -> [[Blocks]] -> TableComponents toTableComponents = toTableComponents' NoNormalization -- | Bundles basic table components into a single value, performing -- normalizations as necessary. toTableComponents' :: TableNormalization - -> [Alignment] -> [Double] -> [Blocks] -> [[Blocks]] + -> [Alignment] -> [Double] -> [[Blocks]] -> [[Blocks]] -> TableComponents toTableComponents' normalization aligns widths heads rows = - let th = TableHead nullAttr (toHeaderRow normalization heads) + let th = TableHead nullAttr (mapMaybe (toHeaderRow normalization) heads) tb = TableBody nullAttr 0 [] (map toRow rows) tf = TableFoot nullAttr [] colspecs = toColSpecs aligns widths @@ -191,7 +192,7 @@ -- 'lineParser', and 'footerParser'. tableWith :: (Stream s m Char, UpdateSourcePos s Char, HasReaderOptions st, Monad mf) - => ParsecT s st m (mf [Blocks], [Alignment], [Int]) -- ^ header parser + => ParsecT s st m (mf [[Blocks]], [Alignment], [Int]) -- ^ header parser -> ([Int] -> ParsecT s st m (mf [Blocks])) -- ^ row parser -> ParsecT s st m sep -- ^ line parser -> ParsecT s st m end -- ^ footer parser @@ -202,7 +203,7 @@ tableWith' :: (Stream s m Char, UpdateSourcePos s Char, HasReaderOptions st, Monad mf) => TableNormalization - -> ParsecT s st m (mf [Blocks], [Alignment], [Int]) -- ^ header parser + -> ParsecT s st m (mf [[Blocks]], [Alignment], [Int]) -- ^ header parser -> ([Int] -> ParsecT s st m (mf [Blocks])) -- ^ row parser -> ParsecT s st m sep -- ^ line parser -> ParsecT s st m end -- ^ footer parser @@ -220,10 +221,10 @@ toRow :: [Blocks] -> Row toRow = Row nullAttr . map B.simpleCell -toHeaderRow :: TableNormalization -> [Blocks] -> [Row] +toHeaderRow :: TableNormalization -> [Blocks] -> Maybe Row toHeaderRow = \case - NoNormalization -> \l -> [toRow l | not (null l)] - NormalizeHeader -> \l -> [toRow l | not (null l) && not (all null l)] + NoNormalization -> \l -> if not (null l) then Just (toRow l) else Nothing + NormalizeHeader -> \l -> if not (all null l) then Just (toRow l) else Nothing -- | Calculate relative widths of table columns, based on indices widthsFromIndices :: Int -- Number of columns on terminal diff --git a/src/Text/Pandoc/Readers.hs b/src/Text/Pandoc/Readers.hs --- a/src/Text/Pandoc/Readers.hs +++ b/src/Text/Pandoc/Readers.hs @@ -65,6 +65,7 @@ , readTypst , readDjot , readPod + , readXML -- * Miscellaneous , getReader , getDefaultExtensions @@ -118,6 +119,7 @@ import Text.Pandoc.Readers.RTF import Text.Pandoc.Readers.Typst import Text.Pandoc.Readers.Djot +import Text.Pandoc.Readers.XML import qualified Text.Pandoc.UTF8 as UTF8 import Text.Pandoc.Sources (ToSources(..), sourcesToText) @@ -174,6 +176,7 @@ ,("djot" , TextReader readDjot) ,("mdoc" , TextReader readMdoc) ,("pod" , TextReader readPod) + ,("xml" , TextReader readXML) ] -- | Retrieve reader, extensions based on format spec (format+extensions). diff --git a/src/Text/Pandoc/Readers/DocBook.hs b/src/Text/Pandoc/Readers/DocBook.hs --- a/src/Text/Pandoc/Readers/DocBook.hs +++ b/src/Text/Pandoc/Readers/DocBook.hs @@ -15,6 +15,7 @@ -} module Text.Pandoc.Readers.DocBook ( readDocBook ) where import Control.Monad (MonadPlus(mplus)) +import Control.Applicative () import Control.Monad.State.Strict ( MonadTrans(lift), StateT(runStateT), @@ -52,6 +53,7 @@ import qualified Data.Map as M import Text.Pandoc.XML.Light import Text.Pandoc.Walk (query) +import Text.Read (readMaybe) {- @@ -931,9 +933,7 @@ "lowerroman" -> LowerRoman "upperroman" -> UpperRoman _ -> Decimal - let start = fromMaybe 1 $ - filterElement (named "listitem") e - >>= safeRead . attrValue "override" + let start = fromMaybe 1 $ safeRead $ attrValue "startingnumber" e orderedListWith (start,listStyle,DefaultDelim) . handleCompact <$> listitems "variablelist" -> definitionList <$> deflistitems @@ -1056,9 +1056,8 @@ cs -> mapMaybe (findAttr (unqual "colname" )) cs let isRow x = named "row" x || named "tr" x headrows <- case filterChild (named "thead") e' of - Just h -> case filterChild isRow h of - Just x -> parseRow colnames x - Nothing -> return [] + Just h -> mapM (parseRow colnames) + $ filterChildren isRow h Nothing -> return [] bodyrows <- case filterChild (named "tbody") e' of Just b -> mapM (parseRow colnames) @@ -1072,7 +1071,7 @@ || x == '.') w if n > 0 then Just n else Nothing let numrows = maybe 0 maximum $ nonEmpty - $ map length bodyrows + $ map length (bodyrows ++ headrows) let aligns = case colspecs of [] -> replicate numrows AlignDefault cs -> map toAlignment cs @@ -1094,11 +1093,10 @@ in ColWidth . scale <$> ws' Nothing -> replicate numrows ColWidthDefault let toRow = Row nullAttr - toHeaderRow l = [toRow l | not (null l)] return $ tableWith (elId,classes,attrs) (simpleCaption $ plain capt) (zip aligns widths) - (TableHead nullAttr $ toHeaderRow headrows) + (TableHead nullAttr $ map toRow headrows) [TableBody nullAttr 0 [] $ map toRow bodyrows] (TableFoot nullAttr []) sect n = sectWith(attrValue "id" e) [] [] n @@ -1172,7 +1170,7 @@ parseRow :: PandocMonad m => [Text] -> Element -> DB m [Cell] parseRow cn = do - let isEntry x = named "entry" x || named "td" x || named "th" x + let isEntry x = named "entry" x || named "td" x || named "th" x mapM (parseEntry cn) . filterChildren isEntry parseEntry :: PandocMonad m => [Text] -> Element -> DB m Cell @@ -1189,9 +1187,18 @@ case (mStrt, mEnd) of (Just start, Just end) -> colDistance start end _ -> 1 + let rowDistance mr = do + case readMaybe $ T.unpack mr :: Maybe Int of + Just moreRow -> RowSpan $ moreRow + 1 + _ -> 1 + let toRowSpan en = do + case findAttr (unqual "morerows") en of + Just moreRow -> rowDistance moreRow + _ -> 1 let colSpan = toColSpan el + let rowSpan = toRowSpan el let align = toAlignment el - (fmap (cell align 1 colSpan) . parseMixed plain . elContent) el + (fmap (cell align rowSpan colSpan) . parseMixed plain . elContent) el getInlines :: PandocMonad m => Element -> DB m Inlines getInlines e' = trimInlines . mconcat <$> diff --git a/src/Text/Pandoc/Readers/Docx.hs b/src/Text/Pandoc/Readers/Docx.hs --- a/src/Text/Pandoc/Readers/Docx.hs +++ b/src/Text/Pandoc/Readers/Docx.hs @@ -305,7 +305,8 @@ emph . go rPr{isItalic = Nothing, isItalicCTL = Nothing} | Just True <- bold rPr = strong . go rPr{isBold = Nothing, isBoldCTL = Nothing} - | Just _ <- rHighlight rPr = + | Just v <- rHighlight rPr + , v /= "none" = spanWith ("",["mark"],[]) . go rPr{rHighlight = Nothing} | Just True <- isSmallCaps rPr = smallcaps . go rPr{isSmallCaps = Nothing} diff --git a/src/Text/Pandoc/Readers/Docx/Parse.hs b/src/Text/Pandoc/Readers/Docx/Parse.hs --- a/src/Text/Pandoc/Readers/Docx/Parse.hs +++ b/src/Text/Pandoc/Readers/Docx/Parse.hs @@ -92,13 +92,13 @@ filterChild, filterChildrenName, filterElementName, + filterElementsName, lookupAttrBy, parseXMLElement, elChildren, QName(QName, qName), Content(Elem), - Element(..), - findElements ) + Element(..)) data ReaderEnv = ReaderEnv { envNotes :: Notes , envComments :: Comments @@ -151,46 +151,26 @@ in concatMapM handler xs -isAltContentRun :: NameSpaces -> Element -> Bool -isAltContentRun ns element - | isElem ns "w" "r" element - , Just _altContentElem <- findChildByName ns "mc" "AlternateContent" element - = True - | otherwise - = False - --- Elements such as are not always preferred --- to be unwrapped. Only if they are part of an AlternateContent --- element, they should be unwrapped. --- This strategy prevents VML images breaking. -unwrapAlternateContentElement :: NameSpaces -> Element -> [Element] -unwrapAlternateContentElement ns element - | isElem ns "mc" "AlternateContent" element - || isElem ns "mc" "Fallback" element - || isElem ns "w" "pict" element - || isElem ns "v" "group" element - || isElem ns "v" "rect" element - || isElem ns "v" "roundrect" element - || isElem ns "v" "shape" element - || isElem ns "v" "textbox" element - || isElem ns "w" "txbxContent" element - = concatMap (unwrapAlternateContentElement ns) (elChildren element) - | otherwise - = unwrapElement ns element - unwrapElement :: NameSpaces -> Element -> [Element] unwrapElement ns element | isElem ns "w" "sdt" element , Just sdtContent <- findChildByName ns "w" "sdtContent" element = concatMap (unwrapElement ns) (elChildren sdtContent) - | isElem ns "w" "r" element - , Just alternateContentElem <- findChildByName ns "mc" "AlternateContent" element - = unwrapAlternateContentElement ns alternateContentElem | isElem ns "w" "smartTag" element = concatMap (unwrapElement ns) (elChildren element) | isElem ns "w" "p" element - , Just (modified, altContentRuns) <- extractChildren element (isAltContentRun ns) - = (unwrapElement ns modified) ++ concatMap (unwrapElement ns) altContentRuns + , textboxes@(_:_) <- findChildrenByName ns "w" "r" element >>= + findChildrenByName ns "mc" "AlternateContent" >>= + findChildrenByName ns "mc" "Fallback" >>= + findChildrenByName ns "w" "pict" >>= + findChildrenByName ns "v" "shape" >>= + findChildrenByName ns "v" "textbox" >>= + findChildrenByName ns "w" "txbxContent" + = concatMap (unwrapElement ns) (concatMap elChildren textboxes) -- handle #9214 + | isElem ns "w" "r" element + , Just fallback <- findChildByName ns "mc" "AlternateContent" element >>= + findChildByName ns "mc" "Fallback" + = [element{ elContent = concatMap (unwrapContent ns) (elContent fallback) }] | otherwise = [element{ elContent = concatMap (unwrapContent ns) (elContent element) }] @@ -453,8 +433,10 @@ entry <- findEntryByPath "_rels/.rels" zf relsElem <- parseXMLFromEntry entry let rels = filterChildrenName (\n -> qName n == "Relationship") relsElem - rel <- find (\e -> findAttr (QName "Type" Nothing Nothing) e == - Just "http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument") + rel <- find (\e -> + case findAttr (QName "Type" Nothing Nothing) e of + Just u -> isNamespace "officeDocument" "relationships/officeDocument" u + Nothing -> False) rels fp <- findAttr (QName "Target" Nothing Nothing) rel -- sometimes there will be a leading slash, which windows seems to @@ -1022,8 +1004,8 @@ elemToParPart' ns element | isElem ns "w" "r" element , Just drawingElem <- findChildByName ns "w" "drawing" element - , pic_ns <- "http://schemas.openxmlformats.org/drawingml/2006/picture" - , picElems <- findElements (QName "pic" (Just pic_ns) (Just "pic")) drawingElem + , picElems <- filterElementsName + (matchQName "drawingml" "picture" (Just "pic") "pic") drawingElem = let (title, alt) = getTitleAndAlt ns drawingElem drawings = map (\el -> ((findBlip el >>= findAttrByName ns "r" "embed"), el)) @@ -1057,15 +1039,15 @@ elemToParPart' ns element | isElem ns "w" "r" element , Just drawingElem <- findChildByName ns "w" "drawing" element - , d_ns <- "http://schemas.openxmlformats.org/drawingml/2006/diagram" - , Just _ <- findElement (QName "relIds" (Just d_ns) (Just "dgm")) drawingElem + , Just _ <- filterElementName + (matchQName "drawingml" "diagram" (Just "dgm") "relIds") drawingElem = return [Diagram] -- Chart elemToParPart' ns element | isElem ns "w" "r" element , Just drawingElem <- findChildByName ns "w" "drawing" element - , c_ns <- "http://schemas.openxmlformats.org/drawingml/2006/chart" - , Just _ <- findElement (QName "chart" (Just c_ns) (Just "c")) drawingElem + , Just _ <- filterElementName + (matchQName "drawingml" "chart" (Just "c") "chart") drawingElem = return [Chart] elemToParPart' ns element | isElem ns "w" "r" element = do @@ -1146,8 +1128,8 @@ childElemToRun :: NameSpaces -> Element -> D [Run] childElemToRun ns element | isElem ns "w" "drawing" element - , pic_ns <- "http://schemas.openxmlformats.org/drawingml/2006/picture" - , picElems <- findElements (QName "pic" (Just pic_ns) (Just "pic")) element + , picElems <- filterElementsName + (matchQName "drawingml" "picture" (Just "pic") "pic") element = let (title, alt) = getTitleAndAlt ns element drawings = map (\el -> ((findBlip el >>= findAttrByName ns "r" "embed"), el)) @@ -1161,13 +1143,13 @@ drawings childElemToRun ns element | isElem ns "w" "drawing" element - , c_ns <- "http://schemas.openxmlformats.org/drawingml/2006/chart" - , Just _ <- findElement (QName "chart" (Just c_ns) (Just "c")) element + , Just _ <- filterElementName + (matchQName "drawingml" "chart" (Just "c") "chart") element = return [InlineChart] childElemToRun ns element | isElem ns "w" "drawing" element - , c_ns <- "http://schemas.openxmlformats.org/drawingml/2006/diagram" - , Just _ <- findElement (QName "relIds" (Just c_ns) (Just "dgm")) element + , Just _ <- filterElementName + (matchQName "drawingml" "diagram" (Just "dgm") "relIds") element = return [InlineDiagram] childElemToRun ns element | isElem ns "w" "footnoteReference" element @@ -1190,15 +1172,6 @@ elemToRun :: NameSpaces -> Element -> D [Run] elemToRun ns element | isElem ns "w" "r" element - , Just altCont <- findChildByName ns "mc" "AlternateContent" element = - do let choices = findChildrenByName ns "mc" "Choice" altCont - choiceChildren = concatMap (take 1 . elChildren) choices - outputs <- mapD (childElemToRun ns) choiceChildren - case outputs of - r : _ -> return r - [] -> throwError WrongElem -elemToRun ns element - | isElem ns "w" "r" element , Just drawingElem <- findChildByName ns "w" "drawing" element = childElemToRun ns drawingElem elemToRun ns element @@ -1358,11 +1331,9 @@ findBlip :: Element -> Maybe Element findBlip el = do - blip <- findElement (QName "blip" (Just a_ns) (Just "a")) el + blip <- filterElementName (matchQName "drawingml" "main" (Just "a") "blip") el -- return svg if present: filterElementName (\(QName tag _ _) -> tag == "svgBlip") el `mplus` pure blip - where - a_ns = "http://schemas.openxmlformats.org/drawingml/2006/main" hasCaptionStyle :: ParagraphStyle -> Bool hasCaptionStyle parstyle = any (isCaptionStyleName . pStyleName) (pStyle parstyle) @@ -1387,3 +1358,23 @@ (qName name == "instrText" && let ws = T.words (strContent el) in ("Table" `elem` ws || "Figure" `elem` ws)) + +isNamespace :: Text -> Text -> Text -> Bool +isNamespace primary secondary url = + -- first try transitional: + case T.stripPrefix "http://schemas.openxmlformats.org/" url of + Just path -> path == primary <> "/2006/" <> secondary + Nothing -> -- then try strict: + case T.stripPrefix "http://purl.oclc.org/ooxml/" url of + Just path -> path == primary <> "/" <> snakeToCamel secondary + Nothing -> False + where + snakeToCamel "custom-properties" = "customProperties" + snakeToCamel "extended-properties" = "extendedProperties" + snakeToCamel x = x + +matchQName :: Text -> Text -> Maybe Text -> Text -> QName -> Bool +matchQName primary secondary mbprefix name (QName name' mbns' mbprefix') = + name == name' && + (isNothing mbprefix || mbprefix' == mbprefix) && + maybe True (isNamespace primary secondary) mbns' diff --git a/src/Text/Pandoc/Readers/Docx/Parse/Styles.hs b/src/Text/Pandoc/Readers/Docx/Parse/Styles.hs --- a/src/Text/Pandoc/Readers/Docx/Parse/Styles.hs +++ b/src/Text/Pandoc/Readers/Docx/Parse/Styles.hs @@ -239,8 +239,8 @@ stringToInteger :: Text -> Maybe Integer stringToInteger s = case Data.Text.Read.decimal s of - Right (x,_) -> Just x - Left _ -> Nothing + Right (x,t) | T.null t -> Just x + _ -> Nothing checkOnOff :: NameSpaces -> Element -> QName -> Maybe Bool checkOnOff ns rPr tag diff --git a/src/Text/Pandoc/Readers/EndNote.hs b/src/Text/Pandoc/Readers/EndNote.hs --- a/src/Text/Pandoc/Readers/EndNote.hs +++ b/src/Text/Pandoc/Readers/EndNote.hs @@ -81,6 +81,8 @@ let items = map toCitationItem $ filterElementsName (name "Cite") tree return $ Citeproc.Citation{ Citeproc.citationId = Nothing + , Citeproc.citationPrefix = Nothing + , Citeproc.citationSuffix = Nothing , Citeproc.citationNoteNumber = Nothing , Citeproc.citationItems = items } diff --git a/src/Text/Pandoc/Readers/HTML.hs b/src/Text/Pandoc/Readers/HTML.hs --- a/src/Text/Pandoc/Readers/HTML.hs +++ b/src/Text/Pandoc/Readers/HTML.hs @@ -659,14 +659,10 @@ else v ] contents <- manyTill pAny (pCloses "pre" <|> eof) let rawText = T.concat $ map tagToText contents - -- drop leading newline if any - let result' = case T.uncons rawText of - Just ('\n', xs) -> xs - _ -> rawText -- drop trailing newline if any - let result = case T.unsnoc result' of + let result = case T.unsnoc rawText of Just (result'', '\n') -> result'' - _ -> result' + _ -> rawText return $ B.codeBlockWith attr result tagToText :: Tag Text -> Text diff --git a/src/Text/Pandoc/Readers/LaTeX.hs b/src/Text/Pandoc/Readers/LaTeX.hs --- a/src/Text/Pandoc/Readers/LaTeX.hs +++ b/src/Text/Pandoc/Readers/LaTeX.hs @@ -52,8 +52,8 @@ import Text.Pandoc.TeX (Tok (..), TokType (..)) import Text.Pandoc.Readers.LaTeX.Parsing import Text.Pandoc.Readers.LaTeX.Citation (citationCommands, cites) -import Text.Pandoc.Readers.LaTeX.Math (dollarsMath, inlineEnvironments, - inlineEnvironment, +import Text.Pandoc.Readers.LaTeX.Math (withMathMode, dollarsMath, + inlineEnvironments, inlineEnvironment, mathDisplay, mathInline, newtheorem, theoremstyle, proof, theoremEnvironment) @@ -371,9 +371,11 @@ , ("hbox", rawInlineOr "hbox" $ processHBox <$> tok) , ("vbox", rawInlineOr "vbox" tok) , ("lettrine", rawInlineOr "lettrine" lettrine) - , ("(", mathInline . untokenize <$> manyTill anyTok (controlSeq ")")) - , ("[", mathDisplay . untokenize <$> manyTill anyTok (controlSeq "]")) - , ("ensuremath", mathInline . untokenize <$> braced) + , ("(", withMathMode + (mathInline . untokenize <$> manyTill anyTok (controlSeq ")"))) + , ("[", withMathMode + (mathDisplay . untokenize <$> manyTill anyTok (controlSeq "]"))) + , ("ensuremath", withMathMode (mathInline . untokenize <$> braced)) , ("texorpdfstring", const <$> tok <*> tok) -- old TeX commands , ("em", extractSpaces emph <$> inlines) @@ -961,6 +963,7 @@ , ("paragraph*", section ("",["unnumbered"],[]) 4) , ("subparagraph", section nullAttr 5) , ("subparagraph*", section ("",["unnumbered"],[]) 5) + , ("minisec", section ("",["unnumbered","unlisted"],[]) 6) -- from KOMA -- beamer slides , ("frametitle", section nullAttr 3) , ("framesubtitle", section nullAttr 4) @@ -1035,8 +1038,8 @@ , ("letter", env "letter" letterContents) , ("minipage", divWith ("",["minipage"],[]) <$> env "minipage" (skipopts *> spaces *> optional braced *> spaces *> blocks)) - , ("figure", env "figure" $ skipopts *> figure') - , ("figure*", env "figure*" $ skipopts *> figure') + , ("figure", env "figure" figure') + , ("figure*", env "figure*" figure') , ("subfigure", env "subfigure" $ skipopts *> tok *> figure') , ("center", divWith ("", ["center"], []) <$> env "center" blocks) , ("quote", blockQuote <$> env "quote" blocks) @@ -1208,15 +1211,18 @@ figure' :: PandocMonad m => LP m Blocks figure' = try $ do + sp + poshint <- option "" $ untokenize <$> bracketedToks + sp resetCaption innerContent <- many $ try (Left <$> label) <|> (Right <$> block) let content = walk go $ mconcat $ snd $ partitionEithers innerContent st <- getState let caption' = fromMaybe B.emptyCaption $ sCaption st let mblabel = sLastLabel st - let attr = case mblabel of - Just lab -> (lab, [], []) - Nothing -> nullAttr + let kvs = [("latex-placement", poshint) | not (T.null poshint)] + let ident = fromMaybe "" mblabel + let attr = (ident, [], kvs) case mblabel of Nothing -> pure () Just lab -> do diff --git a/src/Text/Pandoc/Readers/LaTeX/Inline.hs b/src/Text/Pandoc/Readers/LaTeX/Inline.hs --- a/src/Text/Pandoc/Readers/LaTeX/Inline.hs +++ b/src/Text/Pandoc/Readers/LaTeX/Inline.hs @@ -281,6 +281,7 @@ , ("_", lit "_") , ("{", lit "{") , ("}", lit "}") + , ("-", lit "\x00ad") -- soft hyphen , ("qed", lit "\a0\x25FB") , ("lq", return (str "‘")) , ("rq", return (str "’")) diff --git a/src/Text/Pandoc/Readers/LaTeX/Math.hs b/src/Text/Pandoc/Readers/LaTeX/Math.hs --- a/src/Text/Pandoc/Readers/LaTeX/Math.hs +++ b/src/Text/Pandoc/Readers/LaTeX/Math.hs @@ -1,6 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} module Text.Pandoc.Readers.LaTeX.Math - ( dollarsMath + ( withMathMode + , dollarsMath , inlineEnvironments , inlineEnvironmentNames , inlineEnvironment @@ -28,11 +29,19 @@ import qualified Data.Map as M import Data.Text (Text) +withMathMode :: PandocMonad m => LP m a -> LP m a +withMathMode p = do + oldMathMode <- sMathMode <$> getState + updateState $ \s -> s{ sMathMode = True } + result <- p + updateState $ \s -> s{ sMathMode = oldMathMode } + return result + dollarsMath :: PandocMonad m => LP m Inlines dollarsMath = do symbol '$' display <- option False (True <$ symbol '$') - (do contents <- try $ untokenize <$> pDollarsMath 0 + (do contents <- try $ untokenize <$> withMathMode (pDollarsMath 0) if display then mathDisplay contents <$ symbol '$' else return $ mathInline contents) @@ -70,7 +79,7 @@ "\n\\end{" <> y <> "}" mathEnv :: PandocMonad m => Text -> LP m Text -mathEnv name = do +mathEnv name = withMathMode $ do optional blankline res <- manyTill anyTok (end_ name) return $ stripTrailingNewlines $ untokenize res @@ -89,20 +98,21 @@ inlineEnvironments = M.fromList [ ("displaymath", mathEnvWith id Nothing "displaymath") , ("math", math <$> mathEnv "math") - , ("equation", mathEnvWith id Nothing "equation") - , ("equation*", mathEnvWith id Nothing "equation*") - , ("gather", mathEnvWith id (Just "gathered") "gather") - , ("gather*", mathEnvWith id (Just "gathered") "gather*") - , ("multline", mathEnvWith id (Just "gathered") "multline") - , ("multline*", mathEnvWith id (Just "gathered") "multline*") - , ("eqnarray", mathEnvWith id (Just "aligned") "eqnarray") - , ("eqnarray*", mathEnvWith id (Just "aligned") "eqnarray*") - , ("align", mathEnvWith id (Just "aligned") "align") - , ("align*", mathEnvWith id (Just "aligned") "align*") - , ("alignat", mathEnvWith id (Just "aligned") "alignat") - , ("alignat*", mathEnvWith id (Just "aligned") "alignat*") - , ("flalign", mathEnvWith id (Just "aligned") "flalign") - , ("flalign*", mathEnvWith id (Just "aligned") "flalign*") + , ("equation", mathEnvWith id (Just "equation") "equation") + , ("equation*", mathEnvWith id (Just "equation*") "equation*") + , ("gather", mathEnvWith id (Just "gather") "gather") + , ("gather*", mathEnvWith id (Just "gather*") "gather*") + , ("multline", mathEnvWith id (Just "multline") "multline") + , ("multline*", mathEnvWith id (Just "multline*") "multline*") + , ("eqnarray", mathEnvWith id (Just "eqnarray") "eqnarray") + , ("eqnarray*", mathEnvWith id (Just "eqnarray*") "eqnarray*") + , ("align", mathEnvWith id (Just "align") "align") + , ("align*", mathEnvWith id (Just "align*") "align*") + , ("alignat", mathEnvWith id (Just "alignat") "alignat") + , ("alignat*", mathEnvWith id (Just "alignat*") "alignat*") + , ("flalign", mathEnvWith id (Just "flalign") "flalign") + , ("flalign*", mathEnvWith id (Just "flalign*") "flalign*") + -- the following are not yet handled by texmath, so we use substitutes: , ("dmath", mathEnvWith id Nothing "dmath") , ("dmath*", mathEnvWith id Nothing "dmath*") , ("dgroup", mathEnvWith id (Just "aligned") "dgroup") diff --git a/src/Text/Pandoc/Readers/LaTeX/Parsing.hs b/src/Text/Pandoc/Readers/LaTeX/Parsing.hs --- a/src/Text/Pandoc/Readers/LaTeX/Parsing.hs +++ b/src/Text/Pandoc/Readers/LaTeX/Parsing.hs @@ -157,6 +157,7 @@ , sLogMessages :: [LogMessage] , sIdentifiers :: Set.Set Text , sVerbatimMode :: Bool + , sMathMode :: Bool , sCaption :: Maybe Caption , sInListItem :: Bool , sInTableCell :: Bool @@ -186,6 +187,7 @@ , sLogMessages = [] , sIdentifiers = Set.empty , sVerbatimMode = False + , sMathMode = False , sCaption = Nothing , sInListItem = False , sInTableCell = False @@ -650,6 +652,9 @@ _ -> return ts' trySpecialMacro "iftrue" ts = handleIf True ts trySpecialMacro "iffalse" ts = handleIf False ts +trySpecialMacro "ifmmode" ts = do + mathMode <- sMathMode <$> getState + handleIf mathMode ts trySpecialMacro _ _ = mzero handleIf :: PandocMonad m => Bool -> [Tok] -> LP m [Tok] diff --git a/src/Text/Pandoc/Readers/Man.hs b/src/Text/Pandoc/Readers/Man.hs --- a/src/Text/Pandoc/Readers/Man.hs +++ b/src/Text/Pandoc/Readers/Man.hs @@ -219,7 +219,16 @@ (ControlLine _ args _) <- mmacro "TH" let adjustMeta = case args of - (x:y:z:_) -> setMeta "title" (linePartsToInlines x) . + (x:y:z:a:b:_) -> setMeta "title" (linePartsToInlines x) . + setMeta "section" (linePartsToInlines y) . + setMeta "date" (linePartsToInlines z) . + setMeta "footer" (linePartsToInlines a) . + setMeta "header" (linePartsToInlines b) + [x,y,z,a] -> setMeta "title" (linePartsToInlines x) . + setMeta "section" (linePartsToInlines y) . + setMeta "date" (linePartsToInlines z) . + setMeta "footer" (linePartsToInlines a) + [x,y,z] -> setMeta "title" (linePartsToInlines x) . setMeta "section" (linePartsToInlines y) . setMeta "date" (linePartsToInlines z) [x,y] -> setMeta "title" (linePartsToInlines x) . diff --git a/src/Text/Pandoc/Readers/Markdown.hs b/src/Text/Pandoc/Readers/Markdown.hs --- a/src/Text/Pandoc/Readers/Markdown.hs +++ b/src/Text/Pandoc/Readers/Markdown.hs @@ -845,7 +845,9 @@ return (num, style, delim)) listStart :: PandocMonad m => MarkdownParser m () -listStart = bulletListStart <|> Control.Monad.void (orderedListStart Nothing) +listStart = bulletListStart + <|> Control.Monad.void (orderedListStart Nothing) + <|> defListStart listLine :: PandocMonad m => Int -> MarkdownParser m Text listLine continuationIndent = try $ do @@ -967,67 +969,39 @@ -- definition lists -defListMarker :: PandocMonad m => MarkdownParser m () -defListMarker = do - sps <- nonindentSpaces +defListStart :: PandocMonad m => MarkdownParser m () +defListStart = do + nonindentSpaces char ':' <|> char '~' - tabStop <- getOption readerTabStop - let remaining = tabStop - (T.length sps + 1) - if remaining > 0 - then try (count remaining (char ' ')) <|> string "\t" <|> many1 spaceChar - else mzero - return () + gobbleSpaces 1 <|> () <$ lookAhead newline + try (gobbleAtMostSpaces 3 >> notFollowedBy spaceChar) <|> return () -definitionListItem :: PandocMonad m => Bool -> MarkdownParser m (F (Inlines, [Blocks])) -definitionListItem compact = try $ do +definitionListItem :: PandocMonad m => MarkdownParser m (F (Inlines, [Blocks])) +definitionListItem = try $ do rawLine' <- anyLine - raw <- many1 $ defRawBlock compact term <- parseFromString' (trimInlinesF <$> inlines) rawLine' - contents <- mapM (parseFromString' parseBlocks . (<> "\n")) raw + isTight <- (False <$ blanklines) <|> pure True + fourSpaceRule <- (True <$ guardEnabled Ext_four_space_rule) <|> pure False + contents <- many1 $ listItem fourSpaceRule defListStart optional blanklines - return $ liftM2 (,) term (sequence contents) + return $ liftM2 (,) + term + ((if isTight + then fmap (fmap (fmap paraToPlain)) + else id) (sequence contents)) -defRawBlock :: PandocMonad m => Bool -> MarkdownParser m Text -defRawBlock compact = try $ do - hasBlank <- option False $ blankline >> return True - defListMarker - firstline <- anyLineNewline - let dline = try - ( do notFollowedBy blankline - notFollowedByHtmlCloser - notFollowedByDivCloser - if compact -- laziness not compatible with compact - then () <$ indentSpaces - else (() <$ indentSpaces) - <|> notFollowedBy defListMarker - anyLine ) - rawlines <- many dline - cont <- fmap T.concat $ many $ try $ do - trailing <- option "" blanklines - ln <- indentSpaces >> notFollowedBy blankline >> anyLine - lns <- many dline - return $ trailing <> T.unlines (ln:lns) - return $ trimr (firstline <> T.unlines rawlines <> cont) <> - if hasBlank || not (T.null cont) then "\n\n" else "" +paraToPlain :: Block -> Block +paraToPlain (Para ils) = Plain ils +paraToPlain x = x definitionList :: PandocMonad m => MarkdownParser m (F Blocks) definitionList = try $ do + guardEnabled Ext_definition_lists lookAhead (anyLine >> optional (blankline >> notFollowedBy (Control.Monad.void table)) >> -- don't capture table caption as def list! - defListMarker) - compactDefinitionList <|> normalDefinitionList - -compactDefinitionList :: PandocMonad m => MarkdownParser m (F Blocks) -compactDefinitionList = do - guardEnabled Ext_compact_definition_lists - items <- fmap sequence $ many1 $ definitionListItem True - return $ B.definitionList <$> fmap compactifyDL items - -normalDefinitionList :: PandocMonad m => MarkdownParser m (F Blocks) -normalDefinitionList = do - guardEnabled Ext_definition_lists - items <- fmap sequence $ many1 $ definitionListItem False + defListStart) + items <- fmap sequence $ many1 definitionListItem return $ B.definitionList <$> items -- @@ -1083,8 +1057,12 @@ let alt = case "alt" `lookup` attribs of Just alt' -> B.text alt' _ -> capt - attribs' = filter ((/= "alt") . fst) attribs - figattr = (ident, mempty, mempty) + attribs' = filter ((/= "latex-placement") . fst) + (filter ((/= "alt") . fst) attribs) + figattribs = case lookup "latex-placement" attribs of + Just p -> [("latex-placement", p)] + _ -> mempty + figattr = (ident, mempty, figattribs) caption = B.simpleCaption $ B.plain capt figbody = B.plain $ B.imageWith ("", classes, attribs') url title alt in B.figureWith figattr caption figbody @@ -1230,7 +1208,7 @@ -- one (or zero) line of text. simpleTableHeader :: PandocMonad m => Bool -- ^ Headerless table - -> MarkdownParser m (F [Blocks], [Alignment], [Int]) + -> MarkdownParser m (F [[Blocks]], [Alignment], [Int]) simpleTableHeader headless = try $ do rawContent <- if headless then return "" @@ -1252,7 +1230,7 @@ heads <- fmap sequence $ mapM (parseFromString' (mconcat <$> many plain).trim) rawHeads' - return (heads, aligns, indices) + return (fmap (:[]) heads, aligns, indices) -- Returns an alignment type for a table, based on a list of strings -- (the rows of the column header) and a number (the length of the @@ -1346,7 +1324,7 @@ multilineTableHeader :: PandocMonad m => Bool -- ^ Headerless table - -> MarkdownParser m (F [Blocks], [Alignment], [Int]) + -> MarkdownParser m (F [[Blocks]], [Alignment], [Int]) multilineTableHeader headless = try $ do unless headless $ tableSep >> notFollowedBy blankline @@ -1375,7 +1353,7 @@ else map (T.unlines . map trim) rawHeadsList heads <- fmap sequence $ mapM (parseFromString' (mconcat <$> many plain).trim) rawHeads - return (heads, aligns, indices') + return (fmap (:[]) heads, aligns, indices') -- Parse a grid table: starts with row of '-' on top, then header -- (which may be grid), then the rows, @@ -1420,7 +1398,7 @@ (rows :: F [[Blocks]]) <- sequence <$> mapM (fmap sequence . mapM cellContents) lines'' return $ - toTableComponents' NormalizeHeader aligns widths <$> headCells <*> rows + toTableComponents' NormalizeHeader aligns widths <$> fmap (:[]) headCells <*> rows sepPipe :: PandocMonad m => MarkdownParser m () sepPipe = try $ do @@ -2253,7 +2231,7 @@ rest <- option (return []) $ try $ char ';' >> spnl >> citeList spnl char ']' - notFollowedBy $ oneOf "[(" + notFollowedBy $ oneOf "[({" return $ do suff' <- suff rest' <- rest diff --git a/src/Text/Pandoc/Readers/ODT/Arrows/State.hs b/src/Text/Pandoc/Readers/ODT/Arrows/State.hs --- a/src/Text/Pandoc/Readers/ODT/Arrows/State.hs +++ b/src/Text/Pandoc/Readers/ODT/Arrows/State.hs @@ -17,7 +17,21 @@ faster and easier to implement this way. -} -module Text.Pandoc.Readers.ODT.Arrows.State where +module Text.Pandoc.Readers.ODT.Arrows.State + ( ArrowState(..) + , withState + , modifyState + , ignoringState + , fromState + , extractFromState + , tryModifyState + , withSubStateF + , withSubStateF' + , foldS + , iterateS + , iterateSL + , iterateS' + ) where import Control.Arrow import qualified Control.Category as Cat diff --git a/src/Text/Pandoc/Readers/ODT/Arrows/Utils.hs b/src/Text/Pandoc/Readers/ODT/Arrows/Utils.hs --- a/src/Text/Pandoc/Readers/ODT/Arrows/Utils.hs +++ b/src/Text/Pandoc/Readers/ODT/Arrows/Utils.hs @@ -19,7 +19,40 @@ -} -- We export everything -module Text.Pandoc.Readers.ODT.Arrows.Utils where +module Text.Pandoc.Readers.ODT.Arrows.Utils + ( and2 + , and3 + , and4 + , and5 + , and6 + , liftA2 + , liftA3 + , liftA4 + , liftA5 + , liftA6 + , liftA + , duplicate + , (>>%) + , keepingTheValue + , (^|||) + , (|||^) + , (^|||^) + , (^&&&) + , (&&&^) + , choiceToMaybe + , maybeToChoice + , returnV + , FallibleArrow + , liftAsSuccess + , (>>?) + , (>>?^) + , (>>?^?) + , (^>>?) + , (>>?!) + , (>>?%) + , (>>?%?) + , ifFailedDo + ) where import Prelude hiding (Applicative(..)) import Control.Arrow diff --git a/src/Text/Pandoc/Readers/ODT/Base.hs b/src/Text/Pandoc/Readers/ODT/Base.hs --- a/src/Text/Pandoc/Readers/ODT/Base.hs +++ b/src/Text/Pandoc/Readers/ODT/Base.hs @@ -10,7 +10,11 @@ Core types of the odt reader. -} -module Text.Pandoc.Readers.ODT.Base where +module Text.Pandoc.Readers.ODT.Base + ( ODTConverterState + , XMLReader + , XMLReaderSafe + ) where import Text.Pandoc.Readers.ODT.Generic.XMLConverter import Text.Pandoc.Readers.ODT.Namespaces diff --git a/src/Text/Pandoc/Readers/ODT/ContentReader.hs b/src/Text/Pandoc/Readers/ODT/ContentReader.hs --- a/src/Text/Pandoc/Readers/ODT/ContentReader.hs +++ b/src/Text/Pandoc/Readers/ODT/ContentReader.hs @@ -795,15 +795,26 @@ read_table :: BlockMatcher read_table = matchingElement NsTable "table" $ liftA simpleTable' - $ matchChildContent' [ read_table_row - ] + $ (matchChildContent' [read_table_header]) &&& + (matchChildContent' [read_table_row]) --- | A simple table without a caption or headers --- | Infers the number of headers from rows -simpleTable' :: [[Blocks]] -> Blocks -simpleTable' [] = simpleTable [] [] -simpleTable' (x : rest) = simpleTable (fmap (const defaults) x) (x : rest) - where defaults = fromList [] +-- | A simple table without a caption. +simpleTable' :: ([[Blocks]], [[Blocks]]) -> Blocks +simpleTable' (headers, rows) = + table emptyCaption (replicate numcols defaults) th [tb] tf + where + defaults = (AlignDefault, ColWidthDefault) + numcols = maximum $ map length $ headers ++ rows + toRow = Row nullAttr . map simpleCell + th = TableHead nullAttr $ map toRow headers + tb = TableBody nullAttr 0 [] $ map toRow rows + tf = TableFoot nullAttr [] + +-- +read_table_header :: ElementMatcher [[Blocks]] +read_table_header = matchingElement NsTable "table-header-rows" + $ matchChildContent' [ read_table_row + ] -- read_table_row :: ElementMatcher [[Blocks]] diff --git a/src/Text/Pandoc/Readers/ODT/Generic/Fallible.hs b/src/Text/Pandoc/Readers/ODT/Generic/Fallible.hs --- a/src/Text/Pandoc/Readers/ODT/Generic/Fallible.hs +++ b/src/Text/Pandoc/Readers/ODT/Generic/Fallible.hs @@ -17,7 +17,21 @@ -} -- We export everything -module Text.Pandoc.Readers.ODT.Generic.Fallible where +module Text.Pandoc.Readers.ODT.Generic.Fallible + ( Failure + , Fallible + , maybeToEither + , eitherToMaybe + , recover + , failWith + , failEmpty + , succeedWith + , collapseEither + , chooseMax + , chooseMaxWith + , ChoiceVector(..) + , SuccessList(..) + ) where -- | Default for now. Will probably become a class at some point. type Failure = () diff --git a/src/Text/Pandoc/Readers/ODT/Generic/Namespaces.hs b/src/Text/Pandoc/Readers/ODT/Generic/Namespaces.hs --- a/src/Text/Pandoc/Readers/ODT/Generic/Namespaces.hs +++ b/src/Text/Pandoc/Readers/ODT/Generic/Namespaces.hs @@ -11,7 +11,11 @@ typesafe Haskell namespace identifiers and unsafe "real world" namespaces. -} -module Text.Pandoc.Readers.ODT.Generic.Namespaces where +module Text.Pandoc.Readers.ODT.Generic.Namespaces + ( NameSpaceIRI + , NameSpaceIRIs + , NameSpaceID(..) + ) where import qualified Data.Map as M import Data.Text (Text) diff --git a/src/Text/Pandoc/Readers/ODT/Generic/SetMap.hs b/src/Text/Pandoc/Readers/ODT/Generic/SetMap.hs --- a/src/Text/Pandoc/Readers/ODT/Generic/SetMap.hs +++ b/src/Text/Pandoc/Readers/ODT/Generic/SetMap.hs @@ -10,7 +10,13 @@ A map of values to sets of values. -} -module Text.Pandoc.Readers.ODT.Generic.SetMap where +module Text.Pandoc.Readers.ODT.Generic.SetMap + ( SetMap + , empty + , fromList + , insert + , union3 + ) where import qualified Data.Map as M import qualified Data.Set as S diff --git a/src/Text/Pandoc/Readers/Org/Blocks.hs b/src/Text/Pandoc/Readers/Org/Blocks.hs --- a/src/Text/Pandoc/Readers/Org/Blocks.hs +++ b/src/Text/Pandoc/Readers/Org/Blocks.hs @@ -115,7 +115,7 @@ stringyMetaAttribute :: Monad m => OrgParser m (Text, Text) stringyMetaAttribute = try $ do - metaLineStart + metaLineStart *> notFollowedBy (stringAnyCase "begin_") attrName <- T.toLower <$> many1TillChar nonspaceChar (char ':') skipSpaces attrValue <- anyLine <|> ("" <$ newline) @@ -184,9 +184,9 @@ case T.toLower blkType of "export" -> exportBlock "comment" -> rawBlockLines (const mempty) - "html" -> rawBlockLines (return . B.rawBlock (lowercase blkType)) - "latex" -> rawBlockLines (return . B.rawBlock (lowercase blkType)) - "ascii" -> rawBlockLines (return . B.rawBlock (lowercase blkType)) + "html" -> rawBlockLines (return . B.rawBlock (T.toLower blkType)) + "latex" -> rawBlockLines (return . B.rawBlock (T.toLower blkType)) + "ascii" -> rawBlockLines (return . B.rawBlock (T.toLower blkType)) "example" -> exampleBlock blockAttrs "quote" -> parseBlockLines (fmap B.blockQuote) "verse" -> verseBlock @@ -205,10 +205,11 @@ in fmap $ B.divWith (ident, classes ++ [blkType], kv) where blockHeaderStart :: Monad m => OrgParser m Text - blockHeaderStart = try $ skipSpaces *> stringAnyCase "#+begin_" *> orgArgWord - - lowercase :: Text -> Text - lowercase = T.toLower + blockHeaderStart = try $ do + skipSpaces + metaLineStart + stringAnyCase "begin_" + many1Char (satisfy (not . isSpace)) admonitionBlock :: PandocMonad m => Text -> BlockAttributes -> Text -> OrgParser m (F Blocks) @@ -427,10 +428,11 @@ return (argKey, paramValue) orgParamValue :: Monad m => OrgParser m Text -orgParamValue = try $ fmap T.pack $ +orgParamValue = try $ skipSpaces *> notFollowedBy orgArgKey - *> noneOf "\n\r" `many1Till` endOfValue + *> ((char '"' *> manyChar (noneOf "\n\r\"") <* char '"') <|> + noneOf "\n\r" `many1TillChar` endOfValue) <* skipSpaces where endOfValue = lookAhead $ try (skipSpaces <* oneOf "\n\r") diff --git a/src/Text/Pandoc/Readers/Org/Inlines.hs b/src/Text/Pandoc/Readers/Org/Inlines.hs --- a/src/Text/Pandoc/Readers/Org/Inlines.hs +++ b/src/Text/Pandoc/Readers/Org/Inlines.hs @@ -2,8 +2,8 @@ {-# LANGUAGE OverloadedStrings #-} {- | Module : Text.Pandoc.Readers.Org.Inlines - Copyright : Copyright (C) 2014-2024 Albert Krewinkel - License : GNU GPL, version 2 or above + Copyright : Copyright (C) 2014-2025 Albert Krewinkel + License : GPL-2.0-or-later Maintainer : Albert Krewinkel @@ -88,17 +88,18 @@ , inlineCodeBlock , str , endline + , subscript -- takes precedence over underlined text + , superscript , emphasizedText , code , math , displayMath , verbatim - , subscript - , superscript , inlineLaTeX , exportSnippet , macro - , smart + , smartQuotes + , specialStrings , symbol ] <* (guard =<< newlinesCountWithinLimits) "inline" @@ -199,10 +200,10 @@ citationSuffix d <> B.toList aff' }]) citeItems :: PandocMonad m => OrgParser m (F [Citation]) -citeItems = sequence <$> citeItem `sepBy1` (char ';') +citeItems = sequence <$> sepBy1' citeItem (char ';' <* void (many spaceChar)) citeItem :: PandocMonad m => OrgParser m (F Citation) -citeItem = do +citeItem = try $ do pref <- citePrefix itemKey <- orgCiteKey suff <- citeSuffix @@ -246,37 +247,36 @@ citeSuffix :: PandocMonad m => OrgParser m (F Inlines) citeSuffix = - rawAffix False >>= parseFromString parseSuffix - where - parseSuffix = do - hasSpace <- option False - (True <$ try (spaceChar >> skipSpaces >> lookAhead nonspaceChar)) - ils <- trimInlinesF . mconcat <$> many inline - return $ if hasSpace - then (B.space <>) <$> ils - else ils + rawAffix False >>= parseFromString (mconcat <$> many inline) citeStyle :: PandocMonad m => OrgParser m (CiteStyle, [CiteVariant]) -citeStyle = option (DefStyle, []) $ do - sty <- option DefStyle $ try $ char '/' *> orgCiteStyle +citeStyle = do + sty <- option NilStyle $ try $ char '/' *> orgCiteStyle variants <- option [] $ try $ char '/' *> orgCiteVariants return (sty, variants) orgCiteStyle :: PandocMonad m => OrgParser m CiteStyle -orgCiteStyle = choice $ map try - [ NoAuthorStyle <$ string "noauthor" - , NoAuthorStyle <$ string "na" - , LocatorsStyle <$ string "locators" - , LocatorsStyle <$ char 'l' - , NociteStyle <$ string "nocite" - , NociteStyle <$ char 'n' - , TextStyle <$ string "text" - , TextStyle <$ char 't' - ] +orgCiteStyle = try $ do + s <- many1 letter + case s of + "author" -> pure AuthorStyle + "a" -> pure AuthorStyle + "noauthor" -> pure NoAuthorStyle + "na" -> pure NoAuthorStyle + "nocite" -> pure NociteStyle + "n" -> pure NociteStyle + "text" -> pure TextStyle + "t" -> pure TextStyle + "note" -> pure NoteStyle + "ft" -> pure NoteStyle + "numeric" -> pure NumericStyle + "nb" -> pure NumericStyle + "nil" -> pure NilStyle + _ -> fail $ "Unknown org cite style " <> show s orgCiteVariants :: PandocMonad m => OrgParser m [CiteVariant] orgCiteVariants = - (fullnameVariant `sepBy1` (char '-')) <|> (many1 onecharVariant) + (sepBy1' fullnameVariant (char '-')) <|> (many1 onecharVariant) where fullnameVariant = choice $ map try [ Bare <$ string "bare" @@ -290,11 +290,14 @@ ] data CiteStyle = - NoAuthorStyle + AuthorStyle + | NoAuthorStyle | LocatorsStyle | NociteStyle | TextStyle - | DefStyle + | NoteStyle + | NumericStyle + | NilStyle deriving Show data CiteVariant = @@ -591,11 +594,25 @@ code :: PandocMonad m => OrgParser m (F Inlines) code = return . B.code <$> verbatimBetween '~' +-- | Returns 'True' if the parser position right after a string, and 'False' +-- otherwise. +isAfterString :: PandocMonad m => OrgParser m Bool +isAfterString = do + pos <- getPosition + st <- getState + pure $ getLastStrPos st == Just pos + +-- | Parses subscript markup. Subscripts must be preceded by a string. subscript :: PandocMonad m => OrgParser m (F Inlines) -subscript = fmap B.subscript <$> try (char '_' *> subOrSuperExpr) +subscript = do + guard =<< isAfterString + fmap B.subscript <$> try (char '_' *> subOrSuperExpr) +-- | Parses superscript markup. Superscript must be preceded by a string. superscript :: PandocMonad m => OrgParser m (F Inlines) -superscript = fmap B.superscript <$> try (char '^' *> subOrSuperExpr) +superscript = do + guard =<< isAfterString + fmap B.superscript <$> try (char '^' *> subOrSuperExpr) math :: PandocMonad m => OrgParser m (F Inlines) math = return . B.math <$> choice [ math1CharBetween '$' @@ -890,29 +907,27 @@ escapedComma = try $ char '\\' *> oneOf ",\\" eoa = string ")}}}" -smart :: PandocMonad m => OrgParser m (F Inlines) -smart = choice [doubleQuoted, singleQuoted, orgApostrophe, orgDash, orgEllipses] +smartQuotes :: PandocMonad m => OrgParser m (F Inlines) +smartQuotes = do + guard =<< getExportSetting exportSmartQuotes + choice [doubleQuoted, singleQuoted, orgApostrophe] where - orgDash = do - guardOrSmartEnabled =<< getExportSetting exportSpecialStrings - pure <$> dash <* updatePositions '-' - orgEllipses = do - guardOrSmartEnabled =<< getExportSetting exportSpecialStrings - pure <$> ellipses <* updatePositions '.' orgApostrophe = do - guardEnabled Ext_smart (char '\'' <|> char '\8217') <* updateLastPreCharPos <* updateLastForbiddenCharPos returnF (B.str "\x2019") -guardOrSmartEnabled :: PandocMonad m => Bool -> OrgParser m () -guardOrSmartEnabled b = do - smartExtension <- extensionEnabled Ext_smart <$> getOption readerExtensions - guard (b || smartExtension) +specialStrings :: PandocMonad m => OrgParser m (F Inlines) +specialStrings = do + guard =<< getExportSetting exportSpecialStrings + choice [orgDash, orgEllipses, shyHyphen] + where + shyHyphen = pure <$> (B.str "\173" <$ string "\\-") <* updatePositions '-' + orgDash = pure <$> dash <* updatePositions '-' + orgEllipses = pure <$> ellipses <* updatePositions '.' singleQuoted :: PandocMonad m => OrgParser m (F Inlines) singleQuoted = try $ do - guardOrSmartEnabled =<< getExportSetting exportSmartQuotes singleQuoteStart updatePositions '\'' withQuoteContext InSingleQuote $ @@ -924,7 +939,6 @@ -- in the same paragraph. doubleQuoted :: PandocMonad m => OrgParser m (F Inlines) doubleQuoted = try $ do - guardOrSmartEnabled =<< getExportSetting exportSmartQuotes doubleQuoteStart updatePositions '"' contents <- mconcat <$> many (try $ notFollowedBy doubleQuoteEnd >> inline) diff --git a/src/Text/Pandoc/Readers/Org/Meta.hs b/src/Text/Pandoc/Readers/Org/Meta.hs --- a/src/Text/Pandoc/Readers/Org/Meta.hs +++ b/src/Text/Pandoc/Readers/Org/Meta.hs @@ -252,7 +252,11 @@ where todoKeyword :: Monad m => OrgParser m Text - todoKeyword = many1Char nonspaceChar <* skipSpaces + todoKeyword = do + keyword <- many1Char nonspaceChar + let cleanKeyword = T.takeWhile (/= '(') keyword + skipSpaces + return cleanKeyword todoKeywords :: Monad m => OrgParser m [Text] todoKeywords = try $ diff --git a/src/Text/Pandoc/Readers/Org/ParserState.hs b/src/Text/Pandoc/Readers/Org/ParserState.hs --- a/src/Text/Pandoc/Readers/Org/ParserState.hs +++ b/src/Text/Pandoc/Readers/Org/ParserState.hs @@ -3,7 +3,7 @@ {-# LANGUAGE OverloadedStrings #-} {- | Module : Text.Pandoc.Readers.Org.ParserState - Copyright : Copyright (C) 2014-2024 Albert Krewinkel + Copyright : Copyright (C) 2014-2025 Albert Krewinkel License : GNU GPL, version 2 or above Maintainer : Albert Krewinkel @@ -49,7 +49,7 @@ import Text.Pandoc.Builder (Blocks) import Text.Pandoc.Definition (Meta (..), nullMeta) import Text.Pandoc.Logging -import Text.Pandoc.Options (ReaderOptions (..)) +import Text.Pandoc.Options (ReaderOptions (..), Extension(..), isEnabled) import Text.Pandoc.Parsing (Future, HasIdentifierList (..), HasIncludeFiles (..), HasLastStrPosition (..), HasLogMessages (..), HasMacros (..), @@ -193,7 +193,15 @@ optionsToParserState :: ReaderOptions -> OrgParserState optionsToParserState opts = - def { orgStateOptions = opts } + let exportSettings = defaultExportSettings + { exportSmartQuotes = isEnabled Ext_smart opts || + isEnabled Ext_smart_quotes opts + , exportSpecialStrings = isEnabled Ext_smart opts || + isEnabled Ext_special_strings opts + } + in def { orgStateOptions = opts + , orgStateExportSettings = exportSettings + } registerTodoSequence :: TodoSequence -> OrgParserState -> OrgParserState registerTodoSequence todoSeq st = diff --git a/src/Text/Pandoc/Readers/Org/Parsing.hs b/src/Text/Pandoc/Readers/Org/Parsing.hs --- a/src/Text/Pandoc/Readers/Org/Parsing.hs +++ b/src/Text/Pandoc/Readers/Org/Parsing.hs @@ -1,6 +1,6 @@ {- | Module : Text.Pandoc.Readers.Org.Parsing - Copyright : Copyright (C) 2014-2024 Albert Krewinkel + Copyright : Copyright (C) 2014-2025 Albert Krewinkel License : GNU GPL, version 2 or above Maintainer : Albert Krewinkel @@ -40,6 +40,7 @@ , many1Till , many1TillChar , notFollowedBy' + , sepBy1' , spaceChar , nonspaceChar , skipSpaces @@ -53,6 +54,7 @@ , guardEnabled , updateLastStrPos , notAfterString + , getLastStrPos , ParserState (..) , registerHeader , QuoteContext (..) diff --git a/src/Text/Pandoc/Readers/Pod.hs b/src/Text/Pandoc/Readers/Pod.hs --- a/src/Text/Pandoc/Readers/Pod.hs +++ b/src/Text/Pandoc/Readers/Pod.hs @@ -258,7 +258,11 @@ entity (T.stripPrefix "0" -> Just suf) | Just (n, "") <- oct (T.unpack suf) = lookupEntity $ "#" <> tshow n entity (TR.decimal @Integer -> Right (x, "")) = lookupEntity $ "#" <> tshow x - entity x = lookupEntity x + -- named entities in Commonmark.Entity de facto have to be looked up with + -- the semicolon at the end. perlpodspec says arguments to E<> must be + -- alphanumeric, so an argument that already has a trailing semicolon + -- is bogus anyway, so just paste the semicolon on unconditionally. + entity x = lookupEntity (x <> ";") -- god knows there must be a higher order way of writing this thing, where we -- have multiple different possible parser states within the link argument diff --git a/src/Text/Pandoc/Readers/RST.hs b/src/Text/Pandoc/Readers/RST.hs --- a/src/Text/Pandoc/Readers/RST.hs +++ b/src/Text/Pandoc/Readers/RST.hs @@ -1395,7 +1395,7 @@ -- Parse a table row and return a list of blocks (columns). simpleTableRow :: PandocMonad m => [Int] -> RSTParser m [Blocks] simpleTableRow indices = do - notFollowedBy' simpleTableFooter + notFollowedBy' (blanklines <|> simpleTableFooter) firstLine <- simpleTableRawLine indices conLines <- many $ simpleTableRawLineWithInitialEmptyCell indices let cols = map T.unlines . transpose $ firstLine : conLines ++ @@ -1409,12 +1409,12 @@ simpleTableHeader :: PandocMonad m => Bool -- ^ Headerless table - -> RSTParser m ([Blocks], [Alignment], [Int]) + -> RSTParser m ([[Blocks]], [Alignment], [Int]) simpleTableHeader headless = try $ do optional blanklines rawContent <- if headless - then return "" - else simpleTableSep '=' >> anyLine + then return [""] + else simpleTableSep '=' >> many1 (notFollowedBy (simpleDashedLines '=') >> anyLine) dashes <- if headless then simpleDashedLines '=' else simpleDashedLines '=' <|> simpleDashedLines '-' @@ -1424,8 +1424,8 @@ let aligns = replicate (length lines') AlignDefault let rawHeads = if headless then [] - else simpleTableSplitLine indices rawContent - heads <- mapM ( parseFromString' (mconcat <$> many plain) . trim) rawHeads + else map (simpleTableSplitLine indices) rawContent + heads <- mapM ( mapM $ parseFromString' (mconcat <$> many plain) . trim) rawHeads return (heads, aligns, indices) -- Parse a simple table. @@ -1495,7 +1495,7 @@ escapedChar :: Monad m => RSTParser m Inlines escapedChar = do c <- escaped anyChar - unless (canPrecedeOpener c) $ updateLastStrPos + unless (canPrecedeOpener c) updateLastStrPos return $ if c == ' ' || c == '\n' || c == '\r' -- '\ ' is null in RST then mempty @@ -1509,7 +1509,7 @@ symbol :: Monad m => RSTParser m Inlines symbol = do c <- oneOf specialChars - unless (canPrecedeOpener c) $ updateLastStrPos + unless (canPrecedeOpener c) updateLastStrPos return $ B.str $ T.singleton c -- parses inline code, between codeStart and codeEnd diff --git a/src/Text/Pandoc/Readers/TikiWiki.hs b/src/Text/Pandoc/Readers/TikiWiki.hs --- a/src/Text/Pandoc/Readers/TikiWiki.hs +++ b/src/Text/Pandoc/Readers/TikiWiki.hs @@ -24,8 +24,7 @@ import Data.Text (Text) import qualified Data.Text as T import qualified Text.Pandoc.Builder as B -import Text.Pandoc.Class.CommonState (CommonState (..)) -import Text.Pandoc.Class.PandocMonad (PandocMonad (..)) +import Text.Pandoc.Class.PandocMonad (PandocMonad (..), getVerbosity) import Text.Pandoc.Definition import Text.Pandoc.Logging (Verbosity (..)) import Text.Pandoc.Options @@ -71,7 +70,7 @@ block :: PandocMonad m => TikiWikiParser m B.Blocks block = do - verbosity <- getsCommonState stVerbosity + verbosity <- getVerbosity pos <- getPosition res <- mempty <$ skipMany1 blankline <|> blockElements diff --git a/src/Text/Pandoc/Readers/Typst.hs b/src/Text/Pandoc/Readers/Typst.hs --- a/src/Text/Pandoc/Readers/Typst.hs +++ b/src/Text/Pandoc/Readers/Typst.hs @@ -50,8 +50,9 @@ import Typst.Methods (formatNumber, applyPureFunction) import Typst.Types import qualified Data.Vector as V - --- import Debug.Trace +import System.FilePath (takeDirectory) +import qualified System.FilePath.Windows as Windows +import qualified System.FilePath.Posix as Posix -- | Read Typst from an input string and return a Pandoc document. readTypst :: (PandocMonad m, ToSources a) @@ -73,9 +74,10 @@ case res of Left e -> throwError $ PandocParseError $ tshow e Right content -> do - let labs = findLabels [content] + let content' = fixNesting content + let labs = findLabels [content'] runParserT pPandoc defaultPState{ sLabels = labs } - inputName [content] >>= + inputName [content'] >>= either (throwError . PandocParseError . T.pack . show) pure pBlockElt :: PandocMonad m => P m B.Blocks @@ -90,7 +92,7 @@ ignored ("unknown block element " <> tname <> " at " <> tshow pos) pure mempty - Just handler -> handler mbident fields + Just handler -> handler pos mbident fields _ -> pure mempty pInline :: PandocMonad m => P m B.Inlines @@ -120,15 +122,38 @@ ignored ("unknown inline element " <> tname <> " at " <> tshow pos) pure mempty - Just handler -> handler Nothing (M.mapKeys targetToKey fields) + Just handler -> handler pos Nothing (M.mapKeys targetToKey fields) else do case M.lookup name inlineHandlers of Nothing -> do ignored ("unknown inline element " <> tname <> " at " <> tshow pos) pure mempty - Just handler -> handler Nothing fields + Just handler -> handler pos Nothing fields +-- Pull block elements out of inline elements, e.g. +-- Elt "smallcaps" [ Elt "heading" [..] ] -> +-- Elt "heading" [ Elt "smallcaps" [..]]. See #11017. +fixNesting :: Content -> Content +fixNesting el@(Elt name pos fields) + | Just (VContent elts) <- M.lookup "body" fields + = let elts' = fmap fixNesting elts + fields' = M.insert "body" (VContent elts') fields + in if isInline el + then case getField "body" fields' of + Just ([el'@(Elt name' pos' fields'')] :: Seq Content) + | isBlock el' + , not (isInline el') + , "body" `M.member` fields'' + -> Elt name' pos' $ + M.insert "body" (VContent + (Seq.singleton + (Elt name pos fields''))) + fields' + _ -> Elt name pos fields' + else Elt name pos fields' +fixNesting x = x + pPandoc :: PandocMonad m => P m B.Pandoc pPandoc = do Elt "document" _ fields <- pTok isDocument @@ -208,31 +233,34 @@ blockKeys :: Set.Set Identifier blockKeys = Set.fromList $ M.keys (blockHandlers :: M.Map Identifier - (Maybe Text -> M.Map Identifier Val -> P PandocPure B.Blocks)) + (Maybe SourcePos -> Maybe Text -> + M.Map Identifier Val -> P PandocPure B.Blocks)) inlineKeys :: Set.Set Identifier inlineKeys = Set.fromList $ M.keys (inlineHandlers :: M.Map Identifier - (Maybe Text -> M.Map Identifier Val -> P PandocPure B.Inlines)) + (Maybe SourcePos -> Maybe Text -> + M.Map Identifier Val -> P PandocPure B.Inlines)) blockHandlers :: PandocMonad m => M.Map Identifier - (Maybe Text -> M.Map Identifier Val -> P m B.Blocks) + (Maybe SourcePos -> Maybe Text -> + M.Map Identifier Val -> P m B.Blocks) blockHandlers = M.fromList - [("text", \_ fields -> do + [("text", \_ _ fields -> do body <- getField "body" fields -- sometimes text elements include para breaks notFollowedBy $ void $ pWithContents pInlines body pWithContents pBlocks body) - ,("box", \_ fields -> do + ,("box", \_ _ fields -> do body <- getField "body" fields B.divWith ("", ["box"], []) <$> pWithContents pBlocks body) - ,("heading", \mbident fields -> do + ,("heading", \_ mbident fields -> do body <- getField "body" fields lev <- getField "level" fields <|> pure 1 B.headerWith (fromMaybe "" mbident,[],[]) lev <$> pWithContents pInlines body) - ,("quote", \_ fields -> do + ,("quote", \_ _ fields -> do getField "block" fields >>= guard body <- getField "body" fields >>= pWithContents pBlocks attribution' <- getField "attribution" fields @@ -241,11 +269,11 @@ else (\x -> B.para ("\x2014\xa0" <> x)) <$> (pWithContents pInlines attribution') pure $ B.blockQuote $ body <> attribution) - ,("list", \_ fields -> do + ,("list", \_ _ fields -> do children <- V.toList <$> getField "children" fields B.bulletList <$> mapM (pWithContents pBlocks) children) - ,("list.item", \_ fields -> getField "body" fields >>= pWithContents pBlocks) - ,("enum", \_ fields -> do + ,("list.item", \_ _ fields -> getField "body" fields >>= pWithContents pBlocks) + ,("enum", \_ _ fields -> do children <- V.toList <$> getField "children" fields mbstart <- getField "start" fields start <- case mbstart of @@ -274,8 +302,8 @@ _ -> (B.DefaultStyle, B.DefaultDelim) let listAttr = (start, sty, delim) B.orderedListWith listAttr <$> mapM (pWithContents pBlocks) children) - ,("enum.item", \_ fields -> getField "body" fields >>= pWithContents pBlocks) - ,("terms", \_ fields -> do + ,("enum.item", \_ _ fields -> getField "body" fields >>= pWithContents pBlocks) + ,("terms", \_ _ fields -> do children <- V.toList <$> getField "children" fields B.definitionList <$> mapM @@ -287,38 +315,38 @@ _ -> pure (mempty, []) ) children) - ,("terms.item", \_ fields -> getField "body" fields >>= pWithContents pBlocks) - ,("raw", \mbident fields -> do + ,("terms.item", \_ _ fields -> getField "body" fields >>= pWithContents pBlocks) + ,("raw", \_ mbident fields -> do txt <- T.filter (/= '\r') <$> getField "text" fields mblang <- getField "lang" fields let attr = (fromMaybe "" mbident, maybe [] (\l -> [l]) mblang, []) pure $ B.codeBlockWith attr txt) - ,("parbreak", \_ _ -> pure mempty) - ,("block", \mbident fields -> + ,("parbreak", \_ _ _ -> pure mempty) + ,("block", \_ mbident fields -> maybe id (\ident -> B.divWith (ident, [], [])) mbident <$> (getField "body" fields >>= pWithContents pBlocks)) - ,("place", \_ fields -> do + ,("place", \_ _ fields -> do ignored "parameters of place" getField "body" fields >>= pWithContents pBlocks) - ,("columns", \_ fields -> do + ,("columns", \_ _ fields -> do (cnt :: Integer) <- getField "count" fields B.divWith ("", ["columns-flow"], [("count", T.pack (show cnt))]) <$> (getField "body" fields >>= pWithContents pBlocks)) - ,("rect", \_ fields -> + ,("rect", \_ _ fields -> B.divWith ("", ["rect"], []) <$> (getField "body" fields >>= pWithContents pBlocks)) - ,("circle", \_ fields -> + ,("circle", \_ _ fields -> B.divWith ("", ["circle"], []) <$> (getField "body" fields >>= pWithContents pBlocks)) - ,("ellipse", \_ fields -> + ,("ellipse", \_ _ fields -> B.divWith ("", ["ellipse"], []) <$> (getField "body" fields >>= pWithContents pBlocks)) - ,("polygon", \_ fields -> + ,("polygon", \_ _ fields -> B.divWith ("", ["polygon"], []) <$> (getField "body" fields >>= pWithContents pBlocks)) - ,("square", \_ fields -> + ,("square", \_ _ fields -> B.divWith ("", ["square"], []) <$> (getField "body" fields >>= pWithContents pBlocks)) - ,("align", \_ fields -> do + ,("align", \_ _ fields -> do alignment <- getField "alignment" fields B.divWith ("", [], [("align", repr alignment)]) <$> (getField "body" fields >>= pWithContents pBlocks)) - ,("stack", \_ fields -> do + ,("stack", \_ _ fields -> do (dir :: Direction) <- getField "dir" fields `mplus` pure Ltr rawchildren <- getField "children" fields children <- @@ -333,9 +361,9 @@ B.divWith ("", [], [("stack", repr (VDirection dir))]) $ mconcat $ map (B.divWith ("", [], [])) children) - ,("grid", \mbident fields -> parseTable mbident fields) - ,("table", \mbident fields -> parseTable mbident fields) - ,("figure", \mbident fields -> do + ,("grid", \_ mbident fields -> parseTable mbident fields) + ,("table", \_ mbident fields -> parseTable mbident fields) + ,("figure", \_ mbident fields -> do body <- getField "body" fields >>= pWithContents pBlocks (mbCaption :: Maybe (Seq Content)) <- getField "caption" fields (caption :: B.Blocks) <- maybe mempty (pWithContents pBlocks) mbCaption @@ -345,13 +373,13 @@ (B.Table attr (B.Caption Nothing (B.toList caption)) colspecs thead tbodies tfoot) _ -> B.figureWith (fromMaybe "" mbident, [], []) (B.Caption Nothing (B.toList caption)) body) - ,("line", \_ fields -> + ,("line", \_ _ fields -> case ( M.lookup "start" fields >> M.lookup "end" fields >> M.lookup "angle" fields ) of Nothing -> pure B.horizontalRule _ -> pure mempty) - ,("numbering", \_ fields -> do + ,("numbering", \_ _ fields -> do numStyle <- getField "numbering" fields (nums :: V.Vector Integer) <- getField "numbers" fields let toText v = fromMaybe "" $ fromVal v @@ -364,16 +392,17 @@ Failure _ -> "?" _ -> "?" pure $ B.plain . B.text . mconcat . map toNum $ V.toList nums) - ,("footnote.entry", \_ fields -> + ,("footnote.entry", \_ _ fields -> getField "body" fields >>= pWithContents pBlocks) - ,("pad", \_ fields -> -- ignore paddingy + ,("pad", \_ _ fields -> -- ignore paddingy getField "body" fields >>= pWithContents pBlocks) ] inlineHandlers :: PandocMonad m => - M.Map Identifier (Maybe Text -> M.Map Identifier Val -> P m B.Inlines) + M.Map Identifier (Maybe SourcePos -> Maybe Text -> + M.Map Identifier Val -> P m B.Inlines) inlineHandlers = M.fromList - [("ref", \_ fields -> do + [("ref", \_ _ fields -> do VLabel target <- getField "target" fields supplement' <- getField "supplement" fields supplement <- case supplement' of @@ -384,17 +413,17 @@ pure $ B.text ("[" <> target <> "]") _ -> pure mempty pure $ B.linkWith ("", ["ref"], []) ("#" <> target) "" supplement) - ,("linebreak", \_ _ -> pure B.linebreak) - ,("text", \_ fields -> do + ,("linebreak", \_ _ _ -> pure B.linebreak) + ,("text", \_ _ fields -> do body <- getField "body" fields (mbweight :: Maybe Text) <- getField "weight" fields case mbweight of Just "bold" -> B.strong <$> pWithContents pInlines body _ -> pWithContents pInlines body) - ,("raw", \_ fields -> B.code . T.filter (/= '\r') <$> getField "text" fields) - ,("footnote", \_ fields -> + ,("raw", \_ _ fields -> B.code . T.filter (/= '\r') <$> getField "text" fields) + ,("footnote", \_ _ fields -> B.note <$> (getField "body" fields >>= pWithContents pBlocks)) - ,("cite", \_ fields -> do + ,("cite", \_ _ fields -> do VLabel key <- getField "key" fields (form :: Text) <- getField "form" fields <|> pure "normal" let citation = @@ -409,38 +438,38 @@ B.citationHash = 0 } pure $ B.cite [citation] (B.text $ "[" <> key <> "]")) - ,("lower", \_ fields -> do + ,("lower", \_ _ fields -> do body <- getField "text" fields walk (modString T.toLower) <$> pWithContents pInlines body) - ,("upper", \_ fields -> do + ,("upper", \_ _ fields -> do body <- getField "text" fields walk (modString T.toUpper) <$> pWithContents pInlines body) - ,("emph", \_ fields -> do + ,("emph", \_ _ fields -> do body <- getField "body" fields B.emph <$> pWithContents pInlines body) - ,("strong", \_ fields -> do + ,("strong", \_ _ fields -> do body <- getField "body" fields B.strong <$> pWithContents pInlines body) - ,("sub", \_ fields -> do + ,("sub", \_ _ fields -> do body <- getField "body" fields B.subscript <$> pWithContents pInlines body) - ,("super", \_ fields -> do + ,("super", \_ _ fields -> do body <- getField "body" fields B.superscript <$> pWithContents pInlines body) - ,("strike", \_ fields -> do + ,("strike", \_ _ fields -> do body <- getField "body" fields B.strikeout <$> pWithContents pInlines body) - ,("smallcaps", \_ fields -> do + ,("smallcaps", \_ _ fields -> do body <- getField "body" fields B.smallcaps <$> pWithContents pInlines body) - ,("underline", \_ fields -> do + ,("underline", \_ _ fields -> do body <- getField "body" fields B.underline <$> pWithContents pInlines body) - ,("quote", \_ fields -> do + ,("quote", \_ _ fields -> do (getField "block" fields <|> pure False) >>= guard . not body <- getInlineBody fields >>= pWithContents pInlines pure $ B.doubleQuoted $ B.trimInlines body) - ,("link", \_ fields -> do + ,("link", \_ _ fields -> do dest <- getField "dest" fields src <- case dest of VString t -> pure t @@ -448,7 +477,7 @@ VDict _ -> do ignored "link to location, linking to #" pure "#" - _ -> fail $ "Expected string or label for dest" + _ -> fail "Expected string or label for dest" body <- getField "body" fields description <- if null body @@ -465,9 +494,15 @@ pWithContents (B.fromList . blocksToInlines . B.toList <$> pBlocks) body pure $ B.link src "" description) - ,("image", \_ fields -> do + ,("image", \mbpos _ fields -> do path <- getField "source" fields <|> getField "path" fields alt <- (B.text <$> getField "alt" fields) `mplus` pure mempty + let basedir = maybe "." (takeDirectory . sourceName) mbpos + let isAbsolutePath p = Posix.isAbsolute p || Windows.isAbsolute p + let path' = T.pack $ + if isAbsolutePath path || basedir == "." + then path + else basedir Posix. path (mbwidth :: Maybe Text) <- fmap (renderLength False) <$> getField "width" fields (mbheight :: Maybe Text) <- @@ -478,11 +513,11 @@ maybe [] (\x -> [("width", x)]) mbwidth ++ maybe [] (\x -> [("height", x)]) mbheight ) - pure $ B.imageWith attr path "" alt) - ,("box", \_ fields -> do + pure $ B.imageWith attr path' "" alt) + ,("box", \_ _ fields -> do body <- getField "body" fields B.spanWith ("", ["box"], []) <$> pWithContents pInlines body) - ,("h", \_ fields -> do + ,("h", \_ _ fields -> do amount <- getField "amount" fields `mplus` pure (LExact 1 LEm) let em = case amount of LExact x LEm -> toRational x @@ -490,21 +525,21 @@ LExact x LPt -> toRational x / 12 _ -> 1 / 3 -- guess! pure $ B.text $ getSpaceChars em) - ,("place", \_ fields -> do + ,("place", \_ _ fields -> do ignored "parameters of place" getField "body" fields >>= pWithContents pInlines) - ,("align", \_ fields -> do + ,("align", \_ _ fields -> do alignment <- getField "alignment" fields B.spanWith ("", [], [("align", repr alignment)]) <$> (getField "body" fields >>= pWithContents pInlines)) - ,("sys.version", \_ _ -> pure $ B.text "typst-hs") - ,("math.equation", \_ fields -> do + ,("sys.version", \_ _ _ -> pure $ B.text "typst-hs") + ,("math.equation", \_ _ fields -> do body <- getField "body" fields display <- getField "block" fields (if display then B.displayMath else B.math) . writeTeX <$> pMathMany body) - ,("pad", \_ fields -> -- ignore paddingy + ,("pad", \_ _ fields -> -- ignore paddingy getField "body" fields >>= pWithContents pInlines) - ,("block", \mbident fields -> + ,("block", \_ mbident fields -> maybe id (\ident -> B.spanWith (ident, [], [])) mbident <$> (getField "body" fields >>= pWithContents pInlines)) ] diff --git a/src/Text/Pandoc/Readers/XML.hs b/src/Text/Pandoc/Readers/XML.hs new file mode 100644 --- /dev/null +++ b/src/Text/Pandoc/Readers/XML.hs @@ -0,0 +1,540 @@ +{-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE ScopedTypeVariables #-} + +-- | +-- Module : Text.Pandoc.Readers.XML +-- Copyright : Copyright (C) 2025- Massimiliano Farinella and John MacFarlane +-- License : GNU GPL, version 2 or above +-- +-- Maintainer : Massimiliano Farinella +-- Stability : WIP +-- Portability : portable +-- +-- Conversion of (Pandoc specific) xml to 'Pandoc' document. +module Text.Pandoc.Readers.XML (readXML) where + +import Control.Monad (msum) +import Control.Monad.Except (throwError) +import Control.Monad.State.Strict (StateT (runStateT), modify) +import Data.Char (isSpace) +import Data.Default (Default (..)) +import qualified Data.List as L +import qualified Data.Map as M +import Data.Maybe (catMaybes, fromMaybe, mapMaybe) +import qualified Data.Set as S (Set, fromList, member) +import Data.Text (Text) +import qualified Data.Text as T +import Data.Text.Lazy (fromStrict) +import Data.Version (Version, makeVersion) +import Text.Pandoc.Builder +import Text.Pandoc.Class.PandocMonad +import Text.Pandoc.Error (PandocError (..)) +import Text.Pandoc.Options +import Text.Pandoc.Parsing (ToSources, toSources) +import Text.Pandoc.Sources (sourcesToText) +import Text.Pandoc.Version (pandocVersion) +import Text.Pandoc.XML (lookupEntity) +import Text.Pandoc.XML.Light +import Text.Pandoc.XMLFormat +import Text.Read (readMaybe) + +-- TODO: use xmlPath state to give better context when an error occurs + +type XMLReader m = StateT XMLReaderState m + +data XMLReaderState = XMLReaderState + { xmlApiVersion :: Version, + xmlMeta :: Meta, + xmlContent :: [Content], + xmlPath :: [Text] + } + deriving (Show) + +instance Default XMLReaderState where + def = + XMLReaderState + { xmlApiVersion = pandocVersion, + xmlMeta = mempty, + xmlContent = [], + xmlPath = ["root"] + } + +readXML :: (PandocMonad m, ToSources a) => ReaderOptions -> a -> m Pandoc +readXML _ inp = do + let sources = toSources inp + tree <- + either (throwError . PandocXMLError "") return $ + parseXMLContents (fromStrict . sourcesToText $ sources) + (bs, st') <- flip runStateT (def {xmlContent = tree}) $ mapM parseBlock tree + let blockList = toList $ concatMany bs + return $ Pandoc (xmlMeta st') blockList + +concatMany :: [Many a] -> Many a +concatMany = Many . mconcat . map unMany + +parseBlocks :: (PandocMonad m) => [Content] -> XMLReader m Blocks +parseBlocks contents = concatMany <$> mapM parseBlock contents + +getBlocks :: (PandocMonad m) => Element -> XMLReader m Blocks +getBlocks e = parseBlocks (elContent e) + +elementName :: Element -> Text +elementName e = qName $ elName e + +attrValue :: Text -> Element -> Text +attrValue attr = + fromMaybe "" . maybeAttrValue attr + +maybeAttrValue :: Text -> Element -> Maybe Text +maybeAttrValue attr elt = + lookupAttrBy (\x -> qName x == attr) (elAttribs elt) + +parseBlock :: (PandocMonad m) => Content -> XMLReader m Blocks +parseBlock (Text (CData CDataRaw _ _)) = return mempty -- DOCTYPE +parseBlock (Text (CData _ s _)) = + if T.all isSpace s + then return mempty + else do + throwError $ PandocXMLError "" "non-space characters out of inline context" +parseBlock (CRef x) = do + throwError $ PandocXMLError "" ("reference \"" <> x <> "\" out of inline context") +parseBlock (Elem e) = do + let name = elementName e + in case (name) of + "Pandoc" -> parsePandoc + "?xml" -> return mempty + "blocks" -> getBlocks e + "meta" -> + let entry_els = childrenNamed tgNameMetaMapEntry e + in do + entries <- catMaybes <$> mapM parseMetaMapEntry entry_els + mapM_ (uncurry addMeta) entries + return mempty + "Para" -> para <$> getInlines (elContent e) + "Plain" -> do + ils <- getInlines (elContent e) + return $ singleton . Plain . toList $ ils + "Header" -> (headerWith attr level) <$> getInlines (elContent e) + where + level = textToInt (attrValue atNameLevel e) 1 + attr = filterAttrAttributes [atNameLevel] $ attrFromElement e + "HorizontalRule" -> return horizontalRule + "BlockQuote" -> do + contents <- getBlocks e + return $ blockQuote contents + "Div" -> do + contents <- getBlocks e + return $ divWith (attrFromElement e) contents + "BulletList" -> do + items <- getListItems e + return $ bulletList items + "OrderedList" -> do + items <- getListItems e + return $ orderedListWith (getListAttributes e) items + "DefinitionList" -> do + let items_contents = getContentsOfElements (isElementNamed tgNameDefListItem) (elContent e) + items <- mapM parseDefinitionListItem items_contents + return $ definitionList items + "Figure" -> do + let attr = attrFromElement e + (maybe_caption_el, contents) = partitionFirstChildNamed "Caption" $ elContent e + figure_caption <- case (maybe_caption_el) of + Just (caption_el) -> parseCaption $ elContent caption_el + Nothing -> pure emptyCaption + blocks <- parseBlocks contents + return $ figureWith attr figure_caption blocks + "CodeBlock" -> do + let attr = attrFromElement e + return $ codeBlockWith attr $ strContentRecursive e + "RawBlock" -> do + let format = (attrValue atNameFormat e) + return $ rawBlock format $ strContentRecursive e + "LineBlock" -> do + lins <- mapM getInlines (contentsOfChildren tgNameLineItem (elContent e)) + return $ lineBlock lins + "Table" -> do + -- TODO: check unexpected items + let attr = attrFromElement e + (maybe_caption_el, after_caption) = partitionFirstChildNamed "Caption" $ elContent e + children = elementsWithNames (S.fromList [tgNameColspecs, "TableHead", "TableBody", "TableFoot"]) after_caption + is_element tag el = tag == elementName el + colspecs <- getColspecs $ L.find (is_element tgNameColspecs) children + tbs <- getTableBodies $ filter (is_element "TableBody") children + th <- getTableHead $ L.find (is_element "TableHead") children + tf <- getTableFoot $ L.find (is_element "TableFoot") children + capt <- parseMaybeCaptionElement maybe_caption_el + case colspecs of + Nothing -> return mempty + Just cs -> return $ fromList [Table attr capt cs th tbs tf] + _ -> do + throwError $ PandocXMLError "" ("unexpected element \"" <> name <> "\" in blocks context") + where + parsePandoc = do + let version = maybeAttrValue atNameApiVersion e + apiversion = case (version) of + Just (v) -> makeVersion $ map (read . T.unpack) $ T.splitOn "," v + Nothing -> pandocVersion + in modify $ \st -> st {xmlApiVersion = apiversion} + getBlocks e + +getListItems :: (PandocMonad m) => Element -> XMLReader m [Blocks] +getListItems e = + let items_els = childrenNamed tgNameListItem e + in do + mapM getBlocks items_els + +getContentsOfElements :: (Content -> Bool) -> [Content] -> [[Content]] +getContentsOfElements filter_element contents = mapMaybe element_contents $ filter filter_element contents + where + element_contents :: Content -> Maybe [Content] + element_contents c = case (c) of + Elem e -> Just (elContent e) + _ -> Nothing + +strContentRecursive :: Element -> Text +strContentRecursive = + strContent + . (\e' -> e' {elContent = map elementToStr $ elContent e'}) + +elementToStr :: Content -> Content +elementToStr (Elem e') = Text $ CData CDataText (strContentRecursive e') Nothing +elementToStr x = x + +textToInt :: Text -> Int -> Int +textToInt t deflt = + let safe_to_int :: Text -> Maybe Int + safe_to_int s = readMaybe $ T.unpack s + in case (safe_to_int t) of + Nothing -> deflt + Just (n) -> n + +parseInline :: (PandocMonad m) => Content -> XMLReader m Inlines +parseInline (Text (CData _ s _)) = + return $ text s +parseInline (CRef ref) = + return $ + maybe (text $ T.toUpper ref) text $ + lookupEntity ref +parseInline (Elem e) = + let name = elementName e + in case (name) of + "Space" -> + let count = textToInt (attrValue atNameSpaceCount e) 1 + in return $ fromList $ replicate count Space + "Str" -> return $ fromList [Str $ attrValue atNameStrContent e] + "Emph" -> innerInlines emph + "Strong" -> innerInlines strong + "Strikeout" -> innerInlines strikeout + "Subscript" -> innerInlines subscript + "Superscript" -> innerInlines superscript + "Underline" -> innerInlines underline + "SoftBreak" -> return softbreak + "LineBreak" -> return linebreak + "SmallCaps" -> innerInlines smallcaps + "Quoted" -> case (attrValue atNameQuoteType e) of + "SingleQuote" -> innerInlines singleQuoted + _ -> innerInlines doubleQuoted + "Math" -> case (attrValue atNameMathType e) of + "DisplayMath" -> pure $ displayMath $ strContentRecursive e + _ -> pure $ math $ strContentRecursive e + "Span" -> innerInlines $ spanWith (attrFromElement e) + "Code" -> do + let attr = attrFromElement e + return $ codeWith attr $ strContentRecursive e + "Link" -> innerInlines $ linkWith attr url title + where + url = attrValue atNameLinkUrl e + title = attrValue atNameTitle e + attr = filterAttrAttributes [atNameLinkUrl, atNameTitle] $ attrFromElement e + "Image" -> innerInlines $ imageWith attr url title + where + url = attrValue atNameImageUrl e + title = attrValue atNameTitle e + attr = filterAttrAttributes [atNameImageUrl, atNameTitle] $ attrFromElement e + "RawInline" -> do + let format = (attrValue atNameFormat e) + return $ rawInline format $ strContentRecursive e + "Note" -> do + contents <- getBlocks e + return $ note contents + "Cite" -> + let (maybe_citations_el, contents) = partitionFirstChildNamed tgNameCitations $ elContent e + in case (maybe_citations_el) of + Just citations_el -> do + citations <- parseCitations $ elContent citations_el + (innerInlines' contents) $ cite citations + Nothing -> getInlines contents + _ -> do + throwError $ PandocXMLError "" ("unexpected element \"" <> name <> "\" in inline context") + where + innerInlines' contents f = + f . concatMany + <$> mapM parseInline contents + innerInlines f = innerInlines' (elContent e) f + +getInlines :: (PandocMonad m) => [Content] -> XMLReader m Inlines +getInlines contents = concatMany <$> mapM parseInline contents + +getListAttributes :: Element -> ListAttributes +getListAttributes e = (start, style, delim) + where + start = textToInt (attrValue atNameStart e) 1 + style = case (attrValue atNameNumberStyle e) of + "Example" -> Example + "Decimal" -> Decimal + "LowerRoman" -> LowerRoman + "UpperRoman" -> UpperRoman + "LowerAlpha" -> LowerAlpha + "UpperAlpha" -> UpperAlpha + _ -> DefaultStyle + delim = case (attrValue atNameNumberDelim e) of + "Period" -> Period + "OneParen" -> OneParen + "TwoParens" -> TwoParens + _ -> DefaultDelim + +contentsOfChildren :: Text -> [Content] -> [[Content]] +contentsOfChildren tag contents = mapMaybe childrenElementWithTag contents + where + childrenElementWithTag :: Content -> Maybe [Content] + childrenElementWithTag c = case (c) of + (Elem e) -> if tag == elementName e then Just (elContent e) else Nothing + _ -> Nothing + +alignmentFromText :: Text -> Alignment +alignmentFromText t = case t of + "AlignLeft" -> AlignLeft + "AlignRight" -> AlignRight + "AlignCenter" -> AlignCenter + _ -> AlignDefault + +getColWidth :: Text -> ColWidth +getColWidth txt = case reads (T.unpack txt) of + [(value, "")] -> if value == 0.0 then ColWidthDefault else ColWidth value + _ -> ColWidthDefault + +getColspecs :: (PandocMonad m) => Maybe Element -> XMLReader m (Maybe [ColSpec]) +getColspecs Nothing = pure Nothing +getColspecs (Just cs) = do + return $ Just $ map elementToColSpec (childrenNamed "ColSpec" cs) + where + elementToColSpec e = (alignmentFromText $ attrValue atNameAlignment e, getColWidth $ attrValue atNameColWidth e) + +getTableBody :: (PandocMonad m) => Element -> XMLReader m (Maybe TableBody) +getTableBody body_el = do + let attr = filterAttrAttributes [atNameRowHeadColumns] $ attrFromElement body_el + bh = childrenNamed tgNameBodyHeader body_el + bb = childrenNamed tgNameBodyBody body_el + headcols = textToInt (attrValue atNameRowHeadColumns body_el) 0 + hrows <- mconcat <$> mapM getRows bh + brows <- mconcat <$> mapM getRows bb + return $ Just $ TableBody attr (RowHeadColumns headcols) hrows brows + +getTableBodies :: (PandocMonad m) => [Element] -> XMLReader m [TableBody] +getTableBodies body_elements = do + catMaybes <$> mapM getTableBody body_elements + +getTableHead :: (PandocMonad m) => Maybe Element -> XMLReader m TableHead +getTableHead maybe_e = case maybe_e of + Just e -> do + let attr = attrFromElement e + rows <- getRows e + return $ TableHead attr rows + Nothing -> return $ TableHead nullAttr [] + +getTableFoot :: (PandocMonad m) => Maybe Element -> XMLReader m TableFoot +getTableFoot maybe_e = case maybe_e of + Just e -> do + let attr = attrFromElement e + rows <- getRows e + return $ TableFoot attr rows + Nothing -> return $ TableFoot nullAttr [] + +getCell :: (PandocMonad m) => Element -> XMLReader m Cell +getCell c = do + let alignment = alignmentFromText $ attrValue atNameAlignment c + rowspan = RowSpan $ textToInt (attrValue atNameRowspan c) 1 + colspan = ColSpan $ textToInt (attrValue atNameColspan c) 1 + attr = filterAttrAttributes [atNameAlignment, atNameRowspan, atNameColspan] $ attrFromElement c + blocks <- getBlocks c + return $ Cell attr alignment rowspan colspan (toList blocks) + +getRows :: (PandocMonad m) => Element -> XMLReader m [Row] +getRows e = mapM getRow $ childrenNamed "Row" e + where + getRow r = do + cells <- mapM getCell (childrenNamed "Cell" r) + return $ Row (attrFromElement r) cells + +parseCitations :: (PandocMonad m) => [Content] -> XMLReader m [Citation] +parseCitations contents = do + maybecitations <- mapM getCitation contents + return $ catMaybes maybecitations + where + getCitation :: (PandocMonad m) => Content -> XMLReader m (Maybe Citation) + getCitation content = case (content) of + (Elem e) -> + if qName (elName e) == "Citation" + then do + p <- inlinesOfChildrenNamed tgNameCitationPrefix e + s <- inlinesOfChildrenNamed tgNameCitationSuffix e + return $ + Just + ( Citation + { citationId = attrValue "id" e, + citationPrefix = toList p, + citationSuffix = toList s, + citationMode = case (attrValue atNameCitationMode e) of + "AuthorInText" -> AuthorInText + "SuppressAuthor" -> SuppressAuthor + _ -> NormalCitation, + citationNoteNum = textToInt (attrValue atNameCitationNoteNum e) 0, + citationHash = textToInt (attrValue atNameCitationHash e) 0 + } + ) + else do + return Nothing + _ -> do + return Nothing + where + inlinesOfChildrenNamed tag e = getInlines $ concatMap (\e' -> elContent e') (childrenNamed tag e) + +parseMaybeCaptionElement :: (PandocMonad m) => Maybe Element -> XMLReader m Caption +parseMaybeCaptionElement Nothing = pure emptyCaption +parseMaybeCaptionElement (Just e) = parseCaption $ elContent e + +parseCaption :: (PandocMonad m) => [Content] -> XMLReader m Caption +parseCaption contents = + let (maybe_shortcaption_el, caption_contents) = partitionFirstChildNamed tgNameShortCaption contents + in do + blocks <- parseBlocks caption_contents + case (maybe_shortcaption_el) of + Just shortcaption_el -> do + short_caption <- getInlines (elContent shortcaption_el) + return $ caption (Just $ toList short_caption) blocks + Nothing -> return $ caption Nothing blocks + +parseDefinitionListItem :: (PandocMonad m) => [Content] -> XMLReader m (Inlines, [Blocks]) +parseDefinitionListItem contents = do + let term_contents = getContentsOfElements (isElementNamed tgNameDefListTerm) contents + defs_elements = elementContents $ filter (isElementNamed tgNameDefListDef) contents + term_inlines <- getInlines (concat term_contents) + defs <- mapM getBlocks defs_elements + return (term_inlines, defs) + +elementContents :: [Content] -> [Element] +elementContents contents = mapMaybe toElement contents + where + toElement :: Content -> Maybe Element + toElement (Elem e) = Just e + toElement _ = Nothing + +isElementNamed :: Text -> Content -> Bool +isElementNamed t c = case (c) of + Elem e -> t == elementName e + _ -> False + +childrenNamed :: Text -> Element -> [Element] +childrenNamed tag e = elementContents $ filter (isElementNamed tag) (elContent e) + +elementsWithNames :: S.Set Text -> [Content] -> [Element] +elementsWithNames tags contents = mapMaybe isElementWithNameInSet contents + where + isElementWithNameInSet c = case (c) of + Elem el -> + if (elementName el) `S.member` tags + then Just el + else Nothing + _ -> Nothing + +partitionFirstChildNamed :: Text -> [Content] -> (Maybe Element, [Content]) +partitionFirstChildNamed tag contents = case (contents) of + (Text (CData _ s _) : rest) -> + if T.all isSpace s + then partitionFirstChildNamed tag rest + else (Nothing, contents) + (Elem e : rest) -> + if tag == elementName e + then (Just e, rest) + else (Nothing, contents) + _ -> (Nothing, contents) + +type PandocAttr = (Text, [Text], [(Text, Text)]) + +filterAttributes :: S.Set Text -> [(Text, Text)] -> [(Text, Text)] +filterAttributes to_be_removed a = filter keep_attr a + where + keep_attr (k, _) = not (k `S.member` to_be_removed) + +filterAttrAttributes :: [Text] -> PandocAttr -> PandocAttr +filterAttrAttributes to_be_removed (idn, classes, a) = (idn, classes, filtered) + where + filtered = filterAttributes (S.fromList to_be_removed) a + +attrFromElement :: Element -> PandocAttr +attrFromElement e = filterAttrAttributes ["id", "class"] (idn, classes, attributes) + where + idn = attrValue "id" e + classes = T.words $ attrValue "class" e + attributes = map (\a -> (qName $ attrKey a, attrVal a)) $ elAttribs e + +addMeta :: (PandocMonad m) => (ToMetaValue a) => Text -> a -> XMLReader m () +addMeta field val = modify (setMeta field val) + +instance HasMeta XMLReaderState where + setMeta field v s = s {xmlMeta = setMeta field v (xmlMeta s)} + + deleteMeta field s = s {xmlMeta = deleteMeta field (xmlMeta s)} + +parseMetaMapEntry :: (PandocMonad m) => Element -> XMLReader m (Maybe (Text, MetaValue)) +parseMetaMapEntry e = + let key = attrValue atNameMetaMapEntryKey e + in case (key) of + "" -> pure Nothing + k -> do + maybe_value <- parseMetaMapEntryContents $ elContent e + case (maybe_value) of + Nothing -> return Nothing + Just v -> return $ Just (k, v) + +parseMetaMapEntryContents :: (PandocMonad m) => [Content] -> XMLReader m (Maybe MetaValue) +parseMetaMapEntryContents cs = msum <$> mapM parseMeta cs + +parseMeta :: (PandocMonad m) => Content -> XMLReader m (Maybe MetaValue) +parseMeta (Text (CData CDataRaw _ _)) = return Nothing +parseMeta (Text (CData _ s _)) = + if T.all isSpace s + then return Nothing + else do + throwError $ PandocXMLError "" "non-space characters out of inline context in metadata" +parseMeta (CRef x) = + throwError $ PandocXMLError "" ("reference \"" <> x <> "\" out of inline context") +parseMeta (Elem e) = do + let name = elementName e + in case (name) of + "MetaBool" -> case (attrValue atNameMetaBoolValue e) of + "true" -> return $ Just $ MetaBool True + _ -> return $ Just $ MetaBool False + "MetaString" -> pure Nothing + "MetaInlines" -> do + inlines <- getInlines (elContent e) + return $ Just $ MetaInlines $ toList inlines + "MetaBlocks" -> do + blocks <- getBlocks e + return $ Just $ MetaBlocks $ toList blocks + "MetaList" -> do + maybe_items <- mapM parseMeta $ elContent e + let items = catMaybes maybe_items + in -- TODO: report empty MetaList? + return $ Just $ MetaList items + "MetaMap" -> + let entry_els = childrenNamed tgNameMetaMapEntry e + in do + entries <- catMaybes <$> mapM parseMetaMapEntry entry_els + if null entries + then + -- TODO: report empty MetaMap + return Nothing + else return $ Just $ MetaMap $ M.fromList entries + _ -> do + throwError $ PandocXMLError "" ("unexpected element \"" <> name <> "\" in metadata") diff --git a/src/Text/Pandoc/SelfContained.hs b/src/Text/Pandoc/SelfContained.hs --- a/src/Text/Pandoc/SelfContained.hs +++ b/src/Text/Pandoc/SelfContained.hs @@ -43,7 +43,6 @@ import Data.Maybe (isNothing) import qualified Data.Map as M import Control.Monad.State --- import Debug.Trace isOk :: Char -> Bool isOk c = isAscii c && isAlphaNum c @@ -420,17 +419,22 @@ return res else return raw' return $ Fetched (mime, result) - handler e = case e of - PandocResourceNotFound r -> do - report $ CouldNotFetchResource r "" - return $ CouldNotFetch e - PandocHttpError u er -> do - report $ CouldNotFetchResource u (tshow er) - return $ CouldNotFetch e - _ -> throwError e - - - + handler e + -- If fetch failed and we have a fragment and/or query, + -- try the fetch again without these, since the resource + -- may be local (see #1477, #11021) + | T.any (\c -> c == '?' || c == '#') src && not (isURI src) + = getData mimetype (removeQueryAndFragment src) + | otherwise + = case e of + PandocResourceNotFound r -> do + report $ CouldNotFetchResource r "" + return $ CouldNotFetch e + PandocHttpError u er -> do + report $ CouldNotFetchResource u (tshow er) + return $ CouldNotFetch e + _ -> throwError e + removeQueryAndFragment = T.takeWhile (\c -> c /= '#' && c /= '?') -- | Convert HTML into self-contained HTML, incorporating images, -- scripts, and CSS using data: URIs. diff --git a/src/Text/Pandoc/Templates.hs b/src/Text/Pandoc/Templates.hs --- a/src/Text/Pandoc/Templates.hs +++ b/src/Text/Pandoc/Templates.hs @@ -102,6 +102,7 @@ "native" -> return "" "csljson" -> return "" "json" -> return "" + "xml" -> return "" "fb2" -> return "" "pptx" -> return "" "ipynb" -> return "" diff --git a/src/Text/Pandoc/Translations.hs b/src/Text/Pandoc/Translations.hs --- a/src/Text/Pandoc/Translations.hs +++ b/src/Text/Pandoc/Translations.hs @@ -20,7 +20,8 @@ ) where import Text.Pandoc.Translations.Types -import Text.Pandoc.Class (PandocMonad(..), CommonState(..), toTextM, report) +import Text.Pandoc.Class (PandocMonad(..), toTextM, report) +import Text.Pandoc.Class.CommonState (CommonState(..)) import Text.Pandoc.Data (readDataFile) import Text.Pandoc.Error (PandocError(..)) import Text.Pandoc.Logging (LogMessage(..)) diff --git a/src/Text/Pandoc/Writers.hs b/src/Text/Pandoc/Writers.hs --- a/src/Text/Pandoc/Writers.hs +++ b/src/Text/Pandoc/Writers.hs @@ -76,6 +76,7 @@ , writeTexinfo , writeTextile , writeTypst + , writeXML , writeXWiki , writeZimWiki , getWriter @@ -128,6 +129,7 @@ import Text.Pandoc.Writers.Texinfo import Text.Pandoc.Writers.Textile import Text.Pandoc.Writers.Typst +import Text.Pandoc.Writers.XML import Text.Pandoc.Writers.XWiki import Text.Pandoc.Writers.ZimWiki @@ -203,6 +205,7 @@ ,("chunkedhtml" , ByteStringWriter writeChunkedHTML) ,("djot" , TextWriter writeDjot) ,("ansi" , TextWriter writeANSI) + ,("xml" , TextWriter writeXML) ] -- | Retrieve writer, extensions based on formatSpec (format+extensions). diff --git a/src/Text/Pandoc/Writers/ANSI.hs b/src/Text/Pandoc/Writers/ANSI.hs --- a/src/Text/Pandoc/Writers/ANSI.hs +++ b/src/Text/Pandoc/Writers/ANSI.hs @@ -1,7 +1,8 @@ {-# LANGUAGE OverloadedStrings #-} {- | Module : Text.Pandoc.Writers.ANSI - Copyright : Copyright (C) 2024 Evan Silberman + Copyright : © 2024 Evan Silberman + © 2025 Pandoc contributors License : GNU GPL, version 2 or above Maintainer : John MacFarlane @@ -30,6 +31,7 @@ import qualified Data.Text as T import Data.Text.Lazy (toStrict) import qualified Text.DocLayout as D +import qualified Text.Pandoc.Highlighting as HL hr :: D.HasChars a => D.Doc a hr = rule 20 @@ -76,26 +78,39 @@ metadata <- metaToContext opts (blockListToANSI opts) (inlineListToANSI opts) meta - width <- gets stColumns - let title = titleBlock width metadata + let colwidth = if writerWrapText opts == WrapAuto + then Just $ writerColumns opts + else Nothing + let title = titleBlock colwidth metadata let blocks' = makeSections (writerNumberSections opts) Nothing blocks body <- blockListToANSI opts blocks' notes <- gets $ reverse . stNotes let notemark x = D.literal (tshow (x :: Int) <> ".") <+> D.space let marks = map notemark [1..length notes] let hangWidth = foldr (max . D.offset) 0 marks - let notepretty | not (null notes) = D.cblock width hr $+$ hangMarks hangWidth marks notes - | otherwise = D.empty + let notepretty + | not (null notes) = + (case colwidth of + Nothing -> hr + Just w -> D.cblock w hr) + $+$ hangMarks hangWidth marks notes + | otherwise = D.empty let main = body $+$ notepretty let context = defField "body" main $ defField "titleblock" title metadata return $ case writerTemplate opts of - Nothing -> toStrict $ D.renderANSI (Just width) main - Just tpl -> toStrict $ D.renderANSI (Just width) $ renderTemplate tpl context + Nothing -> toStrict $ D.renderANSI colwidth main + Just tpl -> toStrict $ D.renderANSI colwidth + $ renderTemplate tpl context -titleBlock :: Int -> Context Text -> D.Doc Text -titleBlock width meta = if null most then D.empty else D.cblock width $ most $+$ hr +titleBlock :: Maybe Int -> Context Text -> D.Doc Text +titleBlock width meta = + if null most + then D.empty + else (case width of + Just w -> D.cblock w + Nothing -> id) $ most $+$ hr where title = D.bold (fromMaybe D.empty $ getField "title" meta) subtitle = fromMaybe D.empty $ getField "subtitle" meta @@ -155,15 +170,20 @@ -- Doc Text. blockToANSI opts (CodeBlock attr str) = do table <- gets stInTable - inner <- case (table, writerHighlightStyle opts) of - (_, Nothing) -> return $ defaultStyle str + let highlightWithStyle s = do + let fmt o = formatANSI o s + result = highlight (writerSyntaxMap opts) fmt attr str + return $ case result of + Left _ -> defaultStyle str + Right f -> D.literal f + inner <- case (table, writerHighlightMethod opts) of + (_, NoHighlighting) -> return $ defaultStyle str (True, _) -> return $ defaultStyle str - (False, Just s) -> do - let fmt o = formatANSI o s - result = highlight (writerSyntaxMap opts) fmt attr str - return $ case result of - Left _ -> defaultStyle str - Right f -> D.literal f + (False, Skylighting s) -> highlightWithStyle s + (False, DefaultHighlighting) -> highlightWithStyle HL.defaultStyle + (False, IdiomaticHighlighting) -> do + report $ CouldNotHighlight "no idiomatic highlighting in ANSI" + return $ defaultStyle str return $ nest table inner where defaultStyle = (D.fg D.red) . D.literal nest False = D.nest 4 diff --git a/src/Text/Pandoc/Writers/AsciiDoc.hs b/src/Text/Pandoc/Writers/AsciiDoc.hs --- a/src/Text/Pandoc/Writers/AsciiDoc.hs +++ b/src/Text/Pandoc/Writers/AsciiDoc.hs @@ -416,7 +416,7 @@ let blocksWithTasks = if isLegacy then blocks else (taskListItemToAsciiDoc blocks) - contents <- foldM (addBlock opts) empty blocksWithTasks + contents <- snd <$> foldM (addBlock opts) (False, empty) blocksWithTasks modify $ \s -> s{ bulletListLevel = lev } let marker = text (replicate (lev + 1) '*') return $ marker <> text " " <> listBegin blocksWithTasks <> @@ -433,18 +433,24 @@ listExt = extensionsFromList [Ext_task_lists] addBlock :: PandocMonad m - => WriterOptions -> Doc Text -> Block -> ADW m (Doc Text) -addBlock opts d b = do + => WriterOptions -> (Bool, Doc Text) -> Block -> ADW m (Bool, Doc Text) +addBlock opts (containsContinuation, d) b = do x <- chomp <$> blockToAsciiDoc opts b return $ case b of - BulletList{} -> d <> cr <> x - OrderedList{} -> d <> cr <> x - Para (Math DisplayMath _:_) -> d <> cr <> x - Plain (Math DisplayMath _:_) -> d <> cr <> x - Para{} | isEmpty d -> x - Plain{} | isEmpty d -> x - _ -> d <> cr <> text "+" <> cr <> x + BulletList{} + | containsContinuation -> (False, d <> blankline <> x) -- see #11006 + | otherwise -> (False, d <> cr <> x) + OrderedList (start, sty, _) _ + | containsContinuation + , start == 1 + , sty == DefaultStyle -> (False, d <> blankline <> x) -- see #11006 + | otherwise -> (False, d <> cr <> x) + Para (Math DisplayMath _:_) -> (containsContinuation, d <> cr <> x) + Plain (Math DisplayMath _:_) -> (containsContinuation, d <> cr <> x) + Para{} | isEmpty d -> (containsContinuation, x) + Plain{} | isEmpty d -> (containsContinuation, x) + _ -> (True, d <> cr <> text "+" <> cr <> x) listBegin :: [Block] -> Doc Text listBegin blocks = @@ -464,7 +470,7 @@ orderedListItemToAsciiDoc opts blocks = do lev <- gets orderedListLevel modify $ \s -> s{ orderedListLevel = lev + 1 } - contents <- foldM (addBlock opts) empty blocks + contents <- snd <$> foldM (addBlock opts) (False, empty) blocks modify $ \s -> s{ orderedListLevel = lev } let marker = text (replicate (lev + 1) '.') return $ marker <> text " " <> listBegin blocks <> contents <> cr diff --git a/src/Text/Pandoc/Writers/ConTeXt.hs b/src/Text/Pandoc/Writers/ConTeXt.hs --- a/src/Text/Pandoc/Writers/ConTeXt.hs +++ b/src/Text/Pandoc/Writers/ConTeXt.hs @@ -4,7 +4,7 @@ {-# LANGUAGE ViewPatterns #-} {- | Module : Text.Pandoc.Writers.ConTeXt - Copyright : Copyright (C) 2007-2024 John MacFarlane + Copyright : Copyright (C) 2007-2025 John MacFarlane License : GNU GPL, version 2 or above Maintainer : John MacFarlane @@ -20,7 +20,7 @@ import Data.Char (ord, isDigit) import Data.List (intersperse) import Data.List.NonEmpty (NonEmpty ((:|))) -import Data.Maybe (isNothing, mapMaybe, catMaybes) +import Data.Maybe (mapMaybe, catMaybes) import Data.Monoid (Any (Any, getAny)) import Data.Text (Text) import qualified Data.Text as T @@ -131,10 +131,10 @@ _ -> id) $ defField "emphasis-commands" (mconcat $ Map.elems (stEmphasisCommands st)) - $ (case writerHighlightStyle options of - Just sty | stHighlighting st -> - defField "highlighting-commands" (styleToConTeXt sty) - _ -> id) + $ (case writerHighlightMethod options of + Skylighting sty | stHighlighting st -> + defField "highlighting-commands" (styleToConTeXt sty) + _ -> id) $ (case T.toLower $ lookupMetaString "pdfa" meta of "true" -> resetField "pdfa" (T.pack "1b:2005") _ -> id) metadata @@ -233,9 +233,9 @@ return (literal h) -- blankline because \stoptyping can't have anything after it, inc. '}' ($$ blankline) . flush <$> - if null classes || isNothing (writerHighlightStyle opts) - then pure unhighlighted - else highlighted + case writerHighlightMethod opts of + Skylighting _ | not (null classes) -> pure unhighlighted + _ -> highlighted blockToConTeXt b@(RawBlock f str) | f == Format "context" || f == Format "tex" = return $ literal str <> blankline | otherwise = empty <$ report (BlockNotRendered b) @@ -625,9 +625,9 @@ Right h -> do modify (\st -> st{ stHighlighting = True }) return (text (T.unpack h)) - if isNothing (writerHighlightStyle opts) || null classes - then rawCode - else highlightCode + case writerHighlightMethod opts of + Skylighting _ | not (null classes) -> highlightCode + _ -> rawCode inlineToConTeXt (Quoted SingleQuote lst) = do contents <- inlineListToConTeXt lst return $ "\\quote" <> braces contents diff --git a/src/Text/Pandoc/Writers/Djot.hs b/src/Text/Pandoc/Writers/Djot.hs --- a/src/Text/Pandoc/Writers/Djot.hs +++ b/src/Text/Pandoc/Writers/Djot.hs @@ -100,7 +100,8 @@ blockToDjot (Header lev attr ils) = fmap (D.addAttr (toDjotAttr attr)) . D.heading lev <$> inlinesToDjot ils blockToDjot HorizontalRule = pure D.thematicBreak -blockToDjot (Div (ident,"section":cls,kvs) bls@(Header _ _ ils : _)) = do +blockToDjot (Div (ident,"section":cls,kvs) + (Header lev (_,hcls,hkvs) ils : bls)) = do ilsBs <- D.inlinesToByteString <$> inlinesToDjot ils let ident' = toIdentifier ilsBs let label = D.normalizeLabel ilsBs @@ -112,8 +113,10 @@ fmap (D.addAttr (toDjotAttr (if autoid then "" else ident, filter (/= "section") cls, - filter (\(k,_) -> k /= "wrapper") kvs))) . D.section - <$> blocksToDjot bls + filter (\(k,_) -> k /= "wrapper") kvs) <> + toDjotAttr ("", [c | c <- hcls, c `notElem` cls], hkvs))) + . D.section + <$> blocksToDjot (Header lev mempty ils : bls) blockToDjot (Div attr@(ident,cls,kvs) bls) | Just "1" <- lookup "wrapper" kvs = fmap (D.addAttr diff --git a/src/Text/Pandoc/Writers/DocBook.hs b/src/Text/Pandoc/Writers/DocBook.hs --- a/src/Text/Pandoc/Writers/DocBook.hs +++ b/src/Text/Pandoc/Writers/DocBook.hs @@ -265,7 +265,7 @@ let attribs = [("spacing", "compact") | isTightList lst] inTags True "itemizedlist" attribs <$> listItemsToDocBook opts lst blockToDocBook _ (OrderedList _ []) = return empty -blockToDocBook opts (OrderedList (start, numstyle, _) (first:rest)) = do +blockToDocBook opts (OrderedList (start, numstyle, _) items) = do let numeration = case numstyle of DefaultStyle -> [] Decimal -> [("numeration", "arabic")] @@ -274,17 +274,10 @@ LowerAlpha -> [("numeration", "loweralpha")] UpperRoman -> [("numeration", "upperroman")] LowerRoman -> [("numeration", "lowerroman")] - spacing = [("spacing", "compact") | isTightList (first:rest)] - attribs = numeration <> spacing - items <- if start == 1 - then listItemsToDocBook opts (first:rest) - else do - first' <- blocksToDocBook opts (map plainToPara first) - rest' <- listItemsToDocBook opts rest - return $ - inTags True "listitem" [("override",tshow start)] first' $$ - rest' - return $ inTags True "orderedlist" attribs items + spacing = [("spacing", "compact") | isTightList items] + startnum = [("startingnumber", tshow start) | start /= 1] + attribs = numeration <> spacing <> startnum + inTags True "orderedlist" attribs <$> listItemsToDocBook opts items blockToDocBook opts (DefinitionList lst) = do let attribs = [("spacing", "compact") | isTightList $ concatMap snd lst] inTags True "variablelist" attribs <$> deflistItemsToDocBook opts lst diff --git a/src/Text/Pandoc/Writers/Docx.hs b/src/Text/Pandoc/Writers/Docx.hs --- a/src/Text/Pandoc/Writers/Docx.hs +++ b/src/Text/Pandoc/Writers/Docx.hs @@ -6,7 +6,7 @@ {-# LANGUAGE OverloadedStrings #-} {- | Module : Text.Pandoc.Writers.Docx - Copyright : Copyright (C) 2012-2024 John MacFarlane + Copyright : Copyright (C) 2012-2025 John MacFarlane License : GNU GPL, version 2 or above Maintainer : John MacFarlane @@ -48,6 +48,7 @@ import qualified Text.Pandoc.UTF8 as UTF8 import Text.Pandoc.Definition import Text.Pandoc.Error +import Text.Pandoc.Highlighting (defaultStyle) import Text.Pandoc.MIME (getMimeTypeDef) import Text.Pandoc.Options import Text.Pandoc.Readers.Docx.Parse (extractTarget) @@ -350,7 +351,11 @@ let newstyles = map newParaPropToOpenXml newDynamicParaProps ++ map newTextPropToOpenXml newDynamicTextProps ++ - maybe [] (styleToOpenXml styleMaps) (writerHighlightStyle opts) + (case writerHighlightMethod opts of + Skylighting sty -> styleToOpenXml styleMaps sty + DefaultHighlighting -> styleToOpenXml styleMaps + defaultStyle + _ -> []) let styledoc' = styledoc{ elContent = elContent styledoc ++ map Elem newstyles } let styleEntry = toEntry stylepath epochtime $ renderXml styledoc' diff --git a/src/Text/Pandoc/Writers/Docx/OpenXML.hs b/src/Text/Pandoc/Writers/Docx/OpenXML.hs --- a/src/Text/Pandoc/Writers/Docx/OpenXML.hs +++ b/src/Text/Pandoc/Writers/Docx/OpenXML.hs @@ -9,7 +9,7 @@ {-# LANGUAGE TypeApplications #-} {- | Module : Text.Pandoc.Writers.Docx - Copyright : Copyright (C) 2012-2024 John MacFarlane + Copyright : Copyright (C) 2012-2025 John MacFarlane License : GNU GPL, version 2 or above Maintainer : John MacFarlane @@ -20,7 +20,7 @@ -} module Text.Pandoc.Writers.Docx.OpenXML ( writeOpenXML, maxListLevel ) where -import Control.Monad (when, unless) +import Control.Monad ((>=>), when, unless) import Control.Applicative ((<|>)) import Control.Monad.Except (catchError) import Crypto.Hash (hashWith, SHA1(SHA1)) @@ -30,7 +30,7 @@ import Data.Ord (comparing) import Data.String (fromString) import qualified Data.Map as M -import Data.Maybe (fromMaybe, isNothing, maybeToList, isJust) +import Data.Maybe (fromMaybe, maybeToList, isJust) import Control.Monad.State ( gets, modify, MonadTrans(lift) ) import Control.Monad.Reader ( asks, MonadReader(local) ) import qualified Data.Set as Set @@ -220,6 +220,15 @@ ]) -- w:sdtContent ]] -- w:sdt +-- | Separator element between sections +sectionSeparator :: PandocMonad m => WS m (Maybe Content) +sectionSeparator = do + asks envSectPr >>= \case + Just sectPrElem -> pure $ + Just $ Elem (mknode "w:p" [] (mknode "w:pPr" [] [sectPrElem])) + Nothing -> pure + Nothing + -- | Convert Pandoc document to rendered document contents plus two lists of -- OpenXML elements (footnotes and comments). writeOpenXML :: PandocMonad m @@ -317,7 +326,17 @@ -- | Convert a list of Pandoc blocks to OpenXML. blocksToOpenXML :: (PandocMonad m) => WriterOptions -> [Block] -> WS m [Content] -blocksToOpenXML opts = fmap concat . mapM (blockToOpenXML opts) . separateTables . filter (not . isForeignRawBlock) +blocksToOpenXML opts = + fmap concat . mapM (blockToOpenXML opts) + . separateTables . filter (not . isForeignRawBlock) + >=> + \case + a@(x:xs) -> do + sep <- sectionSeparator + if Just x == sep + then pure xs + else pure a + [] -> pure [] isForeignRawBlock :: Block -> Bool isForeignRawBlock (RawBlock format _) = format /= "openxml" @@ -395,13 +414,9 @@ Nothing -> return [] else return [] contents <- (number ++) <$> inlinesToOpenXML opts lst - sectpr <- asks envSectPr - let addSectionBreak - | isSection - , Just sectPrElem <- sectpr - = (Elem (mknode "w:p" [] - (mknode "w:pPr" [] [sectPrElem])) :) - | otherwise = id + addSectionBreak <- sectionSeparator >>= \case + Just sep | isSection -> pure (sep:) + _ -> pure id addSectionBreak <$> if T.null ident then return [Elem $ mknode "w:p" [] (map Elem paraProps ++ contents)] @@ -874,14 +889,14 @@ maybeToList (lookup toktype tokTypesMap) , mknode "w:t" [("xml:space","preserve")] tok ] withTextPropM (rStyleM "Verbatim Char") - $ if isNothing (writerHighlightStyle opts) - then unhighlighted - else case highlight (writerSyntaxMap opts) - formatOpenXML attrs str of - Right h -> return (map Elem h) - Left msg -> do - unless (T.null msg) $ report $ CouldNotHighlight msg - unhighlighted + $ case writerHighlightMethod opts of + Skylighting _ -> + case highlight (writerSyntaxMap opts) formatOpenXML attrs str of + Right h -> return (map Elem h) + Left msg -> do + unless (T.null msg) $ report $ CouldNotHighlight msg + unhighlighted + _ -> unhighlighted inlineToOpenXML' opts (Note bs) = do notes <- gets stFootnotes notenum <- getUniqueId @@ -1041,6 +1056,7 @@ Just Emf -> ".emf" Just Tiff -> ".tiff" Just Webp -> ".webp" + Just Avif -> ".avif" Nothing -> "" imgpath = "media/" <> ident <> imgext mbMimeType = mt <|> getMimeType (T.unpack imgpath) diff --git a/src/Text/Pandoc/Writers/HTML.hs b/src/Text/Pandoc/Writers/HTML.hs --- a/src/Text/Pandoc/Writers/HTML.hs +++ b/src/Text/Pandoc/Writers/HTML.hs @@ -53,7 +53,7 @@ import Text.Blaze.Html hiding (contents) import Text.Pandoc.Definition import Text.Pandoc.Highlighting (formatHtmlBlock, formatHtml4Block, - formatHtmlInline, highlight, styleToCss) + formatHtmlInline, highlight, styleToCss, defaultStyle) import Text.Pandoc.ImageSize import Text.Pandoc.Options import Text.Pandoc.Shared @@ -356,10 +356,14 @@ let mCss :: Maybe [Text] = lookupContext "css" metadata let context :: Context Text context = (if stHighlighting st - then case writerHighlightStyle opts of - Just sty -> defField "highlighting-css" - (literal $ T.pack $ styleToCss sty) - Nothing -> id + then case writerHighlightMethod opts of + Skylighting sty -> + defField "highlighting-css" + (literal $ T.pack $ styleToCss sty) + DefaultHighlighting -> + defField "highlighting-css" + (literal $ T.pack $ styleToCss defaultStyle) + _ -> id else id) . (if stCsl st then defField "csl-css" True . @@ -845,6 +849,11 @@ if null innerSecs then mempty else nl <> innerContents +blockToHtmlInner opts (Div (ident, classes, kvs) [b]) + | Just "1" <- lookup "wrapper" kvs + -- unwrap "wrapper" div, putting attr on child + = blockToHtmlInner opts b >>= + addAttrs opts (ident, classes, [(k,v) | (k,v) <- kvs, k /= "wrapper"]) blockToHtmlInner opts (Div attr@(ident, classes, kvs') bs) = do html5 <- gets stHtml5 slideVariant <- gets stSlideVariant @@ -943,11 +952,13 @@ adjCode = if tolhs then T.unlines . map ("> " <>) . T.lines $ rawCode else rawCode - hlCode = if isJust (writerHighlightStyle opts) - then highlight (writerSyntaxMap opts) - (if html5 then formatHtmlBlock else formatHtml4Block) - (id'',classes',keyvals) adjCode - else Left "" + highlighted = highlight (writerSyntaxMap opts) + (if html5 then formatHtmlBlock else formatHtml4Block) + (id'',classes',keyvals) adjCode + hlCode = case writerHighlightMethod opts of + Skylighting _ -> highlighted + DefaultHighlighting -> highlighted + _ -> Left "" case hlCode of Left msg -> do unless (T.null msg) $ @@ -1437,11 +1448,12 @@ modify $ \st -> st{ stHighlighting = True } addAttrs opts (ids,[],kvs) $ fromMaybe id sampOrVar h - where hlCode = if isJust (writerHighlightStyle opts) - then highlight - (writerSyntaxMap opts) - formatHtmlInline attr str - else Left "" + where hlCode = case writerHighlightMethod opts of + Skylighting _ -> highlighted + DefaultHighlighting -> highlighted + _ -> Left "" + highlighted = highlight (writerSyntaxMap opts) + formatHtmlInline attr str (sampOrVar,cs') | "sample" `elem` cs = (Just H.samp,"sample" `delete` cs) diff --git a/src/Text/Pandoc/Writers/LaTeX.hs b/src/Text/Pandoc/Writers/LaTeX.hs --- a/src/Text/Pandoc/Writers/LaTeX.hs +++ b/src/Text/Pandoc/Writers/LaTeX.hs @@ -7,7 +7,7 @@ {-# LANGUAGE ViewPatterns #-} {- | Module : Text.Pandoc.Writers.LaTeX - Copyright : Copyright (C) 2006-2024 John MacFarlane + Copyright : Copyright (C) 2006-2025 John MacFarlane License : GNU GPL, version 2 or above Maintainer : John MacFarlane @@ -30,8 +30,9 @@ liftM, when, unless ) +import Crypto.Hash (hashWith, MD5(MD5)) import Data.Containers.ListUtils (nubOrd) -import Data.Char (isDigit) +import Data.Char (isDigit, isAscii) import Data.List (intersperse, (\\)) import Data.Maybe (catMaybes, fromMaybe, isJust, mapMaybe, isNothing) import Data.Monoid (Any (..)) @@ -40,10 +41,11 @@ import Network.URI (unEscapeString) import Text.DocTemplates (FromContext(lookupContext), Val(..), renderTemplate) import Text.Collate.Lang (renderLang) -import Text.Pandoc.Class.PandocMonad (PandocMonad, report, toLang) +import Text.Pandoc.Class.PandocMonad (PandocMonad, getPOSIXTime, lookupEnv, + report, toLang) import Text.Pandoc.Definition import Text.Pandoc.Highlighting (formatLaTeXBlock, formatLaTeXInline, highlight, - styleToLaTeX) + defaultStyle, styleToLaTeX) import Text.Pandoc.ImageSize import Text.Pandoc.Logging import Text.Pandoc.Options @@ -63,7 +65,11 @@ wrapDiv, hypertarget, labelFor, getListingsLanguage, mbBraced) import Text.Pandoc.Writers.Shared +import qualified Data.Attoparsec.Text as A +import qualified Text.Pandoc.UTF8 as UTF8 import qualified Text.Pandoc.Writers.AnnotatedTable as Ann +import Data.Char (isLetter) +import Control.Applicative ((<|>)) -- Work around problems with notes inside emphasis (see #8982) isolateBigNotes :: ([Inline] -> Inline) -> [Inline] -> [Inline] @@ -176,8 +182,22 @@ main <- blockListToLaTeX blocks''' biblioTitle <- inlineListToLaTeX lastHeader st <- get - titleMeta <- stringToLaTeX TextString $ stringify $ docTitle meta + titleMeta <- escapeCommas <$> -- see #10501 + stringToLaTeX TextString (stringify $ docTitle meta) authorsMeta <- mapM (stringToLaTeX TextString . stringify) $ docAuthors meta + -- The trailer ID is as hash used to identify the PDF. Taking control of its + -- value is important when aiming for reproducible PDF generation. Setting + -- `SOURCE_DATE_EPOCH` is the traditional method used to control + -- reproducible builds. There are no cryptographic requirements for the ID, + -- so the 128bits (16 bytes) of MD5 are appropriate. + reproduciblePDF <- isJust <$> lookupEnv "SOURCE_DATE_EPOCH" + trailerID <- do + time <- getPOSIXTime + let hash = T.pack . show . hashWith MD5 $ mconcat + [ UTF8.fromString $ show time + , UTF8.fromText $ render Nothing main + ] + pure $ mconcat [ "<", hash, "> <", hash, ">" ] -- we need a default here since lang is used in template conditionals let hasStringValue x = isJust (getField x metadata :: Maybe (Doc Text)) let geometryFromMargins = mconcat $ intersperse ("," :: Doc Text) $ @@ -212,6 +232,7 @@ defField "verbatim-in-note" (stVerbInNote st) $ defField "tables" (stTable st) $ defField "multirow" (stMultiRow st) $ + defField "cancel" (stCancel st) $ defField "strikeout" (stStrikeout st) $ defField "url" (stUrl st) $ defField "numbersections" (writerNumberSections options) $ @@ -221,15 +242,20 @@ defField "svg" (stSVG st) $ defField "has-chapters" (stHasChapters st) $ defField "has-frontmatter" (documentClass `elem` frontmatterClasses) $ - defField "listings" (writerListings options || stLHS st) $ + defField "listings" (writerHighlightMethod options == + IdiomaticHighlighting + || stLHS st) $ defField "zero-width-non-joiner" (stZwnj st) $ defField "beamer" beamer $ (if stHighlighting st - then case writerHighlightStyle options of - Just sty -> + then case writerHighlightMethod options of + Skylighting sty -> defField "highlighting-macros" (T.stripEnd $ styleToLaTeX sty) - Nothing -> id + DefaultHighlighting -> + defField "highlighting-macros" + (T.stripEnd $ styleToLaTeX defaultStyle) + _ -> id else id) $ (case writerCiteMethod options of Natbib -> defField "biblio-title" biblioTitle . @@ -254,7 +280,10 @@ Just (Just ('A', ds)) | not (T.null ds) && T.all isDigit ds -> resetField "papersize" ("a" <> ds) - _ -> id) + _ -> id) . + (if reproduciblePDF + then defField "pdf-trailer-id" trailerID + else id) $ metadata let babelLang = mblang >>= toBabel let context' = @@ -282,6 +311,10 @@ Nothing -> main Just tpl -> renderTemplate tpl context' +-- Commas in title-meta need to be put in braces; see #10501 +escapeCommas :: Text -> Text +escapeCommas = T.replace "," "{,}" + toSlides :: PandocMonad m => [Block] -> LW m [Block] toSlides bs = do opts <- gets stOptions @@ -469,7 +502,8 @@ ref <- toLabel identifier kvs <- mapM (\(k,v) -> (k,) <$> stringToLaTeX TextString v) keyvalAttr - let params = if writerListings (stOptions st) + let params = if writerHighlightMethod (stOptions st) + == IdiomaticHighlighting then (case getListingsLanguage classes of Just l -> [ "language=" <> mbBraced l ] Nothing -> []) ++ @@ -506,9 +540,12 @@ case () of _ | isEnabled Ext_literate_haskell opts && "haskell" `elem` classes && "literate" `elem` classes -> lhsCodeBlock - | writerListings opts -> listingsCodeBlock - | not (null classes) && isJust (writerHighlightStyle opts) + | writerHighlightMethod opts == IdiomaticHighlighting + -> listingsCodeBlock + | not (null classes), Skylighting _ <- writerHighlightMethod opts -> highlightedCodeBlock + | not (null classes), DefaultHighlighting <- writerHighlightMethod opts + -> highlightedCodeBlock -- we don't want to use \begin{verbatim} if our code -- contains \end{verbatim}: | inNote @@ -616,11 +653,14 @@ blockToLaTeX (Table attr blkCapt specs thead tbodies tfoot) = tableToLaTeX inlineListToLaTeX blockListToLaTeX (Ann.toTable attr blkCapt specs thead tbodies tfoot) -blockToLaTeX (Figure (ident, _, _) captnode body) = do +blockToLaTeX (Figure (ident, _, kvs) captnode body) = do opts <- gets stOptions (capt, captForLof, footnotes) <- getCaption inlineListToLaTeX True captnode lab <- labelFor ident let caption = "\\caption" <> captForLof <> braces capt <> lab + placement = case lookup "latex-placement" kvs of + Just p -> brackets (text (T.unpack p)) + _ -> text "" isSubfigure <- gets stInFigure modify $ \st -> st{ stInFigure = True } @@ -652,7 +692,7 @@ cr <> "\\begin{center}" $$ contents $+$ capt $$ "\\end{center}" _ | isSubfigure -> innards - _ -> cr <> "\\begin{figure}" $$ innards $$ "\\end{figure}") + _ -> cr <> "\\begin{figure}" <> placement $$ innards $$ "\\end{figure}") $$ footnotes toSubfigure :: PandocMonad m => Int -> Block -> LW m (Doc Text) @@ -960,12 +1000,14 @@ -- incorrect results if there is a space (see #5529). let inMbox x = "\\mbox" <> braces x (if inSoul then inMbox else id) <$> - case () of + case writerHighlightMethod opts of _ | inHeading || inItem -> rawCode -- see #5574 - | writerListings opts -> listingsCode - | isJust (writerHighlightStyle opts) && not (null classes) + IdiomaticHighlighting -> listingsCode + Skylighting _ | not (null classes) -> highlightCode - | otherwise -> rawCode + DefaultHighlighting | not (null classes) + -> highlightCode + _noHighlighting -> rawCode inlineToLaTeX (Quoted qt lst) = do contents <- inlineListToLaTeX lst csquotes <- liftM stCsquotes get @@ -1002,10 +1044,16 @@ inlineToLaTeX (Str str) = do setEmptyLine False liftM literal $ stringToLaTeX TextString str +inlineToLaTeX (Math _ str) + | isMathEnv str -- see #9711 + = do setEmptyLine False + when (needsCancel str) $ modify $ \st -> st{ stCancel = True } + pure $ literal str inlineToLaTeX (Math InlineMath str) = do setEmptyLine False inSoul <- gets stInSoulCommand let contents = literal (handleMathComment str) + when (needsCancel str) $ modify $ \st -> st{ stCancel = True } return $ if inSoul -- #9597 then "$" <> contents <> "$" @@ -1014,6 +1062,7 @@ setEmptyLine False inSoul <- gets stInSoulCommand let contents = literal (handleMathComment str) + when (needsCancel str) $ modify $ \st -> st{ stCancel = True } return $ if inSoul -- # 9597 then "$$" <> contents <> "$$" @@ -1053,7 +1102,9 @@ then "\\hyperlink" <> braces (literal lab) <> braces contents else "\\hyperref" <> brackets (literal lab) <> braces contents _ -> case txt of - [Str x] | unEscapeString (T.unpack x) == unEscapeString (T.unpack src) -> -- autolink + [Str x] | T.all isAscii x -- see #8802 + , unEscapeString (T.unpack x) == + unEscapeString (T.unpack src) -> -- autolink do modify $ \s -> s{ stUrl = True } src' <- stringToLaTeX URLString (escapeURI src) return $ literal $ "\\url{" <> src' <> "}" @@ -1289,3 +1340,40 @@ , "galician" , "slovene" ] + +isMathEnv :: Text -> Bool +isMathEnv t = + case T.stripPrefix "\\begin{" t of + Nothing -> False + Just t' -> T.takeWhile (/= '}') t' `elem` + [ "align", "align*" + , "flalign", "flalign*" + , "alignat", "alignat*" + , "dmath", "dmath*" + , "dgroup", "dgroup*" + , "darray", "darray*" + , "gather", "gather*" + , "multline", "multline*" + , "split" + , "subequations" + , "equation", "equation*" + , "eqnarray" + , "displaymath" + ] + +-- True if the math needs the cancel package +needsCancel :: Text -> Bool +needsCancel t = + case A.parseOnly pCancel t of + Right True -> True + _ -> False + where + pCancel = (False <$ A.endOfInput) <|> do + c <- A.anyChar + case c of + '\\' -> do + x <- A.takeWhile isLetter + if x == "cancel" || x == "xcancel" || x == "bcancel" + then return True + else pCancel + _ -> pCancel diff --git a/src/Text/Pandoc/Writers/LaTeX/Types.hs b/src/Text/Pandoc/Writers/LaTeX/Types.hs --- a/src/Text/Pandoc/Writers/LaTeX/Types.hs +++ b/src/Text/Pandoc/Writers/LaTeX/Types.hs @@ -53,6 +53,7 @@ , stIsFirstInDefinition :: Bool -- ^ first block in a defn list , stLang :: Maybe Lang -- ^ lang specified in metadata , stInSoulCommand :: Bool -- ^ in a soul command like ul + , stCancel :: Bool -- ^ true if document uses \cancel } startingState :: WriterOptions -> WriterState @@ -93,4 +94,5 @@ , stIsFirstInDefinition = False , stLang = Nothing , stInSoulCommand = False + , stCancel = False } diff --git a/src/Text/Pandoc/Writers/LaTeX/Util.hs b/src/Text/Pandoc/Writers/LaTeX/Util.hs --- a/src/Text/Pandoc/Writers/LaTeX/Util.hs +++ b/src/Text/Pandoc/Writers/LaTeX/Util.hs @@ -134,6 +134,7 @@ ']' -> emits "{]}" -- optional arguments '\'' -> emitcseq "\\textquotesingle" '\160' -> emits "~" + '\x00AD' -> emits "\\-" -- shy hyphen '\x200B' -> emits "\\hspace{0pt}" -- zero-width space '\x202F' -> emits "\\," '\x2026' | ligatures -> emitcseq "\\ldots" diff --git a/src/Text/Pandoc/Writers/Man.hs b/src/Text/Pandoc/Writers/Man.hs --- a/src/Text/Pandoc/Writers/Man.hs +++ b/src/Text/Pandoc/Writers/Man.hs @@ -362,7 +362,10 @@ formatTok :: WriterOptions -> FormatOptions -> (TokenType, Text) -> Doc Text formatTok wopts _fopts (toktype, t) = let txt = literal (escString wopts t) - styleMap = tokenStyles <$> writerHighlightStyle wopts + styleMap = case writerHighlightMethod wopts of + Skylighting style -> Just $ tokenStyles style + DefaultHighlighting -> Just $ tokenStyles defaultStyle + _ -> Nothing tokStyle = fromMaybe defStyle $ styleMap >>= M.lookup toktype in if toktype == NormalTok then txt diff --git a/src/Text/Pandoc/Writers/Markdown.hs b/src/Text/Pandoc/Writers/Markdown.hs --- a/src/Text/Pandoc/Writers/Markdown.hs +++ b/src/Text/Pandoc/Writers/Markdown.hs @@ -25,7 +25,7 @@ import Control.Monad.Reader ( asks, MonadReader(local) ) import Control.Monad.State.Strict ( gets, modify ) import Data.Default -import Data.List (intersperse, sortOn, union) +import Data.List (intersperse, sortOn, union, find) import Data.List.NonEmpty (nonEmpty, NonEmpty(..)) import qualified Data.Map as M import Data.Maybe (fromMaybe, mapMaybe, isNothing) @@ -369,6 +369,10 @@ -> Block -- ^ Block element -> MD m (Doc Text) blockToMarkdown' opts (Div attrs@(_,classes,_) bs) + | ("sourceCode":_) <- classes + , [CodeBlock (_,"sourceCode":_,_) _] <- bs + -- skip pandoc-generated Div wrappers around code blocks + = blockListToMarkdown opts bs | isEnabled Ext_alerts opts , (cls:_) <- classes , cls `elem` ["note", "tip", "warning", "caution", "important"] @@ -396,8 +400,7 @@ | (take 3 (T.unpack id')) == "ref" -> contents <> blankline | otherwise -> contents <> blankline - | isEnabled Ext_fenced_divs opts && - attrs /= nullAttr -> + | isEnabled Ext_fenced_divs opts -> let attrsToMd = if variant == Commonmark then attrsToMarkdown opts else classOrAttrsToMarkdown opts @@ -589,9 +592,11 @@ attrs = if isEnabled Ext_fenced_code_attributes opts || isEnabled Ext_attributes opts then nowrap $ " " <> classOrAttrsToMarkdown opts attribs - else case attribs of - (_,cls:_,_) -> " " <> literal cls - _ -> empty + else + let (_,cls,_) = attribs + in case getLangFromClasses cls of + Just l -> " " <> literal l + Nothing -> empty blockToMarkdown' opts (BlockQuote blocks) = do variant <- asks envVariant -- if we're writing literate haskell, put a space before the bird tracks @@ -851,26 +856,21 @@ let tabStop = writerTabStop opts variant <- asks envVariant let leader = case variant of - PlainText -> " " - Markua -> ":" - _ -> ": " - let sps = case writerTabStop opts - 3 of - n | n > 0 -> literal $ T.replicate n " " - _ -> literal " " + PlainText -> " " + _ -> ":" + let leadingChars = case tabStop of + -- Always use two leading characters for Markua + n | n >= 2 && variant /= Markua -> n + _ -> 2 + let sps = literal $ T.replicate (leadingChars - 1) " " let isTight = case defs of ((Plain _ : _): _) -> True _ -> False - if isEnabled Ext_compact_definition_lists opts - then do - let contents = vcat $ map (\d -> hang tabStop (leader <> sps) - $ vcat d <> cr) defs' - return $ nowrap labelText <> cr <> contents <> cr - else do - let contents = (if isTight then vcat else vsep) $ map - (\d -> hang tabStop (leader <> sps) $ vcat d) - defs' - return $ blankline <> nowrap labelText $$ - (if isTight then empty else blankline) <> contents <> blankline + let contents = (if isTight then vcat else vsep) $ map + (\d -> hang leadingChars (leader <> sps) $ vcat d) + defs' + return $ blankline <> nowrap labelText $$ + (if isTight then empty else blankline) <> contents <> blankline else return $ nowrap (chomp labelText <> literal " " <> cr) <> vsep (map vsep defs') <> blankline @@ -948,3 +948,14 @@ where go (Div _ bls') n = max (n + 1) (foldr go (n + 1) bls') go _ n = n + +-- Identify the class in a list of classes that corresponds to +-- the language syntax. language-X turns to X. +getLangFromClasses :: [Text] -> Maybe Text +getLangFromClasses cs = + case find ("language-" `T.isPrefixOf`) cs of + Just x -> Just (T.drop 9 x) + Nothing -> + case filter (/= "sourceCode") cs of + (x:_) -> Just x + [] -> Nothing diff --git a/src/Text/Pandoc/Writers/Ms.hs b/src/Text/Pandoc/Writers/Ms.hs --- a/src/Text/Pandoc/Writers/Ms.hs +++ b/src/Text/Pandoc/Writers/Ms.hs @@ -76,7 +76,9 @@ let authorsMeta = map (escapeStr opts . stringify) $ docAuthors meta hasHighlighting <- gets stHighlighting let highlightingMacros = if hasHighlighting - then maybe mempty styleToMs $ writerHighlightStyle opts + then case writerHighlightMethod opts of + Skylighting sty -> styleToMs sty + _ -> mempty else mempty let context = defField "body" main diff --git a/src/Text/Pandoc/Writers/ODT.hs b/src/Text/Pandoc/Writers/ODT.hs --- a/src/Text/Pandoc/Writers/ODT.hs +++ b/src/Text/Pandoc/Writers/ODT.hs @@ -34,7 +34,8 @@ import Text.Pandoc.ImageSize import Text.Pandoc.Logging import Text.Pandoc.MIME (extensionFromMimeType, getMimeType) -import Text.Pandoc.Options (WrapOption (..), WriterOptions (..)) +import Text.Pandoc.Options (WrapOption (..), WriterOptions (..), + HighlightMethod(Skylighting)) import Text.DocLayout import Text.Pandoc.Shared (stringify, tshow) import Text.Pandoc.Version (pandocVersionText) @@ -211,7 +212,9 @@ . maybe id addLang mbLang . transformElement (\qn -> qName qn == "styles" && qPrefix qn == Just "office" ) - (maybe id addHlStyles (writerHighlightStyle opts)) + (case writerHighlightMethod opts of + Skylighting style -> addHlStyles style + _ -> id) $ d ) | otherwise = pure e entries <- mapM goEntry (zEntries arch) diff --git a/src/Text/Pandoc/Writers/OpenDocument.hs b/src/Text/Pandoc/Writers/OpenDocument.hs --- a/src/Text/Pandoc/Writers/OpenDocument.hs +++ b/src/Text/Pandoc/Writers/OpenDocument.hs @@ -21,7 +21,6 @@ import Data.Foldable (find) import Data.List (sortOn, sortBy, foldl') import qualified Data.Map as Map -import Data.Maybe (isNothing) import Data.Ord (comparing) import qualified Data.Set as Set import Data.Text (Text) @@ -390,15 +389,16 @@ OrderedList a b -> setFirstPara >> orderedList a b CodeBlock attrs s -> do setFirstPara - if isNothing (writerHighlightStyle o) - then unhighlighted s - else case highlight (writerSyntaxMap o) formatOpenDocument attrs s of + case writerHighlightMethod o of + Skylighting {} -> + case highlight (writerSyntaxMap o) formatOpenDocument attrs s of Right h -> return $ flush . vcat $ map (inTags True "text:p" [("text:style-name", "Preformatted_20_Text")] . hcat) h Left msg -> do unless (T.null msg) $ report $ CouldNotHighlight msg unhighlighted s + _ -> unhighlighted s Table a bc s th tb tf -> setFirstPara >> table o (Ann.toTable a bc s th tb tf) HorizontalRule -> setFirstPara >> return (selfClosingTag "text:p" @@ -630,14 +630,14 @@ Subscript l -> withTextStyle Sub $ inlinesToOpenDocument o l SmallCaps l -> withTextStyle SmallC $ inlinesToOpenDocument o l Quoted t l -> inQuotes t <$> inlinesToOpenDocument o l - Code attrs s -> if isNothing (writerHighlightStyle o) - then unhighlighted s - else case highlight (writerSyntaxMap o) - formatOpenDocument attrs s of + Code attrs s -> case writerHighlightMethod o of + Skylighting {} -> + case highlight (writerSyntaxMap o) formatOpenDocument attrs s of Right h -> inlinedCode $ mconcat $ mconcat h Left msg -> do unless (T.null msg) $ report $ CouldNotHighlight msg unhighlighted s + _ -> unhighlighted s Math t s -> lift (texMathToInlines t s) >>= inlinesToOpenDocument o Cite _ l -> inlinesToOpenDocument o l diff --git a/src/Text/Pandoc/Writers/Org.hs b/src/Text/Pandoc/Writers/Org.hs --- a/src/Text/Pandoc/Writers/Org.hs +++ b/src/Text/Pandoc/Writers/Org.hs @@ -3,8 +3,8 @@ {- | Module : Text.Pandoc.Writers.Org Copyright : © 2010-2015 Puneeth Chaganti - 2010-2024 John MacFarlane - 2016-2024 Albert Krewinkel + 2010-2025 John MacFarlane + 2016-2025 Albert Krewinkel License : GNU GPL, version 2 or above Maintainer : Albert Krewinkel @@ -71,6 +71,17 @@ let main = body $+$ notes let context = defField "body" main . defField "math" hasMath + . defField "options" + (M.fromList + ((if isEnabled Ext_smart_quotes opts + then (("'", "t"):) + else id) . + (if not (isEnabled Ext_special_strings opts) + then (("-", "nil"):) + else id) + $ ([] :: [(Text, Text)]))) + . defField "option-special-strings" + (isEnabled Ext_special_strings opts) $ metadata return $ render colwidth $ case writerTemplate opts of @@ -89,20 +100,28 @@ let marker = "[fn:" ++ show num ++ "] " return $ hang (length marker) (text marker) contents +-- | Replace Unicode characters with their ASCII representation +replaceSpecialStrings :: Text -> Text +replaceSpecialStrings = + let expand c = case c of + '\x00ad' -> "\\-" + '\x2013' -> "--" + '\x2014' -> "---" + '\x2019' -> "'" + '\x2026' -> "..." + _ -> T.singleton c + in T.concatMap expand + -- | Escape special characters for Org. escapeString :: Text -> Doc Text escapeString t | T.all isAlphaNum t = literal t | otherwise = mconcat $ map escChar (T.unpack t) where - escChar '\x2013' = "--" - escChar '\x2014' = "---" - escChar '\x2019' = "'" - escChar '\x2026' = "..." - escChar c - -- escape special chars with ZERO WIDTH SPACE as org manual suggests - | c == '*' || c == '#' || c == '|' = afterBreak "\x200B" <> char c - | otherwise = char c + -- escape special chars with ZERO WIDTH SPACE as org manual suggests + escChar c = if c == '*' || c == '#' || c == '|' + then afterBreak "\x200B" <> char c + else char c isRawFormat :: Format -> Bool isRawFormat f = @@ -488,10 +507,18 @@ inlineToOrg (SmallCaps lst) = inlineListToOrg lst inlineToOrg (Quoted SingleQuote lst) = do contents <- inlineListToOrg lst - return $ "'" <> contents <> "'" + opts <- gets stOptions + return $ + if isEnabled Ext_smart opts || isEnabled Ext_smart_quotes opts + then "'" <> contents <> "'" + else "‘" <> contents <> "’" inlineToOrg (Quoted DoubleQuote lst) = do contents <- inlineListToOrg lst - return $ "\"" <> contents <> "\"" + opts <- gets stOptions + return $ + if isEnabled Ext_smart opts || isEnabled Ext_smart_quotes opts + then "\"" <> contents <> "\"" + else "“" <> contents <> "”" inlineToOrg (Cite cs lst) = do opts <- gets stOptions if isEnabled Ext_citations opts @@ -522,7 +549,12 @@ return $ "[cite" <> sty <> ":" <> citeItems <> "]" else inlineListToOrg lst inlineToOrg (Code _ str) = return $ "=" <> literal str <> "=" -inlineToOrg (Str str) = return $ escapeString str +inlineToOrg (Str str) = do + opts <- gets stOptions + let str' = if isEnabled Ext_smart opts || isEnabled Ext_special_strings opts + then replaceSpecialStrings str + else str + return $ escapeString str' inlineToOrg (Math t str) = do modify $ \st -> st{ stHasMath = True } return $ if t == InlineMath @@ -547,7 +579,7 @@ case txt of [Str x] | escapeURI x == src -> -- autolink return $ "[[" <> literal (orgPath x) <> "]]" - _ -> do contents <- inlineListToOrg txt + _ -> do contents <- nowrap <$> inlineListToOrg txt return $ "[[" <> literal (orgPath src) <> "][" <> contents <> "]]" inlineToOrg (Image _ _ (source, _)) = return $ "[[" <> literal (orgPath source) <> "]]" diff --git a/src/Text/Pandoc/Writers/Powerpoint/Output.hs b/src/Text/Pandoc/Writers/Powerpoint/Output.hs --- a/src/Text/Pandoc/Writers/Powerpoint/Output.hs +++ b/src/Text/Pandoc/Writers/Powerpoint/Output.hs @@ -915,6 +915,7 @@ Just Emf -> Just ".emf" Just Tiff -> Just ".tiff" Just Webp -> Just ".webp" + Just Avif -> Just ".avif" Nothing -> Nothing let newGlobalId = fromMaybe (maxGlobalId + 1) (M.lookup fp globalIds) diff --git a/src/Text/Pandoc/Writers/Powerpoint/Presentation.hs b/src/Text/Pandoc/Writers/Powerpoint/Presentation.hs --- a/src/Text/Pandoc/Writers/Powerpoint/Presentation.hs +++ b/src/Text/Pandoc/Writers/Powerpoint/Presentation.hs @@ -460,15 +460,17 @@ , pPropIndent = Just 0 } , envRunProps = (envRunProps r){rPropCode = True}}) $ do - mbSty <- writerHighlightStyle <$> asks envOpts + highlightOpt <- writerHighlightMethod <$> asks envOpts synMap <- writerSyntaxMap <$> asks envOpts - case mbSty of - Just sty -> - case highlight synMap (formatSourceLines sty) attr str of - Right pElems -> do pPropsNew <- asks envParaProps - return [Paragraph pPropsNew pElems] - Left _ -> blockToParagraphs $ Para [Str str] - Nothing -> blockToParagraphs $ Para [Str str] + let highlightWithStyle style = do + case highlight synMap (formatSourceLines style) attr str of + Right pElems -> do pPropsNew <- asks envParaProps + return [Paragraph pPropsNew pElems] + Left _ -> blockToParagraphs $ Para [Str str] + case highlightOpt of + Skylighting sty -> highlightWithStyle sty + DefaultHighlighting -> highlightWithStyle defaultStyle + _ -> blockToParagraphs $ Para [Str str] -- We can't yet do incremental lists, but we should render a -- (BlockQuote List) as a list to maintain compatibility with other -- formats. diff --git a/src/Text/Pandoc/Writers/Shared.hs b/src/Text/Pandoc/Writers/Shared.hs --- a/src/Text/Pandoc/Writers/Shared.hs +++ b/src/Text/Pandoc/Writers/Shared.hs @@ -568,6 +568,8 @@ -- | Retrieve the metadata value for a given @key@ -- and extract blocks. +-- +-- Note that an empty list is returned for maps, lists, and booleans. lookupMetaBlocks :: Text -> Meta -> [Block] lookupMetaBlocks key meta = case lookupMeta key meta of @@ -578,6 +580,8 @@ -- | Retrieve the metadata value for a given @key@ -- and extract inlines. +-- +-- Note that an empty list is returned for maps and lists. lookupMetaInlines :: Text -> Meta -> [Inline] lookupMetaInlines key meta = case lookupMeta key meta of @@ -589,6 +593,8 @@ -- | Retrieve the metadata value for a given @key@ -- and convert to String. +-- +-- Note that an empty list is returned for maps, lists, and booleans. lookupMetaString :: Text -> Meta -> Text lookupMetaString key meta = case lookupMeta key meta of diff --git a/src/Text/Pandoc/Writers/Typst.hs b/src/Text/Pandoc/Writers/Typst.hs --- a/src/Text/Pandoc/Writers/Typst.hs +++ b/src/Text/Pandoc/Writers/Typst.hs @@ -1,5 +1,6 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE BangPatterns #-} +{-# LANGUAGE LambdaCase #-} {-# LANGUAGE ScopedTypeVariables #-} {- | Module : Text.Pandoc.Writers.Typst @@ -28,9 +29,12 @@ import Network.URI (unEscapeString) import qualified Data.Text as T import Control.Monad.State ( StateT, evalStateT, gets, modify ) -import Text.Pandoc.Writers.Shared ( metaToContext, defField, resetField, - lookupMetaString ) +import Text.Pandoc.Writers.Shared ( lookupMetaInlines, lookupMetaString, + metaToContext, defField, resetField, + setupTranslations ) import Text.Pandoc.Shared (isTightList, orderedListMarkers, tshow) +import Text.Pandoc.Translations (Term(Abstract), translateTerm) +import Text.Pandoc.Walk (query) import Text.Pandoc.Writers.Math (convertMath) import qualified Text.TeXMath as TM import Text.DocLayout @@ -38,8 +42,9 @@ import Text.Pandoc.Extensions (Extension(..)) import Text.Collate.Lang (Lang(..), parseLang) import Text.Printf (printf) -import Data.Char (isAlphaNum, isDigit) +import Data.Char (isDigit) import Data.Maybe (fromMaybe) +import Unicode.Char (isXIDContinue) -- | Convert Pandoc to Typst. writeTypst :: PandocMonad m => WriterOptions -> Pandoc -> m Text @@ -64,18 +69,27 @@ let colwidth = if writerWrapText options == WrapAuto then Just $ writerColumns options else Nothing + setupTranslations meta metadata <- metaToContext options blocksToTypst (fmap chomp . inlinesToTypst) meta main <- blocksToTypst blocks + abstractTitle <- translateTerm Abstract let toPosition :: CaptionPosition -> Text toPosition CaptionAbove = "top" toPosition CaptionBelow = "bottom" + let nociteIds = query (\case + Cite cs _ -> map citationId cs + _ -> []) + $ lookupMetaInlines "nocite" meta + let context = defField "body" main $ defField "toc" (writerTableOfContents options) $ (if isEnabled Ext_citations options then defField "citations" True + . defField "nocite-ids" (filter (/= "*") nociteIds) + . defField "full-bibliography" ("*" `elem` nociteIds) else id) $ (case lookupMetaString "lang" meta of "" -> id @@ -87,6 +101,7 @@ maybe id (resetField "region") (langRegion l)) $ defField "csl" (lookupMetaString "citation-style" meta) -- #10661 $ defField "smart" (isEnabled Ext_smart options) + $ defField "abstract-title" abstractTitle $ defField "toc-depth" (tshow $ writerTOCDepth options) $ defField "figure-caption-position" (toPosition $ writerFigureCaptionPosition options) @@ -94,9 +109,9 @@ (toPosition $ writerTableCaptionPosition options) $ defField "page-numbering" ("1" :: Text) $ (if writerNumberSections options - then defField "numbering" ("1.1.1.1.1" :: Text) + then defField "section-numbering" ("1.1.1.1.1" :: Text) else id) - $ metadata + metadata return $ render colwidth $ case writerTemplate options of Nothing -> main @@ -107,15 +122,15 @@ where go (k,v) = case T.splitOn ":" k of - "typst":"text":x:[] -> second ((x,v):) - "typst":x:[] -> first ((x,v):) + ["typst", "text", x] -> second ((x,v):) + ["typst", x] -> first ((x,v):) _ -> id formatTypstProp :: (Text, Text) -> Text formatTypstProp (k,v) = k <> ": " <> v toTypstPropsListSep :: [(Text, Text)] -> Doc Text -toTypstPropsListSep = hsep . intersperse "," . (map $ literal . formatTypstProp) +toTypstPropsListSep = hsep . intersperse "," . map (literal . formatTypstProp) toTypstPropsListTerm :: [(Text, Text)] -> Doc Text toTypstPropsListTerm [] = "" @@ -147,12 +162,9 @@ blockToTypst :: PandocMonad m => Block -> TW m (Doc Text) blockToTypst block = case block of - Plain inlines -> do - opts <- gets stOptions - inlinesToTypst (addLineStartEscapes opts inlines) + Plain inlines -> inlinesToTypst inlines Para inlines -> do - opts <- gets stOptions - ($$ blankline) <$> inlinesToTypst (addLineStartEscapes opts inlines) + ($$ blankline) <$> inlinesToTypst inlines Header level (ident,cls,_) inlines -> do contents <- inlinesToTypst inlines let lab = toLabel FreestandingLabel ident @@ -342,9 +354,17 @@ Div (ident,_,kvs) blocks -> do let lab = toLabel FreestandingLabel ident let (typstAttrs,typstTextAttrs) = pickTypstAttrs kvs + -- Handle lang attribute for Div elements + let langAttrs = case lookup "lang" kvs of + Nothing -> [] + Just lang -> case parseLang lang of + Left _ -> [] + Right l -> [("lang", + tshow (langLanguage l))] + let allTypstTextAttrs = typstTextAttrs ++ langAttrs contents <- blocksToTypst blocks return $ "#block" <> toTypstPropsListParens typstAttrs <> "[" - $$ toTypstPoundSetText typstTextAttrs <> contents + $$ toTypstPoundSetText allTypstTextAttrs <> contents $$ ("]" <+> lab) defListItemToTypst :: PandocMonad m => ([Inline], [[Block]]) -> TW m (Doc Text) @@ -483,23 +503,17 @@ textstyle :: PandocMonad m => Doc Text -> [Inline] -> TW m (Doc Text) textstyle s inlines = do - opts <- gets stOptions - (<> endCode) . (s <>) . brackets - <$> inlinesToTypst (addLineStartEscapes opts inlines) + (<> endCode) . (s <>) . brackets . fixInitialAfterBreakEscape + <$> inlinesToTypst inlines -addLineStartEscapes :: WriterOptions -> [Inline] -> [Inline] -addLineStartEscapes opts = go True - where - go True (Str t : xs) - | isOrderedListMarker t = RawInline "typst" "\\" : Str t : go False xs - | Just (c, t') <- T.uncons t - , needsEscapeAtLineStart c - , T.null t' = RawInline "typst" "\\" : Str t : go False xs - go _ (SoftBreak : xs) - | writerWrapText opts == WrapPreserve = SoftBreak : go True xs - go _ (LineBreak : xs) = LineBreak : go True xs - go _ (x : xs) = x : go False xs - go _ [] = [] +fixInitialAfterBreakEscape :: Doc Text -> Doc Text +fixInitialAfterBreakEscape (Concat x y) = + Concat (fixInitialAfterBreakEscape x) y +-- make an initial AfterBreak escape unconditional (it will be rendered +-- in a block [..] and there won't be an actual break to trigger it, but +-- typst still needs the escape) +fixInitialAfterBreakEscape (AfterBreak "\\") = Text 1 "\\" +fixInitialAfterBreakEscape x = x isOrderedListMarker :: Text -> Bool isOrderedListMarker t = not (T.null ds) && rest == "." @@ -509,7 +523,7 @@ escapeTypst smart context t = (case T.uncons t of Just (c, _) - | needsEscapeAtLineStart c + | needsEscapeAtLineStart c || isOrderedListMarker t -> afterBreak "\\" _ -> mempty) <> (literal (T.replace "//" "\\/\\/" @@ -570,7 +584,7 @@ ident' = T.pack $ unEscapeString $ T.unpack ident isIdentChar :: Char -> Bool -isIdentChar c = isAlphaNum c || c == '_' || c == '-' || c == '.' || c == ':' +isIdentChar c = isXIDContinue c || c == '_' || c == '-' || c == '.' || c == ':' toCite :: PandocMonad m => Citation -> TW m (Doc Text) toCite cite = do @@ -596,7 +610,10 @@ [] -> pure mempty suff -> (", supplement: " <>) . brackets <$> inlinesToTypst (eatComma suff) - pure $ "#cite" <> parens (label <> form <> suppl) <> endCode + pure $ (if citationMode cite == SuppressAuthor -- see #11044 + then parens + else id) + $ "#cite" <> parens (label <> form <> suppl) <> endCode doubleQuoted :: Text -> Doc Text doubleQuoted = doubleQuotes . literal . escape diff --git a/src/Text/Pandoc/Writers/XML.hs b/src/Text/Pandoc/Writers/XML.hs new file mode 100644 --- /dev/null +++ b/src/Text/Pandoc/Writers/XML.hs @@ -0,0 +1,365 @@ +{-# LANGUAGE FlexibleContexts #-} +{-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE ScopedTypeVariables #-} + +-- | +-- Module : Text.Pandoc.Writers.XML +-- Copyright : Copyright (C) 2025- Massimiliano Farinella and John MacFarlane +-- License : GNU GPL, version 2 or above +-- +-- Maintainer : Massimiliano Farinella +-- Stability : WIP +-- Portability : portable +-- +-- Conversion of 'Pandoc' documents to (pandoc specific) xml markup. +module Text.Pandoc.Writers.XML (writeXML) where + +import Data.Map (Map, toList) +import Data.Maybe (mapMaybe) +import qualified Data.Text as T +import Data.Version (versionBranch) +import Text.Pandoc.Class.PandocMonad (PandocMonad) +import Text.Pandoc.Definition +import Text.Pandoc.Options (WriterOptions (..)) +import Text.Pandoc.XML.Light +import qualified Text.Pandoc.XML.Light as XML +import Text.Pandoc.XMLFormat +import Text.XML.Light (xml_header) + +type PandocAttr = Text.Pandoc.Definition.Attr + +writeXML :: (PandocMonad m) => WriterOptions -> Pandoc -> m T.Text +writeXML _ doc = do + return $ pandocToXmlText doc + +text_node :: T.Text -> Content +text_node text = Text (CData CDataText text Nothing) + +emptyElement :: T.Text -> Element +emptyElement tag = + Element + { elName = unqual tag, + elAttribs = [], + elContent = [], + elLine = Nothing + } + +elementWithContents :: T.Text -> [Content] -> Element +elementWithContents tag contents = + Element + { elName = unqual tag, + elAttribs = [], + elContent = contents, + elLine = Nothing + } + +elementWithAttributes :: T.Text -> [XML.Attr] -> Element +elementWithAttributes tag attributes = + Element + { elName = unqual tag, + elAttribs = attributes, + elContent = [], + elLine = Nothing + } + +elementWithAttrAndContents :: T.Text -> PandocAttr -> [Content] -> Element +elementWithAttrAndContents tag attr contents = addAttrAttributes attr $ elementWithContents tag contents + +asBlockOfInlines :: Element -> [Content] +asBlockOfInlines el = [Elem el, text_node "\n"] + +asBlockOfBlocks :: Element -> [Content] +asBlockOfBlocks el = [Elem newline_before_first, newline] + where + newline = text_node "\n" + newline_before_first = if null (elContent el) then el else prependContents [newline] el + +itemName :: (Show a) => a -> T.Text +itemName a = T.pack $ takeWhile (/= ' ') (show a) + +intAsText :: Int -> T.Text +intAsText i = T.pack $ show i + +itemAsEmptyElement :: (Show a) => a -> Element +itemAsEmptyElement item = emptyElement $ itemName item + +pandocToXmlText :: Pandoc -> T.Text +pandocToXmlText (Pandoc (Meta meta) blocks) = with_header . with_blocks . with_meta . with_version $ el + where + el = prependContents [text_node "\n"] $ emptyElement "Pandoc" + with_version = addAttribute atNameApiVersion (T.intercalate "," $ map (T.pack . show) $ versionBranch pandocTypesVersion) + with_meta = appendContents (metaMapToXML meta "meta") + with_blocks = appendContents (asBlockOfBlocks $ elementWithContents "blocks" $ blocksToXML blocks) + with_header :: Element -> T.Text + with_header e = T.concat [T.pack xml_header, "\n", showElement e] + +metaMapToXML :: Map T.Text MetaValue -> T.Text -> [Content] +metaMapToXML mmap tag = asBlockOfBlocks $ elementWithContents tag entries + where + entries = concatMap to_entry $ toList mmap + to_entry :: (T.Text, MetaValue) -> [Content] + to_entry (text, metavalue) = asBlockOfBlocks with_key + where + entry = elementWithContents tgNameMetaMapEntry $ metaValueToXML metavalue + with_key = addAttribute atNameMetaMapEntryKey text entry + +metaValueToXML :: MetaValue -> [Content] +metaValueToXML value = + let name = itemName value + el = itemAsEmptyElement value + in case (value) of + MetaBool b -> asBlockOfInlines $ addAttribute atNameMetaBoolValue bool_value el + where + bool_value = if b then "true" else "false" + MetaString s -> asBlockOfInlines $ appendContents [text_node s] el + MetaInlines inlines -> asBlockOfInlines $ appendContents (inlinesToXML inlines) el + MetaBlocks blocks -> asBlockOfBlocks $ appendContents (blocksToXML blocks) el + MetaList items -> asBlockOfBlocks $ appendContents (concatMap metaValueToXML items) el + MetaMap mm -> metaMapToXML mm name + +blocksToXML :: [Block] -> [Content] +blocksToXML blocks = concatMap blockToXML blocks + +inlinesToXML :: [Inline] -> [Content] +inlinesToXML inlines = concatMap inlineContentToContents (ilsToIlsContent inlines []) + +data InlineContent + = NormalInline Inline + | ElSpace Int + | ElStr T.Text + +ilsToIlsContent :: [Inline] -> [InlineContent] -> [InlineContent] +ilsToIlsContent (Space : xs) [] = ilsToIlsContent xs [ElSpace 1] +ilsToIlsContent (Space : xs) (NormalInline Space : cs) = ilsToIlsContent xs (ElSpace 2 : cs) +ilsToIlsContent (Space : xs) (ElSpace n : cs) = ilsToIlsContent xs (ElSpace (n + 1) : cs) +-- empty Str are always encoded as +ilsToIlsContent (Str "" : xs) ilct = ilsToIlsContent xs (ElStr "" : ilct) +-- Str s1, Str s2 -> s1 +ilsToIlsContent (Str s2 : xs) (NormalInline str1@(Str _) : ilct) = ilsToIlsContent xs (ElStr s2 : NormalInline str1 : ilct) +-- +ilsToIlsContent (Str s : xs) ilct = + if T.any (== ' ') s + then ilsToIlsContent xs (ElStr s : ilct) + else ilsToIlsContent xs (NormalInline (Str s) : ilct) +ilsToIlsContent (x : xs) ilct = ilsToIlsContent xs (NormalInline x : ilct) +ilsToIlsContent [] ilct = reverse $ lastSpaceAsElem ilct + where + lastSpaceAsElem :: [InlineContent] -> [InlineContent] + lastSpaceAsElem (NormalInline Space : xs) = ElSpace 1 : xs + lastSpaceAsElem ilcts = ilcts + +inlineContentToContents :: InlineContent -> [Content] +inlineContentToContents (NormalInline il) = inlineToXML il +inlineContentToContents (ElSpace 1) = [Elem $ emptyElement "Space"] +inlineContentToContents (ElSpace n) = [Elem $ addAttribute atNameSpaceCount (intAsText n) (emptyElement "Space")] +inlineContentToContents (ElStr "") = [Elem $ emptyElement "Str"] +inlineContentToContents (ElStr s) = [Elem $ addAttribute atNameStrContent s (emptyElement "Str")] + +asContents :: Element -> [Content] +asContents el = [Elem el] + +wrapBlocks :: T.Text -> [Block] -> [Content] +wrapBlocks tag blocks = asBlockOfBlocks $ elementWithContents tag $ blocksToXML blocks + +wrapArrayOfBlocks :: T.Text -> [[Block]] -> [Content] +wrapArrayOfBlocks tag array = concatMap (wrapBlocks tag) array + +-- wrapInlines :: T.Text -> [Inline] -> [Content] +-- wrapInlines tag inlines = asBlockOfInlines $ element_with_contents tag $ inlinesToXML inlines + +blockToXML :: Block -> [Content] +blockToXML block = + let el = itemAsEmptyElement block + in case (block) of + Para inlines -> asBlockOfInlines $ appendContents (inlinesToXML inlines) el + Header level (idn, cls, attrs) inlines -> asBlockOfInlines $ appendContents (inlinesToXML inlines) with_attr + where + with_attr = addAttrAttributes (idn, cls, attrs ++ [(atNameLevel, intAsText level)]) el + Plain inlines -> asBlockOfInlines $ appendContents (inlinesToXML inlines) el + Div attr blocks -> asBlockOfBlocks $ appendContents (blocksToXML blocks) with_attr + where + with_attr = addAttrAttributes attr el + BulletList items -> asBlockOfBlocks $ appendContents (wrapArrayOfBlocks tgNameListItem items) el + OrderedList (start, style, delim) items -> asBlockOfBlocks $ with_contents . with_attrs $ el + where + with_attrs = + addAttributes + ( validAttributes + [ (atNameStart, intAsText start), + (atNameNumberStyle, itemName style), + (atNameNumberDelim, itemName delim) + ] + ) + with_contents = appendContents (wrapArrayOfBlocks tgNameListItem items) + BlockQuote blocks -> asBlockOfBlocks $ appendContents (blocksToXML blocks) el + HorizontalRule -> asBlockOfInlines el + CodeBlock attr text -> asBlockOfInlines $ with_contents . with_attr $ el + where + with_contents = appendContents [text_node text] + with_attr = addAttrAttributes attr + LineBlock lins -> asBlockOfBlocks $ appendContents (concatMap wrapInlines lins) el + where + wrapInlines inlines = asContents $ appendContents (inlinesToXML inlines) $ emptyElement tgNameLineItem + Table attr caption colspecs thead tbodies tfoot -> asBlockOfBlocks $ with_foot . with_bodies . with_head . with_colspecs . with_caption . with_attr $ el + where + with_attr = addAttrAttributes attr + with_caption = appendContents (captionToXML caption) + with_colspecs = appendContents (colSpecsToXML colspecs) + with_head = appendContents (tableHeadToXML thead) + with_bodies = appendContents (concatMap tableBodyToXML tbodies) + with_foot = appendContents (tableFootToXML tfoot) + Figure attr caption blocks -> asBlockOfBlocks $ with_contents . with_caption . with_attr $ el + where + with_attr = addAttrAttributes attr + with_caption = appendContents (captionToXML caption) + with_contents = appendContents (blocksToXML blocks) + RawBlock (Format format) text -> asContents $ appendContents [text_node text] raw + where + raw = addAttribute atNameFormat format el + DefinitionList items -> asBlockOfBlocks $ appendContents (map definitionListItemToXML items) el + +inlineToXML :: Inline -> [Content] +inlineToXML inline = + let el = itemAsEmptyElement inline + wrapInlines inlines = asContents $ appendContents (inlinesToXML inlines) el + in case (inline) of + Space -> [text_node " "] + Str s -> [text_node s] + Emph inlines -> wrapInlines inlines + Strong inlines -> wrapInlines inlines + Quoted quote_type inlines -> asContents $ appendContents (inlinesToXML inlines) quoted + where + quoted = addAttribute atNameQuoteType (itemName quote_type) el + Underline inlines -> wrapInlines inlines + Strikeout inlines -> wrapInlines inlines + SmallCaps inlines -> wrapInlines inlines + Superscript inlines -> wrapInlines inlines + Subscript inlines -> wrapInlines inlines + SoftBreak -> asContents el + LineBreak -> asContents el + Span attr inlines -> asContents $ appendContents (inlinesToXML inlines) with_attr + where + with_attr = addAttrAttributes attr el + Link (idn, cls, attrs) inlines (url, title) -> asContents $ appendContents (inlinesToXML inlines) with_attr + where + with_attr = addAttrAttributes (idn, cls, attrs ++ [(atNameLinkUrl, url), (atNameTitle, title)]) el + Image (idn, cls, attrs) inlines (url, title) -> asContents $ appendContents (inlinesToXML inlines) with_attr + where + with_attr = addAttrAttributes (idn, cls, attrs ++ [(atNameImageUrl, url), (atNameTitle, title)]) el + RawInline (Format format) text -> asContents $ appendContents [text_node text] raw + where + raw = addAttribute atNameFormat format el + Math math_type text -> asContents $ appendContents [text_node text] math + where + math = addAttribute atNameMathType (itemName math_type) el + Code attr text -> asContents $ appendContents [text_node text] with_attr + where + with_attr = addAttrAttributes attr el + Note blocks -> asContents $ appendContents (blocksToXML blocks) el + Cite citations inlines -> asContents $ appendContents (inlinesToXML inlines) with_citations + where + with_citations = addCitations citations el + +-- TODO: don't let an attribute overwrite id or class +maybeAttribute :: (T.Text, T.Text) -> Maybe XML.Attr +maybeAttribute (_, "") = Nothing +maybeAttribute ("", _) = Nothing +maybeAttribute (name, value) = Just $ XML.Attr (unqual name) value + +validAttributes :: [(T.Text, T.Text)] -> [XML.Attr] +validAttributes pairs = mapMaybe maybeAttribute pairs + +appendContents :: [Content] -> Element -> Element +appendContents newContents el = el {elContent = (elContent el) ++ newContents} + +prependContents :: [Content] -> Element -> Element +prependContents newContents el = el {elContent = newContents ++ (elContent el)} + +addAttributes :: [XML.Attr] -> Element -> Element +addAttributes newAttrs el = el {elAttribs = newAttrs ++ elAttribs el} + +addAttribute :: T.Text -> T.Text -> Element -> Element +addAttribute attr_name attr_value el = el {elAttribs = new_attr : elAttribs el} + where + new_attr = XML.Attr (unqual attr_name) attr_value + +addAttrAttributes :: PandocAttr -> Element -> Element +addAttrAttributes (identifier, classes, attributes) el = addAttributes attrs' el + where + attrs' = mapMaybe maybeAttribute (("id", identifier) : ("class", T.intercalate " " classes) : attributes) + +addCitations :: [Citation] -> Element -> Element +addCitations citations el = appendContents [Elem $ elementWithContents tgNameCitations $ (text_node "\n") : concatMap citation_to_elem citations] el + where + citation_to_elem :: Citation -> [Content] + citation_to_elem citation = asBlockOfInlines with_suffix + where + cit_elem = elementWithAttributes (itemName citation) attrs + prefix = citationPrefix citation + suffix = citationSuffix citation + with_prefix = + if null prefix + then cit_elem + else appendContents [Elem $ elementWithContents tgNameCitationPrefix $ inlinesToXML prefix] cit_elem + with_suffix = + if null suffix + then with_prefix + else appendContents [Elem $ elementWithContents tgNameCitationSuffix $ inlinesToXML suffix] with_prefix + attrs = + map + (\(n, v) -> XML.Attr (unqual n) v) + [ ("id", citationId citation), + (atNameCitationMode, T.pack $ show $ citationMode citation), + (atNameCitationNoteNum, intAsText $ citationNoteNum citation), + (atNameCitationHash, intAsText $ citationHash citation) + ] + +definitionListItemToXML :: ([Inline], [[Block]]) -> Content +definitionListItemToXML (inlines, defs) = Elem $ elementWithContents tgNameDefListItem $ term ++ wrapArrayOfBlocks tgNameDefListDef defs + where + term = asBlockOfInlines $ appendContents (inlinesToXML inlines) $ emptyElement tgNameDefListTerm + +captionToXML :: Caption -> [Content] +captionToXML (Caption short blocks) = asBlockOfBlocks with_short_caption + where + el = elementWithContents "Caption" $ blocksToXML blocks + with_short_caption = case (short) of + Just inlines -> prependContents (asBlockOfInlines $ elementWithContents tgNameShortCaption $ inlinesToXML inlines) el + _ -> el + +colSpecToXML :: (Alignment, ColWidth) -> [Content] +colSpecToXML (align, cw) = asBlockOfInlines colspec + where + colspec = elementWithAttributes "ColSpec" $ validAttributes [(atNameAlignment, itemName align), (atNameColWidth, colwidth)] + colwidth = case (cw) of + ColWidth d -> T.pack $ show d + ColWidthDefault -> "0" + +colSpecsToXML :: [(Alignment, ColWidth)] -> [Content] +colSpecsToXML colspecs = asBlockOfBlocks $ elementWithContents tgNameColspecs $ concatMap colSpecToXML colspecs + +tableHeadToXML :: TableHead -> [Content] +tableHeadToXML (TableHead attr rows) = asBlockOfBlocks $ elementWithAttrAndContents "TableHead" attr $ concatMap rowToXML rows + +tableBodyToXML :: TableBody -> [Content] +tableBodyToXML (TableBody (idn, cls, attrs) (RowHeadColumns headcols) hrows brows) = asBlockOfBlocks $ elementWithAttrAndContents "TableBody" attr children + where + attr = (idn, cls, (atNameRowHeadColumns, intAsText headcols) : attrs) + header_rows = asBlockOfBlocks $ elementWithContents tgNameBodyHeader $ concatMap rowToXML hrows + body_rows = asBlockOfBlocks $ elementWithContents tgNameBodyBody $ concatMap rowToXML brows + children = header_rows ++ body_rows + +tableFootToXML :: TableFoot -> [Content] +tableFootToXML (TableFoot attr rows) = asBlockOfBlocks $ elementWithAttrAndContents "TableFoot" attr $ concatMap rowToXML rows + +rowToXML :: Row -> [Content] +rowToXML (Row attr cells) = asBlockOfBlocks $ elementWithAttrAndContents "Row" attr $ concatMap cellToXML cells + +cellToXML :: Cell -> [Content] +cellToXML (Cell (idn, cls, attrs) alignment (RowSpan rowspan) (ColSpan colspan) blocks) = asBlockOfBlocks $ elementWithAttrAndContents "Cell" attr $ blocksToXML blocks + where + with_alignment a = (atNameAlignment, itemName alignment) : a + with_rowspan a = if rowspan > 1 then (atNameRowspan, intAsText rowspan) : a else a + with_colspan a = if colspan > 1 then (atNameColspan, intAsText colspan) : a else a + attrs' = (with_colspan . with_rowspan . with_alignment) attrs + attr = (idn, cls, attrs') diff --git a/src/Text/Pandoc/XMLFormat.hs b/src/Text/Pandoc/XMLFormat.hs new file mode 100644 --- /dev/null +++ b/src/Text/Pandoc/XMLFormat.hs @@ -0,0 +1,188 @@ +{-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE ScopedTypeVariables #-} + +module Text.Pandoc.XMLFormat + ( atNameAlignment, + atNameApiVersion, + atNameCitationHash, + atNameCitationMode, + atNameCitationNoteNum, + atNameColspan, + atNameColWidth, + atNameFormat, + atNameImageUrl, + atNameLevel, + atNameLinkUrl, + atNameMathType, + atNameMetaBoolValue, + atNameMetaMapEntryKey, + atNameNumberDelim, + atNameNumberStyle, + atNameQuoteType, + atNameRowHeadColumns, + atNameRowspan, + atNameSpaceCount, + atNameStart, + atNameStrContent, + atNameTitle, + tgNameBodyBody, + tgNameBodyHeader, + tgNameCitations, + tgNameCitationPrefix, + tgNameCitationSuffix, + tgNameColspecs, + tgNameDefListDef, + tgNameDefListItem, + tgNameDefListTerm, + tgNameLineItem, + tgNameListItem, + tgNameMetaMapEntry, + tgNameShortCaption, + ) +where + +import Data.Text (Text) + +-- the attribute carrying the API version of pandoc types in the main Pandoc element +atNameApiVersion :: Text +atNameApiVersion = "api-version" + +-- the element of a or entry +tgNameMetaMapEntry :: Text +tgNameMetaMapEntry = "entry" + +-- the attribute carrying the key name of a or entry +atNameMetaMapEntryKey :: Text +atNameMetaMapEntryKey = "key" + +-- the attribute carrying the boolean value ("true" or "false") of a MetaBool +atNameMetaBoolValue :: Text +atNameMetaBoolValue = "value" + +-- level of a Header +atNameLevel :: Text +atNameLevel = "level" + +-- start number of an OrderedList +atNameStart :: Text +atNameStart = "start" + +-- number delimiter of an OrderedList +atNameNumberDelim :: Text +atNameNumberDelim = "number-delim" + +-- number style of an OrderedList +atNameNumberStyle :: Text +atNameNumberStyle = "number-style" + +-- target title in Image and Link +atNameTitle :: Text +atNameTitle = "title" + +-- target url in Image +atNameImageUrl :: Text +atNameImageUrl = "src" + +-- target url in Link +atNameLinkUrl :: Text +atNameLinkUrl = "href" + +-- QuoteType of a Quoted +atNameQuoteType :: Text +atNameQuoteType = "quote-type" + +-- MathType of a Math +atNameMathType :: Text +atNameMathType = "math-type" + +-- format of a RawInline or a RawBlock +atNameFormat :: Text +atNameFormat = "format" + +-- alignment attribute in a ColSpec or in a Cell +atNameAlignment :: Text +atNameAlignment = "alignment" + +-- ColWidth attribute in a ColSpec +atNameColWidth :: Text +atNameColWidth = "col-width" + +-- RowHeadColumns attribute in a TableBody +atNameRowHeadColumns :: Text +atNameRowHeadColumns = "row-head-columns" + +-- RowSpan attribute in a Cell +atNameRowspan :: Text +atNameRowspan = "row-span" + +-- ColSpan attribute in a Cell +atNameColspan :: Text +atNameColspan = "col-span" + +-- the citationMode of a Citation +atNameCitationMode :: Text +atNameCitationMode = "mode" + +-- the citationHash of a Citation +atNameCitationHash :: Text +atNameCitationHash = "hash" + +-- the citationNoteNum of a Citation +atNameCitationNoteNum :: Text +atNameCitationNoteNum = "note-num" + +-- the number of consecutive spaces of the element +atNameSpaceCount :: Text +atNameSpaceCount = "count" + +-- the content of the element +atNameStrContent :: Text +atNameStrContent = "content" + +-- container of Citation elements in Cite inlines +tgNameCitations :: Text +tgNameCitations = "citations" + +-- element around the prefix inlines of a Citation +tgNameCitationPrefix :: Text +tgNameCitationPrefix = "prefix" + +-- element around the suffix inlines of a Citation +tgNameCitationSuffix :: Text +tgNameCitationSuffix = "suffix" + +-- list item for BulletList and OrderedList +tgNameListItem :: Text +tgNameListItem = "item" + +-- list item for DefinitionList +tgNameDefListItem :: Text +tgNameDefListItem = "item" + +-- element around the inlines of the term of a DefinitionList item +tgNameDefListTerm :: Text +tgNameDefListTerm = "term" + +-- element around the blocks of a definition in a DefinitionList item +tgNameDefListDef :: Text +tgNameDefListDef = "def" + +-- optional element of the ShortCaption +tgNameShortCaption :: Text +tgNameShortCaption = "ShortCaption" + +-- element around the ColSpec of a Table +tgNameColspecs :: Text +tgNameColspecs = "colspecs" + +-- element around the header rows of a TableBody +tgNameBodyHeader :: Text +tgNameBodyHeader = "header" + +-- element around the body rows of a TableBody +tgNameBodyBody :: Text +tgNameBodyBody = "body" + +-- element around the inlines of a line in a LineBlock +tgNameLineItem :: Text +tgNameLineItem = "line" diff --git a/test/Tests/Readers/HTML.hs b/test/Tests/Readers/HTML.hs --- a/test/Tests/Readers/HTML.hs +++ b/test/Tests/Readers/HTML.hs @@ -130,10 +130,10 @@ [ test html "attributes in pre > code element" $ "
\nprint('hi')\n
" =?> - codeBlockWith ("a", ["python"], []) "print('hi')" + codeBlockWith ("a", ["python"], []) "\nprint('hi')" , test html "attributes in pre take precedence" $ - "
\nprint('hi mom!')\n
" + "
print('hi mom!')\n
" =?> codeBlockWith ("c", [], []) "print('hi mom!')" ] diff --git a/test/Tests/Readers/Markdown.hs b/test/Tests/Readers/Markdown.hs --- a/test/Tests/Readers/Markdown.hs +++ b/test/Tests/Readers/Markdown.hs @@ -29,10 +29,6 @@ markdownSmart = purely $ readMarkdown def { readerExtensions = enableExtension Ext_smart pandocExtensions } -markdownCDL :: Text -> Pandoc -markdownCDL = purely $ readMarkdown def { readerExtensions = enableExtension - Ext_compact_definition_lists pandocExtensions } - markdownGH :: Text -> Pandoc markdownGH = purely $ readMarkdown def {readerExtensions = enableExtension Ext_wikilinks_title_before_pipe githubMarkdownExtensions } @@ -461,14 +457,14 @@ , "blank space before first def" =: "foo1\n\n : bar\n\nfoo2\n\n : bar2\n : bar3\n" =?> definitionList [ (text "foo1", [para (text "bar")]) - , (text "foo2", [para (text "bar2"), - plain (text "bar3")]) + , (text "foo2", [plain (text "bar2"), + para (text "bar3")]) ] , "blank space before second def" =: "foo1\n : bar\n\nfoo2\n : bar2\n\n : bar3\n" =?> definitionList [ (text "foo1", [plain (text "bar")]) , (text "foo2", [plain (text "bar2"), - para (text "bar3")]) + plain (text "bar3")]) ] , "laziness" =: "foo1\n : bar\nbaz\n : bar2\n" =?> @@ -478,8 +474,8 @@ ] , "no blank space before first of two paragraphs" =: "foo1\n : bar\n\n baz\n" =?> - definitionList [ (text "foo1", [para (text "bar") <> - para (text "baz")]) + definitionList [ (text "foo1", [plain (text "bar") <> + plain (text "baz")]) ] , "first line not indented" =: "foo\n: bar\n" =?> @@ -492,14 +488,6 @@ divWith nullAttr (definitionList [ (text "foo", [bulletList [plain (text "bar")]]) ]) ] - , testGroup "+compact_definition_lists" - [ test markdownCDL "basic compact list" $ - "foo1\n: bar\n baz\nfoo2\n: bar2\n" =?> - definitionList [ (text "foo1", [plain (text "bar" <> softbreak <> - text "baz")]) - , (text "foo2", [plain (text "bar2")]) - ] - ] , testGroup "lists" [ "issue #1154" =: " -
\n first div breaks\n
\n\n \n\n
\n with this div too.\n
\n" @@ -590,4 +578,11 @@ (str "@cita" <> space <> str "[foo]") ) ] + , testGroup "figures" + [ "latex placement" =: + "![caption](img.jpg){latex-placement=\"htbp\" alt=\"alt text\"}" =?> + figureWith ("", [], [("latex-placement", "htbp")]) + (simpleCaption $ plain "caption") + (plain $ image (T.pack "img.jpg") (T.pack "") (text "alt text")) ] + ] diff --git a/test/Tests/Readers/ODT.hs b/test/Tests/Readers/ODT.hs --- a/test/Tests/Readers/ODT.hs +++ b/test/Tests/Readers/ODT.hs @@ -180,6 +180,8 @@ , "referenceToText" , "simpleTable" , "simpleTableWithCaption" + , "simpleTableWithHeader" + , "simpleTableWithMultipleHeaderRows" , "tab" -- , "table" , "textMixedStyles" diff --git a/test/Tests/Readers/Org/Block/Header.hs b/test/Tests/Readers/Org/Block/Header.hs --- a/test/Tests/Readers/Org/Block/Header.hs +++ b/test/Tests/Readers/Org/Block/Header.hs @@ -118,6 +118,13 @@ in headerWith ("compile", [], []) 1 (waiting <> space <> "compile") <> headerWith ("lunch", [], []) 1 (cancelled <> space <> "lunch") <> headerWith ("todo-feature", [], []) 1 (done <> space <> "todo-feature") + + , "Fast access TODO states" =: + T.unlines [ "#+TODO: TODO(t) | DONE(d)" + , "* TODO test" + ] =?> + let todoSpan = spanWith ("", ["todo", "TODO"], []) "TODO" + in headerWith ("test", [], []) 1 (todoSpan <> space <> "test") ] , "Tagged headers" =: diff --git a/test/Tests/Readers/Org/Inline.hs b/test/Tests/Readers/Org/Inline.hs --- a/test/Tests/Readers/Org/Inline.hs +++ b/test/Tests/Readers/Org/Inline.hs @@ -88,6 +88,10 @@ "2^{n-1}" =?> para (str "2" <> superscript "n-1") + , "Superscript-like, but not after string" =: + "a ^caret" =?> + para "a ^caret" + , "Subscript simple expression" =: "a_n" =?> para (str "a" <> subscript "n") @@ -95,6 +99,14 @@ , "Subscript multi char" =: "a_{n+1}" =?> para (str "a" <> subscript "n+1") + + , "Subscript-like, but not after string" =: + "_underscore" =?> + para "_underscore" + + , "Subscript takes precedence before underline" =: + "text_subscript_" =?> + para (str "text" <> subscript "subscript" <> str "_") , "Linebreak" =: "line \\\\ \nbreak" =?> diff --git a/test/Tests/Readers/Org/Inline/Smart.hs b/test/Tests/Readers/Org/Inline/Smart.hs --- a/test/Tests/Readers/Org/Inline/Smart.hs +++ b/test/Tests/Readers/Org/Inline/Smart.hs @@ -47,6 +47,10 @@ ("/foo---/" =?> para (emph "foo—")) + , test orgSmart "Support for shy (soft) hyphen" + ("Ur\\-instinkt" =?> + para "Ur\173instinkt") + , test orgSmart "Single quotes can be followed by emphasized text" ("Singles on the '/meat market/'" =?> para ("Singles on the " <> singleQuoted (emph "meat market"))) diff --git a/test/Tests/Readers/Pod.hs b/test/Tests/Readers/Pod.hs --- a/test/Tests/Readers/Pod.hs +++ b/test/Tests/Readers/Pod.hs @@ -145,6 +145,17 @@ "E" =?> para "»" ] + , testGroup "html" + [ "trade" =: + "E" =?> + para "™" + , "ccaron" =: + "E" =?> + para "č" + , "cent" =: + "E" =?> + para "¢" + ] , testGroup "numeric" [ "decimal" =: "E<162>" =?> @@ -170,6 +181,7 @@ , bogusEntity "0xhh" , bogusEntity "077x" , bogusEntity "0x63 skidoo" + , bogusEntity "trade;" ] ] ] diff --git a/test/Tests/Writers/LaTeX.hs b/test/Tests/Writers/LaTeX.hs --- a/test/Tests/Writers/LaTeX.hs +++ b/test/Tests/Writers/LaTeX.hs @@ -1,7 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} module Tests.Writers.LaTeX (tests) where -import Data.Text (unpack) +import Data.Text (pack, unpack) import Test.Tasty import Test.Tasty.HUnit (HasCallStack) import Tests.Helpers @@ -13,7 +13,7 @@ latex = latexWithOpts def latexListing :: (ToPandoc a) => a -> String -latexListing = latexWithOpts def{ writerListings = True } +latexListing = latexWithOpts def{ writerHighlightMethod = IdiomaticHighlighting } latexWithOpts :: (ToPandoc a) => WriterOptions -> a -> String latexWithOpts opts = unpack . purely (writeLaTeX opts) . toPandoc @@ -241,5 +241,15 @@ , "\\addcontentsline{toc}{paragraph}{header6}" ] ] + ] + , testGroup "figures" + [ "placement" =: + figureWith ("", [], [("latex-placement", "htbp")]) + (simpleCaption $ plain "caption") + (plain $ image (pack "img.jpg") (pack "") (text "alt text")) + =?> + "\\begin{figure}[htbp]\n\\centering\n" + <> "\\pandocbounded{\\includegraphics[keepaspectratio,alt={alt text}]{img.jpg}}\n" + <> "\\caption{caption}\n\\end{figure}" ] ] diff --git a/test/Tests/XML.hs b/test/Tests/XML.hs new file mode 100644 --- /dev/null +++ b/test/Tests/XML.hs @@ -0,0 +1,28 @@ +{-# LANGUAGE OverloadedStrings #-} +{- | +-- Module : Tests.XML +-- Copyright : Copyright (C) 2025- Massimiliano Farinella and John MacFarlane +-- License : GNU GPL, version 2 or above +-- +-- Maintainer : Massimiliano Farinella +-- Stability : WIP +-- Portability : portable +Runs a roundtrip conversion of an AST trough the XML format: +- first from AST to XML (XML Writer), +- then back to AST (XML Reader), +- and checks that the two ASTs are the same +-} +module Tests.XML (tests) where + +import Control.Monad ((>=>)) +import Test.Tasty (TestTree) +import Test.Tasty.QuickCheck +import Tests.Helpers +import Text.Pandoc +import Text.Pandoc.Arbitrary () + +p_xml_roundtrip :: Pandoc -> Bool +p_xml_roundtrip d = d == purely (writeXML def {writerTemplate = Just mempty} >=> readXML def) d + +tests :: [TestTree] +tests = [testProperty "p_xml_roundtrip" p_xml_roundtrip] diff --git a/test/command/10160.md b/test/command/10160.md --- a/test/command/10160.md +++ b/test/command/10160.md @@ -7,5 +7,9 @@ \begin{table}[h] \end{table} ^D -[ Para [ Math DisplayMath "[0,1)" ] ] +[ Para + [ Math + DisplayMath "\\begin{equation}\n [0,1)\n\\end{equation}" + ] +] ``` diff --git a/test/command/10185.md b/test/command/10185.md --- a/test/command/10185.md +++ b/test/command/10185.md @@ -36,7 +36,7 @@ \section{References}\label{references} \protect\phantomsection\label{refs} -\begin{CSLReferences}{1}{0} +\begin{CSLReferences}{1}{1} \bibitem[\citeproctext]{ref-foo} John Doe. n.d. \emph{The Title}. diff --git a/test/command/10338-rst-multiple-header-rows.md b/test/command/10338-rst-multiple-header-rows.md new file mode 100644 --- /dev/null +++ b/test/command/10338-rst-multiple-header-rows.md @@ -0,0 +1,155 @@ +``` +% pandoc -f rst -t native +Multiple Headers +================ + +========== ========= +Header A1 Header A2 +Header B1 Header B2 +========== ========= +body a1 body a1 +body b1 body b2 +========== ========= + +Headless +======== + +========== ========= +body a1 body a1 +body b1 body b2 +========== ========= + +End Section +=========== +^D +[ Header + 1 + ( "multiple-headers" , [] , [] ) + [ Str "Multiple" , Space , Str "Headers" ] +, Table + ( "" , [] , [] ) + (Caption Nothing []) + [ ( AlignDefault , ColWidthDefault ) + , ( AlignDefault , ColWidthDefault ) + ] + (TableHead + ( "" , [] , [] ) + [ Row + ( "" , [] , [] ) + [ Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "Header" , Space , Str "A1" ] ] + , Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "Header" , Space , Str "A2" ] ] + ] + , Row + ( "" , [] , [] ) + [ Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "Header" , Space , Str "B1" ] ] + , Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "Header" , Space , Str "B2" ] ] + ] + ]) + [ TableBody + ( "" , [] , [] ) + (RowHeadColumns 0) + [] + [ Row + ( "" , [] , [] ) + [ Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "body" , Space , Str "a1" ] ] + , Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "body" , Space , Str "a1" ] ] + ] + , Row + ( "" , [] , [] ) + [ Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "body" , Space , Str "b1" ] ] + , Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "body" , Space , Str "b2" ] ] + ] + ] + ] + (TableFoot ( "" , [] , [] ) []) +, Header 1 ( "headless" , [] , [] ) [ Str "Headless" ] +, Table + ( "" , [] , [] ) + (Caption Nothing []) + [ ( AlignDefault , ColWidthDefault ) + , ( AlignDefault , ColWidthDefault ) + ] + (TableHead ( "" , [] , [] ) []) + [ TableBody + ( "" , [] , [] ) + (RowHeadColumns 0) + [] + [ Row + ( "" , [] , [] ) + [ Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "body" , Space , Str "a1" ] ] + , Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "body" , Space , Str "a1" ] ] + ] + , Row + ( "" , [] , [] ) + [ Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "body" , Space , Str "b1" ] ] + , Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "body" , Space , Str "b2" ] ] + ] + ] + ] + (TableFoot ( "" , [] , [] ) []) +, Header + 1 + ( "end-section" , [] , [] ) + [ Str "End" , Space , Str "Section" ] +] +``` diff --git a/test/command/10635.md b/test/command/10635.md new file mode 100644 --- /dev/null +++ b/test/command/10635.md @@ -0,0 +1,24 @@ +# Support for KOMA's `\minisec` command + +``` +% pandoc -f latex -t native +\minisec{Montage} +Zunächst suche man das Mauseloch. +^D +[ Header + 6 + ( "montage" , [ "unnumbered" , "unlisted" ] , [] ) + [ Str "Montage" ] +, Para + [ Str "Zun\228chst" + , Space + , Str "suche" + , Space + , Str "man" + , Space + , Str "das" + , Space + , Str "Mauseloch." + ] +] +``` diff --git a/test/command/10816.md b/test/command/10816.md new file mode 100644 --- /dev/null +++ b/test/command/10816.md @@ -0,0 +1,7 @@ +``` +% pandoc -f markdown -t typst +# Test① +^D += Test① +#label("test①") +``` diff --git a/test/command/10889.md b/test/command/10889.md new file mode 100644 --- /dev/null +++ b/test/command/10889.md @@ -0,0 +1,16 @@ +``` +% pandoc +apple + +: pomaceous + + fruit +^D +
+
apple
+
+

pomaceous

+

fruit

+
+
+``` diff --git a/test/command/10890.md b/test/command/10890.md new file mode 100644 --- /dev/null +++ b/test/command/10890.md @@ -0,0 +1,15 @@ +``` +% pandoc --tab-stop=2 --from=native --to=markdown +[ DefinitionList + [ ( [ Str "apple" ] + , [ [ Para [ Str "pomaceous" ] , Para [ Str "fruit" ] ] ] + ) + ] +] +^D +apple + +: pomaceous + + fruit +``` diff --git a/test/command/10912.md b/test/command/10912.md new file mode 100644 --- /dev/null +++ b/test/command/10912.md @@ -0,0 +1,16 @@ +``` +% pandoc -f docbook -t native + + +Para1 + + +Para2 + + +^D +[ OrderedList + ( 4 , Decimal , DefaultDelim ) + [ [ Para [ Str "Para1" ] ] , [ Para [ Str "Para2" ] ] ] +] +``` diff --git a/test/command/10915.md b/test/command/10915.md new file mode 100644 --- /dev/null +++ b/test/command/10915.md @@ -0,0 +1,8 @@ +``` +% pandoc -f latex +\newcommand{\a}{\ifmmode x \else y \fi} +$\a$ and \a +^D +

x and y

+``` + diff --git a/test/command/10919.md b/test/command/10919.md new file mode 100644 --- /dev/null +++ b/test/command/10919.md @@ -0,0 +1,21 @@ +# Org output with smart quotes turned on + +``` +% pandoc -t org+smart_quotes -s +"It's nice" she said. +^D +#+options: ':t + +"It's nice" she said. +``` + +Same test, but with special strings turned off. +``` +% pandoc -t org+smart_quotes-special_strings -s +"It's nice" she said. +^D +#+options: ':t +#+options: -:nil + +"It’s nice" she said. +``` diff --git a/test/command/10926.md b/test/command/10926.md new file mode 100644 --- /dev/null +++ b/test/command/10926.md @@ -0,0 +1,19 @@ +```` +% pandoc -f html -t gfm +
test
+^D +``` ruby +test +``` +```` + +```` +% pandoc -f html -t gfm +
test
+^D +``` ruby +test +``` +```` diff --git a/test/command/10965.md b/test/command/10965.md new file mode 100644 --- /dev/null +++ b/test/command/10965.md @@ -0,0 +1,47 @@ +``` +% pandoc -f markdown -t typst +::: {lang="en"} +This text should be in English. +::: +^D +#block[ +#set text(lang: "en"); This text should be in English. + +] +``` + +``` +% pandoc -f markdown -t typst +::: {lang="fr"} +Ce texte devrait être en français. +::: +^D +#block[ +#set text(lang: "fr"); Ce texte devrait être en français. + +] +``` + +``` +% pandoc -f markdown -t typst +::: {lang="de-DE"} +Dieser Text sollte auf Deutsch sein. +::: +^D +#block[ +#set text(lang: "de"); Dieser Text sollte auf Deutsch sein. + +] +``` + +``` +% pandoc -f markdown -t typst +::: {lang=""} +This should not have lang set. +::: +^D +#block[ +This should not have lang set. + +] +``` diff --git a/test/command/10983.md b/test/command/10983.md new file mode 100644 --- /dev/null +++ b/test/command/10983.md @@ -0,0 +1,22 @@ +``` +% pandoc --citeproc --csl command/chicago-fullnote-bibliography.csl -t plain +--- +references: +- id: test4 + type: blog-post + title: "Username as author" + author: + - brtw + container-title: "Reddit" + issued: + year: 2004 +suppress-bibliography: true +... + +[@test4] +^D +[1] + +[1] brtw, “Username as Author,” Reddit, 2004. + +``` diff --git a/test/command/10984.md b/test/command/10984.md new file mode 100644 --- /dev/null +++ b/test/command/10984.md @@ -0,0 +1,16 @@ + ``` + % pandoc -f html -t djot +

Hi +^D +{#foo .a .b} +# Hi +``` + +In this one the id is suppressed by the djot writer because +the same one would be automatically generated by the djot reader: +``` +% pandoc -f html -t djot +

Introduction

+^D +## Introduction +``` diff --git a/test/command/11006.md b/test/command/11006.md new file mode 100644 --- /dev/null +++ b/test/command/11006.md @@ -0,0 +1,56 @@ +``` +% pandoc -f html -t asciidoc +
    +
  • +

    Paragraph one

    +

    Paragraph two to force a list continuation

    +
      +
    • First nested
      • +
    • Second nested
      • +
    +
    • +
+ +

How about ordered lists?

+ +
    +
  1. +

    Paragraph one

    +

    Paragraph two to force a list continuation

    +
    1. Nested item
    +
    2. +
+ +

With non-default attributes:

+ +
    +
  1. +

    Paragraph one

    +

    Paragraph two to force a list continuation

    +
    1. Nested item
    +
    2. +
+^D +* Paragraph one ++ +Paragraph two to force a list continuation + +** First nested +** Second nested + +How about ordered lists? + +. Paragraph one ++ +Paragraph two to force a list continuation + +.. Nested item + +With non-default attributes: + +. Paragraph one ++ +Paragraph two to force a list continuation +[start=5] +.. Nested item +``` diff --git a/test/command/11013.md b/test/command/11013.md new file mode 100644 --- /dev/null +++ b/test/command/11013.md @@ -0,0 +1,23 @@ +``` +% pandoc --csl command/chicago-fullnote-bibliography.csl -C --wrap=none -t plain +--- +references: +- id: test10 + type: blog-post + title: "Test 10: Lowercase username in brackets, Works!" + author: + - literal: "[deleted]" + container-title: "Reddit" + issued: + year: 2009 +... + +blah [@test10]. +^D +blah.[1] + +[deleted]. “Test 10: Lowercase Username in Brackets, Works!” Reddit, 2009. + +[1] [deleted], “Test 10: Lowercase Username in Brackets, Works!” Reddit, 2009. + +``` diff --git a/test/command/11014.md b/test/command/11014.md new file mode 100644 --- /dev/null +++ b/test/command/11014.md @@ -0,0 +1,9 @@ +``` +% pandoc -f djot -t html +{.foo} +- bar +^D +
    +
  • bar
    • +
+``` diff --git a/test/command/11017.md b/test/command/11017.md new file mode 100644 --- /dev/null +++ b/test/command/11017.md @@ -0,0 +1,9 @@ +``` +% pandoc -t markdown -f typst +#show heading: smallcaps + += Introduction +^D +# [Introduction]{.smallcaps} + +``` diff --git a/test/command/11046.md b/test/command/11046.md new file mode 100644 --- /dev/null +++ b/test/command/11046.md @@ -0,0 +1,19 @@ +``` +% pandoc --citeproc -t plain+smart --csl command/chicago-note-bibliography.csl +--- +references: +- id: doe + title: Title + type: book + date: 2006 + author: John Doe +... + +blah blah [@doe]---blah blah. +^D +blah blah[1]---blah blah. + +John Doe. Title, n.d. + +[1] John Doe, Title. +``` diff --git a/test/command/11047.md b/test/command/11047.md new file mode 100644 --- /dev/null +++ b/test/command/11047.md @@ -0,0 +1,7 @@ +``` +% pandoc -t typst --wrap=auto +Full-time study: 2001-2003; Thesis submission: Nov 2005; Award: Jul 2006. +^D +Full-time study: 2001-2003; Thesis submission: Nov 2005; Award: Jul +\2006. +``` diff --git a/test/command/11048.md b/test/command/11048.md new file mode 100644 --- /dev/null +++ b/test/command/11048.md @@ -0,0 +1,48 @@ +``` +% pandoc -f biblatex -t markdown -s +@article{khatri2021spooky, + title = {Spooky action at a global distance: analysis of space-based + entanglement distribution for the quantum internet}, + author = {Khatri, Sumeet and Brady, Anthony J and Desporte, Ren{\'e}e A and + Bart, Manon P and Dowling, Jonathan P}, + journal = {{npj} Quantum Information}, + volume = {7}, + number = {1}, + pages = {4}, + doi = {10.1038/s41534-020-00327-5}, + url = {https://doi.org/10.1038/s41534-020-00327-5}, + year = {2021}, + publisher = {Nature Publishing Group UK London}, +} +^D +--- +nocite: "[@*]" +references: +- author: + - family: Khatri + given: Sumeet + - family: Brady + given: Anthony J + - family: Desporte + given: Renée A + - family: Bart + given: Manon P + - family: Dowling + given: Jonathan P + container-title: "[npj]{.nocase} Quantum Information" + doi: 10.1038/s41534-020-00327-5 + id: khatri2021spooky + issue: 1 + issued: 2021 + page: 4 + publisher: Nature Publishing Group UK London + title: "Spooky action at a global distance: Analysis of space-based + entanglement distribution for the quantum internet" + title-short: Spooky action at a global distance + type: article-journal + url: "https://doi.org/10.1038/s41534-020-00327-5" + volume: 7 +--- + + +``` diff --git a/test/command/11090.md b/test/command/11090.md new file mode 100644 --- /dev/null +++ b/test/command/11090.md @@ -0,0 +1,49 @@ +``` +% pandoc -f typst -t native +#include "command/11090/ch1.typ" + +== Chapter Two + +#figure( + image("command/11090/media/image1.png"), + caption: [This is an image.] +) +^D +[ Header + 2 ( "" , [] , [] ) [ Str "Chapter" , Space , Str "One" ] +, Figure + ( "" , [] , [] ) + (Caption + Nothing [ Para [ Str "An" , Space , Str "image." ] ]) + [ Para + [ Image + ( "" , [] , [] ) + [] + ( "command/11090/media/image1.png" , "" ) + ] + ] +, Header + 2 ( "" , [] , [] ) [ Str "Chapter" , Space , Str "Two" ] +, Figure + ( "" , [] , [] ) + (Caption + Nothing + [ Para + [ Str "This" + , Space + , Str "is" + , Space + , Str "an" + , Space + , Str "image." + ] + ]) + [ Para + [ Image + ( "" , [] , [] ) + [] + ( "command/11090/media/image1.png" , "" ) + ] + ] +] +``` diff --git a/test/command/11090/ch1.typ b/test/command/11090/ch1.typ new file mode 100644 --- /dev/null +++ b/test/command/11090/ch1.typ @@ -0,0 +1,6 @@ +== Chapter One + +#figure( + image("media/image1.png"), + caption: [An image.] +) diff --git a/test/command/11113.docx b/test/command/11113.docx new file mode 100644 Binary files /dev/null and b/test/command/11113.docx differ diff --git a/test/command/11113.md b/test/command/11113.md new file mode 100644 --- /dev/null +++ b/test/command/11113.md @@ -0,0 +1,7 @@ +``` +% pandoc command/11113.docx -t plain +^D +Colored:❤️😏📖👌😒🤍 + +BW: 😊︎ +``` diff --git a/test/command/1629.md b/test/command/1629.md --- a/test/command/1629.md +++ b/test/command/1629.md @@ -1,5 +1,5 @@ ``` -% pandoc -t latex --listings +% pandoc -t latex --syntax-highlighting=idiomatic bla bla `a % b` *bla bla `a % b`* diff --git a/test/command/2118.md b/test/command/2118.md --- a/test/command/2118.md +++ b/test/command/2118.md @@ -8,7 +8,7 @@ \end{figure} ^D [ Figure - ( "fig:setminus" , [] , [] ) + ( "fig:setminus" , [] , [ ( "latex-placement" , "ht" ) ] ) (Caption Nothing [ Plain [ Str "Set" , Space , Str "subtraction" ] ]) [ Plain diff --git a/test/command/3113.md b/test/command/3113.md --- a/test/command/3113.md +++ b/test/command/3113.md @@ -11,7 +11,7 @@ [ Para [ Math DisplayMath - "\\begin{aligned}\nA&=&B,\\\\\nC&=&D,\\\\\n%\\end{eqnarray}\n%\\begin{eqnarray}\nE&=&F\n\\end{aligned}" + "\\begin{eqnarray}\nA&=&B,\\\\\nC&=&D,\\\\\n%\\end{eqnarray}\n%\\begin{eqnarray}\nE&=&F\n\\end{eqnarray}" ] ] ``` diff --git a/test/command/3422.md b/test/command/3422.md --- a/test/command/3422.md +++ b/test/command/3422.md @@ -1,7 +1,7 @@ See #3422 ``` -% pandoc -t latex --listings +% pandoc -t latex --syntax-highlighting=idiomatic `int main(int argc, const char *argv[]);`{.c} ^D \passthrough{\lstinline[language=C]!int main(int argc, const char *argv[]);!} diff --git a/test/command/3577.md b/test/command/3577.md --- a/test/command/3577.md +++ b/test/command/3577.md @@ -15,7 +15,7 @@ \caption{Subfigure with Subfloat} \end{figure} ^D -
+
Caption 1
@@ -34,7 +34,7 @@ \caption{Caption 3} \end{figure} ^D -
+
Caption 3
diff --git a/test/command/4164.md b/test/command/4164.md --- a/test/command/4164.md +++ b/test/command/4164.md @@ -8,11 +8,9 @@ Here is inline html: -
- +::: {} ``{=html} bla bla - -
+::: ``` diff --git a/test/command/5420.md b/test/command/5420.md --- a/test/command/5420.md +++ b/test/command/5420.md @@ -1,5 +1,5 @@ ``` -% pandoc --listings -t latex +% pandoc --syntax-highlighting=idiomatic -t latex `int a = 1;`{.cpp style=cpp} ^D \passthrough{\lstinline[language={C++}, style=cpp]!int a = 1;!} diff --git a/test/command/6783.md b/test/command/6783.md --- a/test/command/6783.md +++ b/test/command/6783.md @@ -20,7 +20,7 @@ This is a test(Jupyter 2018). Jupyter, Project. 2018. “Binder 2.0 - Reproducible, Interactive, -Sharable Environments for Science at Scale.” In Proceedings of the 17th +Sharable Environments for Science at Scale.” Proceedings of the 17th Python in Science Conference. ``` diff --git a/test/command/6890.md b/test/command/6890.md --- a/test/command/6890.md +++ b/test/command/6890.md @@ -38,13 +38,11 @@ , citationHash = 0 } ] - [ Str "Fr\252chtel," - , Space - , Str "Budde," + [ Str "Fr\252chtel" , Space - , Str "and" + , Str "et" , Space - , Str "Cyprian" + , Str "al." , Space , Str "(2013)" ] @@ -65,13 +63,11 @@ , citationHash = 0 } ] - [ Str "Fr\252chtel," - , Space - , Str "Budde," + [ Str "Fr\252chtel" , Space - , Str "and" + , Str "et" , Space - , Str "Cyprian" + , Str "al." , Space , Str "(2013)" ] @@ -81,7 +77,7 @@ , Div ( "refs" , [ "references" , "csl-bib-body" , "hanging-indent" ] - , [ ( "entry-spacing" , "0" ) ] + , [] ) [ Div ( "ref-fruchtel-sozialer-2013a" , [ "csl-entry" ] , [] ) @@ -126,10 +122,6 @@ , Str "3rd" , Space , Str "ed." - , Space - , Str "Wiesbaden," - , Space - , Str "Germany:" , Space , Str "Springer" , Space diff --git a/test/command/6951.md b/test/command/6951.md --- a/test/command/6951.md +++ b/test/command/6951.md @@ -14,5 +14,5 @@ Crazy. n.d. -Doe, John. 2005. First Book. Cambridge: Cambridge University Press. +Doe, John. 2005. First Book. Cambridge University Press. ``` diff --git a/test/command/7219.md b/test/command/7219.md --- a/test/command/7219.md +++ b/test/command/7219.md @@ -20,6 +20,6 @@ and also in (1958) Reese, Trevor R. 1958. “Georgia in Anglo-Spanish Diplomacy, 1736-1739.” -William and Mary Quarterly, 3rd series, 15: 168–90. +William and Mary Quarterly, 3rd series, vol. 15: 168–90. ``` diff --git a/test/command/7329.md b/test/command/7329.md --- a/test/command/7329.md +++ b/test/command/7329.md @@ -32,7 +32,7 @@ <> <> -Doe, John. 2005. /First Book/. Cambridge: Cambridge University Press. +Doe, John. 2005. /First Book/. Cambridge University Press. ``` @@ -45,7 +45,7 @@ <> <> -Doe, John. 2005. /First Book/. Cambridge: Cambridge University Press. +Doe, John. 2005. /First Book/. Cambridge University Press. ``` ``` diff --git a/test/command/7512.md b/test/command/7512.md --- a/test/command/7512.md +++ b/test/command/7512.md @@ -4,7 +4,12 @@ [d,\delta]=0. \end{equation*} ^D -[ Para [ Math DisplayMath "[d,\\delta]=0." ] ] +[ Para + [ Math + DisplayMath + "\\begin{equation*}\n[d,\\delta]=0.\n\\end{equation*}" + ] +] ``` ``` diff --git a/test/command/7691.docx b/test/command/7691.docx new file mode 100644 Binary files /dev/null and b/test/command/7691.docx differ diff --git a/test/command/7691.md b/test/command/7691.md new file mode 100644 --- /dev/null +++ b/test/command/7691.md @@ -0,0 +1,5 @@ +``` +% pandoc command/7691.docx +^D +

Some text

+``` diff --git a/test/command/8354.md b/test/command/8354.md --- a/test/command/8354.md +++ b/test/command/8354.md @@ -26,7 +26,7 @@ ^D

References

+role="list">
Fekete, Jean-Daniel, and Juliana Freire. 2020. “Exploring diff --git a/test/command/8745.md b/test/command/8745.md --- a/test/command/8745.md +++ b/test/command/8745.md @@ -16,7 +16,7 @@ ^D [ Para [ Str "Sample" , Space , Str "text" ] , Figure - ( "fig:Mondrian" , [] , [] ) + ( "fig:Mondrian" , [] , [ ( "latex-placement" , "ht" ) ] ) (Caption Nothing [ Plain diff --git a/test/command/8869.md b/test/command/8869.md new file mode 100644 --- /dev/null +++ b/test/command/8869.md @@ -0,0 +1,20 @@ +Org arguments can either be single words, or quoted. + +``` +% pandoc -f org -t native +#+begin_src jupyter-julia :exports both +1 + 2 +#+end_src + +#+begin_src jupyter-julia :exports "both" +1 + 2 +#+end_src +^D +[ CodeBlock + ( "" , [ "julia" , "code" ] , [ ( "exports" , "both" ) ] ) + "1 + 2\n" +, CodeBlock + ( "" , [ "julia" , "code" ] , [ ( "exports" , "both" ) ] ) + "1 + 2\n" +] +``` diff --git a/test/command/8957.md b/test/command/8957.md new file mode 100644 --- /dev/null +++ b/test/command/8957.md @@ -0,0 +1,7 @@ +``` +% pandoc -f latex -t html --wrap=none +![Sample figure caption.](fig.jpg){width="342pt" height="9pc"} +^D +

![Sample figure caption.](fig.jpg)width="342pt" height="9pc"

+ +``` diff --git a/test/command/9000.md b/test/command/9000.md new file mode 100644 --- /dev/null +++ b/test/command/9000.md @@ -0,0 +1,14 @@ +``` +% pandoc -t org --wrap=auto --columns=20 +[This is a link with a long description that should not +wrap](http://example.com) and +some regular text after it so we can see how wrapping works. +^D +[[http://example.com][This is a link with a long description that should not wrap]] +and some regular +text after it so we +can see how wrapping +works. + +``` + diff --git a/test/command/9017.md b/test/command/9017.md --- a/test/command/9017.md +++ b/test/command/9017.md @@ -45,7 +45,8 @@ - DeGroot. 2002. “Probability.” + DeGroot. 2002. + Probability. diff --git a/test/command/9452.md b/test/command/9452.md --- a/test/command/9452.md +++ b/test/command/9452.md @@ -14,6 +14,6 @@ With supplement @something2024[p.~3]. -And just the year #cite(, form: "year");. +And just the year (#cite(, form: "year");). ``` diff --git a/test/command/982.md b/test/command/982.md --- a/test/command/982.md +++ b/test/command/982.md @@ -7,5 +7,9 @@ y=x^2 \EEQ ^D -[ Para [ Math DisplayMath "y=x^2" ] ] +[ Para + [ Math + DisplayMath "\\begin{equation}\ny=x^2\n\\end{equation}" + ] +] ``` diff --git a/test/command/9904.md b/test/command/9904.md --- a/test/command/9904.md +++ b/test/command/9904.md @@ -153,7 +153,7 @@ , Div ( "refs" , [ "references" , "csl-bib-body" , "hanging-indent" ] - , [ ( "entry-spacing" , "0" ) ] + , [] ) [ Div ( "ref-item1" , [ "csl-entry" ] , [] ) @@ -166,8 +166,6 @@ , Space , Emph [ Str "First" , Space , Str "Book" ] , Str "." - , Space - , Str "Cambridge:" , Space , Str "Cambridge" , Space diff --git a/test/command/biblatex-angenendt.md b/test/command/biblatex-angenendt.md --- a/test/command/biblatex-angenendt.md +++ b/test/command/biblatex-angenendt.md @@ -50,7 +50,7 @@ author: - family: Angenendt given: Arnold - container-title: Revue d'Histoire Ecclésiastique + container-title: Revue d'Histoire Eccl[é]{.nocase}siastique id: angenendt issued: 2002 language: de-DE diff --git a/test/command/citeproc-87.md b/test/command/citeproc-87.md --- a/test/command/citeproc-87.md +++ b/test/command/citeproc-87.md @@ -18,7 +18,7 @@ The Quoted is passed to citeproc as a Span ("",["csl-quoted"],[]) so that flipflopping and localization occur. ``` -% pandoc -C -t plain -Mlang=en +% pandoc -C -t plain -Mlang=en --csl command/le-tapuscrit-note.csl --- references: - id: a @@ -30,13 +30,15 @@ Foo [@a 50]. ^D -Foo (Aristotele, n.d., 50). +Foo.[1] -Aristotele. n.d. “Metafisica Et ‘Physica’.” +ARISTOTELE, “Metafisica et ‘Physica’.” + +[1] Aristotele, “Metafisica et ‘Physica’,” p. 50. ``` ``` -% pandoc -C -t plain -Mlang=it +% pandoc -C -t plain -Mlang=it --csl command/le-tapuscrit-note.csl --- references: - id: a @@ -48,8 +50,10 @@ Foo [@a 50]. ^D -Foo (Aristotele, s.d., 50). +Foo.[1] -Aristotele. s.d. «Metafisica et “Physica”». +ARISTOTELE, «Metafisica et “Physica”». + +[1] Aristotele, «Metafisica et “Physica”», p. 50. ``` diff --git a/test/command/lists-inside-definition.md b/test/command/lists-inside-definition.md --- a/test/command/lists-inside-definition.md +++ b/test/command/lists-inside-definition.md @@ -33,6 +33,7 @@ 2. list ^D \begin{description} +\tightlist \item[Definition] Foo diff --git a/test/command/macros.md b/test/command/macros.md --- a/test/command/macros.md +++ b/test/command/macros.md @@ -61,9 +61,9 @@ ^D \(5-67\) -\[\begin{aligned} +\begin{align} x &= y\\ -\end{aligned}\] +\end{align} \emph{hi} diff --git a/test/command/pandoc-citeproc-118.md b/test/command/pandoc-citeproc-118.md --- a/test/command/pandoc-citeproc-118.md +++ b/test/command/pandoc-citeproc-118.md @@ -19,10 +19,9 @@ ^D (Hitchcock 1959) is a spy thriller film. -:::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +:::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-nbn .csl-entry} -Hitchcock, Alfred, dir. 1959. *North by Northwest*. USA: -Metro-Goldwyn-Mayer. +Hitchcock, Alfred, dir. 1959. *North by Northwest*. Metro-Goldwyn-Mayer. ::: :::: ``` diff --git a/test/command/pandoc-citeproc-136.md b/test/command/pandoc-citeproc-136.md --- a/test/command/pandoc-citeproc-136.md +++ b/test/command/pandoc-citeproc-136.md @@ -15,7 +15,7 @@ ^D *Stanze in lode della donna brutta* (1547) is an anoynymous work. -:::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +:::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-stanze .csl-entry} *Stanze in lode della donna brutta*. 1547. Florence. ::: diff --git a/test/command/pandoc-citeproc-14.md b/test/command/pandoc-citeproc-14.md --- a/test/command/pandoc-citeproc-14.md +++ b/test/command/pandoc-citeproc-14.md @@ -51,27 +51,27 @@ References {#references .unnumbered} ========== ^D -Foo (Pelikan 1971b, 1:12). Bar (Pelikan 1971c, 1:12). Baz (Pelikan -1971a, 12). +Foo (Pelikan 1971b, 1:12). Bar (Pelikan 1971c, 12). Baz (Pelikan 1971a, +12). # References {#references .unnumbered} -:::::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +:::::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-CTv1c2 .csl-entry} -Pelikan, Jaroslav. 1971a. "Chapter Two." In *The Christian Tradition: A -History of the Development of Doctrine*, 1:34--56. Chicago: University -of Chicago Press. +Pelikan, Jaroslav. 1971a. "Chapter Two." In *The Emergence of the +Catholic Tradition (100--600)*, vol. 1 of *The Christian Tradition: A +History of the Development of Doctrine*. University of Chicago Press. ::: ::: {#ref-CT .csl-entry} ----------. 1971b. *The Christian Tradition: A History of the Development -of Doctrine*. Chicago: University of Chicago Press. +Pelikan, Jaroslav. 1971b. *The Christian Tradition: A History of the +Development of Doctrine*. University of Chicago Press. ::: ::: {#ref-CTv1 .csl-entry} ----------. 1971c. *The Emergence of the Catholic Tradition (100--600)*. -*The Christian Tradition: A History of the Development of Doctrine*. -Vol. 1. Chicago: University of Chicago Press. +Pelikan, Jaroslav. 1971c. *The Emergence of the Catholic Tradition +(100--600)*. In *The Christian Tradition: A History of the Development +of Doctrine*, vol. 1. University of Chicago Press. ::: :::::: ``` diff --git a/test/command/pandoc-citeproc-175.md b/test/command/pandoc-citeproc-175.md --- a/test/command/pandoc-citeproc-175.md +++ b/test/command/pandoc-citeproc-175.md @@ -35,9 +35,9 @@ > Doe, Jane. 2011. "A Title." *A Magazine*, January--February. -:::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +:::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-item1 .csl-entry} -Doe, Jane. 2011. "A Title." *A Magazine*, January--February 2011. +Doe, Jane. 2011. "A Title." *A Magazine*, January--February, 33--44. ::: :::: ``` diff --git a/test/command/pandoc-citeproc-25.md b/test/command/pandoc-citeproc-25.md --- a/test/command/pandoc-citeproc-25.md +++ b/test/command/pandoc-citeproc-25.md @@ -22,9 +22,9 @@ # References {#references .unnumbered} -:::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +:::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-item1 .csl-entry} -Author, Al. 1998. "Foo Bar Baz: Bazbaz Foo." +Author, Al. 1998. *Foo Bar Baz: Bazbaz Foo*. ::: :::: ``` diff --git a/test/command/pandoc-citeproc-250.md b/test/command/pandoc-citeproc-250.md --- a/test/command/pandoc-citeproc-250.md +++ b/test/command/pandoc-citeproc-250.md @@ -13,9 +13,9 @@ ^D ([Doe, n.d.](#ref-doe)) -:::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +:::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-doe .csl-entry} -Doe. n.d. "Title." +Doe. n.d. *Title*. ::: :::: ``` diff --git a/test/command/pandoc-citeproc-301.md b/test/command/pandoc-citeproc-301.md --- a/test/command/pandoc-citeproc-301.md +++ b/test/command/pandoc-citeproc-301.md @@ -10,16 +10,16 @@ @test; @test2 ^D -"Essays Presented to N.R. Ker (On Art)" (n.d.); "*Test:* An Experiment: -An Abridgement" (n.d.) +*Essays Presented to N.R. Ker (On Art)* (n.d.); **Test:* An Experiment: +An Abridgement* (n.d.) -::::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +::::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-test .csl-entry} -"Essays Presented to N.R. Ker (On Art)." n.d. +*Essays Presented to N.R. Ker (On Art)*. n.d. ::: ::: {#ref-test2 .csl-entry} -"*Test:* An Experiment: An Abridgement." n.d. +**Test:* An Experiment: An Abridgement*. n.d. ::: ::::: ``` diff --git a/test/command/pandoc-citeproc-307.md b/test/command/pandoc-citeproc-307.md --- a/test/command/pandoc-citeproc-307.md +++ b/test/command/pandoc-citeproc-307.md @@ -22,7 +22,7 @@ ^D Bonjour(Bazin 1954) ! -:::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +:::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-bazin_cybernetique_1954 .csl-entry} Bazin, André. 1954. « La Cybernétique d'André Cayatte ». *Cahiers du cinéma*, nᵒ 36 (juin): 22‑27. diff --git a/test/command/pandoc-citeproc-320a.md b/test/command/pandoc-citeproc-320a.md --- a/test/command/pandoc-citeproc-320a.md +++ b/test/command/pandoc-citeproc-320a.md @@ -61,17 +61,17 @@ Uch, Ann. 2000. -‘Udhrī, Jamīl al-. 2000. “Inverted Curly Apostrophe = Opening Single -Curly Quote (for Ayn).” +‘Udhrī, Jamīl al-. 2000. Inverted Curly Apostrophe = Opening Single +Curly Quote (for Ayn). -ʿUdhrī, Jamīl al-. 2000. “Ayn.” +ʿUdhrī, Jamīl al-. 2000. Ayn. -’Udhrī, Jamīl al-. 2000a. “Curly Apostrophe = Closing Single Curly Quote -(for Hamza).” +’Udhrī, Jamīl al-. 2000a. Curly Apostrophe = Closing Single Curly Quote +(for Hamza). -ʾUdhrī, Jamīl al-. 2000. “Hamza.” +ʾUdhrī, Jamīl al-. 2000. Hamza. -’Udhrī, Jamīl al-. 2000b. “Straight Apostrophe.” +’Udhrī, Jamīl al-. 2000b. Straight Apostrophe. Uebel, Joe. 2000. diff --git a/test/command/pandoc-citeproc-325.md b/test/command/pandoc-citeproc-325.md --- a/test/command/pandoc-citeproc-325.md +++ b/test/command/pandoc-citeproc-325.md @@ -16,15 +16,15 @@ [@item1; @item2] ^D -(Smith, n.d.a, n.d.b) +(Smith, n.d.-a, n.d.-b) -::::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +::::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-item1 .csl-entry} -Smith, John. n.d.a. +Smith, John. n.d.-a. ::: ::: {#ref-item2 .csl-entry} ----------. n.d.b. +Smith, John. n.d.-b. ::: ::::: ``` diff --git a/test/command/pandoc-citeproc-327.md b/test/command/pandoc-citeproc-327.md --- a/test/command/pandoc-citeproc-327.md +++ b/test/command/pandoc-citeproc-327.md @@ -45,7 +45,7 @@ ::: {#ref-LiLiaoDongWanHaiYuDiQiDongWuCiJiShengChanLiYanJiuJiShengJingGuaYiXingPingJie2017 .csl-entry} [\[1\] ]{.csl-left-margin}[李轶平, 于旭光, 孙明, 等. [辽东湾海域底栖动物次级生产力研究及生境适宜性评价](http://kns.cnki.net/kns/detail/detail.aspx?QueryID=4&CurRec=4&recid=&FileName=CHAN201706006&DbName=CJFDLAST2018&DbCode=CJFQ&yx=Y&pr=&URLID=21.1110.S.20171129.1725.006)\[J\]. -水产科学, 2017(6): 728--734.]{.csl-right-inline} +水产科学, 2017(6): 728～734.]{.csl-right-inline} ::: :::: ``` diff --git a/test/command/pandoc-citeproc-371.md b/test/command/pandoc-citeproc-371.md --- a/test/command/pandoc-citeproc-371.md +++ b/test/command/pandoc-citeproc-371.md @@ -28,13 +28,13 @@ # References {#references .unnumbered} -::::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +::::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-item2 .csl-entry} Doe, Jane. 2018. *Title Two*. ::: ::: {#ref-item1 .csl-entry} ----------. In press. *Title One*. +Doe, Jane. In press. *Title One*. ::: ::::: ``` diff --git a/test/command/pandoc-citeproc-38.md b/test/command/pandoc-citeproc-38.md --- a/test/command/pandoc-citeproc-38.md +++ b/test/command/pandoc-citeproc-38.md @@ -19,11 +19,11 @@ @a ^D -Doe, Doe, and Roe (2007) +Doe et al. (2007) -:::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +:::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-a .csl-entry} -Doe, Ann, Ben Doe, and Ron Roe. 2007. "Title." +Doe, Ann, Ben Doe, and Ron Roe. 2007. *Title*. ::: :::: ``` diff --git a/test/command/pandoc-citeproc-401.md b/test/command/pandoc-citeproc-401.md --- a/test/command/pandoc-citeproc-401.md +++ b/test/command/pandoc-citeproc-401.md @@ -43,17 +43,17 @@ ^D Haslanger (\[2003\] 2012, \[2000\] 2012) says... -::::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +::::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-haslanger2012FeminismMetaphysicsNegotiating .csl-entry} Haslanger, Sally. (2000) 2012. "Feminism in Metaphysics: Negotiating the Natural." In *Resisting Reality: Social Construction and Social -Critique*, 139--57. Oxford: Oxford University Press. +Critique*. Oxford University Press. ::: ::: {#ref-haslanger2012SocialConstructionDebunking .csl-entry} ----------. (2003) 2012. "Social Construction: The 'Debunking' Project." -In *Resisting Reality: Social Construction and Social Critique*, -113--38. Oxford: Oxford University Press. +Haslanger, Sally. (2003) 2012. "Social Construction: The 'Debunking' +Project." In *Resisting Reality: Social Construction and Social +Critique*. Oxford University Press. ::: ::::: ``` diff --git a/test/command/pandoc-citeproc-408.md b/test/command/pandoc-citeproc-408.md --- a/test/command/pandoc-citeproc-408.md +++ b/test/command/pandoc-citeproc-408.md @@ -27,13 +27,13 @@ ^D (Smith and Smith 2019; Smith 2019) -::::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +::::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-smith1 .csl-entry} -Smith, Mary. 2019. "Foo." +Smith, Mary. 2019. *Foo*. ::: ::: {#ref-smithsmith .csl-entry} -Smith, Mary, and John Smith. 2019. "Foo Bar." +Smith, Mary, and John Smith. 2019. *Foo Bar*. ::: ::::: ``` diff --git a/test/command/pandoc-citeproc-416.md b/test/command/pandoc-citeproc-416.md --- a/test/command/pandoc-citeproc-416.md +++ b/test/command/pandoc-citeproc-416.md @@ -44,17 +44,17 @@ # References {#references .unnumbered} -:::::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +:::::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-item2 .csl-entry} -Doe, J. 2007. "The Title," December 12--13, 2007. +Doe, J. 2007. *The Title*. December 12--13. ::: ::: {#ref-item3 .csl-entry} ----------. 2008. "The Title," 2008. +Doe, J. 2008. *The Title*. ::: ::: {#ref-item1 .csl-entry} ----------. 2010. "The Title," December 13, 2010. +Doe, J. 2010. *The Title*. December 13. ::: :::::: ``` diff --git a/test/command/pandoc-citeproc-47.md b/test/command/pandoc-citeproc-47.md --- a/test/command/pandoc-citeproc-47.md +++ b/test/command/pandoc-citeproc-47.md @@ -69,8 +69,8 @@ References {#references .unnumbered} ========== ^D -Foo (Doe 2000a). Bar (Doe and Poe 2000). Foo (Doe 2000b). Bar (Doe, Loe, -and Toe 2000). +Foo (Doe 2000a). Bar (Doe and Poe 2000). Foo (Doe 2000b). Bar (Doe et +al. 2000). Expected output: @@ -92,13 +92,13 @@ # References {#references .unnumbered} -::::::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +::::::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-doe .csl-entry} Doe, A. 2000a. *Title*. ::: ::: {#ref-doe-ed .csl-entry} ----------, ed. 2000b. *Title*. +Doe, A., ed. 2000b. *Title*. ::: ::: {#ref-doeloetoe .csl-entry} diff --git a/test/command/pandoc-citeproc-51.md b/test/command/pandoc-citeproc-51.md --- a/test/command/pandoc-citeproc-51.md +++ b/test/command/pandoc-citeproc-51.md @@ -33,7 +33,7 @@ ^D Doe (1987--1988); Roe (1987) -::::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +::::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-item1 .csl-entry} Doe, John. 1987--1988. "The Title." *Journal of Something* 3: 12--34. ::: diff --git a/test/command/pandoc-citeproc-64.md b/test/command/pandoc-citeproc-64.md --- a/test/command/pandoc-citeproc-64.md +++ b/test/command/pandoc-citeproc-64.md @@ -1,5 +1,5 @@ ``` -% pandoc --citeproc -t markdown-citations +% pandoc --citeproc -t markdown-citations --csl command/chicago-fullnote-bibliography.csl --- bibliography: - command/biblio.bib @@ -8,17 +8,17 @@ ^D :::::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} -::: {#ref-item1 .csl-entry} -Doe, John. 2005. *First Book*. Cambridge: Cambridge University Press. +::: {#ref-item2 .csl-entry} +Doe, John. "Article." *Journal of Generic Studies* 6 (2006): 33--34. ::: -::: {#ref-item2 .csl-entry} ----------. 2006. "Article." *Journal of Generic Studies* 6: 33--34. +::: {#ref-item1 .csl-entry} +---------. *First Book*. Cambridge: Cambridge University Press, 2005. ::: ::: {#ref-пункт3 .csl-entry} -Doe, John, and Jenny Roe. 2007. "Why Water Is Wet." In *Third Book*, -edited by Sam Smith. Oxford: Oxford University Press. +Doe, John, and Jenny Roe. "Why Water Is Wet." In *Third Book*, edited by +Sam Smith. Oxford: Oxford University Press, 2007. ::: :::::: ``` diff --git a/test/command/pandoc-citeproc-65.md b/test/command/pandoc-citeproc-65.md --- a/test/command/pandoc-citeproc-65.md +++ b/test/command/pandoc-citeproc-65.md @@ -28,11 +28,10 @@ ^D (Stotz 1996--2004) -:::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +:::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-stotz:1996handbuch .csl-entry} Stotz, Peter. 1996--2004. *Handbuch zur lateinischen Sprache des -Mittelalters*. 5 vols. Handbuch der Altertumswissenschaft 2.5. Munich: -Beck. +Mittelalters*. 5 vols. Handbuch der Altertumswissenschaft, 2.5. Beck. ::: :::: ``` diff --git a/test/command/pandoc-citeproc-7.md b/test/command/pandoc-citeproc-7.md --- a/test/command/pandoc-citeproc-7.md +++ b/test/command/pandoc-citeproc-7.md @@ -23,9 +23,9 @@ ^D Author (2011) -:::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +:::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-item1 .csl-entry} -Author, Ann. 2011. "Title." *Journal*, September 24--26, 2011. +Author, Ann. 2011. "Title." *Journal*, September 24--26. ::: :::: ``` diff --git a/test/command/pandoc-citeproc-70.md b/test/command/pandoc-citeproc-70.md --- a/test/command/pandoc-citeproc-70.md +++ b/test/command/pandoc-citeproc-70.md @@ -56,14 +56,14 @@ ^D (Thorndike 1955; Dinkova-Bruun 2009) -::::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +::::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-bruun:2009samuel .csl-entry} Dinkova-Bruun, Greti. 2009. "Samuel Presbyter and the Glosses to His Versification of Psalm 1: An Anti-Church Invective?" In *Florilegium mediaevale: Études offertes à Jacqueline Hamesse à l'occasion de son -éméritat*, edited by José Francisco Meirinhos and Olga Weijers, 155--74. -Textes et études du moyen âge 50. Louvain-la-Neuve: Fédération -Internationale des Instituts d'Études Médiévales. +éméritat*, edited by José Francisco Meirinhos and Olga Weijers. Textes +et études du moyen âge 50. Fédération Internationale des Instituts +d'Études Médiévales. ::: ::: {#ref-thorndike:1955unde .csl-entry} diff --git a/test/command/pandoc-citeproc-76.md b/test/command/pandoc-citeproc-76.md --- a/test/command/pandoc-citeproc-76.md +++ b/test/command/pandoc-citeproc-76.md @@ -44,21 +44,21 @@ ^D Author (1998c), Author (1998d), Author (1998a), Author (1998b) -::::::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +::::::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-item3 .csl-entry} -Author, Al. 1998a. "Foo Bar Baz: A Bazbaz Bar Foo." +Author, Al. 1998a. *Foo Bar Baz: A Bazbaz Bar Foo*. ::: ::: {#ref-item4 .csl-entry} ----------. 1998b. "Foo Bar Baz: An Abazbaz Bar Foo." +Author, Al. 1998b. *Foo Bar Baz: An Abazbaz Bar Foo*. ::: ::: {#ref-item1 .csl-entry} ----------. 1998c. "Foo Bar Baz: Bazbaz Bar Foo." +Author, Al. 1998c. *Foo Bar Baz: Bazbaz Bar Foo*. ::: ::: {#ref-item2 .csl-entry} ----------. 1998d. "Foo Bar Baz: The Bazbaz Bar Foo." +Author, Al. 1998d. *Foo Bar Baz: The Bazbaz Bar Foo*. ::: ::::::: ``` diff --git a/test/command/pandoc-citeproc-87.md b/test/command/pandoc-citeproc-87.md --- a/test/command/pandoc-citeproc-87.md +++ b/test/command/pandoc-citeproc-87.md @@ -108,49 +108,49 @@ References ========== ^D -Doe (2006c) -- webpage, date +Doe (2006a) -- webpage, date -Doe (2006d) -- webpage, date range +Doe (2006c) -- webpage, date range -Doe (2006a) -- webpage, date range YM +Doe (2006b) -- webpage, date range YM Doe (2006--2007) -- webpage, date range across years Doe (2006e) -- article-newspaper -Doe (2006b) -- article-newspaper YM +Doe (2006d) -- article-newspaper YM # References {#references .unnumbered} -::::::::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} -::: {#ref-item2a .csl-entry} -Doe, John. 2006a. "Title." The Web Site. October--November 2006. +::::::::: {#refs .references .csl-bib-body .hanging-indent} +::: {#ref-item1 .csl-entry} +Doe, John. 2006a. "Title." The Web Site, October 26. . ::: -::: {#ref-item3b .csl-entry} ----------. 2006b. "Title." *The Newspaper*, October--November 2006. +::: {#ref-item2a .csl-entry} +Doe, John. 2006b. "Title." The Web Site, October--November. . ::: -::: {#ref-item1 .csl-entry} ----------. 2006c. "Title." The Web Site. October 26, 2006. +::: {#ref-item2 .csl-entry} +Doe, John. 2006c. "Title." The Web Site, October 26--November 27. . ::: -::: {#ref-item2 .csl-entry} ----------. 2006d. "Title." The Web Site. October 26--November 27, 2006. +::: {#ref-item3b .csl-entry} +Doe, John. 2006d. "Title." *The Newspaper*, October--November. . ::: ::: {#ref-item3 .csl-entry} ----------. 2006e. "Title." *The Newspaper*, October 26--November 27, -2006. . +Doe, John. 2006e. "Title." *The Newspaper*, October 26--November 27. +. ::: ::: {#ref-item2b .csl-entry} ----------. 2006--2007. "Title." The Web Site. December 31, 2006--January -1, 2007. . +Doe, John. 2006--2007. "Title." The Web Site, December 31--January 1. +. ::: ::::::::: ``` diff --git a/test/command/pandoc-citeproc-chicago-author-date.md b/test/command/pandoc-citeproc-chicago-author-date.md --- a/test/command/pandoc-citeproc-chicago-author-date.md +++ b/test/command/pandoc-citeproc-chicago-author-date.md @@ -83,18 +83,18 @@ # References {#references .unnumbered} -:::::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +:::::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-item1 .csl-entry} -Doe, John. 2005. *First Book*. Cambridge: Cambridge University Press. +Doe, John. 2005. *First Book*. Cambridge University Press. ::: ::: {#ref-item2 .csl-entry} ----------. 2006. "Article." *Journal of Generic Studies* 6: 33--34. +Doe, John. 2006. "Article." *Journal of Generic Studies* 6: 33--34. ::: ::: {#ref-пункт3 .csl-entry} Doe, John, and Jenny Roe. 2007. "Why Water Is Wet." In *Third Book*, -edited by Sam Smith. Oxford: Oxford University Press. +edited by Sam Smith. Oxford University Press. ::: :::::: diff --git a/test/command/pandoc-citeproc-no-author.md b/test/command/pandoc-citeproc-no-author.md --- a/test/command/pandoc-citeproc-no-author.md +++ b/test/command/pandoc-citeproc-no-author.md @@ -45,29 +45,29 @@ *Magazine* (2012a, 3), *Magazine* (2012b), *Magazine* (2012c), *Magazine* (2012d), *Newspaper* (2012a), *Newspaper* (2012b) -::::::::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +::::::::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-item1 .csl-entry} -*Magazine*. 2012a. "Title A," 2012. +*Magazine*. 2012a. "Title A." ::: ::: {#ref-item2 .csl-entry} ----------. 2012b. "Title B," 2012. +*Magazine*. 2012b. "Title B." ::: ::: {#ref-item3 .csl-entry} ----------. 2012c. "Title C," 2012. +*Magazine*. 2012c. "Title C." ::: ::: {#ref-item4 .csl-entry} ----------. 2012d. "Title D," 2012. +*Magazine*. 2012d. "Title D." ::: ::: {#ref-item5 .csl-entry} -*Newspaper*. 2012a. "Title E," 2012. +*Newspaper*. 2012a. "Title E." ::: ::: {#ref-item6 .csl-entry} ----------. 2012b. "Title F," 2012. +*Newspaper*. 2012b. "Title F." ::: ::::::::: ``` diff --git a/test/command/pandoc-citeproc-number-of-volumes.md b/test/command/pandoc-citeproc-number-of-volumes.md --- a/test/command/pandoc-citeproc-number-of-volumes.md +++ b/test/command/pandoc-citeproc-number-of-volumes.md @@ -21,9 +21,9 @@ ^D Author (2013) -:::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0"} +:::: {#refs .references .csl-bib-body .hanging-indent} ::: {#ref-item1 .csl-entry} -Author, Al. 2013. *Title*. 2 vols. Location: Publisher. +Author, Al. 2013. *Title*. 2 vols. Publisher. ::: :::: ``` diff --git a/test/command/refs.md b/test/command/refs.md --- a/test/command/refs.md +++ b/test/command/refs.md @@ -118,7 +118,7 @@ , Para [ Math DisplayMath - "\\label{eq:Accuracy}\n Accuracy = \\frac{t_p + t_n}{t_p + f_p + f_n + t_n}" + "\\begin{equation}\n \\label{eq:Accuracy}\n Accuracy = \\frac{t_p + t_n}{t_p + f_p + f_n + t_n}\n\\end{equation}" ] ] ``` diff --git a/test/docbook-reader.docbook b/test/docbook-reader.docbook --- a/test/docbook-reader.docbook +++ b/test/docbook-reader.docbook @@ -405,8 +405,8 @@ Fancy list markers - - + + begins with 2 @@ -418,8 +418,8 @@ with a continuation - - + + sublist with roman numerals, starting with 4 @@ -457,13 +457,13 @@ Upper Roman. - - + + Decimal start with 6 - - + + Lower alpha with paren @@ -1624,6 +1624,39 @@ Here's another one. Note the blank line between rows. + + + + + Multiline table with cells spanning multiple columns and rows without caption. + + + + + + + + + + Columns + + + A + B + C + + + + + A1 + B1 + C1 + C2 + + + A2 + B2 + A3 + B3 + + + C3 + diff --git a/test/docbook-reader.native b/test/docbook-reader.native --- a/test/docbook-reader.native +++ b/test/docbook-reader.native @@ -3046,6 +3046,128 @@ ] ] (TableFoot ( "" , [] , [] ) []) + , Para + [ Str "Multiline" + , Space + , Str "table" + , Space + , Str "with" + , Space + , Str "cells" + , Space + , Str "spanning" + , Space + , Str "multiple" + , Space + , Str "columns" + , Space + , Str "and" + , Space + , Str "rows" + , Space + , Str "without" + , Space + , Str "caption." + ] + , Table + ( "" , [] , [] ) + (Caption Nothing []) + [ ( AlignLeft , ColWidthDefault ) + , ( AlignLeft , ColWidthDefault ) + , ( AlignLeft , ColWidthDefault ) + ] + (TableHead + ( "" , [] , [] ) + [ Row + ( "" , [] , [] ) + [ Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 3) + [ Plain [ Str "Columns" ] ] + ] + , Row + ( "" , [] , [] ) + [ Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "A" ] ] + , Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "B" ] ] + , Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "C" ] ] + ] + ]) + [ TableBody + ( "" , [] , [] ) + (RowHeadColumns 0) + [] + [ Row + ( "" , [] , [] ) + [ Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 2) + [ Plain + [ Str "A1" , Space , Str "+" , Space , Str "B1" ] + ] + , Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 2) + (ColSpan 1) + [ Plain + [ Str "C1" , Space , Str "+" , Space , Str "C2" ] + ] + ] + , Row + ( "" , [] , [] ) + [ Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 2) + (ColSpan 2) + [ Plain + [ Str "A2" + , Space + , Str "+" + , Space + , Str "B2" + , Space + , Str "+" + , Space + , Str "A3" + , Space + , Str "+" + , Space + , Str "B3" + ] + ] + ] + , Row + ( "" , [] , [] ) + [ Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "C3" ] ] + ] + ] + ] + (TableFoot ( "" , [] , [] ) []) , OrderedList ( 1 , DefaultStyle , DefaultDelim ) [ [ Para [ Str "A" , Space , Str "Step" ] ] diff --git a/test/docx/golden/block_quotes.docx b/test/docx/golden/block_quotes.docx Binary files a/test/docx/golden/block_quotes.docx and b/test/docx/golden/block_quotes.docx differ diff --git a/test/docx/golden/codeblock.docx b/test/docx/golden/codeblock.docx Binary files a/test/docx/golden/codeblock.docx and b/test/docx/golden/codeblock.docx differ diff --git a/test/docx/golden/comments.docx b/test/docx/golden/comments.docx Binary files a/test/docx/golden/comments.docx and b/test/docx/golden/comments.docx differ diff --git a/test/docx/golden/custom_style_no_reference.docx b/test/docx/golden/custom_style_no_reference.docx Binary files a/test/docx/golden/custom_style_no_reference.docx and b/test/docx/golden/custom_style_no_reference.docx differ diff --git a/test/docx/golden/custom_style_preserve.docx b/test/docx/golden/custom_style_preserve.docx Binary files a/test/docx/golden/custom_style_preserve.docx and b/test/docx/golden/custom_style_preserve.docx differ diff --git a/test/docx/golden/definition_list.docx b/test/docx/golden/definition_list.docx Binary files a/test/docx/golden/definition_list.docx and b/test/docx/golden/definition_list.docx differ diff --git a/test/docx/golden/document-properties-short-desc.docx b/test/docx/golden/document-properties-short-desc.docx Binary files a/test/docx/golden/document-properties-short-desc.docx and b/test/docx/golden/document-properties-short-desc.docx differ diff --git a/test/docx/golden/document-properties.docx b/test/docx/golden/document-properties.docx Binary files a/test/docx/golden/document-properties.docx and b/test/docx/golden/document-properties.docx differ diff --git a/test/docx/golden/headers.docx b/test/docx/golden/headers.docx Binary files a/test/docx/golden/headers.docx and b/test/docx/golden/headers.docx differ diff --git a/test/docx/golden/image.docx b/test/docx/golden/image.docx Binary files a/test/docx/golden/image.docx and b/test/docx/golden/image.docx differ diff --git a/test/docx/golden/inline_code.docx b/test/docx/golden/inline_code.docx Binary files a/test/docx/golden/inline_code.docx and b/test/docx/golden/inline_code.docx differ diff --git a/test/docx/golden/inline_formatting.docx b/test/docx/golden/inline_formatting.docx Binary files a/test/docx/golden/inline_formatting.docx and b/test/docx/golden/inline_formatting.docx differ diff --git a/test/docx/golden/inline_images.docx b/test/docx/golden/inline_images.docx Binary files a/test/docx/golden/inline_images.docx and b/test/docx/golden/inline_images.docx differ diff --git a/test/docx/golden/link_in_notes.docx b/test/docx/golden/link_in_notes.docx Binary files a/test/docx/golden/link_in_notes.docx and b/test/docx/golden/link_in_notes.docx differ diff --git a/test/docx/golden/links.docx b/test/docx/golden/links.docx Binary files a/test/docx/golden/links.docx and b/test/docx/golden/links.docx differ diff --git a/test/docx/golden/lists.docx b/test/docx/golden/lists.docx Binary files a/test/docx/golden/lists.docx and b/test/docx/golden/lists.docx differ diff --git a/test/docx/golden/lists_9994.docx b/test/docx/golden/lists_9994.docx Binary files a/test/docx/golden/lists_9994.docx and b/test/docx/golden/lists_9994.docx differ diff --git a/test/docx/golden/lists_continuing.docx b/test/docx/golden/lists_continuing.docx Binary files a/test/docx/golden/lists_continuing.docx and b/test/docx/golden/lists_continuing.docx differ diff --git a/test/docx/golden/lists_div_bullets.docx b/test/docx/golden/lists_div_bullets.docx Binary files a/test/docx/golden/lists_div_bullets.docx and b/test/docx/golden/lists_div_bullets.docx differ diff --git a/test/docx/golden/lists_multiple_initial.docx b/test/docx/golden/lists_multiple_initial.docx Binary files a/test/docx/golden/lists_multiple_initial.docx and b/test/docx/golden/lists_multiple_initial.docx differ diff --git a/test/docx/golden/lists_restarting.docx b/test/docx/golden/lists_restarting.docx Binary files a/test/docx/golden/lists_restarting.docx and b/test/docx/golden/lists_restarting.docx differ diff --git a/test/docx/golden/nested_anchors_in_header.docx b/test/docx/golden/nested_anchors_in_header.docx Binary files a/test/docx/golden/nested_anchors_in_header.docx and b/test/docx/golden/nested_anchors_in_header.docx differ diff --git a/test/docx/golden/notes.docx b/test/docx/golden/notes.docx Binary files a/test/docx/golden/notes.docx and b/test/docx/golden/notes.docx differ diff --git a/test/docx/golden/raw-blocks.docx b/test/docx/golden/raw-blocks.docx Binary files a/test/docx/golden/raw-blocks.docx and b/test/docx/golden/raw-blocks.docx differ diff --git a/test/docx/golden/raw-bookmarks.docx b/test/docx/golden/raw-bookmarks.docx Binary files a/test/docx/golden/raw-bookmarks.docx and b/test/docx/golden/raw-bookmarks.docx differ diff --git a/test/docx/golden/table_one_row.docx b/test/docx/golden/table_one_row.docx Binary files a/test/docx/golden/table_one_row.docx and b/test/docx/golden/table_one_row.docx differ diff --git a/test/docx/golden/table_with_list_cell.docx b/test/docx/golden/table_with_list_cell.docx Binary files a/test/docx/golden/table_with_list_cell.docx and b/test/docx/golden/table_with_list_cell.docx differ diff --git a/test/docx/golden/tables-default-widths.docx b/test/docx/golden/tables-default-widths.docx Binary files a/test/docx/golden/tables-default-widths.docx and b/test/docx/golden/tables-default-widths.docx differ diff --git a/test/docx/golden/tables.docx b/test/docx/golden/tables.docx Binary files a/test/docx/golden/tables.docx and b/test/docx/golden/tables.docx differ diff --git a/test/docx/golden/tables_separated_with_rawblock.docx b/test/docx/golden/tables_separated_with_rawblock.docx Binary files a/test/docx/golden/tables_separated_with_rawblock.docx and b/test/docx/golden/tables_separated_with_rawblock.docx differ diff --git a/test/docx/golden/task_list.docx b/test/docx/golden/task_list.docx Binary files a/test/docx/golden/task_list.docx and b/test/docx/golden/task_list.docx differ diff --git a/test/docx/golden/track_changes_deletion.docx b/test/docx/golden/track_changes_deletion.docx Binary files a/test/docx/golden/track_changes_deletion.docx and b/test/docx/golden/track_changes_deletion.docx differ diff --git a/test/docx/golden/track_changes_insertion.docx b/test/docx/golden/track_changes_insertion.docx Binary files a/test/docx/golden/track_changes_insertion.docx and b/test/docx/golden/track_changes_insertion.docx differ diff --git a/test/docx/golden/track_changes_move.docx b/test/docx/golden/track_changes_move.docx Binary files a/test/docx/golden/track_changes_move.docx and b/test/docx/golden/track_changes_move.docx differ diff --git a/test/docx/golden/track_changes_scrubbed_metadata.docx b/test/docx/golden/track_changes_scrubbed_metadata.docx Binary files a/test/docx/golden/track_changes_scrubbed_metadata.docx and b/test/docx/golden/track_changes_scrubbed_metadata.docx differ diff --git a/test/docx/golden/unicode.docx b/test/docx/golden/unicode.docx Binary files a/test/docx/golden/unicode.docx and b/test/docx/golden/unicode.docx differ diff --git a/test/docx/golden/verbatim_subsuper.docx b/test/docx/golden/verbatim_subsuper.docx Binary files a/test/docx/golden/verbatim_subsuper.docx and b/test/docx/golden/verbatim_subsuper.docx differ diff --git a/test/man-reader.man b/test/man-reader.man --- a/test/man-reader.man +++ b/test/man-reader.man @@ -1,4 +1,4 @@ -.TH "Pandoc Man tests" "" "Oct 17, 2018" "" "" +.TH "PANDOC-MAN-TESTS" "1" "Oct 17, 2018" "v1.0.1" "Pandoc Tests" .PP This is a set of tests for pandoc. .PP diff --git a/test/man-reader.native b/test/man-reader.native --- a/test/man-reader.native +++ b/test/man-reader.native @@ -6,16 +6,12 @@ , MetaInlines [ Str "Oct" , Space , Str "17," , Space , Str "2018" ] ) - , ( "section" , MetaInlines [] ) - , ( "title" - , MetaInlines - [ Str "Pandoc" - , Space - , Str "Man" - , Space - , Str "tests" - ] + , ( "footer" , MetaInlines [ Str "v1.0.1" ] ) + , ( "header" + , MetaInlines [ Str "Pandoc" , Space , Str "Tests" ] ) + , ( "section" , MetaInlines [ Str "1" ] ) + , ( "title" , MetaInlines [ Str "PANDOC-MAN-TESTS" ] ) ] } [ Para diff --git a/test/odt/native/simpleTable.native b/test/odt/native/simpleTable.native --- a/test/odt/native/simpleTable.native +++ b/test/odt/native/simpleTable.native @@ -1,1 +1,1 @@ -[Table ("",[],[]) (Caption Nothing []) [(AlignDefault,ColWidthDefault),(AlignDefault,ColWidthDefault)] (TableHead ("",[],[]) [Row ("",[],[]) [Cell ("",[],[]) AlignDefault (RowSpan 1) (ColSpan 1) [],Cell ("",[],[]) AlignDefault (RowSpan 1) (ColSpan 1) []]]) [TableBody ("",[],[]) (RowHeadColumns 0) [] [Row ("",[],[]) [Cell ("",[],[]) AlignDefault (RowSpan 1) (ColSpan 1) [Plain [Str "Content"]],Cell ("",[],[]) AlignDefault (RowSpan 1) (ColSpan 1) [Plain [Str "More",Space,Str "content"]]]]] (TableFoot ("",[],[]) []),Para []] +[Table ("",[],[]) (Caption Nothing []) [(AlignDefault,ColWidthDefault),(AlignDefault,ColWidthDefault)] (TableHead ("",[],[]) []) [TableBody ("",[],[]) (RowHeadColumns 0) [] [Row ("",[],[]) [Cell ("",[],[]) AlignDefault (RowSpan 1) (ColSpan 1) [Plain [Str "Content"]],Cell ("",[],[]) AlignDefault (RowSpan 1) (ColSpan 1) [Plain [Str "More",Space,Str "content"]]]]] (TableFoot ("",[],[]) []),Para []] diff --git a/test/odt/native/simpleTableWithCaption.native b/test/odt/native/simpleTableWithCaption.native --- a/test/odt/native/simpleTableWithCaption.native +++ b/test/odt/native/simpleTableWithCaption.native @@ -1,1 +1,1 @@ -[Table ("",[],[]) (Caption Nothing [Para [Str "Table",Space,Str "1:",Space,Str "Some",Space,Str "caption",Space,Str "for",Space,Str "a",Space,Str "table"]]) [(AlignDefault,ColWidthDefault),(AlignDefault,ColWidthDefault)] (TableHead ("",[],[]) [Row ("",[],[]) [Cell ("",[],[]) AlignDefault (RowSpan 1) (ColSpan 1) [],Cell ("",[],[]) AlignDefault (RowSpan 1) (ColSpan 1) []]]) [TableBody ("",[],[]) (RowHeadColumns 0) [] [Row ("",[],[]) [Cell ("",[],[]) AlignDefault (RowSpan 1) (ColSpan 1) [Plain [Str "Content"]],Cell ("",[],[]) AlignDefault (RowSpan 1) (ColSpan 1) [Plain [Str "More",Space,Str "content"]]]]] (TableFoot ("",[],[]) []),Para []] +[Table ("",[],[]) (Caption Nothing [Para [Str "Table",Space,Str "1:",Space,Str "Some",Space,Str "caption",Space,Str "for",Space,Str "a",Space,Str "table"]]) [(AlignDefault,ColWidthDefault),(AlignDefault,ColWidthDefault)] (TableHead ("",[],[]) []) [TableBody ("",[],[]) (RowHeadColumns 0) [] [Row ("",[],[]) [Cell ("",[],[]) AlignDefault (RowSpan 1) (ColSpan 1) [Plain [Str "Content"]],Cell ("",[],[]) AlignDefault (RowSpan 1) (ColSpan 1) [Plain [Str "More",Space,Str "content"]]]]] (TableFoot ("",[],[]) []),Para []] diff --git a/test/odt/native/simpleTableWithHeader.native b/test/odt/native/simpleTableWithHeader.native new file mode 100644 --- /dev/null +++ b/test/odt/native/simpleTableWithHeader.native @@ -0,0 +1,79 @@ +[ Table + ( "" , [] , [] ) + (Caption Nothing []) + [ ( AlignDefault , ColWidthDefault ) + , ( AlignDefault , ColWidthDefault ) + , ( AlignDefault , ColWidthDefault ) + ] + (TableHead + ( "" , [] , [] ) + [ Row + ( "" , [] , [] ) + [ Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "A" ] ] + , Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "B" ] ] + , Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "C" ] ] + ] + ]) + [ TableBody + ( "" , [] , [] ) + (RowHeadColumns 0) + [] + [ Row + ( "" , [] , [] ) + [ Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "1" ] ] + , Cell + ( "" , [] , [] ) AlignDefault (RowSpan 1) (ColSpan 1) [ Plain [] ] + , Cell + ( "" , [] , [] ) AlignDefault (RowSpan 1) (ColSpan 1) [ Plain [] ] + ] + , Row + ( "" , [] , [] ) + [ Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "2" ] ] + , Cell + ( "" , [] , [] ) AlignDefault (RowSpan 1) (ColSpan 1) [ Plain [] ] + , Cell + ( "" , [] , [] ) AlignDefault (RowSpan 1) (ColSpan 1) [ Plain [] ] + ] + , Row + ( "" , [] , [] ) + [ Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "3" ] ] + , Cell + ( "" , [] , [] ) AlignDefault (RowSpan 1) (ColSpan 1) [ Plain [] ] + , Cell + ( "" , [] , [] ) AlignDefault (RowSpan 1) (ColSpan 1) [ Plain [] ] + ] + ] + ] + (TableFoot ( "" , [] , [] ) []) +, Para [] +] diff --git a/test/odt/native/simpleTableWithMultipleHeaderRows.native b/test/odt/native/simpleTableWithMultipleHeaderRows.native new file mode 100644 --- /dev/null +++ b/test/odt/native/simpleTableWithMultipleHeaderRows.native @@ -0,0 +1,125 @@ +[ Table + ( "" , [] , [] ) + (Caption Nothing []) + [ ( AlignDefault , ColWidthDefault ) + , ( AlignDefault , ColWidthDefault ) + , ( AlignDefault , ColWidthDefault ) + ] + (TableHead + ( "" , [] , [] ) + [ Row + ( "" , [] , [] ) + [ Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "A" ] ] + , Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "B" ] ] + , Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "C" ] ] + ] + , Row + ( "" , [] , [] ) + [ Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "I" ] ] + , Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "II" ] ] + , Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "II" ] ] + ] + ]) + [ TableBody + ( "" , [] , [] ) + (RowHeadColumns 0) + [] + [ Row + ( "" , [] , [] ) + [ Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "1" ] ] + , Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [] ] + , Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [] ] + ] + , Row + ( "" , [] , [] ) + [ Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "2" ] ] + , Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [] ] + , Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [] ] + ] + , Row + ( "" , [] , [] ) + [ Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [ Str "3" ] ] + , Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [] ] + , Cell + ( "" , [] , [] ) + AlignDefault + (RowSpan 1) + (ColSpan 1) + [ Plain [] ] + ] + ] + ] + (TableFoot ( "" , [] , [] ) []) +, Para [] +] + diff --git a/test/odt/native/tableWithContents.native b/test/odt/native/tableWithContents.native --- a/test/odt/native/tableWithContents.native +++ b/test/odt/native/tableWithContents.native @@ -1,1 +1,1 @@ -[Table ("",[],[]) (Caption Nothing []) [(AlignDefault,ColWidthDefault),(AlignDefault,ColWidthDefault)] (TableHead ("",[],[]) [Row ("",[],[]) [Cell ("",[],[]) AlignDefault (RowSpan 1) (ColSpan 1) [],Cell ("",[],[]) AlignDefault (RowSpan 1) (ColSpan 1) []]]) [TableBody ("",[],[]) (RowHeadColumns 0) [] [Row ("",[],[]) [Cell ("",[],[]) AlignDefault (RowSpan 1) (ColSpan 1) [Plain [Str "A"]],Cell ("",[],[]) AlignDefault (RowSpan 1) (ColSpan 1) [Plain [Str "B"]]],Row ("",[],[]) [Cell ("",[],[]) AlignDefault (RowSpan 1) (ColSpan 1) [Plain [Str "C"]],Cell ("",[],[]) AlignDefault (RowSpan 1) (ColSpan 1) [Plain [Str "D"]]]]] (TableFoot ("",[],[]) []),Para []] +[Table ("",[],[]) (Caption Nothing []) [(AlignDefault,ColWidthDefault),(AlignDefault,ColWidthDefault)] (TableHead ("",[],[]) []) [TableBody ("",[],[]) (RowHeadColumns 0) [] [Row ("",[],[]) [Cell ("",[],[]) AlignDefault (RowSpan 1) (ColSpan 1) [Plain [Str "A"]],Cell ("",[],[]) AlignDefault (RowSpan 1) (ColSpan 1) [Plain [Str "B"]]],Row ("",[],[]) [Cell ("",[],[]) AlignDefault (RowSpan 1) (ColSpan 1) [Plain [Str "C"]],Cell ("",[],[]) AlignDefault (RowSpan 1) (ColSpan 1) [Plain [Str "D"]]]]] (TableFoot ("",[],[]) []),Para []] diff --git a/test/odt/odt/simpleTableWithHeader.odt b/test/odt/odt/simpleTableWithHeader.odt new file mode 100644 Binary files /dev/null and b/test/odt/odt/simpleTableWithHeader.odt differ diff --git a/test/odt/odt/simpleTableWithMultipleHeaderRows.odt b/test/odt/odt/simpleTableWithMultipleHeaderRows.odt new file mode 100644 Binary files /dev/null and b/test/odt/odt/simpleTableWithMultipleHeaderRows.odt differ diff --git a/test/s5-basic.html b/test/s5-basic.html --- a/test/s5-basic.html +++ b/test/s5-basic.html @@ -11,6 +11,9 @@ My S5 Document