pandoc-plot 0.9.2.0 → 0.9.3.0

raw patch · 29 files changed

+138/−94 lines, 29 filesPVP: major bump suggested

API removals or changes: PVP suggests a major version bump

API changes (from Hackage documentation)

Files

• CHANGELOG.md +8/−0
• MANUAL.md +1/−1
• README.md +1/−3
• docs/MANUAL.html +3/−3
• executable/Main.hs +2/−1
• pandoc-plot.cabal +17/−2
• src/Text/Pandoc/Filter/Plot.hs +2/−39
• src/Text/Pandoc/Filter/Plot/Monad.hs +50/−10
• src/Text/Pandoc/Filter/Plot/Monad/Types.hs +14/−2
• src/Text/Pandoc/Filter/Plot/Renderers.hs +1/−1
• src/Text/Pandoc/Filter/Plot/Renderers/Prelude.hs +2/−31
• src/Text/Pandoc/Filter/Plot/Scripting.hs +3/−0
• tests/Common.hs +19/−1
• tests/Main.hs +2/−0
• tests/fixtures/.config-meta.yml +3/−0
• tests/fixtures/.empty-config.yml +0/−0
• tests/fixtures/.verbose-config.yml +3/−0
• tests/fixtures/plotfile.py +3/−0
• tests/includes/bokeh.py +0/−0
• tests/includes/ggplot2.r +0/−0
• tests/includes/gnuplot.gp +0/−0
• tests/includes/graphviz.dot +1/−0
• tests/includes/mathplot.m +0/−0
• tests/includes/matlabplot.m +1/−0
• tests/includes/matplotlib.py +1/−0
• tests/includes/octave.m +0/−0
• tests/includes/plotly-python.py +1/−0
• tests/includes/plotly-r.r +0/−0
• tests/includes/plotsjl.jl +0/−0

@@ -2,6 +2,14 @@ 
 pandoc-plot uses [Semantic Versioning](http://semver.org/spec/v2.0.0.html)
 
+Release 0.9.3.0
+---------------
+
+* Added executable caching: repeated useage of a particular toolkit will be faster because executables are only looked-for once.
+* Reverting the change from 0.8.1.0: internal machinery of `pandoc-plot` has been moved to the `Text.Pandoc.Filter.Plot.Internal` module, where there is no guarantee of backwards-compatibility after 1.0.0.
+* Removed the `makePlot` function, which could not take advantage of multithreading and other key features of `pandoc-plot`.
+* Fixed an issue where files required for tests were missing from source tarballs.
+
 Release 0.9.2.0
 ---------------
 

@@ -159,7 +159,7 @@ `pandoc-plot` is a command line executable with a few functions. You can take a look at the help using the `-h`/`--help` flag:
 
 ``` bash
-pandoc-plot 0.9.2.0 - generate figures directly in documents
+pandoc-plot 0.9.3.0 - generate figures directly in documents
 
 Usage: pandoc-plot.EXE [(-v|--version) | --full-version | (-m|--manual)] 
                        [COMMAND] [AST]

@@ -2,7 +2,7 @@ 
 ## A Pandoc filter to generate figures from code blocks in documents
 
-[![Hackage version](https://img.shields.io/hackage/v/pandoc-plot.svg)](http://hackage.haskell.org/package/pandoc-plot) [![Stackage version (latest long-term service)](http://stackage.org/package/pandoc-plot/badge/lts)](http://stackage.org/lts/package/pandoc-plot) [![Stackage version (nightly)](http://stackage.org/package/pandoc-plot/badge/nightly)](http://stackage.org/nightly/package/pandoc-plot) [![Conda Version](https://img.shields.io/conda/vn/conda-forge/pandoc-plot.svg)](https://anaconda.org/conda-forge/pandoc-plot) [![license](https://img.shields.io/badge/license-GPLv2+-lightgray.svg)](https://www.gnu.org/licenses/gpl.html) 
+[![Hackage version](https://img.shields.io/hackage/v/pandoc-plot.svg)](http://hackage.haskell.org/package/pandoc-plot) [![Stackage version (latest long-term service)](http://stackage.org/package/pandoc-plot/badge/lts)](http://stackage.org/lts/package/pandoc-plot) [![Conda Version](https://img.shields.io/conda/vn/conda-forge/pandoc-plot.svg)](https://anaconda.org/conda-forge/pandoc-plot) [![license](https://img.shields.io/badge/license-GPLv2+-lightgray.svg)](https://www.gnu.org/licenses/gpl.html) 
 
 `pandoc-plot` turns code blocks present in your documents (Markdown, LaTeX, etc.) into embedded figures, using your plotting toolkit of choice, including Matplotlib, ggplot2, MATLAB, Mathematica, and more.
 
@@ -67,8 +67,6 @@ ``` bash
 pandoc-plot toolkits
 ```
-
-The `toolkits` command is described in its own section below.
 
 **Wish your plotting toolkit of choice was available? Please [raise an
 issue](https://github.com/LaurentRDC/pandoc-plot/issues)\!**

@@ -4,7 +4,7 @@   <meta charset="utf-8" />
   <meta name="generator" content="pandoc" />
   <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
-  <title>pandoc-plot 0.9.2.0 manual</title>
+  <title>pandoc-plot 0.9.3.0 manual</title>
   <style>
     code{white-space: pre-wrap;}
     span.smallcaps{font-variant: small-caps;}
@@ -83,7 +83,7 @@ </head>
 <body>
 <header id="title-block-header">
-<h1 class="title">pandoc-plot 0.9.2.0 manual</h1>
+<h1 class="title">pandoc-plot 0.9.3.0 manual</h1>
 </header>
 <!--
 The file MANUAL.md is automatically generated by the tools/mkmanual.ps1 script. Do not edit manually.
@@ -187,7 +187,7 @@ <div class="sourceCode" id="cb10"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb10-1"><a href="#cb10-1" aria-hidden="true"></a><span class="ex">pandoc</span> --filter pandoc-plot --filter pandoc-crossref -i myfile.md -o myfile.html</span></code></pre></div>
 <h2 id="detailed-usage">Detailed usage</h2>
 <p><code>pandoc-plot</code> is a command line executable with a few functions. You can take a look at the help using the <code>-h</code>/<code>--help</code> flag:</p>
-<div class="sourceCode" id="cb11"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb11-1"><a href="#cb11-1" aria-hidden="true"></a><span class="ex">pandoc-plot</span> 0.9.2.0 - generate figures directly in documents</span>
+<div class="sourceCode" id="cb11"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb11-1"><a href="#cb11-1" aria-hidden="true"></a><span class="ex">pandoc-plot</span> 0.9.3.0 - generate figures directly in documents</span>
 <span id="cb11-2"><a href="#cb11-2" aria-hidden="true"></a></span>
 <span id="cb11-3"><a href="#cb11-3" aria-hidden="true"></a><span class="ex">Usage</span>: pandoc-plot.EXE [(-v<span class="kw">|</span><span class="ex">--version</span>) <span class="kw">|</span> <span class="ex">--full-version</span> <span class="kw">|</span> <span class="kw">(</span><span class="ex">-m</span><span class="kw">|</span><span class="ex">--manual</span><span class="kw">)</span>] </span>
 <span id="cb11-4"><a href="#cb11-4" aria-hidden="true"></a>                       [<span class="ex">COMMAND</span>] [AST]</span>

@@ -30,7 +30,8 @@                                                    plotTransform,
                                                    defaultConfiguration, 
                                                    configuration, Configuration(..),
-                                                   pandocPlotVersion, cls, 
+                                                   pandocPlotVersion, )
+import           Text.Pandoc.Filter.Plot.Internal (cls, 
                                                    supportedSaveFormats, 
                                                    toolkits, readDoc, 
                                                    cleanOutputDirs, 

@@ -1,6 +1,6 @@ cabal-version:  2.2
 name:           pandoc-plot
-version:        0.9.2.0
+version:        0.9.3.0
 synopsis:       A Pandoc filter to include figures generated from code blocks using your plotting toolkit of choice.
 description:    A Pandoc filter to include figures generated from code blocks. Keep the document and code in the same location. Output is captured and included as a figure.
 category:       Text
@@ -21,6 +21,21 @@     docs/MANUAL.html
     stack.yaml
     example-config.yml
+    tests/fixtures/.config-meta.yml
+    tests/fixtures/.empty-config.yml
+    tests/fixtures/.verbose-config.yml
+    tests/fixtures/plotfile.py
+    tests/includes/bokeh.py
+    tests/includes/ggplot2.r
+    tests/includes/gnuplot.gp
+    tests/includes/graphviz.dot
+    tests/includes/mathplot.m
+    tests/includes/matlabplot.m
+    tests/includes/matplotlib.py
+    tests/includes/octave.m
+    tests/includes/plotly-python.py
+    tests/includes/plotly-r.r
+    tests/includes/plotsjl.jl
 
 source-repository head
     type: git
@@ -35,9 +50,9 @@ library
     exposed-modules:
         Text.Pandoc.Filter.Plot
+        Text.Pandoc.Filter.Plot.Internal
     other-modules:
         Paths_pandoc_plot
-        Text.Pandoc.Filter.Plot.Internal
         Text.Pandoc.Filter.Plot.Clean
         Text.Pandoc.Filter.Plot.Configuration
         Text.Pandoc.Filter.Plot.Parse

@@ -77,10 +77,8 @@ @
 -}
 module Text.Pandoc.Filter.Plot (
-    -- * Operating on single Pandoc blocks
-      makePlot
     -- * Operating on whole Pandoc documents
-    , plotTransform
+      plotTransform
     -- * Cleaning output directories
     , cleanOutputDirs
     -- * Runtime configuration
@@ -99,26 +97,10 @@     , supportedSaveFormats
     -- * Version information
     , pandocPlotVersion
-    -- * For testing and internal purposes ONLY
-    -- ** Running the @PlotM@ monad
+    -- * For embedding, testing and internal purposes ONLY. Might change without notice.
     , make
     , makeEither
-    , PlotM
-    , runPlotM
     , PandocPlotError(..)
-    -- ** Code block parameters
-    , InclusionKey(..)
-    , inclusionKeys
-    -- ** Utilities
-    , extension
-    , readDoc
-    , captionReader
-    , cls
-    , configurationPathMeta
-    , executable
-    , Executable(..)
-    -- ** Embedding HTML content
-    , extractPlot
     ) where
 
 import Control.Concurrent.Async.Lifted   (mapConcurrently)
@@ -128,7 +110,6 @@ import Paths_pandoc_plot                 (version)
 
 import Text.Pandoc.Definition            (Pandoc(..), Block)
-import Text.Pandoc.Walk                  (walkM, Walkable)
 
 import Text.Pandoc.Filter.Plot.Internal
 
@@ -143,24 +124,6 @@               -> IO Pandoc
 plotTransform conf (Pandoc meta blocks) = 
     runPlotM conf $ mapConcurrently make blocks >>= return . Pandoc meta
-
-
--- | Highest-level function that can be walked over a Pandoc tree.
--- All code blocks that have the appropriate class names will be considered
--- figures, e.g. @.matplotlib@.
---
--- This function can be made to operation on whole @Pandoc@ documents. However,
--- you should prefer the @plotTransform@ function for whole documents, as it
--- is optimized for parallel operations.
---
--- Failing to render a figure does not stop the filter, so that you may run the filter
--- on documents without having all necessary toolkits installed. In this case, error
--- messages are printed to stderr, and blocks are left unchanged.
-makePlot :: Walkable Block a
-         => Configuration -- ^ Configuration for default values
-         -> a             -- ^ Input block or document
-         -> IO a
-makePlot conf = runPlotM conf . walkM make
 
 
 -- | The version of the pandoc-plot package.

@@ -19,6 +19,8 @@     , runCommand
     -- * Getting file hashes
     , fileHash
+    -- * Getting executables
+    , executable
     -- * Logging
     , Verbosity(..)
     , LogSink(..)
@@ -53,7 +55,7 @@ import           Data.Text.Encoding          (decodeUtf8With)
 import           Data.Text.Encoding.Error    (lenientDecode)
 
-import           System.Directory            (doesFileExist, getModificationTime)
+import           System.Directory            (doesFileExist, getModificationTime, findExecutable)
 import           System.Exit                 (ExitCode (..))
 import           System.Process.Typed        ( readProcessStderr, shell, nullStream
                                              , setStdout, setStderr, byteStringOutput
@@ -90,7 +92,8 @@ -- | Evaluate a @PlotM@ action.
 runPlotM :: Configuration -> PlotM a -> IO a
 runPlotM conf v = do
-    st <- newMVar mempty
+    st <- PlotState <$> newMVar mempty
+                    <*> newMVar mempty
     let verbosity = logVerbosity conf
         sink      = logSink conf 
     withLogger verbosity sink $ 
@@ -113,8 +116,7 @@     logger <- asks envLogger
     when (v >= lVerbosity logger) $ 
         liftIO $ do
-            let lines' = [l' | l' <- T.lines t]
-            forM_ lines' $ \l -> writeChan (lChannel logger) (Just (h <> l <> "\n"))
+            forM_ ( T.lines t) $ \l -> writeChan (lChannel logger) (Just (h <> l <> "\n"))
 
 
 -- | Run a command within the @PlotM@ monad. Stderr stream
@@ -150,22 +152,27 @@     return (ec, processOutput)
 
 
--- Plot state consists of a map of filepaths to hashes
+-- Plot state is used for caching.
+-- One part consists of a map of filepaths to hashes
 -- This allows multiple plots to depend on the same file/directory, and the file hashes
 -- will only be calculated once. This is OK because pandoc-plot will not run for long.
 -- We note that because figures are rendered possibly in parallel, access to 
 -- the state must be synchronized; otherwise, each thread might compute its own
 -- hashes.
+-- The other part is comprised of a map of toolkits to executables (possibly missing)
+-- This means that executable will be search for only once.
 type FileHash  = Word
-type PlotState = MVar (Map FilePath FileHash)
+data PlotState = 
+    PlotState (MVar (Map FilePath FileHash))
+              (MVar (Map Toolkit (Maybe Executable)))
 
 
 -- | Get a filehash. If the file hash has been computed before,
 -- it is reused. Otherwise, the filehash is calculated and stored.
 fileHash :: FilePath -> PlotM FileHash
 fileHash path = do
-    var <- get
-    hashes <- liftIO $ takeMVar var
+    PlotState varHashes varExes <- get
+    hashes <- liftIO $ takeMVar varHashes
     (fh, hashes') <- case M.lookup path hashes of
         Nothing -> do
             debug $ mconcat ["Calculating hash of dependency ", pack path]
@@ -175,8 +182,8 @@         Just h -> do
             debug $ mconcat ["Hash of dependency ", pack path, " already calculated."]
             return (h, hashes)
-    liftIO $ putMVar var hashes'
-    put var
+    liftIO $ putMVar varHashes hashes'
+    put $ PlotState varHashes varExes
     return fh
     where
     -- As a proxy for the state of a file dependency, we use the modification time
@@ -187,6 +194,39 @@         if fileExists
             then liftIO . fmap (fromIntegral . hash . show) . getModificationTime $ fp
             else err (mconcat ["Dependency ", pack fp, " does not exist."]) >> return 0 
+
+
+-- | Get an executable. If the executable has not been used before, 
+-- find it and store where it is. It will be re-used.
+executable :: Toolkit -> PlotM (Maybe Executable)
+executable tk = do
+    name <- exeSelector tk
+    PlotState varHashes varExes <- get
+    exes <- liftIO $ takeMVar varExes
+    (exe', exes') <- case M.lookup tk exes of
+        Nothing -> do
+            debug $ mconcat ["Looking for executable \"", pack name, "\" for ", pack $ show tk]
+            exe' <- liftIO $ findExecutable name >>= return . fmap exeFromPath
+            let exes' = M.insert tk exe' exes
+            return (exe', exes')
+        Just e -> do
+            debug $ mconcat ["Executable \"", pack name, "\" already found."]
+            return (e, exes)
+    liftIO $ putMVar varExes exes'
+    put $ PlotState varHashes varExes
+    return exe'
+    where
+        exeSelector Matplotlib   = asksConfig matplotlibExe  
+        exeSelector PlotlyPython = asksConfig plotlyPythonExe 
+        exeSelector PlotlyR      = asksConfig plotlyRExe      
+        exeSelector Matlab       = asksConfig matlabExe       
+        exeSelector Mathematica  = asksConfig mathematicaExe  
+        exeSelector Octave       = asksConfig octaveExe       
+        exeSelector GGPlot2      = asksConfig ggplot2Exe      
+        exeSelector GNUPlot      = asksConfig gnuplotExe      
+        exeSelector Graphviz     = asksConfig graphvizExe     
+        exeSelector Bokeh        = asksConfig bokehExe         
+        exeSelector Plotsjl      = asksConfig plotsjlExe 
 
 
 -- | The @Configuration@ type holds the default values to use

@@ -25,6 +25,8 @@     , extension
     , toolkits
     , inclusionKeys
+    , Executable(..)
+    , exeFromPath
     -- Utilities
     , isWindows
 ) where
@@ -32,10 +34,11 @@ import           Data.Char              (toLower)
 import           Data.List              (intersperse)
 import           Data.String            (IsString (..))
-import           Data.Text              (Text)
+import           Data.Text              (Text, pack)
 import           Data.Yaml
 
 import           GHC.Generics           (Generic)
+import           System.FilePath        (splitFileName)
 import           System.Info            (os)
 
 import           Text.Pandoc.Definition (Attr)
@@ -58,7 +61,7 @@     | Graphviz
     | Bokeh
     | Plotsjl
-    deriving (Bounded, Eq, Enum, Generic)
+    deriving (Bounded, Eq, Enum, Generic, Ord)
 
 
 -- | This instance should only be used to display toolkit names
@@ -89,6 +92,15 @@ cls Graphviz     = "graphviz"
 cls Bokeh        = "bokeh"
 cls Plotsjl      = "plotsjl"
+
+
+-- | Executable program and directory where it can be found.
+data Executable = Executable FilePath Text
+
+
+exeFromPath :: FilePath -> Executable
+exeFromPath fp = let (dir, name) = splitFileName fp
+                 in Executable dir (pack name)
 
 
 -- | Source context for plotting scripts

@@ -48,7 +48,7 @@ import Text.Pandoc.Filter.Plot.Renderers.Graphviz
 import Text.Pandoc.Filter.Plot.Renderers.Bokeh
 import Text.Pandoc.Filter.Plot.Renderers.Plotsjl
-import Text.Pandoc.Filter.Plot.Renderers.Prelude     (executable, Executable(..), OutputSpec(..))
+import Text.Pandoc.Filter.Plot.Renderers.Prelude     (OutputSpec(..))
 
 import Text.Pandoc.Filter.Plot.Monad
 

@@ -18,31 +18,23 @@     , unpack
     , commandSuccess
     , existsOnPath
-    , executable
     , OutputSpec(..)
-    , Executable(..)
     , appendCapture
     , toRPath
 ) where
 
 import           Data.Maybe                    (isJust)
-import           Data.Text                     (Text, unpack, pack)
+import           Data.Text                     (Text, unpack)
 
 import           System.Directory              (findExecutable)
-import           System.FilePath               (splitFileName, isPathSeparator)
+import           System.FilePath               (isPathSeparator)
 import           System.Exit                   (ExitCode(..))
 
 import           Text.Shakespeare.Text         (st)
 
 import           Text.Pandoc.Filter.Plot.Monad
 
-data Executable = Executable FilePath Text
 
-
-exeFromPath :: FilePath -> Executable
-exeFromPath fp = let (dir, name) = splitFileName fp
-                 in Executable dir (pack name)
-
 -- | Check that the supplied command results in
 -- an exit code of 0 (i.e. no errors)
 commandSuccess :: FilePath -- Directory from which to run the command
@@ -56,27 +48,6 @@ -- | Checks that an executable is available on path, at all.
 existsOnPath :: FilePath -> IO Bool
 existsOnPath fp = findExecutable fp >>= fmap isJust . return
-
-
--- | Try to find the executable and normalise its path.
--- If it cannot be found, it is left unchanged - just in case.
-tryToFindExe :: String -> IO (Maybe Executable)
-tryToFindExe fp = findExecutable fp >>= return . fmap exeFromPath
-
-
--- | Path to (directory, executable) of a toolkit. 
-executable :: Toolkit -> PlotM (Maybe Executable)
-executable Matplotlib   = asksConfig matplotlibExe   >>= liftIO . tryToFindExe 
-executable PlotlyPython = asksConfig plotlyPythonExe >>= liftIO . tryToFindExe 
-executable PlotlyR      = asksConfig plotlyRExe      >>= liftIO . tryToFindExe 
-executable Matlab       = asksConfig matlabExe       >>= liftIO . tryToFindExe 
-executable Mathematica  = asksConfig mathematicaExe  >>= liftIO . tryToFindExe 
-executable Octave       = asksConfig octaveExe       >>= liftIO . tryToFindExe 
-executable GGPlot2      = asksConfig ggplot2Exe      >>= liftIO . tryToFindExe 
-executable GNUPlot      = asksConfig gnuplotExe      >>= liftIO . tryToFindExe 
-executable Graphviz     = asksConfig graphvizExe     >>= liftIO . tryToFindExe 
-executable Bokeh        = asksConfig bokehExe        >>= liftIO . tryToFindExe  
-executable Plotsjl      = asksConfig plotsjlExe      >>= liftIO . tryToFindExe 
 
 
 -- | A shortcut to append capture script fragments to scripts

@@ -91,6 +91,9 @@             target <- figurePath spec
 
             -- Check if executable is present
+            -- Note that checking if the toolkit if fully configured is much more involved,
+            -- and so we only check if the toolkit is appropriately installed if there is
+            -- an error.
             exe <- executable toolkit
             case exe of
                 Nothing -> error $ "Toolkit " <> show toolkit <> " is not installed."

@@ -15,6 +15,7 @@ import           Test.Tasty.HUnit
 
 import           Text.Pandoc.Filter.Plot
+import           Text.Pandoc.Filter.Plot.Internal
 
 import qualified Text.Pandoc.Builder              as B
 import qualified Text.Pandoc.Definition           as B
@@ -32,7 +33,7 @@ 
 defaultTestConfig :: Configuration
 defaultTestConfig = defaultConfiguration { logVerbosity = Silent
-                                         , logSink = StdErr --LogFile "test_log.txt"
+                                         , logSink = StdErr
                                          }
 
 -------------------------------------------------------------------------------
@@ -48,6 +49,23 @@         _ <- runPlotM defaultTestConfig $ make cb
         filesCreated <- length <$> listDirectory tempDir
         assertEqual "" 2 filesCreated
+
+
+-------------------------------------------------------------------------------
+-- Test that plot files and source files are created when the filter is run
+-- and the path of the files involves spaces. See Issue #2
+testFileCreationPathWithSpaces :: Toolkit -> TestTree
+testFileCreationPathWithSpaces tk =
+    testCase "writes output files in appropriate directory (with spaces)" $ do
+        let postfix = unpack . cls $ tk
+        tempDir <- (</> "test-file-creation-with-spaces-   -" <> postfix) <$> getTemporaryDirectory
+        ensureDirectoryExistsAndEmpty tempDir
+
+        let cb = (addDirectory tempDir $ codeBlock tk (trivialContent tk))
+        _ <- runPlotM defaultTestConfig $ make cb
+        filesCreated <- length <$> listDirectory tempDir
+        assertEqual "" 2 filesCreated
+
 
 -------------------------------------------------------------------------------
 -- Test that included files are found within the source

@@ -16,6 +16,7 @@ 
 import           Common
 import           Text.Pandoc.Filter.Plot
+import           Text.Pandoc.Filter.Plot.Internal
 
 main :: IO ()
 main = do
@@ -50,6 +51,7 @@ toolkitSuite tk =
     testGroup (show tk) $
         [ testFileCreation
+        , testFileCreationPathWithSpaces
         , testFileInclusion
         , testSaveFormat
         , testWithSource

@@ -0,0 +1,3 @@+# Just not default values
+dpi: 1000
+directory: ".."

@@ -0,0 +1,3 @@+logging:
+  verbosity: debug
+  filepath: log.txt

@@ -0,0 +1,3 @@+import matplotlib.pyplot as plt
+
+plt.plot([1,2,3,4,5], [5,6,7,8,9])

@@ -0,0 +1,1 @@+// This is a comment in DOT

@@ -0,0 +1,1 @@+% Matlab content

@@ -0,0 +1,1 @@+# matplotlib content

@@ -0,0 +1,1 @@+# plotly content