packages feed

panpipe (empty) → 0.2.0.0

raw patch · 7 files changed

+449/−0 lines, 7 filesdep +QuickCheckdep +basedep +pandocsetup-changed

Dependencies added: QuickCheck, base, pandoc, pandoc-types, panpipe, process, tasty, tasty-quickcheck, temporary, unix

Files

+ LICENSE view
@@ -0,0 +1,1 @@+Public Domain
+ README.md view
@@ -0,0 +1,302 @@+# PanPipe #++This software is released into the Public Domain+  -- Chris Warburton <chriswarbo@gmail.com>, 2014-09-28++## Usage ##++Tested on GNU/Linux, might work on other POSIX systems.++You'll need some way to run Haskell. Check your package manager or go to+https://www.haskell.org/platform/ to get a compiler or a `runhaskell`+interpreter.++You'll also need Pandoc available as a library, which you can get from your+package manager or with `cabal install pandoc`, and will probably want the+`pandoc` command available too.++To use PanPipe, invoke it as a Pandoc "filter", like this:++`pandoc --filter ./panpipe input_file > output_file`++You can also run `panpipe` as a standalone command, but note that its stdio+should be in PanDoc's JSON format, rather than e.g. raw Markdown. You can+convert *to* PanDoc JSON using a command like `pandoc -t json`, and convert+*from* PanDoc JSON using `pandoc -f json`. The `--filter` argument automates+this plumbing.++## Intro ##++PanPipe is a simple Haskell script using PanDoc. It allows code blocks in+PanDoc-compatible documents, eg. Markdown, to be sent to external programs for+processing.++Any code blocks or lines with a "pipe" attribute will have the contents of that+attribute executed as a shell command. The body of the block/line will be piped+to that command's stdin, and the stdout will replace the body of that+block/line. A non-zero exit code will cause PanPipe to exit with that code;+stderr will be sent to PanPipe's stderr.++For example, we can execute shell scripts by piping them to "sh":++````+```{pipe="sh"}+echo "Hello world"+```+````++This will cause "sh" to be called, with 'echo "Hello world"' as its stdin.+It will execute the echo command, to produce 'Hello world' as its stdout. This+will become the new contents of the code block, so in the resulting document+this code block will be replaced by:++````+```+Hello world+```+````++## Usage Notes ##++### Attributes ###++The "pipe" attribute is removed, but other attributes, classes and IDs remain:++````+```{#foo .bar baz="quux" pipe="sh"}+echo 'Hello'+```+````++Will become:++````+```{#foo .bar baz="quux"}+Hello+```+````++### Execution Order ###++PanPipe uses two passes: in the first, all code *blocks* are executed, in the+order they appear in the document. Hence later blocks can rely on the effects of+earlier ones. For example:++````+```{pipe="sh"}+echo "123" > /tmp/blah+echo "hello"+```++```{pipe="sh"}+cat /tmp/blah+```+````++Will become:++````+```+hello+```++```+123+```+````++The second pass executes *inline* code, in the order they appear in the+document.++### Environment ###++Commands will inherit the environment from the shell which calls panpipe, except+they will all be executed in a temporary directory. This makes it easier to+share data between code, without leaving cruft behind:++````+```{pipe="sh"}+echo "hello world" > file1+echo "done"+```++```{pipe="sh"}+cat file1+```+````++Will become:++````+```+done+```++```+hello world+```+````++The temporary directory will contain a symlink called "root" which points to+wherever Pandoc was called from. This allows resources to be shared across+invocations (although it's not recommended to *modify* anything in root).++### Imperative Blocks ###++If you want to execute a block for some effect, but ignore its output, you can+hide the result using a class or attribute:++````+```{.hidden pipe="python -"}+import random+with open('entropy', 'w') as f:+    f.write(str(random.randint(0, 100)))+```+````++When rendered to HTML will produce:++```html+<pre class="hidden"><code></code></pre>+```++### Program Listings ###++A common use-case is to include a program listing in a document *and* show the+results of executing it. You can do this by passing the source code to the Unix+"tee" command, then using a subsequent shell script to run it:++````+```{.python pipe="tee script1.py"}+print "Foo bar baz"+```++```{pipe="sh"}+python script1.py+```+````++Will become:++````+```{.python}+print "Foo bar baz"+```++```+Foo bar baz+```+````++### Changing Block Order ###++Blocks will always be executed in document-order, so you must arrange dependent+blocks appropriately. However, we can display blocks in any order by saving them+to files and dumping them later.++For example, to show the output of a program *before* its source code listing,+we can define the program first, using "tee" to save it to a file and a HTML+class to hide the listing in the resulting document:++````+```{.hidden pipe="tee script2.py"}+print "Hello world"+```+````++Next we can include a block which executes the file we created:++````+```{pipe="sh"}+python script2.py+```+````++Finally we can include a listing by having a block dump the contents of the file+(using ".python" for syntax highlighting):++````+```{.python pipe="sh"}+cat script2.py+```+````++### Inline Snippets ###++PanPipe also works on inline code snippets; for example, my root filesystem is+currently at `` `df -h | grep "/$" | grep -o "[0-9]*%"`{pipe="sh"} `` capacity.++### PanHandle ###++PanPipe keeps the results of script execution inside code blocks/lines, where+they can't interfere with the formatting. If you want to splice some of these+results back into the document, you can use the PanHandle script which was+written to complement PanPipe.++To prevent ambiguity, PanHandle requires data to be in PanDoc's JSON format. We+can convert things to that format using the `-t json` option to the `pandoc`+command. For example, to generate a Markdown list and insert it into the+document, we can do this:++````+```{.unwrap pipe="python - | pandoc -t json"}+for n in range(5):+    print " - Element " + str(n)+```+````++If we run this document through PanPipe, the Python code will output the+following to its stdout:++```+ - Element 0+ - Element 1+ - Element 2+ - Element 3+ - Element 4+```++This will be transformed by `pandoc -t json` into the following:++```+[{"unMeta":{}},[{"t":"BulletList","c":[[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"0"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"1"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"2"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"3"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"4"}]}]]}]]+```++Hence PanPipe will end up giving out a document equivalent to the following:++````+``` {.unwrap}+[{"unMeta":{}},[{"t":"BulletList","c":[[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"0"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"1"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"2"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"3"}]}],[{"t":"Plain","c":[{"t":"Str","c":"Element"},{"t":"Space","c":[]},{"t":"Str","c":"4"}]}]]}]]+```+````++Running *that* code through PanHandle will splice the contents into the document+to give (a JSON equivalent of):++```+ - Element 0+ - Element 1+ - Element 2+ - Element 3+```++Note that this is no longer in a codeblock, so it will render like this:++ - Element 0+ - Element 1+ - Element 2+ - Element 3++Hence we can use PanPipe to obtain or generate data, and PanHandle to splice it+into the document in a sensible way. For example, it's easy to include one+Markdown document inside another:++````+```{.unwrap pipe="sh"}+cat /some/file.md | pandoc -t json+```++```{.unwrap pipe="sh"}+wget -O- http://some.site/some/markdown | pandoc -t json+```+````
+ Setup.hs view
@@ -0,0 +1,2 @@+import Distribution.Simple+main = defaultMain
+ panpipe.cabal view
@@ -0,0 +1,47 @@+name:                panpipe+version:             0.2.0.0+synopsis:            Pandoc filter to execute code blocks+description:         Executes pandoc code blocks annotated with 'pipe'+homepage:            http://chriswarbo.net/essays/activecode+license:             PublicDomain+license-file:        LICENSE+author:              Chris Warburton+maintainer:          chriswarbo@gmail.com+category:            Text+build-type:          Simple+extra-source-files:  README.md+cabal-version:       >=1.10++source-repository head+  type:     git+  location: http://chriswarbo.net/git/panpipe.git+                     +library+  hs-source-dirs:      src+  default-language:    Haskell2010+  exposed-modules:     PanPipe+  build-depends:       base         >=4.7  && <4.10+                     , unix         >=2.7  && <2.8+                     , process      >=1.2  && <1.7+                     , pandoc       >=1.17 && <1.18+                     , pandoc-types+                     , temporary++executable panpipe+  hs-source-dirs:      panpipe+  main-is:             Main.hs+  default-language:    Haskell2010+  build-depends:       base >=4.7 && < 4.10+                     , panpipe++test-suite test+  default-language:    Haskell2010+  type:                exitcode-stdio-1.0+  hs-source-dirs:      test+  main-is:             Main.hs+  build-depends:       base >=4.7 && <4.10+                     , panpipe+                     , QuickCheck+                     , tasty >= 0.7+                     , tasty-quickcheck+                     , pandoc
+ panpipe/Main.hs view
@@ -0,0 +1,4 @@+module Main where+import PanPipe++main = panpipeMain
+ src/PanPipe.hs view
@@ -0,0 +1,51 @@+module PanPipe where++import Control.Applicative+import Data.List+import System.Exit+import System.IO+import System.IO.Temp (withSystemTempDirectory)+import System.Posix+import System.Process+import Text.Pandoc+import Text.Pandoc.JSON   (toJSONFilter)+import Text.Pandoc.Shared (inDirectory)+import Text.Pandoc.Walk   (walkM)++pipeBWith :: (Functor m, Monad m) => (String -> String -> m String)+                                  -> Block+                                  -> m Block+pipeBWith f (CodeBlock as s)+          |  Just (as', p) <- partPipes as = CodeBlock as' <$> f p s+pipeBWith f x = walkM (pipeIWith f) x++pipeB = pipeBWith readShell++pipeIWith :: (Functor m, Monad m) => (String -> String -> m String)+                                  -> Inline+                                  -> m Inline+pipeIWith f (Code as s)+          |  Just (as', p) <- partPipes as = Code as' <$> f p s+pipeIWith f x = return x++pipeI = pipeIWith readShell++readShell :: FilePath -> String -> IO String+readShell p = readProcess "sh" ["-c", p]++partPipes :: Attr -> Maybe (Attr, String)+partPipes (x, y, zs) = case partition (("pipe" ==) . fst) zs of+                            ((_, p):_, zs') -> Just ((x, y, zs'), p)+                            _               -> Nothing++transform :: Pandoc -> IO Pandoc+transform doc = do cwd <-getWorkingDirectory+                   withSystemTempDirectory+                       "panpipe"+                       (\dir -> do createSymbolicLink cwd (dir ++ "/root")+                                   inDirectory dir (transformDoc doc))++transformDoc :: Pandoc -> IO Pandoc+transformDoc = walkM pipeB++panpipeMain = toJSONFilter transform
+ test/Main.hs view
@@ -0,0 +1,42 @@+import           Control.Applicative+import           Data.Maybe+import           PanPipe+import           Text.Pandoc+import           Test.Tasty (defaultMain, testGroup)+import           Test.Tasty.QuickCheck (testProperty)++pipeBPass p s a1 a2 = let at x  = (undefined, undefined, a1 ++ x ++ a2)+                          f a b = [a ++ b]+                          [CodeBlock _ lhs] = pipeBWith f (CodeBlock (at [("pipe", p)]) s)+                      in  lhs == p ++ s++pipeBAttr i c a1 a2 = let at x = (i, c, a1 ++ x ++ a2)+                          [CodeBlock as _] = pipeBWith (\_ _ -> [undefined])+                                                       (CodeBlock (at [("pipe", undefined)])+                                                                  undefined)+                      in  at [] == as++pipeIPass p s a1 a2 = let at x  = (undefined, undefined, a1 ++ x ++ a2)+                          f a b = [a ++ b]+                          [Code _ lhs] = pipeIWith f (Code (at [("pipe", p)]) s)+                      in  lhs == p ++ s++pipeIAttr i c a1 a2 = let at x = (i, c, a1 ++ x ++ a2)+                          [Code as _] = pipeIWith (\_ _ -> [undefined])+                                                  (Code (at [("pipe", undefined)])+                                                        undefined)+                      in at [] == as++nonPipe as = isNothing $ partPipes (undefined, undefined, filter ((/= "pipe") . fst) as)++pipeClass p as = (snd <$> partPipes (undefined, undefined, ("pipe", p):as)) == Just p++main = defaultMain $ testGroup "All tests"+       [+         testProperty "pipeB passes pipe and stdin properly" pipeBPass+       , testProperty "pipeB leaves attributes intact"       pipeBAttr+       , testProperty "pipeI passes pipe and stdin properly" pipeIPass+       , testProperty "pipeI leaves attributes intact"       pipeIAttr+       , testProperty "non-pipes ignored"                    nonPipe+       , testProperty "pipe class found"                     pipeClass+       ]