pandoc-markdown-ghci-filter (empty) → 0.1.0.0

raw patch · 8 files changed

+542/−0 lines, 8 filesdep +QuickCheckdep +aesondep +basesetup-changed

Dependencies added: QuickCheck, aeson, base, containers, ghcid, pandoc, pandoc-markdown-ghci-filter, pandoc-types, tasty, tasty-hunit, tasty-quickcheck, text

Files

• ChangeLog.md +3/−0
• LICENSE +21/−0
• README.md +235/−0
• Setup.hs +2/−0
• app/Main.hs +12/−0
• pandoc-markdown-ghci-filter.cabal +89/−0
• src/CodeBlockExecutor.hs +114/−0
• test/Spec.hs +66/−0

@@ -0,0 +1,3 @@+# Changelog for learn-aeson++## Unreleased changes

@@ -0,0 +1,21 @@+MIT License++Copyright (c) 2019 Guru Devanla++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+SOFTWARE.

@@ -0,0 +1,235 @@+pandoc-markdown-ghci-filter+===========================++A `Pandoc` filter that identifies code blocks(`Haskell`) in Pandoc+supported formats, executes the code in GHCI and embeds the results in+the returned output by updating the AST provided by `Pandoc`.++Quick Overview+==============++Often a markdown(or any `pandoc` supported document) for any `Haskell`+related documentation or a technical blog post involves `code` blocks.+The `code` block could include `definitions` and also a demonstration of+output with an `interactive` prompt. For, example, take this `code`+block:++``` {.haskell code-filter="Off"}+-- README.md++-- definition+increment:: Integer -> Integer+increment x = x + 1++-- interactive prompt to demonstrate the working of definitions so far++>> increment 41+```++It would be nice if this `code` block was automatically evaluated and+`output` of `increment 41` is automatically recorded below+`>> increment 41`, as follows:++``` {.haskell}+-- README.md++-- definition+increment:: Integer -> Integer+increment x = x + 1++-- interactive prompt to demonstrate the working of definitions so far++>> increment 41+42++```++Notice, that the `42` is automatically populated by this filter while+transforming the original document.++To transform the document, we need to run the document through the+`pandoc` filter, as follows:++``` {.shell}++-- set up pandoc_filter to the executable of this program (see Installation)++pandoc -s -t json README.md | pandoc_filter | pandoc -f json -t markdown+```++To read more about how `filter` work, visit the+[this](https://pandoc.org/filters.html) page.++Installation+============++Requirements+------------++    - [Stack](https://docs.haskellstack.org/en/stable/README/)++### From Source++``` {.shell}++git clone https://github.com/gdevanla/pandoc-markdown-ghci-filter.git+cd pandoc-markdown-ghci-filter++stack build+```++### From Stackage/Hackage++``` {.shell}+stack build pandoc-markdown-ghci-filter # executable only available to local stack environment++or++stack install pandoc-markdown-ghci-filter # if you want to across all stack environments+```++Running the filter+==================++``` {.shell}+pandoc -s -t json test.md | pandoc-markdown-ghci-filter-exe | pandoc -f json -t markdown+```++Usage Notes/Caveats+===================++1.  All interactive statements (prefixed with `>>`) need to be preceded+    by `\n` to let the filter respect original new line spacing. If this+    is not followed, `\n` may be truncated.+2.  The program internally wraps all commands inside the GHCi multi-line+    construct `:{..:}`. Therefore, the code segments should not have+    multi-line constructs as part of code blocks.+3.  If you want the filter to ignore a certain `code` block, you can+    turn-off the filter by setting the `code` block attribute as follows++``` {.markdown}++{.haskell code_filter="Off"}++-- do not run this code through GHCi++>> putStrLn "This line will not be expanded by the filter"+```++Note, the default value is "On"++Limitations/Open Issues+=======================++1.  Attaching different formattting properties to `output`.+2.  As explained in `Usage Notes`, all `interactive` statements should+    be preceded by an empty line, for the filter to maintain the `\n`+    characters as given by the input.++More examples+=============++Sample Markdown Before Transformation+-------------------------------------++Sample markdown as fed into filter through `pandoc`.++Example 1+---------++``` {.haskell code-filter="Off"}+import Data.Text++>> putStrLn "This string should show up in the output"+```++Example 2+---------++``` {.haskell code-filter="Off"}+addOne:: Integer -> Integer+addOne x = x + 1++>> addOne 13++multBy2:: Integer -> Integer+multBy2 x = x * 2++>> (addOne 10) + (multBy2 20)+```++Example 3+---------++Any errors that occur while executing statements in the `code` block are+also rendered.++``` {.haskell code-filter="Off"}++wrongFuncDefinition:: Integer -> Integer+wrongFuncDefintion = x + 1++>> functionNotInScope 10+```++Markdown after transformation+-----------------------------++Example 1+---------++``` {.haskell code-filter="On"}+import Data.Text++>> putStrLn "This string should show up in the output"+This string should show up in the output++```++Example 2+---------++``` {.haskell code-filter="On"}+addOne:: Integer -> Integer+addOne x = x + 1++>> addOne 13+14++multBy2:: Integer -> Integer+multBy2 x = x * 2++>> (addOne 10) + (multBy2 20)+51++```++Example 3+---------++Any errors that occur while executing statements in the `code` block are+also rendered.++``` {.haskell code-filter="On"}++wrongFuncDefinition:: Integer -> Integer+wrongFuncDefintion = x + 1++<interactive>:16:1: error:+    The type signature for ‘wrongFuncDefinition’+      lacks an accompanying binding++>> functionNotInScope 10+<interactive>:29:2: error:+    Variable not in scope: functionNotInScope :: Integer -> t++```++*Fun Fact:* This document was generated using this same tool it+describes. This+[README-pre-process.md](https://github.com/gdevanla/pandoc-markdown-ghci-filter/blob/master/README-pre-process.md)+was used to generate this document. Here is the command that was used:++``` {.shell}+pandoc -s -t json README-pre-process.md | stack runhaskell app/Main.hs | pandoc -f json -t markdown > README.md+```

@@ -0,0 +1,2 @@+import Distribution.Simple+main = defaultMain

@@ -0,0 +1,12 @@+module Main where++import Text.Pandoc.JSON+import CodeBlockExecutor++-- test c@(CodeBlock (identifier, classes, key_value) str) = do+--   putStrLn $ show key_value+--   return c+-- test b = return b++main :: IO ()+main = toJSONFilter applyFilterToBlock

@@ -0,0 +1,89 @@+cabal-version: 1.12++-- This file has been generated from package.yaml by hpack version 0.31.2.+--+-- see: https://github.com/sol/hpack+--+-- hash: b97a93d5d47f2db606b3e2dedb5fb9fb3b0c117d16d5f3f45e03d2f19a47ba07++name:           pandoc-markdown-ghci-filter+version:        0.1.0.0+synopsis:       Pandoc-filter to evaluate `code` section in markdown and auto-embed output.+description:    Please see the README on GitHub at <https://github.com/gdevanla/pandoc-markdown-ghci-filter#readme>+category:       program, text, documentation, filter,mit+homepage:       https://github.com/gdevanla/pandoc-markdown-ghci-filter#readme+bug-reports:    https://github.com/gdevanla/pandoc-markdown-ghci-filter/issues+author:         Guru Devanla+maintainer:     gdrvnl@gmail.com+copyright:      2019 Guru Devanla+license:        MIT+license-file:   LICENSE+build-type:     Simple+extra-source-files:+    README.md+    ChangeLog.md++source-repository head+  type: git+  location: https://github.com/gdevanla/pandoc-markdown-ghci-filter++library+  exposed-modules:+      CodeBlockExecutor+  other-modules:+      Paths_pandoc_markdown_ghci_filter+  hs-source-dirs:+      src+  default-extensions: DeriveGeneric OverloadedStrings+  build-depends:+      aeson+    , base >=4.7 && <5+    , containers+    , ghcid+    , pandoc+    , pandoc-types+    , text+  default-language: Haskell2010++executable pandoc-markdown-ghci-filter-exe+  main-is: Main.hs+  other-modules:+      Paths_pandoc_markdown_ghci_filter+  hs-source-dirs:+      app+  default-extensions: DeriveGeneric OverloadedStrings+  ghc-options: -threaded -rtsopts -with-rtsopts=-N -Wall -Wcompat -Wincomplete-record-updates -Wincomplete-uni-patterns -Wredundant-constraints+  build-depends:+      aeson+    , base >=4.7 && <5+    , containers+    , ghcid+    , pandoc+    , pandoc-markdown-ghci-filter+    , pandoc-types+    , text+  default-language: Haskell2010++test-suite pandoc-markdown-ghci-filter-test+  type: exitcode-stdio-1.0+  main-is: Spec.hs+  other-modules:+      Paths_pandoc_markdown_ghci_filter+  hs-source-dirs:+      test+  default-extensions: DeriveGeneric OverloadedStrings+  ghc-options: -threaded -rtsopts -with-rtsopts=-N+  build-depends:+      QuickCheck+    , aeson+    , base >=4.7 && <5+    , containers+    , ghcid+    , pandoc+    , pandoc-markdown-ghci-filter+    , pandoc-types+    , tasty+    , tasty-hunit+    , tasty-quickcheck+    , text+  default-language: Haskell2010

@@ -0,0 +1,114 @@+{-# LANGUAGE OverloadedStrings #-}++-- | Module that communicates with GHCid, splits the commands in the code block,+-- gather the results and emits the trasnformed JSON based Pandoc AST.+++module CodeBlockExecutor+    (+      applyFilterToBlock,+      runCodeBlock,+      processResults+    ) where+++--import GHC.Generics+import Text.Pandoc++import Language.Haskell.Ghcid+import Control.Applicative+import Control.Exception+import Data.String+import Data.List as L+import Data.Map.Strict as M+import Data.Maybe++import qualified Data.Text as T++removeAll:: T.Text -> T.Text -> T.Text+removeAll pat str = if (T.replace pat "" str) == str then str+  else removeAll pat (T.replace pat "" str)++-- | Determines if the command encountered is an `interactive` command.+-- This is one of the conditions that determines whether output needs to be captured.+isInteractive :: T.Text -> Bool+isInteractive cmd = T.isPrefixOf ">>" cmd++-- | This function takes care of placing back the newline characters+-- that may been stripped out before sending the commands to GHCid+updateSuffixForInteractiveCmd :: Data.String.IsString p => T.Text -> p+updateSuffixForInteractiveCmd cmd = if isInteractive cmd then+  if T.last cmd == '\n' then "" else "\n"+  else "\n\n"++-- | Intercalate the commands and respective results. Typically, only errors+-- encountered while running definitions, and outut of interactive commands (+-- i.e commands prefixed with `>>`) are captured. All empty strins are dropped.+intercalateCmdAndResults :: T.Text -> T.Text -> T.Text+intercalateCmdAndResults cmd result =+  T.concat [cmd, updateSuffixForInteractiveCmd cmd, result, trailResult result] where+  trailResult r = if r /= "" then "\n" else ""++-- | Post-processing function that interleaves command and results+processResults :: [T.Text] -- ^ List of commands that were executed+               -> [T.Text] -- ^ List of results for the executed commands+               -> String -- ^ New string that will form the bodyof the modified Code Block.+processResults cmds results =+  let cmd_result = getZipList $ intercalateCmdAndResults <$> ZipList cmds <*> ZipList results+  in+    (T.unpack . T.concat) $ cmd_result++-- | Apply the filter block only if the language attribute+-- is set to `haskell` and `code-filter` property is *not* set to "Off"+--+-- Example:+-- ``` {.haskell code-filter="Off"} ``` will turn off any kind of transformation.+--+-- By default the filer is "On"+applyFilterToBlock:: Block -- ^ The 'Block' yielded by toJSONFilter in "Text.Pandoc.JSON"+                  -> IO Block -- ^ The newly minted 'Block'+applyFilterToBlock c@(CodeBlock (_, classes, key_values) _) = let+  attrs = M.fromList key_values+  haskell_in_class = L.find (== "haskell") classes+  code_filter_flag = maybe "On" id (M.lookup ("code-filter") attrs)+  in+    if code_filter_flag == "On" && isJust haskell_in_class then runCodeBlock c+    else (return c)+applyFilterToBlock b = return b++-- | Run the commands in the 'Block' in one single GHCid session.+runCodeBlock:: Block -- ^ 'Block' to execute. Only 'CodeBlock' is executed.+            -> IO Block -- ^ transformed code block+runCodeBlock (CodeBlock attr str) = bracket startGhciProcess' stopGhci runCommands+  where+    startGhciProcess' = do+      (ghci_handle, _) <- startGhci "stack ghci" (Just ".") (\_ _ -> return ())+      return ghci_handle+    runCommands g = do+      let cmds = L.filter (\s -> s /= "") $ T.splitOn "\n\n" $ T.pack str+      results <- mapM (runCmd g) cmds+      let results''' = processResults cmds results+      return (CodeBlock attr results''')+runCodeBlock b = return b++-- | Executes a command in GHCIid.+runCmd :: Ghci -- ^ Handle to the GHCi process through the GHCid interface+       -> T.Text -- ^ Statement to execute+       -> IO T.Text -- ^ Result of the executed statement+runCmd g cmd = do+  let executeStatement = exec g+      cmd_ = T.concat [":{\n", T.replace ">>" "" cmd, "\n:}\n"]+  result <- executeStatement . T.unpack $ cmd_+  -- we send this PROBE here since GHCi has its own mind on how it prefixes output based on its native needs. By sending the probe we can guess what is the latest prompt and then discard it while processing thye output.+  probe <- exec g ":{\nshow (\"PANDOC_FILTER_PROBE_PROMPT_INTERNAL\"::String)\n:}\n"+  let current_prompt = preparePrompt probe+        where+          preparePrompt probe' =+            let prompt = T.replace " \"\\\"PANDOC_FILTER_PROBE_PROMPT_INTERNAL\\\"\"\n" "" (T.pack . unlines $ probe')+            in+              T.concat [T.takeWhile (/='|') prompt, "|"]+  --putStrLn $ show . unlines $ probe+  --putStrLn $ show current_prompt+      result' = T.stripStart $ removeAll current_prompt (T.pack . unlines $ result)+  --putStrLn $ show result'+  return $ result'

@@ -0,0 +1,66 @@+{-# OPTIONS_GHC -fno-warn-type-defaults #-}+import Test.Tasty.HUnit+import Test.Tasty++import Data.Text as T+import CodeBlockExecutor+import Control.Applicative+++input1 :: [Text]+input1 = [">> putStrLn \"This string should show up in the output\""]+ghci_result1 :: [Text]+ghci_result1 = ["This string should show up in the output"]+combined_result1 :: String+combined_result1 = T.unpack . T.unlines $+  [+    ">> putStrLn \"This string should show up in the output\"",+    "This string should show up in the output"+  ]++input2 :: [Text]+input2 = fmap T.pack [+  "testFunc:: Integer -> Integer\ntestFunc x = x + 1",+  ">> testFunc 13\n",+  "anotherFunc:: Integer -> Integer\nanotherFunc x = x * 2"+  ]++ghci_result2 :: [Text]+ghci_result2 = fmap T.pack ["", "14", ""]+combined_result2 :: String+combined_result2  = T.unpack . T.unlines $+  ["testFunc:: Integer -> Integer",+  "testFunc x = x + 1",+  "",+  ">> testFunc 13",+  "14",+  "anotherFunc:: Integer -> Integer",+  "anotherFunc x = x * 2",+  ""]++fixtures :: [([Text], [Text], String)]+fixtures = [+  (input1, ghci_result1, combined_result1),+  (input2, ghci_result2, combined_result2)]++generateTests :: [TestTree]+generateTests = getZipList $ testProcessResults <$> ZipList fixtures <*> ZipList [1..]++testProcessResults :: Show a => ([Text], [Text], String) -> a -> TestTree+testProcessResults (input, ghci_results, expected) i = testCase ("test_processResults" ++ (show i)) $ do+  let actual = processResults input ghci_results+  assertEqual "processResult" expected actual++-- test_processResults = testCase "test_processResults" (+--   assertEqual "input1" combined_result (processResults [input1] [ghci_result]))++-- test_processResults2 = testCase "test_processResults" (+--   assertEqual "input1" combined_result (processResults [input1] [ghci_result]))++tests:: TestTree+tests = testGroup "Tests" generateTests++main :: IO ()+main = defaultMain tests++--runTestTT $ TestList [TestLabel "test1" test_processResults]