doctest-parallel 0.2 → 0.2.1

raw patch · 12 files changed

+260/−53 lines, 12 filesdep +randomPVP: major bump suggested

API removals or changes: PVP suggests a major version bump

Dependencies added: random

API changes (from Hackage documentation)

Files

• CHANGES.markdown +8/−1
• README.markdown +5/−1
• doctest-parallel.cabal +7/−1
• src/Test/DocTest.hs +67/−14
• src/Test/DocTest/Helpers.hs +11/−4
• src/Test/DocTest/Internal/Options.hs +59/−5
• src/Test/DocTest/Internal/Runner.hs +64/−19
• test/MainSpec.hs +21/−5
• test/RunnerSpec.hs +2/−2
• test/doctests.hs +3/−1
• test/integration/WithCInclude/Bar.hs +12/−0
• test/integration/WithCInclude/include/WithCInclude.h +1/−0

@@ -1,8 +1,15 @@+# 0.2.1+  * C include directories (Cabal field: `include-dirs`) are now passed to GHC when parsing source files ([#7](https://github.com/martijnbastiaan/doctest-parallel/issues/7))+  * A migration guide has been added ([#11](https://github.com/martijnbastiaan/doctest-parallel/issues/11))+  * Test order can be randomized using `--randomize-order`. Test order can be made deterministic by adding an optional `--seed=N` argument ([#12](https://github.com/martijnbastiaan/doctest-parallel/pull/12))+  * Any non-error output can now be surpressed by `--quiet` ([#20](https://github.com/martijnbastiaan/doctest-parallel/pull/20))+  * Doctest can now be called using a record for option passing in addition to command line arguments. See `mainFromCabalWithConfig` and `mainFromLibraryWithConfig`.+ # 0.2 Changes:   * Support for GHC 9.2 has been added ([#4](https://github.com/martijnbastiaan/doctest-parallel/pull/4))   * Support for GHC 8.2 has been dropped ([#3](https://github.com/martijnbastiaan/doctest-parallel/pull/3))-  * The dependency `cabal-install-parsers` has been dropped. This trims the dependency tree quite a bit [#3](https://github.com/martijnbastiaan/doctest-parallel/pull/3)+  * The dependency `cabal-install-parsers` has been dropped. This trims the dependency tree quite a bit ([#3](https://github.com/martijnbastiaan/doctest-parallel/pull/3))   * The Hackage distribution now ships all files necessary to run `doctest-parallel`'s tests (Fixes [#1](https://github.com/martijnbastiaan/doctest-parallel/issues/1), PR [#2](https://github.com/martijnbastiaan/doctest-parallel/pull/2))  # 0.1

@@ -6,6 +6,9 @@ # Installation `doctest-parallel` is available from [Hackage](https://hackage.haskell.org/package/doctest-parallel). It cannot be used as a standalone binary, rather, it expects to be integrated in a Cabal/Stack project. See [examples/](example/README.md) for more information on how to integrate `doctest-parallel` into your project. +# Migrating from `doctest`+See [issue #11](https://github.com/martijnbastiaan/doctest-parallel/issues/11) for more information.+ # Usage Below is a small Haskell module. The module contains a Haddock comment with some examples of interaction. The examples demonstrate how the module is supposed to be used. @@ -329,7 +332,8 @@  * It would be lovely if we could get rid of the needs for `write-ghc-environment-files: always` option for Cabal. To properly do this, I think Cabal should do two things:     1. Deprecate GHC environment files as a way to _implicitly_ setup environments. Instead, environment files should be written to the `dist-newstyle` directory and activated using some subcommand, e.g. `cabal shell`. This avoids the many problems GHC environment files have, while retaining their functionality for people who like them.     2. Any subcommands should be run with `GHC_ENVIRONMENT` set - pointing to the GHC environment file. Like Stack, this would create a hassle free way of using Cabal in combination with projects/executables that use the GHC API (e.g., `clash-ghc`, `doctest-parallel`).- * Cabal needs to expose more information _by default_ in order for `doctest-parallel` to properly work. Specifically, it needs to know the exact `default-extensions`, `ghc-options`, and `CPP` flags the project is compiled with. These options are obtainable by using a custom `Setup.hs`, but this has its own list of problems.+ * It would be nice if Cabal would expose more information _by default_ (probably through auto-generated modules) in order for `doctest-parallel` to properly work. Specifically, it needs to know the exact `default-extensions`, `ghc-options`, and `CPP` flags the project is compiled with. These options are obtainable by using a custom `Setup.hs`, but this has its own list of problems.+   * Alternatively, if comments could be included in and loaded from `.hi` files that'd solve all issues too.  * Hopefully many of the improvements made here can make their way back into `sol/doctest`.   Of course, if you wish to add a feature that's not in this list, please feel free top open a pull request!

@@ -1,7 +1,7 @@ cabal-version: 2.0  name:           doctest-parallel-version:        0.2+version:        0.2.1 synopsis:       Test interactive Haskell examples description:    The doctest program checks examples in source code comments.  It is modeled                 after doctest for Python (<https://docs.python.org/3/library/doctest.html>).@@ -54,6 +54,7 @@     test/parse/setup-empty/*.hs     test/parse/setup-only/*.hs     test/parse/simple/*.hs+    test/integration/WithCInclude/include/WithCInclude.h  source-repository head   type: git@@ -97,6 +98,7 @@     , ghc-paths >=0.1.0.9     , pretty     , process+    , random >= 1.2     , syb >=0.3     , transformers     , unordered-containers@@ -118,6 +120,8 @@   -- ghc-options: -Wall   hs-source-dirs:       test/integration+  include-dirs:+      test/integration/WithCInclude/include   c-sources:       test/integration/WithCbits/foo.c   exposed-modules:@@ -157,6 +161,7 @@     TestSimple.Fib     TrailingWhitespace.Foo     WithCbits.Bar+    WithCInclude.Bar  test-suite spectests   main-is: Spec.hs@@ -185,6 +190,7 @@     , base >=4.5 && <5     , base-compat >=0.7.0     , code-page >=0.1+    , containers     , doctest-parallel     , deepseq     , directory

@@ -1,11 +1,21 @@ {-# LANGUAGE CPP #-}-{-# LANGUAGE LambdaCase #-} {-# LANGUAGE RecordWildCards #-} {-# LANGUAGE NamedFieldPuns #-} {-# LANGUAGE ViewPatterns #-} -module Test.DocTest where+module Test.DocTest+  ( mainFromCabal+  , mainFromLibrary+  , mainFromCabalWithConfig+  , mainFromLibraryWithConfig +  -- * Internal+  , filterModules+  , isSuccess+  , getSeed+  , run+  ) where+ import           Prelude () import           Prelude.Compat @@ -14,6 +24,7 @@ import           Control.Monad (unless) import           System.Exit (exitFailure) import           System.IO+import           System.Random (randomIO)  import qualified Control.Exception as E @@ -40,13 +51,29 @@ -- -- Example: --+-- @ -- mainFromCabal "my-project" =<< getArgs+-- @ -- mainFromCabal :: String -> [String] -> IO () mainFromCabal libName cmdArgs = do   lib <- extractCabalLibrary =<< findCabalPackage libName   mainFromLibrary lib cmdArgs +-- | Run doctest given config.+--+-- Example:+--+-- @+-- mainFromCabal "my-project" defaultConfig+-- @+--+mainFromCabalWithConfig :: String -> Config -> IO ()+mainFromCabalWithConfig libName config = do+  lib <- extractCabalLibrary =<< findCabalPackage libName+  mainFromLibraryWithConfig lib config++-- | Like 'mainFromCabal', but with a given library. mainFromLibrary :: Library -> [String] -> IO () mainFromLibrary lib (parseOptions -> opts) =   case opts of@@ -56,15 +83,20 @@        hPutStrLn stderr "Try `doctest --help' for more information."        exitFailure     Result config -> do-      r <- main lib config `E.catch` \e -> do-        case fromException e of-          Just (UsageError err) -> do-            hPutStrLn stderr ("doctest: " ++ err)-            hPutStrLn stderr "Try `doctest --help' for more information."-            exitFailure-          _ -> E.throwIO e-      unless (isSuccess r) exitFailure+      mainFromLibraryWithConfig lib config +-- | Run doctests with given library and config.+mainFromLibraryWithConfig :: Library -> Config -> IO ()+mainFromLibraryWithConfig lib config = do+  r <- run lib config `E.catch` \e -> do+    case fromException e of+      Just (UsageError err) -> do+        hPutStrLn stderr ("doctest: " ++ err)+        hPutStrLn stderr "Try `doctest --help' for more information."+        exitFailure+      _ -> E.throwIO e+  unless (isSuccess r) exitFailure+ isSuccess :: Summary -> Bool isSuccess s = sErrors s == 0 && sFailures s == 0 @@ -83,15 +115,36 @@   nonExistingMods = Set.toList (wantedMods1 `Set.difference` allMods1)   isSpecifiedMod Module{moduleName} = moduleName `Set.member` wantedMods1 +getSeed+  :: Bool+  -- ^ Whether quiet mode is enabled+  -> Bool+  -- ^ Enable order randomization. If 'False', this function always returns 'Nothing'+  -> Maybe Int+  -- ^ User supplied seed. If 'Nothing', a fresh seed will be generated.+  -> IO (Maybe Int)+  -- ^ Maybe seed to use for order randomization.+getSeed _quiet False _ = pure Nothing+getSeed _quiet True (Just seed) = pure (Just seed)+getSeed quiet True Nothing = do+  -- Using an abslute number to prevent copy+paste errors+  seed <- abs <$> randomIO+  unless quiet $+    putStrLn ("Using freshly generated seed to randomize test order: " <> show seed)+  pure (Just seed) -main :: Library -> Config -> IO Summary-main lib Config{..} = do+-- | Run doctest for given library and config. Produce a summary of all tests.+run :: Library -> Config -> IO Summary+run lib Config{..} = do   let     implicitPrelude = DisableExtension ImplicitPrelude `notElem` libDefaultExtensions lib     (includeArgs, moduleArgs, otherGhciArgs) = libraryToGhciArgs lib     evalGhciArgs = otherGhciArgs ++ ["-XNoImplicitPrelude"] +  seed <- getSeed cfgQuiet cfgRandomizeOrder cfgSeed+   -- get examples from Haddock comments   allModules <- getDocTests (includeArgs ++ moduleArgs ++ otherGhciArgs)-  let modules = filterModules cfgModules allModules-  runModules cfgThreads cfgPreserveIt cfgVerbose implicitPrelude evalGhciArgs modules+  runModules+    cfgThreads cfgPreserveIt cfgVerbose seed implicitPrelude evalGhciArgs+    cfgQuiet (filterModules cfgModules allModules)

@@ -1,6 +1,5 @@ {-# LANGUAGE CPP #-} {-# LANGUAGE MultiWayIf #-}-{-# LANGUAGE NamedFieldPuns #-} {-# LANGUAGE RecordWildCards #-} {-# LANGUAGE LambdaCase #-} @@ -25,7 +24,7 @@ import Distribution.PackageDescription   ( CondTree(CondNode, condTreeData), GenericPackageDescription (condLibrary)   , exposedModules, libBuildInfo, hsSourceDirs, defaultExtensions, package-  , packageDescription, condSubLibraries )+  , packageDescription, condSubLibraries, includeDirs ) import Distribution.Pretty (prettyShow) import Distribution.Verbosity (silent) @@ -38,16 +37,22 @@  data Library = Library   { libSourceDirectories :: [FilePath]+    -- ^ Haskell source directories+  , libCSourceDirectories :: [FilePath]+    -- ^ C source directories   , libModules :: [ModuleName]+    -- ^ Exposed modules   , libDefaultExtensions :: [Extension]+    -- ^ Extensions enabled by default   }   deriving (Show)  -- | Convert a "Library" to arguments suitable to be passed to GHCi. libraryToGhciArgs :: Library -> ([String], [String], [String])-libraryToGhciArgs Library{..} = (srcArgs, modArgs, extArgs)+libraryToGhciArgs Library{..} = (hsSrcArgs <> cSrcArgs, modArgs, extArgs)  where-  srcArgs = map ("-i" <>) libSourceDirectories+  hsSrcArgs = map ("-i" <>) libSourceDirectories+  cSrcArgs = map ("-I" <>) libCSourceDirectories   modArgs = map prettyShow libModules   extArgs = map showExt libDefaultExtensions @@ -144,10 +149,12 @@     let       buildInfo = libBuildInfo lib       sourceDirs = hsSourceDirs buildInfo+      cSourceDirs = includeDirs buildInfo       root = takeDirectory pkgPath     in       pure Library         { libSourceDirectories = map ((root </>) . compatPrettyShow) sourceDirs+        , libCSourceDirectories = map (root </>) cSourceDirs         , libModules = exposedModules lib         , libDefaultExtensions = defaultExtensions buildInfo         }

@@ -24,15 +24,18 @@ usage :: String usage = unlines [     "Usage:"-  , "  doctest [ --fast | --preserve-it | --verbose | -jN ]..."+  , "  doctest [ options ]... [<module>]..."   , "  doctest --help"   , "  doctest --version"   , "  doctest --info"   , ""   , "Options:"   , "  -jN                      number of threads to use"+  , "  --randomize-order        randomize order in which tests are run"+  , "  --seed                   use a specific seed to randomize test order"   , "  --preserve-it            preserve the `it` variable between examples"   , "  --verbose                print each test as it is run"+  , "  --quiet                  only print errors"   , "  --help                   display this help and exit"   , "  --version                output version information and exit"   , "  --info                   output machine-readable version information and exit"@@ -73,10 +76,18 @@   , cfgVerbose :: Bool   -- ^ Verbose output (default: @False@)   , cfgModules :: [ModuleName]-  -- ^ Module names to test+  -- ^ Module names to test. An empty list means "test all modules".   , cfgThreads :: Maybe Int   -- ^ Number of threads to use. Defaults to autodetection based on the number   -- of cores.+  , cfgRandomizeOrder :: Bool+  -- ^ Randomize the order in which test cases in a module are run (default: @False@)+  , cfgSeed :: Maybe Int+  -- ^ Initialize random number generator used to randomize test cases when+  -- 'cfgRandomizeOrder' is set. If set to 'Nothing', a random seed is picked+  -- from a system RNG source on startup.+  , cfgQuiet :: Bool+  -- ^ Only print error messages, no status or progress messages (default: @False@)   } deriving (Show, Eq)  defaultConfig :: Config@@ -85,6 +96,9 @@   , cfgVerbose = False   , cfgModules = []   , cfgThreads = Nothing+  , cfgRandomizeOrder = False+  , cfgSeed = Nothing+  , cfgQuiet = False   }  parseOptions :: [String] -> Result Config@@ -96,17 +110,57 @@       "--help" -> ResultStdout usage       "--info" -> ResultStdout info       "--version" -> ResultStdout versionInfo+      "--randomize-order" -> go config{cfgRandomizeOrder=True} args       "--preserve-it" -> go config{cfgPreserveIt=True} args       "--verbose" -> go config{cfgVerbose=True} args-      ('-':'j':n0) | Just n1 <- parseThreads n0 -> go config{cfgThreads=Just n1} args+      "--quiet" -> go config{cfgQuiet=True} args+      ('-':_) | Just n <- parseSeed arg -> go config{cfgSeed=Just n} args+      ('-':_) | Just n <- parseThreads arg -> go config{cfgThreads=Just n} args       ('-':_) -> ResultStderr ("Unknown command line argument: " <> arg)       mod_ -> go config{cfgModules=mod_ : cfgModules config} args +-- | Parse seed argument+--+-- >>> parseSeed "--seed=6"+-- Just 6+-- >>> parseSeed "--seeeed=6"+-- Nothing+--+parseSeed :: String -> Maybe Int+parseSeed arg = readMaybe =<< parseSpecificFlag arg "seed"+++-- | Parse number of threads argument+--+-- >>> parseThreads "-j6"+-- Just 6+-- >>> parseThreads "-j-2"+-- Nothing+-- >>> parseThreads "-jA"+-- Nothing+-- parseThreads :: String -> Maybe Int-parseThreads n0 = do+parseThreads ('-':'j':n0) = do   n1 <- readMaybe n0   if n1 > 0 then Just n1 else Nothing+parseThreads _ = Nothing +-- | Parse a specific flag with a value, or return 'Nothing'+--+-- >>> parseSpecificFlag "--foo" "foo"+-- Nothing+-- >>> parseSpecificFlag "--foo=" "foo"+-- Nothing+-- >>> parseSpecificFlag "--foo=5" "foo"+-- Just "5"+-- >>> parseSpecificFlag "--foo=5" "bar"+-- Nothing+parseSpecificFlag :: String -> String -> Maybe String+parseSpecificFlag arg flag = do+  case parseFlag arg of+    ('-':'-':f, value) | f == flag -> value+    _ -> Nothing+ -- | Parse a flag into its flag and argument component. -- -- Example:@@ -121,5 +175,5 @@ parseFlag arg =   case break (== '=') arg of     (flag, ['=']) -> (flag, Nothing)-    (flag, ('=':opt)) -> (flag, Just opt)+    (flag, '=':opt) -> (flag, Just opt)     (flag, _) -> (flag, Nothing)

@@ -1,5 +1,7 @@ {-# LANGUAGE CPP #-} {-# LANGUAGE MultiWayIf #-}+{-# LANGUAGE TypeApplications #-}+{-# LANGUAGE NamedFieldPuns #-}  module Test.DocTest.Internal.Runner where @@ -8,11 +10,14 @@ import           Control.Concurrent (Chan, writeChan, readChan, newChan, forkIO) import           Control.Exception (SomeException, catch) import           Control.Monad hiding (forM_)-import           Data.Maybe (fromMaybe)-import           Text.Printf (printf)-import           System.IO (hPutStrLn, hPutStr, stderr, hIsTerminalDevice) import           Data.Foldable (forM_)+import           Data.Function (on)+import           Data.List (sortBy)+import           Data.Maybe (fromMaybe) import           GHC.Conc (numCapabilities)+import           System.IO (hPutStrLn, hPutStr, stderr, hIsTerminalDevice)+import           System.Random (randoms, mkStdGen)+import           Text.Printf (printf)  import           Control.Monad.Trans.State import           Control.Monad.IO.Class@@ -70,35 +75,46 @@   -- ^ Preserve it   -> Bool   -- ^ Verbose+  -> Maybe Int+  -- ^ If 'Just', use seed to randomize test order   -> Bool   -- ^ Implicit Prelude   -> [String]   -- ^ Arguments passed to the GHCi process.+  -> Bool+  -- ^ Quiet mode activated   -> [Module [Located DocTest]]   -- ^ Modules under test   -> IO Summary-runModules nThreads preserveIt verbose implicitPrelude args modules = do+runModules nThreads preserveIt verbose seed implicitPrelude args quiet modules = do   isInteractive <- hIsTerminalDevice stderr    -- Start a thread pool. It sends status updates to this thread through 'output'.   (input, output) <-     makeThreadPool       (fromMaybe numCapabilities nThreads)-      (runModule preserveIt implicitPrelude args)+      (runModule preserveIt seed implicitPrelude args)    -- Send instructions to threads   liftIO (mapM_ (writeChan input) modules)    let     nExamples = (sum . map count) modules-    initState = ReportState 0 isInteractive verbose mempty {sExamples = nExamples}+    initState = ReportState+      { reportStateCount = 0+      , reportStateInteractive = isInteractive+      , reportStateVerbose = verbose+      , reportStateQuiet = quiet+      , reportStateSummary = mempty{sExamples=nExamples}+      } -  ReportState _ _ _ s <- (`execStateT` initState) $ do+  ReportState{reportStateSummary} <- (`execStateT` initState) $ do     consumeUpdates output (length modules)-    verboseReport "# Final summary:"-    gets (show . reportStateSummary) >>= report+    unless quiet $ do+      verboseReport "# Final summary:"+      gets (show . reportStateSummary) >>= report -  return s+  return reportStateSummary  where   consumeUpdates _output 0 = pure ()   consumeUpdates output modsLeft = do@@ -125,6 +141,7 @@   reportStateCount        :: Int     -- ^ characters on the current line , reportStateInteractive  :: Bool    -- ^ should intermediate results be printed? , reportStateVerbose      :: Bool+, reportStateQuiet        :: Bool , reportStateSummary      :: Summary -- ^ test summary } @@ -156,15 +173,27 @@           | otherwise = msg   liftIO (hPutStr stderr str) +-- | Shuffle a list given a seed for an RNG+shuffle :: Int -> [a] -> [a]+shuffle seed xs =+    map snd+  $ sortBy (compare `on` fst)+  $ zip (randoms @Int (mkStdGen seed)) xs+ -- | Run all examples from given module. runModule   :: Bool+  -> Maybe Int   -> Bool   -> [String]   -> Chan ReportUpdate   -> Module [Located DocTest]   -> IO ()-runModule preserveIt implicitPrelude ghciArgs output (Module module_ setup examples) = do+runModule preserveIt (Just seed) implicitPrelude ghciArgs output (Module module_ setup examples) = do+  runModule+    preserveIt Nothing implicitPrelude ghciArgs output+    (Module module_ setup (shuffle seed examples))+runModule preserveIt Nothing implicitPrelude ghciArgs output (Module module_ setup examples) = do   Interpreter.withInterpreter ghciArgs $ \repl -> withCP65001 $ do     -- Try to import this module, if it fails, something is off     importResult <- Interpreter.safeEval repl importModule@@ -242,7 +271,10 @@  reportStart :: Location -> Expression -> String -> Report () reportStart loc expression testType = do-  verboseReport (printf "### Started execution at %s.\n### %s:\n%s" (show loc) testType expression)+  quiet <- gets reportStateQuiet+  unless quiet $+    verboseReport+      (printf "### Started execution at %s.\n### %s:\n%s" (show loc) testType expression)  reportFailure :: FromSetup -> Location -> Expression -> [String] -> Report () reportFailure fromSetup loc expression err = do@@ -275,33 +307,46 @@   report "   * Run with '--write-ghc-environment-files=always'"   report "   * Add 'write-ghc-environment-files: always' to your cabal.project"   report ""-  report " 3. The testsuite executable does not have a dependency on your project library. Please add it to the 'build-depends' section of the testsuite executable."+  report " 3. For Cabal users: Cabal did not generate a GHC environment file in time. This"+  report "    can happen if you use 'cabal test' instead of 'cabal run doctests'. See"+  report "    https://github.com/martijnbastiaan/doctest-parallel/issues/22."   report ""-  report "See the example project at https://github.com/martijnbastiaan/doctest-parallel/tree/master/examples for more information."+  report " 4. The testsuite executable does not have a dependency on your project library. Please"+  report "    add it to the 'build-depends' section of the testsuite executable."+  report ""+  report "See the example project at https://github.com/martijnbastiaan/doctest-parallel/blob/main/example/README.md for more information."   updateSummary FromSetup emptySummary{sErrors=1}  reportSuccess :: FromSetup -> Location -> Report () reportSuccess fromSetup loc = do-  verboseReport (printf "### Successful `%s'!\n" (show loc))+  quiet <- gets reportStateQuiet+  unless quiet $+    verboseReport (printf "### Successful `%s'!\n" (show loc))   updateSummary fromSetup (Summary 0 1 0 0)  verboseReport :: String -> Report () verboseReport xs = do   verbose <- gets reportStateVerbose-  when verbose $ report xs+  quiet <- gets reportStateQuiet+  unless quiet $+    when verbose $+      report xs  updateSummary :: FromSetup -> Summary -> Report () updateSummary FromSetup summary =   -- Suppress counts, except for errors   updateSummary NotFromSetup summary{sExamples=0, sTried=0, sFailures=0} updateSummary NotFromSetup summary = do-  ReportState n f v s <- get-  put (ReportState n f v $ s `mappend` summary)+  ReportState n f v q s <- get+  put (ReportState n f v q $ s `mappend` summary)  reportProgress :: Report () reportProgress = do   verbose <- gets reportStateVerbose-  when (not verbose) $ gets (show . reportStateSummary) >>= report_+  quiet <- gets reportStateQuiet+  unless quiet $+    unless verbose $+      gets (show . reportStateSummary) >>= report_  -- | Run given test group. --

@@ -7,10 +7,12 @@ import           Test.Hspec import           Test.HUnit (assertEqual, Assertion) +import qualified Data.Map as Map import qualified Test.DocTest as DocTest import           Test.DocTest.Helpers (extractSpecificCabalLibrary, findCabalPackage) import           Test.DocTest.Internal.Options import           Test.DocTest.Internal.Runner+import           System.Environment (getEnvironment) import           System.IO.Silently import           System.IO @@ -24,7 +26,7 @@   lib <- extractSpecificCabalLibrary (Just "spectests-modules") pkg   actual <-     hSilence [stderr] $-      DocTest.main lib config{cfgModules=modNames}+      DocTest.run lib config{cfgModules=modNames}   assertEqual (show modNames) expected actual  cases :: Int -> Summary@@ -35,7 +37,17 @@  spec :: Spec spec = do-  describe "doctest" $ do+  env <- runIO getEnvironment+  let+    cDescribe =+      -- Don't run doctests as part of the Stack testsuite yet, pending+      -- https://github.com/commercialhaskell/stack/issues/5662+      if "STACK_EXE" `Map.member` Map.fromList env then+        xdescribe+      else+        describe++  cDescribe "doctest" $ do     it "testSimple" $       doctest ["TestSimple.Fib"]         (cases 1)@@ -109,7 +121,7 @@       doctest ["TrailingWhitespace.Foo"]         (cases 1) -  describe "doctest as a runner for QuickCheck properties" $ do+  cDescribe "doctest as a runner for QuickCheck properties" $ do     it "runs a boolean property" $ do       doctest ["PropertyBool.Foo"]         (cases 1)@@ -134,12 +146,12 @@       doctest ["PropertySetup.Foo"]         (cases 1) -  describe "doctest (module isolation)" $ do+  cDescribe "doctest (module isolation)" $ do     it "should fail due to module isolation" $ do       doctestWithOpts defaultConfig ["ModuleIsolation.TestA", "ModuleIsolation.TestB"]         (cases 2) {sFailures = 1} -  describe "doctest (regression tests)" $ do+  cDescribe "doctest (regression tests)" $ do     it "bugfixOutputToStdErr" $ do       doctest ["BugfixOutputToStdErr.Fib"]         (cases 2)@@ -160,3 +172,7 @@     it "doesn't get confused by doctests using System.IO imports" $ do       doctest ["SystemIoImported.A"]         (cases 2)++    it "correctly handles C import directories" $ do+      doctest ["WithCInclude.Bar"]+        (cases 1)

@@ -12,11 +12,11 @@ main = hspec spec  capture :: Report a -> IO String-capture = fmap fst . hCapture [stderr] . (`execStateT` ReportState 0 True False mempty)+capture = fmap fst . hCapture [stderr] . (`execStateT` ReportState 0 True False False mempty)  -- like capture, but with interactivity set to False capture_ :: Report a -> IO String-capture_ = fmap fst . hCapture [stderr] . (`execStateT` ReportState 0 False False mempty)+capture_ = fmap fst . hCapture [stderr] . (`execStateT` ReportState 0 False False False mempty)  spec :: Spec spec = do

@@ -4,4 +4,6 @@ import System.Environment (getArgs)  main :: IO ()-main = mainFromCabal "doctest-parallel" =<< getArgs+main = do+  args <- getArgs+  mainFromCabal "doctest-parallel" ("--randomize-order":args)

@@ -0,0 +1,12 @@+{-# LANGUAGE CPP #-}++module WithCInclude.Bar where++#include "WithCInclude.h"+++-- |+-- >>> x+-- 42+x :: Int+x = THE_DEFINE

@@ -0,0 +1,1 @@+#define THE_DEFINE 42