# Doctest parallel: Test interactive Haskell examples
`doctest-parallel` is a library that checks [examples in Haddock comments](http://www.haskell.org/haddock/doc/html/ch03s08.html#id566093). It is similar to the [popular Python module with the same name](http://docs.python.org/library/doctest.html).
# Installation
`doctest-parallel` is available from [Hackage](https://hackage.haskell.org/package/doctest-parallel). It cannot be used as a standalone binary, rather, it expects to be integrated in a Cabal/Stack project. See [examples/](example/README.md) for more information on how to integrate `doctest-parallel` into your project.
# Migrating from `doctest`
See [issue #11](https://github.com/martijnbastiaan/doctest-parallel/issues/11) for more information.
# Usage
Below is a small Haskell module. The module contains a Haddock comment with some examples of interaction. The examples demonstrate how the module is supposed to be used.
```haskell
module Fib where
-- | Compute Fibonacci numbers
--
-- Examples:
--
-- >>> fib 10
-- 55
--
-- >>> fib 5
-- 5
fib :: Int -> Int
fib 0 = 0
fib 1 = 1
fib n = fib (n - 1) + fib (n - 2)
```
A comment line starting with `>>>` denotes an _expression_. All comment lines following an expression denote the _result_ of that expression. Result is defined by what a [REPL](http://en.wikipedia.org/wiki/Read-eval-print_loop) (e.g. ghci) prints to `stdout` and `stderr` when evaluating that expression.
`doctest-parallel` will fail on comments that `haddock` also doesn't like. Sometimes (e.g., [#251](https://github.com/sol/doctest/issues/251)), this means that `doctest-parallel` will fail on input that GHC accepts.
## Command line
```
Usage:
doctest [ options ]... [<module>]...
doctest --help
doctest --version
doctest --info
Options:
-jN number of threads to use
--log-level=LEVEL one of: debug, verbose, info, warning, error. Default: info.
--ghc-arg=ARG pass argument to GHC when parsing, pass multiple times for multiple flags
† --implicit-module-import import module before testing it (default)
† --randomize-order randomize order in which tests are run
† --seed=N use a specific seed to randomize test order
† --preserve-it preserve the `it` variable between examples
--nix account for Nix build environments (default)
--quiet set log level to `Error`, shorthand for `--log-level=error`
--verbose set log level to `Verbose`, shorthand for `--log-level=verbose`
--debug set log level to `Debug`, shorthand for `--log-level=debug`
--help display this help and exit
--version output version information and exit
--info output machine-readable version information and exit
Supported inverted options:
--no-nix
† --no-implicit-module-import
† --no-randomize-order (default)
† --no-preserve-it (default)
Options marked with a dagger (†) can also be used to set module level options, using
an ANN pragma like this:
{-# ANN module "doctest-parallel: --no-randomize-order" #-}
```
## Example groups
Examples from a single Haddock comment are grouped together and share the same
scope. E.g. the following works:
```haskell
-- |
-- >>> let x = 23
-- >>> x + 42
-- 65
```
If an example fails, subsequent examples from the same group are skipped. E.g.
for
```haskell
-- |
-- >>> let x = 23
-- >>> let n = x + y
-- >>> print n
```
`print n` is not tried, because `let n = x + y` fails (`y` is not in scope!).
## Setup code
You can put setup code in a [named chunk][named-chunks] with the name `$setup`.
The setup code is run before each example group. If the setup code produces
any errors/failures, all tests from that module are skipped.
Here is an example:
```haskell
module Foo where
import Bar.Baz
-- $setup
-- >>> let x = 23 :: Int
-- |
-- >>> foo + x
-- 65
foo :: Int
foo = 42
```
Note that you should not place setup code in between the module header (`module
... where`) and import declarations. GHC will not be able to parse it ([issue
#167](https://github.com/sol/doctest/issues/167)). It is best to place setup
code right after import declarations, but due to its declarative nature you can
place it anywhere in between top level declarations as well.
## Multi-line input
GHCi supports commands which span multiple lines, and the same syntax works for doctest:
```haskell
-- |
-- >>> :{
-- let
-- x = 1
-- y = 2
-- in x + y + multiline
-- :}
-- 6
multiline = 3
```
Note that `>>>` can be left off for the lines following the first: this is so that
haddock does not strip leading whitespace. The expected output has whitespace
stripped relative to the :}.
Some peculiarities on the ghci side mean that whitespace at the very start is lost.
This breaks the example `broken`, since the x and y aren't aligned from ghci's
perspective. A workaround is to avoid leading space, or add a newline such
that the indentation does not matter:
```haskell
{- | >>> :{
let x = 1
y = 2
in x + y + works
:}
6
-}
works = 3
{- | >>> :{
let x = 1
y = 2
in x + y + broken
:}
3
-}
broken = 3
```
## Multi-line output
If there are no blank lines in the output, multiple lines are handled
automatically.
```haskell
-- | >>> putStr "Hello\nWorld!"
-- Hello
-- World!
```
If however the output contains blank lines, they must be noted
explicitly with `<BLANKLINE>`. For example,
```haskell
import Data.List ( intercalate )
-- | Double-space a paragraph.
--
-- Examples:
--
-- >>> let s1 = "\"Every one of whom?\""
-- >>> let s2 = "\"Every one of whom do you think?\""
-- >>> let s3 = "\"I haven't any idea.\""
-- >>> let paragraph = unlines [s1,s2,s3]
-- >>> putStrLn $ doubleSpace paragraph
-- "Every one of whom?"
-- <BLANKLINE>
-- "Every one of whom do you think?"
-- <BLANKLINE>
-- "I haven't any idea."
--
doubleSpace :: String -> String
doubleSpace = (intercalate "\n\n") . lines
```
## Matching arbitrary output
Any lines containing only three dots (`...`) will match one or more lines with
arbitrary content. For instance,
```haskell
-- |
-- >>> putStrLn "foo\nbar\nbaz"
-- foo
-- ...
-- baz
```
If a line contains three dots and additional content, the three dots will match
anything *within that line*:
```haskell
-- |
-- >>> putStrLn "foo bar baz"
-- foo ... baz
```
## QuickCheck properties
Haddock (since version 2.13.0) has markup support for properties. Doctest can
verify properties with QuickCheck. A simple property looks like this:
```haskell
-- |
-- prop> \n -> abs n == abs (abs (n :: Int))
```
The lambda abstraction is optional and can be omitted:
```haskell
-- |
-- prop> abs n == abs (abs (n :: Int))
```
A complete example that uses setup code is below:
```haskell
module Fib where
-- $setup
-- >>> import Control.Applicative
-- >>> import Test.QuickCheck
-- >>> newtype Small = Small Int deriving Show
-- >>> instance Arbitrary Small where arbitrary = Small . (`mod` 10) <$> arbitrary
-- | Compute Fibonacci numbers
--
-- The following property holds:
--
-- prop> \(Small n) -> fib n == fib (n + 2) - fib (n + 1)
fib :: Int -> Int
fib 0 = 0
fib 1 = 1
fib n = fib (n - 1) + fib (n - 2)
```
If you see an error like the following, ensure that [QuickCheck](http://hackage.haskell.org/package/QuickCheck) is a dependency of your test-suite.
```haskell
<interactive>:39:3:
Not in scope: ‘polyQuickCheck’
In the splice: $(polyQuickCheck (mkName "doctest_prop"))
<interactive>:39:3:
GHC stage restriction:
‘polyQuickCheck’ is used in a top-level splice or annotation,
and must be imported, not defined locally
In the expression: polyQuickCheck (mkName "doctest_prop")
In the splice: $(polyQuickCheck (mkName "doctest_prop"))
```
## Hiding examples from Haddock
You can put examples into [named chunks][named-chunks], and not refer to them
in the export list. That way they will not be part of the generated Haddock
documentation, but Doctest will still find them.
```haskell
-- $
-- >>> 1 + 1
-- 2
```
[named-chunks]: http://www.haskell.org/haddock/doc/html/ch03s05.html
## Using GHC extensions
You can enable GHC extensions using the following syntax:
```haskell
-- >>> :set -XTupleSections
```
If you want to omit the information which language extensions are enabled from
the Doctest examples you can use the method described in [Hiding examples from
Haddock](#hiding-examples-from-haddock), e.g.:
```haskell
-- $
-- >>> :set -XTupleSections
```
[language-pragma]: http://www.haskell.org/ghc/docs/latest/html/users_guide/pragmas.html#language-pragma
## Using GHC plugins
You can enable GHC plugins using the following syntax:
```haskell
-- >>> :set -fplugin The.Plugin
```
## Hiding Prelude
You _hide_ the import of `Prelude` by using:
```haskell
-- >>> :m -Prelude
```
## Per module options
You can override command line flags per module by using a module annotation. For example, if you know a specific module does not support test order randomization, you can disable it with:
```haskell
{-# ANN module "doctest-parallel: --no-randomize-order" #-}
```
## Test non-exposed modules
Generally, `doctest-parallel` cannot test binders that are part of non-exposed modules, unless they are re-exported from exposed modules. By default `doctest-parallel` will fail to do so (and report an error message), because it doesn't track whether functions are re-exported in such a way. To test a re-exported function, add the following to the _non-exposed_ module:
```haskell
{-# ANN module "doctest-parallel: --no-implicit-module-import" #-}
```
This makes `doctest-parallel` omit the usual module import at the start of a test.
Then, before a test -or in `$setup`- add:
```haskell
>>> import Exposed.Module (someFunction)
```
# Relation to [`doctest`](https://github.com/sol/doctest)
This is a fork of [sol/doctest](https://github.com/sol/doctest) that allows running tests in parallel and aims to provide a more robust project integration method. It is not backwards compatible and expects to be setup differently. At the time of writing it has a few advantages over the base project:
* It runs tests in parallel
* It runs tests against compiled code, instead of reinterpreting your whole project
* It isolates examples in modules, ensuring your tests don't accidentally rely on each other
* It parses cabal files to discover modules, no need for custom setup anymore!
* A minor change: it does not count lines in setup blocks as test cases
* A minor change: the testsuite has been ported to v2 commands
All in all, you can expect `doctest-parallel` to run about 1 or 2 orders of magnitude faster than `doctest` for large projects.
# Relation to [`cabal-docspec`](https://github.com/phadej/cabal-extras/tree/master/cabal-docspec)
There is no direct relation between `doctest-parallel` and `cabal-docspec`. They are similar in some ways:
* Both projects load code from precompiled modules
* Both project aim to get rid of the need for custom setups
And different in others:
* As a fork of `doctest`, `doctest-parallel` inherits the testsuite `doctest` accumulated over the years.
* `doctest-parallel` parses Cabal project files, instead of parsing files from `dist-newstyle`. This makes it compatible with Stack, provided a `.cabal` is still present.
* `doctest-parallel` uses the GHC API to parse comments. This should in theory be more reliable (though I doubt it will ever matter in practice).
* `doctest-parallel` runs tests in parallel.
# Development
To run the tests:
```
cabal run spectests
cabal run doctests
```
# Future of this project
* It would be lovely if we could get rid of the needs for `write-ghc-environment-files: always` option for Cabal. To properly do this, I think Cabal should do two things:
1. Deprecate GHC environment files as a way to _implicitly_ setup environments. Instead, environment files should be written to the `dist-newstyle` directory and activated using some subcommand, e.g. `cabal shell`. This avoids the many problems GHC environment files have, while retaining their functionality for people who like them.
2. Any subcommands should be run with `GHC_ENVIRONMENT` set - pointing to the GHC environment file. Like Stack, this would create a hassle free way of using Cabal in combination with projects/executables that use the GHC API (e.g., `clash-ghc`, `doctest-parallel`).
* It would be nice if Cabal would expose more information _by default_ (probably through auto-generated modules) in order for `doctest-parallel` to properly work. Specifically, it needs to know the exact `default-extensions`, `ghc-options`, and `CPP` flags the project is compiled with. These options are obtainable by using a custom `Setup.hs`, but this has its own list of problems.
* Alternatively, if comments could be included in and loaded from `.hi` files that'd solve all issues too.
* Hopefully many of the improvements made here can make their way back into `sol/doctest`.
Of course, if you wish to add a feature that's not in this list, please feel free top open a pull request!
# Contributors
* Adam Vogt
* Anders Persson
* Ankit Ahuja
* Edward Kmett
* Hiroki Hattori
* Joachim Breitner
* João Cristóvão
* Julian Arni
* Kazu Yamamoto
* Levent Erkok
* Luke Murphy
* Matvey Aksenov
* Michael Orlitzky
* Michael Snoyman
* Nick Smallbone
* Phil de Joux
* Sakari Jokinen
* Simon Hengel
* Sönke Hahn