tasty-bench 0.2.5 → 0.3

raw patch · 4 files changed

+465/−139 lines, 4 filesdep ~tastyPVP ok

version bump matches the API change (PVP)

Dependency ranges changed: tasty

API changes (from Hackage documentation)

Files

• README.md +113/−22
• Test/Tasty/Bench.hs +338/−115
• changelog.md +9/−0
• tasty-bench.cabal +5/−2

@@ -1,4 +1,4 @@-# tasty-bench [![Hackage](http://img.shields.io/hackage/v/tasty-bench.svg)](https://hackage.haskell.org/package/tasty-bench) [![Stackage Nightly](http://stackage.org/package/tasty-bench/badge/nightly)](http://stackage.org/nightly/package/tasty-bench)+# tasty-bench [![Hackage](http://img.shields.io/hackage/v/tasty-bench.svg)](https://hackage.haskell.org/package/tasty-bench) [![Stackage LTS](http://stackage.org/package/tasty-bench/badge/lts)](http://stackage.org/lts/package/tasty-bench) [![Stackage Nightly](http://stackage.org/package/tasty-bench/badge/nightly)](http://stackage.org/nightly/package/tasty-bench)  Featherlight benchmark framework (only one file!) for performance measurement with API mimicking [`criterion`](http://hackage.haskell.org/package/criterion)@@ -13,7 +13,9 @@ So if you already depend on `tasty` for a test suite, there is nothing else to install. -Compare this to `criterion` (10+ modules, 50+ dependencies) and `gauge` (40+ modules, depends on `basement` and `vector`).+Compare this to `criterion` (10+ modules, 50+ dependencies) and `gauge` (40+ modules, depends on `basement` and `vector`). A build on a clean machine is up to 16x+faster than `criterion` and up to 4x faster than `gauge`. A build without dependencies+is up to 6x faster than `criterion` and up to 8x faster than `gauge`.  ## How is it possible? @@ -151,10 +153,27 @@ for [68%](https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule) of samples only, double it to estimate the behavior in 95% of cases. -When benchmarking multithreaded algorithms, note-that `tasty-bench` reports total elapsed CPU time across all cores, while-`criterion` and `gauge` print wall-clock time.+## Wall-clock time vs. CPU time +What time are we talking about?+Both `criterion` and `gauge` by default report wall-clock time, which is+affected by any other application which runs concurrently.+While ideally benchmarks are executed on a dedicated server without any other load,+but — let's face the truth — most of developers run benchmarks+on a laptop with a hundred other services and a window manager, and+watch videos while waiting for benchmarks to finish. That's the cause+of a notorious "variance introduced by outliers: 88% (severely inflated)" warning.++To alleviate this issue `tasty-bench` measures CPU time by `getCPUTime`+instead of wall-clock time.+It does not provide a perfect isolation from other processes (e. g.,+if CPU cache is spoiled by others, populating data back from RAM+is your burden), but is significantly more stable.++Caveat: this means that for multithreaded algorithms+`tasty-bench` reports total elapsed CPU time across all cores, while+`criterion` and `gauge` print maximum of core's wall-clock time.+ ## Statistical model  Here is a procedure used by `tasty-bench` to measure execution time:@@ -162,8 +181,10 @@ 1. Set _n_ ← 1. 2. Measure execution time _tₙ_ of _n_ iterations    and execution time _t₂ₙ_ of _2n_ iterations.-3. Find _t_ which minimizes deviation of (_nt_, _2nt_) from (_tₙ_, _t₂ₙ_).-4. If deviation is small enough (see `--stdev` below),+3. Find _t_ which minimizes deviation of (_nt_, _2nt_) from (_tₙ_, _t₂ₙ_),+   namely _t_ ← (_tₙ_ + _2t₂ₙ_) / _5n_.+4. If deviation is small enough (see `--stdev` below)+   or time is running out soon (see `--timeout` below),    return _t_ as a mean execution time. 5. Otherwise set _n_ ← _2n_ and jump back to Step 2. @@ -198,21 +219,24 @@ Configuring RTS to collect GC statistics (e. g., via `cabal bench --benchmark-options '+RTS -T'` or `stack bench --ba '+RTS -T'`) enables `tasty-bench` to estimate and report-memory usage such as allocated and copied bytes:+memory usage:  ``` All   fibonacci numbers     fifth:     OK (2.13s)-       63 ns ± 3.4 ns, 223 B  allocated,   0 B  copied+       63 ns ± 3.4 ns, 223 B  allocated,   0 B  copied, 2.0 MB peak memory     tenth:     OK (1.71s)-      809 ns ±  73 ns, 2.3 KB allocated,   0 B  copied+      809 ns ±  73 ns, 2.3 KB allocated,   0 B  copied, 4.0 MB peak memory     twentieth: OK (3.39s)-      104 μs ± 4.9 μs, 277 KB allocated,  59 B  copied+      104 μs ± 4.9 μs, 277 KB allocated,  59 B  copied, 5.0 MB peak memory  All 3 tests passed (7.25s) ``` +This data is reported as per `RTSStats` fields: `allocated_bytes`, `copied_bytes`+and `max_mem_in_use_bytes`.+ ## Combining tests and benchmarks  When optimizing an existing function, it is important to check that its@@ -282,7 +306,7 @@   Alternatively bake it into   `cabal` file as `ghc-options: "-with-rtsopts=-A32m"`. -  For GHC >= 8.10 consider switching benchmarks to a non-moving garbage collector,+  For GHC ≥ 8.10 consider switching benchmarks to a non-moving garbage collector,   because it decreases GC pauses and corresponding noise: `+RTS --nonmoving-gc`.  * If benchmark results look malformed like below, make sure that you are@@ -301,7 +325,13 @@   Unhandled resource. Probably a bug in the runner you're using.   ``` -  this is probably caused by `env` or `envWithCleanup` affecting benchmarks structure.+or++  ```+  Unexpected state of the resource (NotCreated) in getResource. Report as a tasty bug.+  ```++  this is likely caused by `env` or `envWithCleanup` affecting benchmarks structure.   You can use `env` to read test data from `IO`, but not to read benchmark names   or affect their hierarchy in other way. This is a fundamental restriction of `tasty`   to list and filter benchmarks without launching missiles.@@ -362,14 +392,14 @@  ```bash cabal build --enable-benchmarks my-bench-MYBENCH=`cabal list-bin my-bench`+MYBENCH=$(cabal list-bin my-bench) ```  Now list all benchmark names (hopefully, they do not contain newlines), escape quotes and slashes, and run each of them separately:  ```bash-$MYBENCH -l | sed -e 's/[\"]/\\\\\\&/g' | while read name; do $MYBENCH -p '$0 == "'$name'"'; done+$MYBENCH -l | sed -e 's/[\"]/\\\\\\&/g' | while read -r name; do $MYBENCH -p '$0 == "'"$name"'"'; done ```  ## Comparison against baseline@@ -386,10 +416,6 @@ All.fibonacci numbers.twentieth,81369531,3342646 ``` -Note that columns do not match CSV reports of `criterion` and `gauge`.-If desired, missing columns can be faked with-`awk 'BEGIN {FS=",";OFS=","}; {print $1,$2,$2,$2,$3/2,$3/2,$3/2}'` or similar.- Now modify implementation and rerun benchmarks with `--baseline FILE` key. This produces a report as follows: @@ -415,6 +441,28 @@ [`tasty-rerun`](http://hackage.haskell.org/package/tasty-rerun) package to focus on rerunning failing items only. +Note that columns in CSV report are different from what `criterion` or `gauge`+would produce. If names do not contain commas, missing columns can be faked this way:++```sh+cat tasty-bench.csv \+| awk 'BEGIN {FS=",";OFS=","}; {print $1,$2/1e12,$2/1e12,$2/1e12,$3/2e12,$3/2e12,$3/2e12}' \+| sed '1s/.*/Name,Mean,MeanLB,MeanUB,Stddev,StddevLB,StddevUB/'+```++To fake `gauge` in `--csvraw` mode use++```sh+cat tasty-bench.csv \+| awk 'BEGIN {FS=",";OFS=","}; {print $1,1,$2/1e12,0,$2/1e12,$2/1e12,0,$6+0,0,0,0,0,$4+0,0,$5+0,0,0,0,0}' \+| sed '1s/.*/name,iters,time,cycles,cpuTime,utime,stime,maxrss,minflt,majflt,nvcsw,nivcsw,allocated,numGcs,bytesCopied,mutatorWallSeconds,mutatorCpuSeconds,gcWallSeconds,gcCpuSeconds/'+```++Please refer to `gawk` manual, if you wish to process names with+[commas](https://www.gnu.org/software/gawk/manual/gawk.html#Splitting-By-Content)+or+[quotes](https://www.gnu.org/software/gawk/manual/gawk.html#More-CSV).+ ## Comparison between benchmarks  You can also compare benchmarks to each other without reaching to external tools,@@ -463,8 +511,10 @@ Users can dump results into CSV with `--csv FILE` and plot them using `gnuplot` or other software. But for convenience there is also a built-in quick-and-dirty SVG plotting feature,-which can be invoked by passing `--svg FILE`.+which can be invoked by passing `--svg FILE`. Here is a sample of its output: +![Plotting](./example.svg)+ ## Command-line options  Use `--help` to list command-line options.@@ -482,13 +532,16 @@   in seconds. Use it when benchmarks tend to take too long: `tasty-bench` will make   an effort to report results (even if of subpar quality) before timeout. Setting   timeout too tight (insufficient for at least three iterations)-  will result in a benchmark failure.+  will result in a benchmark failure. One can adjust it locally for a group+  of benchmarks, e. g., `localOption (mkTimeout 100000000)` for 100 seconds.  * `--stdev`    Target relative standard deviation of measurements in percents (5% by default).   Large values correspond to fast and loose benchmarks, and small ones to long and precise.-  If it takes far too long, consider setting `--timeout`,+  It can also be adjusted locally for a group of benchmarks,+  e. g., `localOption (RelStDev 0.02)`.+  If benchmarking takes far too long, consider setting `--timeout`,   which will interrupt benchmarks, potentially before reaching the target deviation.  * `--csv`@@ -504,7 +557,45 @@   Upper bounds of acceptable slow down / speed up in percents. If a benchmark is unacceptably slower / faster than baseline (see `--baseline`),   it will be reported as failed. Can be used in conjunction with   a standard `tasty` option `--hide-successes` to show only problematic benchmarks.+  Both options can be adjusted locally for a group of benchmarks,+  e. g., `localOption (FailIfSlower 0.10)`.  * `--svg`    File to plot results in SVG format.++* `+RTS -T`++  Estimate and report memory usage.++## Custom command-line options++As usual with `tasty`, it is easy to extend benchmarks with custom command-line options.+Here is an example:++```haskell+import Data.Proxy+import Test.Tasty.Bench+import Test.Tasty.Ingredients.Basic+import Test.Tasty.Options+import Test.Tasty.Runners++newtype RandomSeed = RandomSeed Int++instance IsOption RandomSeed where+  defaultValue = RandomSeed 42+  parseValue = fmap RandomSeed . safeRead+  optionName = pure "seed"+  optionHelp = pure "Random seed used in benchmarks"++main :: IO ()+main = do+  let customOpts  = [Option (Proxy :: Proxy RandomSeed)]+      ingredients = includingOptions customOpts : benchIngredients+  opts <- parseOptions ingredients benchmarks+  let RandomSeed seed = lookupOption opts+  defaultMainWithIngredients ingredients benchmarks++benchmarks :: Benchmark+benchmarks = bgroup "All" []+```

@@ -18,7 +18,9 @@ to install.  Compare this to @criterion@ (10+ modules, 50+ dependencies) and @gauge@-(40+ modules, depends on @basement@ and @vector@).+(40+ modules, depends on @basement@ and @vector@). A build on a clean machine is up to 16x+faster than @criterion@ and up to 4x faster than @gauge@. A build without dependencies+is up to 6x faster than @criterion@ and up to 8x faster than @gauge@.  === How is it possible? @@ -146,10 +148,27 @@ <https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule 68%> of samples only, double it to estimate the behavior in 95% of cases. -When benchmarking multithreaded algorithms, note-that @tasty-bench@ reports total elapsed CPU time across all cores, while-@criterion@ and @gauge@ print wall-clock time.+=== Wall-clock time vs. CPU time +What time are we talking about?+Both @criterion@ and @gauge@ by default report wall-clock time, which is+affected by any other application which runs concurrently.+While ideally benchmarks are executed on a dedicated server without any other load,+but — let's face the truth — most of developers run benchmarks+on a laptop with a hundred other services and a window manager, and+watch videos while waiting for benchmarks to finish. That's the cause+of a notorious "variance introduced by outliers: 88% (severely inflated)" warning.++To alleviate this issue @tasty-bench@ measures CPU time by 'getCPUTime'+instead of wall-clock time.+It does not provide a perfect isolation from other processes (e. g.,+if CPU cache is spoiled by others, populating data back from RAM+is your burden), but is significantly more stable.++Caveat: this means that for multithreaded algorithms+@tasty-bench@ reports total elapsed CPU time across all cores, while+@criterion@ and @gauge@ print maximum of core's wall-clock time.+ === Statistical model  Here is a procedure used by @tasty-bench@ to measure execution time:@@ -158,9 +177,10 @@ 2.  Measure execution time \( t_n \) of \( n \) iterations and execution time     \( t_{2n} \) of \( 2n \) iterations. 3.  Find \( t \) which minimizes deviation of \( (nt, 2nt) \) from-    \( (t_n, t_{2n}) \).-4.  If deviation is small enough (see @--stdev@ below), return \( t \) as a-    mean execution time.+    \( (t_n, t_{2n}) \), namely \( t \leftarrow (t_n + 2t_{2n}) / 5n \).+4.  If deviation is small enough (see @--stdev@ below)+    or time is running out soon (see @--timeout@ below),+    return \( t \) as a mean execution time. 5.  Otherwise set \( n \leftarrow 2n \) and jump back to Step 2.  This is roughly similar to the linear regression approach which@@ -191,19 +211,22 @@ Configuring RTS to collect GC statistics (e. g., via @cabal@ @bench@ @--benchmark-options@ @\'+RTS@ @-T\'@ or @stack@ @bench@ @--ba@ @\'+RTS@ @-T\'@) enables @tasty-bench@ to estimate and-report memory usage such as allocated and copied bytes:+report memory usage:  > All >   fibonacci numbers >     fifth:     OK (2.13s)->        63 ns ± 3.4 ns, 223 B  allocated,   0 B  copied+>        63 ns ± 3.4 ns, 223 B  allocated,   0 B  copied, 2.0 MB peak memory >     tenth:     OK (1.71s)->       809 ns ±  73 ns, 2.3 KB allocated,   0 B  copied+>       809 ns ±  73 ns, 2.3 KB allocated,   0 B  copied, 4.0 MB peak memory >     twentieth: OK (3.39s)->       104 μs ± 4.9 μs, 277 KB allocated,  59 B  copied+>       104 μs ± 4.9 μs, 277 KB allocated,  59 B  copied, 5.0 MB peak memory > > All 3 tests passed (7.25s) +This data is reported as per 'RTSStats' fields: 'allocated_bytes', 'copied_bytes'+and 'max_mem_in_use_bytes'.+ === Combining tests and benchmarks  When optimizing an existing function, it is important to check that its@@ -270,7 +293,7 @@     @stack@ @bench@ @--ba@ @\'+RTS@ @-A32m\'@. Alternatively bake it into @cabal@     file as @ghc-options:@ @\"-with-rtsopts=-A32m\"@. -    For GHC >= 8.10 consider switching benchmarks to a non-moving garbage collector,+    For GHC ≥ 8.10 consider switching benchmarks to a non-moving garbage collector,     because it decreases GC pauses and corresponding noise: @+RTS@ @--nonmoving-gc@.  -   If benchmark results look malformed like below, make sure that you@@ -286,7 +309,11 @@      > Unhandled resource. Probably a bug in the runner you're using. -    this is probably caused by 'env' or 'envWithCleanup' affecting+    or++    > Unexpected state of the resource (NotCreated) in getResource. Report as a tasty bug.++    this is likely caused by 'env' or 'envWithCleanup' affecting     benchmarks structure. You can use 'env' to read test data from 'IO',     but not to read benchmark names or affect their hierarchy in other     way. This is a fundamental restriction of @tasty@ to list and filter@@ -343,12 +370,12 @@ @benchmark@ @my-bench@ component, let’s first find its executable:  > cabal build --enable-benchmarks my-bench-> MYBENCH=`cabal list-bin my-bench`+> MYBENCH=$(cabal list-bin my-bench)  Now list all benchmark names (hopefully, they do not contain newlines), escape quotes and slashes, and run each of them separately: -> $MYBENCH -l | sed -e 's/[\"]/\\\\\\&/g' | while read name; do $MYBENCH -p '$0 == "'$name'"'; done+> $MYBENCH -l | sed -e 's/[\"]/\\\\\\&/g' | while read -r name; do $MYBENCH -p '$0 == "'"$name"'"'; done  === Comparison against baseline @@ -362,11 +389,6 @@ > All.fibonacci numbers.tenth,637152,46744 > All.fibonacci numbers.twentieth,81369531,3342646 -Note that columns do not match CSV reports of @criterion@ and @gauge@.-If desired, missing columns can be faked with-@awk@ @\'BEGIN@ @{FS=\",\";OFS=\",\"};@ @{print@ @$1,$2,$2,$2,$3\/2,$3\/2,$3\/2}\'@-or similar.- Now modify implementation and rerun benchmarks with @--baseline@ @FILE@ key. This produces a report as follows: @@ -390,6 +412,19 @@ even [@tasty-rerun@](http://hackage.haskell.org/package/tasty-rerun) package to focus on rerunning failing items only. +Note that columns in CSV report are different from what @criterion@ or @gauge@+would produce. If names do not contain commas, missing columns can be faked this way:++> cat tasty-bench.csv | awk 'BEGIN {FS=",";OFS=","}; {print $1,$2/1e12,$2/1e12,$2/1e12,$3/2e12,$3/2e12,$3/2e12}' | sed '1s/.*/Name,Mean,MeanLB,MeanUB,Stddev,StddevLB,StddevUB/'++To fake @gauge@ in @--csvraw@ mode use++> cat tasty-bench.csv | awk 'BEGIN {FS=",";OFS=","}; {print $1,1,$2/1e12,0,$2/1e12,$2/1e12,0,$6+0,0,0,0,0,$4+0,0,$5+0,0,0,0,0}' | sed '1s/.*/name,iters,time,cycles,cpuTime,utime,stime,maxrss,minflt,majflt,nvcsw,nivcsw,allocated,numGcs,bytesCopied,mutatorWallSeconds,mutatorCpuSeconds,gcWallSeconds,gcCpuSeconds/'++Please refer to @gawk@ manual, if you wish to process names+with [commas](https://www.gnu.org/software/gawk/manual/gawk.html#Splitting-By-Content)+or [quotes](https://www.gnu.org/software/gawk/manual/gawk.html#More-CSV).+ === Comparison between benchmarks  You can also compare benchmarks to each other without reaching to@@ -455,14 +490,18 @@     @tasty-bench@ will make an effort to report results (even if of     subpar quality) before timeout. Setting timeout too tight     (insufficient for at least three iterations) will result in a-    benchmark failure.+    benchmark failure. One can adjust it locally for a group+    of benchmarks, e. g., 'localOption' ('mkTimeout' 100000000) for 100 seconds.  [@--stdev@]:      Target relative standard deviation of measurements in percents (5%     by default). Large values correspond to fast and loose benchmarks,-    and small ones to long and precise. If it takes far too long,-    consider setting @--timeout@, which will interrupt benchmarks,+    and small ones to long and precise.+    It can also be adjusted locally for a group of benchmarks,+    e. g., 'localOption' ('RelStDev' 0.02).+    If benchmarking takes far too long, consider setting @--timeout@,+    which will interrupt benchmarks,     potentially before reaching the target deviation.  [@--csv@]:@@ -481,11 +520,47 @@     @--baseline@), it will be reported as failed. Can be used in     conjunction with a standard @tasty@ option @--hide-successes@ to     show only problematic benchmarks.+    Both options can be adjusted locally for a group of benchmarks,+    e. g., 'localOption' ('FailIfSlower' 0.10).  [@--svg@]:      File to plot results in SVG format. +[@+RTS@ @-T@]:++    Estimate and report memory usage.++=== Custom command-line options++As usual with @tasty@, it is easy to extend benchmarks with custom command-line options.+Here is an example:++> import Data.Proxy+> import Test.Tasty.Bench+> import Test.Tasty.Ingredients.Basic+> import Test.Tasty.Options+> import Test.Tasty.Runners+>+> newtype RandomSeed = RandomSeed Int+>+> instance IsOption RandomSeed where+>   defaultValue = RandomSeed 42+>   parseValue = fmap RandomSeed . safeRead+>   optionName = pure "seed"+>   optionHelp = pure "Random seed used in benchmarks"+>+> main :: IO ()+> main = do+>   let customOpts  = [Option (Proxy :: Proxy RandomSeed)]+>       ingredients = includingOptions customOpts : benchIngredients+>   opts <- parseOptions ingredients benchmarks+>   let RandomSeed seed = lookupOption opts+>   defaultMainWithIngredients ingredients benchmarks+>+> benchmarks :: Benchmark+> benchmarks = bgroup "All" []+ -}  {-# LANGUAGE CPP #-}@@ -502,17 +577,20 @@   , Benchmark   , bench   , bgroup+#if MIN_VERSION_tasty(1,2,0)   , bcompare+#endif   , env   , envWithCleanup   -- * Creating 'Benchmarkable'-  , Benchmarkable+  , Benchmarkable(..)   , nf   , whnf   , nfIO   , whnfIO   , nfAppIO   , whnfAppIO+  , measureCpuTime   -- * Ingredients   , benchIngredients   , consoleBenchReporter@@ -521,6 +599,9 @@   , RelStDev(..)   , FailIfSlower(..)   , FailIfFaster(..)+  , CsvPath(..)+  , BaselinePath(..)+  , SvgPath(..)   ) where  import Prelude hiding (Int, Integer)@@ -533,16 +614,17 @@ import Data.Foldable (foldMap, traverse_) import Data.Int (Int64) import Data.IntMap (IntMap)+#if MIN_VERSION_containers(0,5,0)+import qualified Data.IntMap.Strict as IM+#else import qualified Data.IntMap as IM+#endif import Data.IORef-import Data.List (intercalate, stripPrefix, isPrefixOf, foldl', genericLength, genericDrop)+import Data.List (intercalate, stripPrefix, isPrefixOf, genericLength, genericDrop) import Data.Monoid (All(..), Any(..)) import Data.Proxy import Data.Sequence (Seq, (<|)) import qualified Data.Sequence as Seq-#if MIN_VERSION_containers(0,5,0)-import Data.Set (lookupGE)-#endif import qualified Data.Set as S import Data.Traversable (forM) import Data.Word (Word64)@@ -557,8 +639,10 @@ import Test.Tasty.Ingredients import Test.Tasty.Ingredients.ConsoleReporter import Test.Tasty.Options+#if MIN_VERSION_tasty(1,2,0) import Test.Tasty.Patterns.Eval (eval, asB, withFields) import Test.Tasty.Patterns.Types (Expr (And, StringLit))+#endif import Test.Tasty.Providers import Test.Tasty.Runners import Text.Printf@@ -573,8 +657,16 @@ -- -- E. g., set target relative standard deviation to 2% as follows: --+-- > import Test.Tasty (localOption) -- > localOption (RelStDev 0.02) (bgroup [...]) --+-- If you set 'RelStDev' to infinity,+-- a benchmark will be executed+-- only once and its standard deviation will be recorded as zero.+-- This is rather a blunt approach, but it might be a necessary evil+-- for extremely long benchmarks. If you wish to run all benchmarks+-- only once, use command-line option @--stdev@ @Infinity@.+-- newtype RelStDev = RelStDev Double   deriving (Show, Read, Typeable) @@ -592,6 +684,7 @@ -- -- E. g., set upper bound of acceptable slow down to 10% as follows: --+-- > import Test.Tasty (localOption) -- > localOption (FailIfSlower 0.10) (bgroup [...]) -- newtype FailIfSlower = FailIfSlower Double@@ -611,6 +704,7 @@ -- -- E. g., set upper bound of acceptable speed up to 10% as follows: --+-- > import Test.Tasty (localOption) -- > localOption (FailIfFaster 0.10) (bgroup [...]) -- newtype FailIfFaster = FailIfFaster Double@@ -633,11 +727,13 @@ -- -- Drop-in replacement for 'Criterion.Benchmarkable' and 'Gauge.Benchmarkable'. ---newtype Benchmarkable = Benchmarkable { _unBenchmarkable :: Int64 -> IO () }-  deriving (Typeable)+newtype Benchmarkable = Benchmarkable+  { unBenchmarkable :: Word64 -> IO () -- ^ Run benchmark given number of times.+  } deriving (Typeable) -showPicos :: Word64 -> String-showPicos i+-- | Show picoseconds, fitting number in 3 characters.+showPicos3 :: Word64 -> String+showPicos3 i   | t < 995   = printf "%3.0f ps" t   | t < 995e1 = printf "%3.1f ns" (t / 1e3)   | t < 995e3 = printf "%3.0f ns" (t / 1e3)@@ -647,9 +743,25 @@   | t < 995e9 = printf "%3.0f ms" (t / 1e9)   | otherwise = printf "%4.2f s"  (t / 1e12)   where-    t :: Double-    t = fromIntegral i+    t = word64ToDouble i +-- | Show picoseconds, fitting number in 4 characters.+showPicos4 :: Word64 -> String+showPicos4 i+  | t < 995   = printf "%3.0f  ps"  t+  | t < 995e1 = printf "%4.2f ns"  (t / 1e3)+  | t < 995e2 = printf "%4.1f ns"  (t / 1e3)+  | t < 995e3 = printf "%3.0f  ns" (t / 1e3)+  | t < 995e4 = printf "%4.2f μs"  (t / 1e6)+  | t < 995e5 = printf "%4.1f μs"  (t / 1e6)+  | t < 995e6 = printf "%3.0f  μs" (t / 1e6)+  | t < 995e7 = printf "%4.2f ms"  (t / 1e9)+  | t < 995e8 = printf "%4.1f ms"  (t / 1e9)+  | t < 995e9 = printf "%3.0f  ms" (t / 1e9)+  | otherwise = printf "%4.3f s"   (t / 1e12)+  where+    t = word64ToDouble i+ showBytes :: Word64 -> String showBytes i   | t < 1000                 = printf "%3.0f B " t@@ -666,13 +778,13 @@   | t < 11471568970838126592 = printf "%3.1f EB" (t / 1152921504606846976)   | otherwise                = printf "%3.0f EB" (t / 1152921504606846976)   where-    t :: Double-    t = fromIntegral i+    t = word64ToDouble i  data Measurement = Measurement   { measTime   :: !Word64 -- ^ time in picoseconds   , measAllocs :: !Word64 -- ^ allocations in bytes   , measCopied :: !Word64 -- ^ copied bytes+  , measMaxMem :: !Word64 -- ^ max memory in use   } deriving (Show, Read)  data Estimate = Estimate@@ -688,35 +800,38 @@  prettyEstimate :: Estimate -> String prettyEstimate (Estimate m stdev) =-  showPicos (measTime m) ++ " ± " ++ showPicos (2 * stdev)+  showPicos4 (measTime m)+  ++ (if stdev == 0 then "         " else " ± " ++ showPicos3 (2 * stdev))  prettyEstimateWithGC :: Estimate -> String prettyEstimateWithGC (Estimate m stdev) =-  showPicos (measTime m) ++ " ± " ++ showPicos (2 * stdev)-  ++ ", " ++ showBytes (measAllocs m) ++ " allocated, "-  ++ showBytes (measCopied m) ++ " copied"+  showPicos4 (measTime m)+  ++ (if stdev == 0 then ",          " else " ± " ++ showPicos3 (2 * stdev) ++ ", ")+  ++ showBytes (measAllocs m) ++ " allocated, "+  ++ showBytes (measCopied m) ++ " copied, "+  ++ showBytes (measMaxMem m) ++ " peak memory"  csvEstimate :: Estimate -> String csvEstimate (Estimate m stdev) = show (measTime m) ++ "," ++ show (2 * stdev)  csvEstimateWithGC :: Estimate -> String csvEstimateWithGC (Estimate m stdev) = show (measTime m) ++ "," ++ show (2 * stdev)-  ++ "," ++ show (measAllocs m) ++ "," ++ show (measCopied m)+  ++ "," ++ show (measAllocs m) ++ "," ++ show (measCopied m) ++ "," ++ show (measMaxMem m)  predict   :: Measurement -- ^ time for one run   -> Measurement -- ^ time for two runs   -> Estimate-predict (Measurement t1 a1 c1) (Measurement t2 a2 c2) = Estimate-  { estMean  = Measurement t (fit a1 a2) (fit c1 c2)+predict (Measurement t1 a1 c1 m1) (Measurement t2 a2 c2 m2) = Estimate+  { estMean  = Measurement t (fit a1 a2) (fit c1 c2) (max m1 m2)   , estStdev = truncate (sqrt d :: Double)   }   where     fit x1 x2 = x1 `quot` 5 + 2 * (x2 `quot` 5)     t = fit t1 t2     sqr x = x * x-    d = sqr (fromIntegral t1 -     fromIntegral t)-      + sqr (fromIntegral t2 - 2 * fromIntegral t)+    d = sqr (word64ToDouble t1 -     word64ToDouble t)+      + sqr (word64ToDouble t2 - 2 * word64ToDouble t)  predictPerturbed :: Measurement -> Measurement -> Estimate predictPerturbed t1 t2 = Estimate@@ -730,69 +845,88 @@     hi meas = meas { measTime = measTime meas + prec }     lo meas = meas { measTime = measTime meas - prec } -#if !MIN_VERSION_base(4,10,0)-getRTSStatsEnabled :: IO Bool-#if MIN_VERSION_base(4,6,0)-getRTSStatsEnabled = getGCStatsEnabled+hasGCStats :: Bool+#if MIN_VERSION_base(4,10,0)+hasGCStats = unsafePerformIO getRTSStatsEnabled+#elif MIN_VERSION_base(4,6,0)+hasGCStats = unsafePerformIO getGCStatsEnabled #else-getRTSStatsEnabled = pure False-#endif+hasGCStats = False #endif -getAllocsAndCopied :: IO (Word64, Word64)+getAllocsAndCopied :: IO (Word64, Word64, Word64) getAllocsAndCopied = do-  enabled <- getRTSStatsEnabled-  if not enabled then pure (0, 0) else+  if not hasGCStats then pure (0, 0, 0) else #if MIN_VERSION_base(4,10,0)-    (\s -> (allocated_bytes s, copied_bytes s)) <$> getRTSStats+    (\s -> (allocated_bytes s, copied_bytes s, max_mem_in_use_bytes s)) <$> getRTSStats #elif MIN_VERSION_base(4,6,0)-    (\s -> (fromIntegral $ bytesAllocated s, fromIntegral $ bytesCopied s)) <$> getGCStats+    (\s -> (int64ToWord64 $ bytesAllocated s, int64ToWord64 $ bytesCopied s, int64ToWord64 $ peakMegabytesAllocated s * 1024 * 1024)) <$> getGCStats #else-    pure (0, 0)+    pure (0, 0, 0) #endif -measureTime :: Int64 -> Benchmarkable -> IO Measurement-measureTime n (Benchmarkable act) = do+measure :: Word64 -> Benchmarkable -> IO Measurement+measure n (Benchmarkable act) = do   performGC   startTime <- fromInteger <$> getCPUTime-  (startAllocs, startCopied) <- getAllocsAndCopied+  (startAllocs, startCopied, startMaxMemInUse) <- getAllocsAndCopied   act n   endTime <- fromInteger <$> getCPUTime-  (endAllocs, endCopied) <- getAllocsAndCopied+  (endAllocs, endCopied, endMaxMemInUse) <- getAllocsAndCopied   pure $ Measurement     { measTime   = endTime - startTime     , measAllocs = endAllocs - startAllocs     , measCopied = endCopied - startCopied+    , measMaxMem = max endMaxMemInUse startMaxMemInUse     } -measureTimeUntil :: Timeout -> RelStDev -> Benchmarkable -> IO Estimate-measureTimeUntil timeout (RelStDev targetRelStDev) b = do-  t1 <- measureTime 1 b+measureUntil :: Bool -> Timeout -> RelStDev -> Benchmarkable -> IO Estimate+measureUntil _ _ (RelStDev targetRelStDev) b+  | isInfinite targetRelStDev, targetRelStDev > 0 = do+  t1 <- measure 1 b+  pure $ Estimate { estMean = t1, estStdev = 0 }+measureUntil warnIfNoTimeout timeout (RelStDev targetRelStDev) b = do+  t1 <- measure 1 b   go 1 t1 0   where-    go :: Int64 -> Measurement -> Word64 -> IO Estimate+    go :: Word64 -> Measurement -> Word64 -> IO Estimate     go n t1 sumOfTs = do-      t2 <- measureTime (2 * n) b+      t2 <- measure (2 * n) b -      let Estimate (Measurement meanN allocN copiedN) stdevN = predictPerturbed t1 t2+      let Estimate (Measurement meanN allocN copiedN maxMemN) stdevN = predictPerturbed t1 t2           isTimeoutSoon = case timeout of             NoTimeout -> False             -- multiplying by 12/10 helps to avoid accidental timeouts-            Timeout micros _ -> (sumOfTs + measTime t1 + 3 * measTime t2) `quot` (1000000 * 10 `quot` 12) >= fromInteger micros-          isStDevInTargetRange = stdevN < truncate (max 0 targetRelStDev * fromIntegral meanN)-          scale = (`quot` fromIntegral n)+            Timeout micros _ -> (sumOfTs' + 3 * measTime t2) `quot` (1000000 * 10 `quot` 12) >= fromInteger micros+          isStDevInTargetRange = stdevN < truncate (max 0 targetRelStDev * word64ToDouble meanN)+          scale = (`quot` n)           sumOfTs' = sumOfTs + measTime t1        case timeout of-        NoTimeout | sumOfTs' > 100 * 1000000000000+        NoTimeout | warnIfNoTimeout, sumOfTs' + measTime t2 > 100 * 1000000000000           -> hPutStrLn stderr "This benchmark takes more than 100 seconds. Consider setting --timeout, if this is unexpected (or to silence this warning)."         _ -> pure () -       if isStDevInTargetRange || isTimeoutSoon-        then pure $ Estimate (Measurement (scale meanN) (scale allocN) (scale copiedN)) (scale stdevN)+        then pure $ Estimate+          { estMean  = Measurement (scale meanN) (scale allocN) (scale copiedN) maxMemN+          , estStdev = scale stdevN }         else go (2 * n) t2 sumOfTs' +-- | An internal routine to measure execution time in seconds+-- for a given timeout (put 'NoTimeout', or 'mkTimeout' 100000000 for 100 seconds)+-- and a target relative standard deviation+-- (put 'RelStDev' 0.05 for 5% or 'RelStDev' (1/0) to run only one iteration).+--+-- 'Timeout' takes soft priority over 'RelStDev': this function prefers+-- to finish in time even if at cost of precision. However, timeout is guidance+-- not guarantee: 'measureCpuTime' can take longer, if there is not enough time+-- to run at least thrice or an iteration takes unusually long.+measureCpuTime :: Timeout -> RelStDev -> Benchmarkable -> IO Double+measureCpuTime+    = ((fmap ((/ 1e12) . word64ToDouble . measTime . estMean) .) .)+    . measureUntil False+ instance IsTest Benchmarkable where   testOptions = pure     [ Option (Proxy :: Proxy RelStDev)@@ -803,9 +937,9 @@     ]   run opts b = const $ case getNumThreads (lookupOption opts) of     1 -> do-      est <- measureTimeUntil (lookupOption opts) (lookupOption opts) b+      est <- measureUntil True (lookupOption opts) (lookupOption opts) b       pure $ testPassed $ show (Response est (lookupOption opts) (lookupOption opts))-    _ -> pure $ testFailed "Benchmarks must not be run concurrently. Please pass --jobs 1 and/or avoid +RTS -N."+    _ -> pure $ testFailed "Benchmarks must not be run concurrently. Please pass -j1 and/or avoid +RTS -N."  -- | Attach a name to 'Benchmarkable'. --@@ -824,6 +958,7 @@ bgroup :: String -> [Benchmark] -> Benchmark bgroup = testGroup +#if MIN_VERSION_tasty(1,2,0) -- | Compare benchmarks, reporting relative speed up or slow down. -- -- The first argument is a @tasty@ pattern, which must unambiguously@@ -837,12 +972,13 @@ -- -- This function is a vague reminiscence of @bcompare@, which existed in pre-1.0 -- versions of @criterion@, but their types are incompatible. Under the hood--- 'bcompare' is a thin wrapper over 'after'.+-- 'bcompare' is a thin wrapper over 'after' and requires @tasty-1.2@. -- bcompare :: String -> Benchmark -> Benchmark bcompare s = case parseExpr s of   Nothing -> error $ "Could not parse bcompare pattern " ++ s   Just e  -> after_ AllSucceed (And (StringLit "tasty-bench") e)+#endif  -- | Benchmarks are actually just a regular 'Test.Tasty.TestTree' in disguise. --@@ -866,7 +1002,7 @@ funcToBench frc = (Benchmarkable .) . go   where     go f x n-      | n <= 0    = pure ()+      | n == 0    = pure ()       | otherwise = do         _ <- evaluate (frc (f x))         go f x (n - 1)@@ -944,7 +1080,7 @@ ioToBench frc act = Benchmarkable go   where     go n-      | n <= 0    = pure ()+      | n == 0    = pure ()       | otherwise = do         val <- act         _ <- evaluate (frc val)@@ -955,8 +1091,9 @@ -- and compute its normal form (by means of 'force'). -- -- Pure subexpression of an effectful computation @x@--- may be evaluated only once and get cached; use 'nfAppIO'--- to avoid this.+-- may be evaluated only once and get cached.+-- To avoid surprising results it is usually preferable+-- to use 'nfAppIO' instead. -- -- Note that forcing a normal form requires an additional -- traverse of the structure. In certain scenarios,@@ -978,8 +1115,9 @@ -- and compute its weak head normal form. -- -- Pure subexpression of an effectful computation @x@--- may be evaluated only once and get cached; use 'whnfAppIO'--- to avoid this.+-- may be evaluated only once and get cached.+-- To avoid surprising results it is usually preferable+-- to use 'whnfAppIO' instead. -- -- Computing only a weak head normal form is -- rarely what intuitively is meant by "evaluation".@@ -1001,7 +1139,7 @@ ioFuncToBench frc = (Benchmarkable .) . go   where     go f x n-      | n <= 0    = pure ()+      | n == 0    = pure ()       | otherwise = do         val <- f x         _ <- evaluate (frc val)@@ -1123,7 +1261,32 @@   (void . fin)   (f . unsafePerformIO) -newtype CsvPath = CsvPath { _unCsvPath :: FilePath }+-- | A path to write results in CSV format, populated by @--csv@.+--+-- This is an option of 'csvReporter' and can be set only globally.+-- Modifying it via 'adjustOption' or 'localOption' does not have any effect.+-- One can however pass it to 'tryIngredients' 'benchIngredients'. For example,+-- here is how to set a default CSV location:+--+-- @+-- import Data.Maybe+-- import System.Exit+-- import Test.Tasty.Bench+-- import Test.Tasty.Ingredients+-- import Test.Tasty.Options+-- import Test.Tasty.Runners+--+-- main :: IO ()+-- main = do+--   let benchmarks = bgroup \"All\" ...+--   opts <- parseOptions benchIngredients benchmarks+--   let opts' = changeOption (Just . fromMaybe (CsvPath "foo.csv")) opts+--   case tryIngredients benchIngredients opts' benchmarks of+--     Nothing -> exitFailure+--     Just mb -> mb >>= \b -> if b then exitSuccess else exitFailure+-- @+--+newtype CsvPath = CsvPath FilePath   deriving (Typeable)  instance IsOption (Maybe CsvPath) where@@ -1142,27 +1305,26 @@     let names = testsNames opts tree         namesMap = IM.fromDistinctAscList $ zip [0..] names     pure $ \smap -> do-      case lookupRepeatingElements names of+      case findNonUniqueElement names of         Nothing -> pure ()         Just name -> do -- 'die' is not available before base-4.8           hPutStrLn stderr $ "CSV report cannot proceed, because name '" ++ name ++ "' corresponds to two or more benchmarks. Please disambiguate them."           exitFailure       let augmented = IM.intersectionWith (,) namesMap smap-      hasGCStats <- getRTSStatsEnabled       bracket         (do           h <- openFile path WriteMode           hSetBuffering h LineBuffering           hPutStrLn h $ "Name,Mean (ps),2*Stdev (ps)" ++-            (if hasGCStats then ",Allocated,Copied" else "")+            (if hasGCStats then ",Allocated,Copied,Peak Memory" else "")           pure h         )         hClose         (`csvOutput` augmented)-      pure $ const ((== 0) . statFailures <$> computeStatistics smap)+      pure $ const $ isSuccessful smap -lookupRepeatingElements :: Ord a => [a] -> Maybe a-lookupRepeatingElements = go S.empty+findNonUniqueElement :: Ord a => [a] -> Maybe a+findNonUniqueElement = go S.empty   where     go _ [] = Nothing     go acc (x : xs)@@ -1171,7 +1333,6 @@  csvOutput :: Handle -> IntMap (TestName, TVar Status) -> IO () csvOutput h = traverse_ $ \(name, tv) -> do-  hasGCStats <- getRTSStatsEnabled   let csv = if hasGCStats then csvEstimateWithGC else csvEstimate   r <- atomically $ readTVar tv >>= \s -> case s of Done r -> pure r; _ -> retry   case safeRead (resultDescription r) of@@ -1183,10 +1344,20 @@ encodeCsv :: String -> String encodeCsv xs   | any (`elem` xs) ",\"\n\r"-  = '"' : concatMap (\x -> if x == '"' then "\"\"" else [x]) xs ++ "\""+  = '"' : go xs -- opening quote   | otherwise = xs+  where+    go [] = '"' : [] -- closing quote+    go ('"' : ys) = '"' : '"' : go ys+    go (y : ys) = y : go ys -newtype SvgPath = SvgPath { _unSvgPath :: FilePath }+-- | A path to plot results in SVG format, populated by @--svg@.+--+-- This is an option of 'svgReporter' and can be set only globally.+-- Modifying it via 'adjustOption' or 'localOption' does not have any effect.+-- One can however pass it to 'tryIngredients' 'benchIngredients'.+--+newtype SvgPath = SvgPath FilePath   deriving (Typeable)  instance IsOption (Maybe SvgPath) where@@ -1209,8 +1380,16 @@       svgCollect ref (IM.intersectionWith (,) namesMap smap)       res <- readIORef ref       writeFile path (svgRender (reverse res))-      pure $ const ((== 0) . statFailures <$> computeStatistics smap)+      pure $ const $ isSuccessful smap +isSuccessful :: StatusMap -> IO Bool+isSuccessful = go . IM.elems+  where+    go [] = pure True+    go (tv : tvs) = do+      b <- atomically $ readTVar tv >>= \s -> case s of Done r -> pure (resultSuccessful r); _ -> retry+      if b then go tvs else pure False+ svgCollect :: IORef [(TestName, Estimate)] -> IntMap (TestName, TVar Status) -> IO () svgCollect ref = traverse_ $ \(name, tv) -> do   r <- atomically $ readTVar tv >>= \s -> case s of Done r -> pure r; _ -> retry@@ -1226,12 +1405,12 @@   pairs) ++ footer   where     dropAllPrefix-      | all ("All." `isPrefixOf`) (map fst pairs) = drop 4+      | all (("All." `isPrefixOf`) . fst) pairs = drop 4       | otherwise = id      l = genericLength pairs     findMaxX (Estimate m stdev) = measTime m + 2 * stdev-    xMax = fromIntegral $ maximum $ minBound : map (findMaxX . snd) pairs+    xMax = word64ToDouble $ maximum $ minBound : map (findMaxX . snd) pairs     header = printf "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"%i\" width=\"%f\" font-size=\"%i\" font-family=\"sans-serif\" stroke-width=\"2\">\n<g transform=\"translate(%f 0)\">\n" (svgItemOffset l - 15) svgCanvasWidth svgFontSize svgCanvasMargin     footer = "</g>\n</svg>\n" @@ -1242,7 +1421,7 @@ svgCanvasMargin = 10  svgItemOffset :: Word64 -> Word64-svgItemOffset i = 22 + 55 * fromIntegral i+svgItemOffset i = 22 + 55 * i  svgFontSize :: Word64 svgFontSize = 16@@ -1259,9 +1438,9 @@     x1 = boxWidth - whiskerWidth     x2 = boxWidth + whiskerWidth     deg = (i * 360) `quot` iMax-    glyphWidth = fromIntegral svgFontSize / 2+    glyphWidth = word64ToDouble svgFontSize / 2 -    scale w       = fromIntegral w * (svgCanvasWidth - 2 * svgCanvasMargin) / xMax+    scale w       = word64ToDouble w * (svgCanvasWidth - 2 * svgCanvasMargin) / xMax     boxWidth      = scale (measTime m)     whiskerWidth  = scale (2 * stdev)     boxHeight     = 22@@ -1286,24 +1465,30 @@     longText = printf longTextTemplate       deg       y (encodeSvg name)-      y boxWidth (showPicos (measTime m))+      y boxWidth (showPicos4 (measTime m))     longTextTemplate       =  "<g fill=\"hsl(%i, 100%%, 40%%)\">\n"       ++ "<text y=\"%i\">%s</text>\n"       ++ "<text y=\"%i\" x=\"%f\" text-anchor=\"end\">%s</text>\n"       ++ "</g>\n" -    shortTextContent  = encodeSvg name ++ " " ++ showPicos (measTime m)+    shortTextContent  = encodeSvg name ++ " " ++ showPicos4 (measTime m)     shortText         = printf shortTextTemplate deg y shortTextContent     shortTextTemplate = "<text fill=\"hsl(%i, 100%%, 40%%)\" y=\"%i\">%s</text>\n"  encodeSvg :: String -> String-encodeSvg = concatMap $ \x -> case x of-  '<' -> "&lt;"-  '&' -> "&amp;"-  _   -> [x]+encodeSvg [] = []+encodeSvg ('<' : xs) = '&' : 'l' : 't' : ';' : encodeSvg xs+encodeSvg ('&' : xs) = '&' : 'a' : 'm' : 'p' : ';' : encodeSvg xs+encodeSvg (x : xs) = x : encodeSvg xs -newtype BaselinePath = BaselinePath { _unBaselinePath :: FilePath }+-- | A path to read baseline results in CSV format, populated by @--baseline@.+--+-- This is an option of 'csvReporter' and can be set only globally.+-- Modifying it via 'adjustOption' or 'localOption' does not have any effect.+-- One can however pass it to 'tryIngredients' 'benchIngredients'.+--+newtype BaselinePath = BaselinePath FilePath   deriving (Typeable)  instance IsOption (Maybe BaselinePath) where@@ -1324,8 +1509,7 @@ consoleBenchReporter = modifyConsoleReporter [Option (Proxy :: Proxy (Maybe BaselinePath))] $ \opts -> do   baseline <- case lookupOption opts of     Nothing -> pure S.empty-    Just (BaselinePath path) -> S.fromList . lines <$> (readFile path >>= evaluate . force)-  hasGCStats <- getRTSStatsEnabled+    Just (BaselinePath path) -> S.fromList . joinQuotedFields . lines <$> (readFile path >>= evaluate . force)   let pretty = if hasGCStats then prettyEstimateWithGC else prettyEstimate   pure $ \name depR r -> case safeRead (resultDescription r) of     Nothing  -> r@@ -1335,14 +1519,25 @@       where         slowDown = compareVsBaseline baseline name est         isAcceptable -- ifSlow/ifFast may be infinite, so we cannot 'truncate'-          =  fromIntegral slowDown <=  100 * ifSlow-          && fromIntegral slowDown >= -100 * ifFast+          =  int64ToDouble slowDown <=  100 * ifSlow+          && int64ToDouble slowDown >= -100 * ifFast         bcomp = case depR >>= safeRead . resultDescription of           Nothing -> ""           Just (Response depEst _ _) -> printf ", %.2fx" (estTime est / estTime depEst) +-- | A well-formed CSV entry contains an even number of quotes: 0, 2 or more.+joinQuotedFields :: [String] -> [String]+joinQuotedFields [] = []+joinQuotedFields (x : xs)+  | areQuotesBalanced x = x : joinQuotedFields xs+  | otherwise = case span areQuotesBalanced xs of+    (_, [])      -> [] -- malformed CSV+    (ys, z : zs) -> unlines (x : ys ++ [z]) : joinQuotedFields zs+  where+    areQuotesBalanced = even . length . filter (== '"')+ estTime :: Estimate -> Double-estTime = fromIntegral . measTime . estMean+estTime = word64ToDouble . measTime . estMean  -- | Return slow down in percents. compareVsBaseline :: S.Set String -> TestName -> Estimate -> Int64@@ -1350,11 +1545,10 @@   Nothing -> 0   Just (oldTime, oldDoubleSigma)     -- time and oldTime must be signed integers to use 'abs'-    | abs (time - oldTime) < max (2 * fromIntegral stdev) oldDoubleSigma -> 0+    | abs (time - oldTime) < max (2 * word64ToInt64 stdev) oldDoubleSigma -> 0     | otherwise -> 100 * (time - oldTime) `quot` oldTime   where-    time :: Int64-    time = fromIntegral $ measTime m+    time = word64ToInt64 $ measTime m      mOld :: Maybe (Int64, Int64)     mOld = do@@ -1394,7 +1588,9 @@                    . IM.intersectionWith (\(a, b) c -> (a, b, c)) namesAndDeps   in (modifySMap >=>) <$> cb opts tree   where-    TestReporter desc cb = consoleTestReporter+    (desc, cb) = case consoleTestReporter of+      TestReporter d c -> (d, c)+      _ -> error "modifyConsoleReporter: consoleTestReporter must be TestReporter"      isSingle [a] = Just a     isSingle _ = Nothing@@ -1402,15 +1598,27 @@ testNameSeqs :: OptionSet -> TestTree -> [Seq TestName] testNameSeqs = foldTestTree trivialFold   { foldSingle = const $ const . (:[]) . Seq.singleton+#if MIN_VERSION_tasty(1,4,0)   , foldGroup  = const $ map . (<|)+#else+  , foldGroup  = map . (<|)+#endif   }  testNamesAndDeps :: IntMap (Seq TestName) -> OptionSet -> TestTree -> [(TestName, [IM.Key])] testNamesAndDeps im = foldTestTree trivialFold   { foldSingle = const $ const . (: []) . (, [])+#if MIN_VERSION_tasty(1,4,0)   , foldGroup  = const $ map . first . (++) . (++ ".")   , foldAfter  = const foldDeps+#else+  , foldGroup  = map . first . (++) . (++ ".")+#if MIN_VERSION_tasty(1,2,0)+  , foldAfter  = foldDeps+#endif+#endif   }+#if MIN_VERSION_tasty(1,2,0)   where     foldDeps AllSucceed (And (StringLit "tasty-bench") p) =       map $ second $ (++) $ findMatchingKeys im p@@ -1421,6 +1629,7 @@   foldr (\(k, v) -> if withFields v pat == Right True then (k :) else id) [] $ IM.assocs im   where     pat = eval pattern >>= asB+#endif  postprocessResult     :: (TestName -> Maybe Result -> Result -> Result)@@ -1456,3 +1665,17 @@       adNauseam = doUpdate >>= (`unless` adNauseam)   _ <- forkIO adNauseam   pure $ fmap (\(_, _, _, a) -> a) paired++word64ToDouble :: Word64 -> Double+word64ToDouble = fromIntegral++int64ToDouble :: Int64 -> Double+int64ToDouble = fromIntegral++word64ToInt64 :: Word64 -> Int64+word64ToInt64 = fromIntegral++#if !MIN_VERSION_base(4,10,0) && MIN_VERSION_base(4,6,0)+int64ToWord64 :: Int64 -> Word64+int64ToWord64 = fromIntegral+#endif

@@ -1,3 +1,12 @@+# 0.3++* Report mean time with 3 significant digits.+* Report peak memory usage, when run with `+RTS -T`.+* Run benchmarks only once, if `RelStDev` is infinite.+* Make `Benchmarkable` constructor public.+* Expose `measureCpuTime` helper to run benchmarks manually.+* Expose `CsvPath`, `BaselinePath`, `SvgPath`.+ # 0.2.5  * Fix comparison against baseline.

@@ -1,10 +1,11 @@ name:          tasty-bench-version:       0.2.5+version:       0.3 cabal-version: 1.18 build-type:    Simple license:       MIT license-file:  LICENSE copyright:     2021 Andrew Lelechenko+author:        Andrew Lelechenko maintainer:    Andrew Lelechenko <andrew.lelechenko@gmail.com> homepage:      https://github.com/Bodigrim/tasty-bench bug-reports:   https://github.com/Bodigrim/tasty-bench/issues@@ -36,12 +37,14 @@   ghc-options:      -O2 -Wall -fno-warn-unused-imports   if impl(ghc < 7.10)     ghc-options:    -fcontext-stack=30+  if impl(ghc >= 8.0)+    ghc-options:    -Wcompat -Widentities    build-depends:     base >= 4.3 && < 5,     containers >= 0.4,     deepseq >= 1.1,-    tasty >= 1.2.3+    tasty >= 0.11.3   if impl(ghc < 7.8)     build-depends:       tagged >= 0.2