diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# tasty-bench [![Hackage](http://img.shields.io/hackage/v/tasty-bench.svg)](https://hackage.haskell.org/package/tasty-bench) [![Stackage Nightly](http://stackage.org/package/tasty-bench/badge/nightly)](http://stackage.org/nightly/package/tasty-bench) +# tasty-bench [![Hackage](http://img.shields.io/hackage/v/tasty-bench.svg)](https://hackage.haskell.org/package/tasty-bench) [![Stackage LTS](http://stackage.org/package/tasty-bench/badge/lts)](http://stackage.org/lts/package/tasty-bench) [![Stackage Nightly](http://stackage.org/package/tasty-bench/badge/nightly)](http://stackage.org/nightly/package/tasty-bench) Featherlight benchmark framework (only one file!) for performance measurement with API mimicking [`criterion`](http://hackage.haskell.org/package/criterion) @@ -13,7 +13,9 @@ So if you already depend on `tasty` for a test suite, there is nothing else to install. -Compare this to `criterion` (10+ modules, 50+ dependencies) and `gauge` (40+ modules, depends on `basement` and `vector`). +Compare this to `criterion` (10+ modules, 50+ dependencies) and `gauge` (40+ modules, depends on `basement` and `vector`). A build on a clean machine is up to 16x +faster than `criterion` and up to 4x faster than `gauge`. A build without dependencies +is up to 6x faster than `criterion` and up to 8x faster than `gauge`. ## How is it possible? @@ -151,10 +153,27 @@ for [68%](https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule) of samples only, double it to estimate the behavior in 95% of cases. -When benchmarking multithreaded algorithms, note -that `tasty-bench` reports total elapsed CPU time across all cores, while -`criterion` and `gauge` print wall-clock time. +## Wall-clock time vs. CPU time +What time are we talking about? +Both `criterion` and `gauge` by default report wall-clock time, which is +affected by any other application which runs concurrently. +While ideally benchmarks are executed on a dedicated server without any other load, +but — let's face the truth — most of developers run benchmarks +on a laptop with a hundred other services and a window manager, and +watch videos while waiting for benchmarks to finish. That's the cause +of a notorious "variance introduced by outliers: 88% (severely inflated)" warning. + +To alleviate this issue `tasty-bench` measures CPU time by `getCPUTime` +instead of wall-clock time. +It does not provide a perfect isolation from other processes (e. g., +if CPU cache is spoiled by others, populating data back from RAM +is your burden), but is significantly more stable. + +Caveat: this means that for multithreaded algorithms +`tasty-bench` reports total elapsed CPU time across all cores, while +`criterion` and `gauge` print maximum of core's wall-clock time. + ## Statistical model Here is a procedure used by `tasty-bench` to measure execution time: @@ -162,8 +181,10 @@ 1. Set _n_ ← 1. 2. Measure execution time _tₙ_ of _n_ iterations and execution time _t₂ₙ_ of _2n_ iterations. -3. Find _t_ which minimizes deviation of (_nt_, _2nt_) from (_tₙ_, _t₂ₙ_). -4. If deviation is small enough (see `--stdev` below), +3. Find _t_ which minimizes deviation of (_nt_, _2nt_) from (_tₙ_, _t₂ₙ_), + namely _t_ ← (_tₙ_ + _2t₂ₙ_) / _5n_. +4. If deviation is small enough (see `--stdev` below) + or time is running out soon (see `--timeout` below), return _t_ as a mean execution time. 5. Otherwise set _n_ ← _2n_ and jump back to Step 2. @@ -198,21 +219,24 @@ Configuring RTS to collect GC statistics (e. g., via `cabal bench --benchmark-options '+RTS -T'` or `stack bench --ba '+RTS -T'`) enables `tasty-bench` to estimate and report -memory usage such as allocated and copied bytes: +memory usage: ``` All fibonacci numbers fifth: OK (2.13s) - 63 ns ± 3.4 ns, 223 B allocated, 0 B copied + 63 ns ± 3.4 ns, 223 B allocated, 0 B copied, 2.0 MB peak memory tenth: OK (1.71s) - 809 ns ± 73 ns, 2.3 KB allocated, 0 B copied + 809 ns ± 73 ns, 2.3 KB allocated, 0 B copied, 4.0 MB peak memory twentieth: OK (3.39s) - 104 μs ± 4.9 μs, 277 KB allocated, 59 B copied + 104 μs ± 4.9 μs, 277 KB allocated, 59 B copied, 5.0 MB peak memory All 3 tests passed (7.25s) ``` +This data is reported as per `RTSStats` fields: `allocated_bytes`, `copied_bytes` +and `max_mem_in_use_bytes`. + ## Combining tests and benchmarks When optimizing an existing function, it is important to check that its @@ -282,7 +306,7 @@ Alternatively bake it into `cabal` file as `ghc-options: "-with-rtsopts=-A32m"`. - For GHC >= 8.10 consider switching benchmarks to a non-moving garbage collector, + For GHC ≥ 8.10 consider switching benchmarks to a non-moving garbage collector, because it decreases GC pauses and corresponding noise: `+RTS --nonmoving-gc`. * If benchmark results look malformed like below, make sure that you are @@ -301,7 +325,13 @@ Unhandled resource. Probably a bug in the runner you're using. ``` - this is probably caused by `env` or `envWithCleanup` affecting benchmarks structure. +or + + ``` + Unexpected state of the resource (NotCreated) in getResource. Report as a tasty bug. + ``` + + this is likely caused by `env` or `envWithCleanup` affecting benchmarks structure. You can use `env` to read test data from `IO`, but not to read benchmark names or affect their hierarchy in other way. This is a fundamental restriction of `tasty` to list and filter benchmarks without launching missiles. @@ -362,14 +392,14 @@ ```bash cabal build --enable-benchmarks my-bench -MYBENCH=`cabal list-bin my-bench` +MYBENCH=$(cabal list-bin my-bench) ``` Now list all benchmark names (hopefully, they do not contain newlines), escape quotes and slashes, and run each of them separately: ```bash -$MYBENCH -l | sed -e 's/[\"]/\\\\\\&/g' | while read name; do $MYBENCH -p '$0 == "'$name'"'; done +$MYBENCH -l | sed -e 's/[\"]/\\\\\\&/g' | while read -r name; do $MYBENCH -p '$0 == "'"$name"'"'; done ``` ## Comparison against baseline @@ -386,10 +416,6 @@ All.fibonacci numbers.twentieth,81369531,3342646 ``` -Note that columns do not match CSV reports of `criterion` and `gauge`. -If desired, missing columns can be faked with -`awk 'BEGIN {FS=",";OFS=","}; {print $1,$2,$2,$2,$3/2,$3/2,$3/2}'` or similar. - Now modify implementation and rerun benchmarks with `--baseline FILE` key. This produces a report as follows: @@ -415,6 +441,28 @@ [`tasty-rerun`](http://hackage.haskell.org/package/tasty-rerun) package to focus on rerunning failing items only. +Note that columns in CSV report are different from what `criterion` or `gauge` +would produce. If names do not contain commas, missing columns can be faked this way: + +```sh +cat tasty-bench.csv \ +| awk 'BEGIN {FS=",";OFS=","}; {print $1,$2/1e12,$2/1e12,$2/1e12,$3/2e12,$3/2e12,$3/2e12}' \ +| sed '1s/.*/Name,Mean,MeanLB,MeanUB,Stddev,StddevLB,StddevUB/' +``` + +To fake `gauge` in `--csvraw` mode use + +```sh +cat tasty-bench.csv \ +| awk 'BEGIN {FS=",";OFS=","}; {print $1,1,$2/1e12,0,$2/1e12,$2/1e12,0,$6+0,0,0,0,0,$4+0,0,$5+0,0,0,0,0}' \ +| sed '1s/.*/name,iters,time,cycles,cpuTime,utime,stime,maxrss,minflt,majflt,nvcsw,nivcsw,allocated,numGcs,bytesCopied,mutatorWallSeconds,mutatorCpuSeconds,gcWallSeconds,gcCpuSeconds/' +``` + +Please refer to `gawk` manual, if you wish to process names with +[commas](https://www.gnu.org/software/gawk/manual/gawk.html#Splitting-By-Content) +or +[quotes](https://www.gnu.org/software/gawk/manual/gawk.html#More-CSV). + ## Comparison between benchmarks You can also compare benchmarks to each other without reaching to external tools, @@ -463,8 +511,10 @@ Users can dump results into CSV with `--csv FILE` and plot them using `gnuplot` or other software. But for convenience there is also a built-in quick-and-dirty SVG plotting feature, -which can be invoked by passing `--svg FILE`. +which can be invoked by passing `--svg FILE`. Here is a sample of its output: +![Plotting](./example.svg) + ## Command-line options Use `--help` to list command-line options. @@ -482,13 +532,16 @@ in seconds. Use it when benchmarks tend to take too long: `tasty-bench` will make an effort to report results (even if of subpar quality) before timeout. Setting timeout too tight (insufficient for at least three iterations) - will result in a benchmark failure. + will result in a benchmark failure. One can adjust it locally for a group + of benchmarks, e. g., `localOption (mkTimeout 100000000)` for 100 seconds. * `--stdev` Target relative standard deviation of measurements in percents (5% by default). Large values correspond to fast and loose benchmarks, and small ones to long and precise. - If it takes far too long, consider setting `--timeout`, + It can also be adjusted locally for a group of benchmarks, + e. g., `localOption (RelStDev 0.02)`. + If benchmarking takes far too long, consider setting `--timeout`, which will interrupt benchmarks, potentially before reaching the target deviation. * `--csv` @@ -504,7 +557,45 @@ Upper bounds of acceptable slow down / speed up in percents. If a benchmark is unacceptably slower / faster than baseline (see `--baseline`), it will be reported as failed. Can be used in conjunction with a standard `tasty` option `--hide-successes` to show only problematic benchmarks. + Both options can be adjusted locally for a group of benchmarks, + e. g., `localOption (FailIfSlower 0.10)`. * `--svg` File to plot results in SVG format. + +* `+RTS -T` + + Estimate and report memory usage. + +## Custom command-line options + +As usual with `tasty`, it is easy to extend benchmarks with custom command-line options. +Here is an example: + +```haskell +import Data.Proxy +import Test.Tasty.Bench +import Test.Tasty.Ingredients.Basic +import Test.Tasty.Options +import Test.Tasty.Runners + +newtype RandomSeed = RandomSeed Int + +instance IsOption RandomSeed where + defaultValue = RandomSeed 42 + parseValue = fmap RandomSeed . safeRead + optionName = pure "seed" + optionHelp = pure "Random seed used in benchmarks" + +main :: IO () +main = do + let customOpts = [Option (Proxy :: Proxy RandomSeed)] + ingredients = includingOptions customOpts : benchIngredients + opts <- parseOptions ingredients benchmarks + let RandomSeed seed = lookupOption opts + defaultMainWithIngredients ingredients benchmarks + +benchmarks :: Benchmark +benchmarks = bgroup "All" [] +``` diff --git a/Test/Tasty/Bench.hs b/Test/Tasty/Bench.hs --- a/Test/Tasty/Bench.hs +++ b/Test/Tasty/Bench.hs @@ -18,7 +18,9 @@ to install. Compare this to @criterion@ (10+ modules, 50+ dependencies) and @gauge@ -(40+ modules, depends on @basement@ and @vector@). +(40+ modules, depends on @basement@ and @vector@). A build on a clean machine is up to 16x +faster than @criterion@ and up to 4x faster than @gauge@. A build without dependencies +is up to 6x faster than @criterion@ and up to 8x faster than @gauge@. === How is it possible? @@ -146,10 +148,27 @@ of samples only, double it to estimate the behavior in 95% of cases. -When benchmarking multithreaded algorithms, note -that @tasty-bench@ reports total elapsed CPU time across all cores, while -@criterion@ and @gauge@ print wall-clock time. +=== Wall-clock time vs. CPU time +What time are we talking about? +Both @criterion@ and @gauge@ by default report wall-clock time, which is +affected by any other application which runs concurrently. +While ideally benchmarks are executed on a dedicated server without any other load, +but — let's face the truth — most of developers run benchmarks +on a laptop with a hundred other services and a window manager, and +watch videos while waiting for benchmarks to finish. That's the cause +of a notorious "variance introduced by outliers: 88% (severely inflated)" warning. + +To alleviate this issue @tasty-bench@ measures CPU time by 'getCPUTime' +instead of wall-clock time. +It does not provide a perfect isolation from other processes (e. g., +if CPU cache is spoiled by others, populating data back from RAM +is your burden), but is significantly more stable. + +Caveat: this means that for multithreaded algorithms +@tasty-bench@ reports total elapsed CPU time across all cores, while +@criterion@ and @gauge@ print maximum of core's wall-clock time. + === Statistical model Here is a procedure used by @tasty-bench@ to measure execution time: @@ -158,9 +177,10 @@ 2. Measure execution time \( t_n \) of \( n \) iterations and execution time \( t_{2n} \) of \( 2n \) iterations. 3. Find \( t \) which minimizes deviation of \( (nt, 2nt) \) from - \( (t_n, t_{2n}) \). -4. If deviation is small enough (see @--stdev@ below), return \( t \) as a - mean execution time. + \( (t_n, t_{2n}) \), namely \( t \leftarrow (t_n + 2t_{2n}) / 5n \). +4. If deviation is small enough (see @--stdev@ below) + or time is running out soon (see @--timeout@ below), + return \( t \) as a mean execution time. 5. Otherwise set \( n \leftarrow 2n \) and jump back to Step 2. This is roughly similar to the linear regression approach which @@ -191,19 +211,22 @@ Configuring RTS to collect GC statistics (e. g., via @cabal@ @bench@ @--benchmark-options@ @\'+RTS@ @-T\'@ or @stack@ @bench@ @--ba@ @\'+RTS@ @-T\'@) enables @tasty-bench@ to estimate and -report memory usage such as allocated and copied bytes: +report memory usage: > All > fibonacci numbers > fifth: OK (2.13s) -> 63 ns ± 3.4 ns, 223 B allocated, 0 B copied +> 63 ns ± 3.4 ns, 223 B allocated, 0 B copied, 2.0 MB peak memory > tenth: OK (1.71s) -> 809 ns ± 73 ns, 2.3 KB allocated, 0 B copied +> 809 ns ± 73 ns, 2.3 KB allocated, 0 B copied, 4.0 MB peak memory > twentieth: OK (3.39s) -> 104 μs ± 4.9 μs, 277 KB allocated, 59 B copied +> 104 μs ± 4.9 μs, 277 KB allocated, 59 B copied, 5.0 MB peak memory > > All 3 tests passed (7.25s) +This data is reported as per 'RTSStats' fields: 'allocated_bytes', 'copied_bytes' +and 'max_mem_in_use_bytes'. + === Combining tests and benchmarks When optimizing an existing function, it is important to check that its @@ -270,7 +293,7 @@ @stack@ @bench@ @--ba@ @\'+RTS@ @-A32m\'@. Alternatively bake it into @cabal@ file as @ghc-options:@ @\"-with-rtsopts=-A32m\"@. - For GHC >= 8.10 consider switching benchmarks to a non-moving garbage collector, + For GHC ≥ 8.10 consider switching benchmarks to a non-moving garbage collector, because it decreases GC pauses and corresponding noise: @+RTS@ @--nonmoving-gc@. - If benchmark results look malformed like below, make sure that you @@ -286,7 +309,11 @@ > Unhandled resource. Probably a bug in the runner you're using. - this is probably caused by 'env' or 'envWithCleanup' affecting + or + + > Unexpected state of the resource (NotCreated) in getResource. Report as a tasty bug. + + this is likely caused by 'env' or 'envWithCleanup' affecting benchmarks structure. You can use 'env' to read test data from 'IO', but not to read benchmark names or affect their hierarchy in other way. This is a fundamental restriction of @tasty@ to list and filter @@ -343,12 +370,12 @@ @benchmark@ @my-bench@ component, let’s first find its executable: > cabal build --enable-benchmarks my-bench -> MYBENCH=`cabal list-bin my-bench` +> MYBENCH=$(cabal list-bin my-bench) Now list all benchmark names (hopefully, they do not contain newlines), escape quotes and slashes, and run each of them separately: -> $MYBENCH -l | sed -e 's/[\"]/\\\\\\&/g' | while read name; do $MYBENCH -p '$0 == "'$name'"'; done +> $MYBENCH -l | sed -e 's/[\"]/\\\\\\&/g' | while read -r name; do $MYBENCH -p '$0 == "'"$name"'"'; done === Comparison against baseline @@ -362,11 +389,6 @@ > All.fibonacci numbers.tenth,637152,46744 > All.fibonacci numbers.twentieth,81369531,3342646 -Note that columns do not match CSV reports of @criterion@ and @gauge@. -If desired, missing columns can be faked with -@awk@ @\'BEGIN@ @{FS=\",\";OFS=\",\"};@ @{print@ @$1,$2,$2,$2,$3\/2,$3\/2,$3\/2}\'@ -or similar. - Now modify implementation and rerun benchmarks with @--baseline@ @FILE@ key. This produces a report as follows: @@ -390,6 +412,19 @@ even [@tasty-rerun@](http://hackage.haskell.org/package/tasty-rerun) package to focus on rerunning failing items only. +Note that columns in CSV report are different from what @criterion@ or @gauge@ +would produce. If names do not contain commas, missing columns can be faked this way: + +> cat tasty-bench.csv | awk 'BEGIN {FS=",";OFS=","}; {print $1,$2/1e12,$2/1e12,$2/1e12,$3/2e12,$3/2e12,$3/2e12}' | sed '1s/.*/Name,Mean,MeanLB,MeanUB,Stddev,StddevLB,StddevUB/' + +To fake @gauge@ in @--csvraw@ mode use + +> cat tasty-bench.csv | awk 'BEGIN {FS=",";OFS=","}; {print $1,1,$2/1e12,0,$2/1e12,$2/1e12,0,$6+0,0,0,0,0,$4+0,0,$5+0,0,0,0,0}' | sed '1s/.*/name,iters,time,cycles,cpuTime,utime,stime,maxrss,minflt,majflt,nvcsw,nivcsw,allocated,numGcs,bytesCopied,mutatorWallSeconds,mutatorCpuSeconds,gcWallSeconds,gcCpuSeconds/' + +Please refer to @gawk@ manual, if you wish to process names +with [commas](https://www.gnu.org/software/gawk/manual/gawk.html#Splitting-By-Content) +or [quotes](https://www.gnu.org/software/gawk/manual/gawk.html#More-CSV). + === Comparison between benchmarks You can also compare benchmarks to each other without reaching to @@ -455,14 +490,18 @@ @tasty-bench@ will make an effort to report results (even if of subpar quality) before timeout. Setting timeout too tight (insufficient for at least three iterations) will result in a - benchmark failure. + benchmark failure. One can adjust it locally for a group + of benchmarks, e. g., 'localOption' ('mkTimeout' 100000000) for 100 seconds. [@--stdev@]: Target relative standard deviation of measurements in percents (5% by default). Large values correspond to fast and loose benchmarks, - and small ones to long and precise. If it takes far too long, - consider setting @--timeout@, which will interrupt benchmarks, + and small ones to long and precise. + It can also be adjusted locally for a group of benchmarks, + e. g., 'localOption' ('RelStDev' 0.02). + If benchmarking takes far too long, consider setting @--timeout@, + which will interrupt benchmarks, potentially before reaching the target deviation. [@--csv@]: @@ -481,11 +520,47 @@ @--baseline@), it will be reported as failed. Can be used in conjunction with a standard @tasty@ option @--hide-successes@ to show only problematic benchmarks. + Both options can be adjusted locally for a group of benchmarks, + e. g., 'localOption' ('FailIfSlower' 0.10). [@--svg@]: File to plot results in SVG format. +[@+RTS@ @-T@]: + + Estimate and report memory usage. + +=== Custom command-line options + +As usual with @tasty@, it is easy to extend benchmarks with custom command-line options. +Here is an example: + +> import Data.Proxy +> import Test.Tasty.Bench +> import Test.Tasty.Ingredients.Basic +> import Test.Tasty.Options +> import Test.Tasty.Runners +> +> newtype RandomSeed = RandomSeed Int +> +> instance IsOption RandomSeed where +> defaultValue = RandomSeed 42 +> parseValue = fmap RandomSeed . safeRead +> optionName = pure "seed" +> optionHelp = pure "Random seed used in benchmarks" +> +> main :: IO () +> main = do +> let customOpts = [Option (Proxy :: Proxy RandomSeed)] +> ingredients = includingOptions customOpts : benchIngredients +> opts <- parseOptions ingredients benchmarks +> let RandomSeed seed = lookupOption opts +> defaultMainWithIngredients ingredients benchmarks +> +> benchmarks :: Benchmark +> benchmarks = bgroup "All" [] + -} {-# LANGUAGE CPP #-} @@ -502,17 +577,20 @@ , Benchmark , bench , bgroup +#if MIN_VERSION_tasty(1,2,0) , bcompare +#endif , env , envWithCleanup -- * Creating 'Benchmarkable' - , Benchmarkable + , Benchmarkable(..) , nf , whnf , nfIO , whnfIO , nfAppIO , whnfAppIO + , measureCpuTime -- * Ingredients , benchIngredients , consoleBenchReporter @@ -521,6 +599,9 @@ , RelStDev(..) , FailIfSlower(..) , FailIfFaster(..) + , CsvPath(..) + , BaselinePath(..) + , SvgPath(..) ) where import Prelude hiding (Int, Integer) @@ -533,16 +614,17 @@ import Data.Foldable (foldMap, traverse_) import Data.Int (Int64) import Data.IntMap (IntMap) +#if MIN_VERSION_containers(0,5,0) +import qualified Data.IntMap.Strict as IM +#else import qualified Data.IntMap as IM +#endif import Data.IORef -import Data.List (intercalate, stripPrefix, isPrefixOf, foldl', genericLength, genericDrop) +import Data.List (intercalate, stripPrefix, isPrefixOf, genericLength, genericDrop) import Data.Monoid (All(..), Any(..)) import Data.Proxy import Data.Sequence (Seq, (<|)) import qualified Data.Sequence as Seq -#if MIN_VERSION_containers(0,5,0) -import Data.Set (lookupGE) -#endif import qualified Data.Set as S import Data.Traversable (forM) import Data.Word (Word64) @@ -557,8 +639,10 @@ import Test.Tasty.Ingredients import Test.Tasty.Ingredients.ConsoleReporter import Test.Tasty.Options +#if MIN_VERSION_tasty(1,2,0) import Test.Tasty.Patterns.Eval (eval, asB, withFields) import Test.Tasty.Patterns.Types (Expr (And, StringLit)) +#endif import Test.Tasty.Providers import Test.Tasty.Runners import Text.Printf @@ -573,8 +657,16 @@ -- -- E. g., set target relative standard deviation to 2% as follows: -- +-- > import Test.Tasty (localOption) -- > localOption (RelStDev 0.02) (bgroup [...]) -- +-- If you set 'RelStDev' to infinity, +-- a benchmark will be executed +-- only once and its standard deviation will be recorded as zero. +-- This is rather a blunt approach, but it might be a necessary evil +-- for extremely long benchmarks. If you wish to run all benchmarks +-- only once, use command-line option @--stdev@ @Infinity@. +-- newtype RelStDev = RelStDev Double deriving (Show, Read, Typeable) @@ -592,6 +684,7 @@ -- -- E. g., set upper bound of acceptable slow down to 10% as follows: -- +-- > import Test.Tasty (localOption) -- > localOption (FailIfSlower 0.10) (bgroup [...]) -- newtype FailIfSlower = FailIfSlower Double @@ -611,6 +704,7 @@ -- -- E. g., set upper bound of acceptable speed up to 10% as follows: -- +-- > import Test.Tasty (localOption) -- > localOption (FailIfFaster 0.10) (bgroup [...]) -- newtype FailIfFaster = FailIfFaster Double @@ -633,11 +727,13 @@ -- -- Drop-in replacement for 'Criterion.Benchmarkable' and 'Gauge.Benchmarkable'. -- -newtype Benchmarkable = Benchmarkable { _unBenchmarkable :: Int64 -> IO () } - deriving (Typeable) +newtype Benchmarkable = Benchmarkable + { unBenchmarkable :: Word64 -> IO () -- ^ Run benchmark given number of times. + } deriving (Typeable) -showPicos :: Word64 -> String -showPicos i +-- | Show picoseconds, fitting number in 3 characters. +showPicos3 :: Word64 -> String +showPicos3 i | t < 995 = printf "%3.0f ps" t | t < 995e1 = printf "%3.1f ns" (t / 1e3) | t < 995e3 = printf "%3.0f ns" (t / 1e3) @@ -647,9 +743,25 @@ | t < 995e9 = printf "%3.0f ms" (t / 1e9) | otherwise = printf "%4.2f s" (t / 1e12) where - t :: Double - t = fromIntegral i + t = word64ToDouble i +-- | Show picoseconds, fitting number in 4 characters. +showPicos4 :: Word64 -> String +showPicos4 i + | t < 995 = printf "%3.0f ps" t + | t < 995e1 = printf "%4.2f ns" (t / 1e3) + | t < 995e2 = printf "%4.1f ns" (t / 1e3) + | t < 995e3 = printf "%3.0f ns" (t / 1e3) + | t < 995e4 = printf "%4.2f μs" (t / 1e6) + | t < 995e5 = printf "%4.1f μs" (t / 1e6) + | t < 995e6 = printf "%3.0f μs" (t / 1e6) + | t < 995e7 = printf "%4.2f ms" (t / 1e9) + | t < 995e8 = printf "%4.1f ms" (t / 1e9) + | t < 995e9 = printf "%3.0f ms" (t / 1e9) + | otherwise = printf "%4.3f s" (t / 1e12) + where + t = word64ToDouble i + showBytes :: Word64 -> String showBytes i | t < 1000 = printf "%3.0f B " t @@ -666,13 +778,13 @@ | t < 11471568970838126592 = printf "%3.1f EB" (t / 1152921504606846976) | otherwise = printf "%3.0f EB" (t / 1152921504606846976) where - t :: Double - t = fromIntegral i + t = word64ToDouble i data Measurement = Measurement { measTime :: !Word64 -- ^ time in picoseconds , measAllocs :: !Word64 -- ^ allocations in bytes , measCopied :: !Word64 -- ^ copied bytes + , measMaxMem :: !Word64 -- ^ max memory in use } deriving (Show, Read) data Estimate = Estimate @@ -688,35 +800,38 @@ prettyEstimate :: Estimate -> String prettyEstimate (Estimate m stdev) = - showPicos (measTime m) ++ " ± " ++ showPicos (2 * stdev) + showPicos4 (measTime m) + ++ (if stdev == 0 then " " else " ± " ++ showPicos3 (2 * stdev)) prettyEstimateWithGC :: Estimate -> String prettyEstimateWithGC (Estimate m stdev) = - showPicos (measTime m) ++ " ± " ++ showPicos (2 * stdev) - ++ ", " ++ showBytes (measAllocs m) ++ " allocated, " - ++ showBytes (measCopied m) ++ " copied" + showPicos4 (measTime m) + ++ (if stdev == 0 then ", " else " ± " ++ showPicos3 (2 * stdev) ++ ", ") + ++ showBytes (measAllocs m) ++ " allocated, " + ++ showBytes (measCopied m) ++ " copied, " + ++ showBytes (measMaxMem m) ++ " peak memory" csvEstimate :: Estimate -> String csvEstimate (Estimate m stdev) = show (measTime m) ++ "," ++ show (2 * stdev) csvEstimateWithGC :: Estimate -> String csvEstimateWithGC (Estimate m stdev) = show (measTime m) ++ "," ++ show (2 * stdev) - ++ "," ++ show (measAllocs m) ++ "," ++ show (measCopied m) + ++ "," ++ show (measAllocs m) ++ "," ++ show (measCopied m) ++ "," ++ show (measMaxMem m) predict :: Measurement -- ^ time for one run -> Measurement -- ^ time for two runs -> Estimate -predict (Measurement t1 a1 c1) (Measurement t2 a2 c2) = Estimate - { estMean = Measurement t (fit a1 a2) (fit c1 c2) +predict (Measurement t1 a1 c1 m1) (Measurement t2 a2 c2 m2) = Estimate + { estMean = Measurement t (fit a1 a2) (fit c1 c2) (max m1 m2) , estStdev = truncate (sqrt d :: Double) } where fit x1 x2 = x1 `quot` 5 + 2 * (x2 `quot` 5) t = fit t1 t2 sqr x = x * x - d = sqr (fromIntegral t1 - fromIntegral t) - + sqr (fromIntegral t2 - 2 * fromIntegral t) + d = sqr (word64ToDouble t1 - word64ToDouble t) + + sqr (word64ToDouble t2 - 2 * word64ToDouble t) predictPerturbed :: Measurement -> Measurement -> Estimate predictPerturbed t1 t2 = Estimate @@ -730,69 +845,88 @@ hi meas = meas { measTime = measTime meas + prec } lo meas = meas { measTime = measTime meas - prec } -#if !MIN_VERSION_base(4,10,0) -getRTSStatsEnabled :: IO Bool -#if MIN_VERSION_base(4,6,0) -getRTSStatsEnabled = getGCStatsEnabled +hasGCStats :: Bool +#if MIN_VERSION_base(4,10,0) +hasGCStats = unsafePerformIO getRTSStatsEnabled +#elif MIN_VERSION_base(4,6,0) +hasGCStats = unsafePerformIO getGCStatsEnabled #else -getRTSStatsEnabled = pure False -#endif +hasGCStats = False #endif -getAllocsAndCopied :: IO (Word64, Word64) +getAllocsAndCopied :: IO (Word64, Word64, Word64) getAllocsAndCopied = do - enabled <- getRTSStatsEnabled - if not enabled then pure (0, 0) else + if not hasGCStats then pure (0, 0, 0) else #if MIN_VERSION_base(4,10,0) - (\s -> (allocated_bytes s, copied_bytes s)) <$> getRTSStats + (\s -> (allocated_bytes s, copied_bytes s, max_mem_in_use_bytes s)) <$> getRTSStats #elif MIN_VERSION_base(4,6,0) - (\s -> (fromIntegral $ bytesAllocated s, fromIntegral $ bytesCopied s)) <$> getGCStats + (\s -> (int64ToWord64 $ bytesAllocated s, int64ToWord64 $ bytesCopied s, int64ToWord64 $ peakMegabytesAllocated s * 1024 * 1024)) <$> getGCStats #else - pure (0, 0) + pure (0, 0, 0) #endif -measureTime :: Int64 -> Benchmarkable -> IO Measurement -measureTime n (Benchmarkable act) = do +measure :: Word64 -> Benchmarkable -> IO Measurement +measure n (Benchmarkable act) = do performGC startTime <- fromInteger <$> getCPUTime - (startAllocs, startCopied) <- getAllocsAndCopied + (startAllocs, startCopied, startMaxMemInUse) <- getAllocsAndCopied act n endTime <- fromInteger <$> getCPUTime - (endAllocs, endCopied) <- getAllocsAndCopied + (endAllocs, endCopied, endMaxMemInUse) <- getAllocsAndCopied pure $ Measurement { measTime = endTime - startTime , measAllocs = endAllocs - startAllocs , measCopied = endCopied - startCopied + , measMaxMem = max endMaxMemInUse startMaxMemInUse } -measureTimeUntil :: Timeout -> RelStDev -> Benchmarkable -> IO Estimate -measureTimeUntil timeout (RelStDev targetRelStDev) b = do - t1 <- measureTime 1 b +measureUntil :: Bool -> Timeout -> RelStDev -> Benchmarkable -> IO Estimate +measureUntil _ _ (RelStDev targetRelStDev) b + | isInfinite targetRelStDev, targetRelStDev > 0 = do + t1 <- measure 1 b + pure $ Estimate { estMean = t1, estStdev = 0 } +measureUntil warnIfNoTimeout timeout (RelStDev targetRelStDev) b = do + t1 <- measure 1 b go 1 t1 0 where - go :: Int64 -> Measurement -> Word64 -> IO Estimate + go :: Word64 -> Measurement -> Word64 -> IO Estimate go n t1 sumOfTs = do - t2 <- measureTime (2 * n) b + t2 <- measure (2 * n) b - let Estimate (Measurement meanN allocN copiedN) stdevN = predictPerturbed t1 t2 + let Estimate (Measurement meanN allocN copiedN maxMemN) stdevN = predictPerturbed t1 t2 isTimeoutSoon = case timeout of NoTimeout -> False -- multiplying by 12/10 helps to avoid accidental timeouts - Timeout micros _ -> (sumOfTs + measTime t1 + 3 * measTime t2) `quot` (1000000 * 10 `quot` 12) >= fromInteger micros - isStDevInTargetRange = stdevN < truncate (max 0 targetRelStDev * fromIntegral meanN) - scale = (`quot` fromIntegral n) + Timeout micros _ -> (sumOfTs' + 3 * measTime t2) `quot` (1000000 * 10 `quot` 12) >= fromInteger micros + isStDevInTargetRange = stdevN < truncate (max 0 targetRelStDev * word64ToDouble meanN) + scale = (`quot` n) sumOfTs' = sumOfTs + measTime t1 case timeout of - NoTimeout | sumOfTs' > 100 * 1000000000000 + NoTimeout | warnIfNoTimeout, sumOfTs' + measTime t2 > 100 * 1000000000000 -> hPutStrLn stderr "This benchmark takes more than 100 seconds. Consider setting --timeout, if this is unexpected (or to silence this warning)." _ -> pure () - if isStDevInTargetRange || isTimeoutSoon - then pure $ Estimate (Measurement (scale meanN) (scale allocN) (scale copiedN)) (scale stdevN) + then pure $ Estimate + { estMean = Measurement (scale meanN) (scale allocN) (scale copiedN) maxMemN + , estStdev = scale stdevN } else go (2 * n) t2 sumOfTs' +-- | An internal routine to measure execution time in seconds +-- for a given timeout (put 'NoTimeout', or 'mkTimeout' 100000000 for 100 seconds) +-- and a target relative standard deviation +-- (put 'RelStDev' 0.05 for 5% or 'RelStDev' (1/0) to run only one iteration). +-- +-- 'Timeout' takes soft priority over 'RelStDev': this function prefers +-- to finish in time even if at cost of precision. However, timeout is guidance +-- not guarantee: 'measureCpuTime' can take longer, if there is not enough time +-- to run at least thrice or an iteration takes unusually long. +measureCpuTime :: Timeout -> RelStDev -> Benchmarkable -> IO Double +measureCpuTime + = ((fmap ((/ 1e12) . word64ToDouble . measTime . estMean) .) .) + . measureUntil False + instance IsTest Benchmarkable where testOptions = pure [ Option (Proxy :: Proxy RelStDev) @@ -803,9 +937,9 @@ ] run opts b = const $ case getNumThreads (lookupOption opts) of 1 -> do - est <- measureTimeUntil (lookupOption opts) (lookupOption opts) b + est <- measureUntil True (lookupOption opts) (lookupOption opts) b pure $ testPassed $ show (Response est (lookupOption opts) (lookupOption opts)) - _ -> pure $ testFailed "Benchmarks must not be run concurrently. Please pass --jobs 1 and/or avoid +RTS -N." + _ -> pure $ testFailed "Benchmarks must not be run concurrently. Please pass -j1 and/or avoid +RTS -N." -- | Attach a name to 'Benchmarkable'. -- @@ -824,6 +958,7 @@ bgroup :: String -> [Benchmark] -> Benchmark bgroup = testGroup +#if MIN_VERSION_tasty(1,2,0) -- | Compare benchmarks, reporting relative speed up or slow down. -- -- The first argument is a @tasty@ pattern, which must unambiguously @@ -837,12 +972,13 @@ -- -- This function is a vague reminiscence of @bcompare@, which existed in pre-1.0 -- versions of @criterion@, but their types are incompatible. Under the hood --- 'bcompare' is a thin wrapper over 'after'. +-- 'bcompare' is a thin wrapper over 'after' and requires @tasty-1.2@. -- bcompare :: String -> Benchmark -> Benchmark bcompare s = case parseExpr s of Nothing -> error $ "Could not parse bcompare pattern " ++ s Just e -> after_ AllSucceed (And (StringLit "tasty-bench") e) +#endif -- | Benchmarks are actually just a regular 'Test.Tasty.TestTree' in disguise. -- @@ -866,7 +1002,7 @@ funcToBench frc = (Benchmarkable .) . go where go f x n - | n <= 0 = pure () + | n == 0 = pure () | otherwise = do _ <- evaluate (frc (f x)) go f x (n - 1) @@ -944,7 +1080,7 @@ ioToBench frc act = Benchmarkable go where go n - | n <= 0 = pure () + | n == 0 = pure () | otherwise = do val <- act _ <- evaluate (frc val) @@ -955,8 +1091,9 @@ -- and compute its normal form (by means of 'force'). -- -- Pure subexpression of an effectful computation @x@ --- may be evaluated only once and get cached; use 'nfAppIO' --- to avoid this. +-- may be evaluated only once and get cached. +-- To avoid surprising results it is usually preferable +-- to use 'nfAppIO' instead. -- -- Note that forcing a normal form requires an additional -- traverse of the structure. In certain scenarios, @@ -978,8 +1115,9 @@ -- and compute its weak head normal form. -- -- Pure subexpression of an effectful computation @x@ --- may be evaluated only once and get cached; use 'whnfAppIO' --- to avoid this. +-- may be evaluated only once and get cached. +-- To avoid surprising results it is usually preferable +-- to use 'whnfAppIO' instead. -- -- Computing only a weak head normal form is -- rarely what intuitively is meant by "evaluation". @@ -1001,7 +1139,7 @@ ioFuncToBench frc = (Benchmarkable .) . go where go f x n - | n <= 0 = pure () + | n == 0 = pure () | otherwise = do val <- f x _ <- evaluate (frc val) @@ -1123,7 +1261,32 @@ (void . fin) (f . unsafePerformIO) -newtype CsvPath = CsvPath { _unCsvPath :: FilePath } +-- | A path to write results in CSV format, populated by @--csv@. +-- +-- This is an option of 'csvReporter' and can be set only globally. +-- Modifying it via 'adjustOption' or 'localOption' does not have any effect. +-- One can however pass it to 'tryIngredients' 'benchIngredients'. For example, +-- here is how to set a default CSV location: +-- +-- @ +-- import Data.Maybe +-- import System.Exit +-- import Test.Tasty.Bench +-- import Test.Tasty.Ingredients +-- import Test.Tasty.Options +-- import Test.Tasty.Runners +-- +-- main :: IO () +-- main = do +-- let benchmarks = bgroup \"All\" ... +-- opts <- parseOptions benchIngredients benchmarks +-- let opts' = changeOption (Just . fromMaybe (CsvPath "foo.csv")) opts +-- case tryIngredients benchIngredients opts' benchmarks of +-- Nothing -> exitFailure +-- Just mb -> mb >>= \b -> if b then exitSuccess else exitFailure +-- @ +-- +newtype CsvPath = CsvPath FilePath deriving (Typeable) instance IsOption (Maybe CsvPath) where @@ -1142,27 +1305,26 @@ let names = testsNames opts tree namesMap = IM.fromDistinctAscList $ zip [0..] names pure $ \smap -> do - case lookupRepeatingElements names of + case findNonUniqueElement names of Nothing -> pure () Just name -> do -- 'die' is not available before base-4.8 hPutStrLn stderr $ "CSV report cannot proceed, because name '" ++ name ++ "' corresponds to two or more benchmarks. Please disambiguate them." exitFailure let augmented = IM.intersectionWith (,) namesMap smap - hasGCStats <- getRTSStatsEnabled bracket (do h <- openFile path WriteMode hSetBuffering h LineBuffering hPutStrLn h $ "Name,Mean (ps),2*Stdev (ps)" ++ - (if hasGCStats then ",Allocated,Copied" else "") + (if hasGCStats then ",Allocated,Copied,Peak Memory" else "") pure h ) hClose (`csvOutput` augmented) - pure $ const ((== 0) . statFailures <$> computeStatistics smap) + pure $ const $ isSuccessful smap -lookupRepeatingElements :: Ord a => [a] -> Maybe a -lookupRepeatingElements = go S.empty +findNonUniqueElement :: Ord a => [a] -> Maybe a +findNonUniqueElement = go S.empty where go _ [] = Nothing go acc (x : xs) @@ -1171,7 +1333,6 @@ csvOutput :: Handle -> IntMap (TestName, TVar Status) -> IO () csvOutput h = traverse_ $ \(name, tv) -> do - hasGCStats <- getRTSStatsEnabled let csv = if hasGCStats then csvEstimateWithGC else csvEstimate r <- atomically $ readTVar tv >>= \s -> case s of Done r -> pure r; _ -> retry case safeRead (resultDescription r) of @@ -1183,10 +1344,20 @@ encodeCsv :: String -> String encodeCsv xs | any (`elem` xs) ",\"\n\r" - = '"' : concatMap (\x -> if x == '"' then "\"\"" else [x]) xs ++ "\"" + = '"' : go xs -- opening quote | otherwise = xs + where + go [] = '"' : [] -- closing quote + go ('"' : ys) = '"' : '"' : go ys + go (y : ys) = y : go ys -newtype SvgPath = SvgPath { _unSvgPath :: FilePath } +-- | A path to plot results in SVG format, populated by @--svg@. +-- +-- This is an option of 'svgReporter' and can be set only globally. +-- Modifying it via 'adjustOption' or 'localOption' does not have any effect. +-- One can however pass it to 'tryIngredients' 'benchIngredients'. +-- +newtype SvgPath = SvgPath FilePath deriving (Typeable) instance IsOption (Maybe SvgPath) where @@ -1209,8 +1380,16 @@ svgCollect ref (IM.intersectionWith (,) namesMap smap) res <- readIORef ref writeFile path (svgRender (reverse res)) - pure $ const ((== 0) . statFailures <$> computeStatistics smap) + pure $ const $ isSuccessful smap +isSuccessful :: StatusMap -> IO Bool +isSuccessful = go . IM.elems + where + go [] = pure True + go (tv : tvs) = do + b <- atomically $ readTVar tv >>= \s -> case s of Done r -> pure (resultSuccessful r); _ -> retry + if b then go tvs else pure False + svgCollect :: IORef [(TestName, Estimate)] -> IntMap (TestName, TVar Status) -> IO () svgCollect ref = traverse_ $ \(name, tv) -> do r <- atomically $ readTVar tv >>= \s -> case s of Done r -> pure r; _ -> retry @@ -1226,12 +1405,12 @@ pairs) ++ footer where dropAllPrefix - | all ("All." `isPrefixOf`) (map fst pairs) = drop 4 + | all (("All." `isPrefixOf`) . fst) pairs = drop 4 | otherwise = id l = genericLength pairs findMaxX (Estimate m stdev) = measTime m + 2 * stdev - xMax = fromIntegral $ maximum $ minBound : map (findMaxX . snd) pairs + xMax = word64ToDouble $ maximum $ minBound : map (findMaxX . snd) pairs header = printf "\n\n" (svgItemOffset l - 15) svgCanvasWidth svgFontSize svgCanvasMargin footer = "\n\n" @@ -1242,7 +1421,7 @@ svgCanvasMargin = 10 svgItemOffset :: Word64 -> Word64 -svgItemOffset i = 22 + 55 * fromIntegral i +svgItemOffset i = 22 + 55 * i svgFontSize :: Word64 svgFontSize = 16 @@ -1259,9 +1438,9 @@ x1 = boxWidth - whiskerWidth x2 = boxWidth + whiskerWidth deg = (i * 360) `quot` iMax - glyphWidth = fromIntegral svgFontSize / 2 + glyphWidth = word64ToDouble svgFontSize / 2 - scale w = fromIntegral w * (svgCanvasWidth - 2 * svgCanvasMargin) / xMax + scale w = word64ToDouble w * (svgCanvasWidth - 2 * svgCanvasMargin) / xMax boxWidth = scale (measTime m) whiskerWidth = scale (2 * stdev) boxHeight = 22 @@ -1286,24 +1465,30 @@ longText = printf longTextTemplate deg y (encodeSvg name) - y boxWidth (showPicos (measTime m)) + y boxWidth (showPicos4 (measTime m)) longTextTemplate = "\n" ++ "%s\n" ++ "%s\n" ++ "\n" - shortTextContent = encodeSvg name ++ " " ++ showPicos (measTime m) + shortTextContent = encodeSvg name ++ " " ++ showPicos4 (measTime m) shortText = printf shortTextTemplate deg y shortTextContent shortTextTemplate = "%s\n" encodeSvg :: String -> String -encodeSvg = concatMap $ \x -> case x of - '<' -> "<" - '&' -> "&" - _ -> [x] +encodeSvg [] = [] +encodeSvg ('<' : xs) = '&' : 'l' : 't' : ';' : encodeSvg xs +encodeSvg ('&' : xs) = '&' : 'a' : 'm' : 'p' : ';' : encodeSvg xs +encodeSvg (x : xs) = x : encodeSvg xs -newtype BaselinePath = BaselinePath { _unBaselinePath :: FilePath } +-- | A path to read baseline results in CSV format, populated by @--baseline@. +-- +-- This is an option of 'csvReporter' and can be set only globally. +-- Modifying it via 'adjustOption' or 'localOption' does not have any effect. +-- One can however pass it to 'tryIngredients' 'benchIngredients'. +-- +newtype BaselinePath = BaselinePath FilePath deriving (Typeable) instance IsOption (Maybe BaselinePath) where @@ -1324,8 +1509,7 @@ consoleBenchReporter = modifyConsoleReporter [Option (Proxy :: Proxy (Maybe BaselinePath))] $ \opts -> do baseline <- case lookupOption opts of Nothing -> pure S.empty - Just (BaselinePath path) -> S.fromList . lines <$> (readFile path >>= evaluate . force) - hasGCStats <- getRTSStatsEnabled + Just (BaselinePath path) -> S.fromList . joinQuotedFields . lines <$> (readFile path >>= evaluate . force) let pretty = if hasGCStats then prettyEstimateWithGC else prettyEstimate pure $ \name depR r -> case safeRead (resultDescription r) of Nothing -> r @@ -1335,14 +1519,25 @@ where slowDown = compareVsBaseline baseline name est isAcceptable -- ifSlow/ifFast may be infinite, so we cannot 'truncate' - = fromIntegral slowDown <= 100 * ifSlow - && fromIntegral slowDown >= -100 * ifFast + = int64ToDouble slowDown <= 100 * ifSlow + && int64ToDouble slowDown >= -100 * ifFast bcomp = case depR >>= safeRead . resultDescription of Nothing -> "" Just (Response depEst _ _) -> printf ", %.2fx" (estTime est / estTime depEst) +-- | A well-formed CSV entry contains an even number of quotes: 0, 2 or more. +joinQuotedFields :: [String] -> [String] +joinQuotedFields [] = [] +joinQuotedFields (x : xs) + | areQuotesBalanced x = x : joinQuotedFields xs + | otherwise = case span areQuotesBalanced xs of + (_, []) -> [] -- malformed CSV + (ys, z : zs) -> unlines (x : ys ++ [z]) : joinQuotedFields zs + where + areQuotesBalanced = even . length . filter (== '"') + estTime :: Estimate -> Double -estTime = fromIntegral . measTime . estMean +estTime = word64ToDouble . measTime . estMean -- | Return slow down in percents. compareVsBaseline :: S.Set String -> TestName -> Estimate -> Int64 @@ -1350,11 +1545,10 @@ Nothing -> 0 Just (oldTime, oldDoubleSigma) -- time and oldTime must be signed integers to use 'abs' - | abs (time - oldTime) < max (2 * fromIntegral stdev) oldDoubleSigma -> 0 + | abs (time - oldTime) < max (2 * word64ToInt64 stdev) oldDoubleSigma -> 0 | otherwise -> 100 * (time - oldTime) `quot` oldTime where - time :: Int64 - time = fromIntegral $ measTime m + time = word64ToInt64 $ measTime m mOld :: Maybe (Int64, Int64) mOld = do @@ -1394,7 +1588,9 @@ . IM.intersectionWith (\(a, b) c -> (a, b, c)) namesAndDeps in (modifySMap >=>) <$> cb opts tree where - TestReporter desc cb = consoleTestReporter + (desc, cb) = case consoleTestReporter of + TestReporter d c -> (d, c) + _ -> error "modifyConsoleReporter: consoleTestReporter must be TestReporter" isSingle [a] = Just a isSingle _ = Nothing @@ -1402,15 +1598,27 @@ testNameSeqs :: OptionSet -> TestTree -> [Seq TestName] testNameSeqs = foldTestTree trivialFold { foldSingle = const $ const . (:[]) . Seq.singleton +#if MIN_VERSION_tasty(1,4,0) , foldGroup = const $ map . (<|) +#else + , foldGroup = map . (<|) +#endif } testNamesAndDeps :: IntMap (Seq TestName) -> OptionSet -> TestTree -> [(TestName, [IM.Key])] testNamesAndDeps im = foldTestTree trivialFold { foldSingle = const $ const . (: []) . (, []) +#if MIN_VERSION_tasty(1,4,0) , foldGroup = const $ map . first . (++) . (++ ".") , foldAfter = const foldDeps +#else + , foldGroup = map . first . (++) . (++ ".") +#if MIN_VERSION_tasty(1,2,0) + , foldAfter = foldDeps +#endif +#endif } +#if MIN_VERSION_tasty(1,2,0) where foldDeps AllSucceed (And (StringLit "tasty-bench") p) = map $ second $ (++) $ findMatchingKeys im p @@ -1421,6 +1629,7 @@ foldr (\(k, v) -> if withFields v pat == Right True then (k :) else id) [] $ IM.assocs im where pat = eval pattern >>= asB +#endif postprocessResult :: (TestName -> Maybe Result -> Result -> Result) @@ -1456,3 +1665,17 @@ adNauseam = doUpdate >>= (`unless` adNauseam) _ <- forkIO adNauseam pure $ fmap (\(_, _, _, a) -> a) paired + +word64ToDouble :: Word64 -> Double +word64ToDouble = fromIntegral + +int64ToDouble :: Int64 -> Double +int64ToDouble = fromIntegral + +word64ToInt64 :: Word64 -> Int64 +word64ToInt64 = fromIntegral + +#if !MIN_VERSION_base(4,10,0) && MIN_VERSION_base(4,6,0) +int64ToWord64 :: Int64 -> Word64 +int64ToWord64 = fromIntegral +#endif diff --git a/changelog.md b/changelog.md --- a/changelog.md +++ b/changelog.md @@ -1,3 +1,12 @@ +# 0.3 + +* Report mean time with 3 significant digits. +* Report peak memory usage, when run with `+RTS -T`. +* Run benchmarks only once, if `RelStDev` is infinite. +* Make `Benchmarkable` constructor public. +* Expose `measureCpuTime` helper to run benchmarks manually. +* Expose `CsvPath`, `BaselinePath`, `SvgPath`. + # 0.2.5 * Fix comparison against baseline. diff --git a/tasty-bench.cabal b/tasty-bench.cabal --- a/tasty-bench.cabal +++ b/tasty-bench.cabal @@ -1,10 +1,11 @@ name: tasty-bench -version: 0.2.5 +version: 0.3 cabal-version: 1.18 build-type: Simple license: MIT license-file: LICENSE copyright: 2021 Andrew Lelechenko +author: Andrew Lelechenko maintainer: Andrew Lelechenko homepage: https://github.com/Bodigrim/tasty-bench bug-reports: https://github.com/Bodigrim/tasty-bench/issues @@ -36,12 +37,14 @@ ghc-options: -O2 -Wall -fno-warn-unused-imports if impl(ghc < 7.10) ghc-options: -fcontext-stack=30 + if impl(ghc >= 8.0) + ghc-options: -Wcompat -Widentities build-depends: base >= 4.3 && < 5, containers >= 0.4, deepseq >= 1.1, - tasty >= 1.2.3 + tasty >= 0.11.3 if impl(ghc < 7.8) build-depends: tagged >= 0.2