tasty-bench 0.3.4 → 0.3.5
raw patch · 4 files changed
+265/−164 lines, 4 filesdep ~containersdep ~deepseqdep ~ghc-prim
Dependency ranges changed: containers, deepseq, ghc-prim, tasty
Files
- README.md +27/−12
- changelog.md +5/−0
- src/Test/Tasty/Bench.hs +229/−148
- tasty-bench.cabal +4/−4
README.md view
@@ -40,7 +40,7 @@ is up to 6x faster than `criterion` and up to 8x faster than `gauge`. `tasty-bench` is a native Haskell library and works everywhere, where GHC-does. We support a full range of architectures (`i386`, `amd64`, `armhf`,+does, including WASM. We support a full range of architectures (`i386`, `amd64`, `armhf`, `arm64`, `ppc64le`, `s390x`) and operating systems (Linux, Windows, macOS, FreeBSD, OpenBSD, NetBSD), plus any GHC from 7.0 to 9.6. @@ -171,15 +171,15 @@ Here is a procedure used by `tasty-bench` to measure execution time: -1. Set _n_ ← 1.-2. Measure execution time _tₙ_ of _n_ iterations- and execution time _t₂ₙ_ of _2n_ iterations.-3. Find _t_ which minimizes deviation of (_nt_, _2nt_) from (_tₙ_, _t₂ₙ_),- namely _t_ ← (_tₙ_ + _2t₂ₙ_) / _5n_.+1. Set $n \leftarrow 1$.+2. Measure execution time $t_n$ of $n$ iterations+ and execution time $t_{2n}$ of $2n$ iterations.+3. Find $t$ which minimizes deviation of $(nt,2nt)$ from $(t_n,t_{2n})$,+ namely $t \leftarrow (t_n + 2t_{2n}) / 5n$. 4. If deviation is small enough (see `--stdev` below) or time is running out soon (see `--timeout` below),- return _t_ as a mean execution time.-5. Otherwise set _n_ ← _2n_ and jump back to Step 2.+ return $t$ as a mean execution time.+5. Otherwise set $n \leftarrow 2n$ and jump back to Step 2. This is roughly similar to the linear regression approach which `criterion` takes, but we fit only two last points. This allows us to simplify away all heavy-weight@@ -187,7 +187,7 @@ which are presumably shorter and noisier, do not affect overall result. This is in contrast to `criterion`, which fits all measurements and is biased to use more data points corresponding to shorter runs-(it employs _n_ ← _1.05n_ progression).+(it employs $n \leftarrow 1.05n$ progression). Mean time and its deviation does not say much about the distribution of individual timings. E. g., imagine a computation which@@ -299,9 +299,6 @@ Alternatively bake it into `cabal` file as `ghc-options: "-with-rtsopts=-A32m"`. - For GHC ≥ 8.10 consider switching benchmarks to a non-moving garbage collector,- because it decreases GC pauses and corresponding noise: `+RTS --nonmoving-gc`.- * Never compile benchmarks with `-fstatic-argument-transformation`, because it breaks a trick we use to force GHC into reevaluation of the same function application over and over again.@@ -363,6 +360,24 @@ for reliable performance measurements. Make sure to use `tasty-bench >= 0.3.4` and invoke `Test.Tasty.Bench.defaultMain` and not `Test.Tasty.defaultMain`. Note that `localOption (NumThreads 1)` quashes the warning, but does not eliminate the cause.++* If benchmarks using GHC 9.4.4+ segfault on Windows, check that you+ are not using non-moving garbage collector `--nonmoving-gc`. This is likely caused+ by [GHC issue](https://gitlab.haskell.org/ghc/ghc/-/issues/23003).+ Previous releases of `tasty-bench` recommended enabling `--nonmoving-gc`+ to stabilise benchmarks, but it's discouraged now.++* If you see++ ```+ <stdout>: commitBuffer: invalid argument (cannot encode character '\177')+ ```++ it means that your locale does not support UTF-8. `tasty-bench` makes an effort+ to force locale to UTF-8, but sometimes, when benchmarks are a part of+ a larger application, it's [impossible](https://gitlab.haskell.org/ghc/ghc/-/issues/23606)+ to do so. In such case run `locale -a` to list available locales and set a UTF-8-capable+ one (e. g., `export LANG=C.UTF-8`) before starting benchmarks. ## Isolating interfering benchmarks
changelog.md view
@@ -1,3 +1,8 @@+# 0.3.5++* Support `tasty-1.5`.+* Report benchmarking progress.+ # 0.3.4 * Force single-threaded execution in `defaultMain`.
src/Test/Tasty/Bench.hs view
@@ -4,12 +4,12 @@ License: MIT Featherlight benchmark framework (only one file!) for performance-measurement with API-mimicking [@criterion@](http://hackage.haskell.org/package/criterion)-and [@gauge@](http://hackage.haskell.org/package/gauge).-A prominent feature is built-in comparison against previous runs-and between benchmarks.+measurement with API mimicking+[@criterion@](http://hackage.haskell.org/package/criterion) and+[@gauge@](http://hackage.haskell.org/package/gauge). A prominent feature is+built-in comparison against previous runs and between benchmarks. + === How lightweight is it? There is only one source file "Test.Tasty.Bench" and no non-boot@@ -18,14 +18,16 @@ to install. Compare this to @criterion@ (10+ modules, 50+ dependencies) and @gauge@-(40+ modules, depends on @basement@ and @vector@). A build on a clean machine is up to 16x-faster than @criterion@ and up to 4x faster than @gauge@. A build without dependencies-is up to 6x faster than @criterion@ and up to 8x faster than @gauge@.+(40+ modules, depends on @basement@ and @vector@). A build on a clean+machine is up to 16x faster than @criterion@ and up to 4x faster than+@gauge@. A build without dependencies is up to 6x faster than+@criterion@ and up to 8x faster than @gauge@. -@tasty-bench@ is a native Haskell library and works everywhere, where GHC-does. We support a full range of architectures (@i386@, @amd64@, @armhf@,-@arm64@, @ppc64le@, @s390x@) and operating systems (Linux, Windows, macOS,-FreeBSD, OpenBSD, NetBSD), plus any GHC from 7.0 to 9.6.+@tasty-bench@ is a native Haskell library and works everywhere, where+GHC does, including WASM. We support a full range of architectures+(@i386@, @amd64@, @armhf@, @arm64@, @ppc64le@, @s390x@) and operating+systems (Linux, Windows, macOS, FreeBSD, OpenBSD, NetBSD), plus any GHC+from 7.0 to 9.6. === How is it possible? @@ -116,49 +118,49 @@ > All 3 tests passed (7.25s) The output says that, for instance, the first benchmark was repeatedly-executed for 2.13 seconds (wall-clock time), its predicted mean CPU time was-63 nanoseconds and means of individual samples do not often diverge from it-further than ±3.4 nanoseconds (double standard deviation). Take standard-deviation numbers with a grain of salt; there are lies, damned lies, and-statistics.+executed for 2.13 seconds (wall-clock time), its predicted mean CPU time+was 63 nanoseconds and means of individual samples do not often diverge+from it further than ±3.4 nanoseconds (double standard deviation). Take+standard deviation numbers with a grain of salt; there are lies, damned+lies, and statistics. === Wall-clock time vs. CPU time -What time are we talking about?-Both @criterion@ and @gauge@ by default report wall-clock time, which is-affected by any other application which runs concurrently.-Ideally benchmarks are executed on a dedicated server without any other load,-but — let's face the truth — most of developers run benchmarks-on a laptop with a hundred other services and a window manager, and-watch videos while waiting for benchmarks to finish. That's the cause-of a notorious "variance introduced by outliers: 88% (severely inflated)" warning.+What time are we talking about? Both @criterion@ and @gauge@ by default+report wall-clock time, which is affected by any other application which+runs concurrently. Ideally benchmarks are executed on a dedicated server+without any other load, but — let’s face the truth — most of developers+run benchmarks on a laptop with a hundred other services and a window+manager, and watch videos while waiting for benchmarks to finish. That’s+the cause of a notorious “variance introduced by outliers: 88% (severely+inflated)” warning. To alleviate this issue @tasty-bench@ measures CPU time by 'getCPUTime'-instead of wall-clock time by default.-It does not provide a perfect isolation from other processes (e. g.,-if CPU cache is spoiled by others, populating data back from RAM-is your burden), but is a bit more stable.+instead of wall-clock time by default. It does not provide a perfect+isolation from other processes (e. g., if CPU cache is spoiled by others,+populating data back from RAM is your burden), but is a bit more stable. -Caveat: this means that for multithreaded algorithms-@tasty-bench@ reports total elapsed CPU time across all cores, while-@criterion@ and @gauge@ print maximum of core's wall-clock time.-It also means that by default @tasty-bench@ does not measure time spent out of process,-e. g., calls to other executables. To work around this limitation-use @--time-mode@ command-line option or set it locally via 'TimeMode' option.+Caveat: this means that for multithreaded algorithms @tasty-bench@+reports total elapsed CPU time across all cores, while @criterion@ and+@gauge@ print maximum of core’s wall-clock time. It also means that by+default @tasty-bench@ does not measure time spent out of process, e. g.,+calls to other executables. To work around this limitation use+@--time-mode@ command-line option or set it locally via 'TimeMode'+option. === Statistical model Here is a procedure used by @tasty-bench@ to measure execution time: -1. Set \( n \leftarrow 1 \).-2. Measure execution time \( t_n \) of \( n \) iterations and execution time- \( t_{2n} \) of \( 2n \) iterations.-3. Find \( t \) which minimizes deviation of \( (nt, 2nt) \) from- \( (t_n, t_{2n}) \), namely \( t \leftarrow (t_n + 2t_{2n}) / 5n \).-4. If deviation is small enough (see @--stdev@ below)- or time is running out soon (see @--timeout@ below),- return \( t \) as a mean execution time.-5. Otherwise set \( n \leftarrow 2n \) and jump back to Step 2.+1. Set \(n \leftarrow 1\).+2. Measure execution time \(t_n\) of \(n\) iterations and execution+ time \(t_{2n}\) of \(2n\) iterations.+3. Find \(t\) which minimizes deviation of \((nt,2nt)\) from+ \((t_n,t_{2n})\), namely \(t \leftarrow (t_n + 2t_{2n}) / 5n\).+4. If deviation is small enough (see @--stdev@ below) or time is+ running out soon (see @--timeout@ below), return \(t\) as a mean+ execution time.+5. Otherwise set \(n \leftarrow 2n\) and jump back to Step 2. This is roughly similar to the linear regression approach which @criterion@ takes, but we fit only two last points. This allows us to@@ -166,15 +168,15 @@ earlier measurements, which are presumably shorter and noisier, do not affect overall result. This is in contrast to @criterion@, which fits all measurements and is biased to use more data points corresponding to-shorter runs (it employs \( n \leftarrow 1.05n \) progression).+shorter runs (it employs \(n \leftarrow 1.05n\) progression). -Mean time and its deviation does not say much about the-distribution of individual timings. E. g., imagine a computation which-(according to a coarse system timer) takes either 0 ms or 1 ms with equal-probability. While one would be able to establish that its mean time is 0.5 ms-with a very small deviation, this does not imply that individual measurements-are anywhere near 0.5 ms. Even assuming an infinite precision of a system-timer, the distribution of individual times is not known to be+Mean time and its deviation does not say much about the distribution of+individual timings. E. g., imagine a computation which (according to a+coarse system timer) takes either 0 ms or 1 ms with equal probability.+While one would be able to establish that its mean time is 0.5 ms with a+very small deviation, this does not imply that individual measurements+are anywhere near 0.5 ms. Even assuming an infinite precision of a+system timer, the distribution of individual times is not known to be <https://en.wikipedia.org/wiki/Normal_distribution normal>. Obligatory disclaimer: statistics is a tricky matter, there is no@@ -186,8 +188,8 @@ === Memory usage -Configuring RTS to collect GC statistics-(e. g., via @cabal@ @bench@ @--benchmark-options@ @\'+RTS@ @-T\'@ or+Configuring RTS to collect GC statistics (e. g., via+@cabal@ @bench@ @--benchmark-options@ @\'+RTS@ @-T\'@ or @stack@ @bench@ @--ba@ @\'+RTS@ @-T\'@) enables @tasty-bench@ to estimate and report memory usage: @@ -202,8 +204,8 @@ > > All 3 tests passed (7.25s) -This data is reported as per 'RTSStats' fields: 'allocated_bytes', 'copied_bytes'-and 'max_mem_in_use_bytes'.+This data is reported as per 'RTSStats' fields: 'allocated_bytes',+'copied_bytes' and 'max_mem_in_use_bytes'. === Combining tests and benchmarks @@ -255,9 +257,10 @@ - If benchmarks take too long, set @--timeout@ to limit execution time of individual benchmarks, and @tasty-bench@ will do its best to fit- into a given time frame. Without @--timeout@ we rerun benchmarks until- achieving a target precision set by @--stdev@, which in a noisy- environment of a modern laptop with GUI may take a lot of time.+ into a given time frame. Without @--timeout@ we rerun benchmarks+ until achieving a target precision set by @--stdev@, which in a+ noisy environment of a modern laptop with GUI may take a lot of+ time. While @criterion@ runs each benchmark at least for 5 seconds, @tasty-bench@ is happy to conclude earlier, if it does not@@ -268,15 +271,12 @@ A common source of noisiness is garbage collection. Setting a larger allocation area (/nursery/) is often a good idea, either via @cabal@ @bench@ @--benchmark-options@ @\'+RTS@ @-A32m\'@ or- @stack@ @bench@ @--ba@ @\'+RTS@ @-A32m\'@. Alternatively bake it into @cabal@- file as @ghc-options:@ @\"-with-rtsopts=-A32m\"@.-- For GHC ≥ 8.10 consider switching benchmarks to a non-moving garbage collector,- because it decreases GC pauses and corresponding noise: @+RTS@ @--nonmoving-gc@.+ @stack@ @bench@ @--ba@ @\'+RTS@ @-A32m\'@. Alternatively bake it into+ @cabal@ file as @ghc-options:@ @\"-with-rtsopts=-A32m\"@. -- Never compile benchmarks with @-fstatic-argument-transformation@, because it- breaks a trick we use to force GHC into reevaluation of the same function application- over and over again.+- Never compile benchmarks with @-fstatic-argument-transformation@,+ because it breaks a trick we use to force GHC into reevaluation of+ the same function application over and over again. - If benchmark results look malformed like below, make sure that you are invoking @Test.Tasty.Bench.@'Test.Tasty.Bench.defaultMain' and not@@ -301,21 +301,22 @@ way. This is a fundamental restriction of @tasty@ to list and filter benchmarks without launching missiles. -- If benchmarks fail with @Test dependencies form a loop@- or @Test dependencies have cycles@, this is likely- because of 'bcompare', which compares a benchmark with itself.- Locating a benchmark in a global environment may be tricky, please refer to- [@tasty@ documentation](https://github.com/UnkindPartition/tasty#patterns) for details- and consider using 'locateBenchmark'.+- If benchmarks fail with @Test dependencies form a loop@ or+ @Test dependencies have cycles@, this is likely because of+ 'bcompare', which compares a benchmark with itself. Locating a+ benchmark in a global environment may be tricky, please refer to+ [@tasty@ documentation](https://github.com/UnkindPartition/tasty#patterns)+ for details and consider using 'locateBenchmark'. - When seeing > This benchmark takes more than 100 seconds. Consider setting --timeout, if this is unexpected (or to silence this warning). - do follow the advice: abort benchmarks and pass @-t100@ or similar. Unless you are- benchmarking a very computationally expensive function, a single benchmark should- stabilize after a couple of seconds. This warning is a sign that your environment- is too noisy, in which case @tasty-bench@ will continue trying with exponentially+ do follow the advice: abort benchmarks and pass @-t100@ or similar.+ Unless you are benchmarking a very computationally expensive+ function, a single benchmark should stabilize after a couple of+ seconds. This warning is a sign that your environment is too noisy,+ in which case @tasty-bench@ will continue trying with exponentially longer intervals, often unproductively. - The following error can be thrown when benchmarks are built with@@ -323,28 +324,47 @@ > Benchmarks must not be run concurrently. Please pass -j1 and/or avoid +RTS -N. - The underlying cause is that @tasty@ runs tests concurrently, which is harmful- for reliable performance measurements. Make sure to use @tasty-bench >= 0.3.4@- and invoke @Test.Tasty.Bench.@'Test.Tasty.Bench.defaultMain' and not- @Test.Tasty.@`Test.Tasty.defaultMain`. Note that 'localOption' ('NumThreads' 1)+ The underlying cause is that @tasty@ runs tests concurrently, which+ is harmful for reliable performance measurements. Make sure to use+ @tasty-bench >= 0.3.4@ and invoke @Test.Tasty.Bench.@'Test.Tasty.Bench.defaultMain' and+ not @Test.Tasty.@'Test.Tasty.defaultMain'. Note that 'localOption' ('NumThreads' 1) quashes the warning, but does not eliminate the cause. +- If benchmarks using GHC 9.4.4+ segfault on Windows, check that you+ are not using non-moving garbage collector @--nonmoving-gc@. This is+ likely caused by+ <https://gitlab.haskell.org/ghc/ghc/-/issues/23003 GHC issue>.+ Previous releases of @tasty-bench@ recommended enabling+ @--nonmoving-gc@ to stabilise benchmarks, but it’s discouraged now.++- If you see++ > <stdout>: commitBuffer: invalid argument (cannot encode character '\177')++ it means that your locale does not support UTF-8. @tasty-bench@+ makes an effort to force locale to UTF-8, but sometimes, when+ benchmarks are a part of a larger application, it’s+ <https://gitlab.haskell.org/ghc/ghc/-/issues/23606 impossible> to do+ so. In such case run @locale@ @-a@ to list available locales and set a+ UTF-8-capable one (e. g., @export@ @LANG=C.UTF-8@) before starting+ benchmarks.+ === Isolating interfering benchmarks One difficulty of benchmarking in Haskell is that it is hard to isolate benchmarks so that they do not interfere. Changing the order of-benchmarks or skipping some of them has an effect on heap's layout and+benchmarks or skipping some of them has an effect on heap’s layout and thus affects garbage collection. This issue is well attested in <https://github.com/haskell/criterion/issues/166 both>-<https://github.com/haskell/criterion/issues/60 criterion> and-<https://github.com/vincenthz/hs-gauge/issues/2 gauge>.+[@criterion@](https://github.com/haskell/criterion/issues/60) and+[@gauge@](https://github.com/vincenthz/hs-gauge/issues/2). Usually (but not always) skipping some benchmarks speeds up remaining-ones. That's because once a benchmark allocated heap which for some-reason was not promptly released afterwards (e. g., it forced a-top-level thunk in an underlying library), all further benchmarks are-slowed down by garbage collector processing this additional amount of-live data over and over again.+ones. That’s because once a benchmark allocated heap which for some+reason was not promptly released afterwards (e. g., it forced a top-level+thunk in an underlying library), all further benchmarks are slowed down+by garbage collector processing this additional amount of live data over+and over again. There are several mitigation strategies. First of all, giving garbage collector more breathing space by @+RTS@ @-A32m@ (or more) is often good@@ -378,20 +398,23 @@ > cabal run -v0 all:benches -- -l | sed -e 's/[\"]/\\\\\\&/g' | while read -r name; do cabal run -v0 all:benches -- -p '$0 == "'"$name"'"'; done -This assumes that there is a single benchmark suite in the project-and that benchmark names do not contain newlines.+This assumes that there is a single benchmark suite in the project and+that benchmark names do not contain newlines. === Comparison against baseline -One can compare benchmark results against an earlier run in an automatic way.+One can compare benchmark results against an earlier run in an automatic+way. -When using this feature, it's especially important to compile benchmarks with-@ghc-options:@ [@-fproc-alignment@](https://downloads.haskell.org/ghc/latest/docs/users_guide/debugging.html#ghc-flag--fproc-alignment)@=64@, otherwise results could be skewed by-intermittent changes in cache-line alignment.+When using this feature, it’s especially important to compile benchmarks+with+@ghc-options:@ [@-fproc-alignment@](https://downloads.haskell.org/ghc/latest/docs/users_guide/debugging.html#ghc-flag--fproc-alignment)@=64@,+otherwise results could be skewed by intermittent changes in cache-line+alignment. -Firstly, run @tasty-bench@ with-@--csv@ @FILE@ key to dump results to @FILE@ in CSV format-(it could be a good idea to set smaller @--stdev@, if possible):+Firstly, run @tasty-bench@ with @--csv@ @FILE@ key to dump results to+@FILE@ in CSV format (it could be a good idea to set smaller @--stdev@,+if possible): > Name,Mean (ps),2*Stdev (ps) > All.Fibonacci numbers.fifth,48453,4060@@ -421,13 +444,18 @@ even [@tasty-rerun@](http://hackage.haskell.org/package/tasty-rerun) package to focus on rerunning failing items only. -If you wish to compare two CSV reports non-interactively, here is a handy @awk@ incantation:+If you wish to compare two CSV reports non-interactively, here is a+handy @awk@ incantation: > awk 'BEGIN{FS=",";OFS=",";print "Name,Old,New,Ratio"}FNR==1{trueNF=NF;next}NF<trueNF{print "Benchmark names should not contain newlines";exit 1}FNR==NR{oldTime=$(NF-trueNF+2);NF-=trueNF-1;a[$0]=oldTime;next}{newTime=$(NF-trueNF+2);NF-=trueNF-1;print $0,a[$0],newTime,newTime/a[$0];gs+=log(newTime/a[$0]);gc++}END{if(gc>0)print "Geometric mean,,",exp(gs/gc)}' old.csv new.csv -Note that columns in CSV report are different from what @criterion@ or @gauge@-would produce. If names do not contain commas, missing columns can be faked this way:+A larger shell snippet to compare two @git@ commits can be found in+@compare_benches.sh@. +Note that columns in CSV report are different from what @criterion@ or+@gauge@ would produce. If names do not contain commas, missing columns+can be faked this way:+ > awk 'BEGIN{FS=",";OFS=",";print "Name,Mean,MeanLB,MeanUB,Stddev,StddevLB,StddevUB"}NR==1{trueNF=NF;next}NF<trueNF{print $0;next}{mean=$(NF-trueNF+2);stddev=$(NF-trueNF+3);NF-=trueNF-1;print $0,mean/1e12,mean/1e12,mean/1e12,stddev/2e12,stddev/2e12,stddev/2e12}' To fake @gauge@ in @--csvraw@ mode use@@ -436,8 +464,8 @@ === Comparison between benchmarks -You can also compare benchmarks to each other without any-external tools, all in the comfort of your terminal.+You can also compare benchmarks to each other without any external+tools, all in the comfort of your terminal. > import Test.Tasty.Bench >@@ -467,11 +495,12 @@ To locate a baseline benchmark in a larger suite use 'locateBenchmark'. -One can leverage comparisons between benchmarks to implement portable performance-tests, expressing properties like "this algorithm must be at least twice faster-than that one" or "this operation should not be more than thrice slower than that".-This can be achieved with 'bcompareWithin', which takes an acceptable interval-of performance as an argument.+One can leverage comparisons between benchmarks to implement portable+performance tests, expressing properties like “this algorithm must be at+least twice faster than that one” or “this operation should not be more+than thrice slower than that”. This can be achieved with+'bcompareWithin', which takes an acceptable interval of performance as+an argument. === Plotting results @@ -482,20 +511,24 @@  + === Build flags -Build flags are a brittle subject and users do not normally need to touch them.+Build flags are a brittle subject and users do not normally need to+touch them. -* If you find yourself in an environment, where @tasty@ is not available and you- have access to boot packages only, you can still use @tasty-bench@! Just copy- @Test\/Tasty\/Bench.hs@ to your project (imagine it like a header-only C library).- It will provide you with functions to build 'Benchmarkable' and run them manually- via 'measureCpuTime'. This mode of operation can be also configured- by disabling Cabal flag @tasty@.+- If you find yourself in an environment, where @tasty@ is not+ available and you have access to boot packages only, you can still+ use @tasty-bench@! Just copy @Test\/Tasty\/Bench.hs@ to your project+ (imagine it like a header-only C library). It will provide you with+ functions to build 'Benchmarkable' and run them manually via+ 'measureCpuTime'. This mode of operation can be also configured by+ disabling Cabal flag @tasty@. -* If results are amiss or oscillate wildly and adjusting @--timeout@ and @--stdev@- does not help, you may be interested to investigate individual timings of- successive runs by enabling Cabal flag @debug@. This will pipe raw data into @stderr@.+- If results are amiss or oscillate wildly and adjusting @--timeout@+ and @--stdev@ does not help, you may be interested to investigate+ individual timings of successive runs by enabling Cabal flag+ @debug@. This will pipe raw data into @stderr@. === Command-line options @@ -504,8 +537,8 @@ [@-p@, @--pattern@]: This is a standard @tasty@ option, which allows filtering benchmarks- by a pattern or @awk@ expression. Please refer- to [@tasty@ documentation](https://github.com/UnkindPartition/tasty#patterns)+ by a pattern or @awk@ expression. Please refer to+ [@tasty@ documentation](https://github.com/UnkindPartition/tasty#patterns) for details. [@-t@, @--timeout@]:@@ -515,19 +548,19 @@ @tasty-bench@ will make an effort to report results (even if of subpar quality) before timeout. Setting timeout too tight (insufficient for at least three iterations) will result in a- benchmark failure. One can adjust it locally for a group- of benchmarks, e. g., 'localOption' ('mkTimeout' 100000000) for 100 seconds.+ benchmark failure. One can adjust it locally for a group of+ benchmarks, e. g., 'localOption' ('mkTimeout' 100000000) for 100+ seconds. [@--stdev@]: Target relative standard deviation of measurements in percents (5% by default). Large values correspond to fast and loose benchmarks,- and small ones to long and precise.- It can also be adjusted locally for a group of benchmarks,- e. g., 'localOption' ('RelStDev' 0.02).- If benchmarking takes far too long, consider setting @--timeout@,- which will interrupt benchmarks,- potentially before reaching the target deviation.+ and small ones to long and precise. It can also be adjusted locally+ for a group of benchmarks, e. g., 'localOption' ('RelStDev' 0.02). If+ benchmarking takes far too long, consider setting @--timeout@, which+ will interrupt benchmarks, potentially before reaching the target+ deviation. [@--csv@]: @@ -544,9 +577,9 @@ benchmark is unacceptably slower \/ faster than baseline (see @--baseline@), it will be reported as failed. Can be used in conjunction with a standard @tasty@ option @--hide-successes@ to- show only problematic benchmarks.- Both options can be adjusted locally for a group of benchmarks,- e. g., 'localOption' ('FailIfSlower' 0.10).+ show only problematic benchmarks. Both options can be adjusted+ locally for a group of benchmarks, e. g.,+ 'localOption' ('FailIfSlower' 0.10). [@--svg@]: @@ -554,7 +587,8 @@ [@--time-mode@]: - Whether to measure CPU time (@cpu@, default) or wall-clock time (@wall@).+ Whether to measure CPU time (@cpu@, default) or wall-clock time+ (@wall@). [@+RTS@ @-T@]: @@ -562,8 +596,8 @@ === Custom command-line options -As usual with @tasty@, it is easy to extend benchmarks with custom command-line options.-Here is an example:+As usual with @tasty@, it is easy to extend benchmarks with custom+command-line options. Here is an example: > import Data.Proxy > import Test.Tasty.Bench@@ -730,6 +764,8 @@ String -- ^ textual representation (e. g., @"0.2s"@) | NoTimeout deriving (Show)++type Progress = () #endif @@ -1049,12 +1085,19 @@ pure meas #endif -measureUntil :: TimeMode -> Bool -> Timeout -> RelStDev -> Benchmarkable -> IO Estimate-measureUntil timeMode _ _ (RelStDev targetRelStDev) b+measureUntil+ :: (Progress -> IO ())+ -> TimeMode+ -> Bool+ -> Timeout+ -> RelStDev+ -> Benchmarkable+ -> IO Estimate+measureUntil _ timeMode _ _ (RelStDev targetRelStDev) b | isInfinite targetRelStDev, targetRelStDev > 0 = do t1 <- measure timeMode 1 b pure $ Estimate { estMean = t1, estStdev = 0 }-measureUntil timeMode warnIfNoTimeout timeout (RelStDev targetRelStDev) b = do+measureUntil yieldProgress timeMode warnIfNoTimeout timeout (RelStDev targetRelStDev) b = do t1 <- measure' 1 b go 1 t1 0 where@@ -1073,15 +1116,26 @@ scale = (`quot` n) sumOfTs' = sumOfTs + measTime t1 + let scaledEstimate = Estimate+ { estMean = Measurement (scale meanN) (scale allocN) (scale copiedN) maxMemN+ , estStdev = scale stdevN }++#ifdef MIN_VERSION_tasty+ yieldProgress $ Progress+ { progressText = prettyEstimate scaledEstimate+ , progressPercent = 0.0+ }+#else+ yieldProgress ()+#endif+ case timeout of NoTimeout | warnIfNoTimeout, sumOfTs' + measTime t2 > 100 * 1000000000000 -> hPutStrLn stderr "This benchmark takes more than 100 seconds. Consider setting --timeout, if this is unexpected (or to silence this warning)." _ -> pure () if isStDevInTargetRange || isTimeoutSoon- then pure $ Estimate- { estMean = Measurement (scale meanN) (scale allocN) (scale copiedN) maxMemN- , estStdev = scale stdevN }+ then pure scaledEstimate else go (2 * n) t2 sumOfTs' -- | An internal routine to measure CPU execution time in seconds@@ -1108,7 +1162,7 @@ ( word64ToDouble (measTime (estMean x)) / 1e12 , word64ToDouble (estStdev x) / 1e12 )) .) .)- . measureUntil CpuTime False+ . measureUntil (const $ pure ()) CpuTime False #ifdef MIN_VERSION_tasty @@ -1121,10 +1175,10 @@ , Option (Proxy :: Proxy FailIfFaster) , Option (Proxy :: Proxy TimeMode) ]- run opts b = const $ case getNumThreads (lookupOption opts) of+ run opts b yieldProgress = case getNumThreads (lookupOption opts) of 1 -> do let timeMode = lookupOption opts- est <- measureUntil timeMode True (lookupOption opts) (lookupOption opts) b+ est <- measureUntil yieldProgress timeMode True (lookupOption opts) (lookupOption opts) b let FailIfSlower ifSlower = lookupOption opts FailIfFaster ifFaster = lookupOption opts pure $ testPassed $ show (WithLoHi est (1 - ifFaster) (1 + ifSlower))@@ -1244,7 +1298,13 @@ installSignalHandlers let b = testGroup "All" bs opts <- parseOptions benchIngredients b- case tryIngredients benchIngredients (setOption (NumThreads 1) opts) b of+ let opts' = setOption (NumThreads 1) opts+#if MIN_VERSION_tasty(1,5,0)+ opts'' = setOption (MinDurationToReport 1000000000000) opts'+#else+ opts'' = opts'+#endif+ case tryIngredients benchIngredients opts'' b of Nothing -> exitFailure Just act -> act >>= \x -> if x then exitSuccess else exitFailure @@ -1535,6 +1595,11 @@ -- @Gauge.@'Gauge.env', and involves 'unsafePerformIO'. Consider using -- 'withResource' instead. --+-- When working with a mutable environment, bear in mind that it is threaded+-- through all iterations of a benchmark. @tasty-bench@ does not roll it back+-- or reset, it's user's resposibility. You might have better luck+-- with @Criterion.@'Criterion.perBatchEnv' or @Criterion.@'Criterion.perRunEnv'.+-- -- 'defaultMain' requires that the hierarchy of benchmarks and their names is -- independent of underlying 'IO' actions. While executing 'IO' inside 'bench' -- via 'nfIO' is fine, and reading test data from files via 'env' is also fine,@@ -1931,10 +1996,13 @@ isSingle (Unique a) = Just a isSingle _ = Nothing +-- | Convert a test tree to a list of test names. testNameSeqs :: OptionSet -> TestTree -> [Seq TestName] testNameSeqs = foldTestTree trivialFold { foldSingle = const $ const . (:[]) . Seq.singleton-#if MIN_VERSION_tasty(1,4,0)+#if MIN_VERSION_tasty(1,5,0)+ , foldGroup = const $ (. concat) . map . (<|)+#elif MIN_VERSION_tasty(1,4,0) , foldGroup = const $ map . (<|) #else , foldGroup = map . (<|)@@ -1945,7 +2013,11 @@ testNamesAndDeps im = foldTestTree trivialFold { foldSingle = const $ const . (: []) . (, mempty) #if MIN_VERSION_tasty(1,4,0)+#if MIN_VERSION_tasty(1,5,0)+ , foldGroup = const $ (. concat) . map . first . (++) . (++ ".")+#else , foldGroup = const $ map . first . (++) . (++ ".")+#endif , foldAfter = const foldDeps #else , foldGroup = map . first . (++) . (++ ".")@@ -1995,9 +2067,18 @@ writeTVar oldTV (Done (f name depRes res)) pure (Any True, All True)- -- ignoring Progress nodes, we do not report any- -- it would be helpful to have instance Eq Progress- _ -> pure (Any False, All False)+#if MIN_VERSION_tasty(1,5,0)+ Executing newProgr -> do+ let updated = case old of+ Executing oldProgr -> oldProgr /= newProgr+ _ -> True+ when updated $+ writeTVar oldTV (Executing newProgr)+ pure (Any updated, All False)+#else+ Executing{} -> pure (Any False, All False)+#endif+ NotStarted -> pure (Any False, All False) if anyUpdated || allDone then pure allDone else retry adNauseam = doUpdate >>= (`unless` adNauseam) _ <- forkIO adNauseam
tasty-bench.cabal view
@@ -1,5 +1,5 @@ name: tasty-bench-version: 0.3.4+version: 0.3.5 cabal-version: 1.18 build-type: Simple license: MIT@@ -25,7 +25,7 @@ example.svg README.md -tested-with: GHC == 9.6.1 GHC == 9.4.4, GHC == 9.2.7, GHC==9.0.2, GHC==8.10.7, GHC==8.8.4, GHC==8.6.5, GHC==8.4.4, GHC==8.2.2, GHC==8.0.2, GHC==7.10.3, GHC==7.8.4, GHC==7.6.3, GHC==7.4.2, GHC==7.2.2, GHC==7.0.4+tested-with: GHC == 9.8.1, GHC == 9.6.2, GHC == 9.4.7, GHC == 9.2.8, GHC==9.0.2, GHC==8.10.7, GHC==8.8.4, GHC==8.6.5, GHC==8.4.4, GHC==8.2.2, GHC==8.0.2, GHC==7.10.3, GHC==7.8.4, GHC==7.6.3, GHC==7.4.2, GHC==7.2.2, GHC==7.0.4 source-repository head type: git@@ -56,12 +56,12 @@ build-depends: base >= 4.3 && < 5,- deepseq >= 1.1 && < 1.5,+ deepseq >= 1.1 && < 1.6, ghc-prim < 0.11 if flag(tasty) build-depends: containers >= 0.4 && < 0.7,- tasty >= 1.2.3 && < 1.5+ tasty >= 1.2.3 && < 1.6 if impl(ghc < 7.8) build-depends: tagged >= 0.2 && < 0.9