tasty-bench 0.4.1 → 0.5
raw patch · 4 files changed
+265/−144 lines, 4 filesdep ~basedep ~time
Dependency ranges changed: base, time
Files
- README.md +34/−33
- changelog.md +6/−0
- src/Test/Tasty/Bench.hs +216/−104
- tasty-bench.cabal +9/−7
README.md view
@@ -35,27 +35,29 @@ So if you already depend on `tasty` for a test suite, there is nothing else to install. -Compare this to `criterion` (10+ modules, 50+ dependencies) and `gauge` (40+ modules, depends on `basement` and `vector`). A build on a clean machine is up to 16x+Compare this to `criterion` (10+ modules, 50+ dependencies)+and `gauge` (40+ modules, depends on `basement` and `vector`).+A build on a clean machine is up to 16x faster than `criterion` and up to 4x faster than `gauge`. A build without dependencies is up to 6x faster than `criterion` and up to 8x faster than `gauge`. `tasty-bench` is a native Haskell library and works everywhere, where GHC does, including WASM. We support a full range of architectures (`i386`, `amd64`, `armhf`,-`arm64`, `ppc64le`, `s390x`) and operating systems (Linux, Windows, macOS,-FreeBSD, OpenBSD, NetBSD), plus any GHC from 8.0 to 9.10+`arm64`, `ppc64le`, `s390x`, `riscv64`) and operating systems (Linux, Windows, macOS,+FreeBSD, OpenBSD, NetBSD), plus any GHC from 8.0 to 9.14 (and earlier releases stretch back to GHC 7.0). ## How is it possible? Our benchmarks are literally regular `tasty` tests, so we can leverage all existing machinery for command-line options, resource management, structuring,-listing and filtering benchmarks, running and reporting results. It also means+listing and filtering benchmarks, running them and reporting results. It also means that `tasty-bench` can be used in conjunction with other `tasty` ingredients. Unlike `criterion` and `gauge` we use a very simple statistical model described below. This is arguably a questionable choice, but it works pretty well in practice. A rare developer is sufficiently well-versed in probability theory-to make sense and use of all numbers generated by `criterion`.+to make sense and use of all numbers generously generated by `criterion`. ## How to switch? @@ -80,7 +82,8 @@ ## How to write a benchmark? -Benchmarks are declared in a separate section of `cabal` file:+Benchmarks are declared+in [a separate section of `cabal` file](https://cabal.readthedocs.io/en/stable/cabal-package-description-file.html#benchmarks): ```cabal cabal-version: 2.0@@ -117,7 +120,7 @@ ``` Since `tasty-bench` provides an API compatible with `criterion`,-one can refer to [its documentation](http://www.serpentine.com/criterion/tutorial.html#how-to-write-a-benchmark-suite) for more examples.+one can refer to [its documentation](https://hackage.haskell.org/package/criterion#readme) for more examples. ## How to read results? @@ -127,18 +130,17 @@ ``` All Fibonacci numbers- fifth: OK (2.13s)+ fifth: OK 63 ns ± 3.4 ns- tenth: OK (1.71s)+ tenth: OK 809 ns ± 73 ns- twentieth: OK (3.39s)+ twentieth: OK 104 μs ± 4.9 μs All 3 tests passed (7.25s) ``` -The output says that, for instance, the first benchmark was repeatedly-executed for 2.13 seconds (wall-clock time), its predicted mean CPU time was+The output says that, for instance, in the first benchmark the mean CPU time is 63 nanoseconds and means of individual samples do not often diverge from it further than ±3.4 nanoseconds (double standard deviation). Take standard deviation numbers with a grain of salt; there are lies, damned lies, and@@ -218,11 +220,11 @@ ``` All Fibonacci numbers- fifth: OK (2.13s)+ fifth: OK 63 ns ± 3.4 ns, 223 B allocated, 0 B copied, 2.0 MB peak memory- tenth: OK (1.71s)+ tenth: OK 809 ns ± 73 ns, 2.3 KB allocated, 0 B copied, 4.0 MB peak memory- twentieth: OK (3.39s)+ twentieth: OK 104 μs ± 4.9 μs, 277 KB allocated, 59 B copied, 5.0 MB peak memory All 3 tests passed (7.25s)@@ -281,9 +283,9 @@ ``` All- fibo 20: OK (3.02s)+ fibo 20: OK 104 μs ± 4.9 μs- myFibo 20: OK (1.99s)+ myFibo 20: OK 71 μs ± 5.3 μs myFibo = fibo: FAIL *** Failed! Falsified (after 5 tests and 1 shrink):@@ -296,7 +298,7 @@ We see that `myFibo` is indeed significantly faster than `fibo`, but unfortunately does not do the same thing. One should probably-look for another way to speed up generation of Fibonacci numbers.+look for another way to speed up Fibonacci numbers. ## Troubleshooting @@ -304,7 +306,7 @@ of individual benchmarks, and `tasty-bench` will do its best to fit into a given time frame. Without `--timeout` we rerun benchmarks until achieving a target precision set by `--stdev`, which in a noisy environment- of a modern laptop with GUI may take a lot of time.+ of a modern laptop with hundreds of service processes may take a lot of time. While `criterion` runs each benchmark at least for 5 seconds, `tasty-bench` is happy to conclude earlier, if it does not compromise@@ -324,7 +326,7 @@ * If benchmark results look malformed like below, make sure that you are invoking `Test.Tasty.Bench.defaultMain` and not `Test.Tasty.defaultMain`- (the difference is `consoleBenchReporter` vs. `consoleTestReporter`):+ (the underlying difference is `consoleBenchReporter` vs. `consoleTestReporter`): ``` All@@ -344,10 +346,10 @@ Unexpected state of the resource (NotCreated) in getResource. Report as a tasty bug. ``` - this is likely caused by `env` or `envWithCleanup` affecting benchmarks structure.+ this is likely caused by `env` or `envWithCleanup` affecting the structure of benchmarks. You can use `env` to read test data from `IO`, but not to read benchmark names- or affect their hierarchy in other way. This is a fundamental restriction of `tasty`- to list and filter benchmarks without launching missiles.+ or affect their hierarchy in another way. This is a fundamental restriction of `tasty`+ so that it can list and filter benchmarks without launching missiles. Strict pattern-matching on resource is also prohibited. For instance, if it is a tuple, the second argument of `env` should use a lazy pattern match@@ -386,7 +388,7 @@ * If benchmarks using GHC 9.4.4+ segfault on Windows, check that you are not using non-moving garbage collector `--nonmoving-gc`. This is likely caused- by [GHC issue](https://gitlab.haskell.org/ghc/ghc/-/issues/23003).+ by [GHC issue #23003](https://gitlab.haskell.org/ghc/ghc/-/issues/23003). Previous releases of `tasty-bench` recommended enabling `--nonmoving-gc` to stabilise benchmarks, but it's discouraged now. @@ -404,9 +406,8 @@ ``` it means that your locale does not support UTF-8. `tasty-bench` makes an effort- to force locale to UTF-8, but sometimes, when benchmarks are a part of- a larger application, it's [impossible](https://gitlab.haskell.org/ghc/ghc/-/issues/23606)- to do so. In such case run `locale -a` to list available locales and set a UTF-8-capable+ to force locale to UTF-8, but it's not bulletproof.+ In such case run `locale -a` to list available locales and set a UTF-8-capable one (e. g., `export LANG=C.UTF-8`) before starting benchmarks. ## Isolating interfering benchmarks@@ -492,11 +493,11 @@ ``` All Fibonacci numbers- fifth: OK (0.44s)+ fifth: OK 53 ns ± 2.7 ns, 8% more than baseline- tenth: OK (0.33s)+ tenth: OK 641 ns ± 59 ns, same as baseline- twentieth: OK (0.36s)+ twentieth: OK 77 μs ± 6.4 μs, 5% less than baseline All 3 tests passed (1.50s)@@ -558,11 +559,11 @@ ``` All Fibonacci numbers- fifth: OK (16.56s)+ fifth: OK 121 ns ± 2.6 ns, 0.08x- tenth: OK (6.84s)+ tenth: OK 1.6 μs ± 31 ns- twentieth: OK (6.96s)+ twentieth: OK 203 μs ± 4.1 μs, 128.36x ```
changelog.md view
@@ -1,3 +1,9 @@+# 0.5++* Extend `TimeMode` with `MutatorCpuTime`, `MutatorWallTime` and `CustomTime`.+* Use a more robust strategy to force encoding to UTF-8.+* Add `Generic` and `Data` instances.+ # 0.4.1 * Force GC before collecting `RTSStats`, otherwise measurements are inaccurate.
src/Test/Tasty/Bench.hs view
@@ -25,23 +25,24 @@ @tasty-bench@ is a native Haskell library and works everywhere, where GHC does, including WASM. We support a full range of architectures-(@i386@, @amd64@, @armhf@, @arm64@, @ppc64le@, @s390x@) and operating-systems (Linux, Windows, macOS, FreeBSD, OpenBSD, NetBSD), plus any GHC-from 8.0 to 9.10 (and earlier releases stretch back to GHC 7.0).+(@i386@, @amd64@, @armhf@, @arm64@, @ppc64le@, @s390x@, @riscv64@) and+operating systems (Linux, Windows, macOS, FreeBSD, OpenBSD, NetBSD),+plus any GHC from 8.0 to 9.14 (and earlier releases stretch back to GHC+7.0). === How is it possible? Our benchmarks are literally regular @tasty@ tests, so we can leverage all existing machinery for command-line options, resource management,-structuring, listing and filtering benchmarks, running and reporting-results. It also means that @tasty-bench@ can be used in conjunction-with other @tasty@ ingredients.+structuring, listing and filtering benchmarks, running them and+reporting results. It also means that @tasty-bench@ can be used in+conjunction with other @tasty@ ingredients. Unlike @criterion@ and @gauge@ we use a very simple statistical model described below. This is arguably a questionable choice, but it works pretty well in practice. A rare developer is sufficiently well-versed in-probability theory to make sense and use of all numbers generated by-@criterion@.+probability theory to make sense and use of all numbers generously+generated by @criterion@. === How to switch? @@ -64,7 +65,8 @@ === How to write a benchmark? -Benchmarks are declared in a separate section of @cabal@ file:+Benchmarks are declared in+<https://cabal.readthedocs.io/en/stable/cabal-package-description-file.html#benchmarks a separate section of cabal file>: > cabal-version: 2.0 > name: bench-fibo@@ -98,31 +100,30 @@ Since @tasty-bench@ provides an API compatible with @criterion@, one can refer to-<http://www.serpentine.com/criterion/tutorial.html#how-to-write-a-benchmark-suite its documentation>+<https://hackage.haskell.org/package/criterion#readme its documentation> for more examples. === How to read results? -Running the example above (@cabal@ @bench@ or @stack@ @bench@) results in+Running the example above (@cabal bench@ or @stack bench@) results in the following output: > All > Fibonacci numbers-> fifth: OK (2.13s)+> fifth: OK > 63 ns ± 3.4 ns-> tenth: OK (1.71s)+> tenth: OK > 809 ns ± 73 ns-> twentieth: OK (3.39s)+> twentieth: OK > 104 μs ± 4.9 μs > > All 3 tests passed (7.25s) -The output says that, for instance, the first benchmark was repeatedly-executed for 2.13 seconds (wall-clock time), its predicted mean CPU time-was 63 nanoseconds and means of individual samples do not often diverge-from it further than ±3.4 nanoseconds (double standard deviation). Take-standard deviation numbers with a grain of salt; there are lies, damned-lies, and statistics.+The output says that, for instance, in the first benchmark the mean CPU+time is 63 nanoseconds and means of individual samples do not often+diverge from it further than ±3.4 nanoseconds (double standard+deviation). Take standard deviation numbers with a grain of salt; there+are lies, damned lies, and statistics. === Wall-clock time vs. CPU time @@ -189,17 +190,17 @@ === Memory usage Configuring RTS to collect GC statistics (e. g., via-@cabal@ @bench@ @--benchmark-options@ @\'+RTS@ @-T\'@ or-@stack@ @bench@ @--ba@ @\'+RTS@ @-T\'@) enables @tasty-bench@ to estimate and+@cabal bench --benchmark-options \'+RTS -T\'@ or+@stack bench --ba \'+RTS -T\'@) enables @tasty-bench@ to estimate and report memory usage: > All > Fibonacci numbers-> fifth: OK (2.13s)+> fifth: OK > 63 ns ± 3.4 ns, 223 B allocated, 0 B copied, 2.0 MB peak memory-> tenth: OK (1.71s)+> tenth: OK > 809 ns ± 73 ns, 2.3 KB allocated, 0 B copied, 4.0 MB peak memory-> twentieth: OK (3.39s)+> twentieth: OK > 104 μs ± 4.9 μs, 277 KB allocated, 59 B copied, 5.0 MB peak memory > > All 3 tests passed (7.25s)@@ -258,9 +259,9 @@ This outputs: > All-> fibo 20: OK (3.02s)+> fibo 20: OK > 104 μs ± 4.9 μs-> myFibo 20: OK (1.99s)+> myFibo 20: OK > 71 μs ± 5.3 μs > myFibo = fibo: FAIL > *** Failed! Falsified (after 5 tests and 1 shrink):@@ -272,7 +273,7 @@ We see that @myFibo@ is indeed significantly faster than @fibo@, but unfortunately does not do the same thing. One should probably look for-another way to speed up generation of Fibonacci numbers.+another way to speed up Fibonacci numbers. === Troubleshooting @@ -280,8 +281,8 @@ of individual benchmarks, and @tasty-bench@ will do its best to fit into a given time frame. Without @--timeout@ we rerun benchmarks until achieving a target precision set by @--stdev@, which in a- noisy environment of a modern laptop with GUI may take a lot of- time.+ noisy environment of a modern laptop with hundreds of service+ processes may take a lot of time. While @criterion@ runs each benchmark at least for 5 seconds, @tasty-bench@ is happy to conclude earlier, if it does not@@ -291,9 +292,9 @@ A common source of noisiness is garbage collection. Setting a larger allocation area (/nursery/) is often a good idea, either via- @cabal@ @bench@ @--benchmark-options@ @\'+RTS@ @-A32m\'@ or- @stack@ @bench@ @--ba@ @\'+RTS@ @-A32m\'@. Alternatively bake it into- @cabal@ file as @ghc-options:@ @\"-with-rtsopts=-A32m\"@.+ @cabal bench --benchmark-options \'+RTS -A32m\'@ or+ @stack bench --ba \'+RTS -A32m\'@. Alternatively bake it into+ @cabal@ file as @ghc-options: \"-with-rtsopts=-A32m\"@. - Never compile benchmarks with @-fstatic-argument-transformation@, because it breaks a trick we use to force GHC into reevaluation of@@ -301,8 +302,8 @@ - If benchmark results look malformed like below, make sure that you are invoking @Test.Tasty.Bench.@'Test.Tasty.Bench.defaultMain' and not- @Test.Tasty.@'Test.Tasty.defaultMain' (the difference is 'consoleBenchReporter'- vs. 'consoleTestReporter'):+ @Test.Tasty.@'Test.Tasty.defaultMain' (the underlying difference is+ 'consoleBenchReporter' vs. 'consoleTestReporter'): > All > fibo 20: OK (1.46s)@@ -316,11 +317,11 @@ > Unexpected state of the resource (NotCreated) in getResource. Report as a tasty bug. - this is likely caused by 'env' or 'envWithCleanup' affecting- benchmarks structure. You can use 'env' to read test data from 'IO',- but not to read benchmark names or affect their hierarchy in other- way. This is a fundamental restriction of @tasty@ to list and filter- benchmarks without launching missiles.+ this is likely caused by 'env' or 'envWithCleanup' affecting the+ structure of benchmarks. You can use 'env' to read test data from+ 'IO', but not to read benchmark names or affect their hierarchy in+ another way. This is a fundamental restriction of @tasty@ so that it+ can list and filter benchmarks without launching missiles. Strict pattern-matching on resource is also prohibited. For instance, if it is a tuple, the second argument of 'env' should use@@ -358,7 +359,7 @@ - If benchmarks using GHC 9.4.4+ segfault on Windows, check that you are not using non-moving garbage collector @--nonmoving-gc@. This is likely caused by- <https://gitlab.haskell.org/ghc/ghc/-/issues/23003 GHC issue>.+ <https://gitlab.haskell.org/ghc/ghc/-/issues/23003 GHC issue #23003>. Previous releases of @tasty-bench@ recommended enabling @--nonmoving-gc@ to stabilise benchmarks, but it’s discouraged now. @@ -372,11 +373,9 @@ > <stdout>: commitBuffer: invalid argument (cannot encode character '\956') it means that your locale does not support UTF-8. @tasty-bench@- makes an effort to force locale to UTF-8, but sometimes, when- benchmarks are a part of a larger application, it’s- <https://gitlab.haskell.org/ghc/ghc/-/issues/23606 impossible> to do- so. In such case run @locale@ @-a@ to list available locales and set a- UTF-8-capable one (e. g., @export@ @LANG=C.UTF-8@) before starting+ makes an effort to force locale to UTF-8, but it’s not bulletproof.+ In such case run @locale -a@ to list available locales and set a+ UTF-8-capable one (e. g., @export LANG=C.UTF-8@) before starting benchmarks. === Isolating interfering benchmarks@@ -397,7 +396,7 @@ and over again. There are several mitigation strategies. First of all, giving garbage-collector more breathing space by @+RTS@ @-A32m@ (or more) is often good+collector more breathing space by @+RTS -A32m@ (or more) is often good enough. Further, avoid using top-level bindings to store large test data. Once@@ -442,7 +441,7 @@ otherwise results could be skewed by intermittent changes in cache-line alignment. -Firstly, run @tasty-bench@ with @--csv@ @FILE@ key to dump results to+Firstly, run @tasty-bench@ with @--csv FILE@ key to dump results to @FILE@ in CSV format (it could be a good idea to set smaller @--stdev@, if possible): @@ -451,16 +450,16 @@ > All.Fibonacci numbers.tenth,637152,46744 > All.Fibonacci numbers.twentieth,81369531,3342646 -Now modify implementation and rerun benchmarks with @--baseline@ @FILE@+Now modify implementation and rerun benchmarks with @--baseline FILE@ key. This produces a report as follows: > All > Fibonacci numbers-> fifth: OK (0.44s)+> fifth: OK > 53 ns ± 2.7 ns, 8% more than baseline-> tenth: OK (0.33s)+> tenth: OK > 641 ns ± 59 ns, same as baseline-> twentieth: OK (0.36s)+> twentieth: OK > 77 μs ± 6.4 μs, 5% less than baseline > > All 3 tests passed (1.50s)@@ -516,11 +515,11 @@ > All > Fibonacci numbers-> fifth: OK (16.56s)+> fifth: OK > 121 ns ± 2.6 ns, 0.08x-> tenth: OK (6.84s)+> tenth: OK > 1.6 μs ± 31 ns-> twentieth: OK (6.96s)+> twentieth: OK > 203 μs ± 4.1 μs, 128.36x To locate a baseline benchmark in a larger suite use 'locateBenchmark'.@@ -534,10 +533,10 @@ === Plotting results -Users can dump results into CSV with @--csv@ @FILE@ and plot them using+Users can dump results into CSV with @--csv FILE@ and plot them using @gnuplot@ or other software. But for convenience there is also a built-in quick-and-dirty SVG plotting feature, which can be invoked by-passing @--svg@ @FILE@. Here is a sample of its output:+passing @--svg FILE@. Here is a sample of its output:  @@ -615,7 +614,7 @@ Whether to measure CPU time (@cpu@, default) or wall-clock time (@wall@). -[@+RTS@ @-T@]:+[@+RTS -T@]: Estimate and report memory usage. @@ -650,16 +649,19 @@ > benchmarks = bgroup "All" [] -}- {-# LANGUAGE BangPatterns #-} {-# LANGUAGE CPP #-}+{-# LANGUAGE DeriveDataTypeable #-} {-# LANGUAGE DeriveFunctor #-}+{-# LANGUAGE DeriveGeneric #-} {-# LANGUAGE FlexibleContexts #-} {-# LANGUAGE FlexibleInstances #-} {-# LANGUAGE GeneralizedNewtypeDeriving #-} {-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TupleSections #-} +{- HLINT ignore "Unused LANGUAGE pragma" -}+ module Test.Tasty.Bench ( #ifdef MIN_VERSION_tasty@@ -674,7 +676,7 @@ , envWithCleanup , #endif- -- * Creating 'Benchmarkable'+ -- * Creating t'Benchmarkable' Benchmarkable(..) , nf , whnf@@ -713,6 +715,7 @@ import Control.DeepSeq (NFData, force, rnf) import Control.Exception (bracket, bracket_, evaluate) import Control.Monad (void, unless, guard, (>=>), when)+import Data.Data (Data) import Data.Foldable (foldMap, traverse_) import Data.Int (Int64) import Data.IORef@@ -723,9 +726,14 @@ import Data.Traversable (forM) import Data.Word (Word64) import GHC.Conc+import GHC.Generics (Generic) import GHC.IO.Encoding import GHC.Stats+#if MIN_VERSION_base(4,15,0)+import GHC.Exts (SPEC(..))+#else import GHC.Types (SPEC(..))+#endif import System.CPUTime import System.Exit import System.IO@@ -784,7 +792,7 @@ -- > import Test.Tasty (localOption) -- > localOption (RelStDev 0.02) (bgroup [...]) ----- If you set 'RelStDev' to infinity,+-- If you set t'RelStDev' to infinity, -- a benchmark will be executed -- only once and its standard deviation will be recorded as zero. -- This is rather a blunt approach, but it might be a necessary evil@@ -804,6 +812,10 @@ -- ^ @since 0.4 , Fractional -- ^ @since 0.4+ , Generic+ -- ^ @since 0.5+ , Data+ -- ^ @since 0.5 ) -- | Whether to measure CPU time or wall-clock time.@@ -833,6 +845,22 @@ -- ^ Measure CPU time. | WallTime -- ^ Measure wall-clock time.+ | MutatorCpuTime+ -- ^ Measure CPU time+ -- excluding garbage collection, known as "mutator time".+ --+ -- @since 0.5+ | MutatorWallTime+ -- ^ Measure wall-clock time+ -- excluding garbage collection, known as "mutator time".+ --+ -- @since 0.5+ | CustomTime (IO Word64)+ -- ^ Custom measurement action, returning time in picoseconds.+ deriving+ ( Generic+ -- ^ @since 0.5+ ) #ifdef MIN_VERSION_tasty instance IsOption RelStDev where@@ -865,6 +893,10 @@ -- ^ @since 0.4 , Fractional -- ^ @since 0.4+ , Generic+ -- ^ @since 0.5+ , Data+ -- ^ @since 0.5 ) instance IsOption FailIfSlower where@@ -897,6 +929,10 @@ -- ^ @since 0.4 , Fractional -- ^ @since 0.4+ , Generic+ -- ^ @since 0.5+ , Data+ -- ^ @since 0.5 ) instance IsOption FailIfFaster where@@ -916,19 +952,24 @@ parseValue v = case v of "cpu" -> Just CpuTime "wall" -> Just WallTime+ "mutcpu" -> Just MutatorCpuTime+ "mutwall" -> Just MutatorWallTime _ -> Nothing optionName = pure "time-mode"- optionHelp = pure "Whether to measure CPU time (\"cpu\") or wall-clock time (\"wall\")"- showDefaultValue m = Just $ case m of- CpuTime -> "cpu"- WallTime -> "wall"+ optionHelp = pure "Whether to measure total CPU time (\"cpu\"), total wall-clock time (\"wall\"), or time spent by the mutator (CPU \"mutcpu\" or wall-clock \"mutwall\")"+ showDefaultValue m = case m of+ CpuTime -> Just "cpu"+ WallTime -> Just "wall"+ MutatorCpuTime -> Just "mutcpu"+ MutatorWallTime -> Just "mutwall"+ CustomTime _ -> Nothing #endif -- | Something that can be benchmarked, produced by 'nf', 'whnf', 'nfIO', 'whnfIO', -- 'nfAppIO', 'whnfAppIO' below. ----- Drop-in replacement for @Criterion.@'Criterion.Benchmarkable' and--- @Gauge.@'Gauge.Benchmarkable'.+-- Drop-in replacement for @Criterion.Benchmarkable@ and+-- @Gauge.Benchmarkable@. -- -- @since 0.1 newtype Benchmarkable =@@ -936,18 +977,22 @@ Benchmarkable { unBenchmarkable :: Word64 -> IO () -- ^ Run benchmark given number of times. }+ deriving+ ( Generic+ -- ^ @since 0.5+ ) #ifdef MIN_VERSION_tasty --- | 'defaultMain' forces 'setLocaleEncoding' to 'utf8', but users might+-- | 'defaultMain' forces encoding to 'utf8', but users might -- be running benchmarks outside of it (e. g., via 'defaultMainWithIngredients'). supportsUnicode :: Bool-supportsUnicode = take 3 (textEncodingName enc) == "UTF"+supportsUnicode = maybe False ((== "UTF") . take 3 . textEncodingName) enc #if defined(mingw32_HOST_OS) && unsafePerformIO getConsoleOutputCP == 65001 #endif where- enc = unsafePerformIO getLocaleEncoding+ enc = unsafePerformIO (hGetEncoding stdout) {-# NOINLINE supportsUnicode #-} mu :: Char@@ -1072,7 +1117,7 @@ where prec = max (fromInteger cpuTimePrecision) 1000000000 -- 1 ms hi meas = meas { measTime = measTime meas + prec }- lo meas = meas { measTime = measTime meas - prec }+ lo meas = meas { measTime = if measTime meas > prec then measTime meas - prec else 0 } hasGCStats :: Bool #if MIN_VERSION_base(4,10,0)@@ -1080,6 +1125,7 @@ #else hasGCStats = unsafePerformIO getGCStatsEnabled #endif+{-# NOINLINE hasGCStats #-} getAllocsAndCopied :: IO (Word64, Word64, Word64) getAllocsAndCopied = do@@ -1097,10 +1143,28 @@ getWallTimeSecs = realToFrac <$> getPOSIXTime #endif +getMutatorCpuTime :: IO Word64+#if MIN_VERSION_base(4,10,0)+getMutatorCpuTime = (1000 *) . fromIntegral . mutator_cpu_ns <$> getRTSStats+#else+getMutatorCpuTime = round . (1e12 *) . mutatorCpuSeconds <$> getGCStats+#endif++getMutatorWallTime :: IO Word64+#if MIN_VERSION_base(4,10,0)+getMutatorWallTime = (1000 *) . fromIntegral . mutator_elapsed_ns <$> getRTSStats+#else+getMutatorWallTime = round . (1e12 *) . mutatorWallSeconds <$> getGCStats+#endif++ getTimePicoSecs :: TimeMode -> IO Word64 getTimePicoSecs timeMode = case timeMode of CpuTime -> fromInteger <$> getCPUTime WallTime -> round . (1e12 *) <$> getWallTimeSecs+ MutatorCpuTime -> getMutatorCpuTime+ MutatorWallTime -> getMutatorWallTime+ CustomTime getCustomTime -> getCustomTime measure :: TimeMode -> Word64 -> Benchmarkable -> IO Measurement measure timeMode n (Benchmarkable act) = do@@ -1170,9 +1234,9 @@ -- | An internal routine to measure CPU execution time in seconds -- for a given timeout (put 'NoTimeout', or 'mkTimeout' 100000000 for 100 seconds) -- and a target relative standard deviation--- (put 'RelStDev' 0.05 for 5% or 'RelStDev' (1/0) to run only one iteration).+-- (put v'RelStDev' 0.05 for 5% or v'RelStDev' (1/0) to run only one iteration). ----- 'Timeout' takes soft priority over 'RelStDev': this function prefers+-- t'Timeout' takes soft priority over t'RelStDev': this function prefers -- to finish in time even if at cost of precision. However, timeout is guidance -- not guarantee: 'measureCpuTime' can take longer, if there is not enough time -- to run at least thrice or an iteration takes unusually long.@@ -1213,11 +1277,11 @@ pure $ testPassed $ show (WithLoHi est (1 - ifFaster) (1 + ifSlower)) _ -> pure $ testFailed "Benchmarks must not be run concurrently. Please pass -j1 and/or avoid +RTS -N." --- | Attach a name to 'Benchmarkable'.+-- | Attach a name to t'Benchmarkable'. -- -- This is actually a synonym of 'Test.Tasty.Providers.singleTest' to--- provide an interface compatible with @Criterion.@'Criterion.bench'--- and @Gauge.@'Gauge.bench'.+-- provide an interface compatible with @Criterion.bench@+-- and @Gauge.bench@. -- -- @since 0.1 bench :: String -> Benchmarkable -> Benchmark@@ -1226,8 +1290,8 @@ -- | Attach a name to a group of 'Benchmark'. -- -- This is actually a synonym of 'Test.Tasty.testGroup' to provide an--- interface compatible with @Criterion.@'Criterion.bgroup' and--- @Gauge@.'Gauge.bgroup'.+-- interface compatible with @Criterion.bgroup@ and+-- @Gauge.bgroup@. -- -- @since 0.1 bgroup :: String -> [Benchmark] -> Benchmark@@ -1257,9 +1321,9 @@ -- -- More complex examples: ----- * https://hackage.haskell.org/package/chimera-0.3.3.0/src/bench/Bench.hs--- * https://hackage.haskell.org/package/fast-digits-0.3.1.0/src/bench/Bench.hs--- * https://hackage.haskell.org/package/unicode-data-0.4.0.1/src/bench/Main.hs+-- * https://hackage.haskell.org/package/chimera-0.4.1.0/src/bench/Read.hs+-- * https://hackage.haskell.org/package/fast-digits-0.3.2.0/src/bench/Bench.hs+-- * https://hackage.haskell.org/package/unicode-data-0.8.0/src/bench/Unicode/Char/Bench.hs -- -- @since 0.2.4 bcompare@@ -1282,6 +1346,11 @@ -- E. g., 'bcompareWithin' 2.0 3.0 passes only if a benchmark is at least 2x -- and at most 3x slower than a baseline. --+-- Examples:+--+-- * https://hackage.haskell.org/package/text-2.1.3/src/benchmarks/haskell/Benchmarks/Micro.hs+-- * https://hackage.haskell.org/package/bluefin-algae-0.1.0.2/src/bench/quadratic-counter.hs+-- -- @since 0.3.1 bcompareWithin :: Double -- ^ Lower bound of relative speed up.@@ -1298,16 +1367,20 @@ -- | Benchmarks are actually just a regular 'Test.Tasty.TestTree' in disguise. ----- This is a drop-in replacement for @Criterion.@'Criterion.Benchmark'--- and @Gauge.@'Gauge.Benchmark'.+-- This is a drop-in replacement for @Criterion.Benchmark@+-- and @Gauge.Benchmark@. -- -- @since 0.1 type Benchmark = TestTree -- | Run benchmarks and report results, providing an interface--- compatible with @Criterion.@'Criterion.defaultMain' and--- @Gauge.@'Gauge.defaultMain'.+-- compatible with @Criterion.defaultMain@ and+-- @Gauge.defaultMain@. --+-- An unfortunate consequence of being a drop-in replacement for @criterion@+-- and @gauge@ is that this function diverges from @tasty@,+-- where 'Test.Tasty.defaultMain' takes a single input, not a list of inputs.+-- -- @since 0.1 defaultMain :: [Benchmark] -> IO () defaultMain bs = do@@ -1316,17 +1389,19 @@ bracketUtf8 :: IO a -> IO a bracketUtf8 act = do- prevLocaleEnc <- getLocaleEncoding+ prevStdoutEnc <- hGetEncoding stdout #if defined(mingw32_HOST_OS) codePage <- getConsoleOutputCP bracket_- (setLocaleEncoding utf8 >> setConsoleOutputCP 65001)- (setLocaleEncoding prevLocaleEnc >> setConsoleOutputCP codePage)+ (hSetEncoding stdout utf8+ >> setConsoleOutputCP 65001)+ (maybe (hSetBinaryMode stdout True) (hSetEncoding stdout) prevStdoutEnc+ >> setConsoleOutputCP codePage) act #else bracket_- (setLocaleEncoding utf8)- (setLocaleEncoding prevLocaleEnc)+ (hSetEncoding stdout utf8)+ (maybe (hSetBinaryMode stdout True) (hSetEncoding stdout) prevStdoutEnc) act #endif @@ -1411,6 +1486,9 @@ -- see 'nf' (@\\n@ @->@ @f@ @n@) @x@ instead of 'nf' @f@ @x@. -- Same applies to rewrite rules. --+-- If you suspect that GHC overoptimizes / overspecializes the function call @f@,+-- try defeating it with 'GHC.Exts.noinline' @f@.+-- -- While @tasty-bench@ is capable to perform micro- and even nanobenchmarks, -- such measurements are noisy and involve an overhead. Results are more reliable -- when @f@ @x@ takes at least several milliseconds.@@ -1429,8 +1507,8 @@ -- then 'nf' @f@ @x@ is ill-typed, but you can use 'nf' @(\\y -> f y)@ @x@ -- instead. ----- Drop-in replacement for @Criterion.@'Criterion.nf' and--- @Gauge.@'Gauge.nf'.+-- Drop-in replacement for @Criterion.nf@ and+-- @Gauge.nf@. -- -- @since 0.1 nf :: NFData b => (a -> b) -> a -> Benchmarkable@@ -1460,7 +1538,7 @@ -- This will succeed in a matter of nanoseconds, because weak head -- normal form forces only the first element of the list. ----- Drop-in replacement for @Criterion.@'Criterion.whnf' and @Gauge.@'Gauge.whnf'.+-- Drop-in replacement for @Criterion.whnf@ and @Gauge.whnf@. -- -- @since 0.1 whnf :: (a -> b) -> a -> Benchmarkable@@ -1490,7 +1568,10 @@ -- with results in nanosecond range. -- -- To avoid surprising results it is usually preferable--- to use 'nfAppIO' instead.+-- to use 'nfAppIO' instead. You can also try turning off+-- let floating by+-- [@-fno-full-laziness@](https://downloads.haskell.org/ghc/latest/docs/users_guide/using-optimisation.html#ghc-flag-ffull-laziness),+-- but this is likely to cause more problems than solve. -- -- Remember that forcing a normal form requires an additional -- traverse of the structure. In certain scenarios,@@ -1502,8 +1583,12 @@ -- but just read input data from a file, it is cleaner to -- use 'env' or 'withResource'. ----- Drop-in replacement for @Criterion.@'Criterion.nfIO' and @Gauge.@'Gauge.nfIO'.+-- One handy consequence of having access to `IO` is that you can+-- generate new random inputs for each run using @System.Random.randomIO@+-- or @System.Random.Stateful.uniformM@ @System.Random.Stateful.globalStdGen@. --+-- Drop-in replacement for @Criterion.nfIO@ and @Gauge.nfIO@.+-- -- @since 0.1 nfIO :: NFData a => IO a -> Benchmarkable nfIO = ioToBench rnf@@ -1519,7 +1604,10 @@ -- with results in nanosecond range. -- -- To avoid surprising results it is usually preferable--- to use 'whnfAppIO' instead.+-- to use 'whnfAppIO' instead. You can also try turning off+-- let floating by+-- [@-fno-full-laziness@](https://downloads.haskell.org/ghc/latest/docs/users_guide/using-optimisation.html#ghc-flag-ffull-laziness),+-- but this is likely to cause more problems than solve. -- -- Computing only a weak head normal form is -- rarely what intuitively is meant by "evaluation".@@ -1531,8 +1619,12 @@ -- but just read input data from a file, it is cleaner to -- use 'env' or 'withResource'. ----- Drop-in replacement for @Criterion.@'Criterion.whnfIO' and @Gauge.@'Gauge.whnfIO'.+-- One handy consequence of having access to `IO` is that you can+-- generate new random inputs for each run using @System.Random.randomIO@+-- or @System.Random.Stateful.uniformM@ @System.Random.Stateful.globalStdGen@. --+-- Drop-in replacement for @Criterion.whnfIO@ and @Gauge.whnfIO@.+-- -- @since 0.1 whnfIO :: IO a -> Benchmarkable whnfIO = ioToBench id@@ -1574,8 +1666,12 @@ -- but just read input data from a file, it is cleaner to -- use 'env' or 'withResource'. ----- Drop-in replacement for @Criterion.@'Criterion.nfAppIO' and @Gauge.@'Gauge.nfAppIO'.+-- One handy consequence of having access to `IO` is that you can+-- generate new random inputs for each run using @System.Random.randomIO@+-- or @System.Random.Stateful.uniformM@ @System.Random.Stateful.globalStdGen@. --+-- Drop-in replacement for @Criterion.nfAppIO@ and @Gauge.nfAppIO@.+-- -- @since 0.1 nfAppIO :: NFData b => (a -> IO b) -> a -> Benchmarkable nfAppIO = ioFuncToBench rnf@@ -1604,8 +1700,12 @@ -- but just read input data from a file, it is cleaner to -- use 'env' or 'withResource'. ----- Drop-in replacement for @Criterion.@'Criterion.whnfAppIO' and @Gauge.@'Gauge.whnfAppIO'.+-- One handy consequence of having access to `IO` is that you can+-- generate new random inputs for each run using @System.Random.randomIO@+-- or @System.Random.Stateful.uniformM@ @System.Random.Stateful.globalStdGen@. --+-- Drop-in replacement for @Criterion.whnfAppIO@ and @Gauge.whnfAppIO@.+-- -- @since 0.1 whnfAppIO :: (a -> IO b) -> a -> Benchmarkable whnfAppIO = ioFuncToBench id@@ -1645,14 +1745,14 @@ -- or unboxed @Vector@ are good, boxed arrays are worse, lists and trees are bad. -- -- @Test.Tasty.Bench.@'env' is provided only for the sake of--- compatibility with @Criterion.@'Criterion.env' and--- @Gauge.@'Gauge.env', and involves 'unsafePerformIO'. Consider using+-- compatibility with @Criterion.env@ and+-- @Gauge.env@, and involves 'unsafePerformIO'. Consider using -- 'withResource' instead. -- -- When working with a mutable environment, bear in mind that it is threaded -- through all iterations of a benchmark. @tasty-bench@ does not roll it back -- or reset, it's user's resposibility. You might have better luck--- with @Criterion.@'Criterion.perBatchEnv' or @Criterion.@'Criterion.perRunEnv'.+-- with @Criterion.perBatchEnv@ or @Criterion.perRunEnv@. -- -- 'defaultMain' requires that the hierarchy of benchmarks and their names is -- independent of underlying 'IO' actions. While executing 'IO' inside 'bench'@@ -1674,8 +1774,8 @@ -- to clean up created environment. -- -- Provided only for the sake of compatibility with--- @Criterion.@'Criterion.envWithCleanup' and--- @Gauge.@'Gauge.envWithCleanup', and involves+-- @Criterion.envWithCleanup@ and+-- @Gauge.envWithCleanup@, and involves -- 'unsafePerformIO'. Consider using 'withResource' instead. -- -- @since 0.2@@ -1717,6 +1817,10 @@ -- ^ @since 0.4 , Ord -- ^ @since 0.4+ , Generic+ -- ^ @since 0.5+ , Data+ -- ^ @since 0.5 ) instance IsOption (Maybe CsvPath) where@@ -1795,6 +1899,10 @@ -- ^ @since 0.4 , Ord -- ^ @since 0.4+ , Generic+ -- ^ @since 0.5+ , Data+ -- ^ @since 0.5 ) instance IsOption (Maybe SvgPath) where@@ -1933,6 +2041,10 @@ -- ^ @since 0.4 , Ord -- ^ @since 0.4+ , Generic+ -- ^ @since 0.5+ , Data+ -- ^ @since 0.5 ) instance IsOption (Maybe BaselinePath) where
tasty-bench.cabal view
@@ -1,5 +1,5 @@ name: tasty-bench-version: 0.4.1+version: 0.5 cabal-version: 1.18 build-type: Simple license: MIT@@ -25,7 +25,7 @@ example.svg README.md -tested-with: GHC == 9.12.1, GHC == 9.10.1, GHC == 9.8.4, GHC == 9.6.6, GHC == 9.4.8, GHC == 9.2.8, GHC==9.0.2, GHC==8.10.7, GHC==8.8.4, GHC==8.6.5, GHC==8.4.4, GHC==8.2.2, GHC==8.0.2+tested-with: GHC == 9.14.1, GHC == 9.12.2, GHC == 9.10.3, GHC == 9.8.4, GHC == 9.6.7, GHC == 9.4.8, GHC == 9.2.8, GHC==9.0.2, GHC==8.10.7, GHC==8.8.4, GHC==8.6.5, GHC==8.4.4, GHC==8.2.2, GHC==8.0.2 source-repository head type: git@@ -45,16 +45,18 @@ ghc-options: -O2 -Wall -fno-warn-unused-imports -Wcompat -Widentities build-depends:- base >= 4.3 && < 5,- deepseq >= 1.1 && < 1.6,- ghc-prim < 0.14+ base >= 4.9 && < 5,+ deepseq >= 1.1 && < 1.6+ if impl(ghc < 9.0)+ build-depends:+ ghc-prim < 0.14 if flag(tasty) build-depends:- containers >= 0.5 && < 0.8,+ containers >= 0.5 && < 0.9, tasty >= 1.4 && < 1.6 if impl(ghc < 8.4) build-depends:- time >= 1.2 && < 1.13+ time >= 1.2 && < 2 benchmark bench-fibo default-language: Haskell2010