diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -35,27 +35,29 @@ So if you already depend on `tasty` for a test suite, there is nothing else to install. -Compare this to `criterion` (10+ modules, 50+ dependencies) and `gauge` (40+ modules, depends on `basement` and `vector`). A build on a clean machine is up to 16x +Compare this to `criterion` (10+ modules, 50+ dependencies) +and `gauge` (40+ modules, depends on `basement` and `vector`). +A build on a clean machine is up to 16x faster than `criterion` and up to 4x faster than `gauge`. A build without dependencies is up to 6x faster than `criterion` and up to 8x faster than `gauge`. `tasty-bench` is a native Haskell library and works everywhere, where GHC does, including WASM. We support a full range of architectures (`i386`, `amd64`, `armhf`, -`arm64`, `ppc64le`, `s390x`) and operating systems (Linux, Windows, macOS, -FreeBSD, OpenBSD, NetBSD), plus any GHC from 8.0 to 9.10 +`arm64`, `ppc64le`, `s390x`, `riscv64`) and operating systems (Linux, Windows, macOS, +FreeBSD, OpenBSD, NetBSD), plus any GHC from 8.0 to 9.14 (and earlier releases stretch back to GHC 7.0). ## How is it possible? Our benchmarks are literally regular `tasty` tests, so we can leverage all existing machinery for command-line options, resource management, structuring, -listing and filtering benchmarks, running and reporting results. It also means +listing and filtering benchmarks, running them and reporting results. It also means that `tasty-bench` can be used in conjunction with other `tasty` ingredients. Unlike `criterion` and `gauge` we use a very simple statistical model described below. This is arguably a questionable choice, but it works pretty well in practice. A rare developer is sufficiently well-versed in probability theory -to make sense and use of all numbers generated by `criterion`. +to make sense and use of all numbers generously generated by `criterion`. ## How to switch? @@ -80,7 +82,8 @@ ## How to write a benchmark? -Benchmarks are declared in a separate section of `cabal` file: +Benchmarks are declared +in [a separate section of `cabal` file](https://cabal.readthedocs.io/en/stable/cabal-package-description-file.html#benchmarks): ```cabal cabal-version: 2.0 @@ -117,7 +120,7 @@ ``` Since `tasty-bench` provides an API compatible with `criterion`, -one can refer to [its documentation](http://www.serpentine.com/criterion/tutorial.html#how-to-write-a-benchmark-suite) for more examples. +one can refer to [its documentation](https://hackage.haskell.org/package/criterion#readme) for more examples. ## How to read results? @@ -127,18 +130,17 @@ ``` All Fibonacci numbers - fifth: OK (2.13s) + fifth: OK 63 ns ± 3.4 ns - tenth: OK (1.71s) + tenth: OK 809 ns ± 73 ns - twentieth: OK (3.39s) + twentieth: OK 104 μs ± 4.9 μs All 3 tests passed (7.25s) ``` -The output says that, for instance, the first benchmark was repeatedly -executed for 2.13 seconds (wall-clock time), its predicted mean CPU time was +The output says that, for instance, in the first benchmark the mean CPU time is 63 nanoseconds and means of individual samples do not often diverge from it further than ±3.4 nanoseconds (double standard deviation). Take standard deviation numbers with a grain of salt; there are lies, damned lies, and @@ -218,11 +220,11 @@ ``` All Fibonacci numbers - fifth: OK (2.13s) + fifth: OK 63 ns ± 3.4 ns, 223 B allocated, 0 B copied, 2.0 MB peak memory - tenth: OK (1.71s) + tenth: OK 809 ns ± 73 ns, 2.3 KB allocated, 0 B copied, 4.0 MB peak memory - twentieth: OK (3.39s) + twentieth: OK 104 μs ± 4.9 μs, 277 KB allocated, 59 B copied, 5.0 MB peak memory All 3 tests passed (7.25s) @@ -281,9 +283,9 @@ ``` All - fibo 20: OK (3.02s) + fibo 20: OK 104 μs ± 4.9 μs - myFibo 20: OK (1.99s) + myFibo 20: OK 71 μs ± 5.3 μs myFibo = fibo: FAIL *** Failed! Falsified (after 5 tests and 1 shrink): @@ -296,7 +298,7 @@ We see that `myFibo` is indeed significantly faster than `fibo`, but unfortunately does not do the same thing. One should probably -look for another way to speed up generation of Fibonacci numbers. +look for another way to speed up Fibonacci numbers. ## Troubleshooting @@ -304,7 +306,7 @@ of individual benchmarks, and `tasty-bench` will do its best to fit into a given time frame. Without `--timeout` we rerun benchmarks until achieving a target precision set by `--stdev`, which in a noisy environment - of a modern laptop with GUI may take a lot of time. + of a modern laptop with hundreds of service processes may take a lot of time. While `criterion` runs each benchmark at least for 5 seconds, `tasty-bench` is happy to conclude earlier, if it does not compromise @@ -324,7 +326,7 @@ * If benchmark results look malformed like below, make sure that you are invoking `Test.Tasty.Bench.defaultMain` and not `Test.Tasty.defaultMain` - (the difference is `consoleBenchReporter` vs. `consoleTestReporter`): + (the underlying difference is `consoleBenchReporter` vs. `consoleTestReporter`): ``` All @@ -344,10 +346,10 @@ Unexpected state of the resource (NotCreated) in getResource. Report as a tasty bug. ``` - this is likely caused by `env` or `envWithCleanup` affecting benchmarks structure. + this is likely caused by `env` or `envWithCleanup` affecting the structure of benchmarks. You can use `env` to read test data from `IO`, but not to read benchmark names - or affect their hierarchy in other way. This is a fundamental restriction of `tasty` - to list and filter benchmarks without launching missiles. + or affect their hierarchy in another way. This is a fundamental restriction of `tasty` + so that it can list and filter benchmarks without launching missiles. Strict pattern-matching on resource is also prohibited. For instance, if it is a tuple, the second argument of `env` should use a lazy pattern match @@ -386,7 +388,7 @@ * If benchmarks using GHC 9.4.4+ segfault on Windows, check that you are not using non-moving garbage collector `--nonmoving-gc`. This is likely caused - by [GHC issue](https://gitlab.haskell.org/ghc/ghc/-/issues/23003). + by [GHC issue #23003](https://gitlab.haskell.org/ghc/ghc/-/issues/23003). Previous releases of `tasty-bench` recommended enabling `--nonmoving-gc` to stabilise benchmarks, but it's discouraged now. @@ -404,9 +406,8 @@ ``` it means that your locale does not support UTF-8. `tasty-bench` makes an effort - to force locale to UTF-8, but sometimes, when benchmarks are a part of - a larger application, it's [impossible](https://gitlab.haskell.org/ghc/ghc/-/issues/23606) - to do so. In such case run `locale -a` to list available locales and set a UTF-8-capable + to force locale to UTF-8, but it's not bulletproof. + In such case run `locale -a` to list available locales and set a UTF-8-capable one (e. g., `export LANG=C.UTF-8`) before starting benchmarks. ## Isolating interfering benchmarks @@ -492,11 +493,11 @@ ``` All Fibonacci numbers - fifth: OK (0.44s) + fifth: OK 53 ns ± 2.7 ns, 8% more than baseline - tenth: OK (0.33s) + tenth: OK 641 ns ± 59 ns, same as baseline - twentieth: OK (0.36s) + twentieth: OK 77 μs ± 6.4 μs, 5% less than baseline All 3 tests passed (1.50s) @@ -558,11 +559,11 @@ ``` All Fibonacci numbers - fifth: OK (16.56s) + fifth: OK 121 ns ± 2.6 ns, 0.08x - tenth: OK (6.84s) + tenth: OK 1.6 μs ± 31 ns - twentieth: OK (6.96s) + twentieth: OK 203 μs ± 4.1 μs, 128.36x ``` diff --git a/changelog.md b/changelog.md --- a/changelog.md +++ b/changelog.md @@ -1,3 +1,16 @@ +# 0.5.1 + +* Add `benchCont` combinator. +* Set default timeout to 100 seconds. +* Use CAPI for foreign calls on Windows. +* Workaround `+cpuTimePrecision` on NetBSD. + +# 0.5 + +* Extend `TimeMode` with `MutatorCpuTime`, `MutatorWallTime` and `CustomTime`. +* Use a more robust strategy to force encoding to UTF-8. +* Add `Generic` and `Data` instances. + # 0.4.1 * Force GC before collecting `RTSStats`, otherwise measurements are inaccurate. diff --git a/src/Test/Tasty/Bench.hs b/src/Test/Tasty/Bench.hs --- a/src/Test/Tasty/Bench.hs +++ b/src/Test/Tasty/Bench.hs @@ -25,23 +25,24 @@ @tasty-bench@ is a native Haskell library and works everywhere, where GHC does, including WASM. We support a full range of architectures -(@i386@, @amd64@, @armhf@, @arm64@, @ppc64le@, @s390x@) and operating -systems (Linux, Windows, macOS, FreeBSD, OpenBSD, NetBSD), plus any GHC -from 8.0 to 9.10 (and earlier releases stretch back to GHC 7.0). +(@i386@, @amd64@, @armhf@, @arm64@, @ppc64le@, @s390x@, @riscv64@) and +operating systems (Linux, Windows, macOS, FreeBSD, OpenBSD, NetBSD), +plus any GHC from 8.0 to 9.14 (and earlier releases stretch back to GHC +7.0). === How is it possible? Our benchmarks are literally regular @tasty@ tests, so we can leverage all existing machinery for command-line options, resource management, -structuring, listing and filtering benchmarks, running and reporting -results. It also means that @tasty-bench@ can be used in conjunction -with other @tasty@ ingredients. +structuring, listing and filtering benchmarks, running them and +reporting results. It also means that @tasty-bench@ can be used in +conjunction with other @tasty@ ingredients. Unlike @criterion@ and @gauge@ we use a very simple statistical model described below. This is arguably a questionable choice, but it works pretty well in practice. A rare developer is sufficiently well-versed in -probability theory to make sense and use of all numbers generated by -@criterion@. +probability theory to make sense and use of all numbers generously +generated by @criterion@. === How to switch? @@ -64,7 +65,8 @@ === How to write a benchmark? -Benchmarks are declared in a separate section of @cabal@ file: +Benchmarks are declared in +: > cabal-version: 2.0 > name: bench-fibo @@ -98,31 +100,30 @@ Since @tasty-bench@ provides an API compatible with @criterion@, one can refer to - + for more examples. === How to read results? -Running the example above (@cabal@ @bench@ or @stack@ @bench@) results in +Running the example above (@cabal bench@ or @stack bench@) results in the following output: > All > Fibonacci numbers -> fifth: OK (2.13s) +> fifth: OK > 63 ns ± 3.4 ns -> tenth: OK (1.71s) +> tenth: OK > 809 ns ± 73 ns -> twentieth: OK (3.39s) +> twentieth: OK > 104 μs ± 4.9 μs > > All 3 tests passed (7.25s) -The output says that, for instance, the first benchmark was repeatedly -executed for 2.13 seconds (wall-clock time), its predicted mean CPU time -was 63 nanoseconds and means of individual samples do not often diverge -from it further than ±3.4 nanoseconds (double standard deviation). Take -standard deviation numbers with a grain of salt; there are lies, damned -lies, and statistics. +The output says that, for instance, in the first benchmark the mean CPU +time is 63 nanoseconds and means of individual samples do not often +diverge from it further than ±3.4 nanoseconds (double standard +deviation). Take standard deviation numbers with a grain of salt; there +are lies, damned lies, and statistics. === Wall-clock time vs. CPU time @@ -189,17 +190,17 @@ === Memory usage Configuring RTS to collect GC statistics (e. g., via -@cabal@ @bench@ @--benchmark-options@ @\'+RTS@ @-T\'@ or -@stack@ @bench@ @--ba@ @\'+RTS@ @-T\'@) enables @tasty-bench@ to estimate and +@cabal bench --benchmark-options \'+RTS -T\'@ or +@stack bench --ba \'+RTS -T\'@) enables @tasty-bench@ to estimate and report memory usage: > All > Fibonacci numbers -> fifth: OK (2.13s) +> fifth: OK > 63 ns ± 3.4 ns, 223 B allocated, 0 B copied, 2.0 MB peak memory -> tenth: OK (1.71s) +> tenth: OK > 809 ns ± 73 ns, 2.3 KB allocated, 0 B copied, 4.0 MB peak memory -> twentieth: OK (3.39s) +> twentieth: OK > 104 μs ± 4.9 μs, 277 KB allocated, 59 B copied, 5.0 MB peak memory > > All 3 tests passed (7.25s) @@ -258,9 +259,9 @@ This outputs: > All -> fibo 20: OK (3.02s) +> fibo 20: OK > 104 μs ± 4.9 μs -> myFibo 20: OK (1.99s) +> myFibo 20: OK > 71 μs ± 5.3 μs > myFibo = fibo: FAIL > *** Failed! Falsified (after 5 tests and 1 shrink): @@ -272,7 +273,7 @@ We see that @myFibo@ is indeed significantly faster than @fibo@, but unfortunately does not do the same thing. One should probably look for -another way to speed up generation of Fibonacci numbers. +another way to speed up Fibonacci numbers. === Troubleshooting @@ -280,8 +281,8 @@ of individual benchmarks, and @tasty-bench@ will do its best to fit into a given time frame. Without @--timeout@ we rerun benchmarks until achieving a target precision set by @--stdev@, which in a - noisy environment of a modern laptop with GUI may take a lot of - time. + noisy environment of a modern laptop with hundreds of service + processes may take a lot of time. While @criterion@ runs each benchmark at least for 5 seconds, @tasty-bench@ is happy to conclude earlier, if it does not @@ -291,9 +292,9 @@ A common source of noisiness is garbage collection. Setting a larger allocation area (/nursery/) is often a good idea, either via - @cabal@ @bench@ @--benchmark-options@ @\'+RTS@ @-A32m\'@ or - @stack@ @bench@ @--ba@ @\'+RTS@ @-A32m\'@. Alternatively bake it into - @cabal@ file as @ghc-options:@ @\"-with-rtsopts=-A32m\"@. + @cabal bench --benchmark-options \'+RTS -A32m\'@ or + @stack bench --ba \'+RTS -A32m\'@. Alternatively bake it into + @cabal@ file as @ghc-options: \"-with-rtsopts=-A32m\"@. - Never compile benchmarks with @-fstatic-argument-transformation@, because it breaks a trick we use to force GHC into reevaluation of @@ -301,8 +302,8 @@ - If benchmark results look malformed like below, make sure that you are invoking @Test.Tasty.Bench.@'Test.Tasty.Bench.defaultMain' and not - @Test.Tasty.@'Test.Tasty.defaultMain' (the difference is 'consoleBenchReporter' - vs. 'consoleTestReporter'): + @Test.Tasty.@'Test.Tasty.defaultMain' (the underlying difference is + 'consoleBenchReporter' vs. 'consoleTestReporter'): > All > fibo 20: OK (1.46s) @@ -316,11 +317,11 @@ > Unexpected state of the resource (NotCreated) in getResource. Report as a tasty bug. - this is likely caused by 'env' or 'envWithCleanup' affecting - benchmarks structure. You can use 'env' to read test data from 'IO', - but not to read benchmark names or affect their hierarchy in other - way. This is a fundamental restriction of @tasty@ to list and filter - benchmarks without launching missiles. + this is likely caused by 'env' or 'envWithCleanup' affecting the + structure of benchmarks. You can use 'env' to read test data from + 'IO', but not to read benchmark names or affect their hierarchy in + another way. This is a fundamental restriction of @tasty@ so that it + can list and filter benchmarks without launching missiles. Strict pattern-matching on resource is also prohibited. For instance, if it is a tuple, the second argument of 'env' should use @@ -358,7 +359,7 @@ - If benchmarks using GHC 9.4.4+ segfault on Windows, check that you are not using non-moving garbage collector @--nonmoving-gc@. This is likely caused by - . + . Previous releases of @tasty-bench@ recommended enabling @--nonmoving-gc@ to stabilise benchmarks, but it’s discouraged now. @@ -372,11 +373,9 @@ > : commitBuffer: invalid argument (cannot encode character '\956') it means that your locale does not support UTF-8. @tasty-bench@ - makes an effort to force locale to UTF-8, but sometimes, when - benchmarks are a part of a larger application, it’s - to do - so. In such case run @locale@ @-a@ to list available locales and set a - UTF-8-capable one (e. g., @export@ @LANG=C.UTF-8@) before starting + makes an effort to force locale to UTF-8, but it’s not bulletproof. + In such case run @locale -a@ to list available locales and set a + UTF-8-capable one (e. g., @export LANG=C.UTF-8@) before starting benchmarks. === Isolating interfering benchmarks @@ -397,7 +396,7 @@ and over again. There are several mitigation strategies. First of all, giving garbage -collector more breathing space by @+RTS@ @-A32m@ (or more) is often good +collector more breathing space by @+RTS -A32m@ (or more) is often good enough. Further, avoid using top-level bindings to store large test data. Once @@ -442,7 +441,7 @@ otherwise results could be skewed by intermittent changes in cache-line alignment. -Firstly, run @tasty-bench@ with @--csv@ @FILE@ key to dump results to +Firstly, run @tasty-bench@ with @--csv FILE@ key to dump results to @FILE@ in CSV format (it could be a good idea to set smaller @--stdev@, if possible): @@ -451,16 +450,16 @@ > All.Fibonacci numbers.tenth,637152,46744 > All.Fibonacci numbers.twentieth,81369531,3342646 -Now modify implementation and rerun benchmarks with @--baseline@ @FILE@ +Now modify implementation and rerun benchmarks with @--baseline FILE@ key. This produces a report as follows: > All > Fibonacci numbers -> fifth: OK (0.44s) +> fifth: OK > 53 ns ± 2.7 ns, 8% more than baseline -> tenth: OK (0.33s) +> tenth: OK > 641 ns ± 59 ns, same as baseline -> twentieth: OK (0.36s) +> twentieth: OK > 77 μs ± 6.4 μs, 5% less than baseline > > All 3 tests passed (1.50s) @@ -516,11 +515,11 @@ > All > Fibonacci numbers -> fifth: OK (16.56s) +> fifth: OK > 121 ns ± 2.6 ns, 0.08x -> tenth: OK (6.84s) +> tenth: OK > 1.6 μs ± 31 ns -> twentieth: OK (6.96s) +> twentieth: OK > 203 μs ± 4.1 μs, 128.36x To locate a baseline benchmark in a larger suite use 'locateBenchmark'. @@ -534,10 +533,10 @@ === Plotting results -Users can dump results into CSV with @--csv@ @FILE@ and plot them using +Users can dump results into CSV with @--csv FILE@ and plot them using @gnuplot@ or other software. But for convenience there is also a built-in quick-and-dirty SVG plotting feature, which can be invoked by -passing @--svg@ @FILE@. Here is a sample of its output: +passing @--svg FILE@. Here is a sample of its output: ![Plotting](example.svg) @@ -615,7 +614,7 @@ Whether to measure CPU time (@cpu@, default) or wall-clock time (@wall@). -[@+RTS@ @-T@]: +[@+RTS -T@]: Estimate and report memory usage. @@ -650,16 +649,20 @@ > benchmarks = bgroup "All" [] -} - {-# LANGUAGE BangPatterns #-} +{-# LANGUAGE CApiFFI #-} {-# LANGUAGE CPP #-} +{-# LANGUAGE DeriveDataTypeable #-} {-# LANGUAGE DeriveFunctor #-} +{-# LANGUAGE DeriveGeneric #-} {-# LANGUAGE FlexibleContexts #-} {-# LANGUAGE FlexibleInstances #-} {-# LANGUAGE GeneralizedNewtypeDeriving #-} {-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TupleSections #-} +{- HLINT ignore "Unused LANGUAGE pragma" -} + module Test.Tasty.Bench ( #ifdef MIN_VERSION_tasty @@ -668,13 +671,14 @@ , Benchmark , bench , bgroup + , benchCont , bcompare , bcompareWithin , env , envWithCleanup , #endif - -- * Creating 'Benchmarkable' + -- * Creating t'Benchmarkable' Benchmarkable(..) , nf , whnf @@ -712,7 +716,9 @@ import Control.Arrow (first, second) import Control.DeepSeq (NFData, force, rnf) import Control.Exception (bracket, bracket_, evaluate) -import Control.Monad (void, unless, guard, (>=>), when) +import Control.Monad (void, unless, guard, join, (>=>), when) +import Control.Monad.Trans.Cont +import Data.Data (Data) import Data.Foldable (foldMap, traverse_) import Data.Int (Int64) import Data.IORef @@ -723,9 +729,14 @@ import Data.Traversable (forM) import Data.Word (Word64) import GHC.Conc +import GHC.Generics (Generic) import GHC.IO.Encoding import GHC.Stats +#if MIN_VERSION_base(4,15,0) +import GHC.Exts (SPEC(..)) +#else import GHC.Types (SPEC(..)) +#endif import System.CPUTime import System.Exit import System.IO @@ -784,7 +795,7 @@ -- > import Test.Tasty (localOption) -- > localOption (RelStDev 0.02) (bgroup [...]) -- --- If you set 'RelStDev' to infinity, +-- If you set t'RelStDev' to infinity, -- a benchmark will be executed -- only once and its standard deviation will be recorded as zero. -- This is rather a blunt approach, but it might be a necessary evil @@ -804,6 +815,10 @@ -- ^ @since 0.4 , Fractional -- ^ @since 0.4 + , Generic + -- ^ @since 0.5 + , Data + -- ^ @since 0.5 ) -- | Whether to measure CPU time or wall-clock time. @@ -833,6 +848,22 @@ -- ^ Measure CPU time. | WallTime -- ^ Measure wall-clock time. + | MutatorCpuTime + -- ^ Measure CPU time + -- excluding garbage collection, known as "mutator time". + -- + -- @since 0.5 + | MutatorWallTime + -- ^ Measure wall-clock time + -- excluding garbage collection, known as "mutator time". + -- + -- @since 0.5 + | CustomTime (IO Word64) + -- ^ Custom measurement action, returning time in picoseconds. + deriving + ( Generic + -- ^ @since 0.5 + ) #ifdef MIN_VERSION_tasty instance IsOption RelStDev where @@ -865,6 +896,10 @@ -- ^ @since 0.4 , Fractional -- ^ @since 0.4 + , Generic + -- ^ @since 0.5 + , Data + -- ^ @since 0.5 ) instance IsOption FailIfSlower where @@ -897,6 +932,10 @@ -- ^ @since 0.4 , Fractional -- ^ @since 0.4 + , Generic + -- ^ @since 0.5 + , Data + -- ^ @since 0.5 ) instance IsOption FailIfFaster where @@ -916,19 +955,24 @@ parseValue v = case v of "cpu" -> Just CpuTime "wall" -> Just WallTime + "mutcpu" -> Just MutatorCpuTime + "mutwall" -> Just MutatorWallTime _ -> Nothing optionName = pure "time-mode" - optionHelp = pure "Whether to measure CPU time (\"cpu\") or wall-clock time (\"wall\")" - showDefaultValue m = Just $ case m of - CpuTime -> "cpu" - WallTime -> "wall" + optionHelp = pure "Whether to measure total CPU time (\"cpu\"), total wall-clock time (\"wall\"), or time spent by the mutator (CPU \"mutcpu\" or wall-clock \"mutwall\")" + showDefaultValue m = case m of + CpuTime -> Just "cpu" + WallTime -> Just "wall" + MutatorCpuTime -> Just "mutcpu" + MutatorWallTime -> Just "mutwall" + CustomTime _ -> Nothing #endif -- | Something that can be benchmarked, produced by 'nf', 'whnf', 'nfIO', 'whnfIO', -- 'nfAppIO', 'whnfAppIO' below. -- --- Drop-in replacement for @Criterion.@'Criterion.Benchmarkable' and --- @Gauge.@'Gauge.Benchmarkable'. +-- Drop-in replacement for @Criterion.Benchmarkable@ and +-- @Gauge.Benchmarkable@. -- -- @since 0.1 newtype Benchmarkable = @@ -936,18 +980,22 @@ Benchmarkable { unBenchmarkable :: Word64 -> IO () -- ^ Run benchmark given number of times. } + deriving + ( Generic + -- ^ @since 0.5 + ) #ifdef MIN_VERSION_tasty --- | 'defaultMain' forces 'setLocaleEncoding' to 'utf8', but users might +-- | 'defaultMain' forces encoding to 'utf8', but users might -- be running benchmarks outside of it (e. g., via 'defaultMainWithIngredients'). supportsUnicode :: Bool -supportsUnicode = take 3 (textEncodingName enc) == "UTF" +supportsUnicode = maybe False ((== "UTF") . take 3 . textEncodingName) enc #if defined(mingw32_HOST_OS) && unsafePerformIO getConsoleOutputCP == 65001 #endif where - enc = unsafePerformIO getLocaleEncoding + enc = unsafePerformIO (hGetEncoding stdout) {-# NOINLINE supportsUnicode #-} mu :: Char @@ -1070,16 +1118,26 @@ (estStdev (predict (hi t1) (lo t2))) } where - prec = max (fromInteger cpuTimePrecision) 1000000000 -- 1 ms + prec = max (fromInteger cpuTimePrecision') 1000000000 -- 1 ms hi meas = meas { measTime = measTime meas + prec } - lo meas = meas { measTime = measTime meas - prec } + lo meas = meas { measTime = if measTime meas > prec then measTime meas - prec else 0 } +#ifdef netbsd_HOST_OS +-- cpuTimePrecision seems to fail on NetBSD +cpuTimePrecision' :: Prelude.Integer +cpuTimePrecision' = 0 +#else +cpuTimePrecision' :: Prelude.Integer +cpuTimePrecision' = cpuTimePrecision +#endif + hasGCStats :: Bool #if MIN_VERSION_base(4,10,0) hasGCStats = unsafePerformIO getRTSStatsEnabled #else hasGCStats = unsafePerformIO getGCStatsEnabled #endif +{-# NOINLINE hasGCStats #-} getAllocsAndCopied :: IO (Word64, Word64, Word64) getAllocsAndCopied = do @@ -1097,10 +1155,28 @@ getWallTimeSecs = realToFrac <$> getPOSIXTime #endif +getMutatorCpuTime :: IO Word64 +#if MIN_VERSION_base(4,10,0) +getMutatorCpuTime = (1000 *) . fromIntegral . mutator_cpu_ns <$> getRTSStats +#else +getMutatorCpuTime = round . (1e12 *) . mutatorCpuSeconds <$> getGCStats +#endif + +getMutatorWallTime :: IO Word64 +#if MIN_VERSION_base(4,10,0) +getMutatorWallTime = (1000 *) . fromIntegral . mutator_elapsed_ns <$> getRTSStats +#else +getMutatorWallTime = round . (1e12 *) . mutatorWallSeconds <$> getGCStats +#endif + + getTimePicoSecs :: TimeMode -> IO Word64 getTimePicoSecs timeMode = case timeMode of CpuTime -> fromInteger <$> getCPUTime WallTime -> round . (1e12 *) <$> getWallTimeSecs + MutatorCpuTime -> getMutatorCpuTime + MutatorWallTime -> getMutatorWallTime + CustomTime getCustomTime -> getCustomTime measure :: TimeMode -> Word64 -> Benchmarkable -> IO Measurement measure timeMode n (Benchmarkable act) = do @@ -1170,9 +1246,9 @@ -- | An internal routine to measure CPU execution time in seconds -- for a given timeout (put 'NoTimeout', or 'mkTimeout' 100000000 for 100 seconds) -- and a target relative standard deviation --- (put 'RelStDev' 0.05 for 5% or 'RelStDev' (1/0) to run only one iteration). +-- (put v'RelStDev' 0.05 for 5% or v'RelStDev' (1/0) to run only one iteration). -- --- 'Timeout' takes soft priority over 'RelStDev': this function prefers +-- t'Timeout' takes soft priority over t'RelStDev': this function prefers -- to finish in time even if at cost of precision. However, timeout is guidance -- not guarantee: 'measureCpuTime' can take longer, if there is not enough time -- to run at least thrice or an iteration takes unusually long. @@ -1196,14 +1272,7 @@ #ifdef MIN_VERSION_tasty instance IsTest Benchmarkable where - testOptions = pure - [ Option (Proxy :: Proxy RelStDev) - -- FailIfSlower and FailIfFaster must be options of a test provider rather - -- than options of an ingredient to allow setting them on per-test level. - , Option (Proxy :: Proxy FailIfSlower) - , Option (Proxy :: Proxy FailIfFaster) - , Option (Proxy :: Proxy TimeMode) - ] + testOptions = pure benchOptions run opts b yieldProgress = case getNumThreads (lookupOption opts) of 1 -> do let timeMode = lookupOption opts @@ -1213,11 +1282,21 @@ pure $ testPassed $ show (WithLoHi est (1 - ifFaster) (1 + ifSlower)) _ -> pure $ testFailed "Benchmarks must not be run concurrently. Please pass -j1 and/or avoid +RTS -N." --- | Attach a name to 'Benchmarkable'. +benchOptions :: [OptionDescription] +benchOptions = + [ Option (Proxy :: Proxy RelStDev) + -- FailIfSlower and FailIfFaster must be options of a test provider rather + -- than options of an ingredient to allow setting them on per-test level. + , Option (Proxy :: Proxy FailIfSlower) + , Option (Proxy :: Proxy FailIfFaster) + , Option (Proxy :: Proxy TimeMode) + ] + +-- | Attach a name to t'Benchmarkable'. -- -- This is actually a synonym of 'Test.Tasty.Providers.singleTest' to --- provide an interface compatible with @Criterion.@'Criterion.bench' --- and @Gauge.@'Gauge.bench'. +-- provide an interface compatible with @Criterion.bench@ +-- and @Gauge.bench@. -- -- @since 0.1 bench :: String -> Benchmarkable -> Benchmark @@ -1226,13 +1305,70 @@ -- | Attach a name to a group of 'Benchmark'. -- -- This is actually a synonym of 'Test.Tasty.testGroup' to provide an --- interface compatible with @Criterion.@'Criterion.bgroup' and --- @Gauge@.'Gauge.bgroup'. +-- interface compatible with @Criterion.bgroup@ and +-- @Gauge.bgroup@. -- -- @since 0.1 bgroup :: String -> [Benchmark] -> Benchmark bgroup = testGroup +-- | Run a t'Benchmarkable' inside some larger action. +-- +-- Prefer using 'env' or 'withResource' to avoid benchmarking setup/teardown code. +-- But this isn't always possible, like when using a library that only +-- exposes resources through `bracket`-style continuation passing. +-- Imagine an alternate reality where 'withBinaryFile' can not be +-- split into 'openFile' and 'hClose': +-- +-- > -- Broken: +-- > withResource +-- > (withBinaryFile "NO.txt" WriteMode $ pure) +-- > (pure . const ()) +-- > benchmarkWrites +-- +-- This benchmark writes to a closed file handle, which will not go well. +-- Instead, @benchCont@ allows you to embed a t'Benchmarkable' +-- in a continuation. As a trivial example, +-- +-- > main :: IO () +-- > main = defaultMain +-- > [ benchCont "write syscall" $ ContT benchmarkWrites ] +-- > +-- > benchmarkWrites :: (Benchmarkable -> IO ()) -> IO () +-- > benchmarkWrites runBenchmark = withBinaryFile "/dev/null" WriteMode $ \fh -> do +-- > hSetBuffering fh NoBuffering +-- > runBenchmark $ whnfAppIO putHello fh +-- > -- equivalently, +-- > -- runBenchmark $ whnfIO (putHello fh) +-- > +-- > putHello :: Handle -> IO () +-- > putHello h = hPutStrLn h "hi" +-- +-- Calling the runner more than once is unspecified behavior. +-- Create a separate 'Benchmark' instead. +-- +-- @since 0.5.1 +benchCont :: String -> ContT () IO Benchmarkable -> Benchmark +benchCont = singleTest + +#if !MIN_VERSION_tasty(1,5,4) +instance IsTest (ContT () IO Benchmarkable) where + testOptions = pure benchOptions + run opts (ContT bracketedBenchmark) yieldProgress = do + rr <- newIORef Nothing + let run' :: Benchmarkable -> IO () + run' b = do + r <- run opts b yieldProgress + join . atomicModifyIORef' rr $ \prev -> case prev of + Nothing -> (Just r, pure ()) + p -> (p, error "Benchmarkable called multiple times") + bracketedBenchmark run' + maybeRes <- readIORef rr + pure $ case maybeRes of + Nothing -> testFailed "Provided ContT didn't run the passed Benchmarkable" + Just r -> r +#endif + -- | Compare benchmarks, reporting relative speed up or slow down. -- -- This function is a vague reminiscence of @bcompare@, which existed in pre-1.0 @@ -1257,9 +1393,9 @@ -- -- More complex examples: -- --- * https://hackage.haskell.org/package/chimera-0.3.3.0/src/bench/Bench.hs --- * https://hackage.haskell.org/package/fast-digits-0.3.1.0/src/bench/Bench.hs --- * https://hackage.haskell.org/package/unicode-data-0.4.0.1/src/bench/Main.hs +-- * https://hackage.haskell.org/package/chimera-0.4.1.0/src/bench/Read.hs +-- * https://hackage.haskell.org/package/fast-digits-0.3.2.0/src/bench/Bench.hs +-- * https://hackage.haskell.org/package/unicode-data-0.8.0/src/bench/Unicode/Char/Bench.hs -- -- @since 0.2.4 bcompare @@ -1282,13 +1418,24 @@ -- E. g., 'bcompareWithin' 2.0 3.0 passes only if a benchmark is at least 2x -- and at most 3x slower than a baseline. -- +-- Examples: +-- +-- * https://hackage.haskell.org/package/text-2.1.3/src/benchmarks/haskell/Benchmarks/Micro.hs +-- * https://hackage.haskell.org/package/bluefin-algae-0.1.0.2/src/bench/quadratic-counter.hs +-- -- @since 0.3.1 bcompareWithin - :: Double -- ^ Lower bound of relative speed up. - -> Double -- ^ Upper bound of relative speed up. - -> String -- ^ @tasty@ pattern to locate a baseline benchmark. - -> Benchmark -- ^ Benchmark to compare against baseline. + :: Double + -- ^ Lower bound of relative speed up. + -> Double + -- ^ Upper bound of relative speed up. + -> String + -- ^ @tasty@ pattern, which must unambiguously + -- match a unique baseline benchmark. Consider using 'locateBenchmark' to construct it. -> Benchmark + -- ^ Benchmark + -- to be compared against the baseline benchmark by dividing measured mean times. + -> Benchmark bcompareWithin lo hi s = case parseExpr s of Nothing -> error $ "Could not parse bcompare pattern " ++ s Just e -> after_ AllSucceed (And (StringLit (bcomparePrefix ++ show (lo, hi))) e) @@ -1298,49 +1445,56 @@ -- | Benchmarks are actually just a regular 'Test.Tasty.TestTree' in disguise. -- --- This is a drop-in replacement for @Criterion.@'Criterion.Benchmark' --- and @Gauge.@'Gauge.Benchmark'. +-- This is a drop-in replacement for @Criterion.Benchmark@ +-- and @Gauge.Benchmark@. -- -- @since 0.1 type Benchmark = TestTree -- | Run benchmarks and report results, providing an interface --- compatible with @Criterion.@'Criterion.defaultMain' and --- @Gauge.@'Gauge.defaultMain'. +-- compatible with @Criterion.defaultMain@ and +-- @Gauge.defaultMain@. -- +-- An unfortunate consequence of being a drop-in replacement for @criterion@ +-- and @gauge@ is that this function diverges from @tasty@, +-- where 'Test.Tasty.defaultMain' takes a single input, not a list of inputs. +-- -- @since 0.1 defaultMain :: [Benchmark] -> IO () defaultMain bs = do let act = defaultMain' bs bracketUtf8 act +withDefaultTimeout :: TestTree -> TestTree +withDefaultTimeout = adjustOption $ \opt -> case opt of + Timeout{} -> opt + NoTimeout -> mkTimeout 100000000 + bracketUtf8 :: IO a -> IO a bracketUtf8 act = do - prevLocaleEnc <- getLocaleEncoding + prevStdoutEnc <- hGetEncoding stdout #if defined(mingw32_HOST_OS) codePage <- getConsoleOutputCP bracket_ - (setLocaleEncoding utf8 >> setConsoleOutputCP 65001) - (setLocaleEncoding prevLocaleEnc >> setConsoleOutputCP codePage) + (hSetEncoding stdout utf8 + >> setConsoleOutputCP 65001) + (maybe (hSetBinaryMode stdout True) (hSetEncoding stdout) prevStdoutEnc + >> setConsoleOutputCP codePage) act #else bracket_ - (setLocaleEncoding utf8) - (setLocaleEncoding prevLocaleEnc) + (hSetEncoding stdout utf8) + (maybe (hSetBinaryMode stdout True) (hSetEncoding stdout) prevStdoutEnc) act #endif defaultMain' :: [Benchmark] -> IO () defaultMain' bs = do installSignalHandlers - let b = testGroup "All" bs + let b = withDefaultTimeout $ testGroup "All" bs opts <- parseOptions benchIngredients b let opts' = setOption (NumThreads 1) opts -#if MIN_VERSION_tasty(1,5,0) opts'' = setOption (MinDurationToReport 1000000000000) opts' -#else - opts'' = opts' -#endif case tryIngredients benchIngredients opts'' b of Nothing -> exitFailure Just act -> act >>= \x -> if x then exitSuccess else exitFailure @@ -1411,6 +1565,9 @@ -- see 'nf' (@\\n@ @->@ @f@ @n@) @x@ instead of 'nf' @f@ @x@. -- Same applies to rewrite rules. -- +-- If you suspect that GHC overoptimizes / overspecializes the function call @f@, +-- try defeating it with 'GHC.Exts.noinline' @f@. +-- -- While @tasty-bench@ is capable to perform micro- and even nanobenchmarks, -- such measurements are noisy and involve an overhead. Results are more reliable -- when @f@ @x@ takes at least several milliseconds. @@ -1429,8 +1586,8 @@ -- then 'nf' @f@ @x@ is ill-typed, but you can use 'nf' @(\\y -> f y)@ @x@ -- instead. -- --- Drop-in replacement for @Criterion.@'Criterion.nf' and --- @Gauge.@'Gauge.nf'. +-- Drop-in replacement for @Criterion.nf@ and +-- @Gauge.nf@. -- -- @since 0.1 nf :: NFData b => (a -> b) -> a -> Benchmarkable @@ -1460,7 +1617,7 @@ -- This will succeed in a matter of nanoseconds, because weak head -- normal form forces only the first element of the list. -- --- Drop-in replacement for @Criterion.@'Criterion.whnf' and @Gauge.@'Gauge.whnf'. +-- Drop-in replacement for @Criterion.whnf@ and @Gauge.whnf@. -- -- @since 0.1 whnf :: (a -> b) -> a -> Benchmarkable @@ -1490,7 +1647,10 @@ -- with results in nanosecond range. -- -- To avoid surprising results it is usually preferable --- to use 'nfAppIO' instead. +-- to use 'nfAppIO' instead. You can also try turning off +-- let floating by +-- [@-fno-full-laziness@](https://downloads.haskell.org/ghc/latest/docs/users_guide/using-optimisation.html#ghc-flag-ffull-laziness), +-- but this is likely to cause more problems than solve. -- -- Remember that forcing a normal form requires an additional -- traverse of the structure. In certain scenarios, @@ -1502,8 +1662,12 @@ -- but just read input data from a file, it is cleaner to -- use 'env' or 'withResource'. -- --- Drop-in replacement for @Criterion.@'Criterion.nfIO' and @Gauge.@'Gauge.nfIO'. +-- One handy consequence of having access to `IO` is that you can +-- generate new random inputs for each run using @System.Random.randomIO@ +-- or @System.Random.Stateful.uniformM@ @System.Random.Stateful.globalStdGen@. -- +-- Drop-in replacement for @Criterion.nfIO@ and @Gauge.nfIO@. +-- -- @since 0.1 nfIO :: NFData a => IO a -> Benchmarkable nfIO = ioToBench rnf @@ -1519,7 +1683,10 @@ -- with results in nanosecond range. -- -- To avoid surprising results it is usually preferable --- to use 'whnfAppIO' instead. +-- to use 'whnfAppIO' instead. You can also try turning off +-- let floating by +-- [@-fno-full-laziness@](https://downloads.haskell.org/ghc/latest/docs/users_guide/using-optimisation.html#ghc-flag-ffull-laziness), +-- but this is likely to cause more problems than solve. -- -- Computing only a weak head normal form is -- rarely what intuitively is meant by "evaluation". @@ -1531,8 +1698,12 @@ -- but just read input data from a file, it is cleaner to -- use 'env' or 'withResource'. -- --- Drop-in replacement for @Criterion.@'Criterion.whnfIO' and @Gauge.@'Gauge.whnfIO'. +-- One handy consequence of having access to `IO` is that you can +-- generate new random inputs for each run using @System.Random.randomIO@ +-- or @System.Random.Stateful.uniformM@ @System.Random.Stateful.globalStdGen@. -- +-- Drop-in replacement for @Criterion.whnfIO@ and @Gauge.whnfIO@. +-- -- @since 0.1 whnfIO :: IO a -> Benchmarkable whnfIO = ioToBench id @@ -1574,8 +1745,12 @@ -- but just read input data from a file, it is cleaner to -- use 'env' or 'withResource'. -- --- Drop-in replacement for @Criterion.@'Criterion.nfAppIO' and @Gauge.@'Gauge.nfAppIO'. +-- One handy consequence of having access to `IO` is that you can +-- generate new random inputs for each run using @System.Random.randomIO@ +-- or @System.Random.Stateful.uniformM@ @System.Random.Stateful.globalStdGen@. -- +-- Drop-in replacement for @Criterion.nfAppIO@ and @Gauge.nfAppIO@. +-- -- @since 0.1 nfAppIO :: NFData b => (a -> IO b) -> a -> Benchmarkable nfAppIO = ioFuncToBench rnf @@ -1604,8 +1779,12 @@ -- but just read input data from a file, it is cleaner to -- use 'env' or 'withResource'. -- --- Drop-in replacement for @Criterion.@'Criterion.whnfAppIO' and @Gauge.@'Gauge.whnfAppIO'. +-- One handy consequence of having access to `IO` is that you can +-- generate new random inputs for each run using @System.Random.randomIO@ +-- or @System.Random.Stateful.uniformM@ @System.Random.Stateful.globalStdGen@. -- +-- Drop-in replacement for @Criterion.whnfAppIO@ and @Gauge.whnfAppIO@. +-- -- @since 0.1 whnfAppIO :: (a -> IO b) -> a -> Benchmarkable whnfAppIO = ioFuncToBench id @@ -1645,14 +1824,14 @@ -- or unboxed @Vector@ are good, boxed arrays are worse, lists and trees are bad. -- -- @Test.Tasty.Bench.@'env' is provided only for the sake of --- compatibility with @Criterion.@'Criterion.env' and --- @Gauge.@'Gauge.env', and involves 'unsafePerformIO'. Consider using +-- compatibility with @Criterion.env@ and +-- @Gauge.env@, and involves 'unsafePerformIO'. Consider using -- 'withResource' instead. -- -- When working with a mutable environment, bear in mind that it is threaded -- through all iterations of a benchmark. @tasty-bench@ does not roll it back -- or reset, it's user's resposibility. You might have better luck --- with @Criterion.@'Criterion.perBatchEnv' or @Criterion.@'Criterion.perRunEnv'. +-- with @Criterion.perBatchEnv@ or @Criterion.perRunEnv@. -- -- 'defaultMain' requires that the hierarchy of benchmarks and their names is -- independent of underlying 'IO' actions. While executing 'IO' inside 'bench' @@ -1674,8 +1853,8 @@ -- to clean up created environment. -- -- Provided only for the sake of compatibility with --- @Criterion.@'Criterion.envWithCleanup' and --- @Gauge.@'Gauge.envWithCleanup', and involves +-- @Criterion.envWithCleanup@ and +-- @Gauge.envWithCleanup@, and involves -- 'unsafePerformIO'. Consider using 'withResource' instead. -- -- @since 0.2 @@ -1717,6 +1896,10 @@ -- ^ @since 0.4 , Ord -- ^ @since 0.4 + , Generic + -- ^ @since 0.5 + , Data + -- ^ @since 0.5 ) instance IsOption (Maybe CsvPath) where @@ -1795,6 +1978,10 @@ -- ^ @since 0.4 , Ord -- ^ @since 0.4 + , Generic + -- ^ @since 0.5 + , Data + -- ^ @since 0.5 ) instance IsOption (Maybe SvgPath) where @@ -1933,6 +2120,10 @@ -- ^ @since 0.4 , Ord -- ^ @since 0.4 + , Generic + -- ^ @since 0.5 + , Data + -- ^ @since 0.5 ) instance IsOption (Maybe BaselinePath) where @@ -2064,21 +2255,13 @@ testNameSeqs :: OptionSet -> TestTree -> [Seq TestName] testNameSeqs = foldTestTree trivialFold { foldSingle = const $ const . (:[]) . Seq.singleton -#if MIN_VERSION_tasty(1,5,0) , foldGroup = const $ (. concat) . map . (<|) -#else - , foldGroup = const $ map . (<|) -#endif } testNamesAndDeps :: IntMap (Seq TestName) -> OptionSet -> TestTree -> [(TestName, Unique (WithLoHi IM.Key))] testNamesAndDeps im = foldTestTree trivialFold { foldSingle = const $ const . (: []) . (, mempty) -#if MIN_VERSION_tasty(1,5,0) , foldGroup = const $ (. concat) . map . first . (++) . (++ ".") -#else - , foldGroup = const $ map . first . (++) . (++ ".") -#endif , foldAfter = const foldDeps } where @@ -2124,7 +2307,6 @@ writeTVar oldTV (Done (f name depRes res)) pure (Any True, All True) -#if MIN_VERSION_tasty(1,5,0) Executing newProgr -> do let updated = case old of Executing oldProgr -> oldProgr /= newProgr @@ -2132,9 +2314,6 @@ when updated $ writeTVar oldTV (Executing newProgr) pure (Any updated, All False) -#else - Executing{} -> pure (Any False, All False) -#endif NotStarted -> pure (Any False, All False) if anyUpdated || allDone then pure allDone else retry adNauseam = doUpdate >>= (`unless` adNauseam) @@ -2160,14 +2339,8 @@ #if defined(mingw32_HOST_OS) -#if defined(i386_HOST_ARCH) -#define CCONV stdcall -#else -#define CCONV ccall -#endif - -foreign import CCONV unsafe "windows.h GetConsoleOutputCP" getConsoleOutputCP :: IO Word32 -foreign import CCONV unsafe "windows.h SetConsoleOutputCP" setConsoleOutputCP :: Word32 -> IO () +foreign import capi unsafe "windows.h GetConsoleOutputCP" getConsoleOutputCP :: IO Word32 +foreign import capi unsafe "windows.h SetConsoleOutputCP" setConsoleOutputCP :: Word32 -> IO () #endif diff --git a/tasty-bench.cabal b/tasty-bench.cabal --- a/tasty-bench.cabal +++ b/tasty-bench.cabal @@ -1,5 +1,5 @@ name: tasty-bench -version: 0.4.1 +version: 0.5.1 cabal-version: 1.18 build-type: Simple license: MIT @@ -25,7 +25,7 @@ example.svg README.md -tested-with: GHC == 9.12.1, GHC == 9.10.1, GHC == 9.8.4, GHC == 9.6.6, GHC == 9.4.8, GHC == 9.2.8, GHC==9.0.2, GHC==8.10.7, GHC==8.8.4, GHC==8.6.5, GHC==8.4.4, GHC==8.2.2, GHC==8.0.2 +tested-with: GHC == 9.14.1, GHC == 9.12.2, GHC == 9.10.3, GHC == 9.8.4, GHC == 9.6.7, GHC == 9.4.8, GHC == 9.2.8, GHC==9.0.2, GHC==8.10.7, GHC==8.8.4, GHC==8.6.5, GHC==8.4.4, GHC==8.2.2, GHC==8.0.2 source-repository head type: git @@ -45,16 +45,19 @@ ghc-options: -O2 -Wall -fno-warn-unused-imports -Wcompat -Widentities build-depends: - base >= 4.3 && < 5, + base >= 4.9 && < 5, deepseq >= 1.1 && < 1.6, - ghc-prim < 0.14 + transformers >= 0.4 && < 0.7 + if impl(ghc < 9.0) + build-depends: + ghc-prim < 0.14 if flag(tasty) build-depends: - containers >= 0.5 && < 0.8, - tasty >= 1.4 && < 1.6 + containers >= 0.5 && < 0.9, + tasty >= 1.5 && < 1.6 if impl(ghc < 8.4) build-depends: - time >= 1.2 && < 1.13 + time >= 1.2 && < 2 benchmark bench-fibo default-language: Haskell2010