tasty-bench 0.3.5 → 0.4
raw patch · 5 files changed
+225/−103 lines, 5 filesdep +timedep ~containersdep ~tasty
Dependencies added: time
Dependency ranges changed: containers, tasty
Files
- README.md +27/−9
- changelog.md +16/−0
- compare_benches.sh +5/−1
- src/Test/Tasty/Bench.hs +169/−79
- tasty-bench.cabal +8/−14
README.md view
@@ -42,7 +42,7 @@ `tasty-bench` is a native Haskell library and works everywhere, where GHC does, including WASM. We support a full range of architectures (`i386`, `amd64`, `armhf`, `arm64`, `ppc64le`, `s390x`) and operating systems (Linux, Windows, macOS,-FreeBSD, OpenBSD, NetBSD), plus any GHC from 7.0 to 9.6.+FreeBSD, OpenBSD, NetBSD), plus any GHC from 7.0 to 9.10. ## How is it possible? @@ -227,9 +227,27 @@ All 3 tests passed (7.25s) ``` -This data is reported as per `RTSStats` fields: `allocated_bytes`, `copied_bytes`-and `max_mem_in_use_bytes`.+This data is reported as per [`GHC.Stats.RTSStats`](https://hackage.haskell.org/package/base/docs/GHC-Stats.html#t:RTSStats) fields: +* `allocated_bytes`++ Total size of data ever allocated since the start+ of the benchmark iteration. Even if data was immediately+ garbage collected and freed, it still counts.++* `copied_bytes`++ Total size of data ever copied by GC (because it was alive and kicking)+ since the start of the benchmark iteration. Note that zero bytes often mean+ that the benchmark was too short to trigger GC at all.++* `max_mem_in_use_bytes`++ Peak size of live data since the very start of the process.+ This is a global metric, it cumulatively grows and does not say much+ about individual benchmarks, but rather characterizes heap+ environment in which they are executed.+ ## Combining tests and benchmarks When optimizing an existing function, it is important to check that its@@ -310,7 +328,7 @@ ``` All fibo 20: OK (1.46s)- Response {respEstimate = Estimate {estMean = Measurement {measTime = 87496728, measAllocs = 0, measCopied = 0}, estStdev = 694487}, respIfSlower = FailIfSlower Infinity, respIfFaster = FailIfFaster Infinity}+ WithLoHi (Estimate {estMean = Measurement {measTime = 41529118775, measAllocs = 0, measCopied = 0, measMaxMem = 0}, estStdev = 1595055320}) (-Infinity) Infinity ``` * If benchmarks fail with an error message@@ -330,6 +348,10 @@ or affect their hierarchy in other way. This is a fundamental restriction of `tasty` to list and filter benchmarks without launching missiles. + Strict pattern-matching on resource is also prohibited. For instance,+ if it is a tuple, the second argument of `env` should use a lazy pattern match+ `\~(a, b) -> ...`+ * If benchmarks fail with `Test dependencies form a loop` or `Test dependencies have cycles`, this is likely because of `bcompare`, which compares a benchmark with itself.@@ -484,7 +506,7 @@ If you wish to compare two CSV reports non-interactively, here is a handy `awk` incantation: ```sh-awk 'BEGIN{FS=",";OFS=",";print "Name,Old,New,Ratio"}FNR==1{trueNF=NF;next}NF<trueNF{print "Benchmark names should not contain newlines";exit 1}FNR==NR{oldTime=$(NF-trueNF+2);NF-=trueNF-1;a[$0]=oldTime;next}{newTime=$(NF-trueNF+2);NF-=trueNF-1;print $0,a[$0],newTime,newTime/a[$0];gs+=log(newTime/a[$0]);gc++}END{if(gc>0)print "Geometric mean,,",exp(gs/gc)}' old.csv new.csv+awk 'BEGIN{FS=",";OFS=",";print "Name,Old,New,Ratio"}FNR==1{trueNF=NF;next}NF<trueNF{print "Benchmark names should not contain newlines";exit 1}FNR==NR{oldTime=$(NF-trueNF+2);NF-=trueNF-1;a[$0]=oldTime;next}{newTime=$(NF-trueNF+2);NF-=trueNF-1;if(a[$0]){print $0,a[$0],newTime,newTime/a[$0];gs+=log(newTime/a[$0]);gc++}}END{if(gc>0)print "Geometric mean,,",exp(gs/gc)}' old.csv new.csv ``` A larger shell snippet to compare two `git` commits can be found in `compare_benches.sh`.@@ -563,10 +585,6 @@ It will provide you with functions to build `Benchmarkable` and run them manually via `measureCpuTime`. This mode of operation can be also configured by disabling Cabal flag `tasty`.--* If results are amiss or oscillate wildly and adjusting `--timeout` and `--stdev`- does not help, you may be interested to investigate individual timings of- successive runs by enabling Cabal flag `debug`. This will pipe raw data into `stderr`. ## Command-line options
changelog.md view
@@ -1,3 +1,19 @@+# 0.4++* Switch `nf`, `nfIO` and `nfAppIO` to evaluate outputs to a normal form+ with `rnf` instead of `force`. It means that parts of the output, which have+ already been forced, can be garbage collected early, without waiting for+ the entire output to be allocated at the same time. This decreases+ benchmarking overhead in many scenarios and brings the behaviour in line+ with `criterion`. See [#39](https://github.com/Bodigrim/tasty-bench/issues/39)+ for discussion.+* Drop support of `tasty < 1.4`.+* Make `IO` benchmarks immune to `-fspec-constr-count` limit.+* Decomission `debug` build flag.+* Decomission warning when `--timeout` is absent.+* Add `instance {Eq,Ord,Num,Fractional} {RelStDev,FailIfSlower,FailIfFaster}`.+* Add `instance {Eq,Ord} {CsvPath,SvgPath,BaselinePath}`.+ # 0.3.5 * Support `tasty-1.5`.
compare_benches.sh view
@@ -18,10 +18,14 @@ OLDVSNEWCSV=$(echo "$OLD"-vs-"$NEW".csv | sed -e s#/##g) git checkout -q "$OLDREF" && \+ trap 'git checkout -q "@{-1}" && trap - INT' INT && \ cabal run -v0 benchmarks -- --csv "$OLDCSV" "$@" && \ git checkout -q "$NEWREF" && \+ trap 'git checkout -q "@{-2}" && trap - INT' INT && \ cabal run -v0 benchmarks -- --baseline "$OLDCSV" --csv "$NEWCSV" "$@" && \ git checkout -q "@{-2}" && \ - awk 'BEGIN{FS=",";OFS=",";print "Name,'"$OLD"','"$NEW"',Ratio"}FNR==1{trueNF=NF;next}NF<trueNF{print "Benchmark names should not contain newlines";exit 1}FNR==NR{oldTime=$(NF-trueNF+2);NF-=trueNF-1;a[$0]=oldTime;next}{newTime=$(NF-trueNF+2);NF-=trueNF-1;print $0,a[$0],newTime,newTime/a[$0];gs+=log(newTime/a[$0]);gc++}END{if(gc>0)print "Geometric mean,,",exp(gs/gc)}' "$OLDCSV" "$NEWCSV" > "$OLDVSNEWCSV"+ awk 'BEGIN{FS=",";OFS=",";print "Name,'"$OLD"','"$NEW"',Ratio"}FNR==1{trueNF=NF;next}NF<trueNF{print "Benchmark names should not contain newlines";exit 1}FNR==NR{oldTime=$(NF-trueNF+2);NF-=trueNF-1;a[$0]=oldTime;next}{newTime=$(NF-trueNF+2);NF-=trueNF-1;if(a[$0]){print $0,a[$0],newTime,newTime/a[$0];gs+=log(newTime/a[$0]);gc++}}END{if(gc>0)print "Geometric mean,,",exp(gs/gc)}' "$OLDCSV" "$NEWCSV" > "$OLDVSNEWCSV" && \++ trap - INT }
src/Test/Tasty/Bench.hs view
@@ -27,7 +27,7 @@ GHC does, including WASM. We support a full range of architectures (@i386@, @amd64@, @armhf@, @arm64@, @ppc64le@, @s390x@) and operating systems (Linux, Windows, macOS, FreeBSD, OpenBSD, NetBSD), plus any GHC-from 7.0 to 9.6.+from 7.0 to 9.10. === How is it possible? @@ -204,9 +204,30 @@ > > All 3 tests passed (7.25s) -This data is reported as per 'RTSStats' fields: 'allocated_bytes',-'copied_bytes' and 'max_mem_in_use_bytes'.+This data is reported as per+<https://hackage.haskell.org/package/base/docs/GHC-Stats.html#t:RTSStats GHC.Stats.RTSStats>+fields: +- 'allocated_bytes'++ Total size of data ever allocated since the start of the benchmark+ iteration. Even if data was immediately garbage collected and freed,+ it still counts.++- 'copied_bytes'++ Total size of data ever copied by GC (because it was alive and+ kicking) since the start of the benchmark iteration. Note that zero+ bytes often mean that the benchmark was too short to trigger GC at+ all.++- 'max_mem_in_use_bytes'++ Peak size of live data since the very start of the process. This is+ a global metric, it cumulatively grows and does not say much about+ individual benchmarks, but rather characterizes heap environment in+ which they are executed.+ === Combining tests and benchmarks When optimizing an existing function, it is important to check that its@@ -285,7 +306,7 @@ > All > fibo 20: OK (1.46s)- > Response {respEstimate = Estimate {estMean = Measurement {measTime = 87496728, measAllocs = 0, measCopied = 0}, estStdev = 694487}, respIfSlower = FailIfSlower Infinity, respIfFaster = FailIfFaster Infinity}+ > WithLoHi (Estimate {estMean = Measurement {measTime = 41529118775, measAllocs = 0, measCopied = 0, measMaxMem = 0}, estStdev = 1595055320}) (-Infinity) Infinity - If benchmarks fail with an error message @@ -301,6 +322,10 @@ way. This is a fundamental restriction of @tasty@ to list and filter benchmarks without launching missiles. + Strict pattern-matching on resource is also prohibited. For+ instance, if it is a tuple, the second argument of 'env' should use+ a lazy pattern match @\\~(a, b) -> ...@+ - If benchmarks fail with @Test dependencies form a loop@ or @Test dependencies have cycles@, this is likely because of 'bcompare', which compares a benchmark with itself. Locating a@@ -447,7 +472,7 @@ If you wish to compare two CSV reports non-interactively, here is a handy @awk@ incantation: -> awk 'BEGIN{FS=",";OFS=",";print "Name,Old,New,Ratio"}FNR==1{trueNF=NF;next}NF<trueNF{print "Benchmark names should not contain newlines";exit 1}FNR==NR{oldTime=$(NF-trueNF+2);NF-=trueNF-1;a[$0]=oldTime;next}{newTime=$(NF-trueNF+2);NF-=trueNF-1;print $0,a[$0],newTime,newTime/a[$0];gs+=log(newTime/a[$0]);gc++}END{if(gc>0)print "Geometric mean,,",exp(gs/gc)}' old.csv new.csv+> awk 'BEGIN{FS=",";OFS=",";print "Name,Old,New,Ratio"}FNR==1{trueNF=NF;next}NF<trueNF{print "Benchmark names should not contain newlines";exit 1}FNR==NR{oldTime=$(NF-trueNF+2);NF-=trueNF-1;a[$0]=oldTime;next}{newTime=$(NF-trueNF+2);NF-=trueNF-1;if(a[$0]){print $0,a[$0],newTime,newTime/a[$0];gs+=log(newTime/a[$0]);gc++}}END{if(gc>0)print "Geometric mean,,",exp(gs/gc)}' old.csv new.csv A larger shell snippet to compare two @git@ commits can be found in @compare_benches.sh@.@@ -525,11 +550,6 @@ 'measureCpuTime'. This mode of operation can be also configured by disabling Cabal flag @tasty@. -- If results are amiss or oscillate wildly and adjusting @--timeout@- and @--stdev@ does not help, you may be interested to investigate- individual timings of successive runs by enabling Cabal flag- @debug@. This will pipe raw data into @stderr@.- === Command-line options Use @--help@ to list all command-line options.@@ -632,6 +652,7 @@ {-# LANGUAGE DeriveFunctor #-} {-# LANGUAGE FlexibleContexts #-} {-# LANGUAGE FlexibleInstances #-}+{-# LANGUAGE GeneralizedNewtypeDeriving #-} {-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TupleSections #-} @@ -685,7 +706,7 @@ import qualified Prelude import Control.Applicative import Control.Arrow (first, second)-import Control.DeepSeq (NFData, force)+import Control.DeepSeq (NFData, force, rnf) import Control.Exception (bracket, evaluate) import Control.Monad (void, unless, guard, (>=>), when) import Data.Data (Typeable)@@ -712,10 +733,6 @@ import System.Mem import Text.Printf -#ifdef DEBUG-import Debug.Trace-#endif- #ifdef MIN_VERSION_tasty #if !MIN_VERSION_base(4,8,0) import Data.Monoid (Monoid(..))@@ -744,6 +761,12 @@ import Test.Tasty.Runners #endif +#if MIN_VERSION_base(4,11,0)+import GHC.Clock (getMonotonicTime)+#else+import Data.Time.Clock.POSIX (getPOSIXTime)+#endif+ #if defined(mingw32_HOST_OS) import Data.Word (Word32) #endif@@ -788,7 +811,19 @@ -- -- @since 0.2 newtype RelStDev = RelStDev Double- deriving (Show, Read, Typeable)+ deriving+ ( Eq+ -- ^ @since 0.4+ , Ord+ -- ^ @since 0.4+ , Show+ , Read+ , Num+ -- ^ @since 0.4+ , Fractional+ -- ^ @since 0.4+ , Typeable+ ) -- | Whether to measure CPU time or wall-clock time. -- Normally 'CpuTime' is a better option (and default),@@ -802,15 +837,21 @@ -- > import Test.Tasty (localOption) -- > localOption WallTime (bgroup [...]) ----- section of your cabal file.+-- You can measure both times and report their ratio with the following gadget: --+-- @+-- bgroup \"Foo\"+-- [ localOption WallTime $ bench \"WallTime\" foo+-- , bcompare \"Foo.WallTime\"+-- $ localOption CpuTime $ bench \"CPUTime\" foo+-- ]+-- @+-- -- @since 0.3.2 data TimeMode = CpuTime -- ^ Measure CPU time.-#ifdef MIN_VERSION_tasty | WallTime -- ^ Measure wall-clock time.-#endif deriving (Typeable) #ifdef MIN_VERSION_tasty@@ -833,7 +874,19 @@ -- -- @since 0.2 newtype FailIfSlower = FailIfSlower Double- deriving (Show, Read, Typeable)+ deriving+ ( Eq+ -- ^ @since 0.4+ , Ord+ -- ^ @since 0.4+ , Show+ , Read+ , Num+ -- ^ @since 0.4+ , Fractional+ -- ^ @since 0.4+ , Typeable+ ) instance IsOption FailIfSlower where defaultValue = FailIfSlower (1.0 / 0.0)@@ -854,7 +907,19 @@ -- -- @since 0.2 newtype FailIfFaster = FailIfFaster Double- deriving (Show, Read, Typeable)+ deriving+ ( Eq+ -- ^ @since 0.4+ , Ord+ -- ^ @since 0.4+ , Show+ , Read+ , Num+ -- ^ @since 0.4+ , Fractional+ -- ^ @since 0.4+ , Typeable+ ) instance IsOption FailIfFaster where defaultValue = FailIfFaster (1.0 / 0.0)@@ -876,12 +941,10 @@ _ -> Nothing optionName = pure "time-mode" optionHelp = pure "Whether to measure CPU time (\"cpu\") or wall-clock time (\"wall\")"-#if MIN_VERSION_tasty(1,3,0) showDefaultValue m = Just $ case m of CpuTime -> "cpu" WallTime -> "wall" #endif-#endif -- | Something that can be benchmarked, produced by 'nf', 'whnf', 'nfIO', 'whnfIO', -- 'nfAppIO', 'whnfAppIO' below.@@ -1057,12 +1120,17 @@ pure (0, 0, 0) #endif +getWallTimeSecs :: IO Double+#if MIN_VERSION_base(4,11,0)+getWallTimeSecs = getMonotonicTime+#else+getWallTimeSecs = realToFrac <$> getPOSIXTime+#endif+ getTimePicoSecs :: TimeMode -> IO Word64 getTimePicoSecs timeMode = case timeMode of CpuTime -> fromInteger <$> getCPUTime-#ifdef MIN_VERSION_tasty- WallTime -> round . (1e12 *) <$> getTime-#endif+ WallTime -> round . (1e12 *) <$> getWallTimeSecs measure :: TimeMode -> Word64 -> Benchmarkable -> IO Measurement measure timeMode n (Benchmarkable act) = do@@ -1079,25 +1147,20 @@ , measCopied = endCopied - startCopied , measMaxMem = max endMaxMemInUse startMaxMemInUse }-#ifdef DEBUG- pure $ trace (show n ++ (if n == 1 then " iteration gives " else " iterations give ") ++ show meas) meas-#else pure meas-#endif measureUntil :: (Progress -> IO ()) -> TimeMode- -> Bool -> Timeout -> RelStDev -> Benchmarkable -> IO Estimate-measureUntil _ timeMode _ _ (RelStDev targetRelStDev) b+measureUntil _ timeMode _ (RelStDev targetRelStDev) b | isInfinite targetRelStDev, targetRelStDev > 0 = do t1 <- measure timeMode 1 b pure $ Estimate { estMean = t1, estStdev = 0 }-measureUntil yieldProgress timeMode warnIfNoTimeout timeout (RelStDev targetRelStDev) b = do+measureUntil yieldProgress timeMode timeout (RelStDev targetRelStDev) b = do t1 <- measure' 1 b go 1 t1 0 where@@ -1129,11 +1192,6 @@ yieldProgress () #endif - case timeout of- NoTimeout | warnIfNoTimeout, sumOfTs' + measTime t2 > 100 * 1000000000000- -> hPutStrLn stderr "This benchmark takes more than 100 seconds. Consider setting --timeout, if this is unexpected (or to silence this warning)."- _ -> pure ()- if isStDevInTargetRange || isTimeoutSoon then pure scaledEstimate else go (2 * n) t2 sumOfTs'@@ -1162,7 +1220,7 @@ ( word64ToDouble (measTime (estMean x)) / 1e12 , word64ToDouble (estStdev x) / 1e12 )) .) .)- . measureUntil (const $ pure ()) CpuTime False+ . measureUntil (const $ pure ()) CpuTime #ifdef MIN_VERSION_tasty @@ -1178,7 +1236,7 @@ run opts b yieldProgress = case getNumThreads (lookupOption opts) of 1 -> do let timeMode = lookupOption opts- est <- measureUntil yieldProgress timeMode True (lookupOption opts) (lookupOption opts) b+ est <- measureUntil yieldProgress timeMode (lookupOption opts) (lookupOption opts) b let FailIfSlower ifSlower = lookupOption opts FailIfFaster ifFaster = lookupOption opts pure $ testPassed $ show (WithLoHi est (1 - ifFaster) (1 + ifSlower))@@ -1317,7 +1375,7 @@ #endif funcToBench :: forall a b c. (b -> c) -> (a -> b) -> a -> Benchmarkable-funcToBench frc = (Benchmarkable .) . benchLoop SPEC+funcToBench frc = (Benchmarkable .) . funcToBenchLoop SPEC where -- Here we rely on the fact that GHC (unless spurred by -- -fstatic-argument-transformation) is not smart enough:@@ -1328,25 +1386,25 @@ -- -- For perspective, gauge and criterion < 1.4 mark similar functions as INLINE, -- while criterion >= 1.4 switches to NOINLINE.- -- If we mark `benchLoop` NOINLINE then benchmark results are slightly larger+ -- If we mark `funcToBenchLoop` NOINLINE then benchmark results are slightly larger -- (noticeable in bench-fibo), because the loop body is slightly bigger, -- since GHC does not unbox numbers or inline `Eq @Word64` dictionary. --- -- This function is called `benchLoop` instead of, say, `go`,+ -- This function is called `funcToBenchLoop` instead of, say, `go`, -- so it is easier to spot in Core dumps. -- -- Forcing SpecConst optimization with SPEC makes the behaviour of benchmarks -- independent of -fspec-constr-count.- benchLoop :: SPEC -> (a -> b) -> a -> Word64 -> IO ()- benchLoop !_ f x n+ funcToBenchLoop :: SPEC -> (a -> b) -> a -> Word64 -> IO ()+ funcToBenchLoop !_ f x n | n == 0 = pure () | otherwise = do _ <- evaluate (frc (f x))- benchLoop SPEC f x (n - 1)+ funcToBenchLoop SPEC f x (n - 1) {-# INLINE funcToBench #-} -- | 'nf' @f@ @x@ measures time to compute--- a normal form (by means of 'force', not 'Control.DeepSeq.rnf')+-- a normal form (by means of 'Control.DeepSeq.rnf', not 'Control.DeepSeq.force') -- of an application of @f@ to @x@. -- This does not include time to evaluate @f@ or @x@ themselves. -- Ideally @x@ should be a primitive data type like 'Data.Int.Int'.@@ -1364,6 +1422,10 @@ -- on garbage collector. Also no list fusion will happen. -- A better approach is 'nf' (@\\n@ @->@ 'sum' @[1..n]@) @1000000@. --+-- It is preferable that the return type of the function under measurement+-- is inhabited enough to depend genuinely on all computations and is not simply @b ~ ()@.+-- Otherwise GHC might get aggressive and optimise the payload away.+-- -- If you are measuring an inlinable function, -- it is prudent to ensure that its invocation is fully saturated, -- otherwise inlining will not happen. That's why one can often@@ -1379,9 +1441,9 @@ -- especially when 'NFData' instance is badly written, -- this traversal may take non-negligible time and affect results. ----- 'nf' @f@ is equivalent to 'whnf' ('force' '.' @f@), but not to--- 'whnf' ('Control.DeepSeq.rnf' '.' @f@). The former retains the result--- in memory until it is fully evaluated, while the latter allows+-- 'nf' @f@ is equivalent to 'whnf' ('Control.DeepSeq.rnf' '.' @f@), but not to+-- 'whnf' ('Control.DeepSeq.force' '.' @f@). The latter retains the result+-- in memory until it is fully evaluated, while the former allows -- evaluated parts of the result to be garbage-collected immediately. -- -- For users of @{-# LANGUAGE LinearTypes #-}@: if @f@ is a linear function,@@ -1393,7 +1455,7 @@ -- -- @since 0.1 nf :: NFData b => (a -> b) -> a -> Benchmarkable-nf = funcToBench force+nf = funcToBench rnf {-# INLINE nf #-} -- | 'whnf' @f@ @x@ measures time to compute@@ -1427,21 +1489,27 @@ {-# INLINE whnf #-} ioToBench :: (b -> c) -> IO b -> Benchmarkable-ioToBench frc act = Benchmarkable go+ioToBench frc act = Benchmarkable (ioToBenchLoop SPEC) where- go n+ ioToBenchLoop :: SPEC -> Word64 -> IO ()+ ioToBenchLoop !_ n | n == 0 = pure () | otherwise = do val <- act _ <- evaluate (frc val)- go (n - 1)+ ioToBenchLoop SPEC (n - 1) {-# INLINE ioToBench #-} -- | 'nfIO' @x@ measures time to evaluate side-effects of @x@--- and compute its normal form (by means of 'force', not 'Control.DeepSeq.rnf').+-- and compute its normal form+-- (by means of 'Control.DeepSeq.rnf', not 'Control.DeepSeq.force'). -- -- Pure subexpression of an effectful computation @x@--- may be evaluated only once and get cached.+-- may be evaluated only once and get cached. For example,+-- GHC is likely to float @x@ out of 'nfIO' ('pure' @x@) and+-- evaluate in only once, which leaves 'nfIO' to measure 'pure' only+-- with results in nanosecond range.+-- -- To avoid surprising results it is usually preferable -- to use 'nfAppIO' instead. --@@ -1459,14 +1527,18 @@ -- -- @since 0.1 nfIO :: NFData a => IO a -> Benchmarkable-nfIO = ioToBench force+nfIO = ioToBench rnf {-# INLINE nfIO #-} -- | 'whnfIO' @x@ measures time to evaluate side-effects of @x@ -- and compute its weak head normal form. -- -- Pure subexpression of an effectful computation @x@--- may be evaluated only once and get cached.+-- may be evaluated only once and get cached. For example,+-- GHC is likely to float @x@ out of 'whnfIO' ('pure' @x@) and+-- evaluate in only once, which leaves 'whnfIO' to measure 'pure' only+-- with results in nanosecond range.+-- -- To avoid surprising results it is usually preferable -- to use 'whnfAppIO' instead. --@@ -1487,20 +1559,22 @@ whnfIO = ioToBench id {-# INLINE whnfIO #-} -ioFuncToBench :: (b -> c) -> (a -> IO b) -> a -> Benchmarkable-ioFuncToBench frc = (Benchmarkable .) . go+ioFuncToBench :: forall a b c. (b -> c) -> (a -> IO b) -> a -> Benchmarkable+ioFuncToBench frc = (Benchmarkable .) . ioFuncToBenchLoop SPEC where- go f x n+ ioFuncToBenchLoop :: SPEC -> (a -> IO b) -> a -> Word64 -> IO ()+ ioFuncToBenchLoop !_ f x n | n == 0 = pure () | otherwise = do val <- f x _ <- evaluate (frc val)- go f x (n - 1)+ ioFuncToBenchLoop SPEC f x (n - 1) {-# INLINE ioFuncToBench #-} -- | 'nfAppIO' @f@ @x@ measures time to evaluate side-effects of -- an application of @f@ to @x@--- and compute its normal form (by means of 'force', not 'Control.DeepSeq.rnf').+-- and compute its normal form+-- (by means of 'Control.DeepSeq.rnf', not 'Control.DeepSeq.force'). -- This does not include time to evaluate @f@ or @x@ themselves. -- Ideally @x@ should be a primitive data type like 'Data.Int.Int'. --@@ -1525,7 +1599,7 @@ -- -- @since 0.1 nfAppIO :: NFData b => (a -> IO b) -> a -> Benchmarkable-nfAppIO = ioFuncToBench force+nfAppIO = ioFuncToBench rnf {-# INLINE nfAppIO #-} -- | 'whnfAppIO' @f@ @x@ measures time to evaluate side-effects of@@ -1582,14 +1656,15 @@ -- -- use ----- > import Control.DeepSeq (force)--- > import Control.Exception (evaluate)--- > -- > main :: IO () -- > main = defaultMain--- > [ env (evaluate (force (replicate 1000000 'a'))) $ \largeData ->+-- > [ env (pure (replicate 1000000 'a')) $ \largeData -> -- > bench "large" $ nf length largeData, ... ] --+-- Even with 'env', it's advisable to store input data in as few heap objects+-- as possible. 'Data.Array.ByteArray.ByteArray' (ideally pinned)+-- or unboxed @Vector@ are good, boxed arrays are worse, lists and trees are bad.+-- -- @Test.Tasty.Bench.@'env' is provided only for the sake of -- compatibility with @Criterion.@'Criterion.env' and -- @Gauge.@'Gauge.env', and involves 'unsafePerformIO'. Consider using@@ -1608,6 +1683,10 @@ -- -- > Unhandled resource. Probably a bug in the runner you're using. --+-- Strict pattern-matching on resource is also prohibited. For+-- instance, if it is a tuple, the second argument of 'env' should use+-- a lazy pattern match @\\~(a, b) -> ...@+-- -- @since 0.2 env :: NFData env => IO env -> (env -> Benchmark) -> Benchmark env res = envWithCleanup res (const $ pure ())@@ -1654,7 +1733,13 @@ -- -- @since 0.3 newtype CsvPath = CsvPath FilePath- deriving (Typeable)+ deriving+ ( Eq+ -- ^ @since 0.4+ , Ord+ -- ^ @since 0.4+ , Typeable+ ) instance IsOption (Maybe CsvPath) where defaultValue = Nothing@@ -1727,7 +1812,13 @@ -- -- @since 0.3 newtype SvgPath = SvgPath FilePath- deriving (Typeable)+ deriving+ ( Eq+ -- ^ @since 0.4+ , Ord+ -- ^ @since 0.4+ , Typeable+ ) instance IsOption (Maybe SvgPath) where defaultValue = Nothing@@ -1860,7 +1951,13 @@ -- -- @since 0.3 newtype BaselinePath = BaselinePath FilePath- deriving (Typeable)+ deriving+ ( Eq+ -- ^ @since 0.4+ , Ord+ -- ^ @since 0.4+ , Typeable+ ) instance IsOption (Maybe BaselinePath) where defaultValue = Nothing@@ -2002,27 +2099,20 @@ { foldSingle = const $ const . (:[]) . Seq.singleton #if MIN_VERSION_tasty(1,5,0) , foldGroup = const $ (. concat) . map . (<|)-#elif MIN_VERSION_tasty(1,4,0)- , foldGroup = const $ map . (<|) #else- , foldGroup = map . (<|)+ , foldGroup = const $ map . (<|) #endif } testNamesAndDeps :: IntMap (Seq TestName) -> OptionSet -> TestTree -> [(TestName, Unique (WithLoHi IM.Key))] testNamesAndDeps im = foldTestTree trivialFold { foldSingle = const $ const . (: []) . (, mempty)-#if MIN_VERSION_tasty(1,4,0) #if MIN_VERSION_tasty(1,5,0) , foldGroup = const $ (. concat) . map . first . (++) . (++ ".") #else , foldGroup = const $ map . first . (++) . (++ ".") #endif , foldAfter = const foldDeps-#else- , foldGroup = map . first . (++) . (++ ".")- , foldAfter = foldDeps-#endif } where foldDeps :: DependencyType -> Expr -> [(a, Unique (WithLoHi IM.Key))] -> [(a, Unique (WithLoHi IM.Key))]
tasty-bench.cabal view
@@ -1,5 +1,5 @@ name: tasty-bench-version: 0.3.5+version: 0.4 cabal-version: 1.18 build-type: Simple license: MIT@@ -25,7 +25,7 @@ example.svg README.md -tested-with: GHC == 9.8.1, GHC == 9.6.2, GHC == 9.4.7, GHC == 9.2.8, GHC==9.0.2, GHC==8.10.7, GHC==8.8.4, GHC==8.6.5, GHC==8.4.4, GHC==8.2.2, GHC==8.0.2, GHC==7.10.3, GHC==7.8.4, GHC==7.6.3, GHC==7.4.2, GHC==7.2.2, GHC==7.0.4+tested-with: GHC == 9.10.1, GHC == 9.8.2, GHC == 9.6.6, GHC == 9.4.8, GHC == 9.2.8, GHC==9.0.2, GHC==8.10.7, GHC==8.8.4, GHC==8.6.5, GHC==8.4.4, GHC==8.2.2, GHC==8.0.2 source-repository head type: git@@ -38,12 +38,6 @@ When disabled, reduces API to functions independent of @tasty@: combinators to construct @Benchmarkable@ and @measureCpuTime@. -flag debug- default: False- manual: True- description:- Emit ongoing diagnostic information for benchmarks.- library exposed-modules: Test.Tasty.Bench hs-source-dirs: src@@ -57,17 +51,17 @@ build-depends: base >= 4.3 && < 5, deepseq >= 1.1 && < 1.6,- ghc-prim < 0.11+ ghc-prim < 0.12 if flag(tasty) build-depends:- containers >= 0.4 && < 0.7,- tasty >= 1.2.3 && < 1.6+ containers >= 0.4 && < 0.8,+ tasty >= 1.4 && < 1.6 if impl(ghc < 7.8) build-depends: tagged >= 0.2 && < 0.9-- if flag(debug)- cpp-options: -DDEBUG+ if impl(ghc < 8.4)+ build-depends:+ time >= 1.2 && < 1.13 benchmark bench-fibo default-language: Haskell2010