diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -42,7 +42,7 @@ `tasty-bench` is a native Haskell library and works everywhere, where GHC does, including WASM. We support a full range of architectures (`i386`, `amd64`, `armhf`, `arm64`, `ppc64le`, `s390x`) and operating systems (Linux, Windows, macOS, -FreeBSD, OpenBSD, NetBSD), plus any GHC from 7.0 to 9.6. +FreeBSD, OpenBSD, NetBSD), plus any GHC from 7.0 to 9.10. ## How is it possible? @@ -227,9 +227,27 @@ All 3 tests passed (7.25s) ``` -This data is reported as per `RTSStats` fields: `allocated_bytes`, `copied_bytes` -and `max_mem_in_use_bytes`. +This data is reported as per [`GHC.Stats.RTSStats`](https://hackage.haskell.org/package/base/docs/GHC-Stats.html#t:RTSStats) fields: +* `allocated_bytes` + + Total size of data ever allocated since the start + of the benchmark iteration. Even if data was immediately + garbage collected and freed, it still counts. + +* `copied_bytes` + + Total size of data ever copied by GC (because it was alive and kicking) + since the start of the benchmark iteration. Note that zero bytes often mean + that the benchmark was too short to trigger GC at all. + +* `max_mem_in_use_bytes` + + Peak size of live data since the very start of the process. + This is a global metric, it cumulatively grows and does not say much + about individual benchmarks, but rather characterizes heap + environment in which they are executed. + ## Combining tests and benchmarks When optimizing an existing function, it is important to check that its @@ -310,7 +328,7 @@ ``` All fibo 20: OK (1.46s) - Response {respEstimate = Estimate {estMean = Measurement {measTime = 87496728, measAllocs = 0, measCopied = 0}, estStdev = 694487}, respIfSlower = FailIfSlower Infinity, respIfFaster = FailIfFaster Infinity} + WithLoHi (Estimate {estMean = Measurement {measTime = 41529118775, measAllocs = 0, measCopied = 0, measMaxMem = 0}, estStdev = 1595055320}) (-Infinity) Infinity ``` * If benchmarks fail with an error message @@ -330,6 +348,10 @@ or affect their hierarchy in other way. This is a fundamental restriction of `tasty` to list and filter benchmarks without launching missiles. + Strict pattern-matching on resource is also prohibited. For instance, + if it is a tuple, the second argument of `env` should use a lazy pattern match + `\~(a, b) -> ...` + * If benchmarks fail with `Test dependencies form a loop` or `Test dependencies have cycles`, this is likely because of `bcompare`, which compares a benchmark with itself. @@ -484,7 +506,7 @@ If you wish to compare two CSV reports non-interactively, here is a handy `awk` incantation: ```sh -awk 'BEGIN{FS=",";OFS=",";print "Name,Old,New,Ratio"}FNR==1{trueNF=NF;next}NF0)print "Geometric mean,,",exp(gs/gc)}' old.csv new.csv +awk 'BEGIN{FS=",";OFS=",";print "Name,Old,New,Ratio"}FNR==1{trueNF=NF;next}NF0)print "Geometric mean,,",exp(gs/gc)}' old.csv new.csv ``` A larger shell snippet to compare two `git` commits can be found in `compare_benches.sh`. @@ -563,10 +585,6 @@ It will provide you with functions to build `Benchmarkable` and run them manually via `measureCpuTime`. This mode of operation can be also configured by disabling Cabal flag `tasty`. - -* If results are amiss or oscillate wildly and adjusting `--timeout` and `--stdev` - does not help, you may be interested to investigate individual timings of - successive runs by enabling Cabal flag `debug`. This will pipe raw data into `stderr`. ## Command-line options diff --git a/changelog.md b/changelog.md --- a/changelog.md +++ b/changelog.md @@ -1,3 +1,19 @@ +# 0.4 + +* Switch `nf`, `nfIO` and `nfAppIO` to evaluate outputs to a normal form + with `rnf` instead of `force`. It means that parts of the output, which have + already been forced, can be garbage collected early, without waiting for + the entire output to be allocated at the same time. This decreases + benchmarking overhead in many scenarios and brings the behaviour in line + with `criterion`. See [#39](https://github.com/Bodigrim/tasty-bench/issues/39) + for discussion. +* Drop support of `tasty < 1.4`. +* Make `IO` benchmarks immune to `-fspec-constr-count` limit. +* Decomission `debug` build flag. +* Decomission warning when `--timeout` is absent. +* Add `instance {Eq,Ord,Num,Fractional} {RelStDev,FailIfSlower,FailIfFaster}`. +* Add `instance {Eq,Ord} {CsvPath,SvgPath,BaselinePath}`. + # 0.3.5 * Support `tasty-1.5`. diff --git a/compare_benches.sh b/compare_benches.sh --- a/compare_benches.sh +++ b/compare_benches.sh @@ -18,10 +18,14 @@ OLDVSNEWCSV=$(echo "$OLD"-vs-"$NEW".csv | sed -e s#/##g) git checkout -q "$OLDREF" && \ + trap 'git checkout -q "@{-1}" && trap - INT' INT && \ cabal run -v0 benchmarks -- --csv "$OLDCSV" "$@" && \ git checkout -q "$NEWREF" && \ + trap 'git checkout -q "@{-2}" && trap - INT' INT && \ cabal run -v0 benchmarks -- --baseline "$OLDCSV" --csv "$NEWCSV" "$@" && \ git checkout -q "@{-2}" && \ - awk 'BEGIN{FS=",";OFS=",";print "Name,'"$OLD"','"$NEW"',Ratio"}FNR==1{trueNF=NF;next}NF0)print "Geometric mean,,",exp(gs/gc)}' "$OLDCSV" "$NEWCSV" > "$OLDVSNEWCSV" + awk 'BEGIN{FS=",";OFS=",";print "Name,'"$OLD"','"$NEW"',Ratio"}FNR==1{trueNF=NF;next}NF0)print "Geometric mean,,",exp(gs/gc)}' "$OLDCSV" "$NEWCSV" > "$OLDVSNEWCSV" && \ + + trap - INT } diff --git a/src/Test/Tasty/Bench.hs b/src/Test/Tasty/Bench.hs --- a/src/Test/Tasty/Bench.hs +++ b/src/Test/Tasty/Bench.hs @@ -27,7 +27,7 @@ GHC does, including WASM. We support a full range of architectures (@i386@, @amd64@, @armhf@, @arm64@, @ppc64le@, @s390x@) and operating systems (Linux, Windows, macOS, FreeBSD, OpenBSD, NetBSD), plus any GHC -from 7.0 to 9.6. +from 7.0 to 9.10. === How is it possible? @@ -204,9 +204,30 @@ > > All 3 tests passed (7.25s) -This data is reported as per 'RTSStats' fields: 'allocated_bytes', -'copied_bytes' and 'max_mem_in_use_bytes'. +This data is reported as per + +fields: +- 'allocated_bytes' + + Total size of data ever allocated since the start of the benchmark + iteration. Even if data was immediately garbage collected and freed, + it still counts. + +- 'copied_bytes' + + Total size of data ever copied by GC (because it was alive and + kicking) since the start of the benchmark iteration. Note that zero + bytes often mean that the benchmark was too short to trigger GC at + all. + +- 'max_mem_in_use_bytes' + + Peak size of live data since the very start of the process. This is + a global metric, it cumulatively grows and does not say much about + individual benchmarks, but rather characterizes heap environment in + which they are executed. + === Combining tests and benchmarks When optimizing an existing function, it is important to check that its @@ -285,7 +306,7 @@ > All > fibo 20: OK (1.46s) - > Response {respEstimate = Estimate {estMean = Measurement {measTime = 87496728, measAllocs = 0, measCopied = 0}, estStdev = 694487}, respIfSlower = FailIfSlower Infinity, respIfFaster = FailIfFaster Infinity} + > WithLoHi (Estimate {estMean = Measurement {measTime = 41529118775, measAllocs = 0, measCopied = 0, measMaxMem = 0}, estStdev = 1595055320}) (-Infinity) Infinity - If benchmarks fail with an error message @@ -301,6 +322,10 @@ way. This is a fundamental restriction of @tasty@ to list and filter benchmarks without launching missiles. + Strict pattern-matching on resource is also prohibited. For + instance, if it is a tuple, the second argument of 'env' should use + a lazy pattern match @\\~(a, b) -> ...@ + - If benchmarks fail with @Test dependencies form a loop@ or @Test dependencies have cycles@, this is likely because of 'bcompare', which compares a benchmark with itself. Locating a @@ -447,7 +472,7 @@ If you wish to compare two CSV reports non-interactively, here is a handy @awk@ incantation: -> awk 'BEGIN{FS=",";OFS=",";print "Name,Old,New,Ratio"}FNR==1{trueNF=NF;next}NF0)print "Geometric mean,,",exp(gs/gc)}' old.csv new.csv +> awk 'BEGIN{FS=",";OFS=",";print "Name,Old,New,Ratio"}FNR==1{trueNF=NF;next}NF0)print "Geometric mean,,",exp(gs/gc)}' old.csv new.csv A larger shell snippet to compare two @git@ commits can be found in @compare_benches.sh@. @@ -525,11 +550,6 @@ 'measureCpuTime'. This mode of operation can be also configured by disabling Cabal flag @tasty@. -- If results are amiss or oscillate wildly and adjusting @--timeout@ - and @--stdev@ does not help, you may be interested to investigate - individual timings of successive runs by enabling Cabal flag - @debug@. This will pipe raw data into @stderr@. - === Command-line options Use @--help@ to list all command-line options. @@ -632,6 +652,7 @@ {-# LANGUAGE DeriveFunctor #-} {-# LANGUAGE FlexibleContexts #-} {-# LANGUAGE FlexibleInstances #-} +{-# LANGUAGE GeneralizedNewtypeDeriving #-} {-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TupleSections #-} @@ -685,7 +706,7 @@ import qualified Prelude import Control.Applicative import Control.Arrow (first, second) -import Control.DeepSeq (NFData, force) +import Control.DeepSeq (NFData, force, rnf) import Control.Exception (bracket, evaluate) import Control.Monad (void, unless, guard, (>=>), when) import Data.Data (Typeable) @@ -712,10 +733,6 @@ import System.Mem import Text.Printf -#ifdef DEBUG -import Debug.Trace -#endif - #ifdef MIN_VERSION_tasty #if !MIN_VERSION_base(4,8,0) import Data.Monoid (Monoid(..)) @@ -744,6 +761,12 @@ import Test.Tasty.Runners #endif +#if MIN_VERSION_base(4,11,0) +import GHC.Clock (getMonotonicTime) +#else +import Data.Time.Clock.POSIX (getPOSIXTime) +#endif + #if defined(mingw32_HOST_OS) import Data.Word (Word32) #endif @@ -788,7 +811,19 @@ -- -- @since 0.2 newtype RelStDev = RelStDev Double - deriving (Show, Read, Typeable) + deriving + ( Eq + -- ^ @since 0.4 + , Ord + -- ^ @since 0.4 + , Show + , Read + , Num + -- ^ @since 0.4 + , Fractional + -- ^ @since 0.4 + , Typeable + ) -- | Whether to measure CPU time or wall-clock time. -- Normally 'CpuTime' is a better option (and default), @@ -802,15 +837,21 @@ -- > import Test.Tasty (localOption) -- > localOption WallTime (bgroup [...]) -- --- section of your cabal file. +-- You can measure both times and report their ratio with the following gadget: -- +-- @ +-- bgroup \"Foo\" +-- [ localOption WallTime $ bench \"WallTime\" foo +-- , bcompare \"Foo.WallTime\" +-- $ localOption CpuTime $ bench \"CPUTime\" foo +-- ] +-- @ +-- -- @since 0.3.2 data TimeMode = CpuTime -- ^ Measure CPU time. -#ifdef MIN_VERSION_tasty | WallTime -- ^ Measure wall-clock time. -#endif deriving (Typeable) #ifdef MIN_VERSION_tasty @@ -833,7 +874,19 @@ -- -- @since 0.2 newtype FailIfSlower = FailIfSlower Double - deriving (Show, Read, Typeable) + deriving + ( Eq + -- ^ @since 0.4 + , Ord + -- ^ @since 0.4 + , Show + , Read + , Num + -- ^ @since 0.4 + , Fractional + -- ^ @since 0.4 + , Typeable + ) instance IsOption FailIfSlower where defaultValue = FailIfSlower (1.0 / 0.0) @@ -854,7 +907,19 @@ -- -- @since 0.2 newtype FailIfFaster = FailIfFaster Double - deriving (Show, Read, Typeable) + deriving + ( Eq + -- ^ @since 0.4 + , Ord + -- ^ @since 0.4 + , Show + , Read + , Num + -- ^ @since 0.4 + , Fractional + -- ^ @since 0.4 + , Typeable + ) instance IsOption FailIfFaster where defaultValue = FailIfFaster (1.0 / 0.0) @@ -876,12 +941,10 @@ _ -> Nothing optionName = pure "time-mode" optionHelp = pure "Whether to measure CPU time (\"cpu\") or wall-clock time (\"wall\")" -#if MIN_VERSION_tasty(1,3,0) showDefaultValue m = Just $ case m of CpuTime -> "cpu" WallTime -> "wall" #endif -#endif -- | Something that can be benchmarked, produced by 'nf', 'whnf', 'nfIO', 'whnfIO', -- 'nfAppIO', 'whnfAppIO' below. @@ -1057,12 +1120,17 @@ pure (0, 0, 0) #endif +getWallTimeSecs :: IO Double +#if MIN_VERSION_base(4,11,0) +getWallTimeSecs = getMonotonicTime +#else +getWallTimeSecs = realToFrac <$> getPOSIXTime +#endif + getTimePicoSecs :: TimeMode -> IO Word64 getTimePicoSecs timeMode = case timeMode of CpuTime -> fromInteger <$> getCPUTime -#ifdef MIN_VERSION_tasty - WallTime -> round . (1e12 *) <$> getTime -#endif + WallTime -> round . (1e12 *) <$> getWallTimeSecs measure :: TimeMode -> Word64 -> Benchmarkable -> IO Measurement measure timeMode n (Benchmarkable act) = do @@ -1079,25 +1147,20 @@ , measCopied = endCopied - startCopied , measMaxMem = max endMaxMemInUse startMaxMemInUse } -#ifdef DEBUG - pure $ trace (show n ++ (if n == 1 then " iteration gives " else " iterations give ") ++ show meas) meas -#else pure meas -#endif measureUntil :: (Progress -> IO ()) -> TimeMode - -> Bool -> Timeout -> RelStDev -> Benchmarkable -> IO Estimate -measureUntil _ timeMode _ _ (RelStDev targetRelStDev) b +measureUntil _ timeMode _ (RelStDev targetRelStDev) b | isInfinite targetRelStDev, targetRelStDev > 0 = do t1 <- measure timeMode 1 b pure $ Estimate { estMean = t1, estStdev = 0 } -measureUntil yieldProgress timeMode warnIfNoTimeout timeout (RelStDev targetRelStDev) b = do +measureUntil yieldProgress timeMode timeout (RelStDev targetRelStDev) b = do t1 <- measure' 1 b go 1 t1 0 where @@ -1129,11 +1192,6 @@ yieldProgress () #endif - case timeout of - NoTimeout | warnIfNoTimeout, sumOfTs' + measTime t2 > 100 * 1000000000000 - -> hPutStrLn stderr "This benchmark takes more than 100 seconds. Consider setting --timeout, if this is unexpected (or to silence this warning)." - _ -> pure () - if isStDevInTargetRange || isTimeoutSoon then pure scaledEstimate else go (2 * n) t2 sumOfTs' @@ -1162,7 +1220,7 @@ ( word64ToDouble (measTime (estMean x)) / 1e12 , word64ToDouble (estStdev x) / 1e12 )) .) .) - . measureUntil (const $ pure ()) CpuTime False + . measureUntil (const $ pure ()) CpuTime #ifdef MIN_VERSION_tasty @@ -1178,7 +1236,7 @@ run opts b yieldProgress = case getNumThreads (lookupOption opts) of 1 -> do let timeMode = lookupOption opts - est <- measureUntil yieldProgress timeMode True (lookupOption opts) (lookupOption opts) b + est <- measureUntil yieldProgress timeMode (lookupOption opts) (lookupOption opts) b let FailIfSlower ifSlower = lookupOption opts FailIfFaster ifFaster = lookupOption opts pure $ testPassed $ show (WithLoHi est (1 - ifFaster) (1 + ifSlower)) @@ -1317,7 +1375,7 @@ #endif funcToBench :: forall a b c. (b -> c) -> (a -> b) -> a -> Benchmarkable -funcToBench frc = (Benchmarkable .) . benchLoop SPEC +funcToBench frc = (Benchmarkable .) . funcToBenchLoop SPEC where -- Here we rely on the fact that GHC (unless spurred by -- -fstatic-argument-transformation) is not smart enough: @@ -1328,25 +1386,25 @@ -- -- For perspective, gauge and criterion < 1.4 mark similar functions as INLINE, -- while criterion >= 1.4 switches to NOINLINE. - -- If we mark `benchLoop` NOINLINE then benchmark results are slightly larger + -- If we mark `funcToBenchLoop` NOINLINE then benchmark results are slightly larger -- (noticeable in bench-fibo), because the loop body is slightly bigger, -- since GHC does not unbox numbers or inline `Eq @Word64` dictionary. -- - -- This function is called `benchLoop` instead of, say, `go`, + -- This function is called `funcToBenchLoop` instead of, say, `go`, -- so it is easier to spot in Core dumps. -- -- Forcing SpecConst optimization with SPEC makes the behaviour of benchmarks -- independent of -fspec-constr-count. - benchLoop :: SPEC -> (a -> b) -> a -> Word64 -> IO () - benchLoop !_ f x n + funcToBenchLoop :: SPEC -> (a -> b) -> a -> Word64 -> IO () + funcToBenchLoop !_ f x n | n == 0 = pure () | otherwise = do _ <- evaluate (frc (f x)) - benchLoop SPEC f x (n - 1) + funcToBenchLoop SPEC f x (n - 1) {-# INLINE funcToBench #-} -- | 'nf' @f@ @x@ measures time to compute --- a normal form (by means of 'force', not 'Control.DeepSeq.rnf') +-- a normal form (by means of 'Control.DeepSeq.rnf', not 'Control.DeepSeq.force') -- of an application of @f@ to @x@. -- This does not include time to evaluate @f@ or @x@ themselves. -- Ideally @x@ should be a primitive data type like 'Data.Int.Int'. @@ -1364,6 +1422,10 @@ -- on garbage collector. Also no list fusion will happen. -- A better approach is 'nf' (@\\n@ @->@ 'sum' @[1..n]@) @1000000@. -- +-- It is preferable that the return type of the function under measurement +-- is inhabited enough to depend genuinely on all computations and is not simply @b ~ ()@. +-- Otherwise GHC might get aggressive and optimise the payload away. +-- -- If you are measuring an inlinable function, -- it is prudent to ensure that its invocation is fully saturated, -- otherwise inlining will not happen. That's why one can often @@ -1379,9 +1441,9 @@ -- especially when 'NFData' instance is badly written, -- this traversal may take non-negligible time and affect results. -- --- 'nf' @f@ is equivalent to 'whnf' ('force' '.' @f@), but not to --- 'whnf' ('Control.DeepSeq.rnf' '.' @f@). The former retains the result --- in memory until it is fully evaluated, while the latter allows +-- 'nf' @f@ is equivalent to 'whnf' ('Control.DeepSeq.rnf' '.' @f@), but not to +-- 'whnf' ('Control.DeepSeq.force' '.' @f@). The latter retains the result +-- in memory until it is fully evaluated, while the former allows -- evaluated parts of the result to be garbage-collected immediately. -- -- For users of @{-# LANGUAGE LinearTypes #-}@: if @f@ is a linear function, @@ -1393,7 +1455,7 @@ -- -- @since 0.1 nf :: NFData b => (a -> b) -> a -> Benchmarkable -nf = funcToBench force +nf = funcToBench rnf {-# INLINE nf #-} -- | 'whnf' @f@ @x@ measures time to compute @@ -1427,21 +1489,27 @@ {-# INLINE whnf #-} ioToBench :: (b -> c) -> IO b -> Benchmarkable -ioToBench frc act = Benchmarkable go +ioToBench frc act = Benchmarkable (ioToBenchLoop SPEC) where - go n + ioToBenchLoop :: SPEC -> Word64 -> IO () + ioToBenchLoop !_ n | n == 0 = pure () | otherwise = do val <- act _ <- evaluate (frc val) - go (n - 1) + ioToBenchLoop SPEC (n - 1) {-# INLINE ioToBench #-} -- | 'nfIO' @x@ measures time to evaluate side-effects of @x@ --- and compute its normal form (by means of 'force', not 'Control.DeepSeq.rnf'). +-- and compute its normal form +-- (by means of 'Control.DeepSeq.rnf', not 'Control.DeepSeq.force'). -- -- Pure subexpression of an effectful computation @x@ --- may be evaluated only once and get cached. +-- may be evaluated only once and get cached. For example, +-- GHC is likely to float @x@ out of 'nfIO' ('pure' @x@) and +-- evaluate in only once, which leaves 'nfIO' to measure 'pure' only +-- with results in nanosecond range. +-- -- To avoid surprising results it is usually preferable -- to use 'nfAppIO' instead. -- @@ -1459,14 +1527,18 @@ -- -- @since 0.1 nfIO :: NFData a => IO a -> Benchmarkable -nfIO = ioToBench force +nfIO = ioToBench rnf {-# INLINE nfIO #-} -- | 'whnfIO' @x@ measures time to evaluate side-effects of @x@ -- and compute its weak head normal form. -- -- Pure subexpression of an effectful computation @x@ --- may be evaluated only once and get cached. +-- may be evaluated only once and get cached. For example, +-- GHC is likely to float @x@ out of 'whnfIO' ('pure' @x@) and +-- evaluate in only once, which leaves 'whnfIO' to measure 'pure' only +-- with results in nanosecond range. +-- -- To avoid surprising results it is usually preferable -- to use 'whnfAppIO' instead. -- @@ -1487,20 +1559,22 @@ whnfIO = ioToBench id {-# INLINE whnfIO #-} -ioFuncToBench :: (b -> c) -> (a -> IO b) -> a -> Benchmarkable -ioFuncToBench frc = (Benchmarkable .) . go +ioFuncToBench :: forall a b c. (b -> c) -> (a -> IO b) -> a -> Benchmarkable +ioFuncToBench frc = (Benchmarkable .) . ioFuncToBenchLoop SPEC where - go f x n + ioFuncToBenchLoop :: SPEC -> (a -> IO b) -> a -> Word64 -> IO () + ioFuncToBenchLoop !_ f x n | n == 0 = pure () | otherwise = do val <- f x _ <- evaluate (frc val) - go f x (n - 1) + ioFuncToBenchLoop SPEC f x (n - 1) {-# INLINE ioFuncToBench #-} -- | 'nfAppIO' @f@ @x@ measures time to evaluate side-effects of -- an application of @f@ to @x@ --- and compute its normal form (by means of 'force', not 'Control.DeepSeq.rnf'). +-- and compute its normal form +-- (by means of 'Control.DeepSeq.rnf', not 'Control.DeepSeq.force'). -- This does not include time to evaluate @f@ or @x@ themselves. -- Ideally @x@ should be a primitive data type like 'Data.Int.Int'. -- @@ -1525,7 +1599,7 @@ -- -- @since 0.1 nfAppIO :: NFData b => (a -> IO b) -> a -> Benchmarkable -nfAppIO = ioFuncToBench force +nfAppIO = ioFuncToBench rnf {-# INLINE nfAppIO #-} -- | 'whnfAppIO' @f@ @x@ measures time to evaluate side-effects of @@ -1582,14 +1656,15 @@ -- -- use -- --- > import Control.DeepSeq (force) --- > import Control.Exception (evaluate) --- > -- > main :: IO () -- > main = defaultMain --- > [ env (evaluate (force (replicate 1000000 'a'))) $ \largeData -> +-- > [ env (pure (replicate 1000000 'a')) $ \largeData -> -- > bench "large" $ nf length largeData, ... ] -- +-- Even with 'env', it's advisable to store input data in as few heap objects +-- as possible. 'Data.Array.ByteArray.ByteArray' (ideally pinned) +-- or unboxed @Vector@ are good, boxed arrays are worse, lists and trees are bad. +-- -- @Test.Tasty.Bench.@'env' is provided only for the sake of -- compatibility with @Criterion.@'Criterion.env' and -- @Gauge.@'Gauge.env', and involves 'unsafePerformIO'. Consider using @@ -1608,6 +1683,10 @@ -- -- > Unhandled resource. Probably a bug in the runner you're using. -- +-- Strict pattern-matching on resource is also prohibited. For +-- instance, if it is a tuple, the second argument of 'env' should use +-- a lazy pattern match @\\~(a, b) -> ...@ +-- -- @since 0.2 env :: NFData env => IO env -> (env -> Benchmark) -> Benchmark env res = envWithCleanup res (const $ pure ()) @@ -1654,7 +1733,13 @@ -- -- @since 0.3 newtype CsvPath = CsvPath FilePath - deriving (Typeable) + deriving + ( Eq + -- ^ @since 0.4 + , Ord + -- ^ @since 0.4 + , Typeable + ) instance IsOption (Maybe CsvPath) where defaultValue = Nothing @@ -1727,7 +1812,13 @@ -- -- @since 0.3 newtype SvgPath = SvgPath FilePath - deriving (Typeable) + deriving + ( Eq + -- ^ @since 0.4 + , Ord + -- ^ @since 0.4 + , Typeable + ) instance IsOption (Maybe SvgPath) where defaultValue = Nothing @@ -1860,7 +1951,13 @@ -- -- @since 0.3 newtype BaselinePath = BaselinePath FilePath - deriving (Typeable) + deriving + ( Eq + -- ^ @since 0.4 + , Ord + -- ^ @since 0.4 + , Typeable + ) instance IsOption (Maybe BaselinePath) where defaultValue = Nothing @@ -2002,27 +2099,20 @@ { foldSingle = const $ const . (:[]) . Seq.singleton #if MIN_VERSION_tasty(1,5,0) , foldGroup = const $ (. concat) . map . (<|) -#elif MIN_VERSION_tasty(1,4,0) - , foldGroup = const $ map . (<|) #else - , foldGroup = map . (<|) + , foldGroup = const $ map . (<|) #endif } testNamesAndDeps :: IntMap (Seq TestName) -> OptionSet -> TestTree -> [(TestName, Unique (WithLoHi IM.Key))] testNamesAndDeps im = foldTestTree trivialFold { foldSingle = const $ const . (: []) . (, mempty) -#if MIN_VERSION_tasty(1,4,0) #if MIN_VERSION_tasty(1,5,0) , foldGroup = const $ (. concat) . map . first . (++) . (++ ".") #else , foldGroup = const $ map . first . (++) . (++ ".") #endif , foldAfter = const foldDeps -#else - , foldGroup = map . first . (++) . (++ ".") - , foldAfter = foldDeps -#endif } where foldDeps :: DependencyType -> Expr -> [(a, Unique (WithLoHi IM.Key))] -> [(a, Unique (WithLoHi IM.Key))] diff --git a/tasty-bench.cabal b/tasty-bench.cabal --- a/tasty-bench.cabal +++ b/tasty-bench.cabal @@ -1,5 +1,5 @@ name: tasty-bench -version: 0.3.5 +version: 0.4 cabal-version: 1.18 build-type: Simple license: MIT @@ -25,7 +25,7 @@ example.svg README.md -tested-with: GHC == 9.8.1, GHC == 9.6.2, GHC == 9.4.7, GHC == 9.2.8, GHC==9.0.2, GHC==8.10.7, GHC==8.8.4, GHC==8.6.5, GHC==8.4.4, GHC==8.2.2, GHC==8.0.2, GHC==7.10.3, GHC==7.8.4, GHC==7.6.3, GHC==7.4.2, GHC==7.2.2, GHC==7.0.4 +tested-with: GHC == 9.10.1, GHC == 9.8.2, GHC == 9.6.6, GHC == 9.4.8, GHC == 9.2.8, GHC==9.0.2, GHC==8.10.7, GHC==8.8.4, GHC==8.6.5, GHC==8.4.4, GHC==8.2.2, GHC==8.0.2 source-repository head type: git @@ -38,12 +38,6 @@ When disabled, reduces API to functions independent of @tasty@: combinators to construct @Benchmarkable@ and @measureCpuTime@. -flag debug - default: False - manual: True - description: - Emit ongoing diagnostic information for benchmarks. - library exposed-modules: Test.Tasty.Bench hs-source-dirs: src @@ -57,17 +51,17 @@ build-depends: base >= 4.3 && < 5, deepseq >= 1.1 && < 1.6, - ghc-prim < 0.11 + ghc-prim < 0.12 if flag(tasty) build-depends: - containers >= 0.4 && < 0.7, - tasty >= 1.2.3 && < 1.6 + containers >= 0.4 && < 0.8, + tasty >= 1.4 && < 1.6 if impl(ghc < 7.8) build-depends: tagged >= 0.2 && < 0.9 - - if flag(debug) - cpp-options: -DDEBUG + if impl(ghc < 8.4) + build-depends: + time >= 1.2 && < 1.13 benchmark bench-fibo default-language: Haskell2010