tasty-bench 0.3.2 → 0.3.3
raw patch · 5 files changed
+92/−83 lines, 5 filesdep +ghc-primdep ~tasty
Dependencies added: ghc-prim
Dependency ranges changed: tasty
Files
- README.md +15/−16
- changelog.md +5/−0
- compare_benches.sh +11/−3
- src/Test/Tasty/Bench.hs +56/−60
- tasty-bench.cabal +5/−4
README.md view
@@ -41,8 +41,8 @@ `tasty-bench` is a native Haskell library and works everywhere, where GHC does. We support a full range of architectures (`i386`, `amd64`, `armhf`,-`arm64`, `ppc64le`, `s390x`) and operating systems (Linux, Windows, MacOS,-FreeBSD), plus any GHC from 7.0 to 9.4.+`arm64`, `ppc64le`, `s390x`) and operating systems (Linux, Windows, macOS,+FreeBSD, OpenBSD, NetBSD), plus any GHC from 7.0 to 9.6. ## How is it possible? @@ -93,6 +93,8 @@ type: exitcode-stdio-1.0 build-depends: base, tasty-bench ghc-options: "-with-rtsopts=-A32m"+ if impl(ghc >= 8.6)+ ghc-options: -fproc-alignment=64 ``` And here is `BenchFibo.hs`:@@ -387,27 +389,24 @@ Finally, as an ultimate measure to reduce interference between benchmarks, one can run each of them in a separate process. We do not quite recommend-this approach, but if you are desperate, here is how.--Assuming that a benchmark is declared in `cabal` file as `benchmark my-bench` component,-let's first find its executable:+this approach, but if you are desperate, here is how: ```sh-cabal build --enable-benchmarks my-bench-MYBENCH=$(cabal list-bin my-bench) # available since cabal-3.4+cabal run -v0 all:benches -- -l | sed -e 's/[\"]/\\\\\\&/g' | while read -r name; do cabal run -v0 all:benches -- -p '$0 == "'"$name"'"'; done ``` -Now list all benchmark names (hopefully, they do not contain newlines),-escape quotes and slashes, and run each of them separately:--```sh-$MYBENCH -l | sed -e 's/[\"]/\\\\\\&/g' | while read -r name; do $MYBENCH -p '$0 == "'"$name"'"'; done-```+This assumes that there is a single benchmark suite in the project+and that benchmark names do not contain newlines. ## Comparison against baseline -One can compare benchmark results against an earlier baseline in an automatic way.-To use this feature, first run `tasty-bench` with `--csv FILE` key+One can compare benchmark results against an earlier run in an automatic way.++When using this feature, it's especially important to compile benchmarks with+`ghc-options: `[`-fproc-alignment`](https://downloads.haskell.org/ghc/latest/docs/users_guide/debugging.html#ghc-flag--fproc-alignment)`=64`, otherwise results could be skewed by+intermittent changes in cache-line alignment.++Firstly, run `tasty-bench` with `--csv FILE` key to dump results to `FILE` in CSV format (it could be a good idea to set smaller `--stdev`, if possible):
changelog.md view
@@ -1,3 +1,8 @@+# 0.3.3++* Drop support of `tasty < 1.2.3`.+* Make benchmarks immune to `-fspec-constr-count` limit.+ # 0.3.2 * Add `locateBenchmark` and `mapLeafBenchmarks`.
compare_benches.sh view
@@ -4,17 +4,25 @@ printf "Usage:\n compare_benches oldCommit newCommit ...\nwhere ... is passed to benchmarks directly.\n" return 0 fi+ OLD="$1" shift NEW="$1" shift++ HEADREF=$(git rev-parse --verify HEAD)+ OLDREF=$(git rev-parse --verify "$OLD")+ NEWREF=$(git rev-parse --verify "$NEW")+ OLDCSV=$(echo "$OLD".csv | sed -e s#/##g) NEWCSV=$(echo "$NEW".csv | sed -e s#/##g) OLDVSNEWCSV=$(echo "$OLD"-vs-"$NEW".csv | sed -e s#/##g)- git checkout -q "$OLD" && \++ git checkout -q "$OLDREF" && \ cabal run -v0 benchmarks -- --csv "$OLDCSV" "$@" && \- git checkout -q "$NEW" && \+ git checkout -q "$NEWREF" && \ cabal run -v0 benchmarks -- --baseline "$OLDCSV" --csv "$NEWCSV" "$@" && \- git checkout -q "@{-2}" && \+ git checkout -q "$HEADREF" && \+ awk 'BEGIN{FS=",";OFS=",";print "Name,'"$OLD"','"$NEW"',Ratio"}FNR==1{trueNF=NF;next}NF<trueNF{print "Benchmark names should not contain newlines";exit 1}FNR==NR{oldTime=$(NF-trueNF+2);NF-=trueNF-1;a[$0]=oldTime;next}{newTime=$(NF-trueNF+2);NF-=trueNF-1;print $0,a[$0],newTime,newTime/a[$0];gs+=log(newTime/a[$0]);gc++}END{if(gc>0)print "Geometric mean,,",exp(gs/gc)}' "$OLDCSV" "$NEWCSV" > "$OLDVSNEWCSV" }
src/Test/Tasty/Bench.hs view
@@ -24,8 +24,8 @@ @tasty-bench@ is a native Haskell library and works everywhere, where GHC does. We support a full range of architectures (@i386@, @amd64@, @armhf@,-@arm64@, @ppc64le@, @s390x@) and operating systems (Linux, Windows, MacOS,-FreeBSD), plus any GHC from 7.0 to 9.4.+@arm64@, @ppc64le@, @s390x@) and operating systems (Linux, Windows, macOS,+FreeBSD, OpenBSD, NetBSD), plus any GHC from 7.0 to 9.6. === How is it possible? @@ -75,6 +75,8 @@ > type: exitcode-stdio-1.0 > build-depends: base, tasty-bench > ghc-options: "-with-rtsopts=-A32m"+> if impl(ghc >= 8.6)+> ghc-options: -fproc-alignment=64 And here is @BenchFibo.hs@: @@ -351,23 +353,22 @@ Finally, as an ultimate measure to reduce interference between benchmarks, one can run each of them in a separate process. We do not-quite recommend this approach, but if you are desperate, here is how.+quite recommend this approach, but if you are desperate, here is how: -Assuming that a benchmark is declared in @cabal@ file as-@benchmark@ @my-bench@ component, let’s first find its executable:+> cabal run -v0 all:benches -- -l | sed -e 's/[\"]/\\\\\\&/g' | while read -r name; do cabal run -v0 all:benches -- -p '$0 == "'"$name"'"'; done -> cabal build --enable-benchmarks my-bench-> MYBENCH=$(cabal list-bin my-bench) # available since cabal-3.4+This assumes that there is a single benchmark suite in the project+and that benchmark names do not contain newlines. -Now list all benchmark names (hopefully, they do not contain newlines),-escape quotes and slashes, and run each of them separately:+=== Comparison against baseline -> $MYBENCH -l | sed -e 's/[\"]/\\\\\\&/g' | while read -r name; do $MYBENCH -p '$0 == "'"$name"'"'; done+One can compare benchmark results against an earlier run in an automatic way. -=== Comparison against baseline+When using this feature, it's especially important to compile benchmarks with+@ghc-options:@ [@-fproc-alignment@](https://downloads.haskell.org/ghc/latest/docs/users_guide/debugging.html#ghc-flag--fproc-alignment)@=64@, otherwise results could be skewed by+intermittent changes in cache-line alignment. -One can compare benchmark results against an earlier baseline in an-automatic way. To use this feature, first run @tasty-bench@ with+Firstly, run @tasty-bench@ with @--csv@ @FILE@ key to dump results to @FILE@ in CSV format (it could be a good idea to set smaller @--stdev@, if possible): @@ -570,6 +571,7 @@ -} +{-# LANGUAGE BangPatterns #-} {-# LANGUAGE CPP #-} {-# LANGUAGE DeriveDataTypeable #-} {-# LANGUAGE DeriveFunctor #-}@@ -586,10 +588,8 @@ , Benchmark , bench , bgroup-#if MIN_VERSION_tasty(1,2,0) , bcompare , bcompareWithin-#endif , env , envWithCleanup ,@@ -617,9 +617,7 @@ , SvgPath(..) , TimeMode(..) -- * Utils-#if MIN_VERSION_tasty(1,0,0) , locateBenchmark-#endif , mapLeafBenchmarks #else , Timeout(..)@@ -683,11 +681,9 @@ import Test.Tasty.Ingredients import Test.Tasty.Ingredients.ConsoleReporter import Test.Tasty.Options-#if MIN_VERSION_tasty(1,0,0) import Test.Tasty.Patterns.Eval (eval, asB, withFields) import Test.Tasty.Patterns.Types (Expr (And, Field, IntLit, NF, StringLit, Sub)) import qualified Test.Tasty.Patterns.Types as Patterns-#endif import Test.Tasty.Providers import Test.Tasty.Runners #endif@@ -696,6 +692,15 @@ import Data.Word (Word32) #endif +#if MIN_VERSION_ghc_prim(0,3,1)+import GHC.Types (SPEC(..))+#else+import GHC.Exts (SpecConstrAnnotation(..))++data SPEC = SPEC | SPEC2+{-# ANN type SPEC ForceSpecConstr #-}+#endif+ #ifndef MIN_VERSION_tasty data Timeout = Timeout@@ -740,16 +745,13 @@ -- > localOption WallTime (bgroup [...]) -- -- section of your cabal file.+-- -- @since 0.3.2 data TimeMode = CpuTime -- ^ Measure CPU time. #ifdef MIN_VERSION_tasty-#if MIN_VERSION_tasty(1,2,2) | WallTime- -- ^ Measure wall-clock time. This requires @tasty-1.2.2@, so if you use 'WallTime'- -- it is prudent to add @tasty >= 1.2.2@ to @build-depends@- -- section of your cabal file.-#endif+ -- ^ Measure wall-clock time. #endif deriving (Typeable) @@ -812,20 +814,16 @@ defaultValue = CpuTime parseValue v = case v of "cpu" -> Just CpuTime-#if MIN_VERSION_tasty(1,2,2) "wall" -> Just WallTime-#endif _ -> Nothing optionName = pure "time-mode" optionHelp = pure "Whether to measure CPU time (\"cpu\") or wall-clock time (\"wall\")" #if MIN_VERSION_tasty(1,3,0) showDefaultValue m = Just $ case m of CpuTime -> "cpu"-#if MIN_VERSION_tasty(1,2,2) WallTime -> "wall" #endif #endif-#endif -- | Something that can be benchmarked, produced by 'nf', 'whnf', 'nfIO', 'whnfIO', -- 'nfAppIO', 'whnfAppIO' below.@@ -1005,10 +1003,8 @@ getTimePicoSecs timeMode = case timeMode of CpuTime -> fromInteger <$> getCPUTime #ifdef MIN_VERSION_tasty-#if MIN_VERSION_tasty(1,2,2) WallTime -> round . (1e12 *) <$> getTime #endif-#endif measure :: TimeMode -> Word64 -> Benchmarkable -> IO Measurement measure timeMode n (Benchmarkable act) = do@@ -1122,14 +1118,11 @@ bgroup :: String -> [Benchmark] -> Benchmark bgroup = testGroup -#if MIN_VERSION_tasty(1,2,0) -- | Compare benchmarks, reporting relative speed up or slow down. -- -- This function is a vague reminiscence of @bcompare@, which existed in pre-1.0 -- versions of @criterion@, but their types are incompatible. Under the hood--- 'bcompare' is a thin wrapper over 'after' and requires @tasty-1.2@.--- If you use 'bcompare', it is prudent to add @tasty >= 1.2@ to @build-depends@--- section of your cabal file.+-- 'bcompare' is a thin wrapper over 'after'. -- -- Here is a basic example: --@@ -1187,7 +1180,6 @@ bcomparePrefix :: String bcomparePrefix = "tasty-bench"-#endif -- | Benchmarks are actually just a regular 'Test.Tasty.TestTree' in disguise. --@@ -1223,8 +1215,8 @@ #endif -funcToBench :: (b -> c) -> (a -> b) -> a -> Benchmarkable-funcToBench frc = (Benchmarkable .) . benchLoop+funcToBench :: forall a b c. (b -> c) -> (a -> b) -> a -> Benchmarkable+funcToBench frc = (Benchmarkable .) . benchLoop SPEC where -- Here we rely on the fact that GHC (unless spurred by -- -fstatic-argument-transformation) is not smart enough:@@ -1241,15 +1233,20 @@ -- -- This function is called `benchLoop` instead of, say, `go`, -- so it is easier to spot in Core dumps.- benchLoop f x n+ --+ -- Forcing SpecConst optimization with SPEC makes the behaviour of benchmarks+ -- independent of -fspec-constr-count.+ benchLoop :: SPEC -> (a -> b) -> a -> Word64 -> IO ()+ benchLoop !_ f x n | n == 0 = pure () | otherwise = do _ <- evaluate (frc (f x))- benchLoop f x (n - 1)+ benchLoop SPEC f x (n - 1) {-# INLINE funcToBench #-} -- | 'nf' @f@ @x@ measures time to compute--- a normal form (by means of 'force') of an application of @f@ to @x@.+-- a normal form (by means of 'force', not 'Control.DeepSeq.rnf')+-- of an application of @f@ to @x@. -- This does not include time to evaluate @f@ or @x@ themselves. -- Ideally @x@ should be a primitive data type like 'Data.Int.Int'. --@@ -1276,11 +1273,16 @@ -- such measurements are noisy and involve an overhead. Results are more reliable -- when @f@ @x@ takes at least several milliseconds. ----- Note that forcing a normal form requires an additional+-- Remember that forcing a normal form requires an additional -- traverse of the structure. In certain scenarios (imagine benchmarking 'tail'), -- especially when 'NFData' instance is badly written, -- this traversal may take non-negligible time and affect results. --+-- 'nf' @f@ is equivalent to 'whnf' ('force' '.' @f@), but not to+-- 'whnf' ('Control.DeepSeq.rnf' '.' @f@). The former retains the result+-- in memory until it is fully evaluated, while the latter allows+-- evaluated parts of the result to be garbage-collected immediately.+-- -- Drop-in replacement for @Criterion.@'Criterion.nf' and -- @Gauge.@'Gauge.nf'. --@@ -1331,14 +1333,14 @@ {-# INLINE ioToBench #-} -- | 'nfIO' @x@ measures time to evaluate side-effects of @x@--- and compute its normal form (by means of 'force').+-- and compute its normal form (by means of 'force', not 'Control.DeepSeq.rnf'). -- -- Pure subexpression of an effectful computation @x@ -- may be evaluated only once and get cached. -- To avoid surprising results it is usually preferable -- to use 'nfAppIO' instead. ----- Note that forcing a normal form requires an additional+-- Remember that forcing a normal form requires an additional -- traverse of the structure. In certain scenarios, -- especially when 'NFData' instance is badly written, -- this traversal may take non-negligible time and affect results.@@ -1392,8 +1394,8 @@ {-# INLINE ioFuncToBench #-} -- | 'nfAppIO' @f@ @x@ measures time to evaluate side-effects of--- an application of @f@ to @x@.--- and compute its normal form (by means of 'force').+-- an application of @f@ to @x@+-- and compute its normal form (by means of 'force', not 'Control.DeepSeq.rnf'). -- This does not include time to evaluate @f@ or @x@ themselves. -- Ideally @x@ should be a primitive data type like 'Data.Int.Int'. --@@ -1404,7 +1406,7 @@ -- but if @x@ is not a primitive data type, consider forcing its evaluation -- separately, e. g., via 'env' or 'withResource'. ----- Note that forcing a normal form requires an additional+-- Remember that forcing a normal form requires an additional -- traverse of the structure. In certain scenarios, -- especially when 'NFData' instance is badly written, -- this traversal may take non-negligible time and affect results.@@ -1422,7 +1424,7 @@ {-# INLINE nfAppIO #-} -- | 'whnfAppIO' @f@ @x@ measures time to evaluate side-effects of--- an application of @f@ to @x@.+-- an application of @f@ to @x@ -- and compute its weak head normal form. -- This does not include time to evaluate @f@ or @x@ themselves. -- Ideally @x@ should be a primitive data type like 'Data.Int.Int'.@@ -1902,12 +1904,9 @@ , foldAfter = const foldDeps #else , foldGroup = map . first . (++) . (++ ".")-#if MIN_VERSION_tasty(1,2,0) , foldAfter = foldDeps #endif-#endif }-#if MIN_VERSION_tasty(1,2,0) where foldDeps :: DependencyType -> Expr -> [(a, Unique (WithLoHi IM.Key))] -> [(a, Unique (WithLoHi IM.Key))] foldDeps AllSucceed (And (StringLit xs) p)@@ -1921,7 +1920,6 @@ foldMap (\(k, v) -> if withFields v pat == Right True then Unique k else mempty) $ IM.assocs im where pat = eval pattern >>= asB-#endif postprocessResult :: (TestName -> Maybe (WithLoHi Result) -> Result -> Result)@@ -1998,8 +1996,11 @@ -- This helper is useful for bulk application of 'bcompare'. -- See also 'locateBenchmark'. ----- Real world example: https://hackage.haskell.org/package/text-builder-linear-0.1/src/bench/Main.hs+-- Real world examples: --+-- * https://hackage.haskell.org/package/chimera-0.3.3.0/src/bench/Bench.hs+-- * https://hackage.haskell.org/package/text-builder-linear-0.1/src/bench/Main.hs+-- -- @since 0.3.2 mapLeafBenchmarks :: ([String] -> Benchmark -> Benchmark) -> Benchmark -> Benchmark mapLeafBenchmarks processLeaf = go mempty@@ -2011,11 +2012,8 @@ PlusTestOptions g tt -> PlusTestOptions g (go path tt) WithResource res f -> WithResource res (go path . f) AskOptions f -> AskOptions (go path . f)-#if MIN_VERSION_tasty(1,2,0) After dep expr tt -> After dep expr (go path tt)-#endif -#if MIN_VERSION_tasty(1,0,0) -- | Construct an AWK expression to locate an individual element or elements in 'Benchmark' -- by the suffix of the path. Names are listed in reverse order: -- from 'bench'\'s own name to a name of the outermost 'bgroup'.@@ -2024,11 +2022,10 @@ -- 'bcompare' ('Test.Tasty.Patterns.Printer.printAwkExpr' ('locateBenchmark' @path@)). -- See also 'mapLeafBenchmarks'. ----- This function requires @tasty-1.0@, so if you use 'locateBenchmark'--- it is prudent to add @tasty >= 1.0@ to @build-depends@--- section of your cabal file.+-- Real world examples: ----- Real world example: https://hackage.haskell.org/package/text-builder-linear-0.1/src/bench/Main.hs+-- * https://hackage.haskell.org/package/chimera-0.3.3.0/src/bench/Bench.hs+-- * https://hackage.haskell.org/package/text-builder-linear-0.1/src/bench/Main.hs -- -- @since 0.3.2 locateBenchmark :: [String] -> Expr@@ -2037,5 +2034,4 @@ = foldl1' And $ zipWith (\i name -> Patterns.EQ (Field (Sub NF (IntLit i))) (StringLit name)) [0..] path -#endif #endif
tasty-bench.cabal view
@@ -1,5 +1,5 @@ name: tasty-bench-version: 0.3.2+version: 0.3.3 cabal-version: 1.18 build-type: Simple license: MIT@@ -25,7 +25,7 @@ extra-doc-files: example.svg -tested-with: GHC == 9.4.1, GHC == 9.2.4, GHC==9.0.2, GHC==8.10.7, GHC==8.8.4, GHC==8.6.5, GHC==8.4.4, GHC==8.2.2, GHC==8.0.2, GHC==7.10.3, GHC==7.8.4, GHC==7.6.3, GHC==7.4.2, GHC==7.2.2, GHC==7.0.4+tested-with: GHC == 9.6.1 GHC == 9.4.4, GHC == 9.2.5, GHC==9.0.2, GHC==8.10.7, GHC==8.8.4, GHC==8.6.5, GHC==8.4.4, GHC==8.2.2, GHC==8.0.2, GHC==7.10.3, GHC==7.8.4, GHC==7.6.3, GHC==7.4.2, GHC==7.2.2, GHC==7.0.4 source-repository head type: git@@ -56,11 +56,12 @@ build-depends: base >= 4.3 && < 5,- deepseq >= 1.1+ deepseq >= 1.1,+ ghc-prim if flag(tasty) build-depends: containers >= 0.4,- tasty >= 0.11.3+ tasty >= 1.2.3 if impl(ghc < 7.8) build-depends: tagged >= 0.2