tasty-bench 0.2.4 → 0.2.5

raw patch · 4 files changed

+23/−9 lines, 4 files

Files

• README.md +8/−3
• Test/Tasty/Bench.hs +10/−5
• changelog.md +4/−0
• tasty-bench.cabal +1/−1

@@ -110,7 +110,7 @@  The output says that, for instance, the first benchmark was repeatedly executed for 2.13 seconds (wall time),-its mean time was 63 nanoseconds and,+its mean CPU time was 63 nanoseconds and, assuming ideal precision of a system clock, execution time does not often diverge from the mean further than ±3.4 nanoseconds@@ -132,7 +132,7 @@ One might interpret the second line as saying that 95% of measurements fell into 61.99–63.41 ns interval, but this is wrong. It states that the [OLS regression](https://en.wikipedia.org/wiki/Ordinary_least_squares)-of execution time (which is not exactly the mean time) is most probably+(which is not exactly the mean) of wall execution time is most probably somewhere between 61.99 ns and 63.41 ns, but does not say a thing about individual measurements. To understand how far away a typical measurement deviates@@ -151,6 +151,10 @@ for [68%](https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule) of samples only, double it to estimate the behavior in 95% of cases. +When benchmarking multithreaded algorithms, note+that `tasty-bench` reports total elapsed CPU time across all cores, while+`criterion` and `gauge` print wall-clock time.+ ## Statistical model  Here is a procedure used by `tasty-bench` to measure execution time:@@ -191,7 +195,8 @@  ## Memory usage -Passing `+RTS -T` (via `cabal bench --benchmark-options '+RTS -T'`+Configuring RTS to collect GC statistics+(e. g., via `cabal bench --benchmark-options '+RTS -T'` or `stack bench --ba '+RTS -T'`) enables `tasty-bench` to estimate and report memory usage such as allocated and copied bytes: 

@@ -110,7 +110,7 @@ > All 3 tests passed (7.25s)  The output says that, for instance, the first benchmark was repeatedly-executed for 2.13 seconds (wall time), its mean time was 63 nanoseconds+executed for 2.13 seconds (wall time), its mean CPU time was 63 nanoseconds and, assuming ideal precision of a system clock, execution time does not often diverge from the mean further than ±3.4 nanoseconds (double standard deviation, which for normal distributions corresponds to@@ -128,8 +128,8 @@  One might interpret the second line as saying that 95% of measurements fell into 61.99–63.41 ns interval, but this is wrong. It states that the-<https://en.wikipedia.org/wiki/Ordinary_least_squares OLS regression> of-execution time (which is not exactly the mean time) is most probably+<https://en.wikipedia.org/wiki/Ordinary_least_squares OLS regression>+(which is not exactly the mean time) of wall execution time is most probably somewhere between 61.99 ns and 63.41 ns, but does not say a thing about individual measurements. To understand how far away a typical measurement deviates you need to add\/subtract double standard deviation@@ -146,6 +146,10 @@ <https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule 68%> of samples only, double it to estimate the behavior in 95% of cases. +When benchmarking multithreaded algorithms, note+that @tasty-bench@ reports total elapsed CPU time across all cores, while+@criterion@ and @gauge@ print wall-clock time.+ === Statistical model  Here is a procedure used by @tasty-bench@ to measure execution time:@@ -184,7 +188,8 @@  === Memory usage -Passing @+RTS@ @-T@ (via @cabal@ @bench@ @--benchmark-options@ @\'+RTS@ @-T\'@ or+Configuring RTS to collect GC statistics+(e. g., via @cabal@ @bench@ @--benchmark-options@ @\'+RTS@ @-T\'@ or @stack@ @bench@ @--ba@ @\'+RTS@ @-T\'@) enables @tasty-bench@ to estimate and report memory usage such as allocated and copied bytes: @@ -1403,7 +1408,7 @@ testNamesAndDeps :: IntMap (Seq TestName) -> OptionSet -> TestTree -> [(TestName, [IM.Key])] testNamesAndDeps im = foldTestTree trivialFold   { foldSingle = const $ const . (: []) . (, [])-  , foldGroup  = const $ map . first . flip (++) . (++ ".")+  , foldGroup  = const $ map . first . (++) . (++ ".")   , foldAfter  = const foldDeps   }   where

@@ -1,3 +1,7 @@+# 0.2.5++* Fix comparison against baseline.+ # 0.2.4  * Add a simplistic SVG reporter.

@@ -1,5 +1,5 @@ name:          tasty-bench-version:       0.2.4+version:       0.2.5 cabal-version: 1.18 build-type:    Simple license:       MIT