tasty-bench 0.2.2 → 0.2.3

raw patch · 4 files changed

+314/−39 lines, 4 filesPVP ok

version bump matches the API change (PVP)

API changes (from Hackage documentation)

Files

• README.md +106/−9
• Test/Tasty/Bench.hs +202/−28
• changelog.md +4/−0
• tasty-bench.cabal +2/−2

@@ -1,4 +1,4 @@-# tasty-bench [![Hackage](http://img.shields.io/hackage/v/tasty-bench.svg)](https://hackage.haskell.org/package/tasty-bench)+# tasty-bench [![Hackage](http://img.shields.io/hackage/v/tasty-bench.svg)](https://hackage.haskell.org/package/tasty-bench) [![Stackage Nightly](http://stackage.org/package/tasty-bench/badge/nightly)](http://stackage.org/nightly/package/tasty-bench)  Featherlight benchmark framework (only one file!) for performance measurement with API mimicking [`criterion`](http://hackage.haskell.org/package/criterion)@@ -7,7 +7,7 @@  ## How lightweight is it? -There is only one source file `Test.Tasty.Bench` and no external dependencies+There is only one source file `Test.Tasty.Bench` and no non-boot dependencies except [`tasty`](http://hackage.haskell.org/package/tasty). So if you already depend on `tasty` for a test suite, there is nothing else to install.@@ -61,6 +61,7 @@ benchmark bench-fibo   main-is:       BenchFibo.hs   type:          exitcode-stdio-1.0+  ghc-options:   "-with-rtsopts=-A32m"   build-depends: base, tasty-bench ``` @@ -254,21 +255,117 @@  ## Troubleshooting -If benchmark results look malformed like below, make sure that you are invoking-`Test.Tasty.Bench.defaultMain` and not `Test.Tasty.defaultMain` (the difference-is `consoleBenchReporter` vs. `consoleTestReporter`):+* If benchmarks take too long, set `--timeout` to limit execution time+  of individual benchmarks, and `tasty-bench` will do its best to fit+  into a given time frame. Without `--timeout` we rerun benchmarks until+  achieving a target precision set by `--stdev`, which in a noisy environment+  of a modern laptop with GUI may take a lot of time. +  While `criterion` runs each benchmark at least for 5 seconds,+  `tasty-bench` is happy to conclude earlier, if it does not compromise+  the quality of results. In our experiments `tasty-bench` suites+  tend to finish earlier, even if some individual benchmarks+  take longer than with `criterion`.++  A common source of noisiness is garbage collection. Setting a larger+  allocation area (_nursery_) is often a good idea, either via+  `cabal bench --benchmark-options '+RTS -A32m'` or `stack bench --ba '+RTS -A32m'`.+  Alternatively bake it into+  `cabal` file as `ghc-options: "-with-rtsopts=-A32m"`.++* If benchmark results look malformed like below, make sure that you are+  invoking `Test.Tasty.Bench.defaultMain` and not `Test.Tasty.defaultMain`+  (the difference is `consoleBenchReporter` vs. `consoleTestReporter`):++  ```+  All+    fibo 20:       OK (1.46s)+      Response {respEstimate = Estimate {estMean = Measurement {measTime = 87496728, measAllocs = 0, measCopied = 0}, estStdev = 694487}, respIfSlower = FailIfSlower Infinity, respIfFaster = FailIfFaster Infinity}+  ```++* If benchmarks fail with an error message++  ```+  Unhandled resource. Probably a bug in the runner you're using.+  ```++  this is probably caused by `env` or `envWithCleanup` affecting benchmarks structure.+  You can use `env` to read test data from `IO`, but not to read benchmark names+  or affect their hierarchy in other way. This is a fundamental restriction of `tasty`+  to list and filter benchmarks without launching missiles.++## Isolating interfering benchmarks++One difficulty of benchmarking in Haskell is that it is+hard to isolate benchmarks so that they do not interfere.+Changing the order of benchmarks or skipping some of them+has an effect on heap's layout and thus affects garbage collection.+This issue is well attested in+[`both`](https://github.com/haskell/criterion/issues/166)+[`criterion`](https://github.com/haskell/criterion/issues/60)+and+[`gauge`](https://github.com/vincenthz/hs-gauge/issues/2).++Usually (but not always) skipping some benchmarks speeds up remaining ones.+That's because once a benchmark allocated heap which for some reason+was not promptly released afterwards (e. g., it forced a top-level thunk+in an underlying library), all further benchmarks are slowed down+by garbage collector processing this additional amount of live data+over and over again.++There are several mitigation strategies. First of all, giving garbage collector+more breathing space by `+RTS -A32m` (or more) is often good enough.++Further, avoid using top-level bindings to store large test data. Once such thunks+are forced, they remain allocated forever, which affects detrimentally subsequent+unrelated benchmarks. Treat them as external data, supplied via `env`: instead of++```haskell+largeData :: String+largeData = replicate 1000000 'a'++main :: IO ()+main = defaultMain+  [ bench "large" $ nf length largeData, ... ] ```-All-  fibo 20:       OK (1.46s)-    Response {respEstimate = Estimate {estMean = Measurement {measTime = 87496728, measAllocs = 0, measCopied = 0}, estStdev = 694487}, respIfSlower = FailIfSlower Infinity, respIfFaster = FailIfFaster Infinity}++use++```haskell+import Control.DeepSeq (force)+import Control.Exception (evaluate)++main :: IO ()+main = defaultMain+  [ env (evaluate (force (replicate 1000000 'a'))) $ \largeData ->+    bench "large" $ nf length largeData, ... ] ``` +Finally, as an ultimate measure to reduce interference between benchmarks,+one can run each of them in a separate process. We do not quite recommend+this approach, but if you are desperate, here is how.++Assuming that a benchmark is declared in `cabal` file as `benchmark my-bench` component,+let's first find its executable:++```bash+cabal build --enable-benchmarks my-bench+MYBENCH=`cabal list-bin my-bench`+```++Now list all benchmark names (hopefully, they do not contain newlines),+escape quotes and slashes, and run each of them separately:++```bash+$MYBENCH -l | sed -e 's/[\"]/\\\\\\&/g' | while read name; do $MYBENCH -p '$0 == "'$name'"'; done+```+ ## Comparison against baseline  One can compare benchmark results against an earlier baseline in an automatic way. To use this feature, first run `tasty-bench` with `--csv FILE` key-to dump results to `FILE` in CSV format:+to dump results to `FILE` in CSV format+(it could be a good idea to set smaller `--stdev`, if possible):  ``` Name,Mean (ps),2*Stdev (ps)

@@ -11,7 +11,7 @@  === How lightweight is it? -There is only one source file "Test.Tasty.Bench" and no external+There is only one source file "Test.Tasty.Bench" and no non-boot dependencies except [@tasty@](http://hackage.haskell.org/package/tasty). So if you already depend on @tasty@ for a test suite, there is nothing else to install.@@ -65,6 +65,7 @@ > benchmark bench-fibo >   main-is:       BenchFibo.hs >   type:          exitcode-stdio-1.0+>   ghc-options:   "-with-rtsopts=-A32m" >   build-depends: base, tasty-bench  And here is @BenchFibo.hs@:@@ -242,19 +243,107 @@  === Troubleshooting -If benchmark results look malformed like below, make sure that you are-invoking 'Test.Tasty.Bench.defaultMain' and not 'Test.Tasty.defaultMain'-(the difference is 'consoleBenchReporter' vs. 'consoleTestReporter'):+-   If benchmarks take too long, set @--timeout@ to limit execution time+    of individual benchmarks, and @tasty-bench@ will do its best to fit+    into a given time frame. Without @--timeout@ we rerun benchmarks until+    achieving a target precision set by @--stdev@, which in a noisy+    environment of a modern laptop with GUI may take a lot of time. -> All->   fibo 20:       OK (1.46s)->     Response {respEstimate = Estimate {estMean = Measurement {measTime = 87496728, measAllocs = 0, measCopied = 0}, estStdev = 694487}, respIfSlower = FailIfSlower Infinity, respIfFaster = FailIfFaster Infinity}+    While @criterion@ runs each benchmark at least for 5 seconds,+    @tasty-bench@ is happy to conclude earlier, if it does not+    compromise the quality of results. In our experiments @tasty-bench@+    suites tend to finish earlier, even if some individual benchmarks+    take longer than with @criterion@. +    A common source of noisiness is garbage collection. Setting a larger+    allocation area (/nursery/) is often a good idea, either via+    @cabal@ @bench@ @--benchmark-options@ @\'+RTS@ @-A32m\'@ or+    @stack@ @bench@ @--ba@ @\'+RTS@ @-A32m\'@. Alternatively bake it into @cabal@+    file as @ghc-options:@ @\"-with-rtsopts=-A32m\"@.++-   If benchmark results look malformed like below, make sure that you+    are invoking 'Test.Tasty.Bench.defaultMain' and not+    'Test.Tasty.defaultMain' (the difference is 'consoleBenchReporter'+    vs. 'consoleTestReporter'):++    > All+    >   fibo 20:       OK (1.46s)+    >     Response {respEstimate = Estimate {estMean = Measurement {measTime = 87496728, measAllocs = 0, measCopied = 0}, estStdev = 694487}, respIfSlower = FailIfSlower Infinity, respIfFaster = FailIfFaster Infinity}++-   If benchmarks fail with an error message++    > Unhandled resource. Probably a bug in the runner you're using.++    this is probably caused by 'env' or 'envWithCleanup' affecting+    benchmarks structure. You can use 'env' to read test data from 'IO',+    but not to read benchmark names or affect their hierarchy in other+    way. This is a fundamental restriction of @tasty@ to list and filter+    benchmarks without launching missiles.++=== Isolating interfering benchmarks++One difficulty of benchmarking in Haskell is that it is hard to isolate+benchmarks so that they do not interfere. Changing the order of+benchmarks or skipping some of them has an effect on heap’s layout and+thus affects garbage collection. This issue is well attested in+<https://github.com/haskell/criterion/issues/166 both>+<https://github.com/haskell/criterion/issues/60 criterion> and+<https://github.com/vincenthz/hs-gauge/issues/2 gauge>.++Usually (but not always) skipping some benchmarks speeds up remaining+ones. That’s because once a benchmark allocated heap which for some+reason was not promptly released afterwards (e. g., it forced a+top-level thunk in an underlying library), all further benchmarks are+slowed down by garbage collector processing this additional amount of+live data over and over again.++There are several mitigation strategies. First of all, giving garbage+collector more breathing space by @+RTS@ @-A32m@ (or more) is often good+enough.++Further, avoid using top-level bindings to store large test data. Once+such thunks are forced, they remain allocated forever, which affects+detrimentally subsequent unrelated benchmarks. Treat them as external+data, supplied via 'env': instead of++> largeData :: String+> largeData = replicate 1000000 'a'+>+> main :: IO ()+> main = defaultMain+>   [ bench "large" $ nf length largeData, ... ]++use++> import Control.DeepSeq (force)+> import Control.Exception (evaluate)+>+> main :: IO ()+> main = defaultMain+>   [ env (evaluate (force (replicate 1000000 'a'))) $ \largeData ->+>     bench "large" $ nf length largeData, ... ]++Finally, as an ultimate measure to reduce interference between+benchmarks, one can run each of them in a separate process. We do not+quite recommend this approach, but if you are desperate, here is how.++Assuming that a benchmark is declared in @cabal@ file as+@benchmark@ @my-bench@ component, let’s first find its executable:++> cabal build --enable-benchmarks my-bench+> MYBENCH=`cabal list-bin my-bench`++Now list all benchmark names (hopefully, they do not contain newlines),+escape quotes and slashes, and run each of them separately:++> $MYBENCH -l | sed -e 's/[\"]/\\\\\\&/g' | while read name; do $MYBENCH -p '$0 == "'$name'"'; done+ === Comparison against baseline  One can compare benchmark results against an earlier baseline in an automatic way. To use this feature, first run @tasty-bench@ with-@--csv@ @FILE@ key to dump results to @FILE@ in CSV format:+@--csv@ @FILE@ key to dump results to @FILE@ in CSV format+(it could be a good idea to set smaller @--stdev@, if possible):  > Name,Mean (ps),2*Stdev (ps) > All.fibonacci numbers.fifth,48453,4060@@ -371,8 +460,8 @@  import Prelude hiding (Int, Integer) import Control.Applicative-import Control.DeepSeq-import Control.Exception+import Control.DeepSeq (NFData, force)+import Control.Exception (bracket, evaluate) import Control.Monad (void, unless, guard, (>=>)) import Data.Data (Typeable) import Data.Foldable (foldMap, traverse_)@@ -402,6 +491,7 @@ import Test.Tasty.Providers import Test.Tasty.Runners import Text.Printf+import System.Exit import System.IO import System.IO.Unsafe @@ -689,6 +779,13 @@ -- This does not include time to evaluate @f@ or @x@ themselves. -- Ideally @x@ should be a primitive data type like 'Data.Int.Int'. --+-- The same thunk of @x@ is shared by multiple calls of @f@. We cannot evaluate+-- @x@ beforehand: there is no 'NFData' @a@ constraint, and potentialy @x@ may+-- be an infinite structure. Thus @x@ will be evaluated in course of the first+-- application of @f@. This noisy measurement is to be discarded soon,+-- but if @x@ is not a primitive data type, consider forcing its evaluation+-- separately, e. g., via 'env' or 'withResource'.+-- -- Here is a textbook antipattern: 'nf' 'sum' @[1..1000000]@. -- Since an input list is shared by multiple invocations of 'sum', -- it will be allocated in memory in full, putting immense pressure@@ -721,6 +818,13 @@ -- This does not include time to evaluate @f@ or @x@ themselves. -- Ideally @x@ should be a primitive data type like 'Data.Int.Int'. --+-- The same thunk of @x@ is shared by multiple calls of @f@. We cannot evaluate+-- @x@ beforehand: there is no 'NFData' @a@ constraint, and potentialy @x@ may+-- be an infinite structure. Thus @x@ will be evaluated in course of the first+-- application of @f@. This noisy measurement is to be discarded soon,+-- but if @x@ is not a primitive data type, consider forcing its evaluation+-- separately, e. g., via 'env' or 'withResource'.+-- -- Computing only a weak head normal form is -- rarely what intuitively is meant by "evaluation". -- Beware that many educational materials contain examples with 'whnf':@@ -762,8 +866,9 @@ -- this traversal may take non-negligible time and affect results. -- -- A typical use case is 'nfIO' ('readFile' @"foo.txt"@).--- However, if you need I\/O only to read input data from a file,--- consider using 'env'.+-- However, if your goal is not to benchmark I\/O per se,+-- but just read input data from a file, it is cleaner to+-- use 'env' or 'withResource'. -- -- Drop-in replacement for 'Criterion.nfIO' and 'Gauge.nfIO'. --@@ -783,8 +888,10 @@ -- Unless you understand precisely, what is measured, -- it is recommended to use 'nfIO' instead. ----- Lazy I\/O is treacherous. If you need I\/O only--- to read input data from a file, consider using 'env'.+-- Lazy I\/O is treacherous.+-- If your goal is not to benchmark I\/O per se,+-- but just read input data from a file, it is cleaner to+-- use 'env' or 'withResource'. -- -- Drop-in replacement for 'Criterion.whnfIO' and 'Gauge.whnfIO'. --@@ -809,14 +916,22 @@ -- This does not include time to evaluate @f@ or @x@ themselves. -- Ideally @x@ should be a primitive data type like 'Data.Int.Int'. --+-- The same thunk of @x@ is shared by multiple calls of @f@. We cannot evaluate+-- @x@ beforehand: there is no 'NFData' @a@ constraint, and potentialy @x@ may+-- be an infinite structure. Thus @x@ will be evaluated in course of the first+-- application of @f@. This noisy measurement is to be discarded soon,+-- but if @x@ is not a primitive data type, consider forcing its evaluation+-- separately, e. g., via 'env' or 'withResource'.+-- -- Note that forcing a normal form requires an additional -- traverse of the structure. In certain scenarios, -- especially when 'NFData' instance is badly written, -- this traversal may take non-negligible time and affect results. -- -- A typical use case is 'nfAppIO' 'readFile' @"foo.txt"@.--- However, if you need I\/O only to read input data from a file,--- consider using 'env'.+-- However, if your goal is not to benchmark I\/O per se,+-- but just read input data from a file, it is cleaner to+-- use 'env' or 'withResource'. -- -- Drop-in replacement for 'Criterion.nfAppIO' and 'Gauge.nfAppIO'. --@@ -830,13 +945,22 @@ -- This does not include time to evaluate @f@ or @x@ themselves. -- Ideally @x@ should be a primitive data type like 'Data.Int.Int'. --+-- The same thunk of @x@ is shared by multiple calls of @f@. We cannot evaluate+-- @x@ beforehand: there is no 'NFData' @a@ constraint, and potentialy @x@ may+-- be an infinite structure. Thus @x@ will be evaluated in course of the first+-- application of @f@. This noisy measurement is to be discarded soon,+-- but if @x@ is not a primitive data type, consider forcing its evaluation+-- separately, e. g., via 'env' or 'withResource'.+-- -- Computing only a weak head normal form is -- rarely what intuitively is meant by "evaluation". -- Unless you understand precisely, what is measured, -- it is recommended to use 'nfAppIO' instead. ----- Lazy I\/O is treacherous. If you need I\/O only--- to read input data from a file, consider using 'env'.+-- Lazy I\/O is treacherous.+-- If your goal is not to benchmark I\/O per se,+-- but just read input data from a file, it is cleaner to+-- use 'env' or 'withResource'. -- -- Drop-in replacement for 'Criterion.whnfAppIO' and 'Gauge.whnfAppIO'. --@@ -852,9 +976,39 @@ -- dangling in the heap causes longer garbage collection -- and slows down all benchmarks, even those which do not use it at all. ----- Provided only for the sake of compatibility with 'Criterion.env' and 'Gauge.env',+-- It is instrumental not only for proper 'IO' actions,+-- but also for a large statically-known data as well. Instead of a top-level+-- definition, which once evaluated will slow down garbage collection+-- during all subsequent benchmarks,+--+-- > largeData :: String+-- > largeData = replicate 1000000 'a'+-- >+-- > main :: IO ()+-- > main = defaultMain+-- >   [ bench "large" $ nf length largeData, ... ]+--+-- use+--+-- > import Control.DeepSeq (force)+-- > import Control.Exception (evaluate)+-- >+-- > main :: IO ()+-- > main = defaultMain+-- >   [ env (evaluate (force (replicate 1000000 'a'))) $ \largeData ->+-- >     bench "large" $ nf length largeData, ... ]+--+-- 'env' is provided only for the sake of compatibility with 'Criterion.env' and 'Gauge.env', -- and involves 'unsafePerformIO'. Consider using 'withResource' instead. --+-- 'defaultMain' requires that the hierarchy of benchmarks and their names is+-- independent of underlying 'IO' actions. While executing 'IO' inside 'bench'+-- via 'nfIO' is fine, and reading test data from files via 'env' is also fine,+-- using 'env' to choose benchmarks or their names depending on 'IO' side effects+-- will throw a rather cryptic error message:+--+-- > Unhandled resource. Probably a bug in the runner you're using.+-- env :: NFData env => IO env -> (env -> Benchmark) -> Benchmark env res = envWithCleanup res (const $ pure ()) @@ -887,9 +1041,15 @@ csvReporter = TestReporter [Option (Proxy :: Proxy (Maybe CsvPath))] $   \opts tree -> do     CsvPath path <- lookupOption opts-    let names = IM.fromDistinctAscList $ zip [0..] (testsNames opts tree)+    let names = testsNames opts tree+        namesMap = IM.fromDistinctAscList $ zip [0..] names     pure $ \smap -> do-      let augmented = IM.intersectionWith (,) names smap+      case lookupRepeatingElements names of+        Nothing -> pure ()+        Just name -> do -- 'die' is not available before base-4.8+          hPutStrLn stderr $ "CSV report cannot proceed, because name '" ++ name ++ "' corresponds to two or more benchmarks. Please disambiguate them."+          exitFailure+      let augmented = IM.intersectionWith (,) namesMap smap       hasGCStats <- getRTSStatsEnabled       bracket         (do@@ -903,6 +1063,14 @@         (`csvOutput` augmented)       pure $ const ((== 0) . statFailures <$> computeStatistics smap) +lookupRepeatingElements :: Ord a => [a] -> Maybe a+lookupRepeatingElements = go S.empty+  where+    go _ [] = Nothing+    go acc (x : xs)+      | x `S.member` acc = Just x+      | otherwise = go (S.insert x acc) xs+ csvOutput :: Handle -> IntMap (TestName, TVar Status) -> IO () csvOutput h = traverse_ $ \(name, tv) -> do   hasGCStats <- getRTSStatsEnabled@@ -956,7 +1124,7 @@           && fromIntegral slowDown >= -100 * ifFast  -- | Return slow down in percents.-compareVsBaseline :: S.Set TestName -> TestName -> Estimate -> Int64+compareVsBaseline :: S.Set String -> TestName -> Estimate -> Int64 compareVsBaseline baseline name (Estimate m stdev) = case mOld of   Nothing -> 0   Just (oldTime, oldDoubleSigma)@@ -966,9 +1134,20 @@   where     time :: Int64     time = fromIntegral $ measTime m++    mOld :: Maybe (Int64, Int64)     mOld = do       let prefix = encodeCsv name ++ ","-      line <- lookupGE prefix baseline+      (line, furtherLines) <- S.minView $ snd $ S.split prefix baseline++      case S.minView furtherLines of+        Nothing -> pure ()+        Just (nextLine, _) -> case stripPrefix prefix nextLine of+          Nothing -> pure ()+          -- If there are several lines matching prefix, skip them all.+          -- Should not normally happen, 'csvReporter' prohibits repeating test names.+          Just{}  -> Nothing+       (timeCell, ',' : rest) <- span (/= ',') <$> stripPrefix prefix line       let doubleSigmaCell = takeWhile (/= ',') rest       (,) <$> safeRead timeCell <*> safeRead doubleSigmaCell@@ -981,11 +1160,6 @@  forceFail :: Result -> Result forceFail r = r { resultOutcome = Failure TestFailed, resultShortDescription = "FAIL" }--#if !MIN_VERSION_containers(0,5,0)-lookupGE :: TestName -> S.Set TestName -> Maybe TestName-lookupGE x = fmap fst . S.minView . S.filter (x `isPrefixOf`)-#endif  modifyConsoleReporter :: [OptionDescription] -> (OptionSet -> IO (TestName -> Result -> Result)) -> Ingredient modifyConsoleReporter desc' iof = TestReporter (desc ++ desc') $ \opts tree ->

@@ -1,3 +1,7 @@+# 0.2.3++* Prohibit duplicated benchmark names in CSV reports.+ # 0.2.2  * Remove `NFData` constraint from `whnfIO`.

@@ -1,5 +1,5 @@ name:          tasty-bench-version:       0.2.2+version:       0.2.3 cabal-version: >=1.10 build-type:    Simple license:       MIT@@ -21,7 +21,7 @@   changelog.md   README.md -tested-with: GHC==9.0.1, GHC==8.10.3, GHC==8.8.4, GHC==8.6.5, GHC==8.4.4, GHC==8.2.2, GHC==8.0.2, GHC==7.10.3, GHC==7.8.4, GHC==7.6.3, GHC==7.4.2, GHC==7.2.2, GHC==7.0.4+tested-with: GHC==9.0.1, GHC==8.10.4, GHC==8.8.4, GHC==8.6.5, GHC==8.4.4, GHC==8.2.2, GHC==8.0.2, GHC==7.10.3, GHC==7.8.4, GHC==7.6.3, GHC==7.4.2, GHC==7.2.2, GHC==7.0.4  source-repository head   type: git