diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# tasty-bench [![Hackage](http://img.shields.io/hackage/v/tasty-bench.svg)](https://hackage.haskell.org/package/tasty-bench) +# tasty-bench [![Hackage](http://img.shields.io/hackage/v/tasty-bench.svg)](https://hackage.haskell.org/package/tasty-bench) [![Stackage Nightly](http://stackage.org/package/tasty-bench/badge/nightly)](http://stackage.org/nightly/package/tasty-bench) Featherlight benchmark framework (only one file!) for performance measurement with API mimicking [`criterion`](http://hackage.haskell.org/package/criterion) @@ -7,7 +7,7 @@ ## How lightweight is it? -There is only one source file `Test.Tasty.Bench` and no external dependencies +There is only one source file `Test.Tasty.Bench` and no non-boot dependencies except [`tasty`](http://hackage.haskell.org/package/tasty). So if you already depend on `tasty` for a test suite, there is nothing else to install. @@ -61,6 +61,7 @@ benchmark bench-fibo main-is: BenchFibo.hs type: exitcode-stdio-1.0 + ghc-options: "-with-rtsopts=-A32m" build-depends: base, tasty-bench ``` @@ -254,21 +255,117 @@ ## Troubleshooting -If benchmark results look malformed like below, make sure that you are invoking -`Test.Tasty.Bench.defaultMain` and not `Test.Tasty.defaultMain` (the difference -is `consoleBenchReporter` vs. `consoleTestReporter`): +* If benchmarks take too long, set `--timeout` to limit execution time + of individual benchmarks, and `tasty-bench` will do its best to fit + into a given time frame. Without `--timeout` we rerun benchmarks until + achieving a target precision set by `--stdev`, which in a noisy environment + of a modern laptop with GUI may take a lot of time. + While `criterion` runs each benchmark at least for 5 seconds, + `tasty-bench` is happy to conclude earlier, if it does not compromise + the quality of results. In our experiments `tasty-bench` suites + tend to finish earlier, even if some individual benchmarks + take longer than with `criterion`. + + A common source of noisiness is garbage collection. Setting a larger + allocation area (_nursery_) is often a good idea, either via + `cabal bench --benchmark-options '+RTS -A32m'` or `stack bench --ba '+RTS -A32m'`. + Alternatively bake it into + `cabal` file as `ghc-options: "-with-rtsopts=-A32m"`. + +* If benchmark results look malformed like below, make sure that you are + invoking `Test.Tasty.Bench.defaultMain` and not `Test.Tasty.defaultMain` + (the difference is `consoleBenchReporter` vs. `consoleTestReporter`): + + ``` + All + fibo 20: OK (1.46s) + Response {respEstimate = Estimate {estMean = Measurement {measTime = 87496728, measAllocs = 0, measCopied = 0}, estStdev = 694487}, respIfSlower = FailIfSlower Infinity, respIfFaster = FailIfFaster Infinity} + ``` + +* If benchmarks fail with an error message + + ``` + Unhandled resource. Probably a bug in the runner you're using. + ``` + + this is probably caused by `env` or `envWithCleanup` affecting benchmarks structure. + You can use `env` to read test data from `IO`, but not to read benchmark names + or affect their hierarchy in other way. This is a fundamental restriction of `tasty` + to list and filter benchmarks without launching missiles. + +## Isolating interfering benchmarks + +One difficulty of benchmarking in Haskell is that it is +hard to isolate benchmarks so that they do not interfere. +Changing the order of benchmarks or skipping some of them +has an effect on heap's layout and thus affects garbage collection. +This issue is well attested in +[`both`](https://github.com/haskell/criterion/issues/166) +[`criterion`](https://github.com/haskell/criterion/issues/60) +and +[`gauge`](https://github.com/vincenthz/hs-gauge/issues/2). + +Usually (but not always) skipping some benchmarks speeds up remaining ones. +That's because once a benchmark allocated heap which for some reason +was not promptly released afterwards (e. g., it forced a top-level thunk +in an underlying library), all further benchmarks are slowed down +by garbage collector processing this additional amount of live data +over and over again. + +There are several mitigation strategies. First of all, giving garbage collector +more breathing space by `+RTS -A32m` (or more) is often good enough. + +Further, avoid using top-level bindings to store large test data. Once such thunks +are forced, they remain allocated forever, which affects detrimentally subsequent +unrelated benchmarks. Treat them as external data, supplied via `env`: instead of + +```haskell +largeData :: String +largeData = replicate 1000000 'a' + +main :: IO () +main = defaultMain + [ bench "large" $ nf length largeData, ... ] ``` -All - fibo 20: OK (1.46s) - Response {respEstimate = Estimate {estMean = Measurement {measTime = 87496728, measAllocs = 0, measCopied = 0}, estStdev = 694487}, respIfSlower = FailIfSlower Infinity, respIfFaster = FailIfFaster Infinity} + +use + +```haskell +import Control.DeepSeq (force) +import Control.Exception (evaluate) + +main :: IO () +main = defaultMain + [ env (evaluate (force (replicate 1000000 'a'))) $ \largeData -> + bench "large" $ nf length largeData, ... ] ``` +Finally, as an ultimate measure to reduce interference between benchmarks, +one can run each of them in a separate process. We do not quite recommend +this approach, but if you are desperate, here is how. + +Assuming that a benchmark is declared in `cabal` file as `benchmark my-bench` component, +let's first find its executable: + +```bash +cabal build --enable-benchmarks my-bench +MYBENCH=`cabal list-bin my-bench` +``` + +Now list all benchmark names (hopefully, they do not contain newlines), +escape quotes and slashes, and run each of them separately: + +```bash +$MYBENCH -l | sed -e 's/[\"]/\\\\\\&/g' | while read name; do $MYBENCH -p '$0 == "'$name'"'; done +``` + ## Comparison against baseline One can compare benchmark results against an earlier baseline in an automatic way. To use this feature, first run `tasty-bench` with `--csv FILE` key -to dump results to `FILE` in CSV format: +to dump results to `FILE` in CSV format +(it could be a good idea to set smaller `--stdev`, if possible): ``` Name,Mean (ps),2*Stdev (ps) diff --git a/Test/Tasty/Bench.hs b/Test/Tasty/Bench.hs --- a/Test/Tasty/Bench.hs +++ b/Test/Tasty/Bench.hs @@ -11,7 +11,7 @@ === How lightweight is it? -There is only one source file "Test.Tasty.Bench" and no external +There is only one source file "Test.Tasty.Bench" and no non-boot dependencies except [@tasty@](http://hackage.haskell.org/package/tasty). So if you already depend on @tasty@ for a test suite, there is nothing else to install. @@ -65,6 +65,7 @@ > benchmark bench-fibo > main-is: BenchFibo.hs > type: exitcode-stdio-1.0 +> ghc-options: "-with-rtsopts=-A32m" > build-depends: base, tasty-bench And here is @BenchFibo.hs@: @@ -242,19 +243,107 @@ === Troubleshooting -If benchmark results look malformed like below, make sure that you are -invoking 'Test.Tasty.Bench.defaultMain' and not 'Test.Tasty.defaultMain' -(the difference is 'consoleBenchReporter' vs. 'consoleTestReporter'): +- If benchmarks take too long, set @--timeout@ to limit execution time + of individual benchmarks, and @tasty-bench@ will do its best to fit + into a given time frame. Without @--timeout@ we rerun benchmarks until + achieving a target precision set by @--stdev@, which in a noisy + environment of a modern laptop with GUI may take a lot of time. -> All -> fibo 20: OK (1.46s) -> Response {respEstimate = Estimate {estMean = Measurement {measTime = 87496728, measAllocs = 0, measCopied = 0}, estStdev = 694487}, respIfSlower = FailIfSlower Infinity, respIfFaster = FailIfFaster Infinity} + While @criterion@ runs each benchmark at least for 5 seconds, + @tasty-bench@ is happy to conclude earlier, if it does not + compromise the quality of results. In our experiments @tasty-bench@ + suites tend to finish earlier, even if some individual benchmarks + take longer than with @criterion@. + A common source of noisiness is garbage collection. Setting a larger + allocation area (/nursery/) is often a good idea, either via + @cabal@ @bench@ @--benchmark-options@ @\'+RTS@ @-A32m\'@ or + @stack@ @bench@ @--ba@ @\'+RTS@ @-A32m\'@. Alternatively bake it into @cabal@ + file as @ghc-options:@ @\"-with-rtsopts=-A32m\"@. + +- If benchmark results look malformed like below, make sure that you + are invoking 'Test.Tasty.Bench.defaultMain' and not + 'Test.Tasty.defaultMain' (the difference is 'consoleBenchReporter' + vs. 'consoleTestReporter'): + + > All + > fibo 20: OK (1.46s) + > Response {respEstimate = Estimate {estMean = Measurement {measTime = 87496728, measAllocs = 0, measCopied = 0}, estStdev = 694487}, respIfSlower = FailIfSlower Infinity, respIfFaster = FailIfFaster Infinity} + +- If benchmarks fail with an error message + + > Unhandled resource. Probably a bug in the runner you're using. + + this is probably caused by 'env' or 'envWithCleanup' affecting + benchmarks structure. You can use 'env' to read test data from 'IO', + but not to read benchmark names or affect their hierarchy in other + way. This is a fundamental restriction of @tasty@ to list and filter + benchmarks without launching missiles. + +=== Isolating interfering benchmarks + +One difficulty of benchmarking in Haskell is that it is hard to isolate +benchmarks so that they do not interfere. Changing the order of +benchmarks or skipping some of them has an effect on heap’s layout and +thus affects garbage collection. This issue is well attested in + + and +. + +Usually (but not always) skipping some benchmarks speeds up remaining +ones. That’s because once a benchmark allocated heap which for some +reason was not promptly released afterwards (e. g., it forced a +top-level thunk in an underlying library), all further benchmarks are +slowed down by garbage collector processing this additional amount of +live data over and over again. + +There are several mitigation strategies. First of all, giving garbage +collector more breathing space by @+RTS@ @-A32m@ (or more) is often good +enough. + +Further, avoid using top-level bindings to store large test data. Once +such thunks are forced, they remain allocated forever, which affects +detrimentally subsequent unrelated benchmarks. Treat them as external +data, supplied via 'env': instead of + +> largeData :: String +> largeData = replicate 1000000 'a' +> +> main :: IO () +> main = defaultMain +> [ bench "large" $ nf length largeData, ... ] + +use + +> import Control.DeepSeq (force) +> import Control.Exception (evaluate) +> +> main :: IO () +> main = defaultMain +> [ env (evaluate (force (replicate 1000000 'a'))) $ \largeData -> +> bench "large" $ nf length largeData, ... ] + +Finally, as an ultimate measure to reduce interference between +benchmarks, one can run each of them in a separate process. We do not +quite recommend this approach, but if you are desperate, here is how. + +Assuming that a benchmark is declared in @cabal@ file as +@benchmark@ @my-bench@ component, let’s first find its executable: + +> cabal build --enable-benchmarks my-bench +> MYBENCH=`cabal list-bin my-bench` + +Now list all benchmark names (hopefully, they do not contain newlines), +escape quotes and slashes, and run each of them separately: + +> $MYBENCH -l | sed -e 's/[\"]/\\\\\\&/g' | while read name; do $MYBENCH -p '$0 == "'$name'"'; done + === Comparison against baseline One can compare benchmark results against an earlier baseline in an automatic way. To use this feature, first run @tasty-bench@ with -@--csv@ @FILE@ key to dump results to @FILE@ in CSV format: +@--csv@ @FILE@ key to dump results to @FILE@ in CSV format +(it could be a good idea to set smaller @--stdev@, if possible): > Name,Mean (ps),2*Stdev (ps) > All.fibonacci numbers.fifth,48453,4060 @@ -371,8 +460,8 @@ import Prelude hiding (Int, Integer) import Control.Applicative -import Control.DeepSeq -import Control.Exception +import Control.DeepSeq (NFData, force) +import Control.Exception (bracket, evaluate) import Control.Monad (void, unless, guard, (>=>)) import Data.Data (Typeable) import Data.Foldable (foldMap, traverse_) @@ -402,6 +491,7 @@ import Test.Tasty.Providers import Test.Tasty.Runners import Text.Printf +import System.Exit import System.IO import System.IO.Unsafe @@ -689,6 +779,13 @@ -- This does not include time to evaluate @f@ or @x@ themselves. -- Ideally @x@ should be a primitive data type like 'Data.Int.Int'. -- +-- The same thunk of @x@ is shared by multiple calls of @f@. We cannot evaluate +-- @x@ beforehand: there is no 'NFData' @a@ constraint, and potentialy @x@ may +-- be an infinite structure. Thus @x@ will be evaluated in course of the first +-- application of @f@. This noisy measurement is to be discarded soon, +-- but if @x@ is not a primitive data type, consider forcing its evaluation +-- separately, e. g., via 'env' or 'withResource'. +-- -- Here is a textbook antipattern: 'nf' 'sum' @[1..1000000]@. -- Since an input list is shared by multiple invocations of 'sum', -- it will be allocated in memory in full, putting immense pressure @@ -721,6 +818,13 @@ -- This does not include time to evaluate @f@ or @x@ themselves. -- Ideally @x@ should be a primitive data type like 'Data.Int.Int'. -- +-- The same thunk of @x@ is shared by multiple calls of @f@. We cannot evaluate +-- @x@ beforehand: there is no 'NFData' @a@ constraint, and potentialy @x@ may +-- be an infinite structure. Thus @x@ will be evaluated in course of the first +-- application of @f@. This noisy measurement is to be discarded soon, +-- but if @x@ is not a primitive data type, consider forcing its evaluation +-- separately, e. g., via 'env' or 'withResource'. +-- -- Computing only a weak head normal form is -- rarely what intuitively is meant by "evaluation". -- Beware that many educational materials contain examples with 'whnf': @@ -762,8 +866,9 @@ -- this traversal may take non-negligible time and affect results. -- -- A typical use case is 'nfIO' ('readFile' @"foo.txt"@). --- However, if you need I\/O only to read input data from a file, --- consider using 'env'. +-- However, if your goal is not to benchmark I\/O per se, +-- but just read input data from a file, it is cleaner to +-- use 'env' or 'withResource'. -- -- Drop-in replacement for 'Criterion.nfIO' and 'Gauge.nfIO'. -- @@ -783,8 +888,10 @@ -- Unless you understand precisely, what is measured, -- it is recommended to use 'nfIO' instead. -- --- Lazy I\/O is treacherous. If you need I\/O only --- to read input data from a file, consider using 'env'. +-- Lazy I\/O is treacherous. +-- If your goal is not to benchmark I\/O per se, +-- but just read input data from a file, it is cleaner to +-- use 'env' or 'withResource'. -- -- Drop-in replacement for 'Criterion.whnfIO' and 'Gauge.whnfIO'. -- @@ -809,14 +916,22 @@ -- This does not include time to evaluate @f@ or @x@ themselves. -- Ideally @x@ should be a primitive data type like 'Data.Int.Int'. -- +-- The same thunk of @x@ is shared by multiple calls of @f@. We cannot evaluate +-- @x@ beforehand: there is no 'NFData' @a@ constraint, and potentialy @x@ may +-- be an infinite structure. Thus @x@ will be evaluated in course of the first +-- application of @f@. This noisy measurement is to be discarded soon, +-- but if @x@ is not a primitive data type, consider forcing its evaluation +-- separately, e. g., via 'env' or 'withResource'. +-- -- Note that forcing a normal form requires an additional -- traverse of the structure. In certain scenarios, -- especially when 'NFData' instance is badly written, -- this traversal may take non-negligible time and affect results. -- -- A typical use case is 'nfAppIO' 'readFile' @"foo.txt"@. --- However, if you need I\/O only to read input data from a file, --- consider using 'env'. +-- However, if your goal is not to benchmark I\/O per se, +-- but just read input data from a file, it is cleaner to +-- use 'env' or 'withResource'. -- -- Drop-in replacement for 'Criterion.nfAppIO' and 'Gauge.nfAppIO'. -- @@ -830,13 +945,22 @@ -- This does not include time to evaluate @f@ or @x@ themselves. -- Ideally @x@ should be a primitive data type like 'Data.Int.Int'. -- +-- The same thunk of @x@ is shared by multiple calls of @f@. We cannot evaluate +-- @x@ beforehand: there is no 'NFData' @a@ constraint, and potentialy @x@ may +-- be an infinite structure. Thus @x@ will be evaluated in course of the first +-- application of @f@. This noisy measurement is to be discarded soon, +-- but if @x@ is not a primitive data type, consider forcing its evaluation +-- separately, e. g., via 'env' or 'withResource'. +-- -- Computing only a weak head normal form is -- rarely what intuitively is meant by "evaluation". -- Unless you understand precisely, what is measured, -- it is recommended to use 'nfAppIO' instead. -- --- Lazy I\/O is treacherous. If you need I\/O only --- to read input data from a file, consider using 'env'. +-- Lazy I\/O is treacherous. +-- If your goal is not to benchmark I\/O per se, +-- but just read input data from a file, it is cleaner to +-- use 'env' or 'withResource'. -- -- Drop-in replacement for 'Criterion.whnfAppIO' and 'Gauge.whnfAppIO'. -- @@ -852,9 +976,39 @@ -- dangling in the heap causes longer garbage collection -- and slows down all benchmarks, even those which do not use it at all. -- --- Provided only for the sake of compatibility with 'Criterion.env' and 'Gauge.env', +-- It is instrumental not only for proper 'IO' actions, +-- but also for a large statically-known data as well. Instead of a top-level +-- definition, which once evaluated will slow down garbage collection +-- during all subsequent benchmarks, +-- +-- > largeData :: String +-- > largeData = replicate 1000000 'a' +-- > +-- > main :: IO () +-- > main = defaultMain +-- > [ bench "large" $ nf length largeData, ... ] +-- +-- use +-- +-- > import Control.DeepSeq (force) +-- > import Control.Exception (evaluate) +-- > +-- > main :: IO () +-- > main = defaultMain +-- > [ env (evaluate (force (replicate 1000000 'a'))) $ \largeData -> +-- > bench "large" $ nf length largeData, ... ] +-- +-- 'env' is provided only for the sake of compatibility with 'Criterion.env' and 'Gauge.env', -- and involves 'unsafePerformIO'. Consider using 'withResource' instead. -- +-- 'defaultMain' requires that the hierarchy of benchmarks and their names is +-- independent of underlying 'IO' actions. While executing 'IO' inside 'bench' +-- via 'nfIO' is fine, and reading test data from files via 'env' is also fine, +-- using 'env' to choose benchmarks or their names depending on 'IO' side effects +-- will throw a rather cryptic error message: +-- +-- > Unhandled resource. Probably a bug in the runner you're using. +-- env :: NFData env => IO env -> (env -> Benchmark) -> Benchmark env res = envWithCleanup res (const $ pure ()) @@ -887,9 +1041,15 @@ csvReporter = TestReporter [Option (Proxy :: Proxy (Maybe CsvPath))] $ \opts tree -> do CsvPath path <- lookupOption opts - let names = IM.fromDistinctAscList $ zip [0..] (testsNames opts tree) + let names = testsNames opts tree + namesMap = IM.fromDistinctAscList $ zip [0..] names pure $ \smap -> do - let augmented = IM.intersectionWith (,) names smap + case lookupRepeatingElements names of + Nothing -> pure () + Just name -> do -- 'die' is not available before base-4.8 + hPutStrLn stderr $ "CSV report cannot proceed, because name '" ++ name ++ "' corresponds to two or more benchmarks. Please disambiguate them." + exitFailure + let augmented = IM.intersectionWith (,) namesMap smap hasGCStats <- getRTSStatsEnabled bracket (do @@ -903,6 +1063,14 @@ (`csvOutput` augmented) pure $ const ((== 0) . statFailures <$> computeStatistics smap) +lookupRepeatingElements :: Ord a => [a] -> Maybe a +lookupRepeatingElements = go S.empty + where + go _ [] = Nothing + go acc (x : xs) + | x `S.member` acc = Just x + | otherwise = go (S.insert x acc) xs + csvOutput :: Handle -> IntMap (TestName, TVar Status) -> IO () csvOutput h = traverse_ $ \(name, tv) -> do hasGCStats <- getRTSStatsEnabled @@ -956,7 +1124,7 @@ && fromIntegral slowDown >= -100 * ifFast -- | Return slow down in percents. -compareVsBaseline :: S.Set TestName -> TestName -> Estimate -> Int64 +compareVsBaseline :: S.Set String -> TestName -> Estimate -> Int64 compareVsBaseline baseline name (Estimate m stdev) = case mOld of Nothing -> 0 Just (oldTime, oldDoubleSigma) @@ -966,9 +1134,20 @@ where time :: Int64 time = fromIntegral $ measTime m + + mOld :: Maybe (Int64, Int64) mOld = do let prefix = encodeCsv name ++ "," - line <- lookupGE prefix baseline + (line, furtherLines) <- S.minView $ snd $ S.split prefix baseline + + case S.minView furtherLines of + Nothing -> pure () + Just (nextLine, _) -> case stripPrefix prefix nextLine of + Nothing -> pure () + -- If there are several lines matching prefix, skip them all. + -- Should not normally happen, 'csvReporter' prohibits repeating test names. + Just{} -> Nothing + (timeCell, ',' : rest) <- span (/= ',') <$> stripPrefix prefix line let doubleSigmaCell = takeWhile (/= ',') rest (,) <$> safeRead timeCell <*> safeRead doubleSigmaCell @@ -981,11 +1160,6 @@ forceFail :: Result -> Result forceFail r = r { resultOutcome = Failure TestFailed, resultShortDescription = "FAIL" } - -#if !MIN_VERSION_containers(0,5,0) -lookupGE :: TestName -> S.Set TestName -> Maybe TestName -lookupGE x = fmap fst . S.minView . S.filter (x `isPrefixOf`) -#endif modifyConsoleReporter :: [OptionDescription] -> (OptionSet -> IO (TestName -> Result -> Result)) -> Ingredient modifyConsoleReporter desc' iof = TestReporter (desc ++ desc') $ \opts tree -> diff --git a/changelog.md b/changelog.md --- a/changelog.md +++ b/changelog.md @@ -1,3 +1,7 @@ +# 0.2.3 + +* Prohibit duplicated benchmark names in CSV reports. + # 0.2.2 * Remove `NFData` constraint from `whnfIO`. diff --git a/tasty-bench.cabal b/tasty-bench.cabal --- a/tasty-bench.cabal +++ b/tasty-bench.cabal @@ -1,5 +1,5 @@ name: tasty-bench -version: 0.2.2 +version: 0.2.3 cabal-version: >=1.10 build-type: Simple license: MIT @@ -21,7 +21,7 @@ changelog.md README.md -tested-with: GHC==9.0.1, GHC==8.10.3, GHC==8.8.4, GHC==8.6.5, GHC==8.4.4, GHC==8.2.2, GHC==8.0.2, GHC==7.10.3, GHC==7.8.4, GHC==7.6.3, GHC==7.4.2, GHC==7.2.2, GHC==7.0.4 +tested-with: GHC==9.0.1, GHC==8.10.4, GHC==8.8.4, GHC==8.6.5, GHC==8.4.4, GHC==8.2.2, GHC==8.0.2, GHC==7.10.3, GHC==7.8.4, GHC==7.6.3, GHC==7.4.2, GHC==7.2.2, GHC==7.0.4 source-repository head type: git