criterion 0.2.0 → 0.3.0

raw patch · 4 files changed

+121/−74 lines, 4 filesPVP ok

version bump matches the API change (PVP)

API changes (from Hackage documentation)

Files

• Criterion/Main.hs +114/−69
• Criterion/Types.hs +4/−3
• criterion.cabal +1/−1
• examples/Tiny.hs +2/−1

@@ -12,15 +12,18 @@  module Criterion.Main     (-    -- * Benchmarking pure code-    -- $eval+    -- * How to write benchmarks+    -- $bench -    -- ** Let-floating-    -- $letfloat+    -- ** Benchmarking IO actions+    -- $io -    -- ** Worker-wrapper transformation-    -- $worker+    -- ** Benchmarking pure code+    -- $pure +    -- ** Fully evaluating a result+    -- $rnf+     -- * Types       Benchmarkable(..)     , Benchmark@@ -191,9 +194,9 @@ -- > fib n = fib (n-1) + fib (n-2) -- > -- > main = defaultMain [--- >        bgroup "fib" [ bench "fib 10" $ \n -> fib (10+n-n))--- >                     , bench "fib 35" $ \n -> fib (35+n-n))--- >                     , bench "fib 37" $ \n -> fib (37+n-n))+-- >        bgroup "fib" [ bench "fib 10" $ B fib 10+-- >                     , bench "fib 35" $ B fib 35+-- >                     , bench "fib 37" $ B fib 37 -- >                     ] -- >                    ] defaultMain :: [Benchmark] -> IO ()@@ -213,7 +216,7 @@ -- >            } -- >  -- > main = defaultMainWith myConfig [--- >          bench "fib 30" $ \(n::Int) -> fib (30+n-n)+-- >          bench "fib 30" $ B fib 30 -- >        ] -- -- If you save the above example as @\"Fib.hs\"@, you should be able@@ -247,83 +250,125 @@   printError "Run \"%s --help\" for usage information\n" =<< getProgName   exitWith (ExitFailure 64) --- $eval+-- $bench ----- Because GHC optimises aggressively when compiling with @-O@, it is--- easy to write innocent-looking benchmark code that will only be--- evaluated once, for which all but the first iteration of the timing--- loop will be timing the cost of doing nothing.+-- The 'Benchmarkable' typeclass represents the class of all code that+-- can be benchmarked.  Every instance must run a benchmark a given+-- number of times.  We are most interested in benchmarking two things: ----- The 'Int' parameter that is passed into your benchmark function is--- important: you'll almost certainly need to use it somehow in order--- to ensure that your code will not get optimised away.+-- * 'IO' actions.  Any 'IO' action can be benchmarked directly.+--+-- * Pure functions.  GHC optimises aggressively when compiling with+--   @-O@, so it is easy to write innocent-looking benchmark code that+--   doesn't measure the performance of a pure function at all.  We+--   work around this by benchmarking both a function and its final+--   argument together. --- $letfloat+-- $io ----- The following is an example of innocent-looking code that will not--- benchmark correctly:+-- Any 'IO' action can be benchmarked easily if its type resembles+-- this: ----- > b = bench "fib 10" $ \(_::Int) -> fib 10+-- @+-- 'IO' a+-- @++-- $pure ----- GHC will notice that the body is constant, and use let-floating to--- transform the function into a form more like this:+-- Because GHC optimises aggressively when compiling with @-O@, it is+-- potentially easy to write innocent-looking benchmark code that will+-- only be evaluated once, for which all but the first iteration of+-- the timing loop will be timing the cost of doing nothing. ----- > lvl = fib 10--- > b = bench "fib 10" $ \(::_Int) -> lvl+-- To work around this, we provide two types for benchmarking pure+-- code.  The first is a specialised tuple: ----- Here, it is obvious that the CAF @lvl@ only needs to be evaluated--- once, and this is indeed what happens.  The first iteration in the--- timing loop will measure a realistic time. All other iterations--- will take a few dozen nanoseconds, since the original thunk for--- @lvl@ has already been overwritten with the result of its first--- evaluation.+-- @+-- data 'B' a = forall b. 'B' (a -> b) a+-- @ ----- One somewhat unreliable way to defeat let-floating is to disable it:+-- The second is a specialised tuple named 'B': ----- > {-# OPTIONS_GHC -fno-full-laziness #-}+-- @+-- (a -> b, a)+-- @ ----- If you are trying to benchmark an inlined function, turning off the--- let-floating transformation may end up causing slower code to be--- generated.+-- As both of these types suggest, when you want to benchmark a+-- function, you must supply two values: ----- A much more reliable way to defeat let-floating is to find a way to--- make use of the 'Int' that the benchmarking code passes in.+-- * The first element is the function, saturated with all but its+--   last argument. ----- > bench "fib 10" $ \n -> fib (10+n-n)+-- * The second is the last argument to the function. ----- GHC is not yet smart enough to see that adding and subtracting @n@--- amounts to a no-op. This trick is enough to convince it not to--- let-float the function's body out, since the body is no longer--- constant.---- $worker+-- In practice, it is much easier to use the 'B' tuple than a normal+-- tuple.  Using 'B', the type checker can see when the function type+-- @a -> b@ and its argument type @a@ are the same, whereas code may+-- require an explicit type annotation to make this connection+-- explicit for a regular tuple.  Here is an example that makes the+-- distinction clearer.  Suppose we want to benchmark the following+-- function: ----- Another GHC optimisation is worker-wrapper transformation.  Suppose--- you want to time insertion of key\/value pairs into a map.  You--- might perform the insertion via a (/strict/!) fold:+-- @+-- firstN :: Int -> [Int]+-- firstN k = take k [(0::Int)..]+-- @ ----- > import qualified Data.IntMap as I--- > import Data.List (foldl')--- >--- > intmap :: Int -> I.IntMap Int--- > intmap n = foldl' (\m k -> I.insert k k m) I.empty [0..n]--- >--- > b = bench "intmap 10k" $ \(_::Int) -> intmap 10000+-- So in the easy case, we construct a benchmark as follows: ----- Compile this /without/ @-fno-full-laziness@, and the body of the--- anonymous function we're benchmarking gets let-floated out to the--- top level.+-- @+-- 'B' firstN 1000+-- @ ----- > lvl = intmap 10000--- > b = bench "intmap 10k" $ \(_::Int) -> lvl+-- The compiler will correctly infer that the number 1000 must have+-- the type 'Int', and the type of the expression is ----- Compile it /with/ @-fno-full-laziness@, and let-floating occurs--- /anyway/, this time due to GHC's worker-wrapper transformation.+-- @+-- 'B' ['Int'] 'Int'+-- @ ----- Once again, the response is to use the parameter that the--- benchmarking code passes in.+-- However, say we try to construct a benchmark using a tuple, as+-- follows: ----- > intmap :: Int -> Int -> I.IntMap Int--- > intmap n i = foldl' (\m k -> I.insert k k m) I.empty [0..n+i-i]--- >--- > b = bench "intmap 10k" $ intmap 10000+-- @+-- (firstN, 1000)+-- @+--+-- Since we have written a numeric literal with no explicit type, the+-- compiler will correctly infer a rather general type for this+-- expression:+--+-- @+-- ('Num' a) => ('Int' -> ['Int'], a)+-- @+--+-- This does not match the type @(a -> b, a)@, so we would have to+-- explicitly annotate the number @1000@ as having the type @'Int'@+-- for the typechecker to accept this as a valid benchmarkable+-- expression.++-- $rnf+--+-- The harness for evaluating a pure function only evaluates the+-- result to weak head normal form (WHNF).  If you need the result+-- evaluated all the way to normal form, use the @rnf@ function from+-- the Control.Parallel.Strategies module to force its complete+-- evaluation.+--+-- Using the @firstN@ example from earlier, to naive eyes it /appears/+-- that the following code ought to benchmark the production of the+-- first 1000 list elements:+--+-- @+-- B firstN 1000+-- @+--+-- Because the result is only forced until WHNF is reached, what this+-- /actually/ benchmarks is merely the production of the first list+-- element!  Here is a corrected version:+--+-- @+-- B (rnf . firstN) 1000+-- @++

@@ -1,3 +1,5 @@+{-# LANGUAGE ExistentialQuantification, FlexibleInstances, GADTs #-}+ -- | -- Module      : Criterion.Types -- Copyright   : (c) Bryan O'Sullivan 2009@@ -21,7 +23,6 @@ -- For an action of type @IO a@, the benchmarking harness calls the -- action repeatedly, but does not reduce the result. -{-# LANGUAGE FlexibleInstances, GADTs #-} module Criterion.Types     (       Benchmarkable(..)@@ -43,7 +44,7 @@  -- | A container for a pure function to benchmark, and an argument to -- supply to it each time it is evaluated.-data B a b = B (a -> b) a+data B a = forall b. B (a -> b) a  instance Benchmarkable (a -> b, a) where     run fx@(f,x) n@@ -51,7 +52,7 @@         | otherwise = evaluate (f x) >> run fx (n-1)     {-# INLINE run #-} -instance Benchmarkable (B a b) where+instance Benchmarkable (B a) where     run fx@(B f x) n         | n <= 0    = return ()         | otherwise = evaluate (f x) >> run fx (n-1)

@@ -1,5 +1,5 @@ name:           criterion-version:        0.2.0+version:        0.3.0 synopsis:       Robust, reliable performance measurement and analysis license:        BSD3 license-file:   LICENSE

@@ -13,7 +13,8 @@            }  main = defaultMainWith myConfig [-         bench "fib 30" $ B fib 30+         bench "fib 10" $ B fib 10+       , bench "fib 30" $ B fib 30        , bench "intmap 50k" $ B intmap 50000        , bench "intmap 75k" $ B intmap 75000        ]