bench-show 0.2.2 → 0.3.0

raw patch · 13 files changed

+637/−204 lines, 13 filesdep +optparse-applicativedep +optparse-simpledep +semigroupsnew-component:exe:bench-showPVP ok

version bump matches the API change (PVP)

Dependencies added: optparse-applicative, optparse-simple, semigroups

API changes (from Hackage documentation)

Files

• Changelog.md +38/−0
• README.md +39/−15
• app/Main.hs +187/−0
• bench-show.cabal +52/−22
• lib/BenchShow.hs +4/−4
• lib/BenchShow/Analysis.hs +1/−1
• lib/BenchShow/Common.hs +223/−114
• lib/BenchShow/Graph.hs +41/−3
• lib/BenchShow/Report.hs +26/−8
• lib/BenchShow/Tutorial.hs +9/−3
• stack.yaml +9/−7
• test/Doc.hs +4/−15
• test/Main.hs +4/−12

@@ -1,3 +1,41 @@+## 0.3.0++### Breaking Changes++* The signature of `selectBenchmarks` has changed, use 'Nothing' as the second+  argument of the benchmark generator function to port old code without any+  impact.+* Removed the broken 'Percent' constructor from `GroupStyle`. Use `PercentDiff`+  instead to make relative comparisons.+* The behavior of `PercentDiff` has changed, it now computes the % from the+  lower value instead of from the baseline.+* The default `diffStrategy` has been changed to `SingleEstimator` instead of+  `MinEstimator`.++### Deprecations++* Config fields `title` and `titleAnnotations` have been deprecated, please use+  `mkTitle` instead.++### Bug Fixes++* `GroupStyle Absolute` now honors the `MinEstimator` setting. When+  `MinEstimator` is set, the groups being compared to baseline now display the+  value based on the estimator which provides closest estimate to the baseline.++### Enhancements++* Add a CLI executable to generate textual reports and graphs from criterion or+  gauge csv output file.+* Add `Multiples` as a comparison option, the group being compared is shown as+  a multiple of the baseline.+* Add ability to omit the baseline group from the results when we are doing a+  relative comparison among groups.+* Add the ability to sort the benchmarks using a different criterion than the+  one used to present the benchmarks in the final report output.+* Add `mkTitle` config option to use a function for generating a custom report+  title.+ ## 0.2.2  * Allow additional annotations to title to be controlled via config

@@ -1,29 +1,45 @@ # bench-show -Generate text reports and graphical charts from the benchmark results-generated by `gauge` or `criterion`, showing or comparing benchmarks in-many useful ways. In a few lines of code, we can report time taken, peak-memory usage, allocations, among many other fields; we can group benchmarks-and compare the groups; we can compare benchmarks before and after a change;-we can show absolute or percentage difference from the baseline; we can sort-the results to get the worst affected benchmarks by percentage change.--It can help us in answering questions like the following, visually or-textually:+Generate text reports and graphical charts from the benchmark results generated+by `gauge` or `criterion` and stored in a CSV file. This tool is especially+useful when you have many benchmarks or if you want to compare benchmarks+across multiple packages. You can generate many interesting reports+including: -* Across two benchmark runs, show all the operations that resulted in a-  regression of more than 10%, so that we can quickly identify and fix-  performance problems in our application.+* Show individual reports for all the fields measured e.g. `time taken`, `peak+  memory usage`, `allocations`, among many other fields measured by `gauge`+* Sort benchmark results on a specified criterion e.g. you may want to see the+  biggest cpu hoggers or biggest memory hoggers on top+* Across two benchmark runs (e.g. before and after a change), show all the+  operations that resulted in a regression of more than x% in descending order,+  so that we can quickly identify and fix performance problems in our+  application. * Across two (or more) packages providing similar functionality, show all the   operations where the performance differs by more than 10%, so that we can   critically analyze the packages and choose the right one.  ## Quick Start -Use `gauge` or `criterion` to generate a `results.csv` file, and then use the-following code to generate a textual report or a graph:+Use `gauge` or `criterion` to generate a `results.csv` file, and then use+either the `bench-show` executable or the library APIs to generate textual or+graphical reports. +## Executable++Use `bench-show` executable with `report` and `graph` sub-commands:+ ```+$ bench-show report results.csv+$ bench-show graph results.csv output+```++For advanced usage, control the generated report by the CLI flags.++## Library++Use `report` and `graph` library functions:++``` report "results.csv"  Nothing defaultConfig graph  "results.csv" "output" defaultConfig ```@@ -37,6 +53,10 @@ can select many fields from a `gauge` raw report.  Units of the fields are automatically determined based on the range of values: +```+$ bench-show --presentation Fields report results.csv+```+ ```haskell report "results.csv" Nothing defaultConfig { presentation = Fields } ```@@ -53,6 +73,10 @@ ```  `graph` generates one bar chart per field:++```+$ bench-show --presentation Fields graph results.csv+```  ``` graph "results.csv" "output" defaultConfig

@@ -0,0 +1,187 @@+{-# LANGUAGE CPP #-}+{-# LANGUAGE RecordWildCards #-}+{-# LANGUAGE TemplateHaskell #-}+{-# LANGUAGE TupleSections #-}++#if __GLASGOW_HASKELL__ >= 800+{-# OPTIONS_GHC -Wno-deprecations #-}+#endif++-- |+-- Module      : Main+-- Copyright   : (c) 2019 Composewell Technologies+--               (c) 2019 Sanchayan Maity+-- License     : BSD3+-- Maintainer  : harendra.kumar@gmail.com+-- Stability   : experimental+-- Portability : GHC+--+module Main where++import Data.Semigroup ((<>))+import Options.Applicative.Simple (addCommand, simpleOptions, simpleVersion)+import Paths_bench_show (version)++import Options.Applicative+import BenchShow++-- The command line interface provides subcommands for each task.  There are+-- some common options that apply to all subcommands. Subcommands may have+-- their own specific options as well.++-------------------------------------------------------------------------------+-- Common config options+-------------------------------------------------------------------------------++pVerbose :: Parser Bool+pVerbose = switch $+       long "verbose"+    <> short 'v'+    <> help "Provide more details in the report output"++pOutputDir :: Parser FilePath+pOutputDir = strOption+  ( long "output-dir"+    <> metavar "DIR"+    <> help "Default is current directory" )++pTitle :: Parser String+pTitle = strOption+  ( long "title"+    <> short 't'+    <> metavar "STRING"+    <> help "Title for the report" )++pTitleAnnotation :: Parser TitleAnnotation+pTitleAnnotation = option auto $+       long "title-annotations"+    <> help ("*TitleField*|TitleEstimator|TitleDiff")++pPresentation :: Parser Presentation+pPresentation = option auto $+       long "presentation"+    <> help ("Solo|Fields|*Groups* <*Absolute*|Diff|PercentDiff|Multiples")++pEstimator :: Parser Estimator+pEstimator = option auto $+       long "estimator"+    <> help ("*Median*|Mean|Regression")++pThreshold :: Parser Word+pThreshold = option auto $+       long "threshold"+    <> metavar "PERCENT"+    <> help "Min % diff (default 3) to flag regression or improvement"++pDiffStrategy :: Parser DiffStrategy+pDiffStrategy = option auto $+       long "diff-strategy"+    <> help ("*SingleEstimator*|MinEstimator")++pOmitBaseline :: Parser Bool+pOmitBaseline = switch $+       long "omit-baseline"+    <> help "omit the baseline column in relative comparisons"++-------------------------------------------------------------------------------+-- Build a Config parser for common options+-------------------------------------------------------------------------------++-- Specify a default value for a Maybe inside a functor+fMaybe :: Functor f => a -> f (Maybe a) -> f a+fMaybe a = fmap (maybe a id)++-- | parse an optional field with a default value taken from defaultConfig+parseOptional :: (Config -> a) -> Parser a -> Parser a+parseOptional def parser = fMaybe (def defaultConfig) (optional parser)++pConfig :: Parser Config+pConfig = Config+    <$> parseOptional verbose pVerbose+    <*> optional pOutputDir+    <*> pure Nothing+    <*> optional pTitle+    <*> many pTitleAnnotation+    <*> parseOptional presentation pPresentation+    <*> parseOptional estimator pEstimator+    <*> parseOptional threshold pThreshold+    <*> parseOptional diffStrategy pDiffStrategy+    <*> parseOptional omitBaseline pOmitBaseline+    <*> pure (selectFields defaultConfig)+    <*> pure (fieldRanges defaultConfig)+    <*> pure (fieldTicks defaultConfig)+    <*> pure (classifyBenchmark defaultConfig)+    <*> pure (selectGroups defaultConfig)+    <*> pure (selectBenchmarks defaultConfig)++-------------------------------------------------------------------------------+-- "report" subcommand+-------------------------------------------------------------------------------++data ReportOpts = ReportOpts+    { reportInput :: FilePath+    , reportOutput :: Maybe FilePath+    }++pInputFile :: Parser FilePath+pInputFile = argument str (metavar "INPUT-FILE.CSV")++pOutputFile :: Parser FilePath+pOutputFile = strOption+  (long "output"+  <> short 'o'+  <> help "Output file")++pReportOpts :: Parser ReportOpts+pReportOpts = ReportOpts+    <$> pInputFile+    <*> optional pOutputFile++cmdReport :: ReportOpts -> Config -> IO ()+cmdReport ReportOpts{..} cfg = report reportInput reportOutput cfg++-------------------------------------------------------------------------------+-- "graph" subcommand+-------------------------------------------------------------------------------++data GraphOpts = GraphOpts+    { graphInput :: FilePath+    , graphOutput :: FilePath+    }++pOutputArg :: Parser FilePath+pOutputArg = argument str (metavar "OUTPUT-FILE-PREFIX")++pGraphOpts :: Parser GraphOpts+pGraphOpts = GraphOpts+    <$> pInputFile+    <*> pOutputArg++cmdGraph :: GraphOpts -> Config -> IO ()+cmdGraph GraphOpts{..} cfg = graph graphInput graphOutput cfg++-------------------------------------------------------------------------------+-- Build and run a subcommand parser+-------------------------------------------------------------------------------++cmdLineParser :: Parser Config -> IO ()+cmdLineParser p = do+    (cfg, handler) <- simpleOptions $(simpleVersion version)+        ("Generate reports and graphs from gauge or criterion output.\n" +++        "Default values are shown as *DEFAULT*.")+        "" p cmds++    handler cfg++    where cmds = do+            addCommand "report"+                       "Generate a text report"+                       cmdReport+                       pReportOpts+            addCommand "graph"+                       "Generate a graphical report"+                       cmdGraph+                       pGraphOpts++main :: IO ()+main = cmdLineParser pConfig

@@ -1,40 +1,45 @@ name:               bench-show-version:            0.2.2+version:            0.3.0 license:            BSD3 author:             Harendra Kumar maintainer:         harendra.kumar@gmail.com bug-reports:        https://github.com/composewell/bench-show/issues synopsis:           Show, plot and compare benchmark results description:-  Generate text reports and graphical charts from benchmark results-  generated by @gauge@ or @criterion@, showing or comparing benchmarks in-  many useful ways. In a few lines of code, we can report time taken, peak-  memory usage, allocations, among many other fields; we can arrange benchmarks-  in groups and compare the groups; we can compare benchmark results before and-  after a change; we can show the difference from baseline in absolute terms or-  as a percentage; we can sort the results by percentage change to get the-  worst affected benchmarks.-  .-  @bench-show@ helps us in answering questions like the following, visually or-  textually:+  Generate text reports and graphical charts from the benchmark results generated+  by @gauge@ or @criterion@ and stored in a CSV file. This tool is especially+  useful when you have many benchmarks or if you want to compare benchmarks+  across multiple packages. You can generate many interesting reports+  including:   .-  * Across two benchmark runs, show all the operations that resulted in a-    regression of more than 10%, so that we can quickly identify and fix-    performance problems in our application.-  * Across two (or more) packages (providing similar functionality), show all-    the operations where the performance differs by more than 10%, so that we-    can critically analyze the packages and choose the right one.+  * Show individual reports for all the fields measured e.g. @time taken@, @peak+    memory usage@, @allocations@, among many other fields measured by @gauge@+  * Sort benchmark results on a specified criterion e.g. you may want to see the+    biggest cpu hoggers or biggest memory hoggers on top+  * Across two benchmark runs (e.g. before and after a change), show all the+    operations that resulted in a regression of more than x% in descending+    order, so that we can quickly identify and fix performance problems in our+    application.+  * Across two (or more) packages providing similar functionality, show all the+    operations where the performance differs by more than 10%, so that we can+    critically analyze the packages and choose the right one.   .   Quick Start: Use @gauge@ or @criterion@ to generate a @results.csv@ file, and-  then use the following code to generate a textual report or a graph:+  then use either the @bench-show@ executable or the library APIs to generate+  textual or graphical reports.   .   @+  $ bench-show report results.csv+  $ bench-show graph results.csv output+  @+  .+  @   report "results.csv"  Nothing defaultConfig   graph  "results.csv" "output" defaultConfig   @   .-  For example, you can show can show % regression from a baseline in descending-  order texttually as follows:+  There are many ways to present the reports, for example, you can show can+  show % regression from a baseline in descending order textually as follows:   .   @   (time)(Median)(Diff using min estimator)@@ -55,7 +60,8 @@ category:            Performance, Benchmarking homepage:            https://github.com/composewell/bench-show license-file:        LICENSE-tested-with:           GHC==8.4.4+tested-with:           GHC==8.6.5+                     , GHC==8.4.4                      , GHC==8.2.2                      , GHC==7.10.3 copyright:           2017, 2018 Composewell Technologies@@ -122,6 +128,30 @@         , split             >= 0.2     && < 0.3         , statistics        >= 0.15    && < 0.16         , vector            >= 0.10    && < 0.13++executable bench-show+  default-language: Haskell2010+  hs-source-dirs: app+  main-is: Main.hs+  other-modules: Paths_bench_show+  ghc-options: -Wall+  build-depends:+          base                 >= 4.8     && < 5+        , Chart                >= 1.6     && < 2+        , Chart-diagrams       >= 1.6     && < 2+        , csv                  >= 0.1     && < 0.2+        , filepath             >= 1.3     && < 1.5+        , mwc-random           >= 0.13    && < 0.15+        , directory            >= 1.2     && < 1.4+        , transformers         >= 0.4     && < 0.6+        , ansi-wl-pprint       >= 0.6     && < 0.7+        , split                >= 0.2     && < 0.3+        , statistics           >= 0.15    && < 0.16+        , vector               >= 0.10    && < 0.13+        , semigroups           >= 0.18    && < 0.20+        , optparse-applicative >= 0.14.2  && < 0.14.4+        , optparse-simple      >= 0.1.0   && < 0.2+        , bench-show  test-suite test     type: exitcode-stdio-1.0

@@ -52,12 +52,12 @@ -- from the textual report, the rows (i.e.  the benchmarks) are represented as -- bars in the cluster. ----- When the columns are groups, each report consists of results for a single+-- When the columns are groups, each report presents the results for a single -- benchmarking field for different benchmark groups.  Using 'GroupStyle', we--- can further specify how we want to present the results the groups. We can+-- can further specify how we want to present the results of the groups. We can -- either present absolute values of the field for each group or we can make--- the first group as a baseline and present differences from the baseline for--- the subsequent groups.+-- the first group as a baseline showing absolute values and present difference+-- from the baseline for the subsequent groups. -- -- When the columns are fields, each report consists of results for a single -- benchmarking group. Fields cannot be compared like groups because they are

@@ -270,7 +270,7 @@                     -- better when larger number of samples are taken.  This                     -- cannot be used when the number of samples is less than                     -- 2, in that case a mean value is reported instead.-    deriving (Eq, Show)+    deriving (Eq, Show, Read)  getAnalyzedValue :: Estimator -> AnalyzedField -> Double getAnalyzedValue estimator AnalyzedField{..} =

@@ -49,7 +49,7 @@ import Data.Function ((&), on) import Data.List        (transpose, groupBy, (\\), find, sortBy, elemIndex, intersect,-        intersectBy, concatMap)+        intersectBy) import Data.List.Split (linesBy) import Data.Maybe (fromMaybe, mapMaybe) import Data.Ord (comparing)@@ -78,25 +78,35 @@         label ++         " must not add any new items to the original list. The \         \following items were added: " ++ show added+        ++ "\nOriginal groups: " ++ show old+        ++ "\nNew groups: " ++ show new  -------------------------------------------------------------------------------  data ReportType = TextReport | GraphicalChart  -- | How to show the results for multiple benchmark groups presented in columns--- or bar chart clusters.+-- or bar chart clusters. In relative comparisons, the first group is+-- considered as the baseline and the subsequent groups are compared against+-- the baseline. --+-- /Definition changed in 0.3.0/ -- @since 0.2.0 data GroupStyle =-      Absolute       -- ^ Show absolute field values for all groups-    | Diff        -- ^ Show baseline group values as usual and values for-                     -- the subsequent groups as differences from the baseline-    | Percent     -- ^ Show baseline group values as 100% and values for-                     -- subsequent groups as a percentage of the baseline-    | PercentDiff -- ^ Show baseline group values as usual and values for-                     -- subsequent groups as precentage difference from the-                     -- baseline-    deriving (Eq, Show)+      Absolute    -- ^ Show absolute values of the field for all groups+    | Diff        -- ^ Show baseline group as absolute values and values for+                  -- the subsequent groups as difference from the baseline+    | PercentDiff -- ^ If the value of the group being compared is higher than+                  -- the baseline then display the difference as percentage of+                  -- baseline otherwise display the difference as a percentage+                  -- of the group being compared.+    | Multiples   -- ^ If the value of the group being compared is higher than+                  -- the baseline then display @+(value / baseline value)@+                  -- otherwise display @-(baseline value / value)@. This+                  -- provides a normalized comparison independent of the+                  -- absolute value of a benchmark. Note that 'Multiples' can+                  -- be directly computed using 'PercentDiff' and vice-versa.+    deriving (Eq, Show, Read)  -- | How to present the reports or graphs. Each report presents a number of -- benchmarks as rows, it may have, (1) a single column presenting the values@@ -124,7 +134,7 @@                         -- with all the fields selected by the configuration as                         -- columns or clusters. Output files are named using                         -- @-estimator-groupname@ as suffix.-    deriving (Eq, Show)+    deriving (Eq, Show, Read)  -- | FieldTick is used only in visual charts to generate the major ticks on -- the y-axis. You can specify either the size of a tick ('TickSize') or the@@ -175,13 +185,15 @@     | WorstBest     | BestBest     -}+    deriving (Show, Read)  -- | Additional annotations that can be optionally added to the title of the -- report or graph. -- -- @since 0.2.2+{-# DEPRECATED TitleAnnotation "Please use mkTitle to make a custom title" #-} data TitleAnnotation = TitleField | TitleEstimator | TitleDiff-    deriving (Eq, Show)+    deriving (Eq, Show, Read)  -- | Configuration governing generation of chart. See 'defaultConfig' for the -- default values of these fields.@@ -197,11 +209,19 @@     -- | The directory where the output graph or report file should be placed.     , outputDir   :: Maybe FilePath -    -- | Report title, more information like the plotted field name or+    -- | Function to make a title for the report. The argument to the function+    -- is the benchmark field name for which the report is being made.+    , mkTitle  :: Maybe (String -> String)++    -- | /DEPRECATED: Please use 'mkTitle' instead./+    --+    -- Report title, more information like the plotted field name or     -- the presentation style may be added to it.     , title  :: Maybe String -    -- | Additional annotations to be added to the title+    -- | /DEPRECATED: Please use 'mkTitle' instead./+    --+    -- Additional annotations to be added to the title     , titleAnnotations :: [TitleAnnotation]      -- | How to determine the layout of the report or the chart.@@ -217,6 +237,10 @@     -- | Strategy to compare two runs or groups of benchmarks.     , diffStrategy  :: DiffStrategy +    -- | Omit the baseline group in normalized relative comparisons i.e.+    -- when the 'GroupStyle' is 'PercentDiff' or 'Multiples'.+    , omitBaseline :: Bool+     ---------------------------------------------------------------------------     -- Fields (Columns)     ---------------------------------------------------------------------------@@ -263,54 +287,68 @@     -- Benchmarks (Rows)     --------------------------------------------------------------------------- -    -- | Filter and reorder benchmarks. 'selectBenchmarks' is provided with a-    -- function which is invoked with a sorting column name or index, the-    -- function produces the benchmark names and corresponding values for that-    -- column which can be used as a sorting criterion. The output of-    -- 'selectBenchmarks' is a list of benchmarks in the order in which they-    -- are to be rendered.+    -- | Filter and reorder benchmarks. 'selectBenchmarks' takes a function+    -- argument, the function is invoked with a sorting column name or index+    -- and a 'GroupStyle'. The output of the function is either a 'Right' value+    -- consisting of tuples of the benchmark names and values corresponding to+    -- the given column and style or a 'Left' value indicating an error.+    -- 'selectBenchmarks' can inspect these benchmarks and there values to+    -- produce a filtered and sorted list of benchmark names that are to be+    -- rendered.+    --+    -- The style argument is ignored when the report presentation is not+    -- 'Groups'. When style is 'Nothing', the presentation setting specified in+    -- the configuration is used.+    --+    -- /Signature changed in 0.3.0/     , selectBenchmarks-        :: (SortColumn -> Either String [(String, Double)])+        :: (SortColumn -> Maybe GroupStyle -> Either String [(String, Double)])         -> [String]     } +-- IMPORTANT: If you change the defaults, please change the defaults in the CLI+-- help as well.+-- -- | Default configuration. Use this as the base configuration and modify the -- required fields. The defaults are: -- -- @ --  verbose           = False---  title             = Nothing+--  mkTitle           = Nothing --  titleAnnotations  = [TitleField] --  outputDir         = Nothing --  presentation      = Groups Absolute --  estimator         = Median --  threshold         = 3---  diffStrategy      = MinEstimator+--  diffStrategy      = SingleEstimator+--  omitBaseline      = False --  selectFields      = filter (flip elem ["time", "mean", "maxrss"] . map toLower) --  fieldRanges       = [] --  fieldTicks        = [] --  classifyBenchmark = Just . ("default",) --  selectGroups      = id---  selectBenchmarks  = \f -> either error (map fst) $ f (ColumnIndex 0)+--  selectBenchmarks  = \f -> either error (map fst) $ f (ColumnIndex 0) Nothing -- @ -- -- @since 0.2.0 defaultConfig :: Config defaultConfig = Config     { verbose           = False+    , mkTitle           = Nothing     , title             = Nothing     , titleAnnotations  = [TitleField]     , outputDir         = Nothing     , presentation      = Groups Absolute     , estimator         = Median     , threshold         = 3-    , diffStrategy      = MinEstimator+    , diffStrategy      = SingleEstimator+    , omitBaseline      = False     , selectFields      = filter (flip elem ["time", "mean", "maxrss"] . map toLower)     , fieldRanges       = []     , fieldTicks        = []     , classifyBenchmark = Just . ("default",)     , selectGroups      = id-    , selectBenchmarks  = \f -> either error (map fst) $ f (ColumnIndex 0)+    , selectBenchmarks  = \f -> either error (map fst) $ f (ColumnIndex 0) Nothing     }  -------------------------------------------------------------------------------@@ -389,8 +427,8 @@ fieldUnits :: String -> Double -> GroupStyle -> RelativeUnit fieldUnits fieldName fieldMin style =     case style of-        Percent      -> RelativeUnit "%" 1-        PercentDiff  -> RelativeUnit "%" 1+        Multiples   -> RelativeUnit "x" 1+        PercentDiff -> RelativeUnit "%" 1         _ -> getUnitByFieldName fieldName fieldMin  -------------------------------------------------------------------------------@@ -400,11 +438,19 @@ absoluteDiff :: Num a => a -> a -> a absoluteDiff v1 v2 = v2 - v1 -percentDiff :: (Fractional a, Num a) => a -> a -> a-percentDiff v1 v2 = ((v2 - v1) * 100) / v1+percentDiff :: (Fractional a, Num a, Ord a) => a -> a -> a+percentDiff v1 v2 = ((v2 - v1) * 100) / min v1 v2 -percent :: (Fractional a, Num a) => a -> a -> a-percent v1 v2 = (v2 * 100) / v1+-- We map a fraction x between 0 and 1 to a negative 1/x for plotting on an+-- equal and opposite scale.+fraction :: (Fractional a, Num a, Ord a, Show a) => a -> a -> a+fraction v1 v2 =+    let val = v2 / v1+    in case val of+            x | x <= 0 -> error $ "BenchShow.Common.fraction: negative: " ++ show x+            x | x < 1 -> negate (1 / x)+            x | x >= 1 -> x+            x -> error $ "BenchShow.Common.fraction: unhandled: " ++ show x  cmpTransformColumns :: ReportType                     -> GroupStyle@@ -426,37 +472,41 @@             let firstCol = head cols                 colTransform col = zipWith (mkMinDiff diff) firstCol col             in map colTransform (tail cols)-    in case style of-            Absolute    -> (Nothing, columns)-            Percent     -> (Nothing, cmpWith percent)-            Diff        ->++        diffWith f =                 case diffStrategy of                     MinEstimator ->-                        let (ests, vals) = unzip $ map unzip (cmpMinWith absoluteDiff)+                        let (ests, vals) = unzip $ map unzip (cmpMinWith f)                         in ( Just $ map (const estimator) (head cols) : ests                            , head columns : vals                            )                     SingleEstimator ->-                        (Nothing, head columns : cmpWith absoluteDiff)-            PercentDiff ->-                -- In a comparative graphical chart we cannot show the absolute-                -- values in the baseline column as the units won't match for-                -- the baseline and the diff clusters.-                let baseCol =-                        case rtype of-                            TextReport -> head columns-                            GraphicalChart | length columns == 1 ->-                                head columns-                            GraphicalChart ->-                                map (\(n,_) -> (n,100)) (head columns)-                in case diffStrategy of-                    MinEstimator ->-                        let (ests, vals) = unzip $ map unzip (cmpMinWith percentDiff)-                        in ( Just $ map (const estimator) (head cols) : ests-                           , baseCol : vals-                           )-                    SingleEstimator ->-                       (Nothing, baseCol : cmpWith percentDiff)+                        (Nothing, head columns : cmpWith f)++        relativeDiffWith baseVal f =+            -- In a comparative graphical chart we cannot show the absolute+            -- values in the baseline column as the units won't match for+            -- the baseline and the diff clusters.+            let baseCol =+                    case rtype of+                        TextReport -> head columns+                        GraphicalChart | length columns == 1 ->+                            head columns+                        GraphicalChart ->+                            map (\(n,_) -> (n,baseVal)) (head columns)+            in case diffStrategy of+                MinEstimator ->+                    let (ests, vals) = unzip $ map unzip (cmpMinWith f)+                    in ( Just $ map (const estimator) (head cols) : ests+                       , baseCol : vals+                       )+                SingleEstimator ->+                   (Nothing, baseCol : cmpWith f)+    in case style of+            Absolute    -> diffWith (\_ x -> x)+            Diff        -> diffWith absoluteDiff+            PercentDiff -> relativeDiffWith 100 percentDiff+            Multiples   -> relativeDiffWith 1 fraction     where         verify a b = if a then b else error "bug: benchmark names mismatch"         transformVals = map (map (second (getAnalyzedValue estimator)))@@ -478,20 +528,28 @@                      then (Mean, (n2, meanDiff))                      else (Regression, (n2, regDiff)) -transformColumnNames :: GroupStyle -> [ReportColumn] -> [ReportColumn]-transformColumnNames _ [] = []-transformColumnNames style columns@(h:t) =-    let withDiff = colSuffix baseName h : map (colSuffix diffName) t+columnNameByStyle :: GroupStyle -> [ReportColumn] -> [ReportColumn]+columnNameByStyle _ [] = []+columnNameByStyle style columns@(h:t) =+    let withDiff name = colSuffix baseName h : map (colSuffix name) t     in case style of-            Diff        | length columns > 1 -> withDiff-            PercentDiff | length columns > 1 -> withDiff-            _           -> columns+            Diff        | length columns > 1 -> withDiff diffName+            Multiples   | length columns > 1 -> withDiff fracName+            PercentDiff | length columns > 1 -> withDiff diffName+            _ -> columns      where     colSuffix xl col = col { colName = xl (colName col) }-    baseName        = (++ "(base)")-    diffName        = (++ "(-base)")+    baseName        = id -- (++ "(base)")+    diffName        = (++ " - " ++ colName h) -- "(-base)")+    fracName        = (++ "/" ++ colName h) -- "/base") +columnNameByUnit :: [RelativeUnit] -> [ReportColumn] -> [ReportColumn]+columnNameByUnit units columns =+    let applyUnit col (RelativeUnit label _) =+            col { colName = colName col ++ inParens label }+    in zipWith applyUnit columns units+ -- Represents the data for a single benchmark run data GroupMatrix = GroupMatrix     { groupIndex :: Int@@ -599,8 +657,9 @@ selectBenchmarksByField :: Config                         -> [GroupMatrix]                         -> [[(String, Double)]]+                        -> (GroupStyle -> [[(String, Double)]])                         -> [String]-selectBenchmarksByField Config{..} matrices columns =+selectBenchmarksByField Config{..} matrices columns colsByStyle =     let bmnames = selectBenchmarks extractGroup     in if (null bmnames)        then error $ "selectBenchmarks must select at least one benchmark"@@ -621,17 +680,22 @@         in map getName matrices      -- columns are benchmark groups in this case-    extractGroup (ColumnName (Left name)) =-            let len = length columns-            in if len <= 1-               then extractGroup $ ColumnName (Right (name, 0))-               else Left $ "selectBenchmarks: there are " ++ show len-                    ++ " runs in the input data, please specify the run \-                    \index [0-" ++ show (len - 1)-                    ++ "] along with the group name."-    extractGroup (ColumnName (Right (name, runId))) =-            extractColumnByGroupName name runId-    extractGroup (ColumnIndex n) = extractColumnByGroupIndex n+    extractGroup colSelector style =+            let cols = case style of+                    Nothing -> columns+                    Just s -> colsByStyle s+            in case colSelector of+                ColumnName (Left name) ->+                    let len = length cols+                    in if len <= 1+                       then extractGroup (ColumnName (Right (name, 0))) style+                       else Left $ "selectBenchmarks: there are " ++ show len+                            ++ " runs in the input data, please specify the run \+                            \index [0-" ++ show (len - 1)+                            ++ "] along with the group name."+                ColumnName (Right (name, runId)) ->+                    extractColumnByGroupName name runId style+                ColumnIndex n -> extractColumnByGroupIndex n cols      -- The benchmark field is constant.  Extract all benchmark values for the     -- given field and for the given group.@@ -645,20 +709,20 @@                     True -> res         in foldl foldFunc (0, False) mxs -    extractColumnByGroupName name runId =+    extractColumnByGroupName name runId style =             case findColumnIndex matrices (name, runId) of                 (_, False) -> Left $ "Benchmark group name [" ++ name                             ++ "] and index [" ++ show runId                             ++ "] not found. Available groups are: "                             ++ show grpNames-                (i, True) -> extractGroup (ColumnIndex i)+                (i, True) -> extractGroup (ColumnIndex i) style -    extractColumnByGroupIndex idx =-        let len = length columns+    extractColumnByGroupIndex idx cols =+        let len = length cols         in if idx >= len            then Left $ "Column index must be in the range [0-"                 ++ show (len - 1) ++ "]"-           else Right $ columns !! idx+           else Right $ cols !! idx  selectBenchmarksByGroup :: Config -> GroupMatrix -> [String] selectBenchmarksByGroup Config{..} grp@GroupMatrix{..} =@@ -671,11 +735,11 @@     where      -- columns are benchmark fields in this case-    extractField (ColumnName (Left name)) = extractColumnByFieldName name-    extractField (ColumnName (Right (name, _))) =+    extractField (ColumnName (Left name)) _ = extractColumnByFieldName name+    extractField (ColumnName (Right (name, _))) _ =         -- XXX runId does not make sense for fields         extractColumnByFieldName name-    extractField (ColumnIndex n) = extractColumnByFieldIndex n+    extractField (ColumnIndex n) _ = extractColumnByFieldIndex n      -- The benchmark field is constant.  Extract all benchmark values for the     -- given field and for the given group.@@ -973,12 +1037,30 @@         unless (null absent) $ putStrLn msg         return matrix { groupBenches = newBenches } -prepareGroupMatrices :: Config -> CSV -> [String] -> IO (Int, [GroupMatrix])-prepareGroupMatrices cfg@Config{..} csvlines fields = do-    let (hdr, runs) =+prepareGroupMatrices :: Config+                     -> FilePath+                     -> CSV+                     -> [String]+                     -> IO (Int, [GroupMatrix])+prepareGroupMatrices cfg@Config{..} inputFile csvlines fields = do+    let res@(_, ls) =               sanityCheckCSV csvlines             & splitRuns-            & ensureIterField++    when (null ls) $+        error $ "No benchmark results found in CSV file: " ++ show inputFile++    let checkForData (runId, xs) =+            when (null xs) $+                putStrLn $ "No benchmark results found in the CSV file ["+                        ++ show inputFile+                        ++ "], for runId: "+                        ++ show runId++    mapM_ checkForData (zip [(0 :: Int)..] ls)++    let (hdr, runs) = ensureIterField res+     xs <- sequence $ map (readIterations hdr) runs             & map (filterFields fields)             -- & _filterCommonSubsets@@ -1064,30 +1146,40 @@          (estimators, transformedCols) =             cmpTransformColumns rtype style estimator diffStrategy unsortedCols-        benchmarks = selectBenchmarksByField cfg matrices transformedCols+        transformedColsByStyle s = snd $+            cmpTransformColumns rtype s estimator diffStrategy unsortedCols++        benchmarks = selectBenchmarksByField cfg matrices+                            transformedCols transformedColsByStyle         sortedCols = map (sortValues benchmarks) transformedCols         origSortedCols = map (sortValues benchmarks) unsortedCols          mkColUnits :: [RelativeUnit]         mkColUnits =             let cols =-                    if style == Diff || style == PercentDiff+                    if style /= Absolute                     -- if we consider diff values as well here then the                     -- units will change to potentially very small.                     then [head sortedCols]                     else sortedCols                 minVal = getFieldMin cfg (minimum $ concat cols) field-            in case (rtype, style) of-                -- In case of percentDiff in TextReport we use absolute-                -- values in the baseline column, so the unit is different.-                (TextReport, PercentDiff) ->+                mkPercentColUnitText =                     let unit = fieldUnits field minVal Absolute                         punit = fieldUnits field 1 style -- % unit                     in unit : replicate (length matrices - 1) punit-                (GraphicalChart, PercentDiff) | length matrices == 1 ->-                    [fieldUnits field minVal Absolute]-                _ -> let unit = fieldUnits field minVal style+                mkPercentColUnitGraph = [fieldUnits field minVal Absolute]+                mkAbsoluteUnit =+                     let unit = fieldUnits field minVal style                      in replicate (length matrices) unit+            in case (rtype, style) of+                -- In case of percentDiff in TextReport we use absolute+                -- values in the baseline column, so the unit is different.+                (_, Absolute)   -> mkAbsoluteUnit+                (_, Diff)       -> mkAbsoluteUnit+                (TextReport, _) -> mkPercentColUnitText+                (GraphicalChart, _) | length matrices == 1 ->+                    mkPercentColUnitGraph+                (GraphicalChart, _) -> mkAbsoluteUnit          mkColValues :: [[Double]]         mkColValues =@@ -1102,33 +1194,47 @@                             if runs > 1                             then "(" ++ show (groupIndex x) ++ ")"                             else ""-                    applyUnit name (RelativeUnit label _) =-                        name ++ inParens label-                in zipWith applyUnit (map withSuffix matrices) mkColUnits+                in map withSuffix matrices          columns = getZipList $ ReportColumn                     <$> ZipList mkColNames                     <*> ZipList mkColUnits                     <*> ZipList mkColValues +        removeBaseline xs =+            let rel = case style of+                        Absolute -> False+                        Diff -> False+                        PercentDiff -> True+                        Multiples -> True+            in if omitBaseline && rel && length matrices > 1+               then tail xs+               else xs+     in RawReport             { reportOutputFile = outfile             , reportIdentifier = field             , reportRowIds     = benchmarks-            , reportColumns    = transformColumnNames style columns-            , reportAnalyzed   = zipWith (\x y -> map (scaleAnalyzedField x) y)-                                         mkColUnits origSortedCols-            , reportEstimators = estimators+            , reportColumns    = removeBaseline+                $ columnNameByUnit mkColUnits+                $ columnNameByStyle style columns+            , reportAnalyzed   = removeBaseline $+                zipWith (\x y -> map (scaleAnalyzedField x) y)+                        mkColUnits origSortedCols+            , reportEstimators = fmap removeBaseline estimators             }  showStatusMessage :: Show a => Config -> String -> Maybe a -> IO () showStatusMessage cfg field outfile =-    let atitle = makeTitle field (diffString (presentation cfg)-                                 (diffStrategy cfg)) cfg+    let atitle = case mkTitle cfg of+                    Just f -> " [" ++ f field ++ "]"+                    Nothing ->+                        " [" ++ makeTitle field (diffString (presentation cfg)+                                 (diffStrategy cfg)) cfg ++ "]"     in case outfile of         Just path ->-            putStrLn $ "Creating chart "-                ++ "[" ++ atitle ++ "]"+            putStrLn $ "Creating chart"+                ++ atitle                 ++ " at "                 ++ show path         Nothing -> return ()@@ -1235,9 +1341,12 @@ diffString :: Presentation -> DiffStrategy -> Maybe String diffString style s =     case style of-        Groups Diff        -> Just $ "Diff " ++ showDiffStrategy s-        Groups PercentDiff -> Just $ "Diff " ++ showDiffStrategy s-        _ -> Nothing+        Groups Diff -> Just $ "Diff from Baseline " ++ showDiffStrategy s+        Groups PercentDiff -> Just $ "Diff % of Lower " ++ showDiffStrategy s+        Groups Multiples -> Just $ "Multiples of Baseline " ++ showDiffStrategy s+        Groups Absolute -> Nothing+        Solo -> Nothing+        Fields -> Nothing  inParens :: String -> String inParens str = "(" ++ str ++ ")"

@@ -64,6 +64,32 @@             }]      else columns +-- We do not want to see the band of values between -1 and 1, in fact there are+-- no values possible in that band.  Shift the positive values by -1 and+-- negative values by +1 to map them to a 0 based scale on the graph. We change+-- the labels as well accordingly.+transformFractionValue :: ReportColumn -> ReportColumn+transformFractionValue ReportColumn{..} =+    ReportColumn+        { colName = colName+        , colUnit = colUnit+        , colValues = map (\val ->+            case val of+                x | x >= 1 -> x - 1+                x | x < (-1) -> x + 1+                x -> error $ "BenchShow.Graph.transformFractionValue: unhandled: " ++ show x+                ) colValues+        }++transformFractionLabels :: LinearAxisParams Double+transformFractionLabels =+    ((def :: (LinearAxisParams Double)) { _la_labelf = \xs ->+        let shiftVals v = if v >= 0 then v + 1 else v - 1+            replaceMinus ('-' : ys) = "1/" ++ ys+            replaceMinus ys = ys+        in fmap replaceMinus (_la_labelf def (map shiftVals xs))+    })+ genGroupGraph :: RawReport -> Config -> IO () genGroupGraph RawReport{..} cfg@Config{..} = do     let outputFile  = fromMaybe undefined reportOutputFile@@ -79,7 +105,9 @@             if length reportColumns > 1             then diffString presentation diffStrategy             else Nothing-        atitle = makeTitle reportIdentifier diffStr cfg+        atitle = case mkTitle of+            Just f -> f reportIdentifier+            Nothing -> makeTitle reportIdentifier diffStr cfg      toFile def outputFile $ do         layout_title .= atitle@@ -88,7 +116,17 @@         layout_x_axis . laxis_generate .=             autoIndexAxis (map (map replaceMu . colName) columns)         layout_x_axis . laxis_style . axis_label_style . font_size .= 16+         layout_y_axis . laxis_style . axis_label_style . font_size .= 14+        -- delete the -1x to 1x band of values which are not possible in case+        -- of fraction style+        cols' <-+                case presentation of+                    Groups Multiples -> do+                        layout_y_axis . laxis_generate .= autoScaledAxis+                                        transformFractionLabels+                        return $ map transformFractionValue columns+                    _ -> return columns          layout <- get         case _layout_legend layout of@@ -120,7 +158,7 @@                              (indexes, [], [])          plot $ fmap plotBars $ bars reportRowIds-            $ (addIndexes $ map colValues columns)+            $ (addIndexes $ map colValues cols')  -- | Presents the benchmark results in a CSV input file as graphical bar charts -- according to the provided configuration.  The first parameter is the input@@ -141,7 +179,7 @@ graph inputFile outputFile cfg@Config{..} = do     let dir = fromMaybe "." outputDir     (csvlines, fields) <- prepareToReport inputFile cfg-    (runs, matrices) <- prepareGroupMatrices cfg csvlines fields+    (runs, matrices) <- prepareGroupMatrices cfg inputFile csvlines fields     case presentation of         Groups style ->             forM_ fields $

@@ -27,6 +27,17 @@ import BenchShow.Common import BenchShow.Analysis +multiplesToPercentDiff :: Double -> Double+multiplesToPercentDiff x = (if x > 0 then x - 1 else x + 1) * 100++colorCode :: Word -> Double -> Doc -> Doc+colorCode thresh x =+    if x > fromIntegral thresh+    then dullred+    else if x < (-1) * fromIntegral thresh+         then dullgreen+         else id+ -- XXX in comparative reports render lower than baseline in green and higher -- than baseline in red genGroupReport :: RawReport -> Config -> IO ()@@ -35,7 +46,10 @@             if length reportColumns > 1             then diffString presentation diffStrategy             else Nothing-    putStrLn $ makeTitle reportIdentifier diffStr cfg+    case mkTitle of+        Just _ -> putStrLn $ maybe "" (\f -> f reportIdentifier) mkTitle+        Nothing -> putStrLn $ makeTitle reportIdentifier diffStr cfg+     let benchcol  = "Benchmark" : reportRowIds         groupcols =             let firstCol : tailCols = reportColumns@@ -43,12 +57,10 @@                     let f x = case presentation of                                 Groups Diff ->                                     if x > 0 then dullred else dullgreen-                                Groups PercentDiff ->-                                    if x > fromIntegral threshold-                                    then dullred-                                    else if x < (-1) * fromIntegral threshold-                                         then dullgreen-                                         else id+                                Groups PercentDiff -> colorCode threshold x+                                Groups Multiples ->+                                    let y = multiplesToPercentDiff x+                                    in colorCode threshold y                                 _ -> id                     in map f colValues                 renderTailCols estimators col analyzed =@@ -59,6 +71,7 @@                     in case presentation of                         Groups Diff        -> colored                         Groups PercentDiff -> colored+                        Groups Multiples   -> colored                         _ -> regular             in renderGroupCol (showFirstCol firstCol)              : case reportEstimators of@@ -133,12 +146,17 @@                 in case presentation of                         Groups Diff        -> showDiff                         Groups PercentDiff -> showDiff+                        Groups Multiples ->+                            if val > 0+                            then printf "%.2f" val+                            else printf "1/%.2f" (negate val)                         _ -> printf "%.2f" val              showEstAnnot est =                 case presentation of                     Groups Diff        -> showEstimator est                     Groups PercentDiff -> showEstimator est+                    Groups Multiples   -> showEstimator est                     _ -> ""          in case estimators of@@ -186,7 +204,7 @@ report inputFile outputFile cfg@Config{..} = do     let dir = fromMaybe "." outputDir     (csvlines, fields) <- prepareToReport inputFile cfg-    (runs, matrices) <- prepareGroupMatrices cfg csvlines fields+    (runs, matrices) <- prepareGroupMatrices cfg inputFile csvlines fields     case presentation of         Groups style ->             forM_ fields $

@@ -8,7 +8,13 @@ -- -- BenchShow generates text reports and graphs from benchmarking results. It -- allows you to manipulate the format of the report and the benchmarking data--- to present it in many useful ways.+-- to present it in many useful ways. BenchShow uses robust statistical+-- analysis using three different statistical estimators to provide as stable+-- run-to-run comparison of benchmark results as possible.  For stable results,+-- make sure that you are not executing any other tasks on the benchmark host+-- while benhmarking is going on.  For even more stable results, we recommend+-- using a desktop or server machine instead of a laptop notebook for+-- benchmarking.  module BenchShow.Tutorial     (@@ -271,7 +277,7 @@ --                     reverse --                   $ map fst --                   $ sortBy (comparing snd)---                   $ either error id $ f $ 'ColumnIndex' 1+--                   $ either error id $ f ('ColumnIndex' 1) Nothing --          } -- @ --@@ -322,7 +328,7 @@ --                     reverse --                   $ map fst --                   $ sortBy (comparing snd)---                   $ either error id $ f $ 'ColumnIndex' 1+--                   $ either error id $ f ('ColumnIndex' 1) Nothing --          } -- @ --

@@ -1,10 +1,12 @@-resolver: lts-12.11+resolver: lts-13.25 packages: - '.' extra-deps:-  - Chart-1.9-  - Chart-diagrams-1.9-  - SVGFonts-1.6.0.3-  - statistics-0.15.0.0-  - math-functions-0.3.0.2-  - dense-linear-algebra-0.1.0.0+  - Chart-1.9.1+  - Chart-diagrams-1.9.2++    #- SVGFonts-1.6.0.3+    #- statistics-0.15.0.0+    #- math-functions-0.3.0.2+    #- dense-linear-algebra-0.1.0.0+    #- optparse-applicative-0.14.2.0

@@ -26,17 +26,6 @@         defaultConfig         { classifyBenchmark = classifier } -    graph "test/results-doc.csv" "docs/grouped-percent"-        defaultConfig-        { classifyBenchmark = classifier-        , presentation = Groups Percent-        }-    report "test/results-doc.csv" Nothing-        defaultConfig-        { classifyBenchmark = classifier-        , presentation = Groups Percent-        }-     graph "test/results-doc.csv" "docs/grouped-delta"         defaultConfig         { classifyBenchmark = classifier@@ -97,7 +86,7 @@                     reverse                   $ map fst                   $ sortBy (comparing snd)-                  $ either error id $ f $ ColumnIndex 1+                  $ either error id $ f (ColumnIndex 1) Nothing         }     report "test/results-doc.csv" Nothing         defaultConfig@@ -108,7 +97,7 @@                     reverse                   $ map fst                   $ sortBy (comparing snd)-                  $ either error id $ f $ ColumnIndex 1+                  $ either error id $ f (ColumnIndex 1) Nothing         }      graph@@ -122,7 +111,7 @@                     reverse                   $ map fst                   $ sortBy (comparing snd)-                  $ either error id $ f $ ColumnIndex 1+                  $ either error id $ f (ColumnIndex 1) Nothing         }     report         "test/results-doc-multi.csv"@@ -135,5 +124,5 @@                     reverse                   $ map fst                   $ sortBy (comparing snd)-                  $ either error id $ f $ ColumnIndex 1+                  $ either error id $ f (ColumnIndex 1) Nothing         }

@@ -73,7 +73,7 @@                 let i = intersect (map (last . splitOn "/") prefixes) bs                 in i ++ (bs \\ i)         cfg = defaultConfig-            { title = Just chartTitle+            { mkTitle = Just (\_ -> chartTitle)             , outputDir = Just "charts"             , classifyBenchmark = \bm ->                 case any (`isPrefixOf` bm) prefixes of@@ -82,7 +82,7 @@                         in Just (suffixVersion (xs !! 0), xs !! 1)                     False -> Nothing             , selectBenchmarks = \g -> bsort $-                either error (map fst) $ g (ColumnIndex 0)+                either error (map fst) $ g (ColumnIndex 0) Nothing             , selectGroups = \gs ->                 let gs' = map fst gs                     i = intersect (map suffixVersion packages) gs'@@ -112,10 +112,6 @@                 , presentation = Groups Diff                 , selectFields = (`intersect` ["time"])                 }-    graph "test/results.csvraw" "csvraw-percent"-            cfg { presentation = Groups Percent-                , selectFields = (`intersect` ["time"])-                }     graph "test/results.csvraw" "csvraw-percent-delta"             cfg { presentation = Groups PercentDiff                 , selectFields = (`intersect` ["time"])@@ -144,18 +140,14 @@             cfg { presentation = Groups Absolute                 , selectBenchmarks = \g ->                     either error (map fst . sortBy (compare `on` snd))-                           (g (ColumnIndex 1))-                }--    report "test/results.csv" Nothing-            cfg { presentation = Groups Percent+                           (g (ColumnIndex 1) Nothing)                 }      report "test/results.csv" Nothing             cfg { presentation = Groups PercentDiff                 , selectBenchmarks = \g ->                     either error (map fst . sortBy (compare `on` snd))-                           (g (ColumnIndex 1))+                           (g (ColumnIndex 1) Nothing)                 }     report "test/results.csv" Nothing             cfg { presentation = Groups Diff