summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorharendra <>2019-07-25 18:19:00 (GMT)
committerhdiff <hdiff@hdiff.luite.com>2019-07-25 18:19:00 (GMT)
commitf3e9fe295a50373de989350dc12c6409b20f383c (patch)
tree3d05f4161e039004c9a5d3c83521041955661b1e
parent88774e278ced7b380a363f5d1d9f4d165f54cee0 (diff)
version 0.3.00.3.0
-rwxr-xr-x[-rw-r--r--]Changelog.md38
-rwxr-xr-x[-rw-r--r--]README.md56
-rw-r--r--app/Main.hs187
-rw-r--r--bench-show.cabal74
-rwxr-xr-x[-rw-r--r--]docs/full-median-time.svg0
-rwxr-xr-x[-rw-r--r--]docs/grouped-delta-median-time.svg0
-rwxr-xr-x[-rw-r--r--]docs/grouped-median-time.svg0
-rwxr-xr-x[-rw-r--r--]docs/grouped-percent-delta-coeff-time.svg0
-rwxr-xr-x[-rw-r--r--]docs/grouped-percent-delta-median-time.svg0
-rwxr-xr-x[-rw-r--r--]docs/grouped-percent-delta-sorted-median-time.svg0
-rwxr-xr-x[-rw-r--r--]docs/grouped-single-estimator-coeff-time.svg0
-rwxr-xr-x[-rw-r--r--]docs/regression-percent-descending-median-time.svg0
-rw-r--r--lib/BenchShow.hs8
-rw-r--r--lib/BenchShow/Analysis.hs2
-rw-r--r--lib/BenchShow/Common.hs337
-rw-r--r--lib/BenchShow/Graph.hs44
-rw-r--r--lib/BenchShow/Report.hs34
-rw-r--r--lib/BenchShow/Tutorial.hs12
-rwxr-xr-x[-rw-r--r--]stack-8.2.yaml0
-rwxr-xr-x[-rw-r--r--]stack.yaml16
-rw-r--r--test/Doc.hs19
-rw-r--r--test/Main.hs16
-rwxr-xr-x[-rw-r--r--]test/results-doc-multi.csv0
-rwxr-xr-x[-rw-r--r--]test/results-doc.csv0
-rwxr-xr-x[-rw-r--r--]test/results.csv0
-rwxr-xr-x[-rw-r--r--]test/results.csvraw0
26 files changed, 638 insertions, 205 deletions
diff --git a/Changelog.md b/Changelog.md
index c352c6a..011c622 100644..100755
--- a/Changelog.md
+++ b/Changelog.md
@@ -1,3 +1,41 @@
+## 0.3.0
+
+### Breaking Changes
+
+* The signature of `selectBenchmarks` has changed, use 'Nothing' as the second
+ argument of the benchmark generator function to port old code without any
+ impact.
+* Removed the broken 'Percent' constructor from `GroupStyle`. Use `PercentDiff`
+ instead to make relative comparisons.
+* The behavior of `PercentDiff` has changed, it now computes the % from the
+ lower value instead of from the baseline.
+* The default `diffStrategy` has been changed to `SingleEstimator` instead of
+ `MinEstimator`.
+
+### Deprecations
+
+* Config fields `title` and `titleAnnotations` have been deprecated, please use
+ `mkTitle` instead.
+
+### Bug Fixes
+
+* `GroupStyle Absolute` now honors the `MinEstimator` setting. When
+ `MinEstimator` is set, the groups being compared to baseline now display the
+ value based on the estimator which provides closest estimate to the baseline.
+
+### Enhancements
+
+* Add a CLI executable to generate textual reports and graphs from criterion or
+ gauge csv output file.
+* Add `Multiples` as a comparison option, the group being compared is shown as
+ a multiple of the baseline.
+* Add ability to omit the baseline group from the results when we are doing a
+ relative comparison among groups.
+* Add the ability to sort the benchmarks using a different criterion than the
+ one used to present the benchmarks in the final report output.
+* Add `mkTitle` config option to use a function for generating a custom report
+ title.
+
## 0.2.2
* Allow additional annotations to title to be controlled via config
diff --git a/README.md b/README.md
index ed43321..039d51e 100644..100755
--- a/README.md
+++ b/README.md
@@ -1,27 +1,43 @@
# bench-show
-Generate text reports and graphical charts from the benchmark results
-generated by `gauge` or `criterion`, showing or comparing benchmarks in
-many useful ways. In a few lines of code, we can report time taken, peak
-memory usage, allocations, among many other fields; we can group benchmarks
-and compare the groups; we can compare benchmarks before and after a change;
-we can show absolute or percentage difference from the baseline; we can sort
-the results to get the worst affected benchmarks by percentage change.
-
-It can help us in answering questions like the following, visually or
-textually:
-
-* Across two benchmark runs, show all the operations that resulted in a
- regression of more than 10%, so that we can quickly identify and fix
- performance problems in our application.
+Generate text reports and graphical charts from the benchmark results generated
+by `gauge` or `criterion` and stored in a CSV file. This tool is especially
+useful when you have many benchmarks or if you want to compare benchmarks
+across multiple packages. You can generate many interesting reports
+including:
+
+* Show individual reports for all the fields measured e.g. `time taken`, `peak
+ memory usage`, `allocations`, among many other fields measured by `gauge`
+* Sort benchmark results on a specified criterion e.g. you may want to see the
+ biggest cpu hoggers or biggest memory hoggers on top
+* Across two benchmark runs (e.g. before and after a change), show all the
+ operations that resulted in a regression of more than x% in descending order,
+ so that we can quickly identify and fix performance problems in our
+ application.
* Across two (or more) packages providing similar functionality, show all the
operations where the performance differs by more than 10%, so that we can
critically analyze the packages and choose the right one.
## Quick Start
-Use `gauge` or `criterion` to generate a `results.csv` file, and then use the
-following code to generate a textual report or a graph:
+Use `gauge` or `criterion` to generate a `results.csv` file, and then use
+either the `bench-show` executable or the library APIs to generate textual or
+graphical reports.
+
+## Executable
+
+Use `bench-show` executable with `report` and `graph` sub-commands:
+
+```
+$ bench-show report results.csv
+$ bench-show graph results.csv output
+```
+
+For advanced usage, control the generated report by the CLI flags.
+
+## Library
+
+Use `report` and `graph` library functions:
```
report "results.csv" Nothing defaultConfig
@@ -37,6 +53,10 @@ For advanced usage, control the generated report by modifying the
can select many fields from a `gauge` raw report. Units of the fields are
automatically determined based on the range of values:
+```
+$ bench-show --presentation Fields report results.csv
+```
+
```haskell
report "results.csv" Nothing defaultConfig { presentation = Fields }
```
@@ -55,6 +75,10 @@ streamly/zip 644.33 2.59
`graph` generates one bar chart per field:
```
+$ bench-show --presentation Fields graph results.csv
+```
+
+```
graph "results.csv" "output" defaultConfig
```
diff --git a/app/Main.hs b/app/Main.hs
new file mode 100644
index 0000000..e94b626
--- /dev/null
+++ b/app/Main.hs
@@ -0,0 +1,187 @@
+{-# LANGUAGE CPP #-}
+{-# LANGUAGE RecordWildCards #-}
+{-# LANGUAGE TemplateHaskell #-}
+{-# LANGUAGE TupleSections #-}
+
+#if __GLASGOW_HASKELL__ >= 800
+{-# OPTIONS_GHC -Wno-deprecations #-}
+#endif
+
+-- |
+-- Module : Main
+-- Copyright : (c) 2019 Composewell Technologies
+-- (c) 2019 Sanchayan Maity
+-- License : BSD3
+-- Maintainer : harendra.kumar@gmail.com
+-- Stability : experimental
+-- Portability : GHC
+--
+module Main where
+
+import Data.Semigroup ((<>))
+import Options.Applicative.Simple (addCommand, simpleOptions, simpleVersion)
+import Paths_bench_show (version)
+
+import Options.Applicative
+import BenchShow
+
+-- The command line interface provides subcommands for each task. There are
+-- some common options that apply to all subcommands. Subcommands may have
+-- their own specific options as well.
+
+-------------------------------------------------------------------------------
+-- Common config options
+-------------------------------------------------------------------------------
+
+pVerbose :: Parser Bool
+pVerbose = switch $
+ long "verbose"
+ <> short 'v'
+ <> help "Provide more details in the report output"
+
+pOutputDir :: Parser FilePath
+pOutputDir = strOption
+ ( long "output-dir"
+ <> metavar "DIR"
+ <> help "Default is current directory" )
+
+pTitle :: Parser String
+pTitle = strOption
+ ( long "title"
+ <> short 't'
+ <> metavar "STRING"
+ <> help "Title for the report" )
+
+pTitleAnnotation :: Parser TitleAnnotation
+pTitleAnnotation = option auto $
+ long "title-annotations"
+ <> help ("*TitleField*|TitleEstimator|TitleDiff")
+
+pPresentation :: Parser Presentation
+pPresentation = option auto $
+ long "presentation"
+ <> help ("Solo|Fields|*Groups* <*Absolute*|Diff|PercentDiff|Multiples")
+
+pEstimator :: Parser Estimator
+pEstimator = option auto $
+ long "estimator"
+ <> help ("*Median*|Mean|Regression")
+
+pThreshold :: Parser Word
+pThreshold = option auto $
+ long "threshold"
+ <> metavar "PERCENT"
+ <> help "Min % diff (default 3) to flag regression or improvement"
+
+pDiffStrategy :: Parser DiffStrategy
+pDiffStrategy = option auto $
+ long "diff-strategy"
+ <> help ("*SingleEstimator*|MinEstimator")
+
+pOmitBaseline :: Parser Bool
+pOmitBaseline = switch $
+ long "omit-baseline"
+ <> help "omit the baseline column in relative comparisons"
+
+-------------------------------------------------------------------------------
+-- Build a Config parser for common options
+-------------------------------------------------------------------------------
+
+-- Specify a default value for a Maybe inside a functor
+fMaybe :: Functor f => a -> f (Maybe a) -> f a
+fMaybe a = fmap (maybe a id)
+
+-- | parse an optional field with a default value taken from defaultConfig
+parseOptional :: (Config -> a) -> Parser a -> Parser a
+parseOptional def parser = fMaybe (def defaultConfig) (optional parser)
+
+pConfig :: Parser Config
+pConfig = Config
+ <$> parseOptional verbose pVerbose
+ <*> optional pOutputDir
+ <*> pure Nothing
+ <*> optional pTitle
+ <*> many pTitleAnnotation
+ <*> parseOptional presentation pPresentation
+ <*> parseOptional estimator pEstimator
+ <*> parseOptional threshold pThreshold
+ <*> parseOptional diffStrategy pDiffStrategy
+ <*> parseOptional omitBaseline pOmitBaseline
+ <*> pure (selectFields defaultConfig)
+ <*> pure (fieldRanges defaultConfig)
+ <*> pure (fieldTicks defaultConfig)
+ <*> pure (classifyBenchmark defaultConfig)
+ <*> pure (selectGroups defaultConfig)
+ <*> pure (selectBenchmarks defaultConfig)
+
+-------------------------------------------------------------------------------
+-- "report" subcommand
+-------------------------------------------------------------------------------
+
+data ReportOpts = ReportOpts
+ { reportInput :: FilePath
+ , reportOutput :: Maybe FilePath
+ }
+
+pInputFile :: Parser FilePath
+pInputFile = argument str (metavar "INPUT-FILE.CSV")
+
+pOutputFile :: Parser FilePath
+pOutputFile = strOption
+ (long "output"
+ <> short 'o'
+ <> help "Output file")
+
+pReportOpts :: Parser ReportOpts
+pReportOpts = ReportOpts
+ <$> pInputFile
+ <*> optional pOutputFile
+
+cmdReport :: ReportOpts -> Config -> IO ()
+cmdReport ReportOpts{..} cfg = report reportInput reportOutput cfg
+
+-------------------------------------------------------------------------------
+-- "graph" subcommand
+-------------------------------------------------------------------------------
+
+data GraphOpts = GraphOpts
+ { graphInput :: FilePath
+ , graphOutput :: FilePath
+ }
+
+pOutputArg :: Parser FilePath
+pOutputArg = argument str (metavar "OUTPUT-FILE-PREFIX")
+
+pGraphOpts :: Parser GraphOpts
+pGraphOpts = GraphOpts
+ <$> pInputFile
+ <*> pOutputArg
+
+cmdGraph :: GraphOpts -> Config -> IO ()
+cmdGraph GraphOpts{..} cfg = graph graphInput graphOutput cfg
+
+-------------------------------------------------------------------------------
+-- Build and run a subcommand parser
+-------------------------------------------------------------------------------
+
+cmdLineParser :: Parser Config -> IO ()
+cmdLineParser p = do
+ (cfg, handler) <- simpleOptions $(simpleVersion version)
+ ("Generate reports and graphs from gauge or criterion output.\n" ++
+ "Default values are shown as *DEFAULT*.")
+ "" p cmds
+
+ handler cfg
+
+ where cmds = do
+ addCommand "report"
+ "Generate a text report"
+ cmdReport
+ pReportOpts
+ addCommand "graph"
+ "Generate a graphical report"
+ cmdGraph
+ pGraphOpts
+
+main :: IO ()
+main = cmdLineParser pConfig
diff --git a/bench-show.cabal b/bench-show.cabal
index 39a42dd..2df383d 100644
--- a/bench-show.cabal
+++ b/bench-show.cabal
@@ -1,40 +1,45 @@
name: bench-show
-version: 0.2.2
+version: 0.3.0
license: BSD3
author: Harendra Kumar
maintainer: harendra.kumar@gmail.com
bug-reports: https://github.com/composewell/bench-show/issues
synopsis: Show, plot and compare benchmark results
description:
- Generate text reports and graphical charts from benchmark results
- generated by @gauge@ or @criterion@, showing or comparing benchmarks in
- many useful ways. In a few lines of code, we can report time taken, peak
- memory usage, allocations, among many other fields; we can arrange benchmarks
- in groups and compare the groups; we can compare benchmark results before and
- after a change; we can show the difference from baseline in absolute terms or
- as a percentage; we can sort the results by percentage change to get the
- worst affected benchmarks.
+ Generate text reports and graphical charts from the benchmark results generated
+ by @gauge@ or @criterion@ and stored in a CSV file. This tool is especially
+ useful when you have many benchmarks or if you want to compare benchmarks
+ across multiple packages. You can generate many interesting reports
+ including:
.
- @bench-show@ helps us in answering questions like the following, visually or
- textually:
- .
- * Across two benchmark runs, show all the operations that resulted in a
- regression of more than 10%, so that we can quickly identify and fix
- performance problems in our application.
- * Across two (or more) packages (providing similar functionality), show all
- the operations where the performance differs by more than 10%, so that we
- can critically analyze the packages and choose the right one.
+ * Show individual reports for all the fields measured e.g. @time taken@, @peak
+ memory usage@, @allocations@, among many other fields measured by @gauge@
+ * Sort benchmark results on a specified criterion e.g. you may want to see the
+ biggest cpu hoggers or biggest memory hoggers on top
+ * Across two benchmark runs (e.g. before and after a change), show all the
+ operations that resulted in a regression of more than x% in descending
+ order, so that we can quickly identify and fix performance problems in our
+ application.
+ * Across two (or more) packages providing similar functionality, show all the
+ operations where the performance differs by more than 10%, so that we can
+ critically analyze the packages and choose the right one.
.
Quick Start: Use @gauge@ or @criterion@ to generate a @results.csv@ file, and
- then use the following code to generate a textual report or a graph:
+ then use either the @bench-show@ executable or the library APIs to generate
+ textual or graphical reports.
+ .
+ @
+ $ bench-show report results.csv
+ $ bench-show graph results.csv output
+ @
.
@
report "results.csv" Nothing defaultConfig
graph "results.csv" "output" defaultConfig
@
.
- For example, you can show can show % regression from a baseline in descending
- order texttually as follows:
+ There are many ways to present the reports, for example, you can show can
+ show % regression from a baseline in descending order textually as follows:
.
@
(time)(Median)(Diff using min estimator)
@@ -55,7 +60,8 @@ description:
category: Performance, Benchmarking
homepage: https://github.com/composewell/bench-show
license-file: LICENSE
-tested-with: GHC==8.4.4
+tested-with: GHC==8.6.5
+ , GHC==8.4.4
, GHC==8.2.2
, GHC==7.10.3
copyright: 2017, 2018 Composewell Technologies
@@ -123,6 +129,30 @@ library
, statistics >= 0.15 && < 0.16
, vector >= 0.10 && < 0.13
+executable bench-show
+ default-language: Haskell2010
+ hs-source-dirs: app
+ main-is: Main.hs
+ other-modules: Paths_bench_show
+ ghc-options: -Wall
+ build-depends:
+ base >= 4.8 && < 5
+ , Chart >= 1.6 && < 2
+ , Chart-diagrams >= 1.6 && < 2
+ , csv >= 0.1 && < 0.2
+ , filepath >= 1.3 && < 1.5
+ , mwc-random >= 0.13 && < 0.15
+ , directory >= 1.2 && < 1.4
+ , transformers >= 0.4 && < 0.6
+ , ansi-wl-pprint >= 0.6 && < 0.7
+ , split >= 0.2 && < 0.3
+ , statistics >= 0.15 && < 0.16
+ , vector >= 0.10 && < 0.13
+ , semigroups >= 0.18 && < 0.20
+ , optparse-applicative >= 0.14.2 && < 0.14.4
+ , optparse-simple >= 0.1.0 && < 0.2
+ , bench-show
+
test-suite test
type: exitcode-stdio-1.0
default-language: Haskell2010
diff --git a/docs/full-median-time.svg b/docs/full-median-time.svg
index 631c433..631c433 100644..100755
--- a/docs/full-median-time.svg
+++ b/docs/full-median-time.svg
diff --git a/docs/grouped-delta-median-time.svg b/docs/grouped-delta-median-time.svg
index 39c5d82..39c5d82 100644..100755
--- a/docs/grouped-delta-median-time.svg
+++ b/docs/grouped-delta-median-time.svg
diff --git a/docs/grouped-median-time.svg b/docs/grouped-median-time.svg
index 4742c00..4742c00 100644..100755
--- a/docs/grouped-median-time.svg
+++ b/docs/grouped-median-time.svg
diff --git a/docs/grouped-percent-delta-coeff-time.svg b/docs/grouped-percent-delta-coeff-time.svg
index 87d96da..87d96da 100644..100755
--- a/docs/grouped-percent-delta-coeff-time.svg
+++ b/docs/grouped-percent-delta-coeff-time.svg
diff --git a/docs/grouped-percent-delta-median-time.svg b/docs/grouped-percent-delta-median-time.svg
index 58507f5..58507f5 100644..100755
--- a/docs/grouped-percent-delta-median-time.svg
+++ b/docs/grouped-percent-delta-median-time.svg
diff --git a/docs/grouped-percent-delta-sorted-median-time.svg b/docs/grouped-percent-delta-sorted-median-time.svg
index 03a0e8e..03a0e8e 100644..100755
--- a/docs/grouped-percent-delta-sorted-median-time.svg
+++ b/docs/grouped-percent-delta-sorted-median-time.svg
diff --git a/docs/grouped-single-estimator-coeff-time.svg b/docs/grouped-single-estimator-coeff-time.svg
index 37e893c..37e893c 100644..100755
--- a/docs/grouped-single-estimator-coeff-time.svg
+++ b/docs/grouped-single-estimator-coeff-time.svg
diff --git a/docs/regression-percent-descending-median-time.svg b/docs/regression-percent-descending-median-time.svg
index f53f494..f53f494 100644..100755
--- a/docs/regression-percent-descending-median-time.svg
+++ b/docs/regression-percent-descending-median-time.svg
diff --git a/lib/BenchShow.hs b/lib/BenchShow.hs
index ca0bd85..98dbb74 100644
--- a/lib/BenchShow.hs
+++ b/lib/BenchShow.hs
@@ -52,12 +52,12 @@
-- from the textual report, the rows (i.e. the benchmarks) are represented as
-- bars in the cluster.
--
--- When the columns are groups, each report consists of results for a single
+-- When the columns are groups, each report presents the results for a single
-- benchmarking field for different benchmark groups. Using 'GroupStyle', we
--- can further specify how we want to present the results the groups. We can
+-- can further specify how we want to present the results of the groups. We can
-- either present absolute values of the field for each group or we can make
--- the first group as a baseline and present differences from the baseline for
--- the subsequent groups.
+-- the first group as a baseline showing absolute values and present difference
+-- from the baseline for the subsequent groups.
--
-- When the columns are fields, each report consists of results for a single
-- benchmarking group. Fields cannot be compared like groups because they are
diff --git a/lib/BenchShow/Analysis.hs b/lib/BenchShow/Analysis.hs
index 1cb0e48..827cfb2 100644
--- a/lib/BenchShow/Analysis.hs
+++ b/lib/BenchShow/Analysis.hs
@@ -270,7 +270,7 @@ data Estimator =
-- better when larger number of samples are taken. This
-- cannot be used when the number of samples is less than
-- 2, in that case a mean value is reported instead.
- deriving (Eq, Show)
+ deriving (Eq, Show, Read)
getAnalyzedValue :: Estimator -> AnalyzedField -> Double
getAnalyzedValue estimator AnalyzedField{..} =
diff --git a/lib/BenchShow/Common.hs b/lib/BenchShow/Common.hs
index 5019d94..cdfc90a 100644
--- a/lib/BenchShow/Common.hs
+++ b/lib/BenchShow/Common.hs
@@ -49,7 +49,7 @@ import Data.Foldable (foldl')
import Data.Function ((&), on)
import Data.List
(transpose, groupBy, (\\), find, sortBy, elemIndex, intersect,
- intersectBy, concatMap)
+ intersectBy)
import Data.List.Split (linesBy)
import Data.Maybe (fromMaybe, mapMaybe)
import Data.Ord (comparing)
@@ -78,25 +78,35 @@ filterSanity label old new = do
label ++
" must not add any new items to the original list. The \
\following items were added: " ++ show added
+ ++ "\nOriginal groups: " ++ show old
+ ++ "\nNew groups: " ++ show new
-------------------------------------------------------------------------------
data ReportType = TextReport | GraphicalChart
-- | How to show the results for multiple benchmark groups presented in columns
--- or bar chart clusters.
+-- or bar chart clusters. In relative comparisons, the first group is
+-- considered as the baseline and the subsequent groups are compared against
+-- the baseline.
--
+-- /Definition changed in 0.3.0/
-- @since 0.2.0
data GroupStyle =
- Absolute -- ^ Show absolute field values for all groups
- | Diff -- ^ Show baseline group values as usual and values for
- -- the subsequent groups as differences from the baseline
- | Percent -- ^ Show baseline group values as 100% and values for
- -- subsequent groups as a percentage of the baseline
- | PercentDiff -- ^ Show baseline group values as usual and values for
- -- subsequent groups as precentage difference from the
- -- baseline
- deriving (Eq, Show)
+ Absolute -- ^ Show absolute values of the field for all groups
+ | Diff -- ^ Show baseline group as absolute values and values for
+ -- the subsequent groups as difference from the baseline
+ | PercentDiff -- ^ If the value of the group being compared is higher than
+ -- the baseline then display the difference as percentage of
+ -- baseline otherwise display the difference as a percentage
+ -- of the group being compared.
+ | Multiples -- ^ If the value of the group being compared is higher than
+ -- the baseline then display @+(value / baseline value)@
+ -- otherwise display @-(baseline value / value)@. This
+ -- provides a normalized comparison independent of the
+ -- absolute value of a benchmark. Note that 'Multiples' can
+ -- be directly computed using 'PercentDiff' and vice-versa.
+ deriving (Eq, Show, Read)
-- | How to present the reports or graphs. Each report presents a number of
-- benchmarks as rows, it may have, (1) a single column presenting the values
@@ -124,7 +134,7 @@ data Presentation =
-- with all the fields selected by the configuration as
-- columns or clusters. Output files are named using
-- @-estimator-groupname@ as suffix.
- deriving (Eq, Show)
+ deriving (Eq, Show, Read)
-- | FieldTick is used only in visual charts to generate the major ticks on
-- the y-axis. You can specify either the size of a tick ('TickSize') or the
@@ -175,13 +185,15 @@ data DiffStrategy =
| WorstBest
| BestBest
-}
+ deriving (Show, Read)
-- | Additional annotations that can be optionally added to the title of the
-- report or graph.
--
-- @since 0.2.2
+{-# DEPRECATED TitleAnnotation "Please use mkTitle to make a custom title" #-}
data TitleAnnotation = TitleField | TitleEstimator | TitleDiff
- deriving (Eq, Show)
+ deriving (Eq, Show, Read)
-- | Configuration governing generation of chart. See 'defaultConfig' for the
-- default values of these fields.
@@ -197,11 +209,19 @@ data Config = Config
-- | The directory where the output graph or report file should be placed.
, outputDir :: Maybe FilePath
- -- | Report title, more information like the plotted field name or
+ -- | Function to make a title for the report. The argument to the function
+ -- is the benchmark field name for which the report is being made.
+ , mkTitle :: Maybe (String -> String)
+
+ -- | /DEPRECATED: Please use 'mkTitle' instead./
+ --
+ -- Report title, more information like the plotted field name or
-- the presentation style may be added to it.
, title :: Maybe String
- -- | Additional annotations to be added to the title
+ -- | /DEPRECATED: Please use 'mkTitle' instead./
+ --
+ -- Additional annotations to be added to the title
, titleAnnotations :: [TitleAnnotation]
-- | How to determine the layout of the report or the chart.
@@ -217,6 +237,10 @@ data Config = Config
-- | Strategy to compare two runs or groups of benchmarks.
, diffStrategy :: DiffStrategy
+ -- | Omit the baseline group in normalized relative comparisons i.e.
+ -- when the 'GroupStyle' is 'PercentDiff' or 'Multiples'.
+ , omitBaseline :: Bool
+
---------------------------------------------------------------------------
-- Fields (Columns)
---------------------------------------------------------------------------
@@ -263,54 +287,68 @@ data Config = Config
-- Benchmarks (Rows)
---------------------------------------------------------------------------
- -- | Filter and reorder benchmarks. 'selectBenchmarks' is provided with a
- -- function which is invoked with a sorting column name or index, the
- -- function produces the benchmark names and corresponding values for that
- -- column which can be used as a sorting criterion. The output of
- -- 'selectBenchmarks' is a list of benchmarks in the order in which they
- -- are to be rendered.
+ -- | Filter and reorder benchmarks. 'selectBenchmarks' takes a function
+ -- argument, the function is invoked with a sorting column name or index
+ -- and a 'GroupStyle'. The output of the function is either a 'Right' value
+ -- consisting of tuples of the benchmark names and values corresponding to
+ -- the given column and style or a 'Left' value indicating an error.
+ -- 'selectBenchmarks' can inspect these benchmarks and there values to
+ -- produce a filtered and sorted list of benchmark names that are to be
+ -- rendered.
+ --
+ -- The style argument is ignored when the report presentation is not
+ -- 'Groups'. When style is 'Nothing', the presentation setting specified in
+ -- the configuration is used.
+ --
+ -- /Signature changed in 0.3.0/
, selectBenchmarks
- :: (SortColumn -> Either String [(String, Double)])
+ :: (SortColumn -> Maybe GroupStyle -> Either String [(String, Double)])
-> [String]
}
+-- IMPORTANT: If you change the defaults, please change the defaults in the CLI
+-- help as well.
+--
-- | Default configuration. Use this as the base configuration and modify the
-- required fields. The defaults are:
--
-- @
-- verbose = False
--- title = Nothing
+-- mkTitle = Nothing
-- titleAnnotations = [TitleField]
-- outputDir = Nothing
-- presentation = Groups Absolute
-- estimator = Median
-- threshold = 3
--- diffStrategy = MinEstimator
+-- diffStrategy = SingleEstimator
+-- omitBaseline = False
-- selectFields = filter (flip elem ["time", "mean", "maxrss"] . map toLower)
-- fieldRanges = []
-- fieldTicks = []
-- classifyBenchmark = Just . ("default",)
-- selectGroups = id
--- selectBenchmarks = \f -> either error (map fst) $ f (ColumnIndex 0)
+-- selectBenchmarks = \f -> either error (map fst) $ f (ColumnIndex 0) Nothing
-- @
--
-- @since 0.2.0
defaultConfig :: Config
defaultConfig = Config
{ verbose = False
+ , mkTitle = Nothing
, title = Nothing
, titleAnnotations = [TitleField]
, outputDir = Nothing
, presentation = Groups Absolute
, estimator = Median
, threshold = 3
- , diffStrategy = MinEstimator
+ , diffStrategy = SingleEstimator
+ , omitBaseline = False
, selectFields = filter (flip elem ["time", "mean", "maxrss"] . map toLower)
, fieldRanges = []
, fieldTicks = []
, classifyBenchmark = Just . ("default",)
, selectGroups = id
- , selectBenchmarks = \f -> either error (map fst) $ f (ColumnIndex 0)
+ , selectBenchmarks = \f -> either error (map fst) $ f (ColumnIndex 0) Nothing
}
-------------------------------------------------------------------------------
@@ -389,8 +427,8 @@ getUnitByFieldName fieldName fieldMin =
fieldUnits :: String -> Double -> GroupStyle -> RelativeUnit
fieldUnits fieldName fieldMin style =
case style of
- Percent -> RelativeUnit "%" 1
- PercentDiff -> RelativeUnit "%" 1
+ Multiples -> RelativeUnit "x" 1
+ PercentDiff -> RelativeUnit "%" 1
_ -> getUnitByFieldName fieldName fieldMin
-------------------------------------------------------------------------------
@@ -400,11 +438,19 @@ fieldUnits fieldName fieldMin style =
absoluteDiff :: Num a => a -> a -> a
absoluteDiff v1 v2 = v2 - v1
-percentDiff :: (Fractional a, Num a) => a -> a -> a
-percentDiff v1 v2 = ((v2 - v1) * 100) / v1
+percentDiff :: (Fractional a, Num a, Ord a) => a -> a -> a
+percentDiff v1 v2 = ((v2 - v1) * 100) / min v1 v2
-percent :: (Fractional a, Num a) => a -> a -> a
-percent v1 v2 = (v2 * 100) / v1
+-- We map a fraction x between 0 and 1 to a negative 1/x for plotting on an
+-- equal and opposite scale.
+fraction :: (Fractional a, Num a, Ord a, Show a) => a -> a -> a
+fraction v1 v2 =
+ let val = v2 / v1
+ in case val of
+ x | x <= 0 -> error $ "BenchShow.Common.fraction: negative: " ++ show x
+ x | x < 1 -> negate (1 / x)
+ x | x >= 1 -> x
+ x -> error $ "BenchShow.Common.fraction: unhandled: " ++ show x
cmpTransformColumns :: ReportType
-> GroupStyle
@@ -426,37 +472,41 @@ cmpTransformColumns rtype style estimator diffStrategy cols =
let firstCol = head cols
colTransform col = zipWith (mkMinDiff diff) firstCol col
in map colTransform (tail cols)
- in case style of
- Absolute -> (Nothing, columns)
- Percent -> (Nothing, cmpWith percent)
- Diff ->
+
+ diffWith f =
case diffStrategy of
MinEstimator ->
- let (ests, vals) = unzip $ map unzip (cmpMinWith absoluteDiff)
+ let (ests, vals) = unzip $ map unzip (cmpMinWith f)
in ( Just $ map (const estimator) (head cols) : ests
, head columns : vals
)
SingleEstimator ->
- (Nothing, head columns : cmpWith absoluteDiff)
- PercentDiff ->
- -- In a comparative graphical chart we cannot show the absolute
- -- values in the baseline column as the units won't match for
- -- the baseline and the diff clusters.
- let baseCol =
- case rtype of
- TextReport -> head columns
- GraphicalChart | length columns == 1 ->
- head columns
- GraphicalChart ->
- map (\(n,_) -> (n,100)) (head columns)
- in case diffStrategy of
- MinEstimator ->
- let (ests, vals) = unzip $ map unzip (cmpMinWith percentDiff)
- in ( Just $ map (const estimator) (head cols) : ests
- , baseCol : vals
- )
- SingleEstimator ->
- (Nothing, baseCol : cmpWith percentDiff)
+ (Nothing, head columns : cmpWith f)
+
+ relativeDiffWith baseVal f =
+ -- In a comparative graphical chart we cannot show the absolute
+ -- values in the baseline column as the units won't match for
+ -- the baseline and the diff clusters.
+ let baseCol =
+ case rtype of
+ TextReport -> head columns
+ GraphicalChart | length columns == 1 ->
+ head columns
+ GraphicalChart ->
+ map (\(n,_) -> (n,baseVal)) (head columns)
+ in case diffStrategy of
+ MinEstimator ->
+ let (ests, vals) = unzip $ map unzip (cmpMinWith f)
+ in ( Just $ map (const estimator) (head cols) : ests
+ , baseCol : vals
+ )
+ SingleEstimator ->
+ (Nothing, baseCol : cmpWith f)
+ in case style of
+ Absolute -> diffWith (\_ x -> x)
+ Diff -> diffWith absoluteDiff
+ PercentDiff -> relativeDiffWith 100 percentDiff
+ Multiples -> relativeDiffWith 1 fraction
where
verify a b = if a then b else error "bug: benchmark names mismatch"
transformVals = map (map (second (getAnalyzedValue estimator)))
@@ -478,19 +528,27 @@ cmpTransformColumns rtype style estimator diffStrategy cols =
then (Mean, (n2, meanDiff))
else (Regression, (n2, regDiff))
-transformColumnNames :: GroupStyle -> [ReportColumn] -> [ReportColumn]
-transformColumnNames _ [] = []
-transformColumnNames style columns@(h:t) =
- let withDiff = colSuffix baseName h : map (colSuffix diffName) t
+columnNameByStyle :: GroupStyle -> [ReportColumn] -> [ReportColumn]
+columnNameByStyle _ [] = []
+columnNameByStyle style columns@(h:t) =
+ let withDiff name = colSuffix baseName h : map (colSuffix name) t
in case style of
- Diff | length columns > 1 -> withDiff
- PercentDiff | length columns > 1 -> withDiff
- _ -> columns
+ Diff | length columns > 1 -> withDiff diffName
+ Multiples | length columns > 1 -> withDiff fracName
+ PercentDiff | length columns > 1 -> withDiff diffName
+ _ -> columns
where
colSuffix xl col = col { colName = xl (colName col) }
- baseName = (++ "(base)")
- diffName = (++ "(-base)")
+ baseName = id -- (++ "(base)")
+ diffName = (++ " - " ++ colName h) -- "(-base)")
+ fracName = (++ "/" ++ colName h) -- "/base")
+
+columnNameByUnit :: [RelativeUnit] -> [ReportColumn] -> [ReportColumn]
+columnNameByUnit units columns =
+ let applyUnit col (RelativeUnit label _) =
+ col { colName = colName col ++ inParens label }
+ in zipWith applyUnit columns units
-- Represents the data for a single benchmark run
data GroupMatrix = GroupMatrix
@@ -599,8 +657,9 @@ benchmarkCompareSanity benchmarks GroupMatrix{..} = do
selectBenchmarksByField :: Config
-> [GroupMatrix]
-> [[(String, Double)]]
+ -> (GroupStyle -> [[(String, Double)]])
-> [String]
-selectBenchmarksByField Config{..} matrices columns =
+selectBenchmarksByField Config{..} matrices columns colsByStyle =
let bmnames = selectBenchmarks extractGroup
in if (null bmnames)
then error $ "selectBenchmarks must select at least one benchmark"
@@ -621,17 +680,22 @@ selectBenchmarksByField Config{..} matrices columns =
in map getName matrices
-- columns are benchmark groups in this case
- extractGroup (ColumnName (Left name)) =
- let len = length columns
- in if len <= 1
- then extractGroup $ ColumnName (Right (name, 0))
- else Left $ "selectBenchmarks: there are " ++ show len
- ++ " runs in the input data, please specify the run \
- \index [0-" ++ show (len - 1)
- ++ "] along with the group name."
- extractGroup (ColumnName (Right (name, runId))) =
- extractColumnByGroupName name runId
- extractGroup (ColumnIndex n) = extractColumnByGroupIndex n
+ extractGroup colSelector style =
+ let cols = case style of
+ Nothing -> columns
+ Just s -> colsByStyle s
+ in case colSelector of
+ ColumnName (Left name) ->
+ let len = length cols
+ in if len <= 1
+ then extractGroup (ColumnName (Right (name, 0))) style
+ else Left $ "selectBenchmarks: there are " ++ show len
+ ++ " runs in the input data, please specify the run \
+ \index [0-" ++ show (len - 1)
+ ++ "] along with the group name."
+ ColumnName (Right (name, runId)) ->
+ extractColumnByGroupName name runId style
+ ColumnIndex n -> extractColumnByGroupIndex n cols
-- The benchmark field is constant. Extract all benchmark values for the
-- given field and for the given group.
@@ -645,20 +709,20 @@ selectBenchmarksByField Config{..} matrices columns =
True -> res
in foldl foldFunc (0, False) mxs
- extractColumnByGroupName name runId =
+ extractColumnByGroupName name runId style =
case findColumnIndex matrices (name, runId) of
(_, False) -> Left $ "Benchmark group name [" ++ name
++ "] and index [" ++ show runId
++ "] not found. Available groups are: "
++ show grpNames
- (i, True) -> extractGroup (ColumnIndex i)
+ (i, True) -> extractGroup (ColumnIndex i) style
- extractColumnByGroupIndex idx =
- let len = length columns
+ extractColumnByGroupIndex idx cols =
+ let len = length cols
in if idx >= len
then Left $ "Column index must be in the range [0-"
++ show (len - 1) ++ "]"
- else Right $ columns !! idx
+ else Right $ cols !! idx
selectBenchmarksByGroup :: Config -> GroupMatrix -> [String]
selectBenchmarksByGroup Config{..} grp@GroupMatrix{..} =
@@ -671,11 +735,11 @@ selectBenchmarksByGroup Config{..} grp@GroupMatrix{..} =
where
-- columns are benchmark fields in this case
- extractField (ColumnName (Left name)) = extractColumnByFieldName name
- extractField (ColumnName (Right (name, _))) =
+ extractField (ColumnName (Left name)) _ = extractColumnByFieldName name
+ extractField (ColumnName (Right (name, _))) _ =
-- XXX runId does not make sense for fields
extractColumnByFieldName name
- extractField (ColumnIndex n) = extractColumnByFieldIndex n
+ extractField (ColumnIndex n) _ = extractColumnByFieldIndex n
-- The benchmark field is constant. Extract all benchmark values for the
-- given field and for the given group.
@@ -973,12 +1037,30 @@ selectCommon matrices =
unless (null absent) $ putStrLn msg
return matrix { groupBenches = newBenches }
-prepareGroupMatrices :: Config -> CSV -> [String] -> IO (Int, [GroupMatrix])
-prepareGroupMatrices cfg@Config{..} csvlines fields = do
- let (hdr, runs) =
+prepareGroupMatrices :: Config
+ -> FilePath
+ -> CSV
+ -> [String]
+ -> IO (Int, [GroupMatrix])
+prepareGroupMatrices cfg@Config{..} inputFile csvlines fields = do
+ let res@(_, ls) =
sanityCheckCSV csvlines
& splitRuns
- & ensureIterField
+
+ when (null ls) $
+ error $ "No benchmark results found in CSV file: " ++ show inputFile
+
+ let checkForData (runId, xs) =
+ when (null xs) $
+ putStrLn $ "No benchmark results found in the CSV file ["
+ ++ show inputFile
+ ++ "], for runId: "
+ ++ show runId
+
+ mapM_ checkForData (zip [(0 :: Int)..] ls)
+
+ let (hdr, runs) = ensureIterField res
+
xs <- sequence $ map (readIterations hdr) runs
& map (filterFields fields)
-- & _filterCommonSubsets
@@ -1064,30 +1146,40 @@ prepareGroupsReport cfg@Config{..} style outfile rtype runs field matrices =
(estimators, transformedCols) =
cmpTransformColumns rtype style estimator diffStrategy unsortedCols
- benchmarks = selectBenchmarksByField cfg matrices transformedCols
+ transformedColsByStyle s = snd $
+ cmpTransformColumns rtype s estimator diffStrategy unsortedCols
+
+ benchmarks = selectBenchmarksByField cfg matrices
+ transformedCols transformedColsByStyle
sortedCols = map (sortValues benchmarks) transformedCols
origSortedCols = map (sortValues benchmarks) unsortedCols
mkColUnits :: [RelativeUnit]
mkColUnits =
let cols =
- if style == Diff || style == PercentDiff
+ if style /= Absolute
-- if we consider diff values as well here then the
-- units will change to potentially very small.
then [head sortedCols]
else sortedCols
minVal = getFieldMin cfg (minimum $ concat cols) field
- in case (rtype, style) of
- -- In case of percentDiff in TextReport we use absolute
- -- values in the baseline column, so the unit is different.
- (TextReport, PercentDiff) ->
+ mkPercentColUnitText =
let unit = fieldUnits field minVal Absolute
punit = fieldUnits field 1 style -- % unit
in unit : replicate (length matrices - 1) punit
- (GraphicalChart, PercentDiff) | length matrices == 1 ->
- [fieldUnits field minVal Absolute]
- _ -> let unit = fieldUnits field minVal style
+ mkPercentColUnitGraph = [fieldUnits field minVal Absolute]
+ mkAbsoluteUnit =
+ let unit = fieldUnits field minVal style
in replicate (length matrices) unit
+ in case (rtype, style) of
+ -- In case of percentDiff in TextReport we use absolute
+ -- values in the baseline column, so the unit is different.
+ (_, Absolute) -> mkAbsoluteUnit
+ (_, Diff) -> mkAbsoluteUnit
+ (TextReport, _) -> mkPercentColUnitText
+ (GraphicalChart, _) | length matrices == 1 ->
+ mkPercentColUnitGraph
+ (GraphicalChart, _) -> mkAbsoluteUnit
mkColValues :: [[Double]]
mkColValues =
@@ -1102,33 +1194,47 @@ prepareGroupsReport cfg@Config{..} style outfile rtype runs field matrices =
if runs > 1
then "(" ++ show (groupIndex x) ++ ")"
else ""
- applyUnit name (RelativeUnit label _) =
- name ++ inParens label
- in zipWith applyUnit (map withSuffix matrices) mkColUnits
+ in map withSuffix matrices
columns = getZipList $ ReportColumn
<$> ZipList mkColNames
<*> ZipList mkColUnits
<*> ZipList mkColValues
+ removeBaseline xs =
+ let rel = case style of
+ Absolute -> False
+ Diff -> False
+ PercentDiff -> True
+ Multiples -> True
+ in if omitBaseline && rel && length matrices > 1
+ then tail xs
+ else xs
+
in RawReport
{ reportOutputFile = outfile
, reportIdentifier = field
, reportRowIds = benchmarks
- , reportColumns = transformColumnNames style columns
- , reportAnalyzed = zipWith (\x y -> map (scaleAnalyzedField x) y)
- mkColUnits origSortedCols
- , reportEstimators = estimators
+ , reportColumns = removeBaseline
+ $ columnNameByUnit mkColUnits
+ $ columnNameByStyle style columns
+ , reportAnalyzed = removeBaseline $
+ zipWith (\x y -> map (scaleAnalyzedField x) y)
+ mkColUnits origSortedCols
+ , reportEstimators = fmap removeBaseline estimators
}
showStatusMessage :: Show a => Config -> String -> Maybe a -> IO ()
showStatusMessage cfg field outfile =
- let atitle = makeTitle field (diffString (presentation cfg)
- (diffStrategy cfg)) cfg
+ let atitle = case mkTitle cfg of
+ Just f -> " [" ++ f field ++ "]"
+ Nothing ->
+ " [" ++ makeTitle field (diffString (presentation cfg)
+ (diffStrategy cfg)) cfg ++ "]"
in case outfile of
Just path ->
- putStrLn $ "Creating chart "
- ++ "[" ++ atitle ++ "]"
+ putStrLn $ "Creating chart"
+ ++ atitle
++ " at "
++ show path
Nothing -> return ()
@@ -1235,9 +1341,12 @@ showDiffStrategy s =
diffString :: Presentation -> DiffStrategy -> Maybe String
diffString style s =
case style of
- Groups Diff -> Just $ "Diff " ++ showDiffStrategy s
- Groups PercentDiff -> Just $ "Diff " ++ showDiffStrategy s
- _ -> Nothing
+ Groups Diff -> Just $ "Diff from Baseline " ++ showDiffStrategy s
+ Groups PercentDiff -> Just $ "Diff % of Lower " ++ showDiffStrategy s
+ Groups Multiples -> Just $ "Multiples of Baseline " ++ showDiffStrategy s
+ Groups Absolute -> Nothing
+ Solo -> Nothing
+ Fields -> Nothing
inParens :: String -> String
inParens str = "(" ++ str ++ ")"
diff --git a/lib/BenchShow/Graph.hs b/lib/BenchShow/Graph.hs
index 4ce65d5..d42a76a 100644
--- a/lib/BenchShow/Graph.hs
+++ b/lib/BenchShow/Graph.hs
@@ -64,6 +64,32 @@ transformColumns columns =
}]
else columns
+-- We do not want to see the band of values between -1 and 1, in fact there are
+-- no values possible in that band. Shift the positive values by -1 and
+-- negative values by +1 to map them to a 0 based scale on the graph. We change
+-- the labels as well accordingly.
+transformFractionValue :: ReportColumn -> ReportColumn
+transformFractionValue ReportColumn{..} =
+ ReportColumn
+ { colName = colName
+ , colUnit = colUnit
+ , colValues = map (\val ->
+ case val of
+ x | x >= 1 -> x - 1
+ x | x < (-1) -> x + 1
+ x -> error $ "BenchShow.Graph.transformFractionValue: unhandled: " ++ show x
+ ) colValues
+ }
+
+transformFractionLabels :: LinearAxisParams Double
+transformFractionLabels =
+ ((def :: (LinearAxisParams Double)) { _la_labelf = \xs ->
+ let shiftVals v = if v >= 0 then v + 1 else v - 1
+ replaceMinus ('-' : ys) = "1/" ++ ys
+ replaceMinus ys = ys
+ in fmap replaceMinus (_la_labelf def (map shiftVals xs))
+ })
+
genGroupGraph :: RawReport -> Config -> IO ()
genGroupGraph RawReport{..} cfg@Config{..} = do
let outputFile = fromMaybe undefined reportOutputFile
@@ -79,7 +105,9 @@ genGroupGraph RawReport{..} cfg@Config{..} = do
if length reportColumns > 1
then diffString presentation diffStrategy
else Nothing
- atitle = makeTitle reportIdentifier diffStr cfg
+ atitle = case mkTitle of
+ Just f -> f reportIdentifier
+ Nothing -> makeTitle reportIdentifier diffStr cfg
toFile def outputFile $ do
layout_title .= atitle
@@ -88,7 +116,17 @@ genGroupGraph RawReport{..} cfg@Config{..} = do
layout_x_axis . laxis_generate .=
autoIndexAxis (map (map replaceMu . colName) columns)
layout_x_axis . laxis_style . axis_label_style . font_size .= 16
+
layout_y_axis . laxis_style . axis_label_style . font_size .= 14
+ -- delete the -1x to 1x band of values which are not possible in case
+ -- of fraction style
+ cols' <-
+ case presentation of
+ Groups Multiples -> do
+ layout_y_axis . laxis_generate .= autoScaledAxis
+ transformFractionLabels
+ return $ map transformFractionValue columns
+ _ -> return columns
layout <- get
case _layout_legend layout of
@@ -120,7 +158,7 @@ genGroupGraph RawReport{..} cfg@Config{..} = do
(indexes, [], [])
plot $ fmap plotBars $ bars reportRowIds
- $ (addIndexes $ map colValues columns)
+ $ (addIndexes $ map colValues cols')
-- | Presents the benchmark results in a CSV input file as graphical bar charts
-- according to the provided configuration. The first parameter is the input
@@ -141,7 +179,7 @@ graph :: FilePath -> FilePath -> Config -> IO ()
graph inputFile outputFile cfg@Config{..} = do
let dir = fromMaybe "." outputDir
(csvlines, fields) <- prepareToReport inputFile cfg
- (runs, matrices) <- prepareGroupMatrices cfg csvlines fields
+ (runs, matrices) <- prepareGroupMatrices cfg inputFile csvlines fields
case presentation of
Groups style ->
forM_ fields $
diff --git a/lib/BenchShow/Report.hs b/lib/BenchShow/Report.hs
index 309020b..acf4bdd 100644
--- a/lib/BenchShow/Report.hs
+++ b/lib/BenchShow/Report.hs
@@ -27,6 +27,17 @@ import Text.Printf (printf)
import BenchShow.Common
import BenchShow.Analysis
+multiplesToPercentDiff :: Double -> Double
+multiplesToPercentDiff x = (if x > 0 then x - 1 else x + 1) * 100
+
+colorCode :: Word -> Double -> Doc -> Doc
+colorCode thresh x =
+ if x > fromIntegral thresh
+ then dullred
+ else if x < (-1) * fromIntegral thresh
+ then dullgreen
+ else id
+
-- XXX in comparative reports render lower than baseline in green and higher
-- than baseline in red
genGroupReport :: RawReport -> Config -> IO ()
@@ -35,7 +46,10 @@ genGroupReport RawReport{..} cfg@Config{..} = do
if length reportColumns > 1
then diffString presentation diffStrategy
else Nothing
- putStrLn $ makeTitle reportIdentifier diffStr cfg
+ case mkTitle of
+ Just _ -> putStrLn $ maybe "" (\f -> f reportIdentifier) mkTitle
+ Nothing -> putStrLn $ makeTitle reportIdentifier diffStr cfg
+
let benchcol = "Benchmark" : reportRowIds
groupcols =
let firstCol : tailCols = reportColumns
@@ -43,12 +57,10 @@ genGroupReport RawReport{..} cfg@Config{..} = do
let f x = case presentation of
Groups Diff ->
if x > 0 then dullred else dullgreen
- Groups PercentDiff ->
- if x > fromIntegral threshold
- then dullred
- else if x < (-1) * fromIntegral threshold
- then dullgreen
- else id
+ Groups PercentDiff -> colorCode threshold x
+ Groups Multiples ->
+ let y = multiplesToPercentDiff x
+ in colorCode threshold y
_ -> id
in map f colValues
renderTailCols estimators col analyzed =
@@ -59,6 +71,7 @@ genGroupReport RawReport{..} cfg@Config{..} = do
in case presentation of
Groups Diff -> colored
Groups PercentDiff -> colored
+ Groups Multiples -> colored
_ -> regular
in renderGroupCol (showFirstCol firstCol)
: case reportEstimators of
@@ -133,12 +146,17 @@ genGroupReport RawReport{..} cfg@Config{..} = do
in case presentation of
Groups Diff -> showDiff
Groups PercentDiff -> showDiff
+ Groups Multiples ->
+ if val > 0
+ then printf "%.2f" val
+ else printf "1/%.2f" (negate val)
_ -> printf "%.2f" val
showEstAnnot est =
case presentation of
Groups Diff -> showEstimator est
Groups PercentDiff -> showEstimator est
+ Groups Multiples -> showEstimator est
_ -> ""
in case estimators of
@@ -186,7 +204,7 @@ report :: FilePath -> Maybe FilePath -> Config -> IO ()
report inputFile outputFile cfg@Config{..} = do
let dir = fromMaybe "." outputDir
(csvlines, fields) <- prepareToReport inputFile cfg
- (runs, matrices) <- prepareGroupMatrices cfg csvlines fields
+ (runs, matrices) <- prepareGroupMatrices cfg inputFile csvlines fields
case presentation of
Groups style ->
forM_ fields $
diff --git a/lib/BenchShow/Tutorial.hs b/lib/BenchShow/Tutorial.hs
index 6775580..f12373e 100644
--- a/lib/BenchShow/Tutorial.hs
+++ b/lib/BenchShow/Tutorial.hs
@@ -8,7 +8,13 @@
--
-- BenchShow generates text reports and graphs from benchmarking results. It
-- allows you to manipulate the format of the report and the benchmarking data
--- to present it in many useful ways.
+-- to present it in many useful ways. BenchShow uses robust statistical
+-- analysis using three different statistical estimators to provide as stable
+-- run-to-run comparison of benchmark results as possible. For stable results,
+-- make sure that you are not executing any other tasks on the benchmark host
+-- while benhmarking is going on. For even more stable results, we recommend
+-- using a desktop or server machine instead of a laptop notebook for
+-- benchmarking.
module BenchShow.Tutorial
(
@@ -271,7 +277,7 @@ import BenchShow
-- reverse
-- $ map fst
-- $ sortBy (comparing snd)
--- $ either error id $ f $ 'ColumnIndex' 1
+-- $ either error id $ f ('ColumnIndex' 1) Nothing
-- }
-- @
--
@@ -322,7 +328,7 @@ import BenchShow
-- reverse
-- $ map fst
-- $ sortBy (comparing snd)
--- $ either error id $ f $ 'ColumnIndex' 1
+-- $ either error id $ f ('ColumnIndex' 1) Nothing
-- }
-- @
--
diff --git a/stack-8.2.yaml b/stack-8.2.yaml
index fcce409..fcce409 100644..100755
--- a/stack-8.2.yaml
+++ b/stack-8.2.yaml
diff --git a/stack.yaml b/stack.yaml
index 9b6596d..8285524 100644..100755
--- a/stack.yaml
+++ b/stack.yaml
@@ -1,10 +1,12 @@
-resolver: lts-12.11
+resolver: lts-13.25
packages:
- '.'
extra-deps:
- - Chart-1.9
- - Chart-diagrams-1.9
- - SVGFonts-1.6.0.3
- - statistics-0.15.0.0
- - math-functions-0.3.0.2
- - dense-linear-algebra-0.1.0.0
+ - Chart-1.9.1
+ - Chart-diagrams-1.9.2
+
+ #- SVGFonts-1.6.0.3
+ #- statistics-0.15.0.0
+ #- math-functions-0.3.0.2
+ #- dense-linear-algebra-0.1.0.0
+ #- optparse-applicative-0.14.2.0
diff --git a/test/Doc.hs b/test/Doc.hs
index 0fb5295..a071fbe 100644
--- a/test/Doc.hs
+++ b/test/Doc.hs
@@ -26,17 +26,6 @@ main = do
defaultConfig
{ classifyBenchmark = classifier }
- graph "test/results-doc.csv" "docs/grouped-percent"
- defaultConfig
- { classifyBenchmark = classifier
- , presentation = Groups Percent
- }
- report "test/results-doc.csv" Nothing
- defaultConfig
- { classifyBenchmark = classifier
- , presentation = Groups Percent
- }
-
graph "test/results-doc.csv" "docs/grouped-delta"
defaultConfig
{ classifyBenchmark = classifier
@@ -97,7 +86,7 @@ main = do
reverse
$ map fst
$ sortBy (comparing snd)
- $ either error id $ f $ ColumnIndex 1
+ $ either error id $ f (ColumnIndex 1) Nothing
}
report "test/results-doc.csv" Nothing
defaultConfig
@@ -108,7 +97,7 @@ main = do
reverse
$ map fst
$ sortBy (comparing snd)
- $ either error id $ f $ ColumnIndex 1
+ $ either error id $ f (ColumnIndex 1) Nothing
}
graph
@@ -122,7 +111,7 @@ main = do
reverse
$ map fst
$ sortBy (comparing snd)
- $ either error id $ f $ ColumnIndex 1
+ $ either error id $ f (ColumnIndex 1) Nothing
}
report
"test/results-doc-multi.csv"
@@ -135,5 +124,5 @@ main = do
reverse
$ map fst
$ sortBy (comparing snd)
- $ either error id $ f $ ColumnIndex 1
+ $ either error id $ f (ColumnIndex 1) Nothing
}
diff --git a/test/Main.hs b/test/Main.hs
index 24439ee..66fff4b 100644
--- a/test/Main.hs
+++ b/test/Main.hs
@@ -73,7 +73,7 @@ main = do
let i = intersect (map (last . splitOn "/") prefixes) bs
in i ++ (bs \\ i)
cfg = defaultConfig
- { title = Just chartTitle
+ { mkTitle = Just (\_ -> chartTitle)
, outputDir = Just "charts"
, classifyBenchmark = \bm ->
case any (`isPrefixOf` bm) prefixes of
@@ -82,7 +82,7 @@ main = do
in Just (suffixVersion (xs !! 0), xs !! 1)
False -> Nothing
, selectBenchmarks = \g -> bsort $
- either error (map fst) $ g (ColumnIndex 0)
+ either error (map fst) $ g (ColumnIndex 0) Nothing
, selectGroups = \gs ->
let gs' = map fst gs
i = intersect (map suffixVersion packages) gs'
@@ -112,10 +112,6 @@ main = do
, presentation = Groups Diff
, selectFields = (`intersect` ["time"])
}
- graph "test/results.csvraw" "csvraw-percent"
- cfg { presentation = Groups Percent
- , selectFields = (`intersect` ["time"])
- }
graph "test/results.csvraw" "csvraw-percent-delta"
cfg { presentation = Groups PercentDiff
, selectFields = (`intersect` ["time"])
@@ -144,18 +140,14 @@ main = do
cfg { presentation = Groups Absolute
, selectBenchmarks = \g ->
either error (map fst . sortBy (compare `on` snd))
- (g (ColumnIndex 1))
- }
-
- report "test/results.csv" Nothing
- cfg { presentation = Groups Percent
+ (g (ColumnIndex 1) Nothing)
}
report "test/results.csv" Nothing
cfg { presentation = Groups PercentDiff
, selectBenchmarks = \g ->
either error (map fst . sortBy (compare `on` snd))
- (g (ColumnIndex 1))
+ (g (ColumnIndex 1) Nothing)
}
report "test/results.csv" Nothing
cfg { presentation = Groups Diff
diff --git a/test/results-doc-multi.csv b/test/results-doc-multi.csv
index 3f5e278..3f5e278 100644..100755
--- a/test/results-doc-multi.csv
+++ b/test/results-doc-multi.csv
diff --git a/test/results-doc.csv b/test/results-doc.csv
index b64e7e1..b64e7e1 100644..100755
--- a/test/results-doc.csv
+++ b/test/results-doc.csv
diff --git a/test/results.csv b/test/results.csv
index e57b59b..e57b59b 100644..100755
--- a/test/results.csv
+++ b/test/results.csv
diff --git a/test/results.csvraw b/test/results.csvraw
index 168af4a..168af4a 100644..100755
--- a/test/results.csvraw
+++ b/test/results.csvraw