diff --git a/Changelog.md b/Changelog.md --- a/Changelog.md +++ b/Changelog.md @@ -1,3 +1,41 @@ +## 0.3.0 + +### Breaking Changes + +* The signature of `selectBenchmarks` has changed, use 'Nothing' as the second + argument of the benchmark generator function to port old code without any + impact. +* Removed the broken 'Percent' constructor from `GroupStyle`. Use `PercentDiff` + instead to make relative comparisons. +* The behavior of `PercentDiff` has changed, it now computes the % from the + lower value instead of from the baseline. +* The default `diffStrategy` has been changed to `SingleEstimator` instead of + `MinEstimator`. + +### Deprecations + +* Config fields `title` and `titleAnnotations` have been deprecated, please use + `mkTitle` instead. + +### Bug Fixes + +* `GroupStyle Absolute` now honors the `MinEstimator` setting. When + `MinEstimator` is set, the groups being compared to baseline now display the + value based on the estimator which provides closest estimate to the baseline. + +### Enhancements + +* Add a CLI executable to generate textual reports and graphs from criterion or + gauge csv output file. +* Add `Multiples` as a comparison option, the group being compared is shown as + a multiple of the baseline. +* Add ability to omit the baseline group from the results when we are doing a + relative comparison among groups. +* Add the ability to sort the benchmarks using a different criterion than the + one used to present the benchmarks in the final report output. +* Add `mkTitle` config option to use a function for generating a custom report + title. + ## 0.2.2 * Allow additional annotations to title to be controlled via config diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -1,29 +1,45 @@ # bench-show -Generate text reports and graphical charts from the benchmark results -generated by `gauge` or `criterion`, showing or comparing benchmarks in -many useful ways. In a few lines of code, we can report time taken, peak -memory usage, allocations, among many other fields; we can group benchmarks -and compare the groups; we can compare benchmarks before and after a change; -we can show absolute or percentage difference from the baseline; we can sort -the results to get the worst affected benchmarks by percentage change. - -It can help us in answering questions like the following, visually or -textually: +Generate text reports and graphical charts from the benchmark results generated +by `gauge` or `criterion` and stored in a CSV file. This tool is especially +useful when you have many benchmarks or if you want to compare benchmarks +across multiple packages. You can generate many interesting reports +including: -* Across two benchmark runs, show all the operations that resulted in a - regression of more than 10%, so that we can quickly identify and fix - performance problems in our application. +* Show individual reports for all the fields measured e.g. `time taken`, `peak + memory usage`, `allocations`, among many other fields measured by `gauge` +* Sort benchmark results on a specified criterion e.g. you may want to see the + biggest cpu hoggers or biggest memory hoggers on top +* Across two benchmark runs (e.g. before and after a change), show all the + operations that resulted in a regression of more than x% in descending order, + so that we can quickly identify and fix performance problems in our + application. * Across two (or more) packages providing similar functionality, show all the operations where the performance differs by more than 10%, so that we can critically analyze the packages and choose the right one. ## Quick Start -Use `gauge` or `criterion` to generate a `results.csv` file, and then use the -following code to generate a textual report or a graph: +Use `gauge` or `criterion` to generate a `results.csv` file, and then use +either the `bench-show` executable or the library APIs to generate textual or +graphical reports. +## Executable + +Use `bench-show` executable with `report` and `graph` sub-commands: + ``` +$ bench-show report results.csv +$ bench-show graph results.csv output +``` + +For advanced usage, control the generated report by the CLI flags. + +## Library + +Use `report` and `graph` library functions: + +``` report "results.csv" Nothing defaultConfig graph "results.csv" "output" defaultConfig ``` @@ -37,6 +53,10 @@ can select many fields from a `gauge` raw report. Units of the fields are automatically determined based on the range of values: +``` +$ bench-show --presentation Fields report results.csv +``` + ```haskell report "results.csv" Nothing defaultConfig { presentation = Fields } ``` @@ -53,6 +73,10 @@ ``` `graph` generates one bar chart per field: + +``` +$ bench-show --presentation Fields graph results.csv +``` ``` graph "results.csv" "output" defaultConfig diff --git a/app/Main.hs b/app/Main.hs new file mode 100644 --- /dev/null +++ b/app/Main.hs @@ -0,0 +1,187 @@ +{-# LANGUAGE CPP #-} +{-# LANGUAGE RecordWildCards #-} +{-# LANGUAGE TemplateHaskell #-} +{-# LANGUAGE TupleSections #-} + +#if __GLASGOW_HASKELL__ >= 800 +{-# OPTIONS_GHC -Wno-deprecations #-} +#endif + +-- | +-- Module : Main +-- Copyright : (c) 2019 Composewell Technologies +-- (c) 2019 Sanchayan Maity +-- License : BSD3 +-- Maintainer : harendra.kumar@gmail.com +-- Stability : experimental +-- Portability : GHC +-- +module Main where + +import Data.Semigroup ((<>)) +import Options.Applicative.Simple (addCommand, simpleOptions, simpleVersion) +import Paths_bench_show (version) + +import Options.Applicative +import BenchShow + +-- The command line interface provides subcommands for each task. There are +-- some common options that apply to all subcommands. Subcommands may have +-- their own specific options as well. + +------------------------------------------------------------------------------- +-- Common config options +------------------------------------------------------------------------------- + +pVerbose :: Parser Bool +pVerbose = switch $ + long "verbose" + <> short 'v' + <> help "Provide more details in the report output" + +pOutputDir :: Parser FilePath +pOutputDir = strOption + ( long "output-dir" + <> metavar "DIR" + <> help "Default is current directory" ) + +pTitle :: Parser String +pTitle = strOption + ( long "title" + <> short 't' + <> metavar "STRING" + <> help "Title for the report" ) + +pTitleAnnotation :: Parser TitleAnnotation +pTitleAnnotation = option auto $ + long "title-annotations" + <> help ("*TitleField*|TitleEstimator|TitleDiff") + +pPresentation :: Parser Presentation +pPresentation = option auto $ + long "presentation" + <> help ("Solo|Fields|*Groups* <*Absolute*|Diff|PercentDiff|Multiples") + +pEstimator :: Parser Estimator +pEstimator = option auto $ + long "estimator" + <> help ("*Median*|Mean|Regression") + +pThreshold :: Parser Word +pThreshold = option auto $ + long "threshold" + <> metavar "PERCENT" + <> help "Min % diff (default 3) to flag regression or improvement" + +pDiffStrategy :: Parser DiffStrategy +pDiffStrategy = option auto $ + long "diff-strategy" + <> help ("*SingleEstimator*|MinEstimator") + +pOmitBaseline :: Parser Bool +pOmitBaseline = switch $ + long "omit-baseline" + <> help "omit the baseline column in relative comparisons" + +------------------------------------------------------------------------------- +-- Build a Config parser for common options +------------------------------------------------------------------------------- + +-- Specify a default value for a Maybe inside a functor +fMaybe :: Functor f => a -> f (Maybe a) -> f a +fMaybe a = fmap (maybe a id) + +-- | parse an optional field with a default value taken from defaultConfig +parseOptional :: (Config -> a) -> Parser a -> Parser a +parseOptional def parser = fMaybe (def defaultConfig) (optional parser) + +pConfig :: Parser Config +pConfig = Config + <$> parseOptional verbose pVerbose + <*> optional pOutputDir + <*> pure Nothing + <*> optional pTitle + <*> many pTitleAnnotation + <*> parseOptional presentation pPresentation + <*> parseOptional estimator pEstimator + <*> parseOptional threshold pThreshold + <*> parseOptional diffStrategy pDiffStrategy + <*> parseOptional omitBaseline pOmitBaseline + <*> pure (selectFields defaultConfig) + <*> pure (fieldRanges defaultConfig) + <*> pure (fieldTicks defaultConfig) + <*> pure (classifyBenchmark defaultConfig) + <*> pure (selectGroups defaultConfig) + <*> pure (selectBenchmarks defaultConfig) + +------------------------------------------------------------------------------- +-- "report" subcommand +------------------------------------------------------------------------------- + +data ReportOpts = ReportOpts + { reportInput :: FilePath + , reportOutput :: Maybe FilePath + } + +pInputFile :: Parser FilePath +pInputFile = argument str (metavar "INPUT-FILE.CSV") + +pOutputFile :: Parser FilePath +pOutputFile = strOption + (long "output" + <> short 'o' + <> help "Output file") + +pReportOpts :: Parser ReportOpts +pReportOpts = ReportOpts + <$> pInputFile + <*> optional pOutputFile + +cmdReport :: ReportOpts -> Config -> IO () +cmdReport ReportOpts{..} cfg = report reportInput reportOutput cfg + +------------------------------------------------------------------------------- +-- "graph" subcommand +------------------------------------------------------------------------------- + +data GraphOpts = GraphOpts + { graphInput :: FilePath + , graphOutput :: FilePath + } + +pOutputArg :: Parser FilePath +pOutputArg = argument str (metavar "OUTPUT-FILE-PREFIX") + +pGraphOpts :: Parser GraphOpts +pGraphOpts = GraphOpts + <$> pInputFile + <*> pOutputArg + +cmdGraph :: GraphOpts -> Config -> IO () +cmdGraph GraphOpts{..} cfg = graph graphInput graphOutput cfg + +------------------------------------------------------------------------------- +-- Build and run a subcommand parser +------------------------------------------------------------------------------- + +cmdLineParser :: Parser Config -> IO () +cmdLineParser p = do + (cfg, handler) <- simpleOptions $(simpleVersion version) + ("Generate reports and graphs from gauge or criterion output.\n" ++ + "Default values are shown as *DEFAULT*.") + "" p cmds + + handler cfg + + where cmds = do + addCommand "report" + "Generate a text report" + cmdReport + pReportOpts + addCommand "graph" + "Generate a graphical report" + cmdGraph + pGraphOpts + +main :: IO () +main = cmdLineParser pConfig diff --git a/bench-show.cabal b/bench-show.cabal --- a/bench-show.cabal +++ b/bench-show.cabal @@ -1,40 +1,45 @@ name: bench-show -version: 0.2.2 +version: 0.3.0 license: BSD3 author: Harendra Kumar maintainer: harendra.kumar@gmail.com bug-reports: https://github.com/composewell/bench-show/issues synopsis: Show, plot and compare benchmark results description: - Generate text reports and graphical charts from benchmark results - generated by @gauge@ or @criterion@, showing or comparing benchmarks in - many useful ways. In a few lines of code, we can report time taken, peak - memory usage, allocations, among many other fields; we can arrange benchmarks - in groups and compare the groups; we can compare benchmark results before and - after a change; we can show the difference from baseline in absolute terms or - as a percentage; we can sort the results by percentage change to get the - worst affected benchmarks. - . - @bench-show@ helps us in answering questions like the following, visually or - textually: + Generate text reports and graphical charts from the benchmark results generated + by @gauge@ or @criterion@ and stored in a CSV file. This tool is especially + useful when you have many benchmarks or if you want to compare benchmarks + across multiple packages. You can generate many interesting reports + including: . - * Across two benchmark runs, show all the operations that resulted in a - regression of more than 10%, so that we can quickly identify and fix - performance problems in our application. - * Across two (or more) packages (providing similar functionality), show all - the operations where the performance differs by more than 10%, so that we - can critically analyze the packages and choose the right one. + * Show individual reports for all the fields measured e.g. @time taken@, @peak + memory usage@, @allocations@, among many other fields measured by @gauge@ + * Sort benchmark results on a specified criterion e.g. you may want to see the + biggest cpu hoggers or biggest memory hoggers on top + * Across two benchmark runs (e.g. before and after a change), show all the + operations that resulted in a regression of more than x% in descending + order, so that we can quickly identify and fix performance problems in our + application. + * Across two (or more) packages providing similar functionality, show all the + operations where the performance differs by more than 10%, so that we can + critically analyze the packages and choose the right one. . Quick Start: Use @gauge@ or @criterion@ to generate a @results.csv@ file, and - then use the following code to generate a textual report or a graph: + then use either the @bench-show@ executable or the library APIs to generate + textual or graphical reports. . @ + $ bench-show report results.csv + $ bench-show graph results.csv output + @ + . + @ report "results.csv" Nothing defaultConfig graph "results.csv" "output" defaultConfig @ . - For example, you can show can show % regression from a baseline in descending - order texttually as follows: + There are many ways to present the reports, for example, you can show can + show % regression from a baseline in descending order textually as follows: . @ (time)(Median)(Diff using min estimator) @@ -55,7 +60,8 @@ category: Performance, Benchmarking homepage: https://github.com/composewell/bench-show license-file: LICENSE -tested-with: GHC==8.4.4 +tested-with: GHC==8.6.5 + , GHC==8.4.4 , GHC==8.2.2 , GHC==7.10.3 copyright: 2017, 2018 Composewell Technologies @@ -122,6 +128,30 @@ , split >= 0.2 && < 0.3 , statistics >= 0.15 && < 0.16 , vector >= 0.10 && < 0.13 + +executable bench-show + default-language: Haskell2010 + hs-source-dirs: app + main-is: Main.hs + other-modules: Paths_bench_show + ghc-options: -Wall + build-depends: + base >= 4.8 && < 5 + , Chart >= 1.6 && < 2 + , Chart-diagrams >= 1.6 && < 2 + , csv >= 0.1 && < 0.2 + , filepath >= 1.3 && < 1.5 + , mwc-random >= 0.13 && < 0.15 + , directory >= 1.2 && < 1.4 + , transformers >= 0.4 && < 0.6 + , ansi-wl-pprint >= 0.6 && < 0.7 + , split >= 0.2 && < 0.3 + , statistics >= 0.15 && < 0.16 + , vector >= 0.10 && < 0.13 + , semigroups >= 0.18 && < 0.20 + , optparse-applicative >= 0.14.2 && < 0.14.4 + , optparse-simple >= 0.1.0 && < 0.2 + , bench-show test-suite test type: exitcode-stdio-1.0 diff --git a/lib/BenchShow.hs b/lib/BenchShow.hs --- a/lib/BenchShow.hs +++ b/lib/BenchShow.hs @@ -52,12 +52,12 @@ -- from the textual report, the rows (i.e. the benchmarks) are represented as -- bars in the cluster. -- --- When the columns are groups, each report consists of results for a single +-- When the columns are groups, each report presents the results for a single -- benchmarking field for different benchmark groups. Using 'GroupStyle', we --- can further specify how we want to present the results the groups. We can +-- can further specify how we want to present the results of the groups. We can -- either present absolute values of the field for each group or we can make --- the first group as a baseline and present differences from the baseline for --- the subsequent groups. +-- the first group as a baseline showing absolute values and present difference +-- from the baseline for the subsequent groups. -- -- When the columns are fields, each report consists of results for a single -- benchmarking group. Fields cannot be compared like groups because they are diff --git a/lib/BenchShow/Analysis.hs b/lib/BenchShow/Analysis.hs --- a/lib/BenchShow/Analysis.hs +++ b/lib/BenchShow/Analysis.hs @@ -270,7 +270,7 @@ -- better when larger number of samples are taken. This -- cannot be used when the number of samples is less than -- 2, in that case a mean value is reported instead. - deriving (Eq, Show) + deriving (Eq, Show, Read) getAnalyzedValue :: Estimator -> AnalyzedField -> Double getAnalyzedValue estimator AnalyzedField{..} = diff --git a/lib/BenchShow/Common.hs b/lib/BenchShow/Common.hs --- a/lib/BenchShow/Common.hs +++ b/lib/BenchShow/Common.hs @@ -49,7 +49,7 @@ import Data.Function ((&), on) import Data.List (transpose, groupBy, (\\), find, sortBy, elemIndex, intersect, - intersectBy, concatMap) + intersectBy) import Data.List.Split (linesBy) import Data.Maybe (fromMaybe, mapMaybe) import Data.Ord (comparing) @@ -78,25 +78,35 @@ label ++ " must not add any new items to the original list. The \ \following items were added: " ++ show added + ++ "\nOriginal groups: " ++ show old + ++ "\nNew groups: " ++ show new ------------------------------------------------------------------------------- data ReportType = TextReport | GraphicalChart -- | How to show the results for multiple benchmark groups presented in columns --- or bar chart clusters. +-- or bar chart clusters. In relative comparisons, the first group is +-- considered as the baseline and the subsequent groups are compared against +-- the baseline. -- +-- /Definition changed in 0.3.0/ -- @since 0.2.0 data GroupStyle = - Absolute -- ^ Show absolute field values for all groups - | Diff -- ^ Show baseline group values as usual and values for - -- the subsequent groups as differences from the baseline - | Percent -- ^ Show baseline group values as 100% and values for - -- subsequent groups as a percentage of the baseline - | PercentDiff -- ^ Show baseline group values as usual and values for - -- subsequent groups as precentage difference from the - -- baseline - deriving (Eq, Show) + Absolute -- ^ Show absolute values of the field for all groups + | Diff -- ^ Show baseline group as absolute values and values for + -- the subsequent groups as difference from the baseline + | PercentDiff -- ^ If the value of the group being compared is higher than + -- the baseline then display the difference as percentage of + -- baseline otherwise display the difference as a percentage + -- of the group being compared. + | Multiples -- ^ If the value of the group being compared is higher than + -- the baseline then display @+(value / baseline value)@ + -- otherwise display @-(baseline value / value)@. This + -- provides a normalized comparison independent of the + -- absolute value of a benchmark. Note that 'Multiples' can + -- be directly computed using 'PercentDiff' and vice-versa. + deriving (Eq, Show, Read) -- | How to present the reports or graphs. Each report presents a number of -- benchmarks as rows, it may have, (1) a single column presenting the values @@ -124,7 +134,7 @@ -- with all the fields selected by the configuration as -- columns or clusters. Output files are named using -- @-estimator-groupname@ as suffix. - deriving (Eq, Show) + deriving (Eq, Show, Read) -- | FieldTick is used only in visual charts to generate the major ticks on -- the y-axis. You can specify either the size of a tick ('TickSize') or the @@ -175,13 +185,15 @@ | WorstBest | BestBest -} + deriving (Show, Read) -- | Additional annotations that can be optionally added to the title of the -- report or graph. -- -- @since 0.2.2 +{-# DEPRECATED TitleAnnotation "Please use mkTitle to make a custom title" #-} data TitleAnnotation = TitleField | TitleEstimator | TitleDiff - deriving (Eq, Show) + deriving (Eq, Show, Read) -- | Configuration governing generation of chart. See 'defaultConfig' for the -- default values of these fields. @@ -197,11 +209,19 @@ -- | The directory where the output graph or report file should be placed. , outputDir :: Maybe FilePath - -- | Report title, more information like the plotted field name or + -- | Function to make a title for the report. The argument to the function + -- is the benchmark field name for which the report is being made. + , mkTitle :: Maybe (String -> String) + + -- | /DEPRECATED: Please use 'mkTitle' instead./ + -- + -- Report title, more information like the plotted field name or -- the presentation style may be added to it. , title :: Maybe String - -- | Additional annotations to be added to the title + -- | /DEPRECATED: Please use 'mkTitle' instead./ + -- + -- Additional annotations to be added to the title , titleAnnotations :: [TitleAnnotation] -- | How to determine the layout of the report or the chart. @@ -217,6 +237,10 @@ -- | Strategy to compare two runs or groups of benchmarks. , diffStrategy :: DiffStrategy + -- | Omit the baseline group in normalized relative comparisons i.e. + -- when the 'GroupStyle' is 'PercentDiff' or 'Multiples'. + , omitBaseline :: Bool + --------------------------------------------------------------------------- -- Fields (Columns) --------------------------------------------------------------------------- @@ -263,54 +287,68 @@ -- Benchmarks (Rows) --------------------------------------------------------------------------- - -- | Filter and reorder benchmarks. 'selectBenchmarks' is provided with a - -- function which is invoked with a sorting column name or index, the - -- function produces the benchmark names and corresponding values for that - -- column which can be used as a sorting criterion. The output of - -- 'selectBenchmarks' is a list of benchmarks in the order in which they - -- are to be rendered. + -- | Filter and reorder benchmarks. 'selectBenchmarks' takes a function + -- argument, the function is invoked with a sorting column name or index + -- and a 'GroupStyle'. The output of the function is either a 'Right' value + -- consisting of tuples of the benchmark names and values corresponding to + -- the given column and style or a 'Left' value indicating an error. + -- 'selectBenchmarks' can inspect these benchmarks and there values to + -- produce a filtered and sorted list of benchmark names that are to be + -- rendered. + -- + -- The style argument is ignored when the report presentation is not + -- 'Groups'. When style is 'Nothing', the presentation setting specified in + -- the configuration is used. + -- + -- /Signature changed in 0.3.0/ , selectBenchmarks - :: (SortColumn -> Either String [(String, Double)]) + :: (SortColumn -> Maybe GroupStyle -> Either String [(String, Double)]) -> [String] } +-- IMPORTANT: If you change the defaults, please change the defaults in the CLI +-- help as well. +-- -- | Default configuration. Use this as the base configuration and modify the -- required fields. The defaults are: -- -- @ -- verbose = False --- title = Nothing +-- mkTitle = Nothing -- titleAnnotations = [TitleField] -- outputDir = Nothing -- presentation = Groups Absolute -- estimator = Median -- threshold = 3 --- diffStrategy = MinEstimator +-- diffStrategy = SingleEstimator +-- omitBaseline = False -- selectFields = filter (flip elem ["time", "mean", "maxrss"] . map toLower) -- fieldRanges = [] -- fieldTicks = [] -- classifyBenchmark = Just . ("default",) -- selectGroups = id --- selectBenchmarks = \f -> either error (map fst) $ f (ColumnIndex 0) +-- selectBenchmarks = \f -> either error (map fst) $ f (ColumnIndex 0) Nothing -- @ -- -- @since 0.2.0 defaultConfig :: Config defaultConfig = Config { verbose = False + , mkTitle = Nothing , title = Nothing , titleAnnotations = [TitleField] , outputDir = Nothing , presentation = Groups Absolute , estimator = Median , threshold = 3 - , diffStrategy = MinEstimator + , diffStrategy = SingleEstimator + , omitBaseline = False , selectFields = filter (flip elem ["time", "mean", "maxrss"] . map toLower) , fieldRanges = [] , fieldTicks = [] , classifyBenchmark = Just . ("default",) , selectGroups = id - , selectBenchmarks = \f -> either error (map fst) $ f (ColumnIndex 0) + , selectBenchmarks = \f -> either error (map fst) $ f (ColumnIndex 0) Nothing } ------------------------------------------------------------------------------- @@ -389,8 +427,8 @@ fieldUnits :: String -> Double -> GroupStyle -> RelativeUnit fieldUnits fieldName fieldMin style = case style of - Percent -> RelativeUnit "%" 1 - PercentDiff -> RelativeUnit "%" 1 + Multiples -> RelativeUnit "x" 1 + PercentDiff -> RelativeUnit "%" 1 _ -> getUnitByFieldName fieldName fieldMin ------------------------------------------------------------------------------- @@ -400,11 +438,19 @@ absoluteDiff :: Num a => a -> a -> a absoluteDiff v1 v2 = v2 - v1 -percentDiff :: (Fractional a, Num a) => a -> a -> a -percentDiff v1 v2 = ((v2 - v1) * 100) / v1 +percentDiff :: (Fractional a, Num a, Ord a) => a -> a -> a +percentDiff v1 v2 = ((v2 - v1) * 100) / min v1 v2 -percent :: (Fractional a, Num a) => a -> a -> a -percent v1 v2 = (v2 * 100) / v1 +-- We map a fraction x between 0 and 1 to a negative 1/x for plotting on an +-- equal and opposite scale. +fraction :: (Fractional a, Num a, Ord a, Show a) => a -> a -> a +fraction v1 v2 = + let val = v2 / v1 + in case val of + x | x <= 0 -> error $ "BenchShow.Common.fraction: negative: " ++ show x + x | x < 1 -> negate (1 / x) + x | x >= 1 -> x + x -> error $ "BenchShow.Common.fraction: unhandled: " ++ show x cmpTransformColumns :: ReportType -> GroupStyle @@ -426,37 +472,41 @@ let firstCol = head cols colTransform col = zipWith (mkMinDiff diff) firstCol col in map colTransform (tail cols) - in case style of - Absolute -> (Nothing, columns) - Percent -> (Nothing, cmpWith percent) - Diff -> + + diffWith f = case diffStrategy of MinEstimator -> - let (ests, vals) = unzip $ map unzip (cmpMinWith absoluteDiff) + let (ests, vals) = unzip $ map unzip (cmpMinWith f) in ( Just $ map (const estimator) (head cols) : ests , head columns : vals ) SingleEstimator -> - (Nothing, head columns : cmpWith absoluteDiff) - PercentDiff -> - -- In a comparative graphical chart we cannot show the absolute - -- values in the baseline column as the units won't match for - -- the baseline and the diff clusters. - let baseCol = - case rtype of - TextReport -> head columns - GraphicalChart | length columns == 1 -> - head columns - GraphicalChart -> - map (\(n,_) -> (n,100)) (head columns) - in case diffStrategy of - MinEstimator -> - let (ests, vals) = unzip $ map unzip (cmpMinWith percentDiff) - in ( Just $ map (const estimator) (head cols) : ests - , baseCol : vals - ) - SingleEstimator -> - (Nothing, baseCol : cmpWith percentDiff) + (Nothing, head columns : cmpWith f) + + relativeDiffWith baseVal f = + -- In a comparative graphical chart we cannot show the absolute + -- values in the baseline column as the units won't match for + -- the baseline and the diff clusters. + let baseCol = + case rtype of + TextReport -> head columns + GraphicalChart | length columns == 1 -> + head columns + GraphicalChart -> + map (\(n,_) -> (n,baseVal)) (head columns) + in case diffStrategy of + MinEstimator -> + let (ests, vals) = unzip $ map unzip (cmpMinWith f) + in ( Just $ map (const estimator) (head cols) : ests + , baseCol : vals + ) + SingleEstimator -> + (Nothing, baseCol : cmpWith f) + in case style of + Absolute -> diffWith (\_ x -> x) + Diff -> diffWith absoluteDiff + PercentDiff -> relativeDiffWith 100 percentDiff + Multiples -> relativeDiffWith 1 fraction where verify a b = if a then b else error "bug: benchmark names mismatch" transformVals = map (map (second (getAnalyzedValue estimator))) @@ -478,20 +528,28 @@ then (Mean, (n2, meanDiff)) else (Regression, (n2, regDiff)) -transformColumnNames :: GroupStyle -> [ReportColumn] -> [ReportColumn] -transformColumnNames _ [] = [] -transformColumnNames style columns@(h:t) = - let withDiff = colSuffix baseName h : map (colSuffix diffName) t +columnNameByStyle :: GroupStyle -> [ReportColumn] -> [ReportColumn] +columnNameByStyle _ [] = [] +columnNameByStyle style columns@(h:t) = + let withDiff name = colSuffix baseName h : map (colSuffix name) t in case style of - Diff | length columns > 1 -> withDiff - PercentDiff | length columns > 1 -> withDiff - _ -> columns + Diff | length columns > 1 -> withDiff diffName + Multiples | length columns > 1 -> withDiff fracName + PercentDiff | length columns > 1 -> withDiff diffName + _ -> columns where colSuffix xl col = col { colName = xl (colName col) } - baseName = (++ "(base)") - diffName = (++ "(-base)") + baseName = id -- (++ "(base)") + diffName = (++ " - " ++ colName h) -- "(-base)") + fracName = (++ "/" ++ colName h) -- "/base") +columnNameByUnit :: [RelativeUnit] -> [ReportColumn] -> [ReportColumn] +columnNameByUnit units columns = + let applyUnit col (RelativeUnit label _) = + col { colName = colName col ++ inParens label } + in zipWith applyUnit columns units + -- Represents the data for a single benchmark run data GroupMatrix = GroupMatrix { groupIndex :: Int @@ -599,8 +657,9 @@ selectBenchmarksByField :: Config -> [GroupMatrix] -> [[(String, Double)]] + -> (GroupStyle -> [[(String, Double)]]) -> [String] -selectBenchmarksByField Config{..} matrices columns = +selectBenchmarksByField Config{..} matrices columns colsByStyle = let bmnames = selectBenchmarks extractGroup in if (null bmnames) then error $ "selectBenchmarks must select at least one benchmark" @@ -621,17 +680,22 @@ in map getName matrices -- columns are benchmark groups in this case - extractGroup (ColumnName (Left name)) = - let len = length columns - in if len <= 1 - then extractGroup $ ColumnName (Right (name, 0)) - else Left $ "selectBenchmarks: there are " ++ show len - ++ " runs in the input data, please specify the run \ - \index [0-" ++ show (len - 1) - ++ "] along with the group name." - extractGroup (ColumnName (Right (name, runId))) = - extractColumnByGroupName name runId - extractGroup (ColumnIndex n) = extractColumnByGroupIndex n + extractGroup colSelector style = + let cols = case style of + Nothing -> columns + Just s -> colsByStyle s + in case colSelector of + ColumnName (Left name) -> + let len = length cols + in if len <= 1 + then extractGroup (ColumnName (Right (name, 0))) style + else Left $ "selectBenchmarks: there are " ++ show len + ++ " runs in the input data, please specify the run \ + \index [0-" ++ show (len - 1) + ++ "] along with the group name." + ColumnName (Right (name, runId)) -> + extractColumnByGroupName name runId style + ColumnIndex n -> extractColumnByGroupIndex n cols -- The benchmark field is constant. Extract all benchmark values for the -- given field and for the given group. @@ -645,20 +709,20 @@ True -> res in foldl foldFunc (0, False) mxs - extractColumnByGroupName name runId = + extractColumnByGroupName name runId style = case findColumnIndex matrices (name, runId) of (_, False) -> Left $ "Benchmark group name [" ++ name ++ "] and index [" ++ show runId ++ "] not found. Available groups are: " ++ show grpNames - (i, True) -> extractGroup (ColumnIndex i) + (i, True) -> extractGroup (ColumnIndex i) style - extractColumnByGroupIndex idx = - let len = length columns + extractColumnByGroupIndex idx cols = + let len = length cols in if idx >= len then Left $ "Column index must be in the range [0-" ++ show (len - 1) ++ "]" - else Right $ columns !! idx + else Right $ cols !! idx selectBenchmarksByGroup :: Config -> GroupMatrix -> [String] selectBenchmarksByGroup Config{..} grp@GroupMatrix{..} = @@ -671,11 +735,11 @@ where -- columns are benchmark fields in this case - extractField (ColumnName (Left name)) = extractColumnByFieldName name - extractField (ColumnName (Right (name, _))) = + extractField (ColumnName (Left name)) _ = extractColumnByFieldName name + extractField (ColumnName (Right (name, _))) _ = -- XXX runId does not make sense for fields extractColumnByFieldName name - extractField (ColumnIndex n) = extractColumnByFieldIndex n + extractField (ColumnIndex n) _ = extractColumnByFieldIndex n -- The benchmark field is constant. Extract all benchmark values for the -- given field and for the given group. @@ -973,12 +1037,30 @@ unless (null absent) $ putStrLn msg return matrix { groupBenches = newBenches } -prepareGroupMatrices :: Config -> CSV -> [String] -> IO (Int, [GroupMatrix]) -prepareGroupMatrices cfg@Config{..} csvlines fields = do - let (hdr, runs) = +prepareGroupMatrices :: Config + -> FilePath + -> CSV + -> [String] + -> IO (Int, [GroupMatrix]) +prepareGroupMatrices cfg@Config{..} inputFile csvlines fields = do + let res@(_, ls) = sanityCheckCSV csvlines & splitRuns - & ensureIterField + + when (null ls) $ + error $ "No benchmark results found in CSV file: " ++ show inputFile + + let checkForData (runId, xs) = + when (null xs) $ + putStrLn $ "No benchmark results found in the CSV file [" + ++ show inputFile + ++ "], for runId: " + ++ show runId + + mapM_ checkForData (zip [(0 :: Int)..] ls) + + let (hdr, runs) = ensureIterField res + xs <- sequence $ map (readIterations hdr) runs & map (filterFields fields) -- & _filterCommonSubsets @@ -1064,30 +1146,40 @@ (estimators, transformedCols) = cmpTransformColumns rtype style estimator diffStrategy unsortedCols - benchmarks = selectBenchmarksByField cfg matrices transformedCols + transformedColsByStyle s = snd $ + cmpTransformColumns rtype s estimator diffStrategy unsortedCols + + benchmarks = selectBenchmarksByField cfg matrices + transformedCols transformedColsByStyle sortedCols = map (sortValues benchmarks) transformedCols origSortedCols = map (sortValues benchmarks) unsortedCols mkColUnits :: [RelativeUnit] mkColUnits = let cols = - if style == Diff || style == PercentDiff + if style /= Absolute -- if we consider diff values as well here then the -- units will change to potentially very small. then [head sortedCols] else sortedCols minVal = getFieldMin cfg (minimum $ concat cols) field - in case (rtype, style) of - -- In case of percentDiff in TextReport we use absolute - -- values in the baseline column, so the unit is different. - (TextReport, PercentDiff) -> + mkPercentColUnitText = let unit = fieldUnits field minVal Absolute punit = fieldUnits field 1 style -- % unit in unit : replicate (length matrices - 1) punit - (GraphicalChart, PercentDiff) | length matrices == 1 -> - [fieldUnits field minVal Absolute] - _ -> let unit = fieldUnits field minVal style + mkPercentColUnitGraph = [fieldUnits field minVal Absolute] + mkAbsoluteUnit = + let unit = fieldUnits field minVal style in replicate (length matrices) unit + in case (rtype, style) of + -- In case of percentDiff in TextReport we use absolute + -- values in the baseline column, so the unit is different. + (_, Absolute) -> mkAbsoluteUnit + (_, Diff) -> mkAbsoluteUnit + (TextReport, _) -> mkPercentColUnitText + (GraphicalChart, _) | length matrices == 1 -> + mkPercentColUnitGraph + (GraphicalChart, _) -> mkAbsoluteUnit mkColValues :: [[Double]] mkColValues = @@ -1102,33 +1194,47 @@ if runs > 1 then "(" ++ show (groupIndex x) ++ ")" else "" - applyUnit name (RelativeUnit label _) = - name ++ inParens label - in zipWith applyUnit (map withSuffix matrices) mkColUnits + in map withSuffix matrices columns = getZipList $ ReportColumn <$> ZipList mkColNames <*> ZipList mkColUnits <*> ZipList mkColValues + removeBaseline xs = + let rel = case style of + Absolute -> False + Diff -> False + PercentDiff -> True + Multiples -> True + in if omitBaseline && rel && length matrices > 1 + then tail xs + else xs + in RawReport { reportOutputFile = outfile , reportIdentifier = field , reportRowIds = benchmarks - , reportColumns = transformColumnNames style columns - , reportAnalyzed = zipWith (\x y -> map (scaleAnalyzedField x) y) - mkColUnits origSortedCols - , reportEstimators = estimators + , reportColumns = removeBaseline + $ columnNameByUnit mkColUnits + $ columnNameByStyle style columns + , reportAnalyzed = removeBaseline $ + zipWith (\x y -> map (scaleAnalyzedField x) y) + mkColUnits origSortedCols + , reportEstimators = fmap removeBaseline estimators } showStatusMessage :: Show a => Config -> String -> Maybe a -> IO () showStatusMessage cfg field outfile = - let atitle = makeTitle field (diffString (presentation cfg) - (diffStrategy cfg)) cfg + let atitle = case mkTitle cfg of + Just f -> " [" ++ f field ++ "]" + Nothing -> + " [" ++ makeTitle field (diffString (presentation cfg) + (diffStrategy cfg)) cfg ++ "]" in case outfile of Just path -> - putStrLn $ "Creating chart " - ++ "[" ++ atitle ++ "]" + putStrLn $ "Creating chart" + ++ atitle ++ " at " ++ show path Nothing -> return () @@ -1235,9 +1341,12 @@ diffString :: Presentation -> DiffStrategy -> Maybe String diffString style s = case style of - Groups Diff -> Just $ "Diff " ++ showDiffStrategy s - Groups PercentDiff -> Just $ "Diff " ++ showDiffStrategy s - _ -> Nothing + Groups Diff -> Just $ "Diff from Baseline " ++ showDiffStrategy s + Groups PercentDiff -> Just $ "Diff % of Lower " ++ showDiffStrategy s + Groups Multiples -> Just $ "Multiples of Baseline " ++ showDiffStrategy s + Groups Absolute -> Nothing + Solo -> Nothing + Fields -> Nothing inParens :: String -> String inParens str = "(" ++ str ++ ")" diff --git a/lib/BenchShow/Graph.hs b/lib/BenchShow/Graph.hs --- a/lib/BenchShow/Graph.hs +++ b/lib/BenchShow/Graph.hs @@ -64,6 +64,32 @@ }] else columns +-- We do not want to see the band of values between -1 and 1, in fact there are +-- no values possible in that band. Shift the positive values by -1 and +-- negative values by +1 to map them to a 0 based scale on the graph. We change +-- the labels as well accordingly. +transformFractionValue :: ReportColumn -> ReportColumn +transformFractionValue ReportColumn{..} = + ReportColumn + { colName = colName + , colUnit = colUnit + , colValues = map (\val -> + case val of + x | x >= 1 -> x - 1 + x | x < (-1) -> x + 1 + x -> error $ "BenchShow.Graph.transformFractionValue: unhandled: " ++ show x + ) colValues + } + +transformFractionLabels :: LinearAxisParams Double +transformFractionLabels = + ((def :: (LinearAxisParams Double)) { _la_labelf = \xs -> + let shiftVals v = if v >= 0 then v + 1 else v - 1 + replaceMinus ('-' : ys) = "1/" ++ ys + replaceMinus ys = ys + in fmap replaceMinus (_la_labelf def (map shiftVals xs)) + }) + genGroupGraph :: RawReport -> Config -> IO () genGroupGraph RawReport{..} cfg@Config{..} = do let outputFile = fromMaybe undefined reportOutputFile @@ -79,7 +105,9 @@ if length reportColumns > 1 then diffString presentation diffStrategy else Nothing - atitle = makeTitle reportIdentifier diffStr cfg + atitle = case mkTitle of + Just f -> f reportIdentifier + Nothing -> makeTitle reportIdentifier diffStr cfg toFile def outputFile $ do layout_title .= atitle @@ -88,7 +116,17 @@ layout_x_axis . laxis_generate .= autoIndexAxis (map (map replaceMu . colName) columns) layout_x_axis . laxis_style . axis_label_style . font_size .= 16 + layout_y_axis . laxis_style . axis_label_style . font_size .= 14 + -- delete the -1x to 1x band of values which are not possible in case + -- of fraction style + cols' <- + case presentation of + Groups Multiples -> do + layout_y_axis . laxis_generate .= autoScaledAxis + transformFractionLabels + return $ map transformFractionValue columns + _ -> return columns layout <- get case _layout_legend layout of @@ -120,7 +158,7 @@ (indexes, [], []) plot $ fmap plotBars $ bars reportRowIds - $ (addIndexes $ map colValues columns) + $ (addIndexes $ map colValues cols') -- | Presents the benchmark results in a CSV input file as graphical bar charts -- according to the provided configuration. The first parameter is the input @@ -141,7 +179,7 @@ graph inputFile outputFile cfg@Config{..} = do let dir = fromMaybe "." outputDir (csvlines, fields) <- prepareToReport inputFile cfg - (runs, matrices) <- prepareGroupMatrices cfg csvlines fields + (runs, matrices) <- prepareGroupMatrices cfg inputFile csvlines fields case presentation of Groups style -> forM_ fields $ diff --git a/lib/BenchShow/Report.hs b/lib/BenchShow/Report.hs --- a/lib/BenchShow/Report.hs +++ b/lib/BenchShow/Report.hs @@ -27,6 +27,17 @@ import BenchShow.Common import BenchShow.Analysis +multiplesToPercentDiff :: Double -> Double +multiplesToPercentDiff x = (if x > 0 then x - 1 else x + 1) * 100 + +colorCode :: Word -> Double -> Doc -> Doc +colorCode thresh x = + if x > fromIntegral thresh + then dullred + else if x < (-1) * fromIntegral thresh + then dullgreen + else id + -- XXX in comparative reports render lower than baseline in green and higher -- than baseline in red genGroupReport :: RawReport -> Config -> IO () @@ -35,7 +46,10 @@ if length reportColumns > 1 then diffString presentation diffStrategy else Nothing - putStrLn $ makeTitle reportIdentifier diffStr cfg + case mkTitle of + Just _ -> putStrLn $ maybe "" (\f -> f reportIdentifier) mkTitle + Nothing -> putStrLn $ makeTitle reportIdentifier diffStr cfg + let benchcol = "Benchmark" : reportRowIds groupcols = let firstCol : tailCols = reportColumns @@ -43,12 +57,10 @@ let f x = case presentation of Groups Diff -> if x > 0 then dullred else dullgreen - Groups PercentDiff -> - if x > fromIntegral threshold - then dullred - else if x < (-1) * fromIntegral threshold - then dullgreen - else id + Groups PercentDiff -> colorCode threshold x + Groups Multiples -> + let y = multiplesToPercentDiff x + in colorCode threshold y _ -> id in map f colValues renderTailCols estimators col analyzed = @@ -59,6 +71,7 @@ in case presentation of Groups Diff -> colored Groups PercentDiff -> colored + Groups Multiples -> colored _ -> regular in renderGroupCol (showFirstCol firstCol) : case reportEstimators of @@ -133,12 +146,17 @@ in case presentation of Groups Diff -> showDiff Groups PercentDiff -> showDiff + Groups Multiples -> + if val > 0 + then printf "%.2f" val + else printf "1/%.2f" (negate val) _ -> printf "%.2f" val showEstAnnot est = case presentation of Groups Diff -> showEstimator est Groups PercentDiff -> showEstimator est + Groups Multiples -> showEstimator est _ -> "" in case estimators of @@ -186,7 +204,7 @@ report inputFile outputFile cfg@Config{..} = do let dir = fromMaybe "." outputDir (csvlines, fields) <- prepareToReport inputFile cfg - (runs, matrices) <- prepareGroupMatrices cfg csvlines fields + (runs, matrices) <- prepareGroupMatrices cfg inputFile csvlines fields case presentation of Groups style -> forM_ fields $ diff --git a/lib/BenchShow/Tutorial.hs b/lib/BenchShow/Tutorial.hs --- a/lib/BenchShow/Tutorial.hs +++ b/lib/BenchShow/Tutorial.hs @@ -8,7 +8,13 @@ -- -- BenchShow generates text reports and graphs from benchmarking results. It -- allows you to manipulate the format of the report and the benchmarking data --- to present it in many useful ways. +-- to present it in many useful ways. BenchShow uses robust statistical +-- analysis using three different statistical estimators to provide as stable +-- run-to-run comparison of benchmark results as possible. For stable results, +-- make sure that you are not executing any other tasks on the benchmark host +-- while benhmarking is going on. For even more stable results, we recommend +-- using a desktop or server machine instead of a laptop notebook for +-- benchmarking. module BenchShow.Tutorial ( @@ -271,7 +277,7 @@ -- reverse -- $ map fst -- $ sortBy (comparing snd) --- $ either error id $ f $ 'ColumnIndex' 1 +-- $ either error id $ f ('ColumnIndex' 1) Nothing -- } -- @ -- @@ -322,7 +328,7 @@ -- reverse -- $ map fst -- $ sortBy (comparing snd) --- $ either error id $ f $ 'ColumnIndex' 1 +-- $ either error id $ f ('ColumnIndex' 1) Nothing -- } -- @ -- diff --git a/stack.yaml b/stack.yaml --- a/stack.yaml +++ b/stack.yaml @@ -1,10 +1,12 @@ -resolver: lts-12.11 +resolver: lts-13.25 packages: - '.' extra-deps: - - Chart-1.9 - - Chart-diagrams-1.9 - - SVGFonts-1.6.0.3 - - statistics-0.15.0.0 - - math-functions-0.3.0.2 - - dense-linear-algebra-0.1.0.0 + - Chart-1.9.1 + - Chart-diagrams-1.9.2 + + #- SVGFonts-1.6.0.3 + #- statistics-0.15.0.0 + #- math-functions-0.3.0.2 + #- dense-linear-algebra-0.1.0.0 + #- optparse-applicative-0.14.2.0 diff --git a/test/Doc.hs b/test/Doc.hs --- a/test/Doc.hs +++ b/test/Doc.hs @@ -26,17 +26,6 @@ defaultConfig { classifyBenchmark = classifier } - graph "test/results-doc.csv" "docs/grouped-percent" - defaultConfig - { classifyBenchmark = classifier - , presentation = Groups Percent - } - report "test/results-doc.csv" Nothing - defaultConfig - { classifyBenchmark = classifier - , presentation = Groups Percent - } - graph "test/results-doc.csv" "docs/grouped-delta" defaultConfig { classifyBenchmark = classifier @@ -97,7 +86,7 @@ reverse $ map fst $ sortBy (comparing snd) - $ either error id $ f $ ColumnIndex 1 + $ either error id $ f (ColumnIndex 1) Nothing } report "test/results-doc.csv" Nothing defaultConfig @@ -108,7 +97,7 @@ reverse $ map fst $ sortBy (comparing snd) - $ either error id $ f $ ColumnIndex 1 + $ either error id $ f (ColumnIndex 1) Nothing } graph @@ -122,7 +111,7 @@ reverse $ map fst $ sortBy (comparing snd) - $ either error id $ f $ ColumnIndex 1 + $ either error id $ f (ColumnIndex 1) Nothing } report "test/results-doc-multi.csv" @@ -135,5 +124,5 @@ reverse $ map fst $ sortBy (comparing snd) - $ either error id $ f $ ColumnIndex 1 + $ either error id $ f (ColumnIndex 1) Nothing } diff --git a/test/Main.hs b/test/Main.hs --- a/test/Main.hs +++ b/test/Main.hs @@ -73,7 +73,7 @@ let i = intersect (map (last . splitOn "/") prefixes) bs in i ++ (bs \\ i) cfg = defaultConfig - { title = Just chartTitle + { mkTitle = Just (\_ -> chartTitle) , outputDir = Just "charts" , classifyBenchmark = \bm -> case any (`isPrefixOf` bm) prefixes of @@ -82,7 +82,7 @@ in Just (suffixVersion (xs !! 0), xs !! 1) False -> Nothing , selectBenchmarks = \g -> bsort $ - either error (map fst) $ g (ColumnIndex 0) + either error (map fst) $ g (ColumnIndex 0) Nothing , selectGroups = \gs -> let gs' = map fst gs i = intersect (map suffixVersion packages) gs' @@ -112,10 +112,6 @@ , presentation = Groups Diff , selectFields = (`intersect` ["time"]) } - graph "test/results.csvraw" "csvraw-percent" - cfg { presentation = Groups Percent - , selectFields = (`intersect` ["time"]) - } graph "test/results.csvraw" "csvraw-percent-delta" cfg { presentation = Groups PercentDiff , selectFields = (`intersect` ["time"]) @@ -144,18 +140,14 @@ cfg { presentation = Groups Absolute , selectBenchmarks = \g -> either error (map fst . sortBy (compare `on` snd)) - (g (ColumnIndex 1)) - } - - report "test/results.csv" Nothing - cfg { presentation = Groups Percent + (g (ColumnIndex 1) Nothing) } report "test/results.csv" Nothing cfg { presentation = Groups PercentDiff , selectBenchmarks = \g -> either error (map fst . sortBy (compare `on` snd)) - (g (ColumnIndex 1)) + (g (ColumnIndex 1) Nothing) } report "test/results.csv" Nothing cfg { presentation = Groups Diff