# bench-show
Generate text reports and graphical charts from the benchmark results
generated by `gauge` or `criterion`, showing or comparing benchmarks in
many useful ways. In a few lines of code, we can report time taken, peak
memory usage, allocations, among many other fields; we can group benchmarks
and compare the groups; we can compare benchmarks before and after a change;
we can show absolute or percentage difference from the baseline; we can sort
the results to get the worst affected benchmarks by percentage change.
It can help us in answering questions like the following, visually or
textually:
* Across two benchmark runs, show all the operations that resulted in a
regression of more than 10%, so that we can quickly identify and fix
performance problems in our application.
* Across two (or more) packages providing similar functionality, show all the
operations where the performance differs by more than 10%, so that we can
critically analyze the packages and choose the right one.
## Quick Start
Use `gauge` or `criterion` to generate a `results.csv` file, and then use the
following code to generate a textual report or a graph:
```
report "results.csv" Nothing defaultConfig
graph "results.csv" "output" defaultConfig
```
For advanced usage, control the generated report by modifying the
`defaultConfig`.
## Reports and Charts
`report` with `Fields` presentation style generates a multi-column report. We
can select many fields from a `gauge` raw report. Units of the fields are
automatically determined based on the range of values:
```haskell
report "results.csv" Nothing defaultConfig { presentation = Fields }
```
```
Benchmark time(μs) maxrss(MiB)
------------- -------- -----------
vector/fold 641.62 2.75
streamly/fold 639.96 2.75
vector/map 638.89 2.72
streamly/map 653.36 2.66
vector/zip 651.42 2.58
streamly/zip 644.33 2.59
```
`graph` generates one bar chart per field:
```
graph "results.csv" "output" defaultConfig
```
When the input file contains results from a single benchmark run, by default
all the benchmarks are placed in a single benchmark group named "default".
[](https://github.com/composewell/bench-show/blob/master/docs/full-median-time.svg)
## Grouping
Let's write a benchmark classifier to put the `streamly` and `vector`
benchmarks in their own groups:
```haskell
classifier name =
case splitOn "/" name of
grp : bench -> Just (grp, concat bench)
_ -> Nothing
```
Now we can show the two benchmark groups as separate columns. We can
generate reports comparing different benchmark fields (e.g. `time` and
`maxrss`) for all the groups:
```haskell
report "results.csv" Nothing
defaultConfig { classifyBenchmark = classifier }
```
```
(time)(Median)
Benchmark streamly(μs) vector(μs)
--------- ------------ ----------
fold 639.96 641.62
map 653.36 638.89
zip 644.33 651.42
```
We can do the same graphically as well, just replace `report` with `graph`
in the code above. Each group is placed as a cluster on the graph. Multiple
clusters are placed side by side (i.e. on the same scale) for easy
comparison. For example:
[](https://github.com/composewell/bench-show/blob/master/docs/grouped-median-time.svg)
## Regression, Percentage Difference and Sorting
We can append benchmarks results from multiple runs to the same file. These
runs can then be compared. We can run benchmarks before and after a change
and then report the regressions by percentage change in a sorted order:
Given a results file with two runs, this code generates the report that
follows:
```haskell
report "results.csv" Nothing
defaultConfig
{ classifyBenchmark = classifier
, presentation = Groups PercentDiff
, selectBenchmarks = \f ->
reverse
$ map fst
$ sortBy (comparing snd)
$ either error id $ f $ ColumnIndex 1
}
```
```
(time)(Median)(Diff using min estimator)
Benchmark streamly(0)(μs)(base) streamly(1)(%)(-base)
--------- --------------------- ---------------------
zip 644.33 +23.28
map 653.36 +7.65
fold 639.96 -15.63
```
It tells us that in the second run the worst affected benchmark is zip
taking 23.28 percent more time compared to the baseline.
Graphically:
[](https://github.com/composewell/bench-show/blob/master/docs/regression-percent-descending-median-time.svg)
## Full Documentation and examples
* See the [haddock documentation](http://hackage.haskell.org/package/bench-show) on Hackage
* See the [comprehensive tutorial](http://hackage.haskell.org/package/bench-show) module in the haddock docs
* For examples see the [test directory](https://github.com/composewell/bench-show/tree/master/test) in the package
## Contributions and Feedback
Contributions are welcome! Please see the [TODO.md](TODO.md) file or the
existing [issues](https://github.com/composewell/bench-show/issues) if you want
to pick up something to work on.
Any feedback on improvements or the direction of the package is welcome. You
can always send an email to the
[maintainer](https://github.com/composewell/bench-show/blob/master/bench-show.cabal)
or [raise an issue](https://github.com/composewell/bench-show/issues/new) for
anything you want to suggest or discuss, or send a PR for any change that you
would like to make.