diff --git a/LICENSE b/LICENSE new file mode 100644 --- /dev/null +++ b/LICENSE @@ -0,0 +1,19 @@ +Copyright (c) 2013 Joachim Breitner + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software, and to permit persons to whom the Software is furnished to do +so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md new file mode 100644 --- /dev/null +++ b/README.md @@ -0,0 +1,108 @@ +Gipeda -- the Git Performance Dashboard +======================================= + +What is gipeda? +--------------- + +Gitpeda is a a tool that presents data from your program’s benchmark suite (or +any other source), with nice tables and shiny graphs. + + +It is only a frontend and does not help with or care about collecting the data. +So it is up to you whether you have a polling shell script loop, a post-commit +hook or a elaborate jenkins setup. As long as the performance data ends up in +the `logs/` directory, gipeda is happy. + +Gipeda produces static pages. In fact, the (single) html file and the +accompagning JavaScript code is completely static. Giepda just generates a +large number of json files. This has the advantage of easy deployment: Just put +gipeda in your webspace of copy the files to some static web hosting and you +are done. This putts very little load on your server, is cache friendly and has +no security problems. + +Do you want to see it live? Check out these: + + * [Demo page], visualizing fairly boring stuff about gipedia itself. + * [GHC’s gipeda installation]. + +[Demo page]: http://nomeata.github.io/gipeda/ +[GHC’s gipeda installation]: https://perf.ghc.haskell.org/ + +Setting it up +------------- + + * Clone gipedia somewhere, possibly directly into your webspace. + * Install a Haskell compiler, including the `cablal` tool. + * Install the dependencies: + + cabal install --dependencies-only + + * Compile it: + + cabal install --bindir=. + + * Create a `settings.yaml`. You can look at the example file. + * Clone the repository of your project into `repository/`. A bare clone is + sufficient, e.g. + + git clone --bare git://git.haskell.org/ghc.git repository + + * Download a bunch of JavaScript libraries by runing `./install-jslibs.sh`. + +Gipeda does not work without at least some logs, so lets add them. + +Adding data +----------- + +Gipeda expect simple CSV files for each revision, of the form + + benchmark1;1000 + benchmark2;20.123 + benchmark3;0 + +But likely your benchmark suite does not generate them in this format directly. +Hence, put whatever format you have (text base logs, JUnit reports, whatever) +into the directory `logs`, named `.log`, e.g. +`logs/0279a7d327a3b962ffa93a95d47ea5d9ee31e25c.log`. + +Then create a script `log2csv` that expects the filename of such a log on on +the command line and produces the desired CSV file. + +Running gipeda +-------------- + +With everything in place, you can now run + + ./gipeda + +and it will create a bunch of JSON files in `site/out/`. With `./gipda -j4` +you can parallize it. + +You should do this everytime a new log file appears in `logs/`. You should also +make sure your repository is up-to-date, e.g. by running `git -C repository +pull` or, if it is a bare clone, `git -C repository fetch origin +"+refs/heads/*:refs/heads/*" --prune`. + +Using gipedia +------------- + +Finally, you simply point your browser to the `site/index.html`. The page +should be mostly self-explanatory. If you don’t see anything, it might be +because of the filter in the top-right corner. Try to enable all buttons, even +the `=`. + +To host this on a webserver, just put the `site/` directory in your webspace. + +Bugs, Code, Contact +------------------- + +Please reports bugs and missing features at the [GitHub bugtracker]. This is +also where you can find the [source code]. + +Gipeda was written by [Joachim Breitner] and is licensed under a permissive MIT +[license]. + +[GitHub bugtracker]: https://github.com/nomeata/gipeda/issues +[source code]: https://github.com/nomeata/gipeda +[Joachim Breitner]: http://www.joachim-breitner.de/ +[license]: https://github.com/nomeata/gipeda/blob/LICENSE diff --git a/Setup.hs b/Setup.hs new file mode 100644 --- /dev/null +++ b/Setup.hs @@ -0,0 +1,2 @@ +import Distribution.Simple +main = defaultMain diff --git a/gipeda.cabal b/gipeda.cabal new file mode 100644 --- /dev/null +++ b/gipeda.cabal @@ -0,0 +1,81 @@ +name: gipeda +version: 0.1 +category: Development +synopsis: Git Performance Dashboard +description: + Gitpeda is a a tool that presents data from your program’s benchmark suite + (or any other source), with nice tables and shiny graphs. + . + So it is up to you whether you have a polling shell script loop, a post-commit + hook or a elaborate jenkins setup. As long as the performance data ends up in + the `logs/` directory, gipeda is happy. + . + Gipeda produces static pages. In fact, the (single) html file and the + accompagning JavaScript code is completely static. Giepda just generates a + large number of json files. This has the advantage of easy deployment: Just put + gipeda in your webspace of copy the files to some static web hosting and you + are done. This putts very little load on your server, is cache friendly and has + no security problems. + . + Do you want to see it live? Check out these: + . + * Demo page, visualizing fairly boring stuff about gipedia itself: + http://nomeata.github.io/gipeda/ + * GHC’s gipeda installation: + https://perf.ghc.haskell.org/ +homepage: https://github.com/nomeata/gipeda +license: MIT +license-file: LICENSE +author: Joachim Breitner +maintainer: mail@joachim-breitner.de +build-type: Simple +extra-source-files: + README.md, + site/index.html, + site/js/gipeda.js, + install-jslibs.sh +cabal-version: >=1.10 + +executable gipeda + main-is: + gipeda.hs + + other-modules: + BenchmarkSettings, + BenchmarksInCSV, + BenchNames, + GraphReport, + IndexReport, + JsonSettings, + JsonUtils, + ParentMap, + Paths, + ReadResult, + ReportTypes, + RevReport, + Shake, + Summary, + WithLatestLogs + + build-depends: + base >= 4.6 && <4.9, + bytestring >= 0.10 && <0.11, + containers >= 0.4 && <0.6, + directory >= 1.2 && <1.3, + filepath >= 1.3 && <1.4, + shake >= 0.13 && <0.15, + text >= 0.11 && <1.3, + unordered-containers >= 0.2 && <0.3, + split >= 0.2 && <0.3, + vector >= 0.10 && <0.11, + cassava >= 0.4 && <0.5, + yaml >= 0.8 && <0.9, + aeson >= 0.7 && <0.9 + + hs-source-dirs: src + + default-language: Haskell2010 + +source-repository head + type: git + location: https://github.com/nomeat/gipeda diff --git a/install-jslibs.sh b/install-jslibs.sh new file mode 100644 --- /dev/null +++ b/install-jslibs.sh @@ -0,0 +1,29 @@ +#!/bin/bash + +set -e +set -x + +cd site/js +test -e signals.min.js || + wget -c https://raw.githubusercontent.com/millermedeiros/js-signals/master/dist/signals.min.js +test -e hasher.min.js || + wget -c https://raw.githubusercontent.com/millermedeiros/Hasher/master/dist/js/hasher.min.js +test -e handlebars-v2.0.0.js || + wget -c http://builds.handlebarsjs.com.s3.amazonaws.com/handlebars-v2.0.0.js +test -e jquery-1.11.2.min.js || + wget -c http://code.jquery.com/jquery-1.11.2.min.js +test -e jquery.timeago.js || + wget -c http://timeago.yarp.com/jquery.timeago.js +test -d flot || { + wget -c http://www.flotcharts.org/downloads/flot-0.8.3.zip + unzip flot-0.8.3.zip + rm -f flot-0.8.3.zip +} +test -d bootstrap || { + mkdir bootstrap + cd bootstrap + wget -c https://github.com/twbs/bootstrap/releases/download/v3.3.1/bootstrap-3.3.1-dist.zip + unzip bootstrap-3.3.1-dist.zip + rm -f bootstrap-3.3.1-dist.zip + cd .. + } diff --git a/site/index.html b/site/index.html new file mode 100644 --- /dev/null +++ b/site/index.html @@ -0,0 +1,375 @@ + + + + + + + +Gipeda + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+

+Loading... +

+

(If this does not disappear, then you do not have JavaScript enabled, or something else is broken.)

+
+
+ diff --git a/site/js/gipeda.js b/site/js/gipeda.js new file mode 100644 --- /dev/null +++ b/site/js/gipeda.js @@ -0,0 +1,485 @@ +// Main document +var data = {}; +// Name of the current view +var view = 'index'; +// Options of the current view +var viewData = {}; + +// Nonpersistent settings +var settings = { + benchFilter: { + improvements: true, + boring: false, + regressions: true, + }, + collapsedGroups: [true, false, false, false] +}; + +// Signals + +viewChanged = new signals.Signal() +dataChanged = new signals.Signal() + + +// Routes + +var routes = { + index: + { regex: /^$/, + download: ['out/latest-summaries.json'], + url: function () {return ""}, + }, + complete: + { regex: /^all$/, + download: ['out/all-summaries.json'], + url: function () {return "all"}, + }, + graphIndex: + { regex: /^graphs$/, + download: ['out/benchNames.json'], + url: function () {return "graphs"}, + }, + revision: + { regex: /^revision\/([a-f0-9]+)$/, + viewData: function (match) { return { hash: match[1] }; }, + download: function () { + return ['out/benchNames.json','out/reports/' + viewData.hash + '.json']; + }, + url: function (hash) { return "revision/" + hash; }, + }, + graph: + { regex: /^graph\/(.*)$/, + viewData: function (match, options) { + var highlights = []; + options.forEach(function (opt) { + var m; + if (m = /hl=(.*)/.exec(opt)) { + highlights.push(m[1]); + } + }); + return { + benchName: match[1], + highlights: highlights, + }; + }, + download: function () { + return ['out/latest-summaries.json','out/graphs/' + viewData.benchName + '.json']; + }, + url: function (benchName, hls) { + var comps = [ "graph/" + benchName ]; + $.merge(comps, hls.map(function (hl) { return "hl=" + hl })); + return comps.join(';'); + }, + }, +}; + +function parseRoute (path) { + var options = path.split(";"); + var routePath = options.shift(); + + $.each(routes, function (v,r) { + var match = r.regex.exec(routePath); + if (match) { + view = v; + if (r.viewData) viewData = r.viewData(match, options); + return false; + } else { + return true; + } + }); + viewChanged.dispatch(); +} + +viewChanged.add(function () { + if (routes[view].download) { + d = routes[view].download; + if ($.isFunction(d)) {d = d()} + d.forEach(function (url) { getJSON(url) }); + } +}); + +function handleHashChange(newHash) { + parseRoute(newHash); +} + +// Data + +function commitsFrom(revs, hash, count) { + // Can happen during loading + if (!revs) {return []}; + + if (count == 0) return []; + + var rev = revs[hash]; + if (rev) { + if (rev.summary.parents.length > 0) { + var later = commitsFrom(revs, rev.summary.parents[0], count - 1); + later.unshift(rev); + return later; + } else { + return [ rev ]; + } + } else { + return [] + } +} + + +// Template handling +var templates = {}; + +$(function () { + var template_ids = ["revision", "index", "complete", "graphIndex", "graph", "revTooltip"]; + template_ids.forEach(function(id) { + var source = $("#" + id).html(); + templates[id] = Handlebars.compile(source); + }); + + var partials_ids = ["nav", "summary-icons", "summary-list", "nothing"]; + partials_ids.forEach(function(id) { + var source = $("#" + id).html(); + Handlebars.registerPartial(id, source); + }); + +}); + +Handlebars.registerHelper('revisionLink', function(hash) { + if (!hash) { return "#"; } + return "#" + routes.revision.url(hash); +}); +Handlebars.registerHelper('graphLink', function(benchName, hl) { + if (hl) { + return "#" + routes.graph.url(benchName,[hl]); + } else { + return "#" + routes.graph.url(benchName); + } +}); +Handlebars.registerHelper('diffLink', function(rev1, rev2) { + return data.settings.cgitLink + "/commitdiff/" + rev2 +}); +Handlebars.registerHelper('logLink', function(rev) { + return Handlebars.compile(data.settings.logLink)({rev:rev}); +}); +Handlebars.registerHelper('indexLink', function() { + return "#" + routes.index.url(); +}); +Handlebars.registerHelper('allLink', function() { + return "#" + routes.complete.url(); +}); +Handlebars.registerHelper('graphIndexLink', function() { + return "#" + routes.graphIndex.url(); +}); +Handlebars.registerHelper('recentCommits', function(revisions) { + return commitsFrom(revisions, data.latest, data.settings.limitRecent); +}); +Handlebars.registerHelper('allCommits', function(revisions) { + return commitsFrom(revisions, data.latest, -1); +}); +function shortRev(rev) { + if (!rev) { return ''; } + return rev.substr(0,7); +} +Handlebars.registerHelper('shortRev', shortRev); +Handlebars.registerHelper('iso8601', function(timestamp) { + if (!timestamp) { return '' }; + return new Date(timestamp*1000).toISOString(); +}); +Handlebars.registerHelper('humanDate', function(timestamp) { + return new Date(timestamp*1000).toString(); +}); + +// We cache everything +var jsonSeen = {}; +function getJSON(url, callback, options) { + var opts = { + block: true, + }; + $.extend(opts, options); + if (jsonSeen[url]) { + console.log("Not fetching "+url+" again."); + if (callback) callback(); + } else { + console.log("Fetching "+url+"."); + $.ajax(url, { + success: function (newdata) { + console.log("Fetched "+url+"."); + jsonSeen[url] = true; + $.extend(true, data, newdata); + dataChanged.dispatch(); + if (callback) callback(); + }, + dataType: 'json' + }); + } +} + + +// Views + +function groupStats (benchResults) { + return { + totalCount: benchResults.length, + improvementCount: benchResults.filter(function (br) { return br.changeType == 'Improvement' }).length, + regressionCount: benchResults.filter(function (br) { return br.changeType == 'Regression' }).length, + } +} + +// Some views require the data to be prepared in ways that are +// too complicated for the template, so lets do it here. +dataViewPrepare = { + 'revision': function (data, viewData) { + if (!data.benchGroups || !data.revisions) return {}; + var hash = viewData.hash; + var rev = data.revisions[hash]; + if (!rev) return {}; + if (!rev.benchResults) return {}; + + var groups = data.benchGroups.map(function (group) { + var benchmarks = group.groupMembers.map(function (bn) { + return rev.benchResults[bn] + }).filter(function (br) {return br}); + return { + groupName: group.groupName, + benchResults: benchmarks, + groupStats: groupStats(benchmarks), + }; + }); + return { + rev : rev, + groups : groups, + }; + }, +} + +function load_template () { + console.log('Rebuilding page'); + var context = {}; + $.extend(context, data, viewData, settings); + if (dataViewPrepare[view]){ + $.extend(context, dataViewPrepare[view](data, viewData)); + } + $('#main').html(templates[view](context)); + + $(".nav-loading").toggle(jQuery.active > 0); + + updateBenchFilter(); + updateCollapsedGroups(); + $('abbrv.timeago').timeago(); + + if ($('#benchChart').length) { + setupChart(); + } +} +viewChanged.add(load_template); +dataChanged.add(load_template); + +function setupChart () { + + var commits = commitsFrom(data.revisions, data.latest, data.settings.limitRecent); + var benchName = viewData.benchName; + + $("
").css({ + position: "absolute", + display: "none", + //border: "1px solid #fdd", + padding: "2px", + //"background-color": "#fee", + // opacity: 0.80, + width: '300px', + }).appendTo("#main"); + + var plot = $.plot("#benchChart", + [{ + lines: { show: true, fill: true, fillColor: "rgba(255, 255, 255, 0.8)" }, + points: { show: true, fill: false }, + label: benchName, + data: commits.map(function (x,i){ + if (!x.benchResults) return; + if (!x.benchResults[benchName]) return; + return [commits.length - i, x.benchResults[benchName].value] + }), + }], + { + legend: { + position: 'nw', + }, + grid: { + hoverable: true, + clickable: true, + }, + xaxis: { + // ticks: values.map(function (x,i){return [i,x[0]]}), + tickFormatter: function (i,axis) { + if (i > 0 && i <= commits.length) { + var rev = commits[commits.length - i]; + if (!rev) return ""; + var date = new Date(rev.summary.gitDate * 1000); + return $.timeago(date); + } else { + return ''; + } + } + } + }); + + viewData.highlights.forEach(function (hash) { + commits.forEach(function (rev,i) { + if (rev.summary.hash == hash) { + plot.highlight(0, i); + }; + }); + }); + + + $("#benchChart").bind("plothover", function (event, pos, item) { + if (item) { + var v = item.datapoint[1]; + var i = item.dataIndex; + var rev = commits[i].summary.hash; + var summary = commits[i].summary; + + var tooltipContext = $.extend({value: v}, summary); + + if ($("#tooltip").data('rev') != rev) { + $("#tooltip") + .html(shortRev(rev)) + .data('rev',rev) + .html(templates.revTooltip(tooltipContext)) + .fadeIn(200) + .css({top: item.pageY+10, left: item.pageX-100}) + .show(); + $("#benchChart").css('cursor','pointer'); + } + } else { + $("#tooltip").data('rev',null).hide(); + $("#benchChart").css('cursor','default'); + } + }); + $("#benchChart").bind("plotclick", function (event, pos, item) { + if (item) { + var v = item.datapoint[1]; + var i = item.dataIndex; + var hash = commits[i].summary.hash; + + goTo(routes.revision.url(hash)); + } + }); +} + +function updateCollapsedGroups() { + // Does not work yet... + /* + + var list = settings.collapsedGroups; + console.log(list); + if ($(".panel-collapse").length) { + $(".panel-collapse").each(function (i){ + if (i < list.length) { + console.log(i, list[i], this); + $(this).toggleClass('in', !list[i]); + + $(this).collapse(); + if (list[i]) { + $(this).collapse('hide'); + } else { + $(this).collapse('show'); + } + } + }); + } + */ +} + +function updateBenchFilter() { + var showRegressions = settings.benchFilter.regressions; + var showBoring = settings.benchFilter.boring; + var showImprovements = settings.benchFilter.improvements; + + $('#show-regressions').toggleClass('active', showRegressions); + $('#show-boring').toggleClass('active', showBoring); + $('#show-improvements').toggleClass('active', showImprovements); + + $('tr.row-Regression').toggle(showRegressions); + $('tr.row-Boring').toggle(showBoring); + $('tr.row-Improvement').toggle(showImprovements); + + $('.bench-panel').show().each(function() { + if ($(this).has('tr.row-result:visible').length == 0) { + $(this).hide(); + } + }); + + $('tr.summary-row').addClass('summary-row-collapsed'); + if (showBoring) { + $('tr.summary-row:not(.summary-improvement):not(.summary-regression)') + .removeClass('summary-row-collapsed'); + } + if (showRegressions) { + $('tr.summary-row.summary-regression') + .removeClass('summary-row-collapsed'); + } + if (showImprovements) { + $('tr.summary-row.summary-improvement') + .removeClass('summary-row-collapsed'); + } + $('tr.summary-row').first() + .removeClass('summary-row-collapsed'); + + updateNothingToSee(); +}; + +function updateNothingToSee() { + $('.nothing-to-see') + .toggle(jQuery.active == 0 && $('.bench-panel:visible, tr.summary-row:not(.summary-row-collapsed)').length == 0); +} + +$(function (){ + $('body').on('click', '.benchSelector', function (event) { + $(this).toggleClass('active'); + settings.benchFilter = { + regressions: $('#show-regressions').hasClass('active'), + boring: $('#show-boring').hasClass('active'), + improvements: $('#show-improvements').hasClass('active'), + }; + updateBenchFilter(); + }); +}); + +// Redirection + +function goTo(path) { + console.log("goTo " + path); + hasher.setHash(path); +} + + +// Main setup + +$(function() { + hasher.prependHash = ''; + + $('#loading').hide(); + + $(document).ajaxStart(function () { + $(".nav-loading").show(); + updateNothingToSee(); + }); + $(document).ajaxStop(function () { + $(".nav-loading").hide(); + updateNothingToSee(); + }); + + // Load settins, then figure out what view to use. + $.get("out/latest.txt", function (latest) { + data.latest = latest; + + getJSON("out/settings.json", function (settings) { + $('title').html(data.settings.title + " – Gipeda"); + hasher.changed.add(handleHashChange); + hasher.initialized.add(handleHashChange); + hasher.init(); + }); + }, 'text'); + + +}); diff --git a/src/BenchNames.hs b/src/BenchNames.hs new file mode 100644 --- /dev/null +++ b/src/BenchNames.hs @@ -0,0 +1,33 @@ +{-# LANGUAGE OverloadedStrings #-} +module BenchNames where + +import qualified Data.ByteString.Lazy as BS +import Data.Aeson +import Control.Monad +import qualified Data.Map as M +import qualified Data.Text as T +import Data.List + +import Paths +import JsonUtils +import ReportTypes +import qualified BenchmarkSettings as S + + +benchNamesMain :: [S.BenchName] -> IO () +benchNamesMain benchNames = do + settings <- S.readSettings "settings.yaml" + + let groups = + map (uncurry BenchGroup) $ + sort $ + M.toList $ + M.map sort $ + M.fromListWith (++) + [ (S.group s, [bn]) + | bn <- benchNames + , let s = S.benchSettings settings bn + ] + let doc = emptyGlobalReport { benchGroups = Just groups } + + BS.putStr (encode doc) diff --git a/src/BenchmarkSettings.hs b/src/BenchmarkSettings.hs new file mode 100644 --- /dev/null +++ b/src/BenchmarkSettings.hs @@ -0,0 +1,96 @@ +{-# LANGUAGE OverloadedStrings #-} +module BenchmarkSettings where + +import Data.Yaml +import Data.Aeson +import qualified Data.Vector as V +import Data.Functor +import Control.Applicative +import Control.Monad +import Data.Monoid +import Data.Maybe + + +data NumberType = IntegralNT | SmallIntegralNT | FloatingNT + deriving Show + +type BenchName = String +data BenchSettings = BenchSettings + { smallerIsBetter :: Bool + , unit :: String + , numberType :: NumberType + , group :: String + , threshold :: Double + } + deriving Show + +defaultBenchSettings :: BenchSettings +defaultBenchSettings = BenchSettings True "" IntegralNT "" 3 + +newtype S = S { unS :: BenchName -> BenchSettings } +newtype SM = SM (BenchName -> (BenchSettings -> BenchSettings)) + +instance Monoid SM where + mempty = SM (const id) + mappend (SM f) (SM g) = SM (\n -> g n . f n) + +instance FromJSON NumberType where + parseJSON = withText "type" $ \t -> case t of + "small integral" -> return SmallIntegralNT + "integral" -> return IntegralNT + "float" -> return FloatingNT + v -> fail $ "Unknown \"type\": " ++ show v + +-- Very naive glob, * only at the end +matches :: String -> String -> Bool +matches [] [] = True +matches _ ('*':[]) = True +matches (x:xs) (m:ms) = x == m && matches xs ms + +instance FromJSON SM where + parseJSON (Object o) = do + m <- o .: "match" + mt <- o .:? "type" + mu <- o .:? "unit" + mg <- o .:? "group" + ms <- o .:? "smallerIsBetter" + mth <- o .:? "threshold" + return $ SM $ \n b -> + if n `matches` m then + b { numberType = fromMaybe (numberType b) mt + , unit = fromMaybe (unit b) mu + , group = fromMaybe (group b) mg + , smallerIsBetter = fromMaybe (smallerIsBetter b) ms + , threshold = fromMaybe (threshold b) mth + } + else b + parseJSON _ = mzero + +instance FromJSON S where + parseJSON (Array a) = do + mods <- mapM parseJSON (V.toList a) + let SM sm = mconcat mods + return $ S $ \n -> sm n defaultBenchSettings + +data Settings = Settings + { title :: String + , cgitLink :: String + , logLink :: String + , limitRecent :: Integer + , start :: String + , benchSettings :: BenchName -> BenchSettings + } + +instance FromJSON Settings where + parseJSON (Object v) = + Settings <$> v .: "title" + <*> v .: "cgitLink" + <*> v .: "logLink" + <*> v .: "limitRecent" + <*> v .: "start" + <*> (unS <$> v.: "benchmarks") + parseJSON _ = mzero + + +readSettings :: FilePath -> IO Settings +readSettings fname = either (error.show) id <$> decodeFileEither fname diff --git a/src/BenchmarksInCSV.hs b/src/BenchmarksInCSV.hs new file mode 100644 --- /dev/null +++ b/src/BenchmarksInCSV.hs @@ -0,0 +1,24 @@ +module BenchmarksInCSV where + +import Data.Csv hiding (encode) +import qualified Data.ByteString.Lazy as BS +import qualified Data.Vector as V +import Data.Char +import Data.Functor + +type Benchmark = String + +benchmarksInCSVFile :: FilePath -> IO [Benchmark] +benchmarksInCSVFile fname = benchmarksInCSV <$> BS.readFile fname + +benchmarksInCSV :: BS.ByteString -> [Benchmark] +benchmarksInCSV s = + [ n + | (n,v) <- either error V.toList $ decodeWith ssv NoHeader s + , const True (v::String) + ] + +ssv :: DecodeOptions +ssv = defaultDecodeOptions { + decDelimiter = fromIntegral (ord ';') +} diff --git a/src/GraphReport.hs b/src/GraphReport.hs new file mode 100644 --- /dev/null +++ b/src/GraphReport.hs @@ -0,0 +1,29 @@ +{-# LANGUAGE DeriveGeneric, OverloadedStrings #-} +module GraphReport where + +import Control.Monad +import qualified Data.Map as M +import qualified Data.ByteString.Lazy as BS +import qualified Data.Text as T +import Data.Aeson +import GHC.Generics + +import Paths +import ReadResult +import ReportTypes + +graphReportMain :: [String] -> IO () +graphReportMain (bench:revs) = do + g <- forM revs $ \rev -> do + m <- readCSV rev + let v = M.lookup bench m + return $ T.pack rev .= object + [ "benchResults" .= object + [ T.pack bench .= object + [ "value" .= v ] + ] + ] + let doc = object ["revisions" .= object g] + + BS.putStr (encode doc) + diff --git a/src/IndexReport.hs b/src/IndexReport.hs new file mode 100644 --- /dev/null +++ b/src/IndexReport.hs @@ -0,0 +1,19 @@ +module IndexReport where + +import qualified Data.ByteString.Lazy as BS +import Data.Aeson +import Control.Monad + +import Paths +import JsonUtils + + +indexReportsMain :: [FilePath] -> IO () +indexReportsMain revs = do + g <- forM revs $ \rev -> do + json <- BS.readFile (summaryOf rev) + case eitherDecode json of + Left e -> fail e + Right rep -> return (rep :: Value) + + BS.putStr (encode (merges g)) diff --git a/src/JsonSettings.hs b/src/JsonSettings.hs new file mode 100644 --- /dev/null +++ b/src/JsonSettings.hs @@ -0,0 +1,26 @@ +{-# LANGUAGE OverloadedStrings, RecordWildCards #-} + +module JsonSettings where + +import Data.Aeson +import qualified Data.ByteString.Lazy as BS + +import BenchmarkSettings as S + +jsonSettingsMain :: IO () +jsonSettingsMain = do + Settings {..} <- S.readSettings "settings.yaml" + let o = object + [ "settings" .= object + [ "title" .= title + , "cgitLink" .= cgitLink + , "logLink" .= logLink + , "limitRecent" .= limitRecent + ] + ] + BS.putStr (encode o) + + + + + diff --git a/src/JsonUtils.hs b/src/JsonUtils.hs new file mode 100644 --- /dev/null +++ b/src/JsonUtils.hs @@ -0,0 +1,29 @@ +{-# LANGUAGE OverloadedStrings #-} + +module JsonUtils where + +import qualified Data.Text as T +import qualified Data.HashMap.Strict as HM +import Data.Aeson +import Data.List (foldl') + +mapObject :: (Object -> Object) -> Value -> Value +mapObject f (Object m) = Object (f m) +mapObject _ v = error $ "Not an object " ++ show v + +delete :: [T.Text] -> Value -> Value +delete [] = error "delete []" +delete ["*"] = error "delete [\"*\"]" +delete [k] = mapObject $ HM.delete k +delete ("*":ks) = mapObject $ HM.map (delete ks) +delete (k:ks) = mapObject $ HM.adjust (delete ks) k + +merge :: Value -> Value -> Value +merge (Object m1) (Object m2) = Object $ HM.unionWith merge m2 m1 +merge v1 v2 = error $ "Cannot merge " ++ show v1 ++ " with " ++ show v2 + +merges :: [Value] -> Value +merges = foldl' merge (Object HM.empty) + + + diff --git a/src/ParentMap.hs b/src/ParentMap.hs new file mode 100644 --- /dev/null +++ b/src/ParentMap.hs @@ -0,0 +1,28 @@ +module ParentMap where + +import Data.Csv hiding (encode) +import qualified Data.Map as M +import Data.List.Split +import qualified Data.ByteString.Lazy as BS +import qualified Data.Vector as V +import Data.Char +import Data.Functor + +type Hash = String +type ParentMap = M.Map Hash Hash + +ssvFileToMap :: FilePath -> IO ParentMap +ssvFileToMap fname = ssvToMap <$> BS.readFile fname + +ssvToMap :: BS.ByteString -> M.Map Hash Hash +ssvToMap s = M.fromList + [ (k,p) + | (k,parentList) <- either error V.toList $ decodeWith ssv NoHeader s + , let parents = splitOn " " parentList + , p:_ <- return parents + ] + +ssv :: DecodeOptions +ssv = defaultDecodeOptions { + decDelimiter = fromIntegral (ord ';') +} diff --git a/src/Paths.hs b/src/Paths.hs new file mode 100644 --- /dev/null +++ b/src/Paths.hs @@ -0,0 +1,16 @@ +module Paths where + +import System.FilePath + +type Hash = String + +out :: FilePath +out = "site" "out" + +resultsOf, reportOf, summaryOf, logsOf, graphFile :: Hash -> FilePath + +logsOf hash = "logs" hash <.> "log" +resultsOf hash = out "results" hash <.> "csv" +reportOf hash = out "reports" hash <.> "json" +summaryOf hash = out "summaries" hash <.> "json" +graphFile bench = out "graphs" bench <.> "json" diff --git a/src/ReadResult.hs b/src/ReadResult.hs new file mode 100644 --- /dev/null +++ b/src/ReadResult.hs @@ -0,0 +1,50 @@ +{-# LANGUAGE DeriveGeneric #-} + +module ReadResult where + +import Data.Csv hiding (encode) +import Data.Functor +import Control.Applicative +import qualified Data.Map as M +import qualified Data.ByteString.Lazy as BS +import qualified Data.Vector as V +import System.Directory +import Data.Aeson hiding (decode) +import GHC.Generics +import Data.Char + +import Paths + +type BenchName = String + +data BenchValue = + F Double | + I Integer + deriving (Show, Generic, Eq, Ord) + +type ResultMap = M.Map String BenchValue + +instance ToJSON BenchValue where + toJSON (F d) = toJSON d + toJSON (I i) = toJSON i +instance FromJSON BenchValue where + parseJSON o = (I <$> parseJSON o) <|> (F <$> parseJSON o) + +instance FromField BenchValue where + parseField s = (I <$> parseField s) <|> (F <$> parseField s) + +readCSV :: Hash -> IO ResultMap +readCSV hash = do + ex <- doesFileExist (resultsOf hash) + if not ex + then return M.empty + else do + str <- BS.readFile (resultsOf hash) + let rows = either error id $ decodeWith ssv NoHeader str + return $ M.fromList $ V.toList $ rows + +ssv :: DecodeOptions +ssv = defaultDecodeOptions { + decDelimiter = fromIntegral (ord ';') +} + diff --git a/src/ReportTypes.hs b/src/ReportTypes.hs new file mode 100644 --- /dev/null +++ b/src/ReportTypes.hs @@ -0,0 +1,215 @@ +{-# LANGUAGE DeriveGeneric, ViewPatterns, RecordWildCards, OverloadedStrings #-} + +module ReportTypes where + +import qualified Data.Map as M +import Data.Aeson +import GHC.Generics +import Text.Printf +import Data.List + +import Paths +import ReadResult +import qualified BenchmarkSettings as S + +data ClientSettings = ClientSettings + { title :: String + , cgitLink :: String + , logLink :: String + } + deriving (Generic) +instance ToJSON ClientSettings + +data GlobalReport = GlobalReport + { settings :: Maybe ClientSettings + , benchmarks :: Maybe (M.Map BenchName ()) + , revisions :: Maybe (M.Map Hash RevReport) + , benchGroups :: Maybe [BenchGroup] + } + +instance ToJSON GlobalReport where + toJSON (GlobalReport {..}) = object $ + ( "settings" .=? settings ) ++ + ( "benchmarks" .=? benchmarks ) ++ + ( "revisions" .=? revisions ) ++ + ( "benchGroups" .=? benchGroups ) + where + k .=? Just v = [ k .= toJSON v ] + _ .=? Nothing = [] + + +emptyGlobalReport :: GlobalReport +emptyGlobalReport = GlobalReport Nothing Nothing Nothing Nothing + + +data SummaryStats = SummaryStats + { totalCount :: Int + , improvementCount :: Int + , regressionCount :: Int + } + deriving (Show, Generic) +instance ToJSON SummaryStats +instance FromJSON SummaryStats + +sumStats :: [SummaryStats] -> SummaryStats +sumStats = foldl' go (SummaryStats 0 0 0) + where go (SummaryStats a b c) (SummaryStats a' b' c') = + SummaryStats (a + a') (b + b') (c + c') + +data Summary = Summary + { hash :: Hash + , gitDate :: Integer + , gitSubject :: String + , stats :: SummaryStats + , parents :: [String] + } + deriving (Generic) +instance ToJSON Summary +instance FromJSON Summary + + +data RevReport = RevReport + { summary :: Summary + , gitLog :: String + , benchResults :: M.Map BenchName BenchResult + } + deriving (Generic) +instance ToJSON RevReport +instance FromJSON RevReport + +data ChangeType = Improvement | Boring | Regression + deriving (Generic) +instance ToJSON ChangeType +instance FromJSON ChangeType + +data BenchGroup = BenchGroup + { groupName :: String + , groupMembers :: [BenchName] + } + deriving (Generic) +instance ToJSON BenchGroup +instance FromJSON BenchGroup + +data BenchResult = BenchResult + { name :: String + , value :: BenchValue + , previous :: Maybe BenchValue + , change :: String + , changeType :: ChangeType + , unit :: String + } + deriving (Generic) +instance ToJSON BenchResult +instance FromJSON BenchResult + +invertChangeType :: ChangeType -> ChangeType +invertChangeType Improvement = Regression +invertChangeType Boring = Boring +invertChangeType Regression = Improvement + +type Explanation = (String, ChangeType) + +noExplanation :: Explanation +noExplanation = ("", Boring) + +equalExplanation :: Explanation +equalExplanation = ("=", Boring) + +explainSmallInt :: S.BenchSettings -> Integer -> Integer -> Explanation +explainSmallInt _ i1 i2 + | i2 == i1 = equalExplanation + | i2 > i1 = ("+ " ++ show (i2 - i1), Improvement) + | i2 < i1 = ("- " ++ show (i1 - i2), Regression) + +explainInt :: S.BenchSettings -> Integer -> Integer -> Explanation +explainInt s i1 i2 = explainFloat s (fromIntegral i1) (fromIntegral i2) + +explainFloat :: S.BenchSettings -> Double -> Double -> Explanation +explainFloat _ 0 0 = equalExplanation +explainFloat _ 0 _ = ("+ ∞", Improvement) +explainFloat s f1 f2 = (change, typ) + where + change | abs perc < 0.01 = "=" + | perc >= 0 = printf "+ %.2f%%" perc + | perc < 0 = printf "- %.2f%%" (-perc) + typ | abs perc < th = Boring + | perc >= 0 = Improvement + | perc < 0 = Regression + + perc = 100 * ((f2 - f1) / f1) + th = S.threshold s + +toFloat :: BenchValue -> Double +toFloat (I i) = fromIntegral i +toFloat (F f) = f + +explain :: S.BenchSettings -> BenchValue -> BenchValue -> (String, ChangeType) +explain s@(S.numberType -> S.SmallIntegralNT) (I i1) (I i2) = explainSmallInt s i1 i2 +explain s@(S.numberType -> S.IntegralNT) (I i1) (I i2) = explainInt s i1 i2 +explain s@(S.numberType -> S.FloatingNT) v1 v2 = explainFloat s (toFloat v1) (toFloat v2) +explain _ _ _ = noExplanation + +toResult :: S.BenchSettings -> String -> BenchValue -> Maybe BenchValue -> BenchResult +toResult s name value prev = BenchResult + { name = name + , value = value + , previous = prev + , change = change + , changeType = changeType + , unit = S.unit s + } + where + (change, changeType') = + case prev of + Just p -> explain s p value + Nothing -> noExplanation + changeType | S.smallerIsBetter s = invertChangeType changeType' + | otherwise = changeType' + +toSummaryStats :: [BenchResult] -> SummaryStats +toSummaryStats res = SummaryStats + { totalCount = length res + , improvementCount = length [ () | BenchResult { changeType = Improvement } <- res ] + , regressionCount = length [ () | BenchResult { changeType = Regression } <- res ] + } + + +{- +toGroup :: String -> [BenchResult] -> BenchGroup +toGroup n res = BenchGroup + { groupName = n + , benchResults = res + , groupStats = SummaryStats + { totalCount = length res + , improvementCount = length [ () | BenchResult { changeType = Improvement } <- res ] + , regressionCount = length [ () | BenchResult { changeType = Regression } <- res ] + } + } +-} + +createReport :: + S.Settings -> Hash -> [Hash] -> + ResultMap -> ResultMap -> + String -> String -> Integer -> + RevReport +createReport settings this parents thisM parentM log subject date = RevReport + { summary = Summary + { hash = this + , parents = parents + , stats = toSummaryStats $ M.elems results + , gitSubject = subject + , gitDate = date + } + --, benchGroups = benchGroups + , benchResults = results + , gitLog = log + } + where + results = M.fromList + [ (name, toResult s name value (M.lookup name parentM)) + | (name, value) <- M.toAscList thisM + , let s = S.benchSettings settings name + ] + +summarize :: RevReport -> Summary +summarize (RevReport {..}) = summary diff --git a/src/RevReport.hs b/src/RevReport.hs new file mode 100644 --- /dev/null +++ b/src/RevReport.hs @@ -0,0 +1,37 @@ +{-# LANGUAGE DeriveGeneric #-} + +module RevReport where + +import Data.Functor +import qualified Data.Map as M +import qualified Data.ByteString.Lazy as BS +import Data.Aeson +import Development.Shake.Command + +import ReportTypes +import ReadResult +import qualified BenchmarkSettings as S + +git args = cmd (words "git -C repository" ++ args) + +revReportMain :: [String] -> IO () +revReportMain (this:parents) = do + settings <- S.readSettings "settings.yaml" + + thisM <- readCSV this + parentM <- case parents of + p:_ -> readCSV (head parents) + _ -> return M.empty + + log <- case parents of + p:_ -> fromStdout <$> git ["log", p ++ ".."++ this] + _ -> fromStdout <$> git ["show", "-s", this] + + msg <- fromStdout <$> git ["show", "--format=%s", "-s", this] + date <- read . fromStdout <$> git ["show", "--format=%ct","-s",this] + + let rep = createReport settings this parents thisM parentM log msg date + let doc = emptyGlobalReport { revisions = Just (M.singleton this rep) } + + BS.putStr (encode doc) + diff --git a/src/Shake.hs b/src/Shake.hs new file mode 100644 --- /dev/null +++ b/src/Shake.hs @@ -0,0 +1,208 @@ +{-# LANGUAGE DeriveDataTypeable, GeneralizedNewtypeDeriving #-} +module Shake where + +import Prelude hiding ((*>)) + +import Development.Shake +import Development.Shake.FilePath +import Development.Shake.Classes +import Control.Monad +import qualified Data.Map as M +import Data.Functor +import Data.List + +import Paths hiding (Hash) +import ParentMap +import BenchmarksInCSV +import qualified BenchmarkSettings as S + +{- Global settings -} +cGRAPH_HISTORY :: Integer +cGRAPH_HISTORY = 50 + +git :: (CmdResult b) => String -> [String] -> Action b +git gitcmd args = do + cmd (Traced $ "git " ++ gitcmd) (words "git -C repository" ++ gitcmd : args) + +self :: (CmdResult b) => String -> [String] -> Action b +self name args = do + -- orderOnly ["gipeda"] + cmd (Traced name) "./gipeda" name args + +gitRange :: Action String +gitRange = do + s <- liftIO $ S.readSettings "settings.yaml" + let first = S.start s + [head] <- readFileLines "site/out/head.txt" + return $ first ++ ".." ++ head + +needIfThere :: [FilePath] -> Action [FilePath] +needIfThere files = do + existing <- filterM doesFileExist files + need existing + return existing + +findPred, findPredOrSelf :: ParentMap -> Hash -> Action (Maybe Hash) +findPredOrSelf m h = do + ex <- doesFileExist (logsOf h) + if ex then return (Just h) + else findPred m h +findPred m h = case M.lookup h m of + Just h' -> findPredOrSelf m h' + Nothing -> return Nothing + +findRecent :: ParentMap -> Integer -> FilePath -> Action [FilePath] +findRecent _ 0 _ = return [] +findRecent m n h = do + pM <- findPred m h + (h:) <$> case pM of + Nothing -> return [] + Just p -> findRecent m (n-1) p + +newtype LimitRecent = LimitRecent () + deriving (Show,Typeable,Eq,Hashable,Binary,NFData) + +shakeMain :: IO () +shakeMain = shakeArgs shakeOptions $ do + +{- + "gipeda" *> \out -> do + sources <- getDirectoryFiles "src" ["*.hs"] + need (map ("src" ) sources) + cmd "ghc -isrc --make -O src/gipeda.hs -o" out + want ["gipeda"] +-} + + getLimitRecent <- addOracle $ \(LimitRecent _) -> do + need ["settings.yaml"] + S.limitRecent <$> liftIO (S.readSettings "settings.yaml") + + "reports" ~> do + range <- gitRange + Stdout range <- git "log" ["--format=%H",range] + let hashes = words range + withLogs <- filterM (doesFileExist . logsOf) hashes + need $ map reportOf withLogs + want ["reports"] + + "summaries" ~> do + range <- gitRange + Stdout range <- git "log" ["--format=%H",range] + let hashes = words range + withLogs <- filterM (doesFileExist . logsOf) hashes + need $ map summaryOf withLogs + want ["summaries"] + + "site/out/head.txt" *> \ out -> do + alwaysRerun + Stdout stdout <- git "rev-parse" ["master"] + writeFileChanged out stdout + + + "site/out/history.csv" *> \out -> do + range <- gitRange + Stdout stdout <- git "log" ["--format=%H;%P",range] + writeFileChanged out stdout + want ["site/out/history.csv"] + + history' <- newCache $ \() -> do + orderOnly ["site/out/history.csv"] + liftIO $ ssvFileToMap "site/out/history.csv" + let history = history' () + let pred h = do { hist <- history; findPred hist h } + let predOrSelf h = do { hist <- history; findPredOrSelf hist h } + let recent n h = do { hist <- history; findRecent hist n h } + + "site/out/latest.txt" *> \ out -> do + [head] <- readFileLines "site/out/head.txt" + latestM <- predOrSelf head + case latestM of + Just latest -> + writeFileChanged out latest + Nothing -> + fail "Head has no parent with logs?" + + "graphs" ~> do + [latest] <- readFileLines "site/out/latest.txt" + need [resultsOf latest] + b <- liftIO $ benchmarksInCSVFile (resultsOf latest) + need (map graphFile b) + want ["graphs"] + + "site/out/results/*.csv" *> \out -> do + let hash = takeBaseName out + need [logsOf hash] + Stdout csv <- cmd "./log2csv" (logsOf hash) + writeFile' out csv + + "site/out/graphs//*.json" *> \out -> do + let bench = dropDirectory1 (dropDirectory1 (dropDirectory1 (dropExtension out))) + + [latest] <- readFileLines "site/out/latest.txt" + limitRecent <- getLimitRecent (LimitRecent ()) + r <- recent limitRecent latest + need (map resultsOf r) + + Stdout json <- self "GraphReport" (bench : r) + writeFile' out json + + "site/out/reports/*.json" *> \out -> do + let hash = takeBaseName out + need [resultsOf hash] + + pred <- pred hash + need [resultsOf h | Just h <- return pred] + + Stdout json <- self "RevReport" (hash : [h | Just h <- return pred]) + writeFile' out json + + "site/out/summaries/*.json" *> \out -> do + let hash = takeBaseName out + need [reportOf hash] + + Stdout json <- self "Summary" [hash] + writeFile' out json + + "site/out/latest-summaries.json" *> \out -> do + [latest] <- readFileLines "site/out/latest.txt" + r <- recent cGRAPH_HISTORY latest + need (map summaryOf r) + + Stdout json <- self "IndexReport" r + writeFile' out json + want ["site/out/latest-summaries.json"] + + "site/out/benchNames.json" *> \out -> do + [latest] <- readFileLines "site/out/latest.txt" + need [resultsOf latest] + b <- liftIO $ benchmarksInCSVFile (resultsOf latest) + + need ["settings.yaml"] + + Stdout json <- self "BenchNames" (nub b) + writeFile' out json + want ["site/out/benchNames.json"] + + + + "site/out/all-summaries.json" *> \out -> do + range <- gitRange + Stdout range <- git "log" ["--format=%H",range] + let hashes = words range + withLogs <- filterM (doesFileExist . logsOf) hashes + need (map summaryOf withLogs) + + Stdout json <- self "IndexReport" withLogs + writeFile' out json + want ["site/out/all-summaries.json"] + + "site/out/settings.json" *> \out -> do + need ["settings.yaml"] + + Stdout json <- self "JsonSettings" [] + writeFile' out json + want ["site/out/settings.json"] + + phony "clean" $ do + removeFilesAfter "site/out" ["//*"] + diff --git a/src/Summary.hs b/src/Summary.hs new file mode 100644 --- /dev/null +++ b/src/Summary.hs @@ -0,0 +1,19 @@ +{-# LANGUAGE RecordWildCards, DeriveGeneric, OverloadedStrings #-} +module Summary where + +import qualified Data.ByteString.Lazy as BS +import Data.Aeson + +import Paths +import JsonUtils + +summaryMain :: [FilePath] -> IO () +summaryMain [rev] = do + json <- BS.readFile (reportOf rev) + rep <- case eitherDecode json of + Left e -> fail e + Right rep -> return rep + let rep' = delete ["revisions","*","benchResults"] $ + delete ["revisions","*","gitLog"] $ + rep + BS.putStr (encode rep') diff --git a/src/WithLatestLogs.hs b/src/WithLatestLogs.hs new file mode 100644 --- /dev/null +++ b/src/WithLatestLogs.hs @@ -0,0 +1,19 @@ +module WithLatestLogs where + +import Development.Shake.Command +import System.Directory +import Control.Monad + +import Paths + +git args = cmd (words "git -C repository" ++ args) + +withLatestLogsMain :: [String] -> IO () +withLatestLogsMain (n:args) = do + Stdout out <- git ["log", "--first-parent", "--format=%H", "master", "-n", n] + let revs = words out + + logs <- filterM (doesFileExist . logsOf) revs + + () <- cmd args (map logsOf logs) + return () diff --git a/src/gipeda.hs b/src/gipeda.hs new file mode 100644 --- /dev/null +++ b/src/gipeda.hs @@ -0,0 +1,38 @@ +import System.Environment +import System.IO +import System.Directory +import Control.Monad + +import Shake +import JsonSettings +import RevReport +import WithLatestLogs +import Summary +import IndexReport +import GraphReport +import BenchNames + +{- +We want to build everything into one executable, bit still treat +it as multiple tools. Hence the main function +here calls the various real main functions, defaulting to the shake tool. +-} + +main :: IO () +main = do + args <- getArgs + + ex <- doesFileExist "settings.yaml" + unless ex $ do + hPutStr stderr "Please run this from the same directory as the settings.yaml file." + + + case args of + "IndexReport":opts -> indexReportsMain opts + "JsonSettings":_ -> jsonSettingsMain + "Summary":opts -> summaryMain opts + "RevReport":opts -> revReportMain opts + "GraphReport":opts -> graphReportMain opts + "WithLatestLogs":opts -> withLatestLogsMain opts + "BenchNames":opts -> benchNamesMain opts + _ -> shakeMain -- shake will take the arguments from getArgs