fibon-0.1.0: README

Fibon in a Flash
===================================================================
    $ git clone git://github.com/dmpots/fibon.git
    $ cd fibon
    $ git submodule update --init benchmarks
    $ cabal configure && cabal build
    $ ./dist/build/fibon-run/fibon-run

Introduction
===================================================================
Fibon is a set of tools for running and analysing benchmarks in
Haskell. Most importantly, it includes a set of default benchmarks
taken from the [Hackage][1] open source repository.

Benchmarks
------------------
Fibon makes it easy to use either the fibon benchmarks or your own
set of benchmarks. Benchmarks are stored in the
`benchmarks/Fibon/Benchmarks` directory. This directory is setup as
a [git submodule][2] which means you can easily grab the standard
suite or use a suite kept under your own source control.

Benchmark Groups
------------------
Benchamarks named and organized into groups based on the filesystem
organization. For example, a benchmark in the directory
`benchmarks/Fibon/Benchmarks/Hackage/Agum` will have the name `Agum`
an be in the benchmark group `Hackage`.

Executables
------------------
The fibon package builds three tools
  1. `fibon-run` - runs the benchmarks
  2. `fibon-analyze` - analyzes the results of a run
  2. `fibon-init` - utilty used when adding new benchmarks

Size and Tune
------------------
Fibon benchmarks can be run with two different input sizes: `Test` and
`Ref`. The `Test` size is useful to make sure that a benchmark can
run successsfully, but will not give meaningful timings.

Fibon benchmarks can be run under two different tune settings (e.g.
compiler optimization settings). The `Base` and `Peak` settings can
be configured anyway you want to make the desired comparison.

Directory Structure
--------------------
Source directories
    ./benchmarks -- benchmark code
    ./config     -- config files
    ./lib        -- common files used by several executables
    ./tools      -- source code for executables

Working directories
    ./log        -- logging output from benchmark runs
    ./run        -- working directory for benchmark runs

Getting the Benchmarks
===================================================================
The benchmarks are kept in a separate repository as a [git
submodule][2]. You can get the fibon benchmarks by updating the
submodule from within your fibon working directory

    $ git submodule update --init benchmarks

This will checkout the benchmarks from the fibon-benchmarks
repository and place them in your working copy.

Running Benchmarks
===================================================================
The available benchmarks and configurations are discovered when the
fibon package is configured. Benchmarks are searched for in the
`benchmarks/Fibon/Benchmarks` directory and configuration files are
searched for in the `config` directory. If a configuration file or
benchmark is added, you will need to re-run `cabal configure` to
make them available to the fibon-run tool.

Configuration
---------------
Fibon comes with a default configuration. The default configuration
will run all benchmarks with the `Base` setting of `-O0` and a
`Peak` setting of `-O2` on the `Ref` size. A configuration file can
be used to specify more complicated configurations.

You can get some example configuration by doing
    $ git submodule update --init config

This will checkout a repository of config files. Note that currently
these files contain some user and machine-specific configurations,
but should be a useful starting point.

You can also selectively run benchmarks, groups, sizes, and tune
settings as described below.

Running
---------------
Benchamarks are run with the `fibon-run` tool. Running `fibon-run`
with no arguments will use the default config file. An alternate
config file can be specified with the `-c` flag. Also, you can give
a list of benchmarks or groups to run on the command line. Use
`--help` to see a full list of options.

Running the benchmarks will produce some logging to standard out and
create four output files in the `log` directory.

  1. `*.LOG` - the full log of the run
  1. `*.SUMMARY` - the mean runtimes of each benchmark
  1. `*.RESULTS` - the full results (passed to `fibon-analyse`)

Analyzing Benchmark Results
===================================================================
Benchmarks can be analyzed by the `fibon-analyse` tool.

    $ fibon-analyse log/000.default.RESULTS

Adding New Benchmarks
===================================================================
TODO

Benchmark Notes
===================================================================
Ghc610
  ChameneosRedux
    Does not work with -O0. Gets "thread blocked indefinitely"
    exception

  Mandelbrot
    The Test size gives different result, but the Ref size is ok.
    Think it is just some kind of floating point wibbles.

[1]: http://hackage.haskell.org
[2]: http://www.kernel.org/pub/software/scm/git/docs/user-manual.html#submodules