ipedb-0.2.0.0: README.md

## IpeDB

The IpeDB package contains a library for building and using GHC IPE and cost-centre databases, as well as two command-line tools, `ipedb` and `ccdb`, that provide a command-line interface to the library.

- **Working with IPE data using `ipedb`**
  - [What is IPE data?](#what-is-ipe-data)
  - [How to build a binary with IPE data?](#how-to-build-a-binary-with-ipe-data)
  - [How to build an IPE database with `ipedb`?](#how-to-build-an-ipe-database-with-ipedb)
  - [How to query an IPE database with `ipedb`?](#how-to-query-an-ipe-database-with-ipedb)

- **Working with cost-centre data using `ccdb`**
  - [What is cost-centre data?](#what-is-cost-centre-data)
  - [How to build a binary with cost-centre data?](#how-to-build-a-binary-with-cost-centre-data)
  - [How to build a cost-centre database with `ccdb`?](#how-to-build-a-cost-centre-database-with-ccdb)
  - [How to query a cost-centre database with `ccdb`?](#how-to-query-a-cost-centre-database-with-ccdb)

---

### Working with IPE data using `ipedb`

#### What is IPE data?

Broadly speaking, _Info Tables_ are the runtime representation for Haskell symbols—constructors, functions, etc. An IPE, or _Info Table Provenance Entry_—the T is silent—contains provenance information for its Info Table, such as the symbol's name and its source location. This can be useful for analysing heap profiles generated with [`-hi`](https://well-typed.com/blog/2021/01/first-look-at-hi-profiling-mode/) or stack profiles generated by [`ghc-stack-profiler`](https://hackage.haskell.org/package/ghc-stack-profiler), which by themselves only contain Info Table IDs.

An IPE database is a lookup table from _Info Table ID_ to _Info Table_. An _Info Table ID_ is a stable identifier that can be used to look up the _Info Table_. Since GHC 10, these IDs are determined statically at compile-time, which means that an IPE database can only be reused with the same binary.

> ⚠️ **Warning**:
> IPE data is not portable across builds.
> Prior to GHC 10, an _Info Table ID_ was the actual address of the _Info Table_ in memory. Since these were computed by the linker, they would change for every execution, which meant that IPE databases were not portable across _executions of the same binary_. For binaries built using those older GHC versions, you can only use an IPE database built from an eventlog to analyse that eventlog.
> Stable _Info Table IDs_ were introduced in merge request [!14465](https://gitlab.haskell.org/ghc/ghc/-/merge_requests/14465).

#### How to build a binary with IPE data?

To build a package with IPE data, you must pass `-finfo-table-map` and (optionally, but recommended) `-fdistinct-constructor-tables` to GHC.
You can add the following to your `cabal.project` file to instruct Cabal to build all packages with these options:

```
package *
  ghc-options:
    -finfo-table-map
    -fdistinct-constructor-tables
```

Unfortunately, this won't add IPE data to boot libraries that ship with GHC or to GHC builtins. To get this additional information, you must build GHC with the `ipe` flavour, see GHC [Build flavours](https://gitlab.haskell.org/ghc/ghc/blob/master/hadrian/doc/flavours.md).

#### How to build an IPE database with `ipedb`?

The `ipedb` tool extracts IPE data from a binary's _eventlog_. This is produced when the binary is run with the `-l` RTS option. This means your binary must be built with support for RTS options. You can add the following to your `*.cabal` file:

```cabal
executable my-exe
    -- ...
    ghc-options: -rtsopts
    -- ...
```

You can obtain an eventlog by running your binary with the `-l` RTS option.

```sh
my-exe +RTS -l -olmy.eventlog -RTS
```

The `-ol` RTS option controls the output filename. If this is not provided, it defaults to your program's name with the `.eventlog` extension. If you built your program with GHC >=10, you can pass `-l-aI` to generate an eventlog that _only_ contains IPE events.

Your program does not need to run for long to generate IPE events, as these are emitted during RTS initialisation. If your program has a `--version` or `--help` option, that is more than sufficient.

To generate an IPE database, you run `ipedb index` with the eventlog and an output file:

```sh
ipedb index my.eventlog -o my.ipedb
```

Optionally, you can specify the table format with the `--table-format=<fmt>` flag.
The current implementation supports three formats:

- **lsm**: This format creates a _directory_ that contains an `lsm-tree` snapshot. This format is the fastest to write, since it hard links to the runtime database files, but requires that the session root and the target path are on the same filesystem.
- **tar**: This format creates a tar archive of an **lsm** export. This requires copying the database files.
- **tgz**: This format creates a compressed **tar** export. This format is the slowest to write, since it requires compressing the database files, but the it results in significantly smaller files.

The default table format is **tgz**.

#### How to query an IPE database with `ipedb`?

To query an IPE database, you run `ipedb query` with the IPE database and a series of Info Table IDs:

```sh
ipedb query my.ipedb [IPE_ID...]
```

For example, if we create an IPE database from the eventlog in `test/data/oddball.eventlog`, and run `ipedb query` with `0x100000000`, `0x350000050f`, and `0x1e000006c9`, we get:

```
0x100000000: Just (InfoProv {ipName = "I#_Main_1_con_info", ipClosureDesc = 3, ipTyDesc = "Int", ipLabel = "main", ipModule = "Main", ipSrcLoc = SrcLoc {srcFilePath = Just "app/Main.hs", srcRange = Just (Range'MultiLine {line = 13, column = 1, endLine = 16, endColumn = 11})}})
0x350000050f: Just (InfoProv {ipName = "(,)_GHC.Internal.Data.Data_327_con_info", ipClosureDesc = 4, ipTyDesc = "(,)", ipLabel = "gmapMo.k", ipModule = "GHC.Internal.Data.Data", ipSrcLoc = SrcLoc {srcFilePath = Just "libraries/ghc-internal/src/GHC/Internal/Data/Data.hs", srcRange = Just (Range'OneLine {line = 439, column = 30, endColumn = 44})}})
0x1e000006c9: Just (InfoProv {ipName = "block_caBN_info", ipClosureDesc = 53, ipTyDesc = "", ipLabel = "decodeFloat", ipModule = "Data.Complex", ipSrcLoc = SrcLoc {srcFilePath = Just "libraries/ghc-internal/src/GHC/Internal/Float.hs", srcRange = Just (Range'MultiLine {line = 651, column = 5, endLine = 652, endColumn = 52})}})
```

If you wish to list all IPE data in an IPE database, you can run `ipedb list` with the IPE database:

```sh
ipedb list my.ipedb
```

Be warned that this prints _a lot_ of information.

---

### Working with cost-centre data using `ccdb`

#### What is cost-centre data?

_Cost Centres_ are virtual symbols that correspond closely to the Haskell source symbols. A Haskell binary built with [GHC's profiling](https://downloads.haskell.org/ghc/latest/docs/users_guide/profiling.html) maintains a virtual cost-centre stack at runtime. The cost-centre stack contains entries for all functions annotated by the user with a `{-# SCC myFunctionName #-}` pragma or by some annotation strategy (e.g., `-fprof-auto` or `-fprof-late`). The recommended annotation strategy is `-fprof-late`, see [Late Cost Centre Profiling](https://well-typed.com/blog/2023/03/prof-late/).

_Cost Centres_ are useful when analysing heap profiles generated with [`-hc`](https://downloads.haskell.org/ghc/latest/docs/users_guide/profiling.html#profiling-memory-usage) or stack profiles generated by [`-p`](https://downloads.haskell.org/ghc/latest/docs/users_guide/profiling.html#time-and-allocation-profiling), which by themselves only contain Cost Centre IDs.

A cost-centre database is a lookup table from _Cost Centre ID_ to _Cost Centre_. A _Cost Centre ID_ is a stable identifier that can be used to look up the Cost Centre. When building with the GHC option `-fobject-determinism`, the cost-centre data for a binary is deterministic. This means that if `-fobject-determinism` is passed to GHC and the GHC version, package source, and Cabal build plan are all identical, then the generated cost-centre data should be identical.

#### How to build a binary with cost-centre data?

For a detailed overview of how to build a binary with cost-centre data, see [the GHC users guide](https://downloads.haskell.org/ghc/latest/docs/users_guide/profiling.html#cost-centres-and-cost-centre-stacks). At the time of writing, our recommendation is to add the following to your `cabal.project` file to instruct Cabal to build all packages with late cost-centre profiling:

```
profiling: True

package *
    profiling-detail: late
```

#### How to build a cost-centre database with `ccdb`?

The `ccdb` command-line interface follows the same conventions as `ipedb`. Build your binary with cost-centre data and follow the instructions under [How to build an IPE database with `ipedb`?](#how-to-build-an-ipe-database-with-ipedb) using `ccdb` instead of `ipedb`.

#### How to query a cost-centre database with `ccdb`?

The `ccdb` command-line interface follows the same conventions as `ipedb`. Build your binary with cost-centre data and follow the instructions under [How to build an IPE database with `ipedb`?](#how-to-query-an-ipe-database-with-ipedb) using `ccdb` instead of `ipedb`.