# rocksdb-haskell-jprupp
[![Hackage][hackage-badge]][hackage]
[![CI][ci-badge]][ci]
[![License: BSD-3-Clause][license-badge]][license]
[ci]: https://github.com/jprupp/rocksdb-haskell/actions/workflows/ci.yml
[ci-badge]: https://github.com/jprupp/rocksdb-haskell/actions/workflows/ci.yml/badge.svg?branch=master
[hackage]: https://hackage.haskell.org/package/rocksdb-haskell-jprupp
[hackage-badge]: https://img.shields.io/hackage/v/rocksdb-haskell-jprupp.svg
[license]: https://github.com/jprupp/rocksdb-haskell/blob/master/LICENSE
[license-badge]: https://img.shields.io/badge/license-BSD--3--Clause-blue.svg
Haskell bindings to [RocksDB](https://rocksdb.org), the embedded persistent
key-value store. The package on Hackage is `rocksdb-haskell-jprupp`; the module
to import is `Database.RocksDB`.
The bindings follow the RocksDB C API closely. On top of it they add
bracket-style resource management, so that databases, snapshots and iterators
are always released, even when an exception is thrown.
- [API documentation](https://hackage.haskell.org/package/rocksdb-haskell-jprupp)
- [Changelog](https://github.com/jprupp/rocksdb-haskell/blob/master/CHANGELOG.md)
- [Report a bug](https://github.com/jprupp/rocksdb-haskell/issues)
## Contents
- [Why these bindings](#why-these-bindings)
- [Design principles](#design-principles)
- [Features](#features)
- [Installation](#installation)
- [Quick start](#quick-start)
- [Tutorial: an event log](#tutorial-an-event-log)
- [Concepts](#concepts)
- [Usage guide](#usage-guide)
- [Configuration](#configuration)
- [Module overview](#module-overview)
- [Things to watch out for](#things-to-watch-out-for)
- [Development](#development)
- [Related projects](#related-projects)
- [License](#license)
## Why these bindings
RocksDB is a storage engine: a library that keeps sorted keys and values in
files on local disk, with fast writes, fast point lookups and fast ordered
scans. There is no server to run. It is what you reach for when SQLite's
relational model is more than you need, and an in-memory map is not durable or
big enough.
These bindings exist because the Haskoin Bitcoin indexer needed them. It keeps
the whole Bitcoin block chain indexed by address, transaction and block, which
is hundreds of gigabytes, and serves queries from many threads while new
blocks are written. That use shaped the library:
- it must be safe to share one database between many threads;
- resources must never leak, because the process runs for a long time;
- memory outside the Haskell heap must be boundable, because RocksDB's defaults
let it grow with the size of the database;
- the hot paths, such as reading many keys at once, must cost as little as the
C API allows.
The library is a fork of `leveldb-haskell` and of earlier RocksDB bindings,
maintained as `rocksdb-haskell-jprupp`.
## Design principles
- **Stay close to the C API.** Functions map nearly one to one to RocksDB's C
functions, so RocksDB's own documentation and tuning guides apply directly.
Conveniences such as typed keys live in separate libraries
([rocksdb-query](https://github.com/jprupp/rocksdb-query)). The two
exceptions are `iterChunk`, which walks an iterator, and `writePacked`,
which fills a batch, each in a C function of this package: the cost they
remove is that of calling C from Haskell.
- **Scope every resource.** Databases, snapshots and iterators are C objects
that must be freed once and never touched again. `withDB`, `withSnapshot` and
`withIter` free them when their block ends, even on an exception.
- **Plain `IO`.** No monad transformer constraints. Lift into your own monad
with `liftIO`.
- **Bytes in, bytes out.** Keys and values are strict `ByteString`s, which is
what RocksDB stores. How you serialize them is your choice.
- **Don't crash in C.** RocksDB crashes the process if you misuse an iterator.
The Haskell side checks first and turns those cases into no-ops or
`Nothing`.
- **Expose what bounds memory.** The few settings that turn RocksDB's memory
use into a fixed budget are first-class fields of `Config`.
## Features
| Area | What you get |
| -------------- | ------------------------------------------------------------------------------ |
| Basic access | `get`, `put`, `delete` on strict `ByteString` keys and values |
| Batches | Atomic `write` of many puts and deletes, across column families |
| Packed batches | A batch of any size handed to RocksDB in a few foreign calls, from your buffers |
| Merges | Change a value without reading it: merge records, and a merge operator in C |
| Multi-get | `multiGet` reads many keys in one call to RocksDB's `MultiGet` |
| Column families| Open a database with several independent key spaces; add new ones on open |
| Snapshots | Consistent point-in-time reads across many queries and iterators |
| Iterators | Ordered scans, forwards and backwards, optionally pinned to a snapshot |
| Bounds | Iterators that stop at a key or at the end of a prefix, without reading past it |
| Chunks | Hundreds of entries from one foreign call, in either direction, within a prefix |
| Tuning | Bloom filters, fixed-length prefix extractor, shared LRU block cache |
| Limits | Memtable and WAL size limits, open file limit |
| Administration | Properties (memory use, statistics, SST files), approximate sizes, destroy, repair |
### What is not included
- No transactions (`TransactionDB`, `OptimisticTransactionDB`), TTL
databases, backups or checkpoints.
- No merge operators written in Haskell: RocksDB calls a merge operator
from its own threads, so it is a C function.
- No `DeleteRange` or `SingleDelete`.
- No custom comparators: keys always sort by their bytes.
Pull requests that add bindings for these are welcome.
## Installation
### System library
You need the RocksDB C library and its headers.
```sh
# Debian and Ubuntu
sudo apt-get install librocksdb-dev
# Fedora
sudo dnf install rocksdb-devel
# macOS with Homebrew
brew install rocksdb
# Nix
nix-shell -p rocksdb
```
The library links against `-lrocksdb`. If RocksDB is installed in a
non-standard place, pass its directories with `--extra-include-dirs` and
`--extra-lib-dirs` (Cabal or Stack).
The headers are needed to build, not only the library: the binding of the
Bloom filter policy is compiled against `rocksdb/c.h`, because RocksDB changed
the type of its argument in 6.24 and the header says which one is installed.
### Check the library is found
```sh
ls /usr/include/rocksdb/c.h # or $(brew --prefix)/include/rocksdb/c.h
pkg-config --modversion rocksdb # prints the installed version, if registered
```
### Step by step with Stack
1. Install [Stack](https://docs.haskellstack.org/) and the RocksDB library
(above).
2. Create a project:
```sh
stack new rocks-demo simple
cd rocks-demo
```
3. Add `rocksdb-haskell-jprupp`, `bytestring` and `data-default` to
`build-depends` in `rocks-demo.cabal`.
4. Put the [quick start](#quick-start) program in `src/Main.hs` and run it:
```sh
stack run
```
It creates a database in `/tmp/example` and prints what it reads back.
If the build fails with `cannot find -lrocksdb` or `rocksdb/c.h: No such file`,
the C library is not where the compiler looks. Install the development package,
or pass `--extra-include-dirs` and `--extra-lib-dirs`.
### With Cabal
```cabal
build-depends:
base
, bytestring
, data-default
, rocksdb-haskell-jprupp
```
`data-default` gives you `def` for the default `Config`.
### With Stack
The package is in Stackage nightly snapshots. If your resolver does not have
it, or has an older version, add it to `stack.yaml`:
```yaml
extra-deps:
- rocksdb-haskell-jprupp-2.8.0
```
With `stack --nix`, add RocksDB to the Nix packages:
```yaml
nix:
packages:
- rocksdb
```
## Quick start
```haskell
{-# LANGUAGE LambdaCase #-}
{-# LANGUAGE OverloadedStrings #-}
import Data.Default (def)
import Database.RocksDB
main :: IO ()
main = do
let cfg = def {createIfMissing = True, bloomFilter = True}
withDB "/tmp/example" cfg $ \db -> do
put db "hello" "world"
get db "hello" >>= print -- Just "world"
-- atomic batch
write db [Put "a" "1", Put "b" "2", Del "hello"]
-- many keys at once
multiGet db ["a", "b", "c"] >>= print -- [Just "1",Just "2",Nothing]
-- every entry, in key order
withIter db $ \it -> do
iterFirst it
let loop =
iterEntry it >>= \case
Nothing -> return ()
Just kv -> print kv >> iterNext it >> loop
loop
```
## Tutorial: an event log
This builds a small append-only event log, to show how a RocksDB schema is
designed around sorted keys.
### 1. Design the keys
RocksDB sorts keys by their bytes. Records read together should therefore sort
together. Events are read per user and in time order, so the key is the user
followed by a big-endian timestamp, which sorts numerically:
```haskell
{-# LANGUAGE OverloadedStrings #-}
import Data.ByteString (ByteString)
import qualified Data.ByteString as B
import qualified Data.ByteString.Builder as BB
import qualified Data.ByteString.Lazy as BL
import Data.Word (Word64)
eventKey :: ByteString -> Word64 -> ByteString
eventKey user time =
BL.toStrict . BB.toLazyByteString $
BB.byteString user <> BB.char7 '/' <> BB.word64BE time
```
### 2. Write atomically
An event and a per-user counter must stay consistent, so they go in one batch:
```haskell
import Database.RocksDB
logEvent :: DB -> ByteString -> Word64 -> ByteString -> Int -> IO ()
logEvent db user time payload count =
write db
[ Put (eventKey user time) payload,
Put ("count/" <> user) (BL.toStrict (BB.toLazyByteString (BB.intDec count)))
]
```
If the program dies halfway, either both writes are there or neither is.
### 3. Read a range with an iterator
All events of a user sort together, so one seek and a short scan read them:
```haskell
eventsOf :: DB -> ByteString -> IO [ByteString]
eventsOf db user = withIter db $ \it -> do
let prefix = user <> "/"
iterSeek it prefix
let go acc = do
mkv <- iterEntry it
case mkv of
Just (k, v) | prefix `B.isPrefixOf` k -> iterNext it >> go (v : acc)
_ -> return (reverse acc)
go []
```
A list of thousands of events is read faster in chunks: see
[Reading many entries in one call](#reading-many-entries-in-one-call).
### 4. Read consistently with a snapshot
If another thread logs events while you read the counter and then the events,
the two can disagree. Read both through a snapshot:
```haskell
report :: DB -> ByteString -> IO (Maybe ByteString, [ByteString])
report db user = withSnapshot db $ \snap ->
(,) <$> get snap ("count/" <> user) <*> eventsOf snap user
```
### 5. Tune for size
Once the log holds millions of events, open it with a memory budget and a Bloom
filter for faster point lookups:
```haskell
let cfg =
def
{ createIfMissing = True,
bloomFilter = True,
maxFiles = Just 1000,
blockCacheSize = Just (512 * 1024 * 1024),
cacheIndexAndFilterBlocks = True
}
```
## Concepts
### The database handle
`withDB` and `withDBCF` open a database, pass a `DB` to your function, and close
it when the function returns or throws. The `DB` must not escape that function.
It is safe to use from many threads at the same time.
### Keys and values
Keys and values are strict `ByteString`s. RocksDB sorts keys by their bytes,
which is the order iterators walk them in. To store Haskell types, serialize
them yourself, or use [rocksdb-query](https://github.com/jprupp/rocksdb-query).
### Column families
A database can hold several key spaces called column families, each with its
own configuration. Functions ending in `CF` act on a given column family; the
others act on the default one. The handles are in `columnFamilies db`, in the
order you listed them when opening.
### Snapshots
`withSnapshot db f` passes `f` a `DB` that reads from a frozen view of the
database. Reads and iterators through it do not see later writes. Writes
through it still go to the live database.
### Errors
Failures reported by RocksDB are thrown as `IOError` (built with `userError`).
The message starts with the operation, such as `put: ...` or `open: ...`.
## Usage guide
### Column families
```haskell
withDBCF "/tmp/example-cf" cfg [("users", cfg), ("orders", cfg)] $ \db -> do
let [users, orders] = columnFamilies db
putCF db users "alice" "..."
getCF db orders "1234" >>= print
write db [PutCF users "bob" "...", DelCF orders "1234"]
```
On an empty directory, `withDBCF` creates the database and all the column
families. On an existing database, RocksDB needs every column family that
exists to be listed. To add a new one, list it and set
`createMissingColumnFamilies = True` in the database `Config`.
### Snapshots
```haskell
withSnapshot db $ \snap -> do
a <- get snap "a"
b <- get snap "b" -- same point in time as "a"
withIter snap $ \it -> iterFirst it >> iterEntry it >>= print
```
When the snapshot's lifetime does not fit a scope, use `createSnapshot` and
`releaseSnapshot`. `withIterSnap` and `withIterSnapCF` iterate over a raw
`Snapshot`.
### Iterators
An iterator starts out unpositioned. Move it with `iterFirst`, `iterLast` or
`iterSeek`, then step with `iterNext` and `iterPrev`. `iterKey`, `iterValue` and
`iterEntry` return `Nothing` once it moves past either end.
```haskell
-- every key that starts with "user:"
withIter db $ \it -> do
iterSeek it "user:"
let go acc =
iterEntry it >>= \case
Just (k, v) | "user:" `B.isPrefixOf` k -> iterNext it >> go ((k, v) : acc)
_ -> return (reverse acc)
users <- go []
iterGetError it >>= mapM_ (fail . show) -- tell the end apart from an error
return users
```
`createIterator`, `createIteratorSnap` and `destroyIterator` manage iterators
by hand.
### Bounded iterators
The scan above stops when it reads a key that does not start with `user:`.
RocksDB has to find that key first, and that can cost far more than the scan
itself. A deleted key is not gone: it leaves a marker that stays until
compaction drops it, and an iterator steps over every marker on its way to
the next key that is alive. If the keys after `user:` were a queue that has
been emptied, the scan reads the marker of every job that ever was in it.
`withIterConfig` creates an iterator that RocksDB stops at a bound, without
looking past it:
```haskell
-- every key that starts with "user:", and not one step further
withIterConfig db Nothing def {upperBound = prefixUpperBound "user:"} $ \it -> do
iterSeek it "user:"
let go acc =
iterEntry it >>= \case
Just kv -> iterNext it >> go (kv : acc)
Nothing -> return (reverse acc)
go []
```
The second argument is the column family, or `Nothing` for the default one.
The `IterConfig` has four fields, all off in `def`:
| Field | Type | What it does |
| ------------------- | ------------------ | --------------------------------------------------------------- |
| `upperBound` | `Maybe ByteString` | The iterator stays below this key; the key itself is outside |
| `lowerBound` | `Maybe ByteString` | The iterator does not go below this key; the key itself is inside |
| `prefixSameAsStart` | `Bool` | The iterator stays within the prefix of the key it was sent to |
| `totalOrderSeek` | `Bool` | See every key in order, in a database that has a `prefixLength` |
Which one to use:
- **`upperBound`**, with `prefixUpperBound`, when each range gets its own
iterator. It needs nothing from the configuration of the database, and the
range can be any.
- **`prefixSameAsStart`** when one iterator is sent to many ranges, seek
after seek, which saves creating an iterator for each: the bound moves with
every `iterSeek`. It needs a `prefixLength` in the `Config`, and each range
has to be a whole prefix of that length. It also lets RocksDB use its
prefix filters to skip the files that have nothing for the prefix. Every
key, sought or stored, has to be at least as long as the prefix: see
[Things to watch out for](#things-to-watch-out-for).
- **`totalOrderSeek`** to walk a database that has a `prefixLength` from end
to end. Without it, an iterator there only promises the keys that share
the prefix of the key it was sent to.
- **`withIter`** when what follows the range is alive, or when there is no
range. A bound costs a little when the iterator is created, and gains
nothing there.
How much it matters, in this package's own test: one live key, then 20,000
keys that were written and deleted, then another live key. A scan from just
after the first one, which finds nothing, in microseconds:
| Iterator | Time |
| -------------------------------- | ----- |
| `withIter` | 2,600 |
| `withIterConfig` with an `upperBound` | 2.5 |
`createIteratorConfig` manages such an iterator by hand. It returns the
iterator together with the action that destroys it, which also frees the
bounds: RocksDB keeps pointers to them for as long as the iterator lives.
### Reading many entries in one call
`iterNext` may wait for the disk, so it is a foreign call of the kind that
lets other Haskell threads run meanwhile. For that, the thread gives up its
capability and has to get one back when the call returns. A scan that moves
one step at a time pays this for every entry, and under load, when more
threads want to run than there are capabilities, it waits each time.
`iterChunk` walks the iterator in C and returns up to some hundreds of
entries from one call, with a flag that tells whether more follow:
```haskell
-- every key that starts with "user:", in chunks
withIter db $ \it -> do
iterSeek it "user:"
let go acc = do
(entries, more) <- iterChunk it def {chunkPrefix = "user:"}
let acc' = entries : acc
if more then go acc' else return (concat (reverse acc'))
go []
```
The `ChunkConfig` says what one call reads:
| Field | Default | What it does |
| ---------------- | ------- | ------------------------------------------------------------------- |
| `chunkEntries` | 256 | The most entries to read |
| `chunkBytes` | 1 MiB | Stop before keys and values take more; one entry is always read |
| `chunkPrefix` | empty | Stop at the first key that does not start with it |
| `chunkBackwards` | `False` | Go backwards, as `iterPrev` does |
The iterator is left on the first entry that was not returned, so the next
call continues there. A bound of the iterator, from `withIterConfig`, ends a
chunk as it ends a scan by single steps. And `iterChunk` throws if the
iterator stopped at an error, so there is no `iterGetError` to remember.
For a known number of entries there are short forms, which return just the
list: `iterEntries it n`, `iterEntriesBack it n`, `iterEntriesPrefix it
prefix n` and `iterEntriesPrefixBack it prefix n`. They have no limit of
bytes, so a list shorter than `n` means that the scan is over.
```haskell
-- a page of twenty users, starting after the last one seen
iterSeek it (lastSeen <> "\0")
page <- iterEntriesPrefix it "user:" 20
```
What it saves, in this package's benchmark (`stack bench`): seconds for
every thread to read 200,000 entries of 52 bytes, on four capabilities and
four cores.
| Scan | 1 thread | 16 threads |
| -------------------------------------- | -------- | ---------- |
| Single steps, loop not tail recursive | 0.78 | 3.9 |
| Single steps, loop tail recursive | 0.090 | 0.50 |
| Chunks of 256 entries | 0.048 | 0.25 |
The first row is a loop that builds its list with `(entry :) <$> go`. The
run-time system walks the stack of a thread at every foreign call that may
block, and such a loop leaves a frame there for every entry read so far.
Whatever you read with, keep the result in an argument of the loop.
With more cores than capabilities the chunks gain more, because a thread
that is inside the call holds no capability: with twenty more cores free,
the sixteen threads took 0.12 seconds in chunks and 0.83 in single steps.
Keep to single steps where each move depends on the entry just read, or
where one entry is all you want: `iterEntry` after a seek reads it without
moving the iterator.
### Writing a large batch in a few calls
`write` hands a batch to RocksDB one operation at a time, and each is a
foreign call of the kind that lets other Haskell threads run meanwhile. For
a batch of a few operations that is nothing. For tens of thousands it adds
up, and so do the `ByteString`s, one for each key and each value.
A packed batch is the same operations written one after another into a few
large buffers, the chunks. `writePacked` gives each chunk to a C function of
this package, which adds its operations to the batch, and then writes the
batch as `write` does: atomically, with one write.
```haskell
-- the same as: write db ops
writePacked db [packOps ops]
```
`packOps` packs a list of `BatchOp` into one chunk. It is the simple way,
and it copies every key and value once more. A program that serialises its
keys and values itself can skip the list, the `ByteString`s and the copy,
and write records straight into a buffer of its own:
```haskell
-- one record at a pointer: room for the header, the key, the value, and
-- then the header, now that the lengths are known
packRecord :: Ptr Word8 -> ColumnFamily -> IO (Ptr Word8)
packRecord ptr cf = do
let body = ptr `plusPtr` packedHeaderSize
keyLen <- serialiseKey body
valueLen <- serialiseValue (body `plusPtr` keyLen)
pokePackedHeader ptr (Just cf) PackedPut keyLen valueLen
return (body `plusPtr` (keyLen + valueLen))
```
Here `serialiseKey` and `serialiseValue` stand for whatever writes your
bytes at a pointer and says how many. The next record goes where this one
ended: records need no alignment and nothing goes between them. A chunk is
then a `ByteString` over the part of the buffer that holds whole records.
What to know:
- **All the chunks are one batch.** `writePacked db chunks` applies the
records of every chunk in order, atomically, with one write. Chunks only
decide how much RocksDB is handed in one call: a few hundred kilobytes is
plenty. A program that packs on several cores gives the chunks of each
core, one core after another.
- **A chunk holds whole records.** A record cannot start in one chunk and
end in the next.
- **A chunk is checked before it is used.** One that is cut short, or whose
lengths say more than it holds, is refused with an exception, and nothing
of the batch is written. The check cannot cover the column family handle
of a record.
- **A chunk is only good here and now.** It holds column family handles,
which are pointers of this process and of the open database.
- **A record can be a merge.** `PackedMerge` adds a merge record, as
`Merge` does in a `BatchOp`, for a column family that has a merge
operator: see [Merge records](#merge-records-change-a-value-without-reading-it).
The layout of a record, for a program that writes them by other means, is
in `cbits/rocksdb_haskell.h`. It is this package's own format, and what a
chunk becomes is a series of calls to RocksDB's C API. It is not RocksDB's
internal encoding of a batch, which a binding should not depend on.
What it saves, in this package's benchmark
(`stack bench rocksdb-haskell-jprupp:bench:batch`): milliseconds to write a
batch of 65,000 operations and 6 MB over four column families, on four
capabilities and four cores, alone and next to eight threads that compute.
| Way to write the batch | Alone | Next to busy threads |
| ----------------------------------------- | ----- | -------------------- |
| `write`: a call for each operation | 48 | 420 |
| `writePacked`, with `packOps` in the time | 28 | 115 |
| `writePacked`, chunks packed beforehand | 22 | 28 |
All three end with the same write, which takes about 20 ms here: the rest
is the handing over. Alone, a thread gets its capability back at once
after each call, and still pays 400 ns an operation. Next to threads that
want the capabilities too, as in a server under load, it waits for one
after every call. `packOps` is work in Haskell, which has to share the
capabilities with those threads like any other.
With all 24 hardware threads of the machine free, the batch took `write`
1,160 ms next to the busy threads, and 31 ms from packed chunks.
Keep to `write` for small batches: it is as fast there, and simpler.
### Merge records: change a value without reading it
A put replaces the value of a key. A program that wants to change a value,
to add to a counter or to add an entry to a list, has to read it first.
For a batch of thousands of such changes those reads are most of the work,
and each may wait for the disk.
A **merge record** is a third kind of write, next to a put and a delete. It
says how the value changes, not what it becomes: "add 5", "add this
entry". RocksDB stores the record as it is and reads nothing.
The work is done later by the **merge operator** of the column family: a
function that takes the value that the key had and the records written
since, oldest first, and gives the value that the key has now. RocksDB
calls it when the key is read, and when a compaction brings the records
and the value together. After that the key holds a plain value again.
```haskell
-- a column family of counters, with RocksDB's own operator for them
let counters = def {mergeOperator = Just UInt64Add}
withDBCF "/tmp/example-merge" def {createIfMissing = True} [("hits", counters)] $ \db -> do
let [hits] = columnFamilies db
five = BS.pack [5, 0, 0, 0, 0, 0, 0, 0] -- eight bytes, least significant first
mergeCF db hits "home" five -- no read
mergeCF db hits "home" five
getCF db hits "home" >>= print -- Just "\n\NUL\NUL\NUL\NUL\NUL\NUL\NUL": ten
write db [MergeCF hits "home" five, MergeCF hits "about" five]
```
What to know:
- **Readers see values.** `get`, `multiGet` and iterators return merged
values, in a snapshot too. For a key with records pending they pay for
the merge, and for a look into each table file that holds one of the
records.
- **A put or a delete hides the records before it**, like any older value.
A record written after a delete, or to a key that never had a value, is
merged with no value: the operator decides what that means.
- **The operator is set when the database is opened**, with the
`mergeOperator` field of the `Config` of the column family. The one in
the `Config` of the database is for the default column family.
- **`UInt64Add`** is RocksDB's own: values and records are unsigned 64-bit
numbers in eight bytes, least significant first, added modulo 2^64.
- **`MergeFunctions`** makes an operator of a name and two C functions: one
that works out a value (the full merge), and one that makes a single
record of several (the partial merge), which RocksDB uses to shorten a
list of records before it has the value. They are C because RocksDB
calls them from its own threads, during reads, flushes and compactions.
```c
/* my_merge.c, listed under c-sources of your package */
char *my_full_merge(void *state, const char *key, size_t key_length,
const char *existing_value, size_t existing_value_length,
const char *const *operands_list,
const size_t *operands_list_length, int num_operands,
unsigned char *success, size_t *new_value_length);
char *my_partial_merge(void *state, const char *key, size_t key_length,
const char *const *operands_list,
const size_t *operands_list_length, int num_operands,
unsigned char *success, size_t *new_value_length);
```
```haskell
foreign import ccall "&my_full_merge" myFullMerge :: FunPtr FullMerge
foreign import ccall "&my_partial_merge" myPartialMerge :: FunPtr PartialMerge
mine :: Config
mine = def {mergeOperator = Just (MergeFunctions "my-merge" myFullMerge myPartialMerge)}
```
A function returns the new bytes in memory from `malloc`, which RocksDB
frees, with their length in `*new_value_length` and 1 in `*success`.
`existing_value` is `NULL` when the key has no value. To fail, it stores 0
in both and returns `NULL`: the read of that key then throws. A partial
merge may always fail, and RocksDB then keeps the records as they are.
`cbits/rocksdb_haskell.h` has the two types and the rules in full, and
`test/cbits/merge_test.c` is a small operator to start from.
**Every program that opens the database must set the same operator.**
RocksDB opens a column family that has merge records without one, and
this is what version 11.1 was seen to do then:
| Where the merge records are | What happens without the operator |
| --------------------------- | -------------------------------------------------------------- |
| In the write-ahead log | Dropped when the database is opened, without an error: lost |
| In table files | Kept, also through a compaction, but they cannot be read |
| Read: records and no value | `get` throws; an iterator stops there with an error |
| Read: records over a value | `get` and `multiGet` crash the process |
| Writing a merge record | Refused with an error |
So when a program starts to write merge records to a database that older
versions of it can open, make sure that they no longer can. A new column
family is enough: RocksDB refuses to open a database without all its
column families, and does so before it reads the write-ahead log.
### Properties and maintenance
```haskell
getProperty db Stats >>= mapM_ BS.putStr
getProperty db BlockCacheUsage
getPropertyCF db cf CurSizeAllMemTables
approximateSize db ("a", "z")
```
`destroy` deletes a closed database and `repair` tries to recover a damaged one.
Both take a raw `Options` pointer, which the public API does not let you build
yet, so in practice they are only usable from inside the library.
## Configuration
`Config` has a `Default` instance: all flags off and all sizes left to RocksDB.
| Field | Scope | Meaning |
| ----------------------------- | --------------- | --------------------------------------------------------------- |
| `createIfMissing` | database | Create the database if it does not exist |
| `errorIfExists` | database | Fail if the database already exists |
| `paranoidChecks` | database | Check data aggressively and stop on corruption |
| `createMissingColumnFamilies` | database | Create column families given to `withDBCF` that are missing |
| `maxFiles` | database | Maximum number of open files (default unlimited) |
| `dbWriteBufferSize` | database | Bytes of all memtables together (default unlimited) |
| `blockCacheSize` | database | Bytes of an LRU block cache shared by all column families |
| `maxTotalWalSize` | database | Bytes of write-ahead log that trigger flushes |
| `maxBackgroundJobs` | database | Threads for flushes and compactions together (RocksDB default 2) |
| `writeBufferSize` | column family | Bytes per memtable (RocksDB default 64 MiB) |
| `bloomFilter` | column family | Full Bloom filter with 10 bits per key |
| `prefixLength` | column family | Length of a fixed key prefix for the prefix extractor |
| `cacheIndexAndFilterBlocks` | column family | Keep index and filter blocks in the block cache |
| `partitionIndexAndFilters` | column family | Write index and filter in 4 KiB pieces, loaded as needed |
| `mergeOperator` | column family | How merge records are combined with a value (default: none) |
| `disableAutoCompactions` | column family | No background compaction: for bulk loads, tests and benchmarks |
Database-wide fields are ignored in the configuration of a column family.
The column family fields of the database's own `Config` are for its default
column family.
### Choosing a configuration
RocksDB's defaults are tuned for small databases. Its memory use outside the
Haskell heap comes from three places:
1. **Memtables**, where recent writes collect before being flushed to disk.
Capped per column family by `writeBufferSize` and in total by
`dbWriteBufferSize`.
2. **Index and filter blocks** of every open table file. Without
`cacheIndexAndFilterBlocks` they live outside any cache and grow with the
number of files; `maxFiles` limits how many are open.
3. **The block cache** of recently read data. `blockCacheSize` makes one cache
shared by all column families, so memory goes where reads are.
A reasonable starting point for a large database is a block cache of a few
hundred megabytes to a few gigabytes, `cacheIndexAndFilterBlocks = True`,
`maxFiles` of a few thousand, and `dbWriteBufferSize` of a few hundred
megabytes. Enable `bloomFilter` when most reads are point lookups of keys that
may not exist. A filter takes about 1.25 bytes per key, held with the index
blocks: in the block cache with `cacheIndexAndFilterBlocks`, in memory outside
it otherwise. A filter belongs to a table file and is written with it, so
turning `bloomFilter` on for an existing database only covers the files
written from then on; the others get a filter when compaction rewrites them.
For a large database, setting `maxFiles`, `blockCacheSize`,
`cacheIndexAndFilterBlocks` and `dbWriteBufferSize` puts a ceiling on the memory
RocksDB uses outside the Haskell heap. Without them, index and filter blocks of
every open file and every memtable can grow without bound.
## Module overview
`Database.RocksDB` is the only exposed module. It re-exports:
| Part | Contents |
| ------------------------ | -------------------------------------------------------------- |
| Types | `DB`, `Config`, `BatchOp`, `ColumnFamily`, `Snapshot`, `Iterator`, `Property`, `Range` |
| Merge operators | `MergeOperator`, `FullMerge`, `PartialMerge` |
| Opening | `withDB`, `withDBCF` |
| Reading and writing | `get`, `getCF`, `multiGet`, `multiGetCF`, `put`, `putCF`, `delete`, `deleteCF`, `write` |
| Merging | `merge`, `mergeCF`, and `Merge`, `MergeCF` in a `BatchOp` |
| Packed batches | `writePacked`, `packOps`, `PackedOp`, `packedHeaderSize`, `pokePackedHeader` |
| Snapshots | `withSnapshot`, `createSnapshot`, `releaseSnapshot` |
| Creating iterators | `withIter`, `withIterCF`, `withIterSnap`, `withIterSnapCF` |
| Bounded iterators | `IterConfig`, `withIterConfig`, `createIteratorConfig`, `prefixUpperBound` |
| Moving iterators | `iterSeek`, `iterFirst`, `iterLast`, `iterNext`, `iterPrev` |
| Reading from iterators | `iterKey`, `iterValue`, `iterEntry`, `iterValid`, `iterGetError` |
| Reading in chunks | `iterChunk`, `ChunkConfig` |
| Reading a number of entries | `iterEntries`, `iterEntriesBack`, `iterEntriesPrefix`, `iterEntriesPrefixBack` |
| Administration | `getProperty`, `getPropertyCF`, `approximateSize`, `compactRange`, `destroy`, `repair` |
| Flushing | `flush` |
The FFI bindings (`Database.RocksDB.C`) and helpers (`Database.RocksDB.Internal`)
are hidden modules. `cbits/rocksdb_haskell.c` is the one piece of C in the
package: the functions behind `iterChunk` and `writePacked`, and the one
that makes a merge operator of two functions.
## Things to watch out for
- Do not use a `DB`, snapshot or iterator after the function that gave it to you
has returned. The C object is gone and RocksDB will crash.
- Only one process can open a database at a time. A second open fails with a
lock error.
- Moving an invalid iterator does nothing. Reposition it with `iterSeek`,
`iterFirst` or `iterLast`.
- An iterator that becomes invalid because of corruption looks like one that
reached the end. Check `iterGetError`.
- A scan that stops by looking at the key it has reached steps over every
deleted key that follows its range. Where keys come and go, give the
iterator a bound with `withIterConfig`.
- In a database with a `prefixLength`, an iterator that moves past the
prefix it was sent to, or that starts with `iterFirst` or `iterLast`, may
skip keys. Ask for `totalOrderSeek`, or stay within the prefix with
`prefixSameAsStart`.
- With `prefixSameAsStart`, no key may be shorter than the `prefixLength`:
not the keys you seek, and not the keys in the database. RocksDB takes the
prefix of a key without checking its length and reads the memory that
follows a shorter one, so the iterator stops at an arbitrary place. The
binding cannot check this, because an iterator does not know how it was
made. Where keys can be shorter, use an `upperBound`.
- With `prefixSameAsStart`, `iterFirst` and `iterLast` give the first and
the last prefix of the database, and no more: the iterator stays within
the prefix of the key it finds.
- Destroy an iterator from `createIteratorConfig` with the action that came
with it, not with `destroyIterator`, which would leave its bounds and read
options allocated.
- Calls into RocksDB are `safe` foreign calls, so they do not block other
Haskell threads, but each one costs more than an `unsafe` call, and much
more when threads compete for capabilities. A scan by single steps makes
one for every entry: read ranges with `iterChunk`. A `write` makes one
for every operation: write batches of thousands with `writePacked`.
- A chunk for `writePacked` is checked for its lengths, not for its column
family handles. A handle that is not of the open database crashes the
process, as it does in `PutCF`.
- A loop over an iterator that is not tail recursive gets slower with every
entry, because the run-time system walks the stack at each foreign call.
Keep what has been read in an argument of the loop.
- `iterChunk` and the `iterEntries` functions leave the iterator on the
first entry that they did not return, not on the last one that they did.
- An `iterChunk` with a `chunkPrefix` still moves the iterator to the first
key outside the prefix, as a loop written by hand does. Where deleted keys
follow the range, the iterator needs a bound as well.
- A column family with merge records must get its `mergeOperator` from
every program that opens the database. Without it RocksDB drops the
records that are still in its write-ahead log, and a read of a key that
has a value under merge records crashes the process: see
[Merge records](#merge-records-change-a-value-without-reading-it).
- A merge operator is called from RocksDB's threads, several at a time. It
must be a C function that keeps no state between calls, and must return
memory from `malloc`.
- Releases before 2.6.2 built no Bloom filters on RocksDB 6.24 or later,
whatever `bloomFilter` said: they passed the bits per key as an integer to a
function that had come to take a `double`. A database written by one of them
has no filters in its table files. After upgrading, files get them as they
are written, and the memory the filters need comes with them. The size of
the filters of a column family is in the `rocksdb.aggregated-table-properties`
property, as `filter block size`.
- The compiler does not check a foreign import against RocksDB's header. A
binding with the wrong type builds and runs, with garbage in place of the
argument; the two fixed in 2.6.2 went unnoticed for years.
## Development
### Stack
```sh
git clone https://github.com/jprupp/rocksdb-haskell.git
cd rocksdb-haskell
stack build # build the library
stack test # run the tests
stack bench # time scans and large batches
stack haddock --no-haddock-deps # build the API documentation
```
With `stack --nix`, the `nix` section of `stack.yaml` provides RocksDB.
`bench/Scan.hs` times a scan of 200,000 entries by single steps and in
chunks, from one thread and from sixteen on four capabilities.
`bench/Batch.hs` times a batch of 65,000 operations written with `write`
and with `writePacked`, alone and next to threads that compute. To see what
a machine with four cores does, confine them: `taskset -c 0-3 stack bench`.
To run one of the two: `stack bench rocksdb-haskell-jprupp:bench:batch`.
### Nix
```sh
nix develop
cabal build all
cabal test all
```
The shell includes GHC 9.8.4, cabal, haskell-language-server, fourmolu, hlint
and hoogle.
`package.yaml` is the source of the package description. Change it rather than
`rocksdb-haskell-jprupp.cabal`, which [hpack](https://github.com/sol/hpack)
generates.
The tests are in `test/Spec.hs` and use [hspec](https://hspec.github.io/) and
QuickCheck. Each test runs against a fresh database in a temporary directory,
so they need RocksDB installed and nothing else. There are two kinds:
- **Examples** pin down one behaviour each: what a function returns, when it
throws, what a setting changes in the files RocksDB writes. A test of a
setting checks an effect that can be observed, such as the size of the
filter blocks, because a wrong binding can do nothing without failing.
- **Properties** apply random writes to the database and to a `Map`, and
require that every way of reading the database agrees with the map: point
reads, `multiGet`, iteration in both directions, seeks, snapshots and
column families. Chunks are held to what single steps read, with random
limits, prefixes and bounds. The keys are short and drawn from the bytes a binding is
most likely to mishandle, zero and 255, so that operations often meet on
the same key.
`destroy` and `repair` have no tests: they take the `Options` of the C API,
which this package gives no way to create.
### Contributing
Pull requests are welcome.
- Format new code with fourmolu, which the Nix shell provides.
- Add tests for the behaviour you change.
- When you add or change a binding in `Database.RocksDB.C`, read the
prototype in `rocksdb/c.h` argument by argument: nothing else checks the
Haskell type against it. If the type of an argument differs between
versions of RocksDB, import the function with `capi`.
- Describe the change in `CHANGELOG.md`. Versions follow the
[Haskell Package Versioning Policy](https://pvp.haskell.org/).
## Related projects
- [rocksdb-query](https://github.com/jprupp/rocksdb-query) stores and queries
Haskell types with these bindings.
- [haskoin-store](https://github.com/jprupp/haskoin-store) is a Bitcoin block
chain indexer built on them.
This library started as a fork of
[leveldb-haskell](https://github.com/kim/leveldb-haskell) and of earlier RocksDB
bindings. See [AUTHORS](https://github.com/jprupp/rocksdb-haskell/blob/master/AUTHORS).
## License
rocksdb-haskell-jprupp is released under the
[BSD 3-Clause License](https://github.com/jprupp/rocksdb-haskell/blob/master/LICENSE).