rocksdb-haskell-jprupp 2.6.0 → 2.6.1
raw patch · 8 files changed
+959/−26 lines, 8 filesPVP ok
version bump matches the API change (PVP)
API changes (from Hackage documentation)
Files
- CHANGELOG.md +4/−0
- README.md +518/−8
- rocksdb-haskell-jprupp.cabal +1/−1
- src/Database/RocksDB.hs +85/−0
- src/Database/RocksDB/Base.hs +114/−14
- src/Database/RocksDB/C.hs +115/−0
- src/Database/RocksDB/Internal.hs +66/−1
- src/Database/RocksDB/Iterator.hs +56/−2
CHANGELOG.md view
@@ -1,3 +1,7 @@+## 2.6.1++ * Documentation, lots of it, with detailed explanations.+ ## 2.6.0 * Add `multiGet` and `multiGetCF`, which read the values of many keys in
README.md view
@@ -1,22 +1,532 @@-This library provides Haskell bindings to-[RocksDB](http://rocksdb.org)+# rocksdb-haskell-jprupp +[](https://hackage.haskell.org/package/rocksdb-haskell-jprupp)+[](https://github.com/jprupp/rocksdb-haskell/blob/master/LICENSE)++Haskell bindings to [RocksDB](https://rocksdb.org), the embedded persistent+key-value store. The package on Hackage is `rocksdb-haskell-jprupp`; the module+to import is `Database.RocksDB`.++The bindings follow the RocksDB C API closely. On top of it they add+bracket-style resource management, so that databases, snapshots and iterators+are always released, even when an exception is thrown.++- [API documentation](https://hackage.haskell.org/package/rocksdb-haskell-jprupp)+- [Changelog](https://github.com/jprupp/rocksdb-haskell/blob/master/CHANGELOG.md)+- [Report a bug](https://github.com/jprupp/rocksdb-haskell/issues)++## Contents++- [Why these bindings](#why-these-bindings)+- [Design principles](#design-principles)+- [Features](#features)+- [Installation](#installation)+- [Quick start](#quick-start)+- [Tutorial: an event log](#tutorial-an-event-log)+- [Concepts](#concepts)+- [Usage guide](#usage-guide)+- [Configuration](#configuration)+- [Module overview](#module-overview)+- [Things to watch out for](#things-to-watch-out-for)+- [Development](#development)+- [Related projects](#related-projects)+- [License](#license)++## Why these bindings++RocksDB is a storage engine: a library that keeps sorted keys and values in+files on local disk, with fast writes, fast point lookups and fast ordered+scans. There is no server to run. It is what you reach for when SQLite's+relational model is more than you need, and an in-memory map is not durable or+big enough.++These bindings exist because the Haskoin Bitcoin indexer needed them. It keeps+the whole Bitcoin block chain indexed by address, transaction and block, which+is hundreds of gigabytes, and serves queries from many threads while new+blocks are written. That use shaped the library:++- it must be safe to share one database between many threads;+- resources must never leak, because the process runs for a long time;+- memory outside the Haskell heap must be boundable, because RocksDB's defaults+ let it grow with the size of the database;+- the hot paths, such as reading many keys at once, must cost as little as the+ C API allows.++The library is a fork of `leveldb-haskell` and of earlier RocksDB bindings,+maintained as `rocksdb-haskell-jprupp`.++## Design principles++- **Stay close to the C API.** Functions map nearly one to one to RocksDB's C+ functions, so RocksDB's own documentation and tuning guides apply directly.+ Conveniences such as typed keys live in separate libraries+ ([rocksdb-query](https://github.com/jprupp/rocksdb-query)).+- **Scope every resource.** Databases, snapshots and iterators are C objects+ that must be freed once and never touched again. `withDB`, `withSnapshot` and+ `withIter` free them when their block ends, even on an exception.+- **Plain `IO`.** No monad transformer constraints. Lift into your own monad+ with `liftIO`.+- **Bytes in, bytes out.** Keys and values are strict `ByteString`s, which is+ what RocksDB stores. How you serialize them is your choice.+- **Don't crash in C.** RocksDB crashes the process if you misuse an iterator.+ The Haskell side checks first and turns those cases into no-ops or+ `Nothing`.+- **Expose what bounds memory.** The few settings that turn RocksDB's memory+ use into a fixed budget are first-class fields of `Config`.++## Features++| Area | What you get |+| -------------- | ------------------------------------------------------------------------------ |+| Basic access | `get`, `put`, `delete` on strict `ByteString` keys and values |+| Batches | Atomic `write` of many puts and deletes, across column families |+| Multi-get | `multiGet` reads many keys in one call to RocksDB's `MultiGet` |+| Column families| Open a database with several independent key spaces; add new ones on open |+| Snapshots | Consistent point-in-time reads across many queries and iterators |+| Iterators | Ordered scans, forwards and backwards, optionally pinned to a snapshot |+| Tuning | Bloom filters, fixed-length prefix extractor, shared LRU block cache, memtable and WAL limits, open file limit |+| Administration | Properties (memory use, statistics, SST files), approximate sizes, destroy, repair |++### What is not included++- No transactions (`TransactionDB`, `OptimisticTransactionDB`), merge+ operators, TTL databases, backups or checkpoints.+- No manual compaction, `DeleteRange` or `SingleDelete`.+- No custom comparators: keys always sort by their bytes.++Pull requests that add bindings for these are welcome.++## Installation++### System library++You need the RocksDB C library and its headers.++```sh+# Debian and Ubuntu+sudo apt-get install librocksdb-dev++# Fedora+sudo dnf install rocksdb-devel++# macOS with Homebrew+brew install rocksdb++# Nix+nix-shell -p rocksdb+```++The library links against `-lrocksdb`. If RocksDB is installed in a+non-standard place, pass its directories with `--extra-include-dirs` and+`--extra-lib-dirs` (Cabal or Stack).++### Check the library is found++```sh+ls /usr/include/rocksdb/c.h # or $(brew --prefix)/include/rocksdb/c.h+pkg-config --modversion rocksdb # prints the installed version, if registered+```++### Step by step with Stack++1. Install [Stack](https://docs.haskellstack.org/) and the RocksDB library+ (above).+2. Create a project:++ ```sh+ stack new rocks-demo simple+ cd rocks-demo+ ```++3. Add `rocksdb-haskell-jprupp`, `bytestring` and `data-default` to+ `build-depends` in `rocks-demo.cabal`.+4. Put the [quick start](#quick-start) program in `src/Main.hs` and run it:++ ```sh+ stack run+ ```++ It creates a database in `/tmp/example` and prints what it reads back.++If the build fails with `cannot find -lrocksdb` or `rocksdb/c.h: No such file`,+the C library is not where the compiler looks. Install the development package,+or pass `--extra-include-dirs` and `--extra-lib-dirs`.++### With Cabal++```cabal+build-depends:+ base+ , bytestring+ , data-default+ , rocksdb-haskell-jprupp+```++`data-default` gives you `def` for the default `Config`.++### With Stack++The package is in Stackage nightly snapshots. If your resolver does not have+it, or has an older version, add it to `stack.yaml`:++```yaml+extra-deps:+ - rocksdb-haskell-jprupp-2.6.0+```++With `stack --nix`, add RocksDB to the Nix packages:++```yaml+nix:+ packages:+ - rocksdb+```++## Quick start++```haskell+{-# LANGUAGE LambdaCase #-}+{-# LANGUAGE OverloadedStrings #-}++import Data.Default (def)+import Database.RocksDB++main :: IO ()+main = do+ let cfg = def {createIfMissing = True, bloomFilter = True}+ withDB "/tmp/example" cfg $ \db -> do+ put db "hello" "world"+ get db "hello" >>= print -- Just "world"++ -- atomic batch+ write db [Put "a" "1", Put "b" "2", Del "hello"]++ -- many keys at once+ multiGet db ["a", "b", "c"] >>= print -- [Just "1",Just "2",Nothing]++ -- every entry, in key order+ withIter db $ \it -> do+ iterFirst it+ let loop =+ iterEntry it >>= \case+ Nothing -> return ()+ Just kv -> print kv >> iterNext it >> loop+ loop+```++## Tutorial: an event log++This builds a small append-only event log, to show how a RocksDB schema is+designed around sorted keys.++### 1. Design the keys++RocksDB sorts keys by their bytes. Records read together should therefore sort+together. Events are read per user and in time order, so the key is the user+followed by a big-endian timestamp, which sorts numerically:++```haskell+{-# LANGUAGE OverloadedStrings #-}++import Data.ByteString (ByteString)+import qualified Data.ByteString as B+import qualified Data.ByteString.Builder as BB+import qualified Data.ByteString.Lazy as BL+import Data.Word (Word64)++eventKey :: ByteString -> Word64 -> ByteString+eventKey user time =+ BL.toStrict . BB.toLazyByteString $+ BB.byteString user <> BB.char7 '/' <> BB.word64BE time+```++### 2. Write atomically++An event and a per-user counter must stay consistent, so they go in one batch:++```haskell+import Database.RocksDB++logEvent :: DB -> ByteString -> Word64 -> ByteString -> Int -> IO ()+logEvent db user time payload count =+ write db+ [ Put (eventKey user time) payload,+ Put ("count/" <> user) (BL.toStrict (BB.toLazyByteString (BB.intDec count)))+ ]+```++If the program dies halfway, either both writes are there or neither is.++### 3. Read a range with an iterator++All events of a user sort together, so one seek and a short scan read them:++```haskell+eventsOf :: DB -> ByteString -> IO [ByteString]+eventsOf db user = withIter db $ \it -> do+ let prefix = user <> "/"+ iterSeek it prefix+ let go acc = do+ mkv <- iterEntry it+ case mkv of+ Just (k, v) | prefix `B.isPrefixOf` k -> iterNext it >> go (v : acc)+ _ -> return (reverse acc)+ go []+```++### 4. Read consistently with a snapshot++If another thread logs events while you read the counter and then the events,+the two can disagree. Read both through a snapshot:++```haskell+report :: DB -> ByteString -> IO (Maybe ByteString, [ByteString])+report db user = withSnapshot db $ \snap ->+ (,) <$> get snap ("count/" <> user) <*> eventsOf snap user+```++### 5. Tune for size++Once the log holds millions of events, open it with a memory budget and a Bloom+filter for faster point lookups:++```haskell+let cfg =+ def+ { createIfMissing = True,+ bloomFilter = True,+ maxFiles = Just 1000,+ blockCacheSize = Just (512 * 1024 * 1024),+ cacheIndexAndFilterBlocks = True+ }+```++## Concepts++### The database handle++`withDB` and `withDBCF` open a database, pass a `DB` to your function, and close+it when the function returns or throws. The `DB` must not escape that function.+It is safe to use from many threads at the same time.++### Keys and values++Keys and values are strict `ByteString`s. RocksDB sorts keys by their bytes,+which is the order iterators walk them in. To store Haskell types, serialize+them yourself, or use [rocksdb-query](https://github.com/jprupp/rocksdb-query).++### Column families++A database can hold several key spaces called column families, each with its+own configuration. Functions ending in `CF` act on a given column family; the+others act on the default one. The handles are in `columnFamilies db`, in the+order you listed them when opening.++### Snapshots++`withSnapshot db f` passes `f` a `DB` that reads from a frozen view of the+database. Reads and iterators through it do not see later writes. Writes+through it still go to the live database.++### Errors++Failures reported by RocksDB are thrown as `IOError` (built with `userError`).+The message starts with the operation, such as `put: ...` or `open: ...`.++## Usage guide++### Column families++```haskell+withDBCF "/tmp/example-cf" cfg [("users", cfg), ("orders", cfg)] $ \db -> do+ let [users, orders] = columnFamilies db+ putCF db users "alice" "..."+ getCF db orders "1234" >>= print+ write db [PutCF users "bob" "...", DelCF orders "1234"]+```++On an empty directory, `withDBCF` creates the database and all the column+families. On an existing database, RocksDB needs every column family that+exists to be listed. To add a new one, list it and set+`createMissingColumnFamilies = True` in the database `Config`.++### Snapshots++```haskell+withSnapshot db $ \snap -> do+ a <- get snap "a"+ b <- get snap "b" -- same point in time as "a"+ withIter snap $ \it -> iterFirst it >> iterEntry it >>= print+```++When the snapshot's lifetime does not fit a scope, use `createSnapshot` and+`releaseSnapshot`. `withIterSnap` and `withIterSnapCF` iterate over a raw+`Snapshot`.++### Iterators++An iterator starts out unpositioned. Move it with `iterFirst`, `iterLast` or+`iterSeek`, then step with `iterNext` and `iterPrev`. `iterKey`, `iterValue` and+`iterEntry` return `Nothing` once it moves past either end.++```haskell+-- every key that starts with "user:"+withIter db $ \it -> do+ iterSeek it "user:"+ let go acc =+ iterEntry it >>= \case+ Just (k, v) | "user:" `B.isPrefixOf` k -> iterNext it >> go ((k, v) : acc)+ _ -> return (reverse acc)+ users <- go []+ iterGetError it >>= mapM_ (fail . show) -- tell the end apart from an error+ return users+```++`createIterator`, `createIteratorSnap` and `destroyIterator` manage iterators+by hand.++### Properties and maintenance++```haskell+getProperty db Stats >>= mapM_ BS.putStr+getProperty db BlockCacheUsage+getPropertyCF db cf CurSizeAllMemTables+approximateSize db ("a", "z")+```++`destroy` deletes a closed database and `repair` tries to recover a damaged one.+Both take a raw `Options` pointer, which the public API does not let you build+yet, so in practice they are only usable from inside the library.++## Configuration++`Config` has a `Default` instance: all flags off and all sizes left to RocksDB.++| Field | Scope | Meaning |+| ----------------------------- | --------------- | --------------------------------------------------------------- |+| `createIfMissing` | database | Create the database if it does not exist |+| `errorIfExists` | database | Fail if the database already exists |+| `paranoidChecks` | database | Check data aggressively and stop on corruption |+| `createMissingColumnFamilies` | database | Create column families given to `withDBCF` that are missing |+| `maxFiles` | database | Maximum number of open files (default unlimited) |+| `dbWriteBufferSize` | database | Bytes of all memtables together (default unlimited) |+| `blockCacheSize` | database | Bytes of an LRU block cache shared by all column families |+| `maxTotalWalSize` | database | Bytes of write-ahead log that trigger flushes |+| `writeBufferSize` | column family | Bytes per memtable (RocksDB default 64 MiB) |+| `bloomFilter` | column family | Full Bloom filter with 10 bits per key |+| `prefixLength` | column family | Length of a fixed key prefix for the prefix extractor |+| `cacheIndexAndFilterBlocks` | column family | Keep index and filter blocks in the block cache |++Database-wide fields are ignored in the configuration of a column family.++### Choosing a configuration++RocksDB's defaults are tuned for small databases. Its memory use outside the+Haskell heap comes from three places:++1. **Memtables**, where recent writes collect before being flushed to disk.+ Capped per column family by `writeBufferSize` and in total by+ `dbWriteBufferSize`.+2. **Index and filter blocks** of every open table file. Without+ `cacheIndexAndFilterBlocks` they live outside any cache and grow with the+ number of files; `maxFiles` limits how many are open.+3. **The block cache** of recently read data. `blockCacheSize` makes one cache+ shared by all column families, so memory goes where reads are.++A reasonable starting point for a large database is a block cache of a few+hundred megabytes to a few gigabytes, `cacheIndexAndFilterBlocks = True`,+`maxFiles` of a few thousand, and `dbWriteBufferSize` of a few hundred+megabytes. Enable `bloomFilter` when most reads are point lookups of keys that+may not exist.++For a large database, setting `maxFiles`, `blockCacheSize`,+`cacheIndexAndFilterBlocks` and `dbWriteBufferSize` puts a ceiling on the memory+RocksDB uses outside the Haskell heap. Without them, index and filter blocks of+every open file and every memtable can grow without bound.++## Module overview++`Database.RocksDB` is the only exposed module. It re-exports:++| Part | Contents |+| ------------------------ | -------------------------------------------------------------- |+| Types | `DB`, `Config`, `BatchOp`, `ColumnFamily`, `Snapshot`, `Iterator`, `Property`, `Range` |+| Opening | `withDB`, `withDBCF` |+| Reading and writing | `get`, `getCF`, `multiGet`, `multiGetCF`, `put`, `putCF`, `delete`, `deleteCF`, `write` |+| Snapshots | `withSnapshot`, `createSnapshot`, `releaseSnapshot` |+| Iteration | `withIter`, `withIterCF`, `withIterSnap`, `withIterSnapCF`, `iterSeek`, `iterFirst`, `iterLast`, `iterNext`, `iterPrev`, `iterKey`, `iterValue`, `iterEntry`, `iterValid`, `iterGetError` |+| Administration | `getProperty`, `getPropertyCF`, `approximateSize`, `destroy`, `repair` |++The FFI bindings (`Database.RocksDB.C`) and helpers (`Database.RocksDB.Internal`)+are hidden modules.++## Things to watch out for++- Do not use a `DB`, snapshot or iterator after the function that gave it to you+ has returned. The C object is gone and RocksDB will crash.+- Only one process can open a database at a time. A second open fails with a+ lock error.+- Moving an invalid iterator does nothing. Reposition it with `iterSeek`,+ `iterFirst` or `iterLast`.+- An iterator that becomes invalid because of corruption looks like one that+ reached the end. Check `iterGetError`.+- Calls into RocksDB are `safe` foreign calls, so they do not block other+ Haskell threads, but each one costs a little more than an `unsafe` call.+ ## Development ### Stack -```bash-stack build-stack test+```sh+git clone https://github.com/jprupp/rocksdb-haskell.git+cd rocksdb-haskell++stack build # build the library+stack test # run the tests+stack haddock --no-haddock-deps # build the API documentation ``` +With `stack --nix`, the `nix` section of `stack.yaml` provides RocksDB.+ ### Nix -```bash+```sh nix develop cabal build all cabal test all ``` -The shell includes GHC 9.8.4, cabal, haskell-language-server, fourmolu,-hlint, and hoogle.+The shell includes GHC 9.8.4, cabal, haskell-language-server, fourmolu, hlint+and hoogle.++`package.yaml` is the source of the package description. Change it rather than+`rocksdb-haskell-jprupp.cabal`, which [hpack](https://github.com/sol/hpack)+generates.++The tests are in `test/Spec.hs` and use [hspec](https://hspec.github.io/) and+QuickCheck. Each test runs against a fresh database in a temporary directory.++### Contributing++Pull requests are welcome.++- Format new code with fourmolu, which the Nix shell provides.+- Add tests for the behaviour you change.+- Describe the change in `CHANGELOG.md`. Versions follow the+ [Haskell Package Versioning Policy](https://pvp.haskell.org/).++## Related projects++- [rocksdb-query](https://github.com/jprupp/rocksdb-query) stores and queries+ Haskell types with these bindings.+- [haskoin-store](https://github.com/jprupp/haskoin-store) is a Bitcoin block+ chain indexer built on them.++This library started as a fork of+[leveldb-haskell](https://github.com/kim/leveldb-haskell) and of earlier RocksDB+bindings. See [AUTHORS](https://github.com/jprupp/rocksdb-haskell/blob/master/AUTHORS).++## License++rocksdb-haskell-jprupp is released under the+[BSD 3-Clause License](https://github.com/jprupp/rocksdb-haskell/blob/master/LICENSE).
rocksdb-haskell-jprupp.cabal view
@@ -5,7 +5,7 @@ -- see: https://github.com/sol/hpack name: rocksdb-haskell-jprupp-version: 2.6.0+version: 2.6.1 synopsis: Haskell bindings for RocksDB description: See README at <https://github.com/jprupp/rocksdb-haskell#readme> category: Database, FFI
src/Database/RocksDB.hs view
@@ -1,3 +1,88 @@+-- |+-- Module : Database.RocksDB+-- Copyright : (c) 2012-2013 The leveldb-haskell Authors+-- (c) 2014-2020 The rocksdb-haskell Authors+-- (c) 2025 JP Rupp+-- License : BSD3+-- Maintainer : jprupp@protonmail.ch+-- Stability : experimental+-- Portability : non-portable+--+-- Haskell bindings to <https://rocksdb.org RocksDB>, an embedded persistent+-- key-value store.+--+-- = Purpose+--+-- RocksDB is a fast embedded key-value store: a library, not a server, that+-- keeps sorted keys and values in files on local disk. These bindings let a+-- Haskell program use it as its storage engine. They were written for, and are+-- used by, the Haskoin Bitcoin indexer, which keeps hundreds of+-- gigabytes in RocksDB and reads them concurrently from many threads.+--+-- = Design+--+-- * __Close to the C API.__ The functions map almost one to one to RocksDB's C+-- functions, so the RocksDB documentation applies directly and nothing is+-- hidden behind an abstraction you would have to learn separately. Higher+-- level conveniences, such as typed keys and values, belong in libraries on+-- top, like @rocksdb-query@.+--+-- * __Resources are scoped.__ A database, a snapshot and an iterator are C+-- objects that must be freed exactly once and never used afterwards. Every+-- such resource has a @with@ function that frees it when the scope ends, even+-- on an exception. Manual @create@ and @destroy@ functions exist for the cases+-- where a lifetime cannot be a scope.+--+-- * __Plain 'IO'.__ Every function runs in 'IO'. Lifting into an application+-- monad is left to the caller, which keeps the bindings free of monad+-- transformer dependencies and keeps their types simple.+--+-- * __Bytes in, bytes out.__ Keys and values are strict+-- 'Data.ByteString.ByteString's, the representation RocksDB itself uses.+-- Serialization is a separate concern.+--+-- * __Memory you can bound.__ RocksDB keeps a lot of memory outside the Haskell+-- heap. 'Config' exposes the settings that put a ceiling on it, because an+-- indexer that grows until the machine runs out of memory is not usable.+--+-- This is the only module that needs to be imported. It re-exports+-- "Database.RocksDB.Base", which in turn re-exports the iterator functions.+--+-- = Quick start+--+-- @+-- -- with OverloadedStrings+-- import Data.Default (def)+-- import Database.RocksDB+--+-- main :: IO ()+-- main =+-- 'withDB' "\/tmp\/example" def{'createIfMissing' = True} $ \\db -> do+-- 'put' db "hello" "world"+-- 'get' db "hello" >>= print -- Just "world"+-- 'write' db ['Put' "a" "1", 'Put' "b" "2", 'Del' "hello"]+-- 'withIter' db $ \\it -> do+-- 'iterFirst' it+-- 'iterEntry' it >>= print -- Just ("a","1")+-- @+--+-- = Keys and values+--+-- Keys and values are strict 'Data.ByteString.ByteString's. RocksDB orders keys+-- lexicographically by their bytes, which is the order used by iterators.+--+-- = Column families+--+-- A database can hold several independent key spaces called column families.+-- Open such a database with 'withDBCF'. The handles for the extra column+-- families are found in 'columnFamilies', in the same order as they were passed+-- to 'withDBCF'. Functions ending in @CF@ act on a given column family; the+-- others act on the default column family.+--+-- = Errors+--+-- Errors reported by RocksDB are thrown as 'IOError' exceptions created with+-- 'userError', with a message that names the failing operation. module Database.RocksDB (module Base) where import Database.RocksDB.Base as Base
src/Database/RocksDB/Base.hs view
@@ -10,7 +10,14 @@ -- -- RocksDB Haskell binding. ----- The API closely follows the C-API of RocksDB.+-- The API closely follows the C-API of RocksDB. All resources (databases,+-- snapshots, iterators) are managed with bracket-style functions such as+-- 'withDB', 'withSnapshot' and 'withIter', which release them when the+-- supplied action returns or throws. Manual variants such as 'createSnapshot'+-- and 'createIterator' exist for cases where the lifetime of a resource cannot+-- be expressed as a scope.+--+-- A 'DB' handle is safe to use from many threads at once. module Database.RocksDB.Base ( -- * Exported Types@@ -69,10 +76,15 @@ import Foreign.Storable import System.Directory --- | Properties exposed by RocksDB+-- | Properties exposed by RocksDB. Read them with 'getProperty' or+-- 'getPropertyCF'. Every value is returned by RocksDB as text. data Property+ -- | @rocksdb.num-files-at-level\<N\>@: number of SST files at level @N@ = NumFilesAtLevel Int+ -- | @rocksdb.stats@: multi-line human-readable statistics | Stats+ -- | @rocksdb.sstables@: description of the SST files that make up the+ -- database | SSTables -- | @rocksdb.estimate-table-readers-mem@: memory for index and filter -- blocks of open table files, outside the block cache@@ -85,16 +97,36 @@ | PropertyName String deriving (Eq, Show) +-- | A single operation in an atomic batch written with 'write'.+--+-- Batches are how a program keeps several keys consistent with each other: an+-- index and the data it points to, for example, must change together or not at+-- all. RocksDB applies a batch atomically and in one write to its log, so a batch+-- is also much faster than the same writes made one by one. data BatchOp = Put !ByteString !ByteString+ -- ^ set a key to a value in the default column family | Del !ByteString+ -- ^ delete a key from the default column family | PutCF !ColumnFamily !ByteString !ByteString+ -- ^ set a key to a value in a column family | DelCF !ColumnFamily !ByteString+ -- ^ delete a key from a column family deriving (Eq, Show) --- | Open a database.+-- | Open a database at the given path and pass its handle to a function. ----- The returned handle will be automatically released with 'close'--- when the function exits.+-- The handle is closed automatically when the function returns or throws, and+-- must not be used afterwards. If 'createIfMissing' is set, the directory is+-- created first if needed. Throws an 'IOError' if the database cannot be+-- opened, for example if it does not exist and 'createIfMissing' is not set,+-- or if it is locked by another process.+--+-- Only the default column family is opened. A database that has other column+-- families must be opened with 'withDBCF'.+--+-- Opening is expensive and a database can only be opened by one process at a+-- time, so open it once at program start and share the 'DB' between threads for+-- the life of the program. withDB :: FilePath -> Config -> (DB -> IO a) -> IO a withDB path config f = withBlockCache config $ \cache ->@@ -116,6 +148,25 @@ , writeOpts = write_opts } +-- | Open a database with column families and pass its handle to a function.+--+-- The second argument configures the database and its default column family.+-- The third argument lists the other column families by name, each with its own+-- 'Config'. Their handles are stored in 'columnFamilies', in the same order.+--+-- If the database directory is empty, a new database is created along with all+-- the listed column families. Otherwise the existing database is opened with+-- those column families; if some of them do not exist yet, the open fails unless+-- 'createMissingColumnFamilies' is set in the database 'Config'. RocksDB also+-- requires every column family that exists in the database to be listed.+--+-- The handle and all column family handles are released when the function+-- returns or throws.+--+-- Column families are separate key spaces in one database. They share the+-- write-ahead log, so a batch can change several of them atomically, but each+-- has its own files, memtables and tuning. Use them to keep data of different+-- shapes apart, such as an index with short keys and a table of large values. withDBCF :: FilePath -> Config -> [(String, Config)]@@ -197,6 +248,16 @@ -- | Run an action with a snapshot of the database. -- The 'DB' object is not valid after the action ends.+--+-- The 'DB' passed to the action reads from a consistent point-in-time view of+-- the database: 'get', 'multiGet' and iterators created from it do not see+-- writes made after the snapshot was taken. Writes through it still go to the+-- live database.+--+-- Without a snapshot, two reads made one after the other can see different+-- states of the database if another thread writes in between. Anything that+-- reads several related keys and expects them to agree, such as an index and the+-- records it points to, should read through a snapshot. withSnapshot :: DB -> (DB -> IO a) -> IO a withSnapshot db f = bracket (createSnapshot db) releaseSnapshot (f . fst)@@ -217,7 +278,8 @@ destroyReadOpts read_opts c_rocksdb_release_snapshot db_ptr snap_ptr --- | Get a DB property.+-- | Get a property of the default column family, or of the database as a+-- whole. Returns 'Nothing' if RocksDB does not know the property. getProperty :: DB -> Property -> IO (Maybe ByteString) getProperty DB{rocksDB = db_ptr} p = withCString (propertyName p) $ \name ->@@ -230,6 +292,8 @@ withCString (propertyName p) $ \name -> c_rocksdb_property_value_cf db_ptr cf name >>= propertyResult +-- | Turn a property value returned by RocksDB into a 'ByteString', freeing the+-- C string. propertyResult :: CString -> IO (Maybe ByteString) propertyResult val_ptr | val_ptr == nullPtr = return Nothing@@ -238,6 +302,7 @@ freeCString val_ptr return (Just res) +-- | The name RocksDB uses for a property. propertyName :: Property -> String propertyName = \case NumFilesAtLevel i -> "rocksdb.num-files-at-level" ++ show i@@ -248,23 +313,31 @@ BlockCacheUsage -> "rocksdb.block-cache-usage" PropertyName n -> n --- | Destroy the given RocksDB database.+-- | Destroy the given RocksDB database, deleting all of its files. The+-- database must not be open. Throws an 'IOError' on failure. destroy :: FilePath -> Options -> IO () destroy path opts_ptr = withCString path $ \path_ptr -> throwIfErr "destroy" $ c_rocksdb_destroy_db opts_ptr path_ptr --- | Repair the given RocksDB database.+-- | Try to recover as much data as possible from a corrupted database. Some+-- data may be lost. The database must not be open. Throws an 'IOError' on+-- failure. repair :: FilePath -> Options -> IO () repair path opts_ptr = withCString path $ \path_ptr -> throwIfErr "repair" $ c_rocksdb_repair_db opts_ptr path_ptr +-- | A key range given as @(start, limit)@. The start key is included and the+-- limit key is excluded.+-- -- TODO: support [Range], like C API does type Range = (ByteString, ByteString) --- | Inspect the approximate sizes of the different levels.+-- | Approximate number of bytes of file system space used by the keys in a+-- 'Range' of the default column family. Data still in memtables may not be+-- counted. approximateSize :: DB -> Range -> IO Int64 approximateSize DB{rocksDB = db_ptr} (from, to) = BU.unsafeUseAsCStringLen from $ \(from_ptr, flen) ->@@ -283,13 +356,17 @@ where toInt64 = return . fromIntegral --- | Write a key/value pair.+-- | Write a key/value pair to the default column family, replacing any+-- previous value. Throws an 'IOError' on failure. put :: DB -> ByteString -> ByteString -> IO () put db = putCommon db Nothing +-- | Like 'put', but write to the given column family. putCF :: DB -> ColumnFamily -> ByteString -> ByteString -> IO () putCF db cf = putCommon db (Just cf) +-- | Write a key/value pair to the default column family, or to a column family+-- if one is given. putCommon :: DB -> Maybe ColumnFamily -> ByteString -> ByteString -> IO () putCommon DB{rocksDB = db_ptr, writeOpts = write_opts} mcf key value = BU.unsafeUseAsCStringLen key $ \(key_ptr, klen) ->@@ -304,13 +381,17 @@ key_ptr (intToCSize klen) val_ptr (intToCSize vlen) --- | Read a value by key.+-- | Read a value by key from the default column family. Returns 'Nothing' if+-- the key is not found. Throws an 'IOError' on failure. get :: DB -> ByteString -> IO (Maybe ByteString) get db = getCommon db Nothing +-- | Like 'get', but read from the given column family. getCF :: DB -> ColumnFamily -> ByteString -> IO (Maybe ByteString) getCF db cf = getCommon db (Just cf) +-- | Read a value from the default column family, or from a column family if one+-- is given. getCommon :: DB -> Maybe ColumnFamily -> ByteString -> IO (Maybe ByteString) getCommon DB{rocksDB = db_ptr, readOpts = read_opts} mcf key = BU.unsafeUseAsCStringLen key $ \(key_ptr, klen) ->@@ -333,7 +414,12 @@ -- | Read the values of many keys in one call. RocksDB looks the keys up -- together, sharing the work of finding the files and blocks that hold -- them, which is faster than reading them one by one. The values are in the--- order of the keys.+-- order of the keys, with 'Nothing' for keys that are not found. Duplicate keys+-- are allowed. Throws an 'IOError' if any of the lookups fails.+--+-- For a query that needs hundreds of keys at once, such as the balances of many+-- addresses, this cuts the time spent finding blocks and crossing into C to a+-- fraction of separate 'get' calls. multiGet :: DB -> [ByteString] -> IO [Maybe ByteString] multiGet db = multiGetCommon db Nothing @@ -341,6 +427,8 @@ multiGetCF :: DB -> ColumnFamily -> [ByteString] -> IO [Maybe ByteString] multiGetCF db cf = multiGetCommon db (Just cf) +-- | Read many values from the default column family, or from a column family if+-- one is given. multiGetCommon :: DB -> Maybe ColumnFamily -> [ByteString] -> IO [Maybe ByteString] multiGetCommon _ _ [] = return [] multiGetCommon DB{rocksDB = db_ptr, readOpts = read_opts} mcf keys =@@ -380,13 +468,17 @@ m : _ -> throwIO $ userError $ "multi_get: " ++ m [] -> return res +-- | Delete a key from the default column family. Deleting a key that does not+-- exist is not an error. Throws an 'IOError' on failure. delete :: DB -> ByteString -> IO () delete db = deleteCommon db Nothing +-- | Like 'delete', but delete from the given column family. deleteCF :: DB -> ColumnFamily -> ByteString -> IO () deleteCF db cf = deleteCommon db (Just cf) --- | Delete a key/value pair.+-- | Delete a key/value pair from the default column family, or from a column+-- family if one is given. deleteCommon :: DB -> Maybe ColumnFamily -> ByteString -> IO () deleteCommon DB{rocksDB = db_ptr, writeOpts = write_opts} mcf key = BU.unsafeUseAsCStringLen key $ \(key_ptr, klen) ->@@ -394,7 +486,13 @@ Just cf -> c_rocksdb_delete_cf db_ptr write_opts cf key_ptr (intToCSize klen) Nothing -> c_rocksdb_delete db_ptr write_opts key_ptr (intToCSize klen) --- | Perform a batch mutation.+-- | Perform a batch mutation. All operations in the list are applied atomically:+-- either all of them are written or none are. They are applied in order, so a+-- later operation on a key overrides an earlier one. Throws an 'IOError' on+-- failure.+--+-- The keys and values are kept alive until RocksDB has copied them, because+-- they are handed to C as pointers into Haskell memory. write :: DB -> [BatchOp] -> IO () write DB{rocksDB = db_ptr, writeOpts = write_opts} batch = bracket@@ -443,6 +541,8 @@ touch (DelCF _ (PS p _ _)) = touchForeignPtr p +-- | Marshal a list of strings into C strings that are valid while the function+-- runs. withStrings :: [String] -> ([CString] -> IO a) -> IO a withStrings ss f = go [] ss
src/Database/RocksDB/C.hs view
@@ -9,6 +9,18 @@ -- Stability : experimental -- Portability : non-portable --+-- Raw FFI bindings to the RocksDB C API declared in @rocksdb/c.h@. This module+-- is internal; use "Database.RocksDB" instead.+--+-- Each @c_rocksdb_*@ function is a direct binding to the C function of the same+-- name without the @c_@ prefix. Functions that take an 'ErrPtr' report failure+-- by storing a @malloc@-allocated error message in it, which the caller must+-- free. See 'Database.RocksDB.Internal.throwIfErr'.+--+-- The bindings are @safe@ foreign calls. RocksDB calls can take a long time,+-- for example when they wait for disk or for a compaction to make room, and a+-- @safe@ call lets other Haskell threads keep running meanwhile. The small+-- extra cost per call is worth that in a concurrent program. module Database.RocksDB.C ( RocksDB@@ -105,41 +117,72 @@ import Foreign.C.String import Foreign.C.Types +-- | Opaque C database object. data LRocksDB+-- | Opaque C column family handle. data LColumnFamily+-- | Opaque C iterator. data LIterator+-- | Opaque C database or column family options. data LOptions+-- | Opaque C block-based table options. data LBlockBasedOptions+-- | Opaque C read options. data LReadOpts+-- | Opaque C snapshot. data LSnapshot+-- | Opaque C write batch. data LWriteBatch+-- | Opaque C write options. data LWriteOpts+-- | Opaque C filter policy. data LFilterPolicy+-- | Opaque C slice transform (prefix extractor). data LSliceTransform+-- | Opaque C block cache. data LCache +-- | Pointer to an open database. type RocksDB = Ptr LRocksDB+-- | Pointer to a column family handle. type ColumnFamily = Ptr LColumnFamily+-- | Pointer to database or column family options. type Options = Ptr LOptions+-- | Pointer to a batch of writes applied atomically. type WriteBatch = Ptr LWriteBatch+-- | Pointer to a key transformation, used to extract prefixes. type SliceTransform = Ptr LSliceTransform+-- | Pointer to options for reads and iterators. type ReadOpts = Ptr LReadOpts+-- | Pointer to options for writes. type WriteOpts = Ptr LWriteOpts+-- | Pointer to a consistent point-in-time view of a database. type Snapshot = Ptr LSnapshot+-- | Pointer to an iterator over keys. type Iterator = Ptr LIterator+-- | Pointer to a filter policy, such as a bloom filter. type FilterPolicy = Ptr LFilterPolicy+-- | Pointer to block-based table options. type BlockBasedOptions = Ptr LBlockBasedOptions+-- | Pointer to a block cache. type Cache = Ptr LCache +-- | Pointer to a location where a C function stores an error message, or leaves @NULL@ on success. type ErrPtr = Ptr CString+-- | Path of a database, as a C string. type DBName = CString+-- | Name of a column family, as a C string. type CFName = CString+-- | Pointer to the bytes of a key. Its length is passed separately. type Key = CString+-- | Pointer to the bytes of a value. Its length is passed separately. type Val = CString +-- | Binding to @rocksdb_open@. foreign import ccall safe "rocksdb/c.h rocksdb_open" c_rocksdb_open :: Options -> DBName -> ErrPtr -> IO RocksDB +-- | Binding to @rocksdb_open_column_families@. foreign import ccall safe "rocksdb/c.h rocksdb_open_column_families" c_rocksdb_open_column_families :: Options -> DBName@@ -150,9 +193,11 @@ -> ErrPtr -> IO RocksDB +-- | Binding to @rocksdb_close@. foreign import ccall safe "rocksdb/c.h rocksdb_close" c_rocksdb_close :: RocksDB -> IO () +-- | Binding to @rocksdb_create_column_family@. foreign import ccall safe "rocksdb/c.h rocksdb_create_column_family" c_rocksdb_create_column_family :: RocksDB -> Options@@ -160,15 +205,18 @@ -> ErrPtr -> IO ColumnFamily +-- | Binding to @rocksdb_drop_column_family@. foreign import ccall safe "rocksdb/c.h rocksdb_drop_column_family" c_rocksdb_drop_column_family :: RocksDB -> ColumnFamily -> ErrPtr -> IO () +-- | Binding to @rocksdb_column_family_handle_destroy@. foreign import ccall safe "rocksdb/c.h rocksdb_column_family_handle_destroy" c_rocksdb_column_family_handle_destroy :: ColumnFamily -> IO () +-- | Binding to @rocksdb_put@. foreign import ccall safe "rocksdb/c.h rocksdb_put" c_rocksdb_put :: RocksDB -> WriteOpts@@ -177,6 +225,7 @@ -> ErrPtr -> IO () +-- | Binding to @rocksdb_put_cf@. foreign import ccall safe "rocksdb/c.h rocksdb_put_cf" c_rocksdb_put_cf :: RocksDB -> WriteOpts@@ -186,6 +235,7 @@ -> ErrPtr -> IO () +-- | Binding to @rocksdb_delete@. foreign import ccall safe "rocksdb/c.h rocksdb_delete" c_rocksdb_delete :: RocksDB -> WriteOpts@@ -193,6 +243,7 @@ -> ErrPtr -> IO () +-- | Binding to @rocksdb_delete_cf@. foreign import ccall safe "rocksdb/c.h rocksdb_delete_cf" c_rocksdb_delete_cf :: RocksDB -> WriteOpts@@ -201,6 +252,7 @@ -> ErrPtr -> IO () +-- | Binding to @rocksdb_write@. foreign import ccall safe "rocksdb/c.h rocksdb_write" c_rocksdb_write :: RocksDB -> WriteOpts@@ -210,6 +262,8 @@ -- | Returns NULL if not found. A malloc()ed array otherwise. -- Stores the length of the array in *vallen.+--+-- Binding to @rocksdb_get@. foreign import ccall safe "rocksdb/c.h rocksdb_get" c_rocksdb_get :: RocksDB -> ReadOpts@@ -218,6 +272,7 @@ -> ErrPtr -> IO CString +-- | Binding to @rocksdb_get_cf@. foreign import ccall safe "rocksdb/c.h rocksdb_get_cf" c_rocksdb_get_cf :: RocksDB -> ReadOpts@@ -227,6 +282,7 @@ -> ErrPtr -> IO CString +-- | Binding to @rocksdb_multi_get@. foreign import ccall safe "rocksdb/c.h rocksdb_multi_get" c_rocksdb_multi_get :: RocksDB -> ReadOpts@@ -238,6 +294,7 @@ -> ErrPtr -> IO () +-- | Binding to @rocksdb_multi_get_cf@. foreign import ccall safe "rocksdb/c.h rocksdb_multi_get_cf" c_rocksdb_multi_get_cf :: RocksDB -> ReadOpts@@ -250,20 +307,26 @@ -> ErrPtr -> IO () +-- | Binding to @rocksdb_create_snapshot@. foreign import ccall safe "rocksdb/c.h rocksdb_create_snapshot" c_rocksdb_create_snapshot :: RocksDB -> IO Snapshot +-- | Binding to @rocksdb_release_snapshot@. foreign import ccall safe "rocksdb/c.h rocksdb_release_snapshot" c_rocksdb_release_snapshot :: RocksDB -> Snapshot -> IO () -- | Returns NULL if property name is unknown. Else returns a pointer to a -- malloc()-ed null-terminated value.+--+-- Binding to @rocksdb_property_value@. foreign import ccall safe "rocksdb/c.h rocksdb_property_value" c_rocksdb_property_value :: RocksDB -> CString -> IO CString +-- | Binding to @rocksdb_property_value_cf@. foreign import ccall safe "rocksdb/c.h rocksdb_property_value_cf" c_rocksdb_property_value_cf :: RocksDB -> ColumnFamily -> CString -> IO CString +-- | Binding to @rocksdb_approximate_sizes@. foreign import ccall safe "rocksdb/c.h rocksdb_approximate_sizes" c_rocksdb_approximate_sizes :: RocksDB -> CInt@@ -276,9 +339,11 @@ -- ^ array of approx. sizes of ranges -> IO () +-- | Binding to @rocksdb_destroy_db@. foreign import ccall safe "rocksdb/c.h rocksdb_destroy_db" c_rocksdb_destroy_db :: Options -> DBName -> ErrPtr -> IO () +-- | Binding to @rocksdb_repair_db@. foreign import ccall safe "rocksdb/c.h rocksdb_repair_db" c_rocksdb_repair_db :: Options -> DBName -> ErrPtr -> IO () @@ -287,44 +352,56 @@ -- Iterator -- +-- | Binding to @rocksdb_create_iterator@. foreign import ccall safe "rocksdb/c.h rocksdb_create_iterator" c_rocksdb_create_iterator :: RocksDB -> ReadOpts -> IO Iterator +-- | Binding to @rocksdb_create_iterator_cf@. foreign import ccall safe "rocksdb/c.h rocksdb_create_iterator_cf" c_rocksdb_create_iterator_cf :: RocksDB -> ReadOpts -> ColumnFamily -> IO Iterator +-- | Binding to @rocksdb_iter_destroy@. foreign import ccall safe "rocksdb/c.h rocksdb_iter_destroy" c_rocksdb_iter_destroy :: Iterator -> IO () +-- | Binding to @rocksdb_iter_valid@. foreign import ccall safe "rocksdb/c.h rocksdb_iter_valid" c_rocksdb_iter_valid :: Iterator -> IO CUChar +-- | Binding to @rocksdb_iter_seek_to_first@. foreign import ccall safe "rocksdb/c.h rocksdb_iter_seek_to_first" c_rocksdb_iter_seek_to_first :: Iterator -> IO () +-- | Binding to @rocksdb_iter_seek_to_last@. foreign import ccall safe "rocksdb/c.h rocksdb_iter_seek_to_last" c_rocksdb_iter_seek_to_last :: Iterator -> IO () +-- | Binding to @rocksdb_iter_seek@. foreign import ccall safe "rocksdb/c.h rocksdb_iter_seek" c_rocksdb_iter_seek :: Iterator -> Key -> CSize -> IO () +-- | Binding to @rocksdb_iter_next@. foreign import ccall safe "rocksdb/c.h rocksdb_iter_next" c_rocksdb_iter_next :: Iterator -> IO () +-- | Binding to @rocksdb_iter_prev@. foreign import ccall safe "rocksdb/c.h rocksdb_iter_prev" c_rocksdb_iter_prev :: Iterator -> IO () +-- | Binding to @rocksdb_iter_key@. foreign import ccall safe "rocksdb/c.h rocksdb_iter_key" c_rocksdb_iter_key :: Iterator -> Ptr CSize -> IO Key +-- | Binding to @rocksdb_iter_value@. foreign import ccall safe "rocksdb/c.h rocksdb_iter_value" c_rocksdb_iter_value :: Iterator -> Ptr CSize -> IO Val +-- | Binding to @rocksdb_iter_get_error@. foreign import ccall safe "rocksdb/c.h rocksdb_iter_get_error" c_rocksdb_iter_get_error :: Iterator -> ErrPtr -> IO () @@ -333,21 +410,26 @@ -- Write batch -- +-- | Binding to @rocksdb_writebatch_create@. foreign import ccall safe "rocksdb/c.h rocksdb_writebatch_create" c_rocksdb_writebatch_create :: IO WriteBatch +-- | Binding to @rocksdb_writebatch_destroy@. foreign import ccall safe "rocksdb/c.h rocksdb_writebatch_destroy" c_rocksdb_writebatch_destroy :: WriteBatch -> IO () +-- | Binding to @rocksdb_writebatch_clear@. foreign import ccall safe "rocksdb/c.h rocksdb_writebatch_clear" c_rocksdb_writebatch_clear :: WriteBatch -> IO () +-- | Binding to @rocksdb_writebatch_put@. foreign import ccall safe "rocksdb/c.h rocksdb_writebatch_put" c_rocksdb_writebatch_put :: WriteBatch -> Key -> CSize -> Val -> CSize -> IO () +-- | Binding to @rocksdb_writebatch_put_cf@. foreign import ccall safe "rocksdb/c.h rocksdb_writebatch_put_cf" c_rocksdb_writebatch_put_cf :: WriteBatch -> ColumnFamily@@ -355,10 +437,12 @@ -> Val -> CSize -> IO () +-- | Binding to @rocksdb_writebatch_delete@. foreign import ccall safe "rocksdb/c.h rocksdb_writebatch_delete" c_rocksdb_writebatch_delete :: WriteBatch -> Key -> CSize -> IO () +-- | Binding to @rocksdb_writebatch_delete_cf@. foreign import ccall safe "rocksdb/c.h rocksdb_writebatch_delete_cf" c_rocksdb_writebatch_delete_cf :: WriteBatch -> ColumnFamily@@ -370,36 +454,47 @@ -- Options -- +-- | Binding to @rocksdb_options_create@. foreign import ccall safe "rocksdb/c.h rocksdb_options_create" c_rocksdb_options_create :: IO Options +-- | Binding to @rocksdb_options_destroy@. foreign import ccall "rocksdb/c.h rocksdb_options_destroy" c_rocksdb_options_destroy :: Options -> IO () +-- | Binding to @rocksdb_options_set_create_if_missing@. foreign import ccall safe "rocksdb/c.h rocksdb_options_set_create_if_missing" c_rocksdb_options_set_create_if_missing :: Options -> CBool -> IO () +-- | Binding to @rocksdb_options_set_error_if_exists@. foreign import ccall safe "rocksdb/c.h rocksdb_options_set_error_if_exists" c_rocksdb_options_set_error_if_exists :: Options -> CBool -> IO () +-- | Binding to @rocksdb_options_set_paranoid_checks@. foreign import ccall safe "rocksdb/c.h rocksdb_options_set_paranoid_checks" c_rocksdb_options_set_paranoid_checks :: Options -> CBool -> IO () +-- | Binding to @rocksdb_options_set_max_open_files@. foreign import ccall safe "rocksdb/c.h rocksdb_options_set_max_open_files" c_rocksdb_options_set_max_open_files :: Options -> CInt -> IO () +-- | Binding to @rocksdb_options_set_write_buffer_size@. foreign import ccall safe "rocksdb/c.h rocksdb_options_set_write_buffer_size" c_rocksdb_options_set_write_buffer_size :: Options -> CSize -> IO () +-- | Binding to @rocksdb_options_set_db_write_buffer_size@. foreign import ccall safe "rocksdb/c.h rocksdb_options_set_db_write_buffer_size" c_rocksdb_options_set_db_write_buffer_size :: Options -> CSize -> IO () +-- | Binding to @rocksdb_options_set_max_total_wal_size@. foreign import ccall safe "rocksdb/c.h rocksdb_options_set_max_total_wal_size" c_rocksdb_options_set_max_total_wal_size :: Options -> Word64 -> IO () +-- | Binding to @rocksdb_options_set_create_missing_column_families@. foreign import ccall safe "rocksdb/c.h rocksdb_options_set_create_missing_column_families" c_rocksdb_options_set_create_missing_column_families :: Options -> CBool -> IO () +-- | Binding to @rocksdb_options_set_prefix_extractor@. foreign import ccall safe "rocksdb/c.h rocksdb_options_set_prefix_extractor" c_rocksdb_options_set_prefix_extractor :: Options -> SliceTransform@@ -409,38 +504,46 @@ -- Block-Based Options -- +-- | Binding to @rocksdb_block_based_options_create@. foreign import ccall safe "rocksdb/c.h rocksdb_block_based_options_create" c_rocksdb_block_based_options_create :: IO BlockBasedOptions +-- | Binding to @rocksdb_block_based_options_destroy@. foreign import ccall safe "rocksdb/c.h rocksdb_block_based_options_destroy" c_rocksdb_block_based_options_destroy :: BlockBasedOptions -> IO () +-- | Binding to @rocksdb_block_based_options_set_filter_policy@. foreign import ccall safe "rocksdb/c.h rocksdb_block_based_options_set_filter_policy" c_rocksdb_block_based_options_set_filter_policy :: BlockBasedOptions -> FilterPolicy -> IO () +-- | Binding to @rocksdb_block_based_options_set_block_cache@. foreign import ccall safe "rocksdb/c.h rocksdb_block_based_options_set_block_cache" c_rocksdb_block_based_options_set_block_cache :: BlockBasedOptions -> Cache -> IO () +-- | Binding to @rocksdb_block_based_options_set_cache_index_and_filter_blocks@. foreign import ccall safe "rocksdb/c.h rocksdb_block_based_options_set_cache_index_and_filter_blocks" c_rocksdb_block_based_options_set_cache_index_and_filter_blocks :: BlockBasedOptions -> CUChar -> IO () +-- | Binding to @rocksdb_block_based_options_set_cache_index_and_filter_blocks_with_high_priority@. foreign import ccall safe "rocksdb/c.h rocksdb_block_based_options_set_cache_index_and_filter_blocks_with_high_priority" c_rocksdb_block_based_options_set_cache_index_and_filter_blocks_with_high_priority :: BlockBasedOptions -> CUChar -> IO () +-- | Binding to @rocksdb_block_based_options_set_pin_l0_filter_and_index_blocks_in_cache@. foreign import ccall safe "rocksdb/c.h rocksdb_block_based_options_set_pin_l0_filter_and_index_blocks_in_cache" c_rocksdb_block_based_options_set_pin_l0_filter_and_index_blocks_in_cache :: BlockBasedOptions -> CUChar -> IO () +-- | Binding to @rocksdb_options_set_block_based_table_factory@. foreign import ccall safe "rocksdb/c.h rocksdb_options_set_block_based_table_factory" c_rocksdb_options_set_block_based_table_factory :: Options -> BlockBasedOptions@@ -450,9 +553,11 @@ -- Block Cache -- +-- | Binding to @rocksdb_cache_create_lru@. foreign import ccall safe "rocksdb/c.h rocksdb_cache_create_lru" c_rocksdb_cache_create_lru :: CSize -> IO Cache +-- | Binding to @rocksdb_cache_destroy@. foreign import ccall safe "rocksdb/c.h rocksdb_cache_destroy" c_rocksdb_cache_destroy :: Cache -> IO () @@ -460,9 +565,11 @@ -- Prefix Extractor -- +-- | Binding to @rocksdb_slicetransform_create_fixed_prefix@. foreign import ccall safe "rocksdb/c.h rocksdb_slicetransform_create_fixed_prefix" c_rocksdb_slicetransform_create_fixed_prefix :: CSize -> IO SliceTransform +-- | Binding to @rocksdb_slicetransform_destroy@. foreign import ccall safe "rocksdb/c.h rocksdb_slicetransform_destroy" c_rocksdb_slicetransform_destroy :: SliceTransform -> IO () @@ -470,9 +577,11 @@ -- Bloom Filter -- +-- | Binding to @rocksdb_filterpolicy_destroy@. foreign import ccall safe "rocksdb/c.h rocksdb_filterpolicy_destroy" c_rocksdb_filterpolicy_destroy :: FilterPolicy -> IO () +-- | Binding to @rocksdb_filterpolicy_create_bloom_full@. foreign import ccall safe "rocksdb/c.h rocksdb_filterpolicy_create_bloom_full" c_rocksdb_filterpolicy_create_bloom_full :: CInt -> IO FilterPolicy @@ -480,12 +589,15 @@ -- Read options -- +-- | Binding to @rocksdb_readoptions_create@. foreign import ccall safe "rocksdb/c.h rocksdb_readoptions_create" c_rocksdb_readoptions_create :: IO ReadOpts +-- | Binding to @rocksdb_readoptions_destroy@. foreign import ccall safe "rocksdb/c.h rocksdb_readoptions_destroy" c_rocksdb_readoptions_destroy :: ReadOpts -> IO () +-- | Binding to @rocksdb_readoptions_set_snapshot@. foreign import ccall safe "rocksdb/c.h rocksdb_readoptions_set_snapshot" c_rocksdb_readoptions_set_snapshot :: ReadOpts -> Snapshot -> IO () @@ -494,9 +606,11 @@ -- Write options -- +-- | Binding to @rocksdb_writeoptions_create@. foreign import ccall safe "rocksdb/c.h rocksdb_writeoptions_create" c_rocksdb_writeoptions_create :: IO WriteOpts +-- | Binding to @rocksdb_writeoptions_destroy@. foreign import ccall safe "rocksdb/c.h rocksdb_writeoptions_destroy" c_rocksdb_writeoptions_destroy :: WriteOpts -> IO () @@ -504,5 +618,6 @@ -- Free -- +-- | Binding to @rocksdb_free@. foreign import ccall safe "rocksdb/c.h rocksdb_free" c_rocksdb_free :: CString -> IO ()
src/Database/RocksDB/Internal.hs view
@@ -8,6 +8,9 @@ -- Stability : experimental -- Portability : non-portable --+-- Internal types and helpers shared by the other modules. The 'DB' and 'Config'+-- types are re-exported by "Database.RocksDB"; everything else is an+-- implementation detail. module Database.RocksDB.Internal ( Config (..)@@ -43,21 +46,57 @@ import Foreign.C.String import Foreign.C.Types +-- | Handle to an open database. Obtain one with 'Database.RocksDB.withDB' or+-- 'Database.RocksDB.withDBCF'. It is only valid inside the function passed to+-- those, and can be shared between threads. data DB = DB { rocksDB :: !RocksDB+ -- ^ pointer to the C database object , columnFamilies :: ![ColumnFamily]+ -- ^ handles of the column families opened with+ -- 'Database.RocksDB.withDBCF', in the order they were given;+ -- empty for 'Database.RocksDB.withDB' , readOpts :: !ReadOpts+ -- ^ read options used by reads and iterators; set to read from+ -- a snapshot by 'Database.RocksDB.withSnapshot' , writeOpts :: !WriteOpts+ -- ^ write options used by all writes } +-- | Configuration of a database or column family. Use 'def' for the defaults+-- and override the fields you need:+--+-- @+-- def { 'createIfMissing' = True, 'bloomFilter' = True }+-- @+--+-- Some fields only apply to the database as a whole and are ignored in the+-- configuration of a column family. This is noted on each field.+--+-- Most fields exist to control memory. RocksDB's defaults suit a small database:+-- each column family gets its own cache, the index and filter blocks of every+-- open file stay in memory, and memtables are not capped as a whole. On a large+-- database these grow without bound. Setting 'maxFiles', 'blockCacheSize',+-- 'cacheIndexAndFilterBlocks' and 'dbWriteBufferSize' turns that into a fixed+-- budget. data Config = Config { createIfMissing :: !Bool+ -- ^ create the database if it does not exist , errorIfExists :: !Bool+ -- ^ fail to open if the database already exists , paranoidChecks :: !Bool+ -- ^ check data aggressively and stop early on any+ -- corruption , createMissingColumnFamilies :: !Bool -- ^ create the column families passed to 'withDBCF' -- that an existing database does not have yet , maxFiles :: !(Maybe Int)+ -- ^ maximum number of files RocksDB keeps open+ -- ('Nothing': RocksDB default, unlimited) , prefixLength :: !(Maybe Int)+ -- ^ length in bytes of a fixed key prefix, used by+ -- bloom filters and prefix seeks , bloomFilter :: !Bool+ -- ^ use a full bloom filter with 10 bits per key to+ -- skip files that cannot contain a key , writeBufferSize :: !(Maybe Int) -- ^ bytes per memtable (RocksDB default 64 MiB); -- per column family@@ -84,6 +123,7 @@ -- configs } deriving (Eq, Show) +-- | All flags off and all sizes left to RocksDB defaults. instance Default Config where def = Config { createIfMissing = False , errorIfExists = False@@ -99,11 +139,13 @@ , maxTotalWalSize = Nothing } +-- | Create RocksDB options from a 'Config' and pass them to a function. The+-- options are destroyed when it returns. withOptions :: Config -> (Options -> IO a) -> IO a withOptions = withOptionsCache Nothing -- | Options of a database or column family that use this block cache, if--- any.+-- any. The options are destroyed when the function returns. withOptionsCache :: Maybe Cache -> Config -> (Options -> IO a) -> IO a withOptionsCache cache Config {..} f = with_opts $ \opts -> do when (bloomFilter || isJust cache || cacheIndexAndFilterBlocks) $@@ -146,6 +188,8 @@ c_rocksdb_options_destroy +-- | Create options for several column families at once, in the same order as+-- the configurations. withOptionsCF :: [Config] -> ([Options] -> IO a) -> IO a withOptionsCF = withOptionsCFCache Nothing @@ -159,6 +203,9 @@ -- | The block cache of 'blockCacheSize', if it is set. The database and -- column families that use it keep it alive after this returns.+--+-- A single cache for the whole database lets RocksDB give memory to whichever+-- column family is busy, instead of fixing a share for each. withBlockCache :: Config -> (Maybe Cache -> IO a) -> IO a withBlockCache Config {blockCacheSize = Nothing} f = f Nothing withBlockCache Config {blockCacheSize = Just n} f =@@ -166,6 +213,8 @@ c_rocksdb_cache_destroy (f . Just) +-- | Create read options, reading from a snapshot if one is given, and pass them+-- to a function. They are destroyed when it returns. withReadOpts :: Maybe Snapshot -> (ReadOpts -> IO a) -> IO a withReadOpts maybe_snap_ptr = bracket@@ -184,15 +233,25 @@ destroyReadOpts :: ReadOpts -> IO () destroyReadOpts = c_rocksdb_readoptions_destroy +-- | Create default write options and pass them to a function. They are+-- destroyed when it returns. withWriteOpts :: (WriteOpts -> IO a) -> IO a withWriteOpts = bracket c_rocksdb_writeoptions_create c_rocksdb_writeoptions_destroy +-- | Free a C string allocated by RocksDB. freeCString :: CString -> IO () freeCString = c_rocksdb_free +-- | Run a C function that takes an error pointer as its last argument. If the+-- function sets an error, free it and throw an 'IOError' whose message starts+-- with the given operation name.+--+-- Every C function that can fail reports it this way, so this is the one place+-- where RocksDB errors become Haskell exceptions, and where their messages are+-- freed. throwIfErr :: String -> (ErrPtr -> IO a) -> IO a throwIfErr s f = alloca $ \err_ptr -> do poke err_ptr nullPtr@@ -204,27 +263,33 @@ throwIO $ userError $ s ++ ": " ++ err return res +-- | Convert a 'Bool' to a C boolean. boolToCBool :: Bool -> CBool boolToCBool True = 1 boolToCBool False = 0 {-# INLINE boolToCBool #-} +-- | Convert a 'CSize' to an 'Int'. cSizeToInt :: CSize -> Int cSizeToInt = fromIntegral {-# INLINE cSizeToInt #-} +-- | Convert an 'Int' to a 'CSize'. intToCSize :: Int -> CSize intToCSize = fromIntegral {-# INLINE intToCSize #-} +-- | Convert an 'Int' to a 'CInt'. intToCInt :: Int -> CInt intToCInt = fromIntegral {-# INLINE intToCInt #-} +-- | Convert a 'CInt' to an 'Int'. cIntToInt :: CInt -> Int cIntToInt = fromIntegral {-# INLINE cIntToInt #-} +-- | Convert a 'Bool' to @1@ or @0@. boolToNum :: Num b => Bool -> b boolToNum True = fromIntegral (1 :: Int) boolToNum False = fromIntegral (0 :: Int)
src/Database/RocksDB/Iterator.hs view
@@ -11,6 +11,41 @@ -- Portability : non-portable -- -- Iterating over key ranges.+--+-- An 'Iterator' walks the keys of a database or column family in ascending+-- byte order. It starts out not positioned anywhere; call 'iterFirst',+-- 'iterLast' or 'iterSeek' to position it, then 'iterNext' or 'iterPrev' to+-- move. Read the current entry with 'iterKey', 'iterValue' or 'iterEntry',+-- which return 'Nothing' once the iterator has moved past either end.+--+-- @+-- -- all entries whose key starts with "user:"+-- usersFrom :: DB -> IO [(ByteString, ByteString)]+-- usersFrom db = 'withIter' db $ \\it -> do+-- 'iterSeek' it "user:"+-- let go acc = 'iterEntry' it >>= \\case+-- Just (k, v) | "user:" \`B.isPrefixOf\` k -> 'iterNext' it >> go ((k, v) : acc)+-- _ -> return (reverse acc)+-- go []+-- @+--+-- After iterating, check 'iterGetError' to tell the end of the data apart from+-- an error such as corruption.+--+-- = Why iterators+--+-- Sorted keys are what make RocksDB more than a hash table. Keys that share a+-- prefix are stored next to each other, so all the records of one user, or all+-- the transactions of one address, can be read with one seek and a short scan.+-- Designing keys so that the records you read together sort together is the+-- most important performance decision in a RocksDB schema.+--+-- = Safety+--+-- The C API crashes the process if an invalid iterator is moved, and the bytes+-- it returns are only valid until it moves again. This module guards against+-- both: 'iterNext' and 'iterPrev' do nothing on an invalid iterator, and keys+-- and values are copied into Haskell memory before they are returned. module Database.RocksDB.Iterator ( Iterator, -- * Bracket-style iteration@@ -59,6 +94,7 @@ withIter :: DB -> (Iterator -> IO a) -> IO a withIter db = withIterCommon db Nothing Nothing +-- | Like 'withIter', but iterate over the given column family. withIterCF :: DB -> ColumnFamily -> (Iterator -> IO a) -> IO a withIterCF db cf = withIterCommon db Nothing (Just cf) @@ -88,6 +124,8 @@ -> IO a withIterSnapCF db msnap cf = withIterCommon db msnap (Just cf) +-- | Create an iterator over the default column family, or over a column family+-- if one is given, reading from a snapshot if one is given. withIterCommon :: DB -> Maybe Snapshot@@ -116,7 +154,11 @@ Just cf -> c_rocksdb_create_iterator_cf rocks_db ro cf Nothing -> c_rocksdb_create_iterator rocks_db ro --- | Manually create unmanaged iterator.+-- | Manually create unmanaged iterator, over the default column family or over+-- a column family if one is given. It uses the 'readOpts' of the 'DB', so if+-- the 'DB' comes from 'Database.RocksDB.withSnapshot' it reads from that+-- snapshot. It must be released with 'destroyIterator', and must not outlive+-- the 'DB'. createIterator :: DB -> Maybe ColumnFamily -> IO Iterator createIterator DB{rocksDB = rocks_db, readOpts = read_opts} mcf = throwErrnoIfNull "create_iterator" $ case mcf of@@ -157,7 +199,8 @@ Just cf -> c_rocksdb_create_iterator_cf rocks_db ro cf Nothing -> c_rocksdb_create_iterator rocks_db ro --- | Destroy unmanaged iterator.+-- | Destroy unmanaged iterator created with 'createIterator' or+-- 'createIteratorSnap'. It must not be used afterwards. destroyIterator :: Iterator -> IO () destroyIterator = c_rocksdb_iter_destroy @@ -192,6 +235,9 @@ -- If the iterator is not valid, this function does nothing: RocksDB does not -- allow moving an invalid iterator, so reposition it with 'iterSeek', -- 'iterFirst' or 'iterLast' instead.+--+-- The check costs one cheap call and turns a crash in C into a harmless no-op,+-- which is the right trade for a Haskell binding. iterNext :: Iterator -> IO () iterNext it = do valid <- iterValid it@@ -210,6 +256,9 @@ -- | Return the key for the current entry if the iterator is currently -- positioned at an entry, ie. 'iterValid'.+--+-- The key is copied, because RocksDB reuses the memory behind it as soon as the+-- iterator moves. iterKey :: Iterator -> IO (Maybe ByteString) iterKey it = iterString it c_rocksdb_iter_key @@ -229,6 +278,9 @@ -- | Check for errors -- -- Note that this captures somewhat severe errors such as a corrupted database.+-- An iterator that becomes invalid because of an error looks the same as one+-- that reached the end of the data, so call this after iterating to tell them+-- apart. Returns 'Nothing' if there was no error. iterGetError :: Iterator -> IO (Maybe ByteString) iterGetError iter_ptr = alloca $ \err_ptr -> do poke err_ptr nullPtr@@ -242,6 +294,8 @@ -- Internal -- +-- | Read a C string from the iterator using the given accessor, if the+-- iterator is valid. The string is copied, since it belongs to the iterator. iterString :: Iterator -> (Iterator -> Ptr CSize -> IO CString) ->