packages feed

rocksdb-haskell-jprupp-2.8.0: CHANGELOG.md

## 2.8.0

 * `maxBackgroundJobs`, a new field of `Config`: how many threads RocksDB may
   use at once to flush memtables and to compact files, where its default is
   two, one of each. A database that has a backlog of compactions, after a
   bulk load for instance, works through it with one thread unless it is
   given more. More threads only help where the disk and the cores have room
   for them: a compaction that already waits for a slow disk is not made
   faster by a second one, and both take from the reads.
 * `partitionIndexAndFilters`, another new field of `Config`: the index and
   the filter of each table file are written in pieces of 4 KiB under a small
   index of their own. Each is otherwise one block that grows with the file,
   megabytes for a file of 64 MiB with a filter, and a database that has more
   of them than its block cache holds loads and drops them at nearly every
   lookup. Measured on a database of 1.6 TB with a cache of 1 GiB: the cache
   held the index and filter of 400 of 32,000 files and three megabytes of
   data, and a lookup took 0.8 ms of processor time on an idle disk. With
   pieces, a lookup loads the two that it needs. It applies to files written
   from then on; existing files keep their layout until compacted.
 * `compactRange` writes again every table file of a range of keys, those of
   the last level included, so that the files a database already has get the
   layout of its present `Config`: a Bloom filter, or
   `partitionIndexAndFilters`. RocksDB's own compactions rarely reach the
   last level, where most of the data is.
 * Many entries of an iterator in one foreign call. `iterNext` and `iterPrev`
   can wait for the disk, so each is a call for which the Haskell thread
   gives up its capability and has to get one back, and a scan made one for
   every entry. `iterChunk` walks the iterator in C, in a small function
   that now comes with the package (`cbits/rocksdb_haskell.c`), and returns
   up to 256 entries or 1 MiB from one call, forwards or backwards, within a
   prefix if one is given, with a flag that tells whether more follow. A
   `ChunkConfig` sets those limits. `iterEntries`, `iterEntriesBack`,
   `iterEntriesPrefix` and `iterEntriesPrefixBack` are short forms for a
   known number of entries. All of them leave the iterator on the first
   entry that they did not return, respect the bounds of an `IterConfig`,
   and throw if the iterator stopped at an error. The single steps are
   unchanged.

   Seconds for every thread to read 200,000 entries of 52 bytes, on four
   capabilities and four cores (`stack bench`):

   | Scan                                  | 1 thread | 16 threads |
   | ------------------------------------- | -------- | ---------- |
   | Single steps, loop not tail recursive | 0.78     | 3.9        |
   | Single steps, loop tail recursive     | 0.090    | 0.50       |
   | Chunks of 256 entries                 | 0.048    | 0.25       |

   With all 24 hardware threads of the machine free, the sixteen threads
   took 4.4, 0.83 and 0.12 seconds: threads inside the call hold no
   capability, so chunks gain more where there are more cores than
   capabilities. The first row is a loop that returns its list with
   `(entry :) <$> go`; the run-time system walks the stack at every such
   call, and that loop leaves a frame on it for every entry.
 * A large batch in a few foreign calls. `write` hands RocksDB the
   operations of a batch one call at a time, each a call for which the
   Haskell thread gives up its capability and has to get one back. For the
   65,000 keys of a Bitcoin block that was more than half of the time of
   the batch, and nearly all of it in a program whose other threads were
   busy. `writePacked` takes the batch as chunks of packed records, and a
   second function of `cbits/rocksdb_haskell.c` adds the operations of a
   chunk to the batch in one call. The batch is then written as before:
   atomically, with one write, however many chunks it came in.

   `packOps` packs a list of `BatchOp` into a chunk. A program that
   serialises its keys and values itself can write records straight into
   its own buffers instead, with no `ByteString` for each: it leaves
   `packedHeaderSize` bytes, serialises the key and the value after them,
   and fills in the header with `pokePackedHeader` once the lengths are
   known. Records are not aligned, so the next one starts where the last
   one ended.

   A chunk is checked in C before anything of it is used. One that is cut
   short, that has a length reaching past its end, an operation that does
   not exist or a delete with a value, makes `writePacked` throw, with
   nothing written.

   A record can also ask for a merge (`PackedMerge`), which is handed to
   `rocksdb_writebatch_merge`. The binding still has no way to give a
   column family a merge operator, so RocksDB refuses such a batch; the
   format has it so that it need not change later.

   The format is this package's own. It is not RocksDB's internal encoding
   of a batch (`rocksdb_writebatch_create_from`), which would save a copy
   and tie the binding to the insides of RocksDB.

   Milliseconds to write a batch of 65,000 operations and 6 MB over four
   column families, on four capabilities and four cores
   (`stack bench rocksdb-haskell-jprupp:bench:batch`):

   | Way to write the batch                    | Alone | Next to 8 busy threads |
   | ----------------------------------------- | ----- | ---------------------- |
   | `write`: a call for each operation        | 48    | 420                    |
   | `writePacked`, with `packOps` in the time | 28    | 115                    |
   | `writePacked`, chunks packed beforehand   | 22    | 28                     |

   All three end with the same write, of about 20 ms. Alone, the calls of
   `write` cost 400 ns an operation. Next to threads that compute, the
   thread waits for a capability after each of its 65,000 calls. With all
   24 hardware threads of the machine free, that took `write` 1,160 ms,
   and packed chunks 31.
 * Merge records and merge operators. A merge record is a third kind of
   write, next to a put and a delete: it says how a value changes, and
   RocksDB stores it without reading the value. The merge operator of the
   column family works out the value when the key is read, and when a
   compaction brings the records and the value together. A program that
   adds to counters or to lists by the thousand no longer reads each one
   first.
   - `mergeOperator`, a new field of `Config`, gives a column family its
     operator, or the default one when it is in the `Config` of the
     database. `UInt64Add` is RocksDB's own, for counters. `MergeFunctions`
     makes one of a name and the addresses of two C functions, a full merge
     and a partial merge, with the types `FullMerge` and `PartialMerge`.
     They are C functions because RocksDB calls them from its own threads;
     `cbits/rocksdb_haskell.h` says what they must do, and a third function
     of `cbits/rocksdb_haskell.c` makes the operator around them.
   - `Merge` and `MergeCF` are new constructors of `BatchOp`, written by
     `write` and packed by `packOps` as `PackedMerge` records. `merge` and
     `mergeCF` write one record. Code that matches on every `BatchOp` has
     two more cases.
   - `get`, `multiGet`, iterators and chunks return merged values, and
     need no change.
   - A column family with merge records must be given its operator by
     every program that opens the database. RocksDB 11.1 opens it without:
     it then drops the merge records that are still in the write-ahead
     log, with no error, and a read of a key with a value under merge
     records crashes the process. The README has the whole table.
 * `flush` writes the memtable of a column family to a table file and waits
   for it, and `disableAutoCompactions`, a new field of `Config`, stops
   RocksDB from compacting a column family in the background. They are for
   tests and benchmarks that need data in table files, a file for each
   flush, and for bulk loads that end with `compactRange`.
 * The `Config` record has new fields, so code that builds one without `def`
   has to set them; code that starts from `def` needs no change. This is why
   the version is 2.8.0 and not 2.7.2.

## 2.7.1

 * Documentation and tests only; the library behaves as in 2.7.0.
 * The documentation of `prefixSameAsStart` was wrong on one point and silent
   on another. It said that the option does nothing after `iterFirst` or
   `iterLast`: it does, the iterator stays within the prefix of the key that
   it finds there. And it did not say that every key, sought or stored, has
   to be at least as long as the `prefixLength`: RocksDB takes the prefix of
   a key without checking its length, and reads past the end of a shorter
   one. Both are now in the Haddock and in the README, and the first has
   tests.

## 2.7.0

 * Bounded iterators. `withIterConfig` creates an iterator that RocksDB itself
   stops, as an `IterConfig` says: at a `lowerBound` or an `upperBound`, or at
   the end of the prefix that it was sent to, with `prefixSameAsStart`. Until
   now a scan of a range could only stop by reading the first key past it,
   and when the keys past it had been deleted, RocksDB stepped over the marker
   of every one to find that key. A scan of an empty range followed by 20,000
   deleted keys took 2,600 microseconds; with an upper bound it takes 2.5.
   `prefixUpperBound` gives the bound that ends a prefix, and
   `createIteratorConfig` manages such an iterator by hand.
 * `totalOrderSeek`, also in `IterConfig`, walks every key in order in a
   database that has a `prefixLength`, where a plain iterator only promises
   the keys of one prefix.
 * The `DB` record has a new field, `snapshot`, with the snapshot that its
   read options read from. Code that builds a `DB` by hand has to set it;
   code that gets one from `withDB`, `withDBCF` or `withSnapshot` needs no
   change. This is why the version is 2.7.0 and not 2.6.5.

## 2.6.4

 * Iterators read rows about three times faster. `iterValid`, `iterKey` and
   `iterValue` now reach RocksDB through unsafe foreign calls. The three only
   read what the iterator already holds, and reading one row with `iterEntry`
   and `iterNext` makes five of them; as safe calls each cost about 0.4
   microseconds. Walking 700,000 rows took 2.8 microseconds a row before and
   0.8 after. The calls that move an iterator, which can read from disk, are
   safe calls as before.

## 2.6.3

 * Documentation only; the library behaves as in 2.6.2.
 * References in the Haddock that named both a type and its constructor, such
   as `DB` and `Config`, now say that they mean the type, and two that named
   nothing in scope are fixed: the documentation builds without warnings about
   ambiguous or unresolved identifiers.
 * The documentation of `withIter` said to set a snapshot in the read options,
   which the API has no way to do. It now says how: create the iterator from
   the `DB` that `withSnapshot` provides.
 * Remove an unused import.
 * The test suite grows from 17 tests to 51. It now covers deletes, batches,
   `withSnapshot`, iteration backwards, empty keys and values, binary data,
   and the errors of opening a database, none of which had a test. Six of the
   tests are QuickCheck properties that compare the database with a `Map`
   after random writes. The suite also compiles without warnings.
 * A badge in the README shows the state of CI on `master`.

## 2.6.2

 * Build the Bloom filters that `bloomFilter` asks for. The bits per key were
   passed to `rocksdb_filterpolicy_create_bloom_full` as an integer, which is
   what RocksDB took until 6.23; since 6.24 the function takes a `double`. An
   integer and a `double` travel in different registers, so RocksDB read
   something else as the bits per key and built no filters: table files had
   `filter block size=0`, and a lookup of an absent key read a data block in
   every file that could hold it. Nothing failed, and no test noticed. In a
   measurement with 2,600,000 keys, such a lookup took about 5.5 microseconds
   without the filters and 1.5 with them.

   The function is now imported with the `capi` calling convention, through a
   stub compiled against `rocksdb/c.h`, so the C compiler passes the argument
   as the installed RocksDB declares it, before or after 6.24. The headers
   were already needed to install RocksDB for development; they are now
   needed to build this package too.

   Databases opened with `bloomFilter` start building filters when they are
   opened with this version: for the table files written from then on, at
   about 1.25 bytes per key. Files written before have none until compaction
   rewrites them.
 * Pass the error pointer that `rocksdb_approximate_sizes` takes since RocksDB
   6.16. `approximateSize` called it with one argument too few, leaving RocksDB
   to read a pointer that was never passed, and to write an error message
   through it if the call failed. It now throws an `IOError` on failure.
 * Both were found by comparing every foreign import with the `rocksdb/c.h` of
   RocksDB 11.1: the other 70 match.
 * Tests that table files have filters when asked for and not otherwise, and
   of `approximateSize`, which had none.

## 2.6.1

 * Documentation, lots of it, with detailed explanations.

## 2.6.0

 * Add `multiGet` and `multiGetCF`, which read the values of many keys in
   one call to RocksDB's `MultiGet`.
 * Add `blockCacheSize` to `Config`: the bytes of a least-recently-used block
   cache shared by the database and all its column families. Without it
   RocksDB gives each column family its own default cache, of 32 MiB in
   RocksDB 8.
 * Add `cacheIndexAndFilterBlocks` to `Config`, to keep the index and filter
   blocks of a column family in the block cache, at high priority, with
   those of level 0 pinned, instead of in memory outside the cache for every
   open file.
 * Stop leaking the block-based table options of every database and column
   family opened with a Bloom filter.

## 2.5.0

 * Add the `EstimateTableReadersMem`, `CurSizeAllMemTables` and
   `BlockCacheUsage` properties, and `PropertyName` for any other property by
   its name.
 * Add `getPropertyCF` to read a property of a column family.
 * Add `maxTotalWalSize` (`max_total_wal_size`) to `Config`, to bound the
   write-ahead log.
 * Add `createMissingColumnFamilies` (`create_missing_column_families`) to
   `Config`, to add column families to an existing database.
 * Destroy the handle of the default column family when `withDBCF` opens an
   existing database. It was leaked, and RocksDB built with assertions
   aborted when the database was closed.

## 2.4.0

 * Add `writeBufferSize` (`write_buffer_size`) and `dbWriteBufferSize`
   (`db_write_buffer_size`) to `Config`, to bound memtable memory.

## 2.3.1

 * Do not call next/prev on invalid iterators (fixes assertion failure).

## 2.3.0

 * Remove UnliftIO and remain in IO monad.

## 2.2.0

 * Move Iterator module to plain IO.
 * Properly manage read options in createSnapshot.
 * Add nix flake for development.
 * API allows manual database lifecycle management.

## 2.1.7

 * Simplify author name.

## 2.1.6

 * Refactor memory use further.
 * Improve tests.

## 2.1.5

 * Fix errors not releasing memory.

## 2.1.4

 * Add ResourceT support for snapshots and iterators.

## 2.1.3

 * Correct bug where database was ignoring prefix length config.


## 2.1.2

 * Correct bug when opening database with column families.


## 2.1.1

 * Expose ColumnFamily type.
 

## 2.1.0

 * Add support for column families.


## 2.0.0

 * Fork into a different project
 * Rewrite most of the code to make it faster and easier to use
 * Remove many features that I don't need or want to maintain


## 1.0.1

 * Add support for UTF-8 characters in a database's path


## 1.0.0

 * First version by Serokell


## 0.3x.0

 * ResourceT is no longer compulsory


## 0.2.0

 * requires LevelDB v1.7
 * support for filter policy (LevelDB v1.5), either custom or using the built-in
   bloom filter implementation
 * write batch values no longer require a `memcpy` to be early-finalizer-safe
   (introduced in 0.1.1)


## 0.1.0

 * memory (foreign pointers) is managed through
   [ResourceT](http://hackage.haskell.org/package/resourcet). Note that this
   requires to lift monadic actions inside the `MonadResource` monad, see the
   examples.
 * links against shared library (LevelDB v1.3 or higher)
 * LevelDB 1.3 API fully supported (including custom comparators, excluding
   custom environments)


## 0.0.x

 * experimental releases