packages feed

rocksdb-haskell-jprupp 2.6.1 → 2.6.2

raw patch · 7 files changed

+169/−22 lines, 7 filesPVP ok

version bump matches the API change (PVP)

API changes (from Hackage documentation)

Files

CHANGELOG.md view
@@ -1,3 +1,34 @@+## 2.6.2++ * Build the Bloom filters that `bloomFilter` asks for. The bits per key were+   passed to `rocksdb_filterpolicy_create_bloom_full` as an integer, which is+   what RocksDB took until 6.23; since 6.24 the function takes a `double`. An+   integer and a `double` travel in different registers, so RocksDB read+   something else as the bits per key and built no filters: table files had+   `filter block size=0`, and a lookup of an absent key read a data block in+   every file that could hold it. Nothing failed, and no test noticed. In a+   measurement with 2,600,000 keys, such a lookup took about 5.5 microseconds+   without the filters and 1.5 with them.++   The function is now imported with the `capi` calling convention, through a+   stub compiled against `rocksdb/c.h`, so the C compiler passes the argument+   as the installed RocksDB declares it, before or after 6.24. The headers+   were already needed to install RocksDB for development; they are now+   needed to build this package too.++   Databases opened with `bloomFilter` start building filters when they are+   opened with this version: for the table files written from then on, at+   about 1.25 bytes per key. Files written before have none until compaction+   rewrites them.+ * Pass the error pointer that `rocksdb_approximate_sizes` takes since RocksDB+   6.16. `approximateSize` called it with one argument too few, leaving RocksDB+   to read a pointer that was never passed, and to write an error message+   through it if the call failed. It now throws an `IOError` on failure.+ * Both were found by comparing every foreign import with the `rocksdb/c.h` of+   RocksDB 11.1: the other 70 match.+ * Tests that table files have filters when asked for and not otherwise, and+   of `approximateSize`, which had none.+ ## 2.6.1   * Documentation, lots of it, with detailed explanations.
README.md view
@@ -1,8 +1,13 @@ # rocksdb-haskell-jprupp -[![Hackage](https://img.shields.io/hackage/v/rocksdb-haskell-jprupp.svg)](https://hackage.haskell.org/package/rocksdb-haskell-jprupp)-[![License: BSD-3-Clause](https://img.shields.io/badge/license-BSD--3--Clause-blue.svg)](https://github.com/jprupp/rocksdb-haskell/blob/master/LICENSE)+[![Hackage][hackage-badge]][hackage]+[![License: BSD-3-Clause][license-badge]][license] +[hackage]: https://hackage.haskell.org/package/rocksdb-haskell-jprupp+[hackage-badge]: https://img.shields.io/hackage/v/rocksdb-haskell-jprupp.svg+[license]: https://github.com/jprupp/rocksdb-haskell/blob/master/LICENSE+[license-badge]: https://img.shields.io/badge/license-BSD--3--Clause-blue.svg+ Haskell bindings to [RocksDB](https://rocksdb.org), the embedded persistent key-value store. The package on Hackage is `rocksdb-haskell-jprupp`; the module to import is `Database.RocksDB`.@@ -84,7 +89,8 @@ | Column families| Open a database with several independent key spaces; add new ones on open     | | Snapshots      | Consistent point-in-time reads across many queries and iterators               | | Iterators      | Ordered scans, forwards and backwards, optionally pinned to a snapshot         |-| Tuning         | Bloom filters, fixed-length prefix extractor, shared LRU block cache, memtable and WAL limits, open file limit |+| Tuning         | Bloom filters, fixed-length prefix extractor, shared LRU block cache           |+| Limits         | Memtable and WAL size limits, open file limit                                  | | Administration | Properties (memory use, statistics, SST files), approximate sizes, destroy, repair |  ### What is not included@@ -120,6 +126,10 @@ non-standard place, pass its directories with `--extra-include-dirs` and `--extra-lib-dirs` (Cabal or Stack). +The headers are needed to build, not only the library: the binding of the+Bloom filter policy is compiled against `rocksdb/c.h`, because RocksDB changed+the type of its argument in 6.24 and the header says which one is installed.+ ### Check the library is found  ```sh@@ -171,7 +181,7 @@  ```yaml extra-deps:-  - rocksdb-haskell-jprupp-2.6.0+  - rocksdb-haskell-jprupp-2.6.2 ```  With `stack --nix`, add RocksDB to the Nix packages:@@ -437,7 +447,11 @@ hundred megabytes to a few gigabytes, `cacheIndexAndFilterBlocks = True`, `maxFiles` of a few thousand, and `dbWriteBufferSize` of a few hundred megabytes. Enable `bloomFilter` when most reads are point lookups of keys that-may not exist.+may not exist. A filter takes about 1.25 bytes per key, held with the index+blocks: in the block cache with `cacheIndexAndFilterBlocks`, in memory outside+it otherwise. A filter belongs to a table file and is written with it, so+turning `bloomFilter` on for an existing database only covers the files+written from then on; the others get a filter when compaction rewrites them.  For a large database, setting `maxFiles`, `blockCacheSize`, `cacheIndexAndFilterBlocks` and `dbWriteBufferSize` puts a ceiling on the memory@@ -454,7 +468,9 @@ | Opening                  | `withDB`, `withDBCF`                                           | | Reading and writing      | `get`, `getCF`, `multiGet`, `multiGetCF`, `put`, `putCF`, `delete`, `deleteCF`, `write` | | Snapshots                | `withSnapshot`, `createSnapshot`, `releaseSnapshot`            |-| Iteration                | `withIter`, `withIterCF`, `withIterSnap`, `withIterSnapCF`, `iterSeek`, `iterFirst`, `iterLast`, `iterNext`, `iterPrev`, `iterKey`, `iterValue`, `iterEntry`, `iterValid`, `iterGetError` |+| Creating iterators       | `withIter`, `withIterCF`, `withIterSnap`, `withIterSnapCF`     |+| Moving iterators         | `iterSeek`, `iterFirst`, `iterLast`, `iterNext`, `iterPrev`    |+| Reading from iterators   | `iterKey`, `iterValue`, `iterEntry`, `iterValid`, `iterGetError` | | Administration           | `getProperty`, `getPropertyCF`, `approximateSize`, `destroy`, `repair` |  The FFI bindings (`Database.RocksDB.C`) and helpers (`Database.RocksDB.Internal`)@@ -472,6 +488,16 @@   reached the end. Check `iterGetError`. - Calls into RocksDB are `safe` foreign calls, so they do not block other   Haskell threads, but each one costs a little more than an `unsafe` call.+- Releases before 2.6.2 built no Bloom filters on RocksDB 6.24 or later,+  whatever `bloomFilter` said: they passed the bits per key as an integer to a+  function that had come to take a `double`. A database written by one of them+  has no filters in its table files. After upgrading, files get them as they+  are written, and the memory the filters need comes with them. The size of+  the filters of a column family is in the `rocksdb.aggregated-table-properties`+  property, as `filter block size`.+- The compiler does not check a foreign import against RocksDB's header. A+  binding with the wrong type builds and runs, with garbage in place of the+  argument; the two fixed in 2.6.2 went unnoticed for years.  ## Development @@ -512,6 +538,10 @@  - Format new code with fourmolu, which the Nix shell provides. - Add tests for the behaviour you change.+- When you add or change a binding in `Database.RocksDB.C`, read the+  prototype in `rocksdb/c.h` argument by argument: nothing else checks the+  Haskell type against it. If the type of an argument differs between+  versions of RocksDB, import the function with `capi`. - Describe the change in `CHANGELOG.md`. Versions follow the   [Haskell Package Versioning Policy](https://pvp.haskell.org/).
rocksdb-haskell-jprupp.cabal view
@@ -5,7 +5,7 @@ -- see: https://github.com/sol/hpack  name:           rocksdb-haskell-jprupp-version:        2.6.1+version:        2.6.2 synopsis:       Haskell bindings for RocksDB description:    See README at <https://github.com/jprupp/rocksdb-haskell#readme> category:       Database, FFI
src/Database/RocksDB/Base.hs view
@@ -337,7 +337,7 @@  -- | Approximate number of bytes of file system space used by the keys in a -- 'Range' of the default column family. Data still in memtables may not be--- counted.+-- counted. Throws an 'IOError' on failure. approximateSize :: DB -> Range -> IO Int64 approximateSize DB{rocksDB = db_ptr} (from, to) =     BU.unsafeUseAsCStringLen from $ \(from_ptr, flen) ->@@ -346,15 +346,13 @@     withArray [intToCSize flen]   $ \flen_ptrs        ->     withArray [to_ptr]            $ \to_ptrs          ->     withArray [intToCSize tlen]   $ \tlen_ptrs        ->-    allocaArray 1                 $ \size_ptrs        -> do-        c_rocksdb_approximate_sizes db_ptr 1-                                    from_ptrs flen_ptrs-                                    to_ptrs tlen_ptrs-                                    size_ptrs-        fmap head $ peekArray 1 size_ptrs >>= mapM toInt64--    where-        toInt64 = return . fromIntegral+    alloca                        $ \size_ptr         -> do+        throwIfErr "approximate_sizes" $+            c_rocksdb_approximate_sizes db_ptr 1+                                        from_ptrs flen_ptrs+                                        to_ptrs tlen_ptrs+                                        size_ptr+        fromIntegral <$> peek size_ptr  -- | Write a key/value pair to the default column family, replacing any -- previous value. Throws an 'IOError' on failure.
src/Database/RocksDB/C.hs view
@@ -1,3 +1,4 @@+{-# LANGUAGE CApiFFI                  #-} {-# LANGUAGE CPP                      #-} {-# LANGUAGE EmptyDataDecls           #-} {-# LANGUAGE ForeignFunctionInterface #-}@@ -21,6 +22,21 @@ -- for example when they wait for disk or for a compaction to make room, and a -- @safe@ call lets other Haskell threads keep running meanwhile. The small -- extra cost per call is worth that in a concurrent program.+--+-- = Keeping the types in step with the header+--+-- The compiler does not check a foreign import against @rocksdb\/c.h@: it+-- takes the Haskell type on trust. An argument of the wrong kind, or a+-- missing one, compiles, links and runs, and RocksDB then works with garbage+-- in its place. RocksDB has changed the types of functions that already+-- existed, twice among the ones bound here: the error pointer added to+-- @rocksdb_approximate_sizes@ in 6.16, and the bits per key of the Bloom+-- filter policies, an @int@ until 6.24 and a @double@ since. When adding or+-- reviewing a binding, read its prototype in the header, argument by+-- argument. Where a type differs between versions of RocksDB, import the+-- function with @capi@ instead of @ccall@, as+-- 'c_rocksdb_filterpolicy_create_bloom_full' is: the C compiler then reads+-- the header that is installed, and converts the argument.  module Database.RocksDB.C     ( RocksDB@@ -327,6 +343,11 @@   c_rocksdb_property_value_cf :: RocksDB -> ColumnFamily -> CString -> IO CString  -- | Binding to @rocksdb_approximate_sizes@.+--+-- The last argument receives an error, as in every function that can fail.+-- RocksDB added it in 6.16. A binding without it leaves the C function to+-- read a pointer that was never passed, and to write through it on failure.+-- Older versions do not take the argument, and ignore it. foreign import ccall safe "rocksdb/c.h rocksdb_approximate_sizes"   c_rocksdb_approximate_sizes :: RocksDB                               -> CInt@@ -337,6 +358,7 @@                                  -- ^ range limit keys (array)                               -> Ptr Word64                                  -- ^ array of approx. sizes of ranges+                              -> ErrPtr                               -> IO ()  -- | Binding to @rocksdb_destroy_db@.@@ -532,7 +554,8 @@                                                                   -> IO ()  -- | Binding to @rocksdb_block_based_options_set_cache_index_and_filter_blocks_with_high_priority@.-foreign import ccall safe "rocksdb/c.h rocksdb_block_based_options_set_cache_index_and_filter_blocks_with_high_priority"+foreign import ccall safe+  "rocksdb/c.h rocksdb_block_based_options_set_cache_index_and_filter_blocks_with_high_priority"   c_rocksdb_block_based_options_set_cache_index_and_filter_blocks_with_high_priority :: BlockBasedOptions                                                                                      -> CUChar                                                                                      -> IO ()@@ -582,8 +605,19 @@   c_rocksdb_filterpolicy_destroy :: FilterPolicy -> IO ()  -- | Binding to @rocksdb_filterpolicy_create_bloom_full@.-foreign import ccall safe "rocksdb/c.h rocksdb_filterpolicy_create_bloom_full"-  c_rocksdb_filterpolicy_create_bloom_full :: CInt -> IO FilterPolicy+--+-- The bits per key are a @double@ in C since RocksDB 6.24, and were an @int@+-- before. Passed as the wrong one of the two, they travel in the wrong+-- register: the function reads whatever the other register holds, takes it+-- for the bits per key, and with too few of them builds no filter at all.+-- Nothing fails; lookups of absent keys are just as slow as without a filter.+--+-- So this import alone uses the @capi@ calling convention. GHC then calls the+-- function through a stub that it compiles against @rocksdb\/c.h@, and the C+-- compiler converts the argument to the type that the installed RocksDB+-- declares, whichever it is.+foreign import capi safe "rocksdb/c.h rocksdb_filterpolicy_create_bloom_full"+  c_rocksdb_filterpolicy_create_bloom_full :: CDouble -> IO FilterPolicy  -- -- Read options
src/Database/RocksDB/Internal.hs view
@@ -96,7 +96,8 @@                        -- bloom filters and prefix seeks                      , bloomFilter     :: !Bool                        -- ^ use a full bloom filter with 10 bits per key to-                       -- skip files that cannot contain a key+                       -- skip files that cannot contain a key; the filters+                       -- take about 1.25 bytes of memory per key                      , writeBufferSize :: !(Maybe Int)                        -- ^ bytes per memtable (RocksDB default 64 MiB);                        -- per column family
test/Spec.hs view
@@ -56,7 +56,10 @@           `shouldReturn` [Just "one50", Nothing, Just "one1", Just "one100", Nothing, Just "one50"]         multiGet db [] `shouldReturn` []         ks <- forM [1 .. 2000 :: Int] $ \i -> return (C.pack (show (i `mod` 150)))-        multiGet db ks >>= (`shouldBe` map (\k -> if read (C.unpack k) `elem` [1 .. 100 :: Int] then Just ("v" <> k) else Nothing) ks)+        let expected k+              | read (C.unpack k) `elem` [1 .. 100 :: Int] = Just ("v" <> k)+              | otherwise = Nothing+        multiGet db ks `shouldReturn` map expected ks     describe "Multithreading" $ do       it "stores and retrieve items from multiple threads" $ \db -> do         let key i = C.pack $ printf "key_%04d" i@@ -264,6 +267,44 @@             getCF db two (key 20000) `shouldReturn` Just value             files <- getProperty db SSTables             files `shouldSatisfy` maybe False (not . C.null)+    describe "Bloom filters" $ do+      it "are built for a family that asks for one, and only for it" $ \_ ->+        withSystemTempDirectory "rocksdb-tests-bloom" $ \path -> do+          let open = def {createIfMissing = True, createMissingColumnFamilies = True}+              families = [("with", def {bloomFilter = True}), ("without", def)]+              key :: Int -> C.ByteString+              key i = C.pack (printf "k%07d" i)+              -- bytes of filter blocks in the table files of a family+              filters d cf =+                getPropertyCF d cf (PropertyName "rocksdb.aggregated-table-properties")+                  >>= maybe (fail "no table properties") (return . tableProperty "filter block size")+              both d = case columnFamilies d of+                [a, b] -> return (a, b)+                _ -> fail "expected two column families"+          withDBCF path open families $ \db -> do+            (with, without) <- both db+            forM_ [1 .. 5000] $ \i -> putCF db with (key i) "value" >> putCF db without (key i) "value"+          -- opening the database again moves what was written from the+          -- write-ahead log to table files, which is where filters are+          withDBCF path open families $ \db -> do+            (with, without) <- both db+            -- ten bits for each of the 5000 keys is 6250 bytes+            filters db with >>= (`shouldSatisfy` maybe False (> 5000))+            filters db without `shouldReturn` Just 0+            getCF db with (key 2500) `shouldReturn` Just "value"+            getCF db with "absent" `shouldReturn` Nothing+    describe "Approximate sizes" $ do+      it "estimates the space used by a range of keys" $ \_ ->+        withSystemTempDirectory "rocksdb-tests-sizes" $ \path -> do+          let open = def {createIfMissing = True}+              key :: Int -> C.ByteString+              key i = C.pack (printf "k%07d" i)+          withDB path open $ \db ->+            forM_ [1 .. 5000] $ \i -> put db (key i) (C.pack (printf "%0100d" i))+          -- sizes are those of table files; see the test of Bloom filters+          withDB path open $ \db -> do+            approximateSize db (key 1, key 5000) >>= (`shouldSatisfy` (> 0))+            approximateSize db ("x", "y") `shouldReturn` 0     describe "Properties" $ do       it "reports memory use, per database and per column family" $ \db -> do         let one = head (columnFamilies db)@@ -321,3 +362,15 @@             let two = columnFamilies db !! 1             putCF db two "new" "cf"             getCF db two "new" `shouldReturn` Just "cf"++-- | A number in the text of the @rocksdb.aggregated-table-properties@+-- property, which reads @name=value; name=value; ...@.+tableProperty :: C.ByteString -> C.ByteString -> Maybe Int+tableProperty name text =+  listToMaybe+    [ n+    | field <- C.split ';' text,+      let (k, v) = C.break (== '=') (C.dropWhile (== ' ') field),+      k == name,+      Just (n, _) <- [C.readInt (C.drop 1 v)]+    ]