streamly-lmdb 0.5.0 → 0.6.0
raw patch · 6 files changed
+35/−29 lines, 6 filesdep +streamly-coredep ~bytestringdep ~streamlyPVP ok
version bump matches the API change (PVP)
Dependencies added: streamly-core
Dependency ranges changed: bytestring, streamly
API changes (from Hackage documentation)
Files
- ChangeLog.md +4/−0
- LICENSE +1/−1
- README.md +11/−6
- src/Streamly/External/LMDB.hs +2/−6
- streamly-lmdb.cabal +15/−14
- test/Streamly/External/LMDB/Tests.hs +2/−2
ChangeLog.md view
@@ -1,3 +1,7 @@+## 0.6.0++* Updated for Streamly 0.9.0.+ ## 0.5.0 * Added the ability to close databases and environments.
LICENSE view
@@ -1,4 +1,4 @@-Copyright Shlok Datye (c) 2022+Copyright Shlok Datye (c) 2023 All rights reserved.
README.md view
@@ -19,6 +19,9 @@ module Main where +import Data.Function ((&))+import qualified Streamly.Data.Fold as F+import qualified Streamly.Data.Stream.Prelude as S import Streamly.External.LMDB ( Limits (mapSize), WriteOptions (writeTransactionSize),@@ -31,7 +34,6 @@ tebibyte, writeLMDB, )-import qualified Streamly.Prelude as S main :: IO () main = do@@ -59,14 +61,17 @@ -- ("foo","b") let unfold' = readLMDB db Nothing defaultReadOptions let readStream = S.unfold unfold' undefined- S.mapM_ print readStream+ S.mapM print readStream+ & S.fold F.drain ``` ## Benchmarks -See `bench/README.md`. Summary (with rough figures from our machine<sup>†</sup>):+See `bench/README.md`. Summary (with rough figures from our machine<sup>†</sup> using GHC 9.2.7 [GHC 8.10.7]): -* **Reading.** For reading a fully cached LMDB database, this library (when `unsafeReadLMDB` is used instead of `readLMDB`) has roughly a 15 ns/pair overhead compared to plain Haskell `IO` code, which has roughly another 10 ns/pair overhead compared to C. (The first two being similar fulfills the promise of [streamly](https://hackage.haskell.org/package/streamly) and stream fusion.) We deduce that if your total workload per pair takes longer than around 25 ns, your bottleneck will not be your usage of this library as opposed to C.-* **Writing**. Writing with plain Haskell `IO` code and with this library is, respectively, around 30% and 50% slower than writing with C. We have not dug further into these differences because this write performance is currently good enough for our purposes.+* **Reading.** For reading a fully cached LMDB database, this library (when `unsafeReadLMDB` is used instead of `readLMDB`) has roughly a 5 ns/pair [15 ns/pair] overhead compared to plain Haskell `IO` code, which has roughly another 10 ns/pair overhead compared to C. (The first two being similar fulfills the promise of [streamly](https://hackage.haskell.org/package/streamly) and stream fusion.) We deduce that if your total workload per pair takes longer than around 15 ns [25 ns], your bottleneck will not be your usage of this library as opposed to C.+* **Writing**. Writing with plain Haskell `IO` code and with this library is, respectively, roughly 15% [30%] and 50% slower than writing with C. We have not dug further into these differences because this write performance is currently good enough for our purposes. -<sup>†</sup> [Linode](https://linode.com); Debian 10, Dedicated 32GB: 16 CPU, 640GB Storage, 32GB RAM.+(There have apparently been some performance improvements between GHC 8.10.7 and 9.2.7.)++<sup>†</sup> April 2023; [Linode](https://linode.com); Debian 11, Dedicated 32GB: 16 CPU, 640GB Storage, 32GB RAM.
src/Streamly/External/LMDB.hs view
@@ -30,11 +30,6 @@ -- * Database -- | After creating an environment, one creates within it one or more databases.- --- -- Note: We currently have no functionality here for closing and disposing of databases or- -- environments because we have had no need for it yet. In any case, it is a common practice- -- with LMDB to create one’s environments and databases once and reuse them for the remainder of- -- the program’s execution. Database, getDatabase, clearDatabase,@@ -126,7 +121,7 @@ import Streamly.Internal.Data.Fold (Fold (Fold), Step (Partial)) import Streamly.Internal.Data.IOFinalizer (newIOFinalizer, runIOFinalizer) import Streamly.Internal.Data.Stream.StreamD.Type (Step (Stop, Yield))-import Streamly.Internal.Data.Unfold (supply)+import Streamly.Internal.Data.Unfold (lmap) import Streamly.Internal.Data.Unfold.Type (Unfold (Unfold)) newtype Environment mode = Environment (Ptr MDB_env)@@ -314,6 +309,7 @@ (Forward, Just _) -> (mdb_set_range, mdb_next) (Backward, Nothing) -> (mdb_last, mdb_prev) (Backward, Just _) -> (mdb_set_range, mdb_prev)+ supply = lmap . const in supply firstOp $ Unfold ( \(op, pcurs, pk, pv, ref) -> do
streamly-lmdb.cabal view
@@ -1,20 +1,15 @@-cabal-version: 1.12---- This file has been generated from package.yaml by hpack version 0.34.6.------ see: https://github.com/sol/hpack-+cabal-version: 3.0 name: streamly-lmdb-version: 0.5.0+version: 0.6.0 synopsis: Stream data to or from LMDB databases using the streamly library. description: Please see the README on GitHub at <https://github.com/shlok/streamly-lmdb#readme> category: Database, Streaming, Streamly-homepage: https://github.com/shlok/streamly-lmdb#readme+homepage: https://github.com/shlok/streamly-lmdb bug-reports: https://github.com/shlok/streamly-lmdb/issues author: Shlok Datye maintainer: sd-haskell@quant.is-copyright: 2022 Shlok Datye-license: BSD3+copyright: 2023 Shlok Datye+license: BSD-3-Clause license-file: LICENSE build-type: Simple extra-source-files:@@ -32,6 +27,8 @@ Streamly.External.LMDB.Internal.Foreign other-modules: Paths_streamly_lmdb+ autogen-modules:+ Paths_streamly_lmdb hs-source-dirs: src ghc-options: -Wall@@ -46,8 +43,9 @@ build-depends: async >=2.2.2 && <2.3 , base >=4.7 && <5- , bytestring >=0.10.10.0 && <0.11- , streamly ==0.8.*+ , bytestring >=0.10.10.0 && <0.12+ , streamly ==0.9.*+ , streamly-core ==0.1.* default-language: Haskell2010 test-suite streamly-lmdb-test@@ -56,6 +54,8 @@ other-modules: Streamly.External.LMDB.Tests Paths_streamly_lmdb+ autogen-modules:+ Paths_streamly_lmdb hs-source-dirs: test ghc-options: -Wall -threaded -rtsopts -with-rtsopts=-N@@ -65,9 +65,10 @@ QuickCheck >=2.13.2 && <2.15 , async >=2.2.2 && <2.3 , base >=4.7 && <5- , bytestring >=0.10.10.0 && <0.11+ , bytestring >=0.10.10.0 && <0.12 , directory >=1.3.6.0 && <1.4- , streamly ==0.8.*+ , streamly ==0.9.*+ , streamly-core ==0.1.* , streamly-lmdb , tasty >=1.2.3 && <1.5 , tasty-quickcheck >=0.10.1.1 && <0.11
test/Streamly/External/LMDB/Tests.hs view
@@ -41,8 +41,8 @@ mdb_txn_begin, mdb_txn_commit, )-import Streamly.Prelude (fromList, toList, unfold)-import qualified Streamly.Prelude as S+import Streamly.Data.Stream.Prelude (fromList, toList, unfold)+import qualified Streamly.Data.Stream.Prelude as S import Test.QuickCheck (Gen, NonEmptyList (..), choose, elements, frequency) import Test.QuickCheck.Monadic (PropertyM, monadicIO, pick, run) import Test.Tasty (TestTree)