esqueleto-streaming (empty) → 0.1.0.0

raw patch · 6 files changed

+139/−0 lines, 6 filesdep +basedep +conduitdep +esqueletosetup-changed

Dependencies added: base, conduit, esqueleto, persistent-postgresql, persistent-postgresql-streaming, resourcet, transformers

Files

CHANGELOG.md +6/−0
LICENSE +30/−0
README.md +17/−0
Setup.hs +2/−0
esqueleto-streaming.cabal +36/−0
src/Database/Esqueleto/PostgreSQL/Streaming.hs +48/−0

+ CHANGELOG.md view

@@ -0,0 +1,6 @@+# Changelog for esqueleto-streaming++# 0.1.0.0++  * Exposed `Database.Esqueleto.PostgreSQL.Streaming`+    * Added `selectCursor`

+ LICENSE view

@@ -0,0 +1,30 @@+Copyright Supercede Ltd. (c) 2021++All rights reserved.++Redistribution and use in source and binary forms, with or without+modification, are permitted provided that the following conditions are met:++    * Redistributions of source code must retain the above copyright+      notice, this list of conditions and the following disclaimer.++    * Redistributions in binary form must reproduce the above+      copyright notice, this list of conditions and the following+      disclaimer in the documentation and/or other materials provided+      with the distribution.++    * Neither the name of Isaac van Bakel nor the names of other+      contributors may be used to endorse or promote products derived+      from this software without specific prior written permission.++THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

+ README.md view

@@ -0,0 +1,17 @@+# esqueleto-streaming++This library allows for memory-constant streaming of the results of [`esqueleto`](https://hackage.haskell.org/package/esqueleto) queries from PostgreSQL databases.++This code makes use of the PostgreSQL-only [cursors](https://www.postgresql.org/docs/current/plpgsql-cursors.html), which allow for batched access to the result set of a query at speeds comparable to loading all the results into memory at once.++See the [main project README](https://github.com/SupercedeTech/persistent-postgresql-streaming/blob/main/README.md) for more.++## Streaming results from Esqueleto queries++The main function of this library is `selectCursor`, which can be used in place of [`select`](https://hackage.haskell.org/package/esqueleto-3.5.3.0/docs/Database-Esqueleto.html#v:select) to run an Esqueleto query. `selectCursor` runs in a `ConduitT` - consuming the conduit will pull results from the database in constant memory.++## FAQ++### Why is `selectCursor` so slow?++Have you configured PostgreSQL correctly? See the section in the [README](https://github.com/SupercedeTech/persistent-postgresql-streaming/blob/main/README.md) about it.

+ Setup.hs view

@@ -0,0 +1,2 @@+import Distribution.Simple+main = defaultMain

+ esqueleto-streaming.cabal view

@@ -0,0 +1,36 @@+name:                esqueleto-streaming+version:             0.1.0.0+synopsis:            Memory-constant streaming of Esqueleto results from PostgreSQL+description:+  This library provides a set of APIs for performing queries in Esqueleto+  in constant memory, streaming the results using @conduit@.+  .+  The library relies on PostgreSQL-specific features to avoid loading all the+  results of a query into memory at once. This allows for accessing results of+  millions of rows from Haskell without a memory blow-up.+homepage:            https://github.com/SupercedeTech/persistent-postgresql-streaming#readme+license:             BSD3+license-file:        LICENSE+author:              Isaac van Bakel+maintainer:          support@supercede.com+copyright:           (c) 2021 Supercede Ltd.+category:            Conduit, Database+build-type:          Simple+extra-source-files:  CHANGELOG.md README.md+cabal-version:       >=1.10++library+  hs-source-dirs:      src+  exposed-modules:     Database.Esqueleto.PostgreSQL.Streaming+  build-depends:       base >= 4.7 && < 5+                     , conduit >= 1 && < 1.4+                     , esqueleto >= 3.5 && < 3.6+                     , persistent-postgresql-streaming >= 0.1 && < 0.2+                     , persistent-postgresql >= 2.13.2 && < 2.14+                     , resourcet >= 0.3 && < 2+                     , transformers >= 0.5 && < 0.6+  default-language:    Haskell2010++source-repository head+  type:     git+  location: https://github.com/SupercedeTech/persistent-postgresql-streaming

+ src/Database/Esqueleto/PostgreSQL/Streaming.hs view

@@ -0,0 +1,48 @@+{-# LANGUAGE FlexibleContexts #-}+{-# LANGUAGE ScopedTypeVariables #-}++module Database.Esqueleto.PostgreSQL.Streaming+  ( selectCursor+  ) where++import Control.Monad.IO.Class (MonadIO)+import Control.Monad.Trans.Class (lift)+import Control.Monad.Trans.Reader (ReaderT(..), ask)+import Control.Monad.Trans.Resource (MonadResource)+import Data.Conduit (ConduitT)+import Data.Conduit.Lift (runReaderC)+import Database.Esqueleto.Internal.Internal+import Database.Persist.Postgresql+import Database.Persist.Postgresql.Streaming.Internal (rawSelectStream)++-- | Execute an @esqueleto@ @SELECT@ query against a PostgreSQL backend and+-- return a stream of the results in constant memory, using the PostgreSQL+-- cursor feature.+--+-- NB: this function is likely to perform worse than 'select' on small+-- result sets.+--+-- NB: for large result sets, this function is likely to perform very poorly+-- unless you manually configure the @cursor_tuple_fraction@ setting to be+-- close to @1@. This setting decides how much PostgreSQL will prioritise+-- returning the first rows quickly vs. returning all rows efficiently. See+--+-- https://postgresqlco.nf/doc/en/param/cursor_tuple_fraction/+--+-- for more.+selectCursor+  ::+    ( SqlSelect a r+    , MonadIO m+    , MonadResource m+    , BackendCompatible (RawPostgresql SqlBackend) backend+    )+  => SqlQuery a+  -> ConduitT () r (ReaderT backend m) ()+selectCursor query = do+  backend@(RawPostgresql conn _) <- lift $ projectBackend <$> ask+  let (queryTextBuilder, vals) = toRawSql SELECT (conn, initialIdentState) query+      queryText = builderToText queryTextBuilder+  runReaderC backend+    (rawSelectStream sqlSelectProcessRow queryText vals)+