cassava-conduit 0.0.1 → 0.1.0

raw patch · 4 files changed

+195/−6 lines, 4 filesPVP ok

version bump matches the API change (PVP)

API changes (from Hackage documentation)

Files

• CHANGELOG.md +8/−0
• README.md +142/−0
• cassava-conduit.cabal +4/−4
• src/Data/Csv/Conduit.hs +41/−2

@@ -0,0 +1,8 @@+# 0.1.x++## 0.0.1 -> 0.1.0++``` Haskell+fromNamedCsv :: (Show a, Monad m, FromNamedRecord a, MonadError CsvParseError m) => DecodeOptions -> Conduit BS.ByteString m a+fromNamedCsvStreamError :: (Monad m, FromNamedRecord a) => DecodeOptions -> Conduit BS.ByteString m (Either CsvParseError a)+```

@@ -0,0 +1,142 @@+# cassava-conduit [![Build Status](https://img.shields.io/travis/domdere/cassava-conduit.svg?style=flat)](https://travis-ci.org/domdere/cassava-conduit) [![Hackage](https://img.shields.io/hackage/v/cassava-conduit.svg?style=flat)](https://hackage.haskell.org/package/cassava-conduit)++Conduit interface for cassava package++Streaming to CSV is not 100% complete at this stage, and doesn't support encoding to CSV with a header yet++## Example Usage++### The examples project++There is a project containing some examples of the usage, but the gist is here:++``` Haskell++import Data.Csv+import Data.Conduit+import Data.Csv.Conduit++data InputRecord = ...++instance FromRecord InputRecord where+    ...++data OutputRecord = ...++instance ToRecord OutputRecord where+    ...++decodeOpts :: Word8 -> DecodeOptions++encodeOpts :: Word8 -> EncodeOptions++processInput :: InputRecord -> OutputRecord++-- |+--  A Conduit pipeline that streams from '../exampledata/sampleinput.psv', decodes it from a pipe seperated format,+--  processes it with 'processInput' and the encodes it to pipe seperated format and streams it out to '../exampledata/sampleoutput.psv'+--  The first time it encounters a parse error, it will stop streaming and return the error, dropping any decoded records that came through in that batch also...+--+conduitPipeline :: (MonadError CsvParseError m, MonadResource m) => m ()+conduitPipeline = sourceFile "../exampledata/sampleinput.psv" $$ fromCsv (decodeOpts $ fromIntegral $ ord '|') HasHeader =$= map processInput =$= toCsv (encodeOpts $ fromIntegral $ ord '|') =$= sinkFile "../exampledata/sampleoutput.psv"++main :: IO ()+main = do+    res <- runEitherT $ bimapEitherT showError id $ runResourceT conduitPipeline+    either putStrLn return res+```++#### Building the examples project++```+$ cd examples+$ cabal sandbox init+$ cabal sandbox add-source ../+$ cabal install --only-dependencies+$ cabal build+```++## Building the project++Install the dependencies first with either:++    cabal install --only-dependencies++If you do not wish to build tests or benchmarks, or:++    cabal install --only-dependencies --enable-tests++If you want to be able to build the tests, or:++    cabal install --only-dependencies --enable-benchmarks++If you wish to build the benchmarks.++The project must be "configured" at least once everytime `cassava-conduit.cabal` changes, this can be done with:++    cabal configure++If you wish to run the unit tests you will have to run:++    cabal configure --enable-tests++If you wish to run benchmarks you will have to run:++    cabal configure --enable-benchmarks++At the moment there are issues with using both flags at the same time.  Its recommended that you use one flag at a time, use `cabal-dev` or `cabal sandbox` +(see below), and clear your sandbox when switching configurations from one to the other++Then finally build it with:++    cabal build++See `cabal build --help` for more build options.++## Running Unit Tests++**After** running `cabal build`, you can run the unit tests with the command:++    cabal test++## Adding Unit tests++Unit tests are written with [**doctest**] [doctest-github], for instructions on how to add unit tests+see the **doctest** [**User Guide**] [doctest-userguide].++Currently only files in the `src/` directory are searched for tests, it is assumed that the code in `main/`+is a thin layer of code that uses modules from `src/`.++## Running Benchmarks++**After** running `cabal configure --enable-benchmarks` and `cabal build`, the following command will run the benchmarks:++    cabal bench++For newer versions of `cabal`, `cabal bench` will run a `cabal build` automatically if necessary..++## Development: Cabal Dependency Hell?++Cabal's great, but its got its own warts, and when you are developing a few different projects with their own dependency chains, sometimes installing all your libraries to the same place causes problems,++### Cabal version < 1.18++Consider trying [`cabal-dev`] [cabal-dev].  Install it with `cabal install cabal-dev`++In terms of using it, all thats required is replacing `cabal` with `cabal-dev` in all the above command lines.++It will download and install all the dependencies for your project and install them in a `cabal-dev/` directory in your project directory, and they will only be used for this project.++### Cabal version >= 1.18++Cabal version `1.18` and onwards supports sandboxes, which is basically the same idea as `cabal-dev`.++In terms of using it all the commands remain the same, just run `cabal sandbox init` in the root directory of the project before running any of them.++------++The related `cabal-dev` and `sandbox` artifacts are already contained in the `.gitignore` file.++[cabal-dev]: https://github.com/creswick/cabal-dev "creswick/cabal-dev on GitHub.com"+[doctest-github]: https://github.com/sol/doctest-haskell "sol/doctest-haskell on GitHub.com"+[doctest-userguide]: https://github.com/sol/doctest-haskell/blob/master/README.markdown#usage "doctest Usage Guide"

@@ -1,5 +1,5 @@ name:               cassava-conduit-version:            0.0.1+version:            0.1.0 license:            BSD3 license-file:       etc/LICENCE.md author:             Dom De Re@@ -8,12 +8,12 @@ synopsis:           Conduit interface for cassava package category:           Data description:        Conduit interface for cassava package-homepage:           https://github.com/domdere/cassava-conduit/issues+homepage:           https://github.com/domdere/cassava-conduit bug-reports:        https://github.com/domdere/cassava-conduit/issues cabal-version:      >= 1.18 build-type:         Custom---extra-source-files: etc/CONTRIBUTORS,---                    etc/CREDITS+extra-source-files: README.md+                ,   CHANGELOG.md  source-repository       head     type:               git

@@ -14,7 +14,9 @@         CsvParseError(..)     -- * Conduits     ,   fromCsv+    ,   fromNamedCsv     ,   fromCsvStreamError+    ,   fromNamedCsvStreamError     ,   toCsv     ) where @@ -26,26 +28,55 @@ import qualified Data.ByteString.Lazy as BSL import Data.Conduit ( Conduit, await, yield ) import Data.Conduit.List ( map, mapM )-import Data.Csv ( FromRecord(..), ToRecord(..), DecodeOptions, EncodeOptions, HasHeader, encodeWith )-import Data.Csv.Incremental ( Parser(..), decodeWith )+import Data.Csv ( FromNamedRecord, FromRecord, ToRecord, DecodeOptions, EncodeOptions, HasHeader, encodeWith )+import Data.Csv.Incremental ( HeaderParser(..), Parser(..), decodeByNameWith, decodeWith ) import Data.Foldable ( mapM_ )  data CsvParseError =         CsvParseError BS.ByteString String     |   IncrementalError String +-- |+-- Streams parsed records, Errors are not received in the stream but instead after the pipeline is executed,+-- If you want to handle errors as they come and resume, see `fromCsvStreamError`+-- fromCsv :: (Show a, Monad m, FromRecord a, MonadError CsvParseError m) => DecodeOptions -> HasHeader -> Conduit BS.ByteString m a fromCsv opts h = {-# SCC fromCsv_p #-} terminatingStreamParser $ decodeWith opts h +-- |+-- Parses an instance of `FromNamedRecord`, this conduit drops the Header+--+-- Errors are not seen in the pipeline but rather at the end after executing the pipeline, if you want to handle the errors+-- as they occur, try `fromNamedCsvStreamError` instead.+--+fromNamedCsv :: (Show a, Monad m, FromNamedRecord a, MonadError CsvParseError m) => DecodeOptions -> Conduit BS.ByteString m a+fromNamedCsv opts = {-# SCC fromNamedCsv_p #-} terminatingStreamHeaderParser $ decodeByNameWith opts++-- |+-- Same as `fromCsv` but allows for errors to be handled in the pipeline instead+-- fromCsvStreamError :: (Monad m, FromRecord a) => DecodeOptions -> HasHeader -> Conduit BS.ByteString m (Either CsvParseError a) fromCsvStreamError opts h = {-# SCC fromCsvStreamError_p #-} streamParser $ decodeWith opts h +-- |+-- Like `fromNamedCsvStream` but allows for errors to be handled in the pipeline itself.+--+fromNamedCsvStreamError :: (Monad m, FromNamedRecord a) => DecodeOptions -> Conduit BS.ByteString m (Either CsvParseError a)+fromNamedCsvStreamError opts = {-# SCC fromCsvStreamError_p #-} streamHeaderParser $ decodeByNameWith opts +-- |+-- Streams from csv to text, does not create headers...+-- toCsv :: (Monad m, ToRecord a) => EncodeOptions -> Conduit a m BS.ByteString toCsv opts = {-# SCC toCsv_p #-} map $ BSL.toStrict . encodeWith opts . pure  -- helpers +streamHeaderParser :: (Monad m) => HeaderParser (Parser a) -> Conduit BS.ByteString m (Either CsvParseError a)+streamHeaderParser (FailH rest errMsg)  = {-# SCC streamHeaderParser_FailH_p #-} yield $ Left $ CsvParseError rest errMsg+streamHeaderParser (PartialH p)         = {-# SCC streamHeaderParser_PartialH_p #-} await >>= maybe (return ()) (streamHeaderParser . p)+streamHeaderParser (DoneH _ p)          = {-# SCC streamHeaderParser_DoneH_p #-} streamParser p+ streamParser :: (Monad m) => Parser a -> Conduit BS.ByteString m (Either CsvParseError a) streamParser (Fail rest errMsg) = yield $ Left $ CsvParseError rest errMsg streamParser (Many rs p) = do@@ -55,6 +86,14 @@     more <- await     maybe (return ()) (streamParser . p) more streamParser (Done rs) = mapM_ (yield . first IncrementalError) rs++terminatingStreamHeaderParser+    :: (Show a, Monad m, MonadError CsvParseError m)+    => HeaderParser (Parser a)+    -> Conduit BS.ByteString m a+terminatingStreamHeaderParser (FailH rest errMsg)   = {-# SCC terminatingStreamHeaderParser_FailH_p #-} mapM $ const $ throwError $ CsvParseError rest errMsg+terminatingStreamHeaderParser (PartialH p)          = {-# SCC terminatingStreamHeaderParser_PartialH_p #-} await >>= maybe (return ()) (terminatingStreamHeaderParser . p)+terminatingStreamHeaderParser (DoneH _ p)           = {-# SCC terminatingStreamHeaderParser_DoneH_p #-} terminatingStreamParser p  terminatingStreamParser     :: (Show a, Monad m, MonadError CsvParseError m)