diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,8 @@ +# 0.1.x + +## 0.0.1 -> 0.1.0 + +``` Haskell +fromNamedCsv :: (Show a, Monad m, FromNamedRecord a, MonadError CsvParseError m) => DecodeOptions -> Conduit BS.ByteString m a +fromNamedCsvStreamError :: (Monad m, FromNamedRecord a) => DecodeOptions -> Conduit BS.ByteString m (Either CsvParseError a) +``` diff --git a/README.md b/README.md new file mode 100644 --- /dev/null +++ b/README.md @@ -0,0 +1,142 @@ +# cassava-conduit [![Build Status](https://img.shields.io/travis/domdere/cassava-conduit.svg?style=flat)](https://travis-ci.org/domdere/cassava-conduit) [![Hackage](https://img.shields.io/hackage/v/cassava-conduit.svg?style=flat)](https://hackage.haskell.org/package/cassava-conduit) + +Conduit interface for cassava package + +Streaming to CSV is not 100% complete at this stage, and doesn't support encoding to CSV with a header yet + +## Example Usage + +### The examples project + +There is a project containing some examples of the usage, but the gist is here: + +``` Haskell + +import Data.Csv +import Data.Conduit +import Data.Csv.Conduit + +data InputRecord = ... + +instance FromRecord InputRecord where + ... + +data OutputRecord = ... + +instance ToRecord OutputRecord where + ... + +decodeOpts :: Word8 -> DecodeOptions + +encodeOpts :: Word8 -> EncodeOptions + +processInput :: InputRecord -> OutputRecord + +-- | +-- A Conduit pipeline that streams from '../exampledata/sampleinput.psv', decodes it from a pipe seperated format, +-- processes it with 'processInput' and the encodes it to pipe seperated format and streams it out to '../exampledata/sampleoutput.psv' +-- The first time it encounters a parse error, it will stop streaming and return the error, dropping any decoded records that came through in that batch also... +-- +conduitPipeline :: (MonadError CsvParseError m, MonadResource m) => m () +conduitPipeline = sourceFile "../exampledata/sampleinput.psv" $$ fromCsv (decodeOpts $ fromIntegral $ ord '|') HasHeader =$= map processInput =$= toCsv (encodeOpts $ fromIntegral $ ord '|') =$= sinkFile "../exampledata/sampleoutput.psv" + +main :: IO () +main = do + res <- runEitherT $ bimapEitherT showError id $ runResourceT conduitPipeline + either putStrLn return res +``` + +#### Building the examples project + +``` +$ cd examples +$ cabal sandbox init +$ cabal sandbox add-source ../ +$ cabal install --only-dependencies +$ cabal build +``` + +## Building the project + +Install the dependencies first with either: + + cabal install --only-dependencies + +If you do not wish to build tests or benchmarks, or: + + cabal install --only-dependencies --enable-tests + +If you want to be able to build the tests, or: + + cabal install --only-dependencies --enable-benchmarks + +If you wish to build the benchmarks. + +The project must be "configured" at least once everytime `cassava-conduit.cabal` changes, this can be done with: + + cabal configure + +If you wish to run the unit tests you will have to run: + + cabal configure --enable-tests + +If you wish to run benchmarks you will have to run: + + cabal configure --enable-benchmarks + +At the moment there are issues with using both flags at the same time. Its recommended that you use one flag at a time, use `cabal-dev` or `cabal sandbox` +(see below), and clear your sandbox when switching configurations from one to the other + +Then finally build it with: + + cabal build + +See `cabal build --help` for more build options. + +## Running Unit Tests + +**After** running `cabal build`, you can run the unit tests with the command: + + cabal test + +## Adding Unit tests + +Unit tests are written with [**doctest**] [doctest-github], for instructions on how to add unit tests +see the **doctest** [**User Guide**] [doctest-userguide]. + +Currently only files in the `src/` directory are searched for tests, it is assumed that the code in `main/` +is a thin layer of code that uses modules from `src/`. + +## Running Benchmarks + +**After** running `cabal configure --enable-benchmarks` and `cabal build`, the following command will run the benchmarks: + + cabal bench + +For newer versions of `cabal`, `cabal bench` will run a `cabal build` automatically if necessary.. + +## Development: Cabal Dependency Hell? + +Cabal's great, but its got its own warts, and when you are developing a few different projects with their own dependency chains, sometimes installing all your libraries to the same place causes problems, + +### Cabal version < 1.18 + +Consider trying [`cabal-dev`] [cabal-dev]. Install it with `cabal install cabal-dev` + +In terms of using it, all thats required is replacing `cabal` with `cabal-dev` in all the above command lines. + +It will download and install all the dependencies for your project and install them in a `cabal-dev/` directory in your project directory, and they will only be used for this project. + +### Cabal version >= 1.18 + +Cabal version `1.18` and onwards supports sandboxes, which is basically the same idea as `cabal-dev`. + +In terms of using it all the commands remain the same, just run `cabal sandbox init` in the root directory of the project before running any of them. + +------ + +The related `cabal-dev` and `sandbox` artifacts are already contained in the `.gitignore` file. + +[cabal-dev]: https://github.com/creswick/cabal-dev "creswick/cabal-dev on GitHub.com" +[doctest-github]: https://github.com/sol/doctest-haskell "sol/doctest-haskell on GitHub.com" +[doctest-userguide]: https://github.com/sol/doctest-haskell/blob/master/README.markdown#usage "doctest Usage Guide" diff --git a/cassava-conduit.cabal b/cassava-conduit.cabal --- a/cassava-conduit.cabal +++ b/cassava-conduit.cabal @@ -1,5 +1,5 @@ name: cassava-conduit -version: 0.0.1 +version: 0.1.0 license: BSD3 license-file: etc/LICENCE.md author: Dom De Re @@ -8,12 +8,12 @@ synopsis: Conduit interface for cassava package category: Data description: Conduit interface for cassava package -homepage: https://github.com/domdere/cassava-conduit/issues +homepage: https://github.com/domdere/cassava-conduit bug-reports: https://github.com/domdere/cassava-conduit/issues cabal-version: >= 1.18 build-type: Custom ---extra-source-files: etc/CONTRIBUTORS, --- etc/CREDITS +extra-source-files: README.md + , CHANGELOG.md source-repository head type: git diff --git a/src/Data/Csv/Conduit.hs b/src/Data/Csv/Conduit.hs --- a/src/Data/Csv/Conduit.hs +++ b/src/Data/Csv/Conduit.hs @@ -14,7 +14,9 @@ CsvParseError(..) -- * Conduits , fromCsv + , fromNamedCsv , fromCsvStreamError + , fromNamedCsvStreamError , toCsv ) where @@ -26,26 +28,55 @@ import qualified Data.ByteString.Lazy as BSL import Data.Conduit ( Conduit, await, yield ) import Data.Conduit.List ( map, mapM ) -import Data.Csv ( FromRecord(..), ToRecord(..), DecodeOptions, EncodeOptions, HasHeader, encodeWith ) -import Data.Csv.Incremental ( Parser(..), decodeWith ) +import Data.Csv ( FromNamedRecord, FromRecord, ToRecord, DecodeOptions, EncodeOptions, HasHeader, encodeWith ) +import Data.Csv.Incremental ( HeaderParser(..), Parser(..), decodeByNameWith, decodeWith ) import Data.Foldable ( mapM_ ) data CsvParseError = CsvParseError BS.ByteString String | IncrementalError String +-- | +-- Streams parsed records, Errors are not received in the stream but instead after the pipeline is executed, +-- If you want to handle errors as they come and resume, see `fromCsvStreamError` +-- fromCsv :: (Show a, Monad m, FromRecord a, MonadError CsvParseError m) => DecodeOptions -> HasHeader -> Conduit BS.ByteString m a fromCsv opts h = {-# SCC fromCsv_p #-} terminatingStreamParser $ decodeWith opts h +-- | +-- Parses an instance of `FromNamedRecord`, this conduit drops the Header +-- +-- Errors are not seen in the pipeline but rather at the end after executing the pipeline, if you want to handle the errors +-- as they occur, try `fromNamedCsvStreamError` instead. +-- +fromNamedCsv :: (Show a, Monad m, FromNamedRecord a, MonadError CsvParseError m) => DecodeOptions -> Conduit BS.ByteString m a +fromNamedCsv opts = {-# SCC fromNamedCsv_p #-} terminatingStreamHeaderParser $ decodeByNameWith opts + +-- | +-- Same as `fromCsv` but allows for errors to be handled in the pipeline instead +-- fromCsvStreamError :: (Monad m, FromRecord a) => DecodeOptions -> HasHeader -> Conduit BS.ByteString m (Either CsvParseError a) fromCsvStreamError opts h = {-# SCC fromCsvStreamError_p #-} streamParser $ decodeWith opts h +-- | +-- Like `fromNamedCsvStream` but allows for errors to be handled in the pipeline itself. +-- +fromNamedCsvStreamError :: (Monad m, FromNamedRecord a) => DecodeOptions -> Conduit BS.ByteString m (Either CsvParseError a) +fromNamedCsvStreamError opts = {-# SCC fromCsvStreamError_p #-} streamHeaderParser $ decodeByNameWith opts +-- | +-- Streams from csv to text, does not create headers... +-- toCsv :: (Monad m, ToRecord a) => EncodeOptions -> Conduit a m BS.ByteString toCsv opts = {-# SCC toCsv_p #-} map $ BSL.toStrict . encodeWith opts . pure -- helpers +streamHeaderParser :: (Monad m) => HeaderParser (Parser a) -> Conduit BS.ByteString m (Either CsvParseError a) +streamHeaderParser (FailH rest errMsg) = {-# SCC streamHeaderParser_FailH_p #-} yield $ Left $ CsvParseError rest errMsg +streamHeaderParser (PartialH p) = {-# SCC streamHeaderParser_PartialH_p #-} await >>= maybe (return ()) (streamHeaderParser . p) +streamHeaderParser (DoneH _ p) = {-# SCC streamHeaderParser_DoneH_p #-} streamParser p + streamParser :: (Monad m) => Parser a -> Conduit BS.ByteString m (Either CsvParseError a) streamParser (Fail rest errMsg) = yield $ Left $ CsvParseError rest errMsg streamParser (Many rs p) = do @@ -55,6 +86,14 @@ more <- await maybe (return ()) (streamParser . p) more streamParser (Done rs) = mapM_ (yield . first IncrementalError) rs + +terminatingStreamHeaderParser + :: (Show a, Monad m, MonadError CsvParseError m) + => HeaderParser (Parser a) + -> Conduit BS.ByteString m a +terminatingStreamHeaderParser (FailH rest errMsg) = {-# SCC terminatingStreamHeaderParser_FailH_p #-} mapM $ const $ throwError $ CsvParseError rest errMsg +terminatingStreamHeaderParser (PartialH p) = {-# SCC terminatingStreamHeaderParser_PartialH_p #-} await >>= maybe (return ()) (terminatingStreamHeaderParser . p) +terminatingStreamHeaderParser (DoneH _ p) = {-# SCC terminatingStreamHeaderParser_DoneH_p #-} terminatingStreamParser p terminatingStreamParser :: (Show a, Monad m, MonadError CsvParseError m)