csv-conduit 0.6.3 → 0.6.5

raw patch · 6 files changed

+167/−149 lines, 6 filesdep −attoparsec-conduitdep ~resourcetPVP ok

version bump matches the API change (PVP)

Dependencies removed: attoparsec-conduit

Dependency ranges changed: resourcet

API changes (from Hackage documentation)

Files

• README.markdown +0/−120
• README.md +120/−0
• csv-conduit.cabal +7/−8
• src/Data/CSV/Conduit.hs +26/−7
• src/Data/CSV/Conduit/Conversion.hs +2/−2
• src/Data/CSV/Conduit/Parser/ByteString.hs +12/−12

@@ -1,120 +0,0 @@-# README--## CSV Files and Haskell--CSV files are the de-facto standard in many cases of data transfer,-particularly when dealing with enterprise application or disparate database-systems.--While there are a number of csv libraries in Haskell, at the time of-this project's start, there wasn't one that provided all of the-following:--* Full flexibility in quote characters, separators, input/output-* Constant space operation-* Robust parsing and error resiliency-* Battle-tested reliability in real-world datasets-* Fast operation-* Convenient interface that supports a variety of use cases--Over time, people created other plausible CSV packages like cassava.-The major benefit from this library remains to be:--* Direct participation in the conduit ecosystem, which is now quite-  large, and all the benefits that come with it.-* Flexibility in CSV format definition.-* Resiliency to errors in the input data.---## This package--csv-conduit is a conduit-based CSV parsing library that is easy to-use, flexible and fast. It leverages the conduit infrastructure to-provide constant-space operation, which is quite critical in many real-world use cases.--For example, you can use http-conduit to download a CSV file from the-internet and plug its Source into intoCSV to stream-convert the-download into the Row data type and do something with it as the data-streams, that is without having to download the entire file to disk-first.---## Author & Contributors--- Ozgun Ataman (@ozataman)-- Daniel Bergey (@bergey)-- BJTerry (@BJTerry)-- Mike Craig (@mkscrg)-- Daniel Corson (@dancor)-- Dmitry Dzhus (@dzhus)-- Niklas Hambüchen (@nh2)---### Introduction--* The CSVeable typeclass implements the key operations.-* CSVeable is parameterized on both a stream type and a target CSV row type.-* There are 2 basic row types and they implement *exactly* the same operations,-  so you can chose the right one for the job at hand:-  - type MapRow t = Map t t-  - type Row t = [t]-* You basically use the Conduits defined in this library to do the-  parsing from a CSV stream and rendering back into a CSV stream.-* Use the full flexibility and modularity of conduits for sources and sinks.--### Speed--While fast operation is of concern, I have so far cared more about correct-operation and a flexible API. Please let me know if you notice any performance-regressions or optimization opportunities.---### Usage Examples---#### Example #1: Basics Using Convenience API--    {-# LANGUAGE OverloadedStrings #-}--    import Data.Conduit-    import Data.Conduit.Binary-    import Data.Conduit.List as CL-    import Data.CSV.Conduit-    import Data.Text (Text)-    -    -- Just reverse te columns-    myProcessor :: Monad m => Conduit (Row Text) m (Row Text)-    myProcessor = CL.map reverse-    -    test :: IO ()-    test = runResourceT $ -      transformCSV defCSVSettings -                   (sourceFile "input.csv") -                   myProcessor-                   (sinkFile "output.csv")---#### Example #2: Basics Using Conduit API--    {-# LANGUAGE OverloadedStrings #-}--    import Data.Conduit-    import Data.Conduit.Binary-    import Data.CSV.Conduit-    import Data.Text (Text)--    myProcessor :: Conduit (Row Text) m (Row Text)-    myProcessor = undefined-    -    -- Let's simply stream from a file, parse the CSV, reserialize it-    -- and push back into another file.-    test :: IO ()-    test = runResourceT $ -      sourceFile "test/BigFile.csv" $= -      intoCSV defCSVSettings $=-      myProcessor $=-      fromCSV defCSVSettings $$-      sinkFile "test/BigFileOut.csv"--

@@ -0,0 +1,120 @@+# README [![Build Status](https://travis-ci.org/ozataman/csv-conduit.svg?branch=master)](https://travis-ci.org/ozataman/csv-conduit)++## CSV Files and Haskell++CSV files are the de-facto standard in many cases of data transfer,+particularly when dealing with enterprise application or disparate database+systems.++While there are a number of csv libraries in Haskell, at the time of+this project's start, there wasn't one that provided all of the+following:++* Full flexibility in quote characters, separators, input/output+* Constant space operation+* Robust parsing and error resiliency+* Battle-tested reliability in real-world datasets+* Fast operation+* Convenient interface that supports a variety of use cases++Over time, people created other plausible CSV packages like cassava.+The major benefit from this library remains to be:++* Direct participation in the conduit ecosystem, which is now quite+  large, and all the benefits that come with it.+* Flexibility in CSV format definition.+* Resiliency to errors in the input data.+++## This package++csv-conduit is a conduit-based CSV parsing library that is easy to+use, flexible and fast. It leverages the conduit infrastructure to+provide constant-space operation, which is quite critical in many real+world use cases.++For example, you can use http-conduit to download a CSV file from the+internet and plug its Source into intoCSV to stream-convert the+download into the Row data type and do something with it as the data+streams, that is without having to download the entire file to disk+first.+++## Author & Contributors++- Ozgun Ataman (@ozataman)+- Daniel Bergey (@bergey)+- BJTerry (@BJTerry)+- Mike Craig (@mkscrg)+- Daniel Corson (@dancor)+- Dmitry Dzhus (@dzhus)+- Niklas Hambüchen (@nh2)+++### Introduction++* The CSVeable typeclass implements the key operations.+* CSVeable is parameterized on both a stream type and a target CSV row type.+* There are 2 basic row types and they implement *exactly* the same operations,+  so you can chose the right one for the job at hand:+  - type MapRow t = Map t t+  - type Row t = [t]+* You basically use the Conduits defined in this library to do the+  parsing from a CSV stream and rendering back into a CSV stream.+* Use the full flexibility and modularity of conduits for sources and sinks.++### Speed++While fast operation is of concern, I have so far cared more about correct+operation and a flexible API. Please let me know if you notice any performance+regressions or optimization opportunities.+++### Usage Examples+++#### Example #1: Basics Using Convenience API++    {-# LANGUAGE OverloadedStrings #-}++    import Data.Conduit+    import Data.Conduit.Binary+    import Data.Conduit.List as CL+    import Data.CSV.Conduit+    import Data.Text (Text)+    +    -- Just reverse te columns+    myProcessor :: Monad m => Conduit (Row Text) m (Row Text)+    myProcessor = CL.map reverse+    +    test :: IO ()+    test = runResourceT $ +      transformCSV defCSVSettings +                   (sourceFile "input.csv") +                   myProcessor+                   (sinkFile "output.csv")+++#### Example #2: Basics Using Conduit API++    {-# LANGUAGE OverloadedStrings #-}++    import Data.Conduit+    import Data.Conduit.Binary+    import Data.CSV.Conduit+    import Data.Text (Text)++    myProcessor :: Conduit (Row Text) m (Row Text)+    myProcessor = undefined+    +    -- Let's simply stream from a file, parse the CSV, reserialize it+    -- and push back into another file.+    test :: IO ()+    test = runResourceT $ +      sourceFile "test/BigFile.csv" $= +      intoCSV defCSVSettings $=+      myProcessor $=+      fromCSV defCSVSettings $$+      sinkFile "test/BigFileOut.csv"++

@@ -1,5 +1,5 @@ Name:                csv-conduit-Version:             0.6.3+Version:             0.6.5 Synopsis:            A flexible, fast, conduit-based CSV parser library for Haskell. Homepage:            http://github.com/ozataman/csv-conduit License:             BSD3@@ -55,7 +55,7 @@   extra-source-files:-  README.markdown+  README.md   test/test.csv   test/Test.hs   test/Bench.hs@@ -73,13 +73,12 @@   ghc-options: -Wall -funbox-strict-fields   hs-source-dirs: src   build-depends:-      attoparsec >= 0.10-    , attoparsec-conduit >= 0.5.0.2-    , base >= 4 && < 5+      attoparsec             >= 0.10+    , base                   >= 4 && < 5     , bytestring-    , conduit >= 1.0 && < 2.0+    , conduit                >= 1.0 && < 2.0     , conduit-extra-    , containers >= 0.3+    , containers             >= 0.3     , monad-control     , text     , data-default@@ -91,7 +90,7 @@     , mtl     , mmorph     , primitive-    , resourcet+    , resourcet              >= 1.1.2.1   ghc-prof-options: -fprof-auto    if impl(ghc >= 7.2.1)

@@ -15,6 +15,7 @@     , readCSVFile     , writeCSVFile     , transformCSV+    , transformCSV'     , mapCSVFile     , writeHeaders @@ -31,7 +32,6 @@  ------------------------------------------------------------------------------- import           Control.Exception- import           Control.Monad.Morph import           Control.Monad.Primitive import           Control.Monad.ST@@ -362,6 +362,23 @@   -------------------------------------------------------------------------------+-- | Like transformCSV' but uses the same settings for both input and+-- output.+transformCSV+    :: (MonadThrow m, CSV s a, CSV s' b)+    => CSVSettings+    -- ^ Settings to be used for both input and output+    -> Source m s+    -- ^ A raw stream data source. Ex: 'sourceFile inFile'+    -> Conduit a m b+    -- ^ A transforming conduit+    -> Sink s' m ()+    -- ^ A raw stream data sink. Ex: 'sinkFile outFile'+    -> m ()+transformCSV set = transformCSV' set set+++------------------------------------------------------------------------------- -- | General purpose CSV transformer. Apply a list-like processing -- function from 'Data.Conduit.List' to the rows of a CSV stream. You -- need to provide a stream data source, a transformer and a stream@@ -372,11 +389,13 @@ -- -- Example - map a function over the rows of a CSV file: ----- > transformCSV set (sourceFile inFile) (C.map f) (sinkFile outFile)-transformCSV+-- > transformCSV setIn setOut (sourceFile inFile) (C.map f) (sinkFile outFile)+transformCSV'     :: (MonadThrow m, CSV s a, CSV s' b)     => CSVSettings-    -- ^ Settings to be used for both input and output+    -- ^ Settings to be used for input+    -> CSVSettings+    -- ^ Settings to be used for output     -> Source m s     -- ^ A raw stream data source. Ex: 'sourceFile inFile'     -> Conduit a m b@@ -384,11 +403,11 @@     -> Sink s' m ()     -- ^ A raw stream data sink. Ex: 'sinkFile outFile'     -> m ()-transformCSV set source c sink =+transformCSV' setIn setOut source c sink =     source $=-    intoCSV set $=+    intoCSV setIn $=     c $=-    fromCSV set $$+    fromCSV setOut $$     sink  

@@ -55,8 +55,8 @@ import Control.Applicative (Alternative, Applicative, (<*>), (<$>), (<|>),                                        empty, pure) import Control.Monad (MonadPlus, mplus, mzero)-import Data.Attoparsec.Char8 (double, parseOnly)-import qualified Data.Attoparsec.Char8 as A8+import Data.Attoparsec.ByteString.Char8 (double, parseOnly)+import qualified Data.Attoparsec.ByteString.Char8 as A8 import qualified Data.ByteString as B import qualified Data.ByteString.Char8 as B8 import qualified Data.ByteString.Lazy as L

@@ -1,4 +1,4 @@-{-| +{-|    This module exports the underlying Attoparsec row parser. This is helpful if   you want to do some ad-hoc CSV string parsing.@@ -14,12 +14,12 @@  ------------------------------------------------------------------------------- import           Control.Applicative-import           Control.Monad (mzero)-import           Data.Attoparsec as P hiding (take)-import qualified Data.Attoparsec.Char8 as C8-import           Data.ByteString.Char8 (ByteString)-import qualified Data.ByteString.Char8 as B8-import           Data.Word (Word8)+import           Control.Monad                    (mzero)+import           Data.Attoparsec.ByteString       as P hiding (take)+import qualified Data.Attoparsec.ByteString.Char8 as C8+import           Data.ByteString.Char8            (ByteString)+import qualified Data.ByteString.Char8            as B8+import           Data.Word                        (Word8) ------------------------------------------------------------------------------- import           Data.CSV.Conduit.Types @@ -60,11 +60,11 @@   badrow :: Parser (Maybe (Row ByteString))-badrow = P.takeWhile (not . C8.isEndOfLine) *> +badrow = P.takeWhile (not . C8.isEndOfLine) *>          (C8.endOfLine <|> C8.endOfInput) *> return Nothing  csvrow :: CSVSettings -> Parser (Maybe (Row ByteString))-csvrow c = +csvrow c =   let rowbody = (quotedField' <|> field c) `sepBy` C8.char (csvSep c)       properrow = rowbody <* (C8.endOfLine <|> P.endOfInput)       quotedField' = case csvQuoteChar c of@@ -75,17 +75,17 @@     return $ Just res  field :: CSVSettings -> Parser ByteString-field s = P.takeWhile (isFieldChar s) +field s = P.takeWhile (isFieldChar s)  isFieldChar :: CSVSettings -> Word8 -> Bool isFieldChar s = notInClass xs'   where xs = csvSep s : "\n\r"-        xs' = case csvQuoteChar s of +        xs' = case csvQuoteChar s of           Nothing -> xs           Just x -> x : xs  quotedField :: Char -> Parser ByteString-quotedField c = +quotedField c =   let quoted = string dbl *> return c       dbl = B8.pack [c,c]   in do