wire-streams 0.0.2.0 → 0.1.0.0
raw patch · 9 files changed
+194/−308 lines, 9 filesdep +binary-parsersdep ~basedep ~binarydep ~bytestring
Dependencies added: binary-parsers
Dependency ranges changed: base, binary, bytestring, cereal, io-streams
Files
- CHANGELOG +12/−0
- CHANGElOG +0/−7
- README.md +32/−39
- bench/System/IO/Streams/Cereal.hs +126/−0
- dist/build/encode-decode-cerealStub/encode-decode-cerealStub-tmp/encode-decode-cerealStub.hs +0/−5
- src/System/IO/Streams/Binary.hs +8/−15
- src/System/IO/Streams/Cereal.hs +0/−126
- test/EncodeDecodeCereal.hs +0/−87
- wire-streams.cabal +16/−29
+ CHANGELOG view
@@ -0,0 +1,12 @@+# v0.1.0.0+++ Using binary-parsers package which speeds up binary code, remove cereal support since binary is much faster now.++ DecodeException now contain unconsumed buffer.++# v0.0.2.0++Add binary support, unify API, optimize cereal's putToStream/putOutputStream, add benchmark.++# v0.0.1.0++Clean up code, rewrite main parsing function, fix broken benchmark, and add more functions.
− CHANGElOG
@@ -1,7 +0,0 @@-# v0.0.2.0--Add binary support, unify API, optimize cereal's putToStream/putOutputStream, add benchmark.--# v0.0.1.0--Clean up code, rewrite main parsing function, fix broken benchmark, and add more functions.
README.md view
@@ -4,60 +4,53 @@ [](http://hackage.haskell.org/package/wire-streams) [](https://travis-ci.org/winterland1989/wire-streams) -One stop solution to serialize/deserialize [io-streams](http://hackage.haskell.org/package/io-streams):--+ `System.IO.Streams.Cereal` use [cereal](http://hackage.haskell.org/package/cereal) to serialize/deserialize, cereal provides sanner default to `Double`(IEEE-754), and `ShortByteString` support.--+ `System.IO.Streams.Binary` use [binary](http://hackage.haskell.org/package/binary) to serialize/deserialize, binary provide some useful helpers currently not available in cereal(`getLazyByteStringNul`).--This package is rewritten from [cereal-io-streams](https://github.com/Soostone/cereal-io-streams) and [binary-streams](https://github.com/jonpetterbergman/binary-streams) with following changes:+Serialize/deserialize `ByteString` streams from [io-streams](http://hackage.haskell.org/package/io-streams) using [binary](http://hackage.haskell.org/package/binary) package. -+ Completely rewrite cereal/io-streams adapter.-+ Clean and unify APIs. -+ Add more test and benchmark.+From v0.1 cereal is removed because [a binary performance problem](https://github.com/winterland1989/binary-parsers/blob/master/Data/Binary/Parser.hs#L187) is taken care of by [binary-parsers](https://github.com/winterland1989/binary-parsers) package, now binary is much faster and more featureful. -Both cereal and binary are top notch serialize/deserialize libaries, you wouldn't go wrong with either choice. This package mainly serve my purpose to develop native mysql adapter, but also provide a benchmark/comparsion across cereal and binary. here's benchmark result against [cereal-conduit](http://hackage.haskell.org/package/cereal-conduit):+Benchmark+--------- ``` benchmarking decode one element wire-streams/cereal/1000 items-time 126.7 ns (125.1 ns .. 128.2 ns)+time 127.6 ns (126.4 ns .. 129.2 ns) 0.999 R² (0.998 R² .. 0.999 R²)-mean 127.4 ns (126.1 ns .. 128.9 ns)-std dev 4.887 ns (4.122 ns .. 6.214 ns)-variance introduced by outliers: 58% (severely inflated)+mean 128.1 ns (126.6 ns .. 130.1 ns)+std dev 6.019 ns (4.441 ns .. 9.245 ns)+variance introduced by outliers: 67% (severely inflated) benchmarking decode one element wire-streams/binary/1000 items-time 218.4 ns (216.8 ns .. 220.0 ns)- 0.999 R² (0.999 R² .. 1.000 R²)-mean 217.5 ns (215.8 ns .. 219.2 ns)-std dev 5.588 ns (4.589 ns .. 7.044 ns)-variance introduced by outliers: 37% (moderately inflated)+time 90.49 ns (89.77 ns .. 91.11 ns)+ 0.999 R² (0.999 R² .. 0.999 R²)+mean 90.42 ns (89.44 ns .. 91.41 ns)+std dev 3.371 ns (2.711 ns .. 4.433 ns)+variance introduced by outliers: 57% (severely inflated) benchmarking decode one element cereal-conduit/1000 items-time 318.5 ns (314.7 ns .. 322.1 ns)- 0.999 R² (0.999 R² .. 0.999 R²)-mean 319.2 ns (316.1 ns .. 322.7 ns)-std dev 11.37 ns (8.824 ns .. 15.09 ns)-variance introduced by outliers: 53% (severely inflated)+time 345.2 ns (325.0 ns .. 382.3 ns)+ 0.953 R² (0.896 R² .. 0.999 R²)+mean 338.6 ns (329.5 ns .. 379.4 ns)+std dev 52.15 ns (14.85 ns .. 114.3 ns)+variance introduced by outliers: 95% (severely inflated) benchmarking decode 1000 elements from wire-streams/cereal/1000 items-time 99.61 μs (98.56 μs .. 100.9 μs)- 0.997 R² (0.994 R² .. 0.999 R²)-mean 100.4 μs (98.83 μs .. 102.5 μs)-std dev 6.321 μs (4.136 μs .. 9.830 μs)-variance introduced by outliers: 64% (severely inflated)+time 97.24 μs (95.95 μs .. 98.57 μs)+ 0.999 R² (0.998 R² .. 0.999 R²)+mean 97.75 μs (96.78 μs .. 98.68 μs)+std dev 3.170 μs (2.640 μs .. 3.906 μs)+variance introduced by outliers: 31% (moderately inflated) benchmarking decode 1000 elements from wire-streams/binary/1000 items-time 189.3 μs (187.0 μs .. 191.5 μs)+time 65.24 μs (64.69 μs .. 65.94 μs) 0.999 R² (0.998 R² .. 0.999 R²)-mean 189.3 μs (187.5 μs .. 190.9 μs)-std dev 5.868 μs (4.966 μs .. 7.174 μs)-variance introduced by outliers: 27% (moderately inflated)+mean 66.09 μs (65.35 μs .. 67.62 μs)+std dev 3.601 μs (2.014 μs .. 6.391 μs)+variance introduced by outliers: 58% (severely inflated) benchmarking decode 1000 elements cereal-conduit/1000 items-time 203.3 μs (201.1 μs .. 205.7 μs)- 0.998 R² (0.996 R² .. 0.999 R²)-mean 204.0 μs (201.3 μs .. 207.9 μs)-std dev 10.38 μs (7.759 μs .. 14.73 μs)-variance introduced by outliers: 49% (moderately inflated)+time 198.4 μs (195.4 μs .. 201.3 μs)+ 0.998 R² (0.997 R² .. 0.999 R²)+mean 200.4 μs (197.9 μs .. 203.7 μs)+std dev 9.718 μs (7.707 μs .. 12.77 μs)+variance introduced by outliers: 47% (moderately inflated) ```
+ bench/System/IO/Streams/Cereal.hs view
@@ -0,0 +1,126 @@+{-# LANGUAGE DeriveDataTypeable #-}+{-# LANGUAGE OverloadedStrings #-}++-----------------------------------------------------------------------------+-- |+-- Module : Sytem.IO.Streams.Cereal+-- Copyright : Soostone Inc, Winterland+-- License : BSD3+--+-- Maintainer : Winterland+-- Stability : experimental+--+-- Use cereal to encode/decode io-streams.+----------------------------------------------------------------------------++module System.IO.Streams.Cereal (+ -- * single element encode/decode+ getFromStream+ , decodeFromStream+ , putToStream+ -- * 'InputStream' encode/decode+ , getInputStream+ , decodeInputStream+ -- * 'OutputStream' encode+ , putOutputStream+ , encodeOutputStream+ -- * exception type+ , DecodeException(..)+ ) where++-------------------------------------------------------------------------------++import Control.Exception (Exception, throwIO)+import Control.Monad (unless)+import Data.ByteString (ByteString)+import qualified Data.ByteString.Char8 as S+import Data.Serialize+import Data.Typeable+import qualified System.IO.Streams as Streams+import System.IO.Streams.Core++-------------------------------------------------------------------------------++-- | An Exception raised when cereal decoding fails.+data DecodeException = DecodeException String+ deriving (Typeable)++instance Show DecodeException where+ show (DecodeException s) = "System.IO.Streams.Cereal: cereal decode exception: " ++ s++instance Exception DecodeException++-------------------------------------------------------------------------------++-- | write a instance of 'Serialize' to an 'OutputStream'+--+putToStream :: Serialize a => Maybe a -> OutputStream ByteString -> IO ()+putToStream Nothing = Streams.write Nothing+putToStream (Just a) = (Streams.writeLazyByteString . runPutLazy . put) a+{-# INLINE putToStream #-}++-------------------------------------------------------------------------------++-- | Take a 'Get' and an 'InputStream' and decode a+-- value. Consumes only as much input as necessary to decode the+-- value. Unconsumed input will be unread. If there is+-- an error while deserializing, a 'DecodeException' is thrown, and+-- unconsumed part will be unread. To simplify upstream generation,+-- all empty 'ByteString' will be filtered out and not passed to cereal,+-- only EOFs/Nothing will close a cereal decoder.+--+-- Examples:+--+-- >>> import qualified System.IO.Streams as Streams+-- >>> getFromStream (get :: Get String) =<< Streams.fromByteString (Data.ByteString.drop 1 $ runPut $ put "encode me")+-- *** Exception: System.IO.Streams.Cereal: cereal decode exception: too few bytes+-- From: demandInput+-- <BLANKLINE>+--+getFromStream :: Get a -> InputStream ByteString -> IO (Maybe a)+getFromStream g is =+ Streams.read is >>= maybe (return Nothing) (go . runGetPartial g)+ where+ go (Fail msg s) = do+ unless (S.null s) (Streams.unRead s is)+ throwIO (DecodeException msg)+ go (Done r s) = do+ unless (S.null s) (Streams.unRead s is)+ return (Just r)+ go c@(Partial cont) =+ Streams.read is >>= maybe (go (cont S.empty)) -- use 'empty' to notify cereal ending.+ (\ s -> if S.null s then go c else go (cont s))+{-# INLINE getFromStream #-}++-- | typeclass version of 'getFromStream'+decodeFromStream :: Serialize a => InputStream ByteString -> IO (Maybe a)+decodeFromStream = getFromStream get+{-# INLINE decodeFromStream #-}++-------------------------------------------------------------------------------++-- | Convert a stream of individual encoded 'ByteString's to a stream+-- of Results. Throws a 'DecodeException' on error.+getInputStream :: Get a -> InputStream ByteString -> IO (InputStream a)+getInputStream g is = makeInputStream (getFromStream g is)+{-# INLINE getInputStream #-}++-- | typeclass version of 'getInputStream'+decodeInputStream :: Serialize a => InputStream ByteString -> IO (InputStream a)+decodeInputStream = getInputStream get+{-# INLINE decodeInputStream #-}++-------------------------------------------------------------------------------++-- | create an 'OutputStream' of serializable values from an 'OutputStream'+-- of bytestrings with a 'Putter'.+putOutputStream :: Putter a -> OutputStream ByteString -> IO (OutputStream a)+putOutputStream p os = Streams.makeOutputStream $ \ ma ->+ case ma of Nothing -> Streams.write Nothing os+ Just a -> Streams.writeLazyByteString (runPutLazy (p a)) os+{-# INLINE putOutputStream #-}++-- | typeclass version of 'putOutputStream'+encodeOutputStream :: Serialize a => OutputStream ByteString -> IO (OutputStream a)+encodeOutputStream = putOutputStream put+{-# INLINE encodeOutputStream #-}
− dist/build/encode-decode-cerealStub/encode-decode-cerealStub-tmp/encode-decode-cerealStub.hs
@@ -1,5 +0,0 @@-module Main ( main ) where-import Distribution.Simple.Test.LibV09 ( stubMain )-import EncodeDecodeCereal ( tests )-main :: IO ()-main = stubMain tests
src/System/IO/Streams/Binary.hs view
@@ -33,9 +33,8 @@ import Control.Exception (Exception, throwIO) import Control.Monad (unless) import Data.Binary (Binary, get, put)-import Data.Binary.Get (ByteOffset, Decoder (..), Get,- pushChunk, pushEndOfInput,- runGetIncremental)+import qualified Data.Binary.Parser as P+import Data.Binary.Get (ByteOffset, Decoder(..), Get) import Data.Binary.Put (runPut, Put) import Data.ByteString (ByteString) import qualified Data.ByteString as S@@ -49,12 +48,12 @@ -- | An Exception raised when binary decoding fails. -- -- it contains offset information where cereal don't.-data DecodeException = DecodeException ByteOffset String+data DecodeException = DecodeException ByteString ByteOffset String deriving (Typeable) instance Show DecodeException where- show (DecodeException offset message) =- "System.IO.Streams.Binary: binary decode exception: offset " ++ show offset ++ ", " ++ show message+ show (DecodeException buf offset message) =+ "DecodeException\nbuf:" ++ show buf ++ "\noffset:" ++ show offset ++ "\nmessage:" ++ show message instance Exception DecodeException @@ -81,20 +80,14 @@ -- *** Exception: System.IO.Streams.Binary: binary decode exception: offset 16, "not enough bytes" -- getFromStream :: Get a -> InputStream ByteString -> IO (Maybe a)-getFromStream g is = do- let decoder = runGetIncremental g- Streams.read is >>= maybe (return Nothing)- (\s -> if S.null s then go decoder else go $ pushChunk decoder s)+getFromStream g is = Streams.read is >>= maybe (return Nothing) (go . P.parse g) where go (Fail s offset message) = do unless (S.null s) (Streams.unRead s is)- throwIO $ DecodeException offset message+ throwIO $ DecodeException s offset message go (Done s _ x) = do unless (S.null s) (Streams.unRead s is) return (Just x)- go decoder' =- Streams.read is >>=- maybe (go $ pushEndOfInput decoder')- (\s -> if S.null s then go decoder' else go $ pushChunk decoder' s)+ go (Partial p) = Streams.read is >>= go . p {-# INLINE getFromStream #-} -- | typeclass version of 'getFromStream'
− src/System/IO/Streams/Cereal.hs
@@ -1,126 +0,0 @@-{-# LANGUAGE DeriveDataTypeable #-}-{-# LANGUAGE OverloadedStrings #-}---------------------------------------------------------------------------------- |--- Module : Sytem.IO.Streams.Cereal--- Copyright : Soostone Inc, Winterland--- License : BSD3------ Maintainer : Winterland--- Stability : experimental------ Use cereal to encode/decode io-streams.-------------------------------------------------------------------------------module System.IO.Streams.Cereal (- -- * single element encode/decode- getFromStream- , decodeFromStream- , putToStream- -- * 'InputStream' encode/decode- , getInputStream- , decodeInputStream- -- * 'OutputStream' encode- , putOutputStream- , encodeOutputStream- -- * exception type- , DecodeException(..)- ) where-----------------------------------------------------------------------------------import Control.Exception (Exception, throwIO)-import Control.Monad (unless)-import Data.ByteString (ByteString)-import qualified Data.ByteString.Char8 as S-import Data.Serialize-import Data.Typeable-import qualified System.IO.Streams as Streams-import System.IO.Streams.Core------------------------------------------------------------------------------------- | An Exception raised when cereal decoding fails.-data DecodeException = DecodeException String- deriving (Typeable)--instance Show DecodeException where- show (DecodeException s) = "System.IO.Streams.Cereal: cereal decode exception: " ++ s--instance Exception DecodeException------------------------------------------------------------------------------------- | write a instance of 'Serialize' to an 'OutputStream'----putToStream :: Serialize a => Maybe a -> OutputStream ByteString -> IO ()-putToStream Nothing = Streams.write Nothing-putToStream (Just a) = (Streams.writeLazyByteString . runPutLazy . put) a-{-# INLINE putToStream #-}------------------------------------------------------------------------------------- | Take a 'Get' and an 'InputStream' and decode a--- value. Consumes only as much input as necessary to decode the--- value. Unconsumed input will be unread. If there is--- an error while deserializing, a 'DecodeException' is thrown, and--- unconsumed part will be unread. To simplify upstream generation,--- all empty 'ByteString' will be filtered out and not passed to cereal,--- only EOFs/Nothing will close a cereal decoder.------ Examples:------ >>> import qualified System.IO.Streams as Streams--- >>> getFromStream (get :: Get String) =<< Streams.fromByteString (Data.ByteString.drop 1 $ runPut $ put "encode me")--- *** Exception: System.IO.Streams.Cereal: cereal decode exception: too few bytes--- From: demandInput--- <BLANKLINE>----getFromStream :: Get a -> InputStream ByteString -> IO (Maybe a)-getFromStream g is =- Streams.read is >>= maybe (return Nothing) (go . runGetPartial g)- where- go (Fail msg s) = do- unless (S.null s) (Streams.unRead s is)- throwIO (DecodeException msg)- go (Done r s) = do- unless (S.null s) (Streams.unRead s is)- return (Just r)- go c@(Partial cont) =- Streams.read is >>= maybe (go (cont S.empty)) -- use 'empty' to notify cereal ending.- (\ s -> if S.null s then go c else go (cont s))-{-# INLINE getFromStream #-}---- | typeclass version of 'getFromStream'-decodeFromStream :: Serialize a => InputStream ByteString -> IO (Maybe a)-decodeFromStream = getFromStream get-{-# INLINE decodeFromStream #-}------------------------------------------------------------------------------------- | Convert a stream of individual encoded 'ByteString's to a stream--- of Results. Throws a 'DecodeException' on error.-getInputStream :: Get a -> InputStream ByteString -> IO (InputStream a)-getInputStream g is = makeInputStream (getFromStream g is)-{-# INLINE getInputStream #-}---- | typeclass version of 'getInputStream'-decodeInputStream :: Serialize a => InputStream ByteString -> IO (InputStream a)-decodeInputStream = getInputStream get-{-# INLINE decodeInputStream #-}------------------------------------------------------------------------------------- | create an 'OutputStream' of serializable values from an 'OutputStream'--- of bytestrings with a 'Putter'.-putOutputStream :: Putter a -> OutputStream ByteString -> IO (OutputStream a)-putOutputStream p os = Streams.makeOutputStream $ \ ma ->- case ma of Nothing -> Streams.write Nothing os- Just a -> Streams.writeLazyByteString (runPutLazy (p a)) os-{-# INLINE putOutputStream #-}---- | typeclass version of 'putOutputStream'-encodeOutputStream :: Serialize a => OutputStream ByteString -> IO (OutputStream a)-encodeOutputStream = putOutputStream put-{-# INLINE encodeOutputStream #-}
− test/EncodeDecodeCereal.hs
@@ -1,87 +0,0 @@-{-# LANGUAGE ScopedTypeVariables #-}-module EncodeDecodeCereal ( tests ) where--import Control.Exception (catch,evaluate)-import Data.Serialize (Serialize)-import Data.ByteString (ByteString)-import qualified Data.ByteString as S-import Distribution.TestSuite (Test)-import Distribution.TestSuite.QuickCheck (testProperty)-import System.IO.Streams (write)-import System.IO.Streams.Cereal (decodeInputStream,- encodeOutputStream,- DecodeException)-import System.IO.Streams.List (outputToList,- fromList,- toList,- writeList)-import Test.QuickCheck.Property (Property)-import Test.QuickCheck.Monadic (monadicIO,- assert,- run)----- Using binary-streams, decode from a list of bytestrings-decode :: Serialize a => [ByteString] -> IO [a]-decode ss = fromList ss >>= decodeInputStream >>= toList---- Using binary-streams, encode to a list of bytestrings-encode :: Serialize a => [a] -> IO [ByteString]-encode xs = outputToList $ \os ->- do- bos <- encodeOutputStream os- writeList xs bos- write Nothing bos---- Encode something, then decode it and make sure we get the same thing back.-encodeDecodeEq :: (Serialize a,Eq a) => [a] -> Property-encodeDecodeEq xs = monadicIO $ do- xs' <- run go- assert $ xs == xs'- where go = encode xs >>= decode---- corrupt something, remove the last byte of the last bytestring-corrupt :: [ByteString] -> [ByteString]-corrupt = reverse . go . reverse- where go (h:t) = ((S.reverse $ S.drop 1 $ S.reverse h):t)- go [] = []---- Encode something, corrupt the encoded data, and make sure we get a--- decode error when we try do decode it.-encodeDecodeError :: forall a. (Serialize a,Eq a) => [a] -> Property-encodeDecodeError [] = monadicIO $ return ()-encodeDecodeError xs = monadicIO $ do- run $ catch go $ \(_ :: DecodeException) -> return ()- where go =- do- bList <- encode xs- (xs' :: [a]) <- decode $ corrupt bList- evaluate xs'- fail "decoding succeeded when it should fail"--tests :: IO [Test]-tests =- return [testProperty "encode-decode-equality Int"- (encodeDecodeEq :: [Int] -> Property),- testProperty "encode-decode-equality String"- (encodeDecodeEq :: [String] -> Property),- testProperty "encode-decode-equality Maybe Int"- (encodeDecodeEq :: [Maybe Int] -> Property),- testProperty "encode-decode-equality Either Int String"- (encodeDecodeEq :: [Either Int String] -> Property),- testProperty "encode-decode-equality (Int,Int)"- (encodeDecodeEq :: [(Int,Int)] -> Property),- testProperty "encode-decode-equality (String,String)"- (encodeDecodeEq :: [(Int,Int)] -> Property),- testProperty "encode-decode-error Int"- (encodeDecodeError :: [Int] -> Property),- testProperty "encode-decode-error String"- (encodeDecodeError :: [String] -> Property),- testProperty "encode-decode-error Maybe Int"- (encodeDecodeError :: [Maybe Int] -> Property),- testProperty "encode-decode-error Either Int String"- (encodeDecodeError :: [Either Int String] -> Property),- testProperty "encode-decode-error (Int,Int)"- (encodeDecodeError :: [(Int,Int)] -> Property),- testProperty "encode-decode-error (String,String)"- (encodeDecodeError :: [(String,String)] -> Property)]
wire-streams.cabal view
@@ -1,45 +1,35 @@ name: wire-streams-version: 0.0.2.0+version: 0.1.0.0 synopsis: Use cereal or binary with io-streams. description: Use cereal or binary with io-streams. license: BSD3 license-file: LICENSE-author: Michael Xavier, Petter Bergman, Winterland+author: Winterland maintainer: winterland1989@gmail.com-copyright: Soostone Inc, Petter Bergman, Winterland+copyright: Winterland category: Data, Parsing, IO-Streams build-type: Simple-extra-source-files: README.md, CHANGElOG+extra-source-files: README.md, CHANGELOG cabal-version: >=1.10+homepage: https://github.com/winterland1989/wire-streams -library- exposed-modules: System.IO.Streams.Cereal- System.IO.Streams.Binary+source-repository head+ type: git+ location: git://github.com/winterland1989/wire-streams.git - build-depends: base >= 4 && < 5- , bytestring >= 0.10.2.0- , cereal >= 0.5 && < 0.6- , binary >= 0.6 && < 0.9+library+ exposed-modules: System.IO.Streams.Binary+ build-depends: base == 4.*+ , binary == 0.8.*+ , bytestring == 0.10.*+ , binary-parsers >= 0.2.1 , io-streams >= 1.2+ hs-source-dirs: src ghc-options: -Wall default-language: Haskell2010 -test-suite encode-decode-cereal- type: detailed-0.9- test-module: EncodeDecodeCereal- build-depends: base- , bytestring- , cereal- , io-streams- , wire-streams- , QuickCheck- , Cabal >= 1.10- , cabal-test-quickcheck- hs-source-dirs: test- default-language: Haskell2010- test-suite encode-decode-binary type: detailed-0.9 test-module: EncodeDecodeBinary@@ -57,6 +47,7 @@ benchmark bench type: exitcode-stdio-1.0 main-is: Main.hs+ other-modules: System.IO.Streams.Cereal hs-source-dirs: bench default-language: Haskell2010 build-depends: base@@ -72,7 +63,3 @@ , transformers ghc-options: -rtsopts -Wall--source-repository head- type: git- location: git://github.com/winterland1989/wire-streams.git