diff --git a/streaming-bytestring.cabal b/streaming-bytestring.cabal --- a/streaming-bytestring.cabal +++ b/streaming-bytestring.cabal @@ -1,7 +1,8 @@ name: streaming-bytestring -version: 0.1.0.0 -synopsis: Effectful sequences of bytes. -description: This is an implementation of effectful, monadic bytestrings, +version: 0.1.0.1 +synopsis: Lazy bytestring done right +description: This is an implementation of effectful memory-constrained + bytestrings, or byte streams, and streaming bytestring manipulation, adequate for non-lazy-io. . Interoperation with @pipes@ uses this isomorphism: @@ -14,25 +15,32 @@ > IOStreams.unfoldM Streaming.unconsChunk :: ByteString IO () -> IO (InputStream ByteString) > Streaming.reread IOStreams.read :: InputStream ByteString -> ByteString IO () . - and similarly for other streaming io libraries. + and similarly for other streaming io libraries. . + A tutorial module is in the works; + + is a sequence of simplified implementations of familiar shell utilities. + It closely follows those at the end of the + . + It is generally much simpler; in some case simpler than what + you would write with lazy bytestrings. + . The implementation follows the details of @Data.ByteString.Lazy@ and @Data.ByteString.Lazy.Char8@ - as far as is possible, substituting the type + as far as is possible, replacing the lazy bytestring type: . - > data ByteString m r = Empty r - > | Chunk Strict.ByteString (ByteString m r) - > | Go (m (ByteString m r)) + > data ByteString = Empty | Chunk Strict.ByteString ByteString . - for the type - . - > data ByteString = Empty - > | Chunk Strict.ByteString ByteString + with the minimal effectful variant . - found in @Data.ByteString.Lazy.Internal@. (Constructors are necessarily hidden in - internal modules in both cases.) As a lazy bytestring is implemented internally + > data ByteString m r = Empty r | Chunk Strict.ByteString (ByteString m r) | Go (m (ByteString m r)) + . + (Constructors are necessarily hidden in internal modules in both cases.) + As a lazy bytestring is implemented internally by a sort of list of strict bytestring chunks, a streaming bytestring is - implemented as a /producer/ or /generator/ of strict bytestring chunks. + simply implemented as a /producer/ or /generator/ of strict bytestring chunks. + Most operations are defined by simply adding a line to what we find in + @Data.ByteString.Lazy@. . Something like this alteration of type is of course obvious and mechanical, once the idea of an effectful bytestring type is contemplated and lazy io is rejected. @@ -79,8 +87,7 @@ > time ./benchlines streaming >> /dev/null > real 0m1.531s . - The pipes environment (to which this library basically belongs) - would express the latter as + The pipes environment would express the latter as . > Pipes.intercalates (Pipes.yield "!\n") . view Pipes.lines . @@ -89,24 +96,21 @@ > time ./benchlines pipes >> /dev/null > real 0m6.353s . - The difference, I think, is mostly that this library depends - the @streaming@ library, which is used in place of @free@ to - express the splitting and division of byte streams. - . - Indeed even if I unwrap and re-wrap with the above-mentioned isomorphism - . - > Pipes.unfoldr Streaming.nextChunk . Streaming.intercalate "!\n" . Streaming.lines . Streaming.unfoldrChunks Pipe.next - . - I get an excellent speed-up: - . - > $ time ./benchlines pipes_stream >> /dev/null - > real 0m3.393s + The difference is not intrinsic to pipes, but is mostly that + this library depends the @streaming@ library, which is used in place + of @free@ to express the (streaming) splitting and division of byte streams. + Those elementary concepts are catastrophically mishandled in the streaming io libraries + other than pipes; already the @enumerator@ and @iteratee@ libraries + were completely defeated by it: see e.g. the implementation of + . + This will concatenate strict text forever, if that's what is coming + in. . Though we barely alter signatures in @Data.ByteString.Lazy@ more than is required by the types, the point of view that emerges is very much that of @pipes-bytestring@ and @pipes-group@. In particular - we have the correspondences + we have the correspondences: . > Lazy.splitAt :: Int -> ByteString -> (ByteString, ByteString) > Streaming.splitAt :: Int -> ByteString m r -> ByteString m (ByteString m r) @@ -125,9 +129,15 @@ way of structuring material you might take a look at the tutorial for and the examples in the documentation for the streaming library. See also - - implementations of the shell-like examples from the @io-streams@ tutorial. + + of the shell-like examples mentioned above. . + The modules for applying attoparsec parsers and making http clients should + properly be in separate packages, but are included for the moment + to simplify experimentation. They simply replicate the corresponding pipes + modules. Those modules don't involve pipes, but only the + @Producer ByteString m r@ type, which pipes globally reserves + to mean exactly what is here called @ByteString m r@. license: BSD3 license-file: LICENSE