streaming-bytestring 0.1.0.0 → 0.1.0.1
raw patch · 1 files changed
+42/−32 lines, 1 files
Files
- streaming-bytestring.cabal +42/−32
streaming-bytestring.cabal view
@@ -1,7 +1,8 @@ name: streaming-bytestring-version: 0.1.0.0-synopsis: Effectful sequences of bytes.-description: This is an implementation of effectful, monadic bytestrings,+version: 0.1.0.1+synopsis: Lazy bytestring done right+description: This is an implementation of effectful memory-constrained + bytestrings, or byte streams, and streaming bytestring manipulation, adequate for non-lazy-io. . Interoperation with @pipes@ uses this isomorphism:@@ -14,25 +15,32 @@ > IOStreams.unfoldM Streaming.unconsChunk :: ByteString IO () -> IO (InputStream ByteString) > Streaming.reread IOStreams.read :: InputStream ByteString -> ByteString IO () .- and similarly for other streaming io libraries.+ and similarly for other streaming io libraries. .+ A tutorial module is in the works; + <https://gist.github.com/michaelt/6c6843e6dd8030e95d58 here> + is a sequence of simplified implementations of familiar shell utilities. + It closely follows those at the end of the+ <http://hackage.haskell.org/package/io-streams-1.3.2.0/docs/System-IO-Streams-Tutorial.html io-streams tutorial>.+ It is generally much simpler; in some case simpler than what+ you would write with lazy bytestrings. + . The implementation follows the details of @Data.ByteString.Lazy@ and @Data.ByteString.Lazy.Char8@- as far as is possible, substituting the type+ as far as is possible, replacing the lazy bytestring type: .- > data ByteString m r = Empty r - > | Chunk Strict.ByteString (ByteString m r) - > | Go (m (ByteString m r))+ > data ByteString = Empty | Chunk Strict.ByteString ByteString .- for the type- . - > data ByteString = Empty - > | Chunk Strict.ByteString ByteString+ with the minimal effectful variant .- found in @Data.ByteString.Lazy.Internal@. (Constructors are necessarily hidden in - internal modules in both cases.) As a lazy bytestring is implemented internally + > data ByteString m r = Empty r | Chunk Strict.ByteString (ByteString m r) | Go (m (ByteString m r))+ .+ (Constructors are necessarily hidden in internal modules in both cases.) + As a lazy bytestring is implemented internally by a sort of list of strict bytestring chunks, a streaming bytestring is - implemented as a /producer/ or /generator/ of strict bytestring chunks.+ simply implemented as a /producer/ or /generator/ of strict bytestring chunks.+ Most operations are defined by simply adding a line to what we find in+ @Data.ByteString.Lazy@. . Something like this alteration of type is of course obvious and mechanical, once the idea of an effectful bytestring type is contemplated and lazy io is rejected.@@ -79,8 +87,7 @@ > time ./benchlines streaming >> /dev/null > real 0m1.531s . - The pipes environment (to which this library basically belongs) - would express the latter as + The pipes environment would express the latter as . > Pipes.intercalates (Pipes.yield "!\n") . view Pipes.lines .@@ -89,24 +96,21 @@ > time ./benchlines pipes >> /dev/null > real 0m6.353s .- The difference, I think, is mostly that this library depends - the @streaming@ library, which is used in place of @free@ to - express the splitting and division of byte streams. - .- Indeed even if I unwrap and re-wrap with the above-mentioned isomorphism- .- > Pipes.unfoldr Streaming.nextChunk . Streaming.intercalate "!\n" . Streaming.lines . Streaming.unfoldrChunks Pipe.next- .- I get an excellent speed-up:- .- > $ time ./benchlines pipes_stream >> /dev/null- > real 0m3.393s+ The difference is not intrinsic to pipes, but is mostly that + this library depends the @streaming@ library, which is used in place + of @free@ to express the (streaming) splitting and division of byte streams. + Those elementary concepts are catastrophically mishandled in the streaming io libraries + other than pipes; already the @enumerator@ and @iteratee@ libraries+ were completely defeated by it: see e.g. the implementation of + <http://hackage.haskell.org/package/enumerator-0.4.20/docs/Data-Enumerator-Text.html#v:splitWhen splitWhen and lines>.+ This will concatenate strict text forever, if that's what is coming+ in. . Though we barely alter signatures in @Data.ByteString.Lazy@ more than is required by the types, the point of view that emerges is very much that of @pipes-bytestring@ and @pipes-group@. In particular- we have the correspondences+ we have the correspondences: . > Lazy.splitAt :: Int -> ByteString -> (ByteString, ByteString) > Streaming.splitAt :: Int -> ByteString m r -> ByteString m (ByteString m r)@@ -125,9 +129,15 @@ way of structuring material you might take a look at the tutorial for <http://hackage.haskell.org/package/pipes-group-1.0.2/docs/Pipes-Group-Tutorial.html pipes-group> and the examples in the documentation for the streaming library. See also- <https://gist.github.com/michaelt/6c6843e6dd8030e95d58 these> - implementations of the shell-like examples from the @io-streams@ tutorial.+ <https://gist.github.com/michaelt/6c6843e6dd8030e95d58 simple implementations> + of the shell-like examples mentioned above. .+ The modules for applying attoparsec parsers and making http clients should+ properly be in separate packages, but are included for the moment+ to simplify experimentation. They simply replicate the corresponding pipes+ modules. Those modules don't involve pipes, but only the + @Producer ByteString m r@ type, which pipes globally reserves + to mean exactly what is here called @ByteString m r@. license: BSD3 license-file: LICENSE