diff --git a/Pipes/Text.hs b/Pipes/Text.hs --- a/Pipes/Text.hs +++ b/Pipes/Text.hs @@ -1,98 +1,10 @@ {-# LANGUAGE RankNTypes, TypeFamilies, BangPatterns, Trustworthy #-} -{-| This package provides @pipes@ utilities for \'text streams\', which are - streams of 'Text' chunks. The individual chunks are uniformly @strict@, and thus you - will generally want @Data.Text@ in scope. But the type @Producer Text m r@ is - in some ways the pipes equivalent of the lazy @Text@ type. - This module provides many functions equivalent in one way or another to - the 'pure' functions in - . - They transform, divide, group and fold text streams. Though @Producer Text m r@ - is the type of \'effectful Text\', the functions in this module are \'pure\' - in the sense that they are uniformly monad-independent. - Simple IO operations are defined in @Pipes.Text.IO@ -- as lazy IO @Text@ - operations are in @Data.Text.Lazy.IO@. Interoperation with @ByteString@ - is provided in @Pipes.Text.Encoding@, which parallels @Data.Text.Lazy.Encoding@. - - The Text type exported by @Data.Text.Lazy@ is basically '[Text]'. The implementation - is arranged so that the individual strict 'Text' chunks are kept to a reasonable size; - the user is not aware of the divisions between the connected 'Text' chunks. - So also here: the functions in this module are designed to operate on streams that - are insensitive to text boundaries. This means that they may freely split - text into smaller texts and /discard empty texts/. However, the objective is - that they should /never concatenate texts/ in order to provide strict upper - bounds on memory usage. - - For example, to stream only the first three lines of 'stdin' to 'stdout' you - might write: - -> import Pipes -> import qualified Pipes.Text as Text -> import qualified Pipes.Text.IO as Text -> import Pipes.Group -> import Lens.Family -> -> main = runEffect $ takeLines 3 Text.stdin >-> Text.stdout -> where -> takeLines n = Text.unlines . takes' n . view Text.lines -> -- or equivalently: -> -- takeLines n = over Text.lines (takes' n) - - The above program will never bring more than one chunk of text (~ 32 KB) into - memory, no matter how long the lines are. - - As this example shows, one superficial difference from @Data.Text.Lazy@ - is that many of the operations, like 'lines', - are \'lensified\'; this has a number of advantages where it is possible, in particular - it facilitates their use with 'Parser's of Text (in the general - - sense.) - Each such expression, e.g. 'lines', 'chunksOf' or 'splitAt', reduces to the - intuitively corresponding function when used with @view@ or @(^.)@. The lens combinators - you will find indispensible are \'view\'/ '(^.)', 'zoom' and probably 'over', which - are supplied by both and - - - A more important difference the example reveals is in the types closely associated with - the central type, @Producer Text m r@. In @Data.Text@ and @Data.Text.Lazy@ - we find functions like - -> splitAt :: Int -> Text -> (Text, Text) -> lines :: Int -> Text -> [Text] -> chunksOf :: Int -> Text -> [Text] - - which relate a Text with a pair or list of Texts. The corresponding functions here (taking - account of \'lensification\') are - -> view . splitAt :: (Monad m, Integral n) => n -> Producer Text m r -> Producer Text.Text m (Producer Text.Text m r) -> view lines :: Monad m => Producer Text m r -> FreeT (Producer Text m) m r -> view . chunksOf :: (Monad m, Integral n) => n -> Producer Text m r -> FreeT (Producer Text m) m r - - In the type @Producer Text m (Producer Text m r)@ the second - element of the \'pair\' of of \'effectful Texts\' cannot simply be retrieved - with 'snd'. This is an \'effectful\' pair, and one must work through the effects - of the first element to arrive at the second Text stream. Similarly in @FreeT (Producer Text m) m r@, - which corresponds with @[Text]@, on cannot simply drop 10 Producers and take the others; - we can only get to the ones we want to take by working through their predecessors. - - Some of the types may be more readable if you imagine that we have introduced - our own type synonyms - -> type Text m r = Producer T.Text m r -> type Texts m r = FreeT (Producer T.Text m) m r - - Then we would think of the types above as - -> view . splitAt :: (Monad m, Integral n) => n -> Text m r -> Text m (Text m r) -> view lines :: (Monad m) => Text m r -> Texts m r -> view . chunksOf :: (Monad m, Integral n) => n -> Text m r -> Texts m r - - which brings one closer to the types of the similar functions in @Data.Text.Lazy@ - --} - module Pipes.Text ( + -- * Introduction + -- $intro + -- * Producers fromLazy @@ -221,6 +133,172 @@ words, writeFile ) +{- $intro + + * /Effectful Text/ + + This package provides @pipes@ utilities for /text streams/, understood as + streams of 'Text' chunks. The individual chunks are uniformly /strict/, and thus you + will generally want @Data.Text@ in scope. But the type @Producer Text m r@ as we + are using it is a sort of pipes equivalent of the lazy @Text@ type. + + This particular module provides many functions equivalent in one way or another to + the pure functions in + . + They transform, divide, group and fold text streams. Though @Producer Text m r@ + is the type of \'effectful Text\', the functions in this module are \'pure\' + in the sense that they are uniformly monad-independent. + Simple /IO/ operations are defined in @Pipes.Text.IO@ -- as lazy IO @Text@ + operations are in @Data.Text.Lazy.IO@. Inter-operation with @ByteString@ + is provided in @Pipes.Text.Encoding@, which parallels @Data.Text.Lazy.Encoding@. + + The Text type exported by @Data.Text.Lazy@ is basically that of a lazy list of + strict Text: the implementation is arranged so that the individual strict 'Text' + chunks are kept to a reasonable size; the user is not aware of the divisions + between the connected 'Text' chunks. + So also here: the functions in this module are designed to operate on streams that + are insensitive to text boundaries. This means that they may freely split + text into smaller texts and /discard empty texts/. The objective, though, is + that they should /never concatenate texts/ in order to provide strict upper + bounds on memory usage. + + For example, to stream only the first three lines of 'stdin' to 'stdout' you + might write: + +> import Pipes +> import qualified Pipes.Text as Text +> import qualified Pipes.Text.IO as Text +> import Pipes.Group (takes') +> import Lens.Family +> +> main = runEffect $ takeLines 3 Text.stdin >-> Text.stdout +> where +> takeLines n = Text.unlines . takes' n . view Text.lines + + The above program will never bring more than one chunk of text (~ 32 KB) into + memory, no matter how long the lines are. + + * /Lenses/ + + As this example shows, one superficial difference from @Data.Text.Lazy@ + is that many of the operations, like 'lines', + are \'lensified\'; this has a number of advantages (where it is possible), in particular + it facilitates their use with 'Parser's of Text (in the general + + sense.) + Each such lens, e.g. 'lines', 'chunksOf' or 'splitAt', reduces to the + intuitively corresponding function when used with @view@ or @(^.)@. + + Note similarly that many equivalents of 'Text -> Text' functions are exported here as 'Pipe's. + They reduce to the intuitively corresponding functions when used with '(>->)'. Thus something like + +> stripLines = Text.unlines . Group.maps (>-> Text.stripStart) . view Text.lines + + would drop the leading white space from each line. + + The lens combinators + you will find indispensible are @view@ / @(^.)@), @zoom@ and probably @over@. These + are supplied by both and + The use of 'zoom' is explained + in + and to some extent in the @Pipes.Text.Encoding@ module here. The use of + @over@ is simple, illustrated by the fact that we can rewrite @stripLines@ above as + +> stripLines = over Text.lines $ maps (>-> stripStart) + + + * Special types: @Producer Text m (Producer Text m r)@ and @FreeT (Producer Text m) m r@ + + These simple 'lines' examples reveal a more important difference from @Data.Text.Lazy@ . + This is in the types that are most closely associated with our central text type, + @Producer Text m r@. In @Data.Text@ and @Data.Text.Lazy@ we find functions like + +> splitAt :: Int -> Text -> (Text, Text) +> lines :: Text -> [Text] +> chunksOf :: Int -> Text -> [Text] + + which relate a Text with a pair of Texts or a list of Texts. + The corresponding functions here (taking account of \'lensification\') are + +> view . splitAt :: (Monad m, Integral n) => n -> Producer Text m r -> Producer Text m (Producer Text m r) +> view lines :: Monad m => Producer Text m r -> FreeT (Producer Text m) m r +> view . chunksOf :: (Monad m, Integral n) => n -> Producer Text m r -> FreeT (Producer Text m) m r + + Some of the types may be more readable if you imagine that we have introduced + our own type synonyms + +> type Text m r = Producer T.Text m r +> type Texts m r = FreeT (Producer T.Text m) m r + + Then we would think of the types above as + +> view . splitAt :: (Monad m, Integral n) => n -> Text m r -> Text m (Text m r) +> view lines :: (Monad m) => Text m r -> Texts m r +> view . chunksOf :: (Monad m, Integral n) => n -> Text m r -> Texts m r + + which brings one closer to the types of the similar functions in @Data.Text.Lazy@ + + In the type @Producer Text m (Producer Text m r)@ the second + element of the \'pair\' of effectful Texts cannot simply be retrieved + with something like 'snd'. This is an \'effectful\' pair, and one must work + through the effects of the first element to arrive at the second Text stream, even + if you are proposing to throw the Text in the first element away. + Note that we use Control.Monad.join to fuse the pair back together, since it specializes to + +> join :: Monad m => Producer Text m (Producer m r) -> Producer m r + + The return type of 'lines', 'words', 'chunksOf' and the other "splitter" functions, + @FreeT (Producer m Text) m r@ -- our @Texts m r@ -- is the type of (effectful) + lists of (effectful) texts. The type @([Text],r)@ might be seen to gather + together things of the forms: + +> r +> (Text,r) +> (Text, (Text, r)) +> (Text, (Text, (Text, r))) +> (Text, (Text, (Text, (Text, r)))) +> ... + + We might also have identified the sum of those types with @Free ((,) Text) r@ + -- or, more absurdly, @FreeT ((,) Text) Identity r@. Similarly, @FreeT (Producer Text m) m r@ + encompasses all the members of the sequence: + +> m r +> Producer Text m r +> Producer Text m (Producer Text m r) +> Producer Text m (Producer Text m (Producer Text m r)) +> ... + + One might think that + +> lines :: Monad m => Lens' (Producer Text m r) (FreeT (Producer Text m) m r) +> view . lines :: Monad m => Producer Text m r -> FreeT (Producer Text m) m r + + should really have the type + +> lines :: Monad m => Pipe Text Text m r + + as e.g. 'toUpper' does. But this would spoil the control we are + attempting to maintain over the size of chunks. It is in fact just + as unreasonable to want such a pipe as to want + +> Data.Text.Lazy.lines :: Text -> Text + + to 'rechunk' the strict Text chunks inside the lazy Text to respect + line boundaries. In fact we have + +> Data.Text.Lazy.lines :: Text -> [Text] +> Prelude.lines :: String -> [String] + + where the elements of the list are themselves lazy Texts or Strings; the use + of @FreeT (Producer Text m) m r@ is simply the 'effectful' version of this. + + The @Pipes.Group@ module, which can generally be imported without qualification, + provides many functions for working with things of type @FreeT (Producer a m) m r@ + + + -} + -- | Convert a lazy 'TL.Text' into a 'Producer' of strict 'Text's fromLazy :: (Monad m) => TL.Text -> Producer' Text m () fromLazy = foldrChunks (\e a -> yield e >> a) (return ()) @@ -275,7 +353,7 @@ -- | @toCaseFold@, @toLower@, @toUpper@ and @stripStart@ are standard 'Text' utilities, -- here acting as 'Text' pipes, rather as they would on a lazy text -toCaseFold :: Monad m => Pipe Text Text m () +toCaseFold :: Monad m => Pipe Text Text m r toCaseFold = P.map T.toCaseFold {-# INLINEABLE toCaseFold #-} @@ -285,7 +363,7 @@ -- | lowercase incoming 'Text' -toLower :: Monad m => Pipe Text Text m () +toLower :: Monad m => Pipe Text Text m r toLower = P.map T.toLower {-# INLINEABLE toLower #-} @@ -294,7 +372,7 @@ #-} -- | uppercase incoming 'Text' -toUpper :: Monad m => Pipe Text Text m () +toUpper :: Monad m => Pipe Text Text m r toUpper = P.map T.toUpper {-# INLINEABLE toUpper #-} diff --git a/Pipes/Text/Encoding.hs b/Pipes/Text/Encoding.hs --- a/Pipes/Text/Encoding.hs +++ b/Pipes/Text/Encoding.hs @@ -2,16 +2,18 @@ -- | This module uses the stream decoding functions from Michael Snoyman's new -- --- package to define decoding functions and lenses. +-- package to define decoding functions and lenses. The exported names +-- conflict with names in @Data.Text.Encoding@ but the module can otherwise be +-- imported unqualified. module Pipes.Text.Encoding ( -- * The Lens or Codec type -- $lenses Codec - -- * Viewing the Text in a ByteString - -- $codecs , decode + -- * \'Viewing\' the Text in a byte stream + -- $codecs , utf8 , utf8Pure , utf16LE @@ -55,7 +57,6 @@ import Data.Word (Word8) import Pipes - type Lens' a b = forall f . Functor f => (b -> f b) -> (a -> f a) {- $lenses @@ -66,8 +67,9 @@ > type Lens' a b = forall f . Functor f => (b -> f b) -> (a -> f a) - is just an alias for an ordinary Prelude type. Thus you use any codec with - the @view@ / @(^.)@ and @zoom@ functions from those libraries. + is just an alias for a Prelude type. Thus you use any particular codec with + the @view@ / @(^.)@ , @zoom@ and @over@ functions from either of those libraries; + we presuppose neither since we already have access to the types they require. -} @@ -76,11 +78,12 @@ . Monad m => Lens' (Producer ByteString m r) (Producer Text m (Producer ByteString m r)) - + {- | 'decode' is just the ordinary @view@ or @(^.)@ of the lens libraries; - exported here for convience + exported here under a name appropriate to the material. All of these are + the same: -> decode utf8 p = decodeUtf8 p = view utf8 p = p ^. utf +> decode utf8 p = decodeUtf8 p = view utf8 p = p ^. utf8 -} @@ -88,6 +91,85 @@ decode codec a = getConstant (codec Constant a) +{- $codecs + + Each Codec-lens looks into a byte stream that is supposed to contain text. + The particular \'Codec\' lenses are named in accordance with the expected + encoding, 'utf8', 'utf16LE' etc. To turn a Codec into an ordinary function, + use @view@ / @(^.)@ -- here also called 'decode': + +> view utf8 :: Producer ByteString m r -> Producer Text m (Producer ByteString m r) +> decode utf8 Byte.stdin :: Producer Text IO (Producer ByteString IO r) +> Bytes.stdin ^. utf8 :: Producer Text IO (Producer ByteString IO r) + + Uses of a codec with @view@ or @(^.)@ or 'decode' can always be replaced by the specialized + decoding functions exported here, e.g. + +> decodeUtf8 :: Producer ByteString m r -> Producer Text m (Producer ByteString m r) +> decodeUtf8 Byte.stdin :: Producer Text IO (Producer ByteString IO r) + + The stream of text that a @Codec@ \'sees\' in the stream of bytes begins at its head. + At any point of decoding failure, the stream of text ends and reverts to (returns) + the original byte stream. Thus if the first bytes are already + un-decodable, the whole ByteString producer will be returned, i.e. + +> view utf8 bytestream + + will just come to the same as + +> return bytestream + + Where there is no decoding failure, the return value of the text stream will be + an empty byte stream followed by its own return value. In all cases you must + deal with the fact that it is a /ByteString producer/ that is returned, even if + it can be thrown away with @Control.Monad.void@ + +> void (Bytes.stdin ^. utf8) :: Producer Text IO () + + @zoom@ converts a Text parser into a ByteString parser: + +> zoom utf8 drawChar :: Monad m => StateT (Producer ByteString m r) m (Maybe Char) + + or, using the type synonymn from @Pipes.Parse@: + +> zoom utf8 drawChar :: Monad m => Parser ByteString m (Maybe Char) + + Thus we can define a ByteString parser like this: + +> withNextByte :: Parser ByteString m (Maybe Char, Maybe Word8))) +> withNextByte = do char_ <- zoom utf8 Text.drawChar +> byte_ <- Bytes.peekByte +> return (char_, byte_) + + Though @withNextByte@ is partly defined with a Text parser 'drawChar'; + but it is a ByteString parser; it will return the first valid utf8-encoded + Char in a ByteString, whatever its length, + and the first byte of the next character, if they exist. Because + we \'draw\' one and \'peek\' at the other, the parser as a whole only + advances one Char's length along the bytestring, whatever that length may be. + See the slightly more complex example \'decode.hs\' in the + + discussion of this type of byte stream parsing. + -} + +utf8 :: Codec +utf8 = mkCodec decodeUtf8 TE.encodeUtf8 + +utf8Pure :: Codec +utf8Pure = mkCodec decodeUtf8Pure TE.encodeUtf8 + +utf16LE :: Codec +utf16LE = mkCodec decodeUtf16LE TE.encodeUtf16LE + +utf16BE :: Codec +utf16BE = mkCodec decodeUtf16BE TE.encodeUtf16BE + +utf32LE :: Codec +utf32LE = mkCodec decodeUtf32LE TE.encodeUtf32LE + +utf32BE :: Codec +utf32BE = mkCodec decodeUtf32BE TE.encodeUtf32BE + decodeStream :: Monad m => (B.ByteString -> DecodeResult) -> Producer ByteString m r -> Producer Text m (Producer ByteString m r) @@ -177,75 +259,6 @@ -> Codec mkCodec dec enc = \k p0 -> fmap (\p -> join (for p (yield . enc))) (k (dec p0)) - -{- $codecs - - Each codec/lens looks into a byte stream that is supposed to contain text. - The particular \'Codec\' lenses are named in accordance with the expected - encoding, 'utf8', 'utf16LE' etc. @view@ / @(^.)@ -- here also called 'decode' -- - turns a Codec into a function: - -> view utf8 :: Producer ByteString m r -> Producer Text m (Producer ByteString m r) -> decode utf8 Byte.stdin :: Producer Text IO (Producer ByteString IO r) -> Bytes.stdin ^. utf8 :: Producer Text IO (Producer ByteString IO r) - - Uses of a codec with @view@ / @(^.)@ / 'decode' can always be replaced by the specialized - decoding functions exported here, e.g. - -> decodeUtf8 :: Producer ByteString m r -> Producer Text m (Producer ByteString m r) -> decodeUtf8 Byte.stdin :: Producer Text IO (Producer ByteString IO r) - - The stream of text a @Codec@ \'sees\' in the stream of bytes begins at its head. - At any point of decoding failure, the stream of text ends and reverts to (returns) - the original byte stream. Thus if the first bytes are already - un-decodable, the whole ByteString producer will be returned, i.e. - -> view utf8 bytestream - - will just come to the same as - -> return bytestream - - Where there is no decoding failure, the return value of the text stream will be - an empty byte stream followed by its own return value. In all cases you must - deal with the fact that it is a ByteString producer that is returned, even if - it can be thrown away with @Control.Monad.void@ - -> void (Bytes.stdin ^. utf8) :: Producer Text IO () - - @zoom@ converts a Text parser into a ByteString parser: - -> zoom utf8 drawChar :: Monad m => StateT (Producer ByteString m r) m (Maybe Char) -> -> withNextByte :: Parser ByteString m (Maybe Char, Maybe Word8))) -> withNextByte = do char_ <- zoom utf8 Text.drawChar -> byte_ <- Bytes.peekByte -> return (char_, byte_) - - @withNextByte@ will return the first valid Char in a ByteString, - and the first byte of the next character, if they exists. Because - we \'draw\' one and \'peek\' at the other, the parser as a whole only - advances one Char's length along the bytestring. - - -} - -utf8 :: Codec -utf8 = mkCodec decodeUtf8 TE.encodeUtf8 - -utf8Pure :: Codec -utf8Pure = mkCodec decodeUtf8Pure TE.encodeUtf8 - -utf16LE :: Codec -utf16LE = mkCodec decodeUtf16LE TE.encodeUtf16LE - -utf16BE :: Codec -utf16BE = mkCodec decodeUtf16BE TE.encodeUtf16BE - -utf32LE :: Codec -utf32LE = mkCodec decodeUtf32LE TE.encodeUtf32LE - -utf32BE :: Codec -utf32BE = mkCodec decodeUtf32BE TE.encodeUtf32BE {- $ascii diff --git a/Pipes/Text/IO.hs b/Pipes/Text/IO.hs --- a/Pipes/Text/IO.hs +++ b/Pipes/Text/IO.hs @@ -35,14 +35,15 @@ {- $textio Where pipes IO replaces lazy IO, @Producer Text m r@ replaces lazy 'Text'. This module exports some convenient functions for producing and consuming - pipes 'Text' in IO, with caveats described below. The main points are as in - + pipes 'Text' in IO, namely, 'readFile', 'writeFile', 'fromHandle', 'toHandle', + 'stdin' and 'stdout'. Some caveats described below. - An 'IO.Handle' can be associated with a 'Producer' or 'Consumer' according as it is read or written to. - - To stream to or from 'IO.Handle's, one can use 'fromHandle' or 'toHandle'. For - example, the following program copies a document from one file to another: - + The main points are as in + + + An 'IO.Handle' can be associated with a 'Producer' or 'Consumer' according + as it is read or written to. + > import Pipes > import qualified Pipes.Text as Text > import qualified Pipes.Text.IO as Text @@ -62,11 +63,12 @@ > > main = runSafeT $ runEffect $ Text.readFile "inFile.txt" >-> Text.writeFile "outFile.txt" - You can stream to and from 'stdin' and 'stdout' using the predefined 'stdin' + Finally, you can stream to and from 'stdin' and 'stdout' using the predefined 'stdin' and 'stdout' pipes, as with the following \"echo\" program: > main = runEffect $ Text.stdin >-> Text.stdout + -} @@ -78,11 +80,11 @@ * Like the functions in @Data.Text.IO@, they attempt to work with the system encoding. - * Like the functions in @Data.Text.IO@, they are slower than ByteString operations. Where + * Like the functions in @Data.Text.IO@, they significantly slower than ByteString operations. Where you know what encoding you are working with, use @Pipes.ByteString@ and @Pipes.Text.Encoding@ instead, e.g. @view utf8 Bytes.stdin@ instead of @Text.stdin@ - * Like the functions in @Data.Text.IO@ , they use Text exceptions. + * Like the functions in @Data.Text.IO@ , they use Text exceptions, not the standard Pipes protocols. Something like diff --git a/pipes-text.cabal b/pipes-text.cabal --- a/pipes-text.cabal +++ b/pipes-text.cabal @@ -1,5 +1,5 @@ name: pipes-text -version: 0.0.0.8 +version: 0.0.0.9 synopsis: Text pipes. description: * This package will be in a draft, or testing, phase until version 0.0.1. Please report any installation difficulties, or any wisdom about the api, on the github page or the .