packages feed

pipes-text 0.0.0.8 → 0.0.0.9

raw patch · 4 files changed

+276/−183 lines, 4 files

Files

Pipes/Text.hs view
@@ -1,98 +1,10 @@ {-# LANGUAGE RankNTypes, TypeFamilies, BangPatterns, Trustworthy #-} -{-| This package provides @pipes@ utilities for \'text streams\', which are-    streams of 'Text' chunks. The individual chunks are uniformly @strict@, and thus you -    will generally want @Data.Text@ in scope.  But the type @Producer Text m r@ is-    in some ways the pipes equivalent of the lazy @Text@ type. -    This module provides many functions equivalent in one way or another to -    the 'pure' functions in -    <https://hackage.haskell.org/package/text-1.1.0.0/docs/Data-Text-Lazy.html Data.Text.Lazy>. -    They transform, divide, group and fold text streams. Though @Producer Text m r@ -    is the type of \'effectful Text\', the functions in this module are \'pure\' -    in the sense that they are uniformly monad-independent.-    Simple IO operations are defined in @Pipes.Text.IO@ -- as lazy IO @Text@ -    operations are in @Data.Text.Lazy.IO@. Interoperation with @ByteString@ -    is provided in @Pipes.Text.Encoding@, which parallels @Data.Text.Lazy.Encoding@. --    The Text type exported by @Data.Text.Lazy@ is basically '[Text]'. The implementation-    is arranged so that the individual strict 'Text' chunks are kept to a reasonable size; -    the user is not aware of the divisions between the connected 'Text' chunks. -    So also here: the functions in this module are designed to operate on streams that-    are insensitive to text boundaries.  This means that they may freely split-    text into smaller texts and /discard empty texts/.  However, the objective is -    that they should /never concatenate texts/ in order to provide strict upper -    bounds on memory usage. --    For example, to stream only the first three lines of 'stdin' to 'stdout' you-    might write:--> import Pipes-> import qualified Pipes.Text as Text-> import qualified Pipes.Text.IO as Text-> import Pipes.Group-> import Lens.Family -> -> main = runEffect $ takeLines 3 Text.stdin >-> Text.stdout->   where ->     takeLines n = Text.unlines . takes' n . view Text.lines->  -- or equivalently: ->  -- takeLines n = over Text.lines (takes' n)--    The above program will never bring more than one chunk of text (~ 32 KB) into-    memory, no matter how long the lines are.-    -    As this example shows, one superficial difference from @Data.Text.Lazy@ -    is that many of the operations, like 'lines',-    are \'lensified\'; this has a number of advantages where it is possible, in particular -    it facilitates their use with 'Parser's of Text (in the general -    <http://hackage.haskell.org/package/pipes-parse-3.0.1/docs/Pipes-Parse-Tutorial.html pipes-parse> -    sense.) -    Each such expression, e.g. 'lines', 'chunksOf' or 'splitAt', reduces to the -    intuitively corresponding function when used with @view@ or @(^.)@.  The lens combinators-    you will find indispensible are \'view\'/ '(^.)', 'zoom' and probably 'over', which-    are supplied by both <http://hackage.haskell.org/package/lens lens> and -    <http://hackage.haskell.org/package/lens-family lens-family>-    -    A more important difference the example reveals is in the types closely associated with-    the central type, @Producer Text m r@.  In @Data.Text@ and @Data.Text.Lazy@-    we find functions like-    ->   splitAt :: Int -> Text -> (Text, Text)->   lines :: Int -> Text -> [Text]->   chunksOf :: Int -> Text -> [Text]--    which relate a Text with a pair or list of Texts. The corresponding functions here (taking-    account of \'lensification\') are -    ->   view . splitAt :: (Monad m, Integral n) => n -> Producer Text m r -> Producer Text.Text m (Producer Text.Text m r)->   view lines :: Monad m => Producer Text m r -> FreeT (Producer Text m) m r->   view . chunksOf ::  (Monad m, Integral n) => n -> Producer Text m r -> FreeT (Producer Text m) m r--    In the type @Producer Text m (Producer Text m r)@ the second -    element of the \'pair\' of of \'effectful Texts\' cannot simply be retrieved -    with 'snd'. This is an \'effectful\' pair, and one must work through the effects-    of the first element to arrive at the second Text stream. Similarly in @FreeT (Producer Text m) m r@,-    which corresponds with @[Text]@, on cannot simply drop 10 Producers and take the others;-    we can only get to the ones we want to take by working through their predecessors.-    -    Some of the types may be more readable if you imagine that we have introduced-    our own type synonyms-    ->   type Text m r = Producer T.Text m r->   type Texts m r = FreeT (Producer T.Text m) m r--    Then we would think of the types above as-    ->   view . splitAt :: (Monad m, Integral n) => n -> Text m r -> Text m (Text m r)->   view lines :: (Monad m) => Text m r -> Texts m r->   view . chunksOf :: (Monad m, Integral n) => n -> Text m r -> Texts m r--    which brings one closer to the types of the similar functions in @Data.Text.Lazy@---}- module Pipes.Text  (+    -- * Introduction+    -- $intro+         -- * Producers     fromLazy @@ -221,6 +133,172 @@     words,     writeFile ) +{- $intro++    * /Effectful Text/++    This package provides @pipes@ utilities for /text streams/, understood as+    streams of 'Text' chunks. The individual chunks are uniformly /strict/, and thus you +    will generally want @Data.Text@ in scope.  But the type @Producer Text m r@ as we+    are using it is a sort of pipes equivalent of the lazy @Text@ type. +    +    This particular module provides many functions equivalent in one way or another to +    the pure functions in +    <https://hackage.haskell.org/package/text-1.1.0.0/docs/Data-Text-Lazy.html Data.Text.Lazy>. +    They transform, divide, group and fold text streams. Though @Producer Text m r@ +    is the type of \'effectful Text\', the functions in this module are \'pure\' +    in the sense that they are uniformly monad-independent.+    Simple /IO/ operations are defined in @Pipes.Text.IO@ -- as lazy IO @Text@ +    operations are in @Data.Text.Lazy.IO@. Inter-operation with @ByteString@ +    is provided in @Pipes.Text.Encoding@, which parallels @Data.Text.Lazy.Encoding@. ++    The Text type exported by @Data.Text.Lazy@ is basically that of a lazy list of +    strict Text: the implementation is arranged so that the individual strict 'Text' +    chunks are kept to a reasonable size; the user is not aware of the divisions +    between the connected 'Text' chunks. +    So also here: the functions in this module are designed to operate on streams that+    are insensitive to text boundaries. This means that they may freely split+    text into smaller texts and /discard empty texts/.  The objective, though, is +    that they should /never concatenate texts/ in order to provide strict upper +    bounds on memory usage.  ++    For example, to stream only the first three lines of 'stdin' to 'stdout' you+    might write:++> import Pipes+> import qualified Pipes.Text as Text+> import qualified Pipes.Text.IO as Text+> import Pipes.Group (takes')+> import Lens.Family +> +> main = runEffect $ takeLines 3 Text.stdin >-> Text.stdout+>   where +>     takeLines n = Text.unlines . takes' n . view Text.lines++    The above program will never bring more than one chunk of text (~ 32 KB) into+    memory, no matter how long the lines are.++    * /Lenses/++    As this example shows, one superficial difference from @Data.Text.Lazy@ +    is that many of the operations, like 'lines',+    are \'lensified\'; this has a number of advantages (where it is possible), in particular +    it facilitates their use with 'Parser's of Text (in the general +    <http://hackage.haskell.org/package/pipes-parse-3.0.1/docs/Pipes-Parse-Tutorial.html pipes-parse> +    sense.) +    Each such lens, e.g. 'lines', 'chunksOf' or 'splitAt', reduces to the +    intuitively corresponding function when used with @view@ or @(^.)@. ++    Note similarly that many equivalents of 'Text -> Text' functions are exported here as 'Pipe's.+    They reduce to the intuitively corresponding functions when used with '(>->)'. Thus something like++>  stripLines = Text.unlines . Group.maps (>-> Text.stripStart) . view Text.lines ++    would drop the leading white space from each line. ++    The lens combinators+    you will find indispensible are @view@ / @(^.)@), @zoom@ and probably @over@. These+    are supplied by both <http://hackage.haskell.org/package/lens lens> and +    <http://hackage.haskell.org/package/lens-family lens-family> The use of 'zoom' is explained+    in <http://hackage.haskell.org/package/pipes-parse-3.0.1/docs/Pipes-Parse-Tutorial.html Pipes.Parse.Tutorial> +    and to some extent in the @Pipes.Text.Encoding@ module here. The use of+    @over@ is simple, illustrated by the fact that we can rewrite @stripLines@ above as++>  stripLines = over Text.lines $ maps (>-> stripStart)+++    * Special types: @Producer Text m (Producer Text m r)@ and @FreeT (Producer Text m) m r@+    +    These simple 'lines' examples reveal a more important difference from @Data.Text.Lazy@ . +    This is in the types that are most closely associated with our central text type, +    @Producer Text m r@.  In @Data.Text@ and @Data.Text.Lazy@ we find functions like++>   splitAt  :: Int -> Text -> (Text, Text)+>   lines    ::        Text -> [Text]+>   chunksOf :: Int -> Text -> [Text]++    which relate a Text with a pair of Texts or a list of Texts. +    The corresponding functions here (taking account of \'lensification\') are ++>   view . splitAt  :: (Monad m, Integral n) => n -> Producer Text m r -> Producer Text m (Producer Text m r)+>   view lines      :: Monad m               =>      Producer Text m r -> FreeT (Producer Text m) m r+>   view . chunksOf :: (Monad m, Integral n) => n -> Producer Text m r -> FreeT (Producer Text m) m r++    Some of the types may be more readable if you imagine that we have introduced+    our own type synonyms++>   type Text m r  = Producer T.Text m r+>   type Texts m r = FreeT (Producer T.Text m) m r++    Then we would think of the types above as++>   view . splitAt  :: (Monad m, Integral n) => n -> Text m r -> Text m (Text m r)+>   view lines      :: (Monad m)             =>      Text m r -> Texts m r+>   view . chunksOf :: (Monad m, Integral n) => n -> Text m r -> Texts m r++    which brings one closer to the types of the similar functions in @Data.Text.Lazy@++    In the type @Producer Text m (Producer Text m r)@ the second +    element of the \'pair\' of effectful Texts cannot simply be retrieved +    with something like 'snd'. This is an \'effectful\' pair, and one must work +    through the effects of the first element to arrive at the second Text stream, even+    if you are proposing to throw the Text in the first element away. +    Note that we use Control.Monad.join to fuse the pair back together, since it specializes to ++>    join :: Monad m => Producer Text m (Producer m r) -> Producer m r++    The return type of 'lines', 'words', 'chunksOf' and the other "splitter" functions,+    @FreeT (Producer m Text) m r@ -- our @Texts m r@ -- is the type of (effectful)+    lists of (effectful) texts. The type @([Text],r)@ might be seen to gather+    together things of the forms:++> r+> (Text,r)+> (Text, (Text, r))+> (Text, (Text, (Text, r)))+> (Text, (Text, (Text, (Text, r))))+> ...++    We might also have identified the sum of those types with @Free ((,) Text) r@ +    -- or, more absurdly, @FreeT ((,) Text) Identity r@. Similarly, @FreeT (Producer Text m) m r@+    encompasses all the members of the sequence:+   +> m r+> Producer Text m r+> Producer Text m (Producer Text m r)+> Producer Text m (Producer Text m (Producer Text m r))+> ...++    One might think that ++>   lines :: Monad m => Lens' (Producer Text m r) (FreeT (Producer Text m) m r)+>   view . lines :: Monad m => Producer Text m r -> FreeT (Producer Text m) m r++    should really have the type+    +>   lines :: Monad m => Pipe Text Text m r++    as e.g. 'toUpper' does. But this would spoil the control we are +    attempting to maintain over the size of chunks. It is in fact just +    as unreasonable to want such a pipe as to want++> Data.Text.Lazy.lines :: Text -> Text ++    to 'rechunk' the strict Text chunks inside the lazy Text to respect +    line boundaries. In fact we have ++> Data.Text.Lazy.lines :: Text -> [Text]+> Prelude.lines :: String -> [String]++    where the elements of the list are themselves lazy Texts or Strings; the use+    of @FreeT (Producer Text m) m r@ is simply the 'effectful' version of this. +    +    The @Pipes.Group@ module, which can generally be imported without qualification,+    provides many functions for working with things of type @FreeT (Producer a m) m r@+    +   +   -}+ -- | Convert a lazy 'TL.Text' into a 'Producer' of strict 'Text's fromLazy :: (Monad m) => TL.Text -> Producer' Text m () fromLazy  = foldrChunks (\e a -> yield e >> a) (return ()) @@ -275,7 +353,7 @@  -- | @toCaseFold@, @toLower@, @toUpper@ and @stripStart@ are standard 'Text' utilities,  -- here acting as 'Text' pipes, rather as they would  on a lazy text-toCaseFold :: Monad m => Pipe Text Text m ()+toCaseFold :: Monad m => Pipe Text Text m r toCaseFold = P.map T.toCaseFold {-# INLINEABLE toCaseFold #-} @@ -285,7 +363,7 @@   -- | lowercase incoming 'Text'-toLower :: Monad m => Pipe Text Text m ()+toLower :: Monad m => Pipe Text Text m r toLower = P.map T.toLower {-# INLINEABLE toLower #-} @@ -294,7 +372,7 @@   #-}  -- | uppercase incoming 'Text'-toUpper :: Monad m => Pipe Text Text m ()+toUpper :: Monad m => Pipe Text Text m r toUpper = P.map T.toUpper {-# INLINEABLE toUpper #-}
Pipes/Text/Encoding.hs view
@@ -2,16 +2,18 @@  -- | This module uses the stream decoding functions from Michael Snoyman's new --  <http://hackage.haskell.org/package/text-stream-decode text-stream-decode> ---  package to define decoding functions and lenses.  +--  package to define decoding functions and lenses.  The exported names+--  conflict with names in @Data.Text.Encoding@ but the module can otherwise be +--  imported unqualified.   module Pipes.Text.Encoding     (      -- * The Lens or Codec type     -- $lenses     Codec-    -- * Viewing the Text in a ByteString-    -- $codecs     , decode+    -- * \'Viewing\' the Text in a byte stream+    -- $codecs     , utf8     , utf8Pure     , utf16LE@@ -55,7 +57,6 @@ import Data.Word (Word8) import Pipes - type Lens' a b = forall f . Functor f => (b -> f b) -> (a -> f a)  {- $lenses@@ -66,8 +67,9 @@      >   type Lens' a b = forall f . Functor f => (b -> f b) -> (a -> f a) -    is just an alias for an ordinary Prelude type.  Thus you use any codec with-    the @view@ / @(^.)@ and @zoom@ functions from those libraries.+    is just an alias for a Prelude type. Thus you use any particular codec with+    the @view@ / @(^.)@ , @zoom@ and @over@ functions from either of those libraries;+    we presuppose neither since we already have access to the types they require.      -} @@ -76,11 +78,12 @@     .  Monad m     => Lens' (Producer ByteString m r)              (Producer Text m (Producer ByteString m r))-             + {- | 'decode' is just the ordinary @view@ or @(^.)@ of the lens libraries;-      exported here for convience+      exported here under a name appropriate to the material. All of these are+      the same:  ->    decode utf8 p = decodeUtf8 p = view utf8 p = p ^. utf+>    decode utf8 p = decodeUtf8 p = view utf8 p = p ^. utf8  -} @@ -88,6 +91,85 @@ decode codec a = getConstant (codec Constant a)  +{- $codecs+    +    Each Codec-lens looks into a byte stream that is supposed to contain text.+    The particular \'Codec\' lenses are named in accordance with the expected +    encoding, 'utf8', 'utf16LE' etc. To turn a Codec into an ordinary function, +    use @view@ / @(^.)@ -- here also called 'decode':++>   view utf8 :: Producer ByteString m r -> Producer Text m (Producer ByteString m r)+>   decode utf8 Byte.stdin :: Producer Text IO (Producer ByteString IO r)+>   Bytes.stdin ^. utf8 ::  Producer Text IO (Producer ByteString IO r)++    Uses of a codec with @view@ or @(^.)@ or 'decode' can always be replaced by the specialized +    decoding functions exported here, e.g. ++>   decodeUtf8 ::  Producer ByteString m r -> Producer Text m (Producer ByteString m r)+>   decodeUtf8 Byte.stdin :: Producer Text IO (Producer ByteString IO r)++    The stream of text that a @Codec@ \'sees\' in the stream of bytes begins at its head. +    At any point of decoding failure, the stream of text ends and reverts to (returns) +    the original byte stream. Thus if the first bytes are already+    un-decodable, the whole ByteString producer will be returned, i.e.++>   view utf8 bytestream ++    will just come to the same as ++>   return bytestream++    Where there is no decoding failure, the return value of the text stream will be+    an empty byte stream followed by its own return value.  In all cases you must+    deal with the fact that it is a /ByteString producer/ that is returned, even if+    it can be thrown away with @Control.Monad.void@++>   void (Bytes.stdin ^. utf8) :: Producer Text IO ()+    +    @zoom@ converts a Text parser into a ByteString parser:++>   zoom utf8 drawChar :: Monad m => StateT (Producer ByteString m r) m (Maybe Char)++    or, using the type synonymn from @Pipes.Parse@:+    +>   zoom utf8 drawChar :: Monad m => Parser ByteString m (Maybe Char)++    Thus we can define a ByteString parser like this:+    +>   withNextByte :: Parser ByteString m (Maybe Char, Maybe Word8))) +>   withNextByte = do char_ <- zoom utf8 Text.drawChar+>                     byte_ <- Bytes.peekByte+>                     return (char_, byte_)++     Though @withNextByte@ is partly defined with a Text parser 'drawChar'; +     but it is a ByteString parser; it will return the first valid utf8-encoded +     Char in a ByteString, whatever its length, +     and the first byte of the next character, if they exist. Because +     we \'draw\' one and \'peek\' at the other, the parser as a whole only +     advances one Char's length along the bytestring, whatever that length may be.+     See the slightly more complex example \'decode.hs\' in the +     <http://www.haskellforall.com/2014/02/pipes-parse-30-lens-based-parsing.html#batteries-included haskellforall> +     discussion of this type of byte stream parsing.+    -}++utf8 :: Codec+utf8 = mkCodec decodeUtf8 TE.encodeUtf8++utf8Pure :: Codec+utf8Pure = mkCodec decodeUtf8Pure TE.encodeUtf8++utf16LE :: Codec+utf16LE = mkCodec decodeUtf16LE TE.encodeUtf16LE++utf16BE :: Codec+utf16BE = mkCodec decodeUtf16BE TE.encodeUtf16BE++utf32LE :: Codec+utf32LE = mkCodec decodeUtf32LE TE.encodeUtf32LE++utf32BE :: Codec+utf32BE = mkCodec decodeUtf32BE TE.encodeUtf32BE+ decodeStream :: Monad m         => (B.ByteString -> DecodeResult)         -> Producer ByteString m r -> Producer Text m (Producer ByteString m r)@@ -177,75 +259,6 @@         -> Codec mkCodec dec enc = \k p0 -> fmap (\p -> join (for p (yield . enc)))  (k (dec p0)) --{- $codecs-    -    Each codec/lens looks into a byte stream that is supposed to contain text.-    The particular \'Codec\' lenses are named in accordance with the expected -    encoding, 'utf8', 'utf16LE' etc. @view@ / @(^.)@ -- here also called 'decode' -- -    turns a Codec into a function:-->   view utf8 :: Producer ByteString m r -> Producer Text m (Producer ByteString m r)->   decode utf8 Byte.stdin :: Producer Text IO (Producer ByteString IO r)->   Bytes.stdin ^. utf8 ::  Producer Text IO (Producer ByteString IO r)--    Uses of a codec with @view@ / @(^.)@ / 'decode' can always be replaced by the specialized -    decoding functions exported here, e.g. -->   decodeUtf8 ::  Producer ByteString m r -> Producer Text m (Producer ByteString m r)->   decodeUtf8 Byte.stdin :: Producer Text IO (Producer ByteString IO r)--    The stream of text a @Codec@ \'sees\' in the stream of bytes begins at its head. -    At any point of decoding failure, the stream of text ends and reverts to (returns) -    the original byte stream. Thus if the first bytes are already-    un-decodable, the whole ByteString producer will be returned, i.e.-->   view utf8 bytestream --    will just come to the same as -->   return bytestream--    Where there is no decoding failure, the return value of the text stream will be-    an empty byte stream followed by its own return value.  In all cases you must-    deal with the fact that it is a ByteString producer that is returned, even if-    it can be thrown away with @Control.Monad.void@-->   void (Bytes.stdin ^. utf8) :: Producer Text IO ()-    -    @zoom@ converts a Text parser into a ByteString parser:-->   zoom utf8 drawChar :: Monad m => StateT (Producer ByteString m r) m (Maybe Char)-> ->   withNextByte :: Parser ByteString m (Maybe Char, Maybe Word8))) ->   withNextByte = do char_ <- zoom utf8 Text.drawChar->                     byte_ <- Bytes.peekByte->                     return (char_, byte_)--     @withNextByte@ will return the first valid Char in a ByteString, -     and the first byte of the next character, if they exists. Because -     we \'draw\' one and \'peek\' at the other, the parser as a whole only -     advances one Char's length along the bytestring.--    -}--utf8 :: Codec-utf8 = mkCodec decodeUtf8 TE.encodeUtf8--utf8Pure :: Codec-utf8Pure = mkCodec decodeUtf8Pure TE.encodeUtf8--utf16LE :: Codec-utf16LE = mkCodec decodeUtf16LE TE.encodeUtf16LE--utf16BE :: Codec-utf16BE = mkCodec decodeUtf16BE TE.encodeUtf16BE--utf32LE :: Codec-utf32LE = mkCodec decodeUtf32LE TE.encodeUtf32LE--utf32BE :: Codec-utf32BE = mkCodec decodeUtf32BE TE.encodeUtf32BE   {- $ascii
Pipes/Text/IO.hs view
@@ -35,14 +35,15 @@ {- $textio     Where pipes IO replaces lazy IO, @Producer Text m r@ replaces lazy 'Text'.      This module exports some convenient functions for producing and consuming -    pipes 'Text' in IO, with caveats described below. The main points are as in -    <https://hackage.haskell.org/package/pipes-bytestring-1.0.0/docs/Pipes-ByteString.html @Pipes.ByteString@>+    pipes 'Text' in IO, namely, 'readFile', 'writeFile', 'fromHandle', 'toHandle', +    'stdin' and 'stdout'.  Some caveats described below.      -    An 'IO.Handle' can be associated with a 'Producer' or 'Consumer' according as it is read or written to.--    To stream to or from 'IO.Handle's, one can use 'fromHandle' or 'toHandle'.  For-    example, the following program copies a document from one file to another:-+    The main points are as in +    <https://hackage.haskell.org/package/pipes-bytestring-1.0.0/docs/Pipes-ByteString.html Pipes.ByteString>+    +    An 'IO.Handle' can be associated with a 'Producer' or 'Consumer' according +    as it is read or written to.+     > import Pipes > import qualified Pipes.Text as Text > import qualified Pipes.Text.IO as Text@@ -62,11 +63,12 @@ > > main = runSafeT $ runEffect $ Text.readFile "inFile.txt" >-> Text.writeFile "outFile.txt" -    You can stream to and from 'stdin' and 'stdout' using the predefined 'stdin'+    Finally, you can stream to and from 'stdin' and 'stdout' using the predefined 'stdin'     and 'stdout' pipes, as with the following \"echo\" program:  > main = runEffect $ Text.stdin >-> Text.stdout + -}  @@ -78,11 +80,11 @@      * Like the functions in @Data.Text.IO@, they attempt to work with the system encoding.    -    * Like the functions in @Data.Text.IO@, they are slower than ByteString operations. Where+    * Like the functions in @Data.Text.IO@, they significantly slower than ByteString operations. Where        you know what encoding you are working with, use @Pipes.ByteString@ and @Pipes.Text.Encoding@ instead,        e.g. @view utf8 Bytes.stdin@ instead of @Text.stdin@   -    * Like the functions in  @Data.Text.IO@ , they use Text exceptions. +    * Like the functions in  @Data.Text.IO@ , they use Text exceptions, not the standard Pipes protocols.      Something like
pipes-text.cabal view
@@ -1,5 +1,5 @@ name:                pipes-text-version:             0.0.0.8+version:             0.0.0.9 synopsis:            Text pipes. description:         * This package will be in a draft, or testing, phase until version 0.0.1. Please report any installation difficulties, or any wisdom about the api, on the github page or the <https://groups.google.com/forum/#!forum/haskell-pipes pipes list>                      .