packages feed

pipes-text 0.0.0.7 → 0.0.0.8

raw patch · 5 files changed

+305/−115 lines, 5 files

Files

Pipes/Text.hs view
@@ -1,63 +1,95 @@ {-# LANGUAGE RankNTypes, TypeFamilies, BangPatterns, Trustworthy #-} -{-| This module provides @pipes@ utilities for \"text streams\", which are-    streams of 'Text' chunks. The individual chunks are uniformly @strict@, but -    a 'Producer' can be converted to and from lazy 'Text's, though this is generally -    unwise.  Where pipes IO replaces lazy IO, 'Producer Text m r' replaces lazy 'Text'.-    An 'IO.Handle' can be associated with a 'Producer' or 'Consumer' according as it is read or written to.+{-| This package provides @pipes@ utilities for \'text streams\', which are+    streams of 'Text' chunks. The individual chunks are uniformly @strict@, and thus you +    will generally want @Data.Text@ in scope.  But the type @Producer Text m r@ is+    in some ways the pipes equivalent of the lazy @Text@ type. -    To stream to or from 'IO.Handle's, one can use 'fromHandle' or 'toHandle'.  For-    example, the following program copies a document from one file to another:+    This module provides many functions equivalent in one way or another to +    the 'pure' functions in +    <https://hackage.haskell.org/package/text-1.1.0.0/docs/Data-Text-Lazy.html Data.Text.Lazy>. +    They transform, divide, group and fold text streams. Though @Producer Text m r@ +    is the type of \'effectful Text\', the functions in this module are \'pure\' +    in the sense that they are uniformly monad-independent.+    Simple IO operations are defined in @Pipes.Text.IO@ -- as lazy IO @Text@ +    operations are in @Data.Text.Lazy.IO@. Interoperation with @ByteString@ +    is provided in @Pipes.Text.Encoding@, which parallels @Data.Text.Lazy.Encoding@.  -> import Pipes-> import qualified Pipes.Text as Text-> import qualified Pipes.Text.IO as Text-> import System.IO->-> main =->     withFile "inFile.txt"  ReadMode  $ \hIn  ->->     withFile "outFile.txt" WriteMode $ \hOut ->->     runEffect $ Text.fromHandle hIn >-> Text.toHandle hOut+    The Text type exported by @Data.Text.Lazy@ is basically '[Text]'. The implementation+    is arranged so that the individual strict 'Text' chunks are kept to a reasonable size; +    the user is not aware of the divisions between the connected 'Text' chunks. +    So also here: the functions in this module are designed to operate on streams that+    are insensitive to text boundaries.  This means that they may freely split+    text into smaller texts and /discard empty texts/.  However, the objective is +    that they should /never concatenate texts/ in order to provide strict upper +    bounds on memory usage.  -To stream from files, the following is perhaps more Prelude-like (note that it uses Pipes.Safe):+    For example, to stream only the first three lines of 'stdin' to 'stdout' you+    might write:  > import Pipes > import qualified Pipes.Text as Text > import qualified Pipes.Text.IO as Text-> import Pipes.Safe->-> main = runSafeT $ runEffect $ Text.readFile "inFile.txt" >-> Text.writeFile "outFile.txt"--    You can stream to and from 'stdin' and 'stdout' using the predefined 'stdin'-    and 'stdout' pipes, as with the following \"echo\" program:--> main = runEffect $ Text.stdin >-> Text.stdout+> import Pipes.Group+> import Lens.Family +> +> main = runEffect $ takeLines 3 Text.stdin >-> Text.stdout+>   where +>     takeLines n = Text.unlines . takes' n . view Text.lines+>  -- or equivalently: +>  -- takeLines n = over Text.lines (takes' n) -    You can also translate pure lazy 'TL.Text's to and from pipes:+    The above program will never bring more than one chunk of text (~ 32 KB) into+    memory, no matter how long the lines are.+    +    As this example shows, one superficial difference from @Data.Text.Lazy@ +    is that many of the operations, like 'lines',+    are \'lensified\'; this has a number of advantages where it is possible, in particular +    it facilitates their use with 'Parser's of Text (in the general +    <http://hackage.haskell.org/package/pipes-parse-3.0.1/docs/Pipes-Parse-Tutorial.html pipes-parse> +    sense.) +    Each such expression, e.g. 'lines', 'chunksOf' or 'splitAt', reduces to the +    intuitively corresponding function when used with @view@ or @(^.)@.  The lens combinators+    you will find indispensible are \'view\'/ '(^.)', 'zoom' and probably 'over', which+    are supplied by both <http://hackage.haskell.org/package/lens lens> and +    <http://hackage.haskell.org/package/lens-family lens-family>+    +    A more important difference the example reveals is in the types closely associated with+    the central type, @Producer Text m r@.  In @Data.Text@ and @Data.Text.Lazy@+    we find functions like+    +>   splitAt :: Int -> Text -> (Text, Text)+>   lines :: Int -> Text -> [Text]+>   chunksOf :: Int -> Text -> [Text] -> main = runEffect $ Text.fromLazy (TL.pack "Hello, world!\n") >-> Text.stdout+    which relate a Text with a pair or list of Texts. The corresponding functions here (taking+    account of \'lensification\') are +    +>   view . splitAt :: (Monad m, Integral n) => n -> Producer Text m r -> Producer Text.Text m (Producer Text.Text m r)+>   view lines :: Monad m => Producer Text m r -> FreeT (Producer Text m) m r+>   view . chunksOf ::  (Monad m, Integral n) => n -> Producer Text m r -> FreeT (Producer Text m) m r -    In addition, this module provides many functions equivalent to lazy-    'Text' functions so that you can transform or fold text streams.  For-    example, to stream only the first three lines of 'stdin' to 'stdout' you-    might write:+    In the type @Producer Text m (Producer Text m r)@ the second +    element of the \'pair\' of of \'effectful Texts\' cannot simply be retrieved +    with 'snd'. This is an \'effectful\' pair, and one must work through the effects+    of the first element to arrive at the second Text stream. Similarly in @FreeT (Producer Text m) m r@,+    which corresponds with @[Text]@, on cannot simply drop 10 Producers and take the others;+    we can only get to the ones we want to take by working through their predecessors.+    +    Some of the types may be more readable if you imagine that we have introduced+    our own type synonyms+    +>   type Text m r = Producer T.Text m r+>   type Texts m r = FreeT (Producer T.Text m) m r -> import Pipes-> import qualified Pipes.Text as Text-> import qualified Pipes.Parse as Parse->-> main = runEffect $ takeLines 3 Text.stdin >-> Text.stdout->   where->     takeLines n = Text.unlines . Parse.takeFree n . Text.lines+    Then we would think of the types above as+    +>   view . splitAt :: (Monad m, Integral n) => n -> Text m r -> Text m (Text m r)+>   view lines :: (Monad m) => Text m r -> Texts m r+>   view . chunksOf :: (Monad m, Integral n) => n -> Text m r -> Texts m r -    The above program will never bring more than one chunk of text (~ 32 KB) into-    memory, no matter how long the lines are.+    which brings one closer to the types of the similar functions in @Data.Text.Lazy@ -    Note that functions in this library are designed to operate on streams that-    are insensitive to text boundaries.  This means that they may freely split-    text into smaller texts, /discard empty texts/.  However, apart from the -    special case of 'concatMap', they will /never concatenate texts/ in order -    to provide strict upper bounds on memory usage -- with the single exception of 'concatMap'.   -}  module Pipes.Text  (@@ -131,8 +163,7 @@     , unwords      -- * Re-exports-    , Lens'-    , Iso'+    -- $reexports     , module Data.ByteString     , module Data.Text     , module Data.Profunctor@@ -889,5 +920,12 @@ unwords = intercalate (yield $ T.singleton ' ') {-# INLINABLE unwords #-} ++{- $reexports+    +    @Data.Text@ re-exports the 'Text' type.++    @Pipes.Parse@ re-exports 'input', 'concat', 'FreeT' (the type) and the 'Parse' synonym. +-}
Pipes/Text/Encoding.hs view
@@ -1,16 +1,17 @@- {-# LANGUAGE RankNTypes, BangPatterns #-}--- | --- This module uses the stream decoding functions from the text-stream-decoding package--- to define decoding functions and lenses.+-- | This module uses the stream decoding functions from Michael Snoyman's new+--  <http://hackage.haskell.org/package/text-stream-decode text-stream-decode> +--  package to define decoding functions and lenses.    module Pipes.Text.Encoding     ( -    -- * Lens type-    -- $producers+    -- * The Lens or Codec type+    -- $lenses     Codec-    -- * Standard lenses for viewing Text in ByteString+    -- * Viewing the Text in a ByteString+    -- $codecs+    , decode     , utf8     , utf8Pure     , utf16LE@@ -18,12 +19,20 @@     , utf32LE     , utf32BE     -- * Non-lens decoding functions +    -- $decoders     , decodeUtf8     , decodeUtf8Pure     , decodeUtf16LE     , decodeUtf16BE     , decodeUtf32LE     , decodeUtf32BE+    -- * Re-encoding functions+    -- $encoders+    , encodeUtf8+    , encodeUtf16LE+    , encodeUtf16BE+    , encodeUtf32LE+    , encodeUtf32BE     -- * Functions for latin and ascii text     -- $ascii     , encodeAscii@@ -33,6 +42,7 @@     )      where +import Data.Functor.Constant (Constant(..)) import Data.Char (ord) import Data.ByteString as B  import Data.ByteString (ByteString)@@ -46,40 +56,38 @@ import Pipes  +type Lens' a b = forall f . Functor f => (b -> f b) -> (a -> f a) -{- $producers-    The 'Codec' type is just an aliased standard Prelude type. It is more or -    less the Lens\' type of the-    standard lens libraries, @lens@ and @lens-families@ so you can use -    the @view@ or @(^.)@ and @zoom@ functions from those libraries.+{- $lenses+    The 'Codec' type is a simple specializion of +    the @Lens'@ type synonymn used by the standard lens libraries, +    <http://hackage.haskell.org/package/lens lens> and +    <http://hackage.haskell.org/package/lens-family lens-family>. That type,      -    Each looks into a byte stream that is expected to contain text.-    The stream of text they 'see' in a bytestream ends by returning the original byte stream -    beginning at the point of failure, or the empty bytestream with its return value.-    They are named in accordance with the expected encoding, 'utf8', 'utf16LE' etc.+>   type Lens' a b = forall f . Functor f => (b -> f b) -> (a -> f a) ->   view utf8 :: Producer ByteString m r -> Producer Text m (Producer ByteString m r)->   Bytes.stdin ^. utf8 ::  Producer Text m (Producer ByteString m r)+    is just an alias for an ordinary Prelude type.  Thus you use any codec with+    the @view@ / @(^.)@ and @zoom@ functions from those libraries. -    @zoom@ converts a Text parser into a ByteString parser:-    ->   zoom utf8 drawChar :: Monad m => StateT (Producer ByteString m r) m (Maybe Char)-> ->   withNextByte :: Parser ByteString m (Maybe Char, Maybe Word8))) ->   withNextByte = do char_ <- zoom utf8 Text.drawChar->                     byte_ <- Bytes.peekByte->                     return (char_, byte_)+    -} -     @withNextByte@ will return the first valid Char in a ByteString, -     and the first byte of the next character, if they exists; because -     we draw one and peek at the other, we only advance one Char's length-     along the bytestring.+type Codec+    =  forall m r+    .  Monad m+    => Lens' (Producer ByteString m r)+             (Producer Text m (Producer ByteString m r))+             +{- | 'decode' is just the ordinary @view@ or @(^.)@ of the lens libraries;+      exported here for convience +>    decode utf8 p = decodeUtf8 p = view utf8 p = p ^. utf+ -}-type Codec  = forall f m r . (Functor f , Monad m ) => -     (Producer Text m (Producer ByteString m r) -> f (Producer Text m (Producer ByteString m r)))-     -> Producer ByteString m r -> f (Producer ByteString m r ) +decode :: ((b -> Constant b b) -> (a -> Constant b a)) -> a -> b+decode codec a = getConstant (codec Constant a)++ decodeStream :: Monad m         => (B.ByteString -> DecodeResult)         -> Producer ByteString m r -> Producer Text m (Producer ByteString m r)@@ -95,6 +103,22 @@                                                                  p') {-# INLINABLE decodeStream#-} +{- $decoders+   These are functions with the simple type:+   +>   decodeUtf8 :: Monad m => Producer ByteString m r -> Producer Text m (Producer ByteString m r)++   Thus in general ++>     decodeUtf8 = view utf8+>     decodeUtf16LE = view utf16LE++   and so forth, but these forms+   may be more convenient (and give better type errors!) where lenses are+   not desired.+-}++ decodeUtf8 :: Monad m => Producer ByteString m r -> Producer Text m (Producer ByteString m r) decodeUtf8 = decodeStream streamUtf8 {-# INLINE decodeUtf8 #-}@@ -119,6 +143,34 @@ decodeUtf32BE = decodeStream streamUtf32BE {-# INLINE decodeUtf32BE #-} ++{- $encoders+   These are simply defined +   +>      encodeUtf8 = yield . TE.encodeUtf8+   +   They are intended for use with 'for'+   +>      for Text.stdin encodeUtf8 :: Producer ByteString IO ()++   which would have the effect of +   +>      Text.stdin >-> Pipes.Prelude.map (TE.encodeUtf8)++   using the encoding functions from Data.Text.Encoding +-}++encodeUtf8 :: Monad m => Text -> Producer ByteString m ()+encodeUtf8 = yield . TE.encodeUtf8+encodeUtf16LE :: Monad m => Text -> Producer ByteString m ()+encodeUtf16LE = yield . TE.encodeUtf16LE+encodeUtf16BE :: Monad m => Text -> Producer ByteString m ()+encodeUtf16BE = yield . TE.encodeUtf16BE+encodeUtf32LE :: Monad m => Text -> Producer ByteString m ()+encodeUtf32LE = yield . TE.encodeUtf32LE+encodeUtf32BE :: Monad m => Text -> Producer ByteString m ()+encodeUtf32BE = yield . TE.encodeUtf32BE+ mkCodec :: (forall r m . Monad m =>             Producer ByteString m r -> Producer Text m (Producer ByteString m r ))         -> (Text -> ByteString)@@ -126,11 +178,57 @@ mkCodec dec enc = \k p0 -> fmap (\p -> join (for p (yield . enc)))  (k (dec p0))  -{- | An improper lens into a byte stream expected to be UTF-8 encoded; the associated-   text stream ends by returning the original bytestream beginning at the point of failure,-   or the empty bytestring for a well-encoded text. -   -}+{- $codecs+    +    Each codec/lens looks into a byte stream that is supposed to contain text.+    The particular \'Codec\' lenses are named in accordance with the expected +    encoding, 'utf8', 'utf16LE' etc. @view@ / @(^.)@ -- here also called 'decode' -- +    turns a Codec into a function: +>   view utf8 :: Producer ByteString m r -> Producer Text m (Producer ByteString m r)+>   decode utf8 Byte.stdin :: Producer Text IO (Producer ByteString IO r)+>   Bytes.stdin ^. utf8 ::  Producer Text IO (Producer ByteString IO r)++    Uses of a codec with @view@ / @(^.)@ / 'decode' can always be replaced by the specialized +    decoding functions exported here, e.g. ++>   decodeUtf8 ::  Producer ByteString m r -> Producer Text m (Producer ByteString m r)+>   decodeUtf8 Byte.stdin :: Producer Text IO (Producer ByteString IO r)++    The stream of text a @Codec@ \'sees\' in the stream of bytes begins at its head. +    At any point of decoding failure, the stream of text ends and reverts to (returns) +    the original byte stream. Thus if the first bytes are already+    un-decodable, the whole ByteString producer will be returned, i.e.++>   view utf8 bytestream ++    will just come to the same as ++>   return bytestream++    Where there is no decoding failure, the return value of the text stream will be+    an empty byte stream followed by its own return value.  In all cases you must+    deal with the fact that it is a ByteString producer that is returned, even if+    it can be thrown away with @Control.Monad.void@++>   void (Bytes.stdin ^. utf8) :: Producer Text IO ()+    +    @zoom@ converts a Text parser into a ByteString parser:++>   zoom utf8 drawChar :: Monad m => StateT (Producer ByteString m r) m (Maybe Char)+> +>   withNextByte :: Parser ByteString m (Maybe Char, Maybe Word8))) +>   withNextByte = do char_ <- zoom utf8 Text.drawChar+>                     byte_ <- Bytes.peekByte+>                     return (char_, byte_)++     @withNextByte@ will return the first valid Char in a ByteString, +     and the first byte of the next character, if they exists. Because +     we \'draw\' one and \'peek\' at the other, the parser as a whole only +     advances one Char's length along the bytestring.++    -}+ utf8 :: Codec utf8 = mkCodec decodeUtf8 TE.encodeUtf8 @@ -157,7 +255,7 @@ -}  ---  'encodeAscii' reduces as much of your stream of 'Text' actually is ascii to a byte stream,+-- | 'encodeAscii' reduces as much of your stream of 'Text' actually is ascii to a byte stream, --   returning the rest of the 'Text' at the first non-ascii 'Char'  encodeAscii :: Monad m => Producer Text m r -> Producer ByteString m (Producer Text m r)
Pipes/Text/IO.hs view
@@ -1,34 +1,14 @@ {-#LANGUAGE RankNTypes#-}--- | The operations exported here are a convenience, like the similar operations in ---   @Data.Text.IO@ , or rather, @Data.Text.Lazy.IO@, since @Producer Text m r@ is---   'effectful text' and something like the pipes equivalent of lazy Text.------   * Like the functions in @Data.Text.IO@, they attempt to work with the system encoding. ---   ---   * Like the functions in @Data.Text.IO@, they are slower than ByteString operations. Where---      you know what encoding you are working with, use @Pipes.ByteString@ and @Pipes.Text.Encoding@ instead,---      e.g. @view utf8 Bytes.stdin@ instead of @Text.stdin@---   ---   * Like the functions in  @Data.Text.IO@ , they use Text exceptions. ------  Something like ---  ---   >  view utf8 . Bytes.fromHandle :: Handle -> Producer Text IO (Producer ByteString m ()) --- ---  yields a stream of Text, and follows---  standard pipes protocols by reverting to (i.e. returning) the underlying byte stream---  upon reaching any decoding error. (See especially the pipes-binary package.) ------ By contrast, something like --- ---  > Text.fromHandle :: Handle -> Producer Text IO () --- --- supplies a stream of text returning '()', which is convenient for many tasks, --- but violates the pipes @pipes-binary@ approach to decoding errors and --- throws an exception of the kind characteristic of the @text@ library instead. + module Pipes.Text.IO     ( +   -- * Text IO+   -- $textio+   +   -- * Caveats+   -- $caveats+       -- * Producers    fromHandle    , stdin@@ -52,6 +32,76 @@ import Pipes.Safe (MonadSafe(..), Base(..)) import Prelude hiding (readFile, writeFile) +{- $textio+    Where pipes IO replaces lazy IO, @Producer Text m r@ replaces lazy 'Text'. +    This module exports some convenient functions for producing and consuming +    pipes 'Text' in IO, with caveats described below. The main points are as in +    <https://hackage.haskell.org/package/pipes-bytestring-1.0.0/docs/Pipes-ByteString.html @Pipes.ByteString@>+    +    An 'IO.Handle' can be associated with a 'Producer' or 'Consumer' according as it is read or written to.++    To stream to or from 'IO.Handle's, one can use 'fromHandle' or 'toHandle'.  For+    example, the following program copies a document from one file to another:++> import Pipes+> import qualified Pipes.Text as Text+> import qualified Pipes.Text.IO as Text+> import System.IO+>+> main =+>     withFile "inFile.txt"  ReadMode  $ \hIn  ->+>     withFile "outFile.txt" WriteMode $ \hOut ->+>     runEffect $ Text.fromHandle hIn >-> Text.toHandle hOut++To stream from files, the following is perhaps more Prelude-like (note that it uses Pipes.Safe):++> import Pipes+> import qualified Pipes.Text as Text+> import qualified Pipes.Text.IO as Text+> import Pipes.Safe+>+> main = runSafeT $ runEffect $ Text.readFile "inFile.txt" >-> Text.writeFile "outFile.txt"++    You can stream to and from 'stdin' and 'stdout' using the predefined 'stdin'+    and 'stdout' pipes, as with the following \"echo\" program:++> main = runEffect $ Text.stdin >-> Text.stdout++-}+++{- $caveats++    The operations exported here are a convenience, like the similar operations in +    @Data.Text.IO@  (or rather, @Data.Text.Lazy.IO@, since, again, @Producer Text m r@ is+    'effectful text' and something like the pipes equivalent of lazy Text.)++    * Like the functions in @Data.Text.IO@, they attempt to work with the system encoding. +  +    * Like the functions in @Data.Text.IO@, they are slower than ByteString operations. Where+       you know what encoding you are working with, use @Pipes.ByteString@ and @Pipes.Text.Encoding@ instead,+       e.g. @view utf8 Bytes.stdin@ instead of @Text.stdin@+  +    * Like the functions in  @Data.Text.IO@ , they use Text exceptions. ++   Something like + +>  view utf8 . Bytes.fromHandle :: Handle -> Producer Text IO (Producer ByteString m ()) ++   yields a stream of Text, and follows+   standard pipes protocols by reverting to (i.e. returning) the underlying byte stream+   upon reaching any decoding error. (See especially the pipes-binary package.) ++  By contrast, something like ++> Text.fromHandle :: Handle -> Producer Text IO () ++  supplies a stream of text returning '()', which is convenient for many tasks, +  but violates the pipes @pipes-binary@ approach to decoding errors and +  throws an exception of the kind characteristic of the @text@ library instead.+++-}  {-| Convert a 'IO.Handle' into a text stream using a text size      determined by the good sense of the text library. Note with the remarks
README.md view
@@ -1,6 +1,10 @@ pipes-text ========== -This package follows the rule `pipes-text : pipes-bytestring :: text : bytestring` It has three modules, `Pipes.Text` , `Pipes.Text.Encoding` and `Pipes.Text.IO`; the division has more or less the significance it has in the `text` library. +This package follows the rule:++    pipes-text : pipes-bytestring :: text : bytestring++The division of three modules, `Pipes.Text` , `Pipes.Text.Encoding` and `Pipes.Text.IO` has more or less the significance it has in the `text` library.   Note that the module `Pipes.Text.IO` uses version 0.11.3 or later of the `text` library. (It thus works with the version of `text` that came with the 2013 Haskell Platform. To use an older `text`, install with the flag `-fnoio`
pipes-text.cabal view
@@ -1,7 +1,7 @@ name:                pipes-text-version:             0.0.0.7+version:             0.0.0.8 synopsis:            Text pipes.-description:         * This package will be in a draft, or testing, phase until version 0.0.1. Please report any installation difficulties, or any wisdom about the api, on the github page!+description:         * This package will be in a draft, or testing, phase until version 0.0.1. Please report any installation difficulties, or any wisdom about the api, on the github page or the <https://groups.google.com/forum/#!forum/haskell-pipes pipes list>                      .                      This organization of the package follows the rule                       .