diff --git a/Pipes/Text.hs b/Pipes/Text.hs --- a/Pipes/Text.hs +++ b/Pipes/Text.hs @@ -1,63 +1,95 @@ {-# LANGUAGE RankNTypes, TypeFamilies, BangPatterns, Trustworthy #-} -{-| This module provides @pipes@ utilities for \"text streams\", which are - streams of 'Text' chunks. The individual chunks are uniformly @strict@, but - a 'Producer' can be converted to and from lazy 'Text's, though this is generally - unwise. Where pipes IO replaces lazy IO, 'Producer Text m r' replaces lazy 'Text'. - An 'IO.Handle' can be associated with a 'Producer' or 'Consumer' according as it is read or written to. +{-| This package provides @pipes@ utilities for \'text streams\', which are + streams of 'Text' chunks. The individual chunks are uniformly @strict@, and thus you + will generally want @Data.Text@ in scope. But the type @Producer Text m r@ is + in some ways the pipes equivalent of the lazy @Text@ type. - To stream to or from 'IO.Handle's, one can use 'fromHandle' or 'toHandle'. For - example, the following program copies a document from one file to another: + This module provides many functions equivalent in one way or another to + the 'pure' functions in + . + They transform, divide, group and fold text streams. Though @Producer Text m r@ + is the type of \'effectful Text\', the functions in this module are \'pure\' + in the sense that they are uniformly monad-independent. + Simple IO operations are defined in @Pipes.Text.IO@ -- as lazy IO @Text@ + operations are in @Data.Text.Lazy.IO@. Interoperation with @ByteString@ + is provided in @Pipes.Text.Encoding@, which parallels @Data.Text.Lazy.Encoding@. -> import Pipes -> import qualified Pipes.Text as Text -> import qualified Pipes.Text.IO as Text -> import System.IO -> -> main = -> withFile "inFile.txt" ReadMode $ \hIn -> -> withFile "outFile.txt" WriteMode $ \hOut -> -> runEffect $ Text.fromHandle hIn >-> Text.toHandle hOut + The Text type exported by @Data.Text.Lazy@ is basically '[Text]'. The implementation + is arranged so that the individual strict 'Text' chunks are kept to a reasonable size; + the user is not aware of the divisions between the connected 'Text' chunks. + So also here: the functions in this module are designed to operate on streams that + are insensitive to text boundaries. This means that they may freely split + text into smaller texts and /discard empty texts/. However, the objective is + that they should /never concatenate texts/ in order to provide strict upper + bounds on memory usage. -To stream from files, the following is perhaps more Prelude-like (note that it uses Pipes.Safe): + For example, to stream only the first three lines of 'stdin' to 'stdout' you + might write: > import Pipes > import qualified Pipes.Text as Text > import qualified Pipes.Text.IO as Text -> import Pipes.Safe -> -> main = runSafeT $ runEffect $ Text.readFile "inFile.txt" >-> Text.writeFile "outFile.txt" - - You can stream to and from 'stdin' and 'stdout' using the predefined 'stdin' - and 'stdout' pipes, as with the following \"echo\" program: - -> main = runEffect $ Text.stdin >-> Text.stdout +> import Pipes.Group +> import Lens.Family +> +> main = runEffect $ takeLines 3 Text.stdin >-> Text.stdout +> where +> takeLines n = Text.unlines . takes' n . view Text.lines +> -- or equivalently: +> -- takeLines n = over Text.lines (takes' n) - You can also translate pure lazy 'TL.Text's to and from pipes: + The above program will never bring more than one chunk of text (~ 32 KB) into + memory, no matter how long the lines are. + + As this example shows, one superficial difference from @Data.Text.Lazy@ + is that many of the operations, like 'lines', + are \'lensified\'; this has a number of advantages where it is possible, in particular + it facilitates their use with 'Parser's of Text (in the general + + sense.) + Each such expression, e.g. 'lines', 'chunksOf' or 'splitAt', reduces to the + intuitively corresponding function when used with @view@ or @(^.)@. The lens combinators + you will find indispensible are \'view\'/ '(^.)', 'zoom' and probably 'over', which + are supplied by both and + + + A more important difference the example reveals is in the types closely associated with + the central type, @Producer Text m r@. In @Data.Text@ and @Data.Text.Lazy@ + we find functions like + +> splitAt :: Int -> Text -> (Text, Text) +> lines :: Int -> Text -> [Text] +> chunksOf :: Int -> Text -> [Text] -> main = runEffect $ Text.fromLazy (TL.pack "Hello, world!\n") >-> Text.stdout + which relate a Text with a pair or list of Texts. The corresponding functions here (taking + account of \'lensification\') are + +> view . splitAt :: (Monad m, Integral n) => n -> Producer Text m r -> Producer Text.Text m (Producer Text.Text m r) +> view lines :: Monad m => Producer Text m r -> FreeT (Producer Text m) m r +> view . chunksOf :: (Monad m, Integral n) => n -> Producer Text m r -> FreeT (Producer Text m) m r - In addition, this module provides many functions equivalent to lazy - 'Text' functions so that you can transform or fold text streams. For - example, to stream only the first three lines of 'stdin' to 'stdout' you - might write: + In the type @Producer Text m (Producer Text m r)@ the second + element of the \'pair\' of of \'effectful Texts\' cannot simply be retrieved + with 'snd'. This is an \'effectful\' pair, and one must work through the effects + of the first element to arrive at the second Text stream. Similarly in @FreeT (Producer Text m) m r@, + which corresponds with @[Text]@, on cannot simply drop 10 Producers and take the others; + we can only get to the ones we want to take by working through their predecessors. + + Some of the types may be more readable if you imagine that we have introduced + our own type synonyms + +> type Text m r = Producer T.Text m r +> type Texts m r = FreeT (Producer T.Text m) m r -> import Pipes -> import qualified Pipes.Text as Text -> import qualified Pipes.Parse as Parse -> -> main = runEffect $ takeLines 3 Text.stdin >-> Text.stdout -> where -> takeLines n = Text.unlines . Parse.takeFree n . Text.lines + Then we would think of the types above as + +> view . splitAt :: (Monad m, Integral n) => n -> Text m r -> Text m (Text m r) +> view lines :: (Monad m) => Text m r -> Texts m r +> view . chunksOf :: (Monad m, Integral n) => n -> Text m r -> Texts m r - The above program will never bring more than one chunk of text (~ 32 KB) into - memory, no matter how long the lines are. + which brings one closer to the types of the similar functions in @Data.Text.Lazy@ - Note that functions in this library are designed to operate on streams that - are insensitive to text boundaries. This means that they may freely split - text into smaller texts, /discard empty texts/. However, apart from the - special case of 'concatMap', they will /never concatenate texts/ in order - to provide strict upper bounds on memory usage -- with the single exception of 'concatMap'. -} module Pipes.Text ( @@ -131,8 +163,7 @@ , unwords -- * Re-exports - , Lens' - , Iso' + -- $reexports , module Data.ByteString , module Data.Text , module Data.Profunctor @@ -889,5 +920,12 @@ unwords = intercalate (yield $ T.singleton ' ') {-# INLINABLE unwords #-} + +{- $reexports + + @Data.Text@ re-exports the 'Text' type. + + @Pipes.Parse@ re-exports 'input', 'concat', 'FreeT' (the type) and the 'Parse' synonym. +-} diff --git a/Pipes/Text/Encoding.hs b/Pipes/Text/Encoding.hs --- a/Pipes/Text/Encoding.hs +++ b/Pipes/Text/Encoding.hs @@ -1,16 +1,17 @@ - {-# LANGUAGE RankNTypes, BangPatterns #-} --- | --- This module uses the stream decoding functions from the text-stream-decoding package --- to define decoding functions and lenses. +-- | This module uses the stream decoding functions from Michael Snoyman's new +-- +-- package to define decoding functions and lenses. module Pipes.Text.Encoding ( - -- * Lens type - -- $producers + -- * The Lens or Codec type + -- $lenses Codec - -- * Standard lenses for viewing Text in ByteString + -- * Viewing the Text in a ByteString + -- $codecs + , decode , utf8 , utf8Pure , utf16LE @@ -18,12 +19,20 @@ , utf32LE , utf32BE -- * Non-lens decoding functions + -- $decoders , decodeUtf8 , decodeUtf8Pure , decodeUtf16LE , decodeUtf16BE , decodeUtf32LE , decodeUtf32BE + -- * Re-encoding functions + -- $encoders + , encodeUtf8 + , encodeUtf16LE + , encodeUtf16BE + , encodeUtf32LE + , encodeUtf32BE -- * Functions for latin and ascii text -- $ascii , encodeAscii @@ -33,6 +42,7 @@ ) where +import Data.Functor.Constant (Constant(..)) import Data.Char (ord) import Data.ByteString as B import Data.ByteString (ByteString) @@ -46,40 +56,38 @@ import Pipes +type Lens' a b = forall f . Functor f => (b -> f b) -> (a -> f a) -{- $producers - The 'Codec' type is just an aliased standard Prelude type. It is more or - less the Lens\' type of the - standard lens libraries, @lens@ and @lens-families@ so you can use - the @view@ or @(^.)@ and @zoom@ functions from those libraries. +{- $lenses + The 'Codec' type is a simple specializion of + the @Lens'@ type synonymn used by the standard lens libraries, + and + . That type, - Each looks into a byte stream that is expected to contain text. - The stream of text they 'see' in a bytestream ends by returning the original byte stream - beginning at the point of failure, or the empty bytestream with its return value. - They are named in accordance with the expected encoding, 'utf8', 'utf16LE' etc. +> type Lens' a b = forall f . Functor f => (b -> f b) -> (a -> f a) -> view utf8 :: Producer ByteString m r -> Producer Text m (Producer ByteString m r) -> Bytes.stdin ^. utf8 :: Producer Text m (Producer ByteString m r) + is just an alias for an ordinary Prelude type. Thus you use any codec with + the @view@ / @(^.)@ and @zoom@ functions from those libraries. - @zoom@ converts a Text parser into a ByteString parser: - -> zoom utf8 drawChar :: Monad m => StateT (Producer ByteString m r) m (Maybe Char) -> -> withNextByte :: Parser ByteString m (Maybe Char, Maybe Word8))) -> withNextByte = do char_ <- zoom utf8 Text.drawChar -> byte_ <- Bytes.peekByte -> return (char_, byte_) + -} - @withNextByte@ will return the first valid Char in a ByteString, - and the first byte of the next character, if they exists; because - we draw one and peek at the other, we only advance one Char's length - along the bytestring. +type Codec + = forall m r + . Monad m + => Lens' (Producer ByteString m r) + (Producer Text m (Producer ByteString m r)) + +{- | 'decode' is just the ordinary @view@ or @(^.)@ of the lens libraries; + exported here for convience +> decode utf8 p = decodeUtf8 p = view utf8 p = p ^. utf + -} -type Codec = forall f m r . (Functor f , Monad m ) => - (Producer Text m (Producer ByteString m r) -> f (Producer Text m (Producer ByteString m r))) - -> Producer ByteString m r -> f (Producer ByteString m r ) +decode :: ((b -> Constant b b) -> (a -> Constant b a)) -> a -> b +decode codec a = getConstant (codec Constant a) + + decodeStream :: Monad m => (B.ByteString -> DecodeResult) -> Producer ByteString m r -> Producer Text m (Producer ByteString m r) @@ -95,6 +103,22 @@ p') {-# INLINABLE decodeStream#-} +{- $decoders + These are functions with the simple type: + +> decodeUtf8 :: Monad m => Producer ByteString m r -> Producer Text m (Producer ByteString m r) + + Thus in general + +> decodeUtf8 = view utf8 +> decodeUtf16LE = view utf16LE + + and so forth, but these forms + may be more convenient (and give better type errors!) where lenses are + not desired. +-} + + decodeUtf8 :: Monad m => Producer ByteString m r -> Producer Text m (Producer ByteString m r) decodeUtf8 = decodeStream streamUtf8 {-# INLINE decodeUtf8 #-} @@ -119,6 +143,34 @@ decodeUtf32BE = decodeStream streamUtf32BE {-# INLINE decodeUtf32BE #-} + +{- $encoders + These are simply defined + +> encodeUtf8 = yield . TE.encodeUtf8 + + They are intended for use with 'for' + +> for Text.stdin encodeUtf8 :: Producer ByteString IO () + + which would have the effect of + +> Text.stdin >-> Pipes.Prelude.map (TE.encodeUtf8) + + using the encoding functions from Data.Text.Encoding +-} + +encodeUtf8 :: Monad m => Text -> Producer ByteString m () +encodeUtf8 = yield . TE.encodeUtf8 +encodeUtf16LE :: Monad m => Text -> Producer ByteString m () +encodeUtf16LE = yield . TE.encodeUtf16LE +encodeUtf16BE :: Monad m => Text -> Producer ByteString m () +encodeUtf16BE = yield . TE.encodeUtf16BE +encodeUtf32LE :: Monad m => Text -> Producer ByteString m () +encodeUtf32LE = yield . TE.encodeUtf32LE +encodeUtf32BE :: Monad m => Text -> Producer ByteString m () +encodeUtf32BE = yield . TE.encodeUtf32BE + mkCodec :: (forall r m . Monad m => Producer ByteString m r -> Producer Text m (Producer ByteString m r )) -> (Text -> ByteString) @@ -126,11 +178,57 @@ mkCodec dec enc = \k p0 -> fmap (\p -> join (for p (yield . enc))) (k (dec p0)) -{- | An improper lens into a byte stream expected to be UTF-8 encoded; the associated - text stream ends by returning the original bytestream beginning at the point of failure, - or the empty bytestring for a well-encoded text. - -} +{- $codecs + + Each codec/lens looks into a byte stream that is supposed to contain text. + The particular \'Codec\' lenses are named in accordance with the expected + encoding, 'utf8', 'utf16LE' etc. @view@ / @(^.)@ -- here also called 'decode' -- + turns a Codec into a function: +> view utf8 :: Producer ByteString m r -> Producer Text m (Producer ByteString m r) +> decode utf8 Byte.stdin :: Producer Text IO (Producer ByteString IO r) +> Bytes.stdin ^. utf8 :: Producer Text IO (Producer ByteString IO r) + + Uses of a codec with @view@ / @(^.)@ / 'decode' can always be replaced by the specialized + decoding functions exported here, e.g. + +> decodeUtf8 :: Producer ByteString m r -> Producer Text m (Producer ByteString m r) +> decodeUtf8 Byte.stdin :: Producer Text IO (Producer ByteString IO r) + + The stream of text a @Codec@ \'sees\' in the stream of bytes begins at its head. + At any point of decoding failure, the stream of text ends and reverts to (returns) + the original byte stream. Thus if the first bytes are already + un-decodable, the whole ByteString producer will be returned, i.e. + +> view utf8 bytestream + + will just come to the same as + +> return bytestream + + Where there is no decoding failure, the return value of the text stream will be + an empty byte stream followed by its own return value. In all cases you must + deal with the fact that it is a ByteString producer that is returned, even if + it can be thrown away with @Control.Monad.void@ + +> void (Bytes.stdin ^. utf8) :: Producer Text IO () + + @zoom@ converts a Text parser into a ByteString parser: + +> zoom utf8 drawChar :: Monad m => StateT (Producer ByteString m r) m (Maybe Char) +> +> withNextByte :: Parser ByteString m (Maybe Char, Maybe Word8))) +> withNextByte = do char_ <- zoom utf8 Text.drawChar +> byte_ <- Bytes.peekByte +> return (char_, byte_) + + @withNextByte@ will return the first valid Char in a ByteString, + and the first byte of the next character, if they exists. Because + we \'draw\' one and \'peek\' at the other, the parser as a whole only + advances one Char's length along the bytestring. + + -} + utf8 :: Codec utf8 = mkCodec decodeUtf8 TE.encodeUtf8 @@ -157,7 +255,7 @@ -} --- 'encodeAscii' reduces as much of your stream of 'Text' actually is ascii to a byte stream, +-- | 'encodeAscii' reduces as much of your stream of 'Text' actually is ascii to a byte stream, -- returning the rest of the 'Text' at the first non-ascii 'Char' encodeAscii :: Monad m => Producer Text m r -> Producer ByteString m (Producer Text m r) diff --git a/Pipes/Text/IO.hs b/Pipes/Text/IO.hs --- a/Pipes/Text/IO.hs +++ b/Pipes/Text/IO.hs @@ -1,34 +1,14 @@ {-#LANGUAGE RankNTypes#-} --- | The operations exported here are a convenience, like the similar operations in --- @Data.Text.IO@ , or rather, @Data.Text.Lazy.IO@, since @Producer Text m r@ is --- 'effectful text' and something like the pipes equivalent of lazy Text. --- --- * Like the functions in @Data.Text.IO@, they attempt to work with the system encoding. --- --- * Like the functions in @Data.Text.IO@, they are slower than ByteString operations. Where --- you know what encoding you are working with, use @Pipes.ByteString@ and @Pipes.Text.Encoding@ instead, --- e.g. @view utf8 Bytes.stdin@ instead of @Text.stdin@ --- --- * Like the functions in @Data.Text.IO@ , they use Text exceptions. --- --- Something like --- --- > view utf8 . Bytes.fromHandle :: Handle -> Producer Text IO (Producer ByteString m ()) --- --- yields a stream of Text, and follows --- standard pipes protocols by reverting to (i.e. returning) the underlying byte stream --- upon reaching any decoding error. (See especially the pipes-binary package.) --- --- By contrast, something like --- --- > Text.fromHandle :: Handle -> Producer Text IO () --- --- supplies a stream of text returning '()', which is convenient for many tasks, --- but violates the pipes @pipes-binary@ approach to decoding errors and --- throws an exception of the kind characteristic of the @text@ library instead. + module Pipes.Text.IO ( + -- * Text IO + -- $textio + + -- * Caveats + -- $caveats + -- * Producers fromHandle , stdin @@ -52,6 +32,76 @@ import Pipes.Safe (MonadSafe(..), Base(..)) import Prelude hiding (readFile, writeFile) +{- $textio + Where pipes IO replaces lazy IO, @Producer Text m r@ replaces lazy 'Text'. + This module exports some convenient functions for producing and consuming + pipes 'Text' in IO, with caveats described below. The main points are as in + + + An 'IO.Handle' can be associated with a 'Producer' or 'Consumer' according as it is read or written to. + + To stream to or from 'IO.Handle's, one can use 'fromHandle' or 'toHandle'. For + example, the following program copies a document from one file to another: + +> import Pipes +> import qualified Pipes.Text as Text +> import qualified Pipes.Text.IO as Text +> import System.IO +> +> main = +> withFile "inFile.txt" ReadMode $ \hIn -> +> withFile "outFile.txt" WriteMode $ \hOut -> +> runEffect $ Text.fromHandle hIn >-> Text.toHandle hOut + +To stream from files, the following is perhaps more Prelude-like (note that it uses Pipes.Safe): + +> import Pipes +> import qualified Pipes.Text as Text +> import qualified Pipes.Text.IO as Text +> import Pipes.Safe +> +> main = runSafeT $ runEffect $ Text.readFile "inFile.txt" >-> Text.writeFile "outFile.txt" + + You can stream to and from 'stdin' and 'stdout' using the predefined 'stdin' + and 'stdout' pipes, as with the following \"echo\" program: + +> main = runEffect $ Text.stdin >-> Text.stdout + +-} + + +{- $caveats + + The operations exported here are a convenience, like the similar operations in + @Data.Text.IO@ (or rather, @Data.Text.Lazy.IO@, since, again, @Producer Text m r@ is + 'effectful text' and something like the pipes equivalent of lazy Text.) + + * Like the functions in @Data.Text.IO@, they attempt to work with the system encoding. + + * Like the functions in @Data.Text.IO@, they are slower than ByteString operations. Where + you know what encoding you are working with, use @Pipes.ByteString@ and @Pipes.Text.Encoding@ instead, + e.g. @view utf8 Bytes.stdin@ instead of @Text.stdin@ + + * Like the functions in @Data.Text.IO@ , they use Text exceptions. + + Something like + +> view utf8 . Bytes.fromHandle :: Handle -> Producer Text IO (Producer ByteString m ()) + + yields a stream of Text, and follows + standard pipes protocols by reverting to (i.e. returning) the underlying byte stream + upon reaching any decoding error. (See especially the pipes-binary package.) + + By contrast, something like + +> Text.fromHandle :: Handle -> Producer Text IO () + + supplies a stream of text returning '()', which is convenient for many tasks, + but violates the pipes @pipes-binary@ approach to decoding errors and + throws an exception of the kind characteristic of the @text@ library instead. + + +-} {-| Convert a 'IO.Handle' into a text stream using a text size determined by the good sense of the text library. Note with the remarks diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -1,6 +1,10 @@ pipes-text ========== -This package follows the rule `pipes-text : pipes-bytestring :: text : bytestring` It has three modules, `Pipes.Text` , `Pipes.Text.Encoding` and `Pipes.Text.IO`; the division has more or less the significance it has in the `text` library. +This package follows the rule: + + pipes-text : pipes-bytestring :: text : bytestring + +The division of three modules, `Pipes.Text` , `Pipes.Text.Encoding` and `Pipes.Text.IO` has more or less the significance it has in the `text` library. Note that the module `Pipes.Text.IO` uses version 0.11.3 or later of the `text` library. (It thus works with the version of `text` that came with the 2013 Haskell Platform. To use an older `text`, install with the flag `-fnoio` diff --git a/pipes-text.cabal b/pipes-text.cabal --- a/pipes-text.cabal +++ b/pipes-text.cabal @@ -1,7 +1,7 @@ name: pipes-text -version: 0.0.0.7 +version: 0.0.0.8 synopsis: Text pipes. -description: * This package will be in a draft, or testing, phase until version 0.0.1. Please report any installation difficulties, or any wisdom about the api, on the github page! +description: * This package will be in a draft, or testing, phase until version 0.0.1. Please report any installation difficulties, or any wisdom about the api, on the github page or the . This organization of the package follows the rule .