pipes-text 0.0.2.5 → 1.0.0
raw patch · 10 files changed
+1612/−1564 lines, 10 filesdep ~bytestringdep ~pipes-groupdep ~pipes-parsesetup-changednew-uploader
Dependency ranges changed: bytestring, pipes-group, pipes-parse, pipes-safe, streaming-commons
Files
- CHANGES.md +55/−0
- Pipes/Prelude/Text.hs +138/−147
- Pipes/Text.hs +504/−496
- Pipes/Text/Encoding.hs +399/−370
- Pipes/Text/IO.hs +126/−139
- Pipes/Text/Tutorial.hs +291/−308
- README.md +6/−5
- Setup.hs +1/−0
- changelog +0/−49
- pipes-text.cabal +92/−50
+ CHANGES.md view
@@ -0,0 +1,55 @@+# Version 1.0.0.0++* New maintainer: Peter Jones++* Widen version bounds for dependencies.++* The `noio` flag is deprecated and will be removed in the next major release.++# Version 0.0.0.12++* Opposing lenses for `lines` and `unlines` and `words` and `unwords`.+ Brought closer in line with `pipes-bytestring` again. Removed `count`, which+ was wrong. Scrapped `Iso` and the `profunctors` dependency.++# Version 0.0.0.11++* Updated to use streaming-commons in place of text-stream-decoding.++# Version 0.0.0.10++* Documentation changes.+++# Version 0.0.0.9++* Documentation changes.++# Version 0.0.0.7++ * Used the new text-stream-decoding package+ * Separated IO and Encoding modules adding flag -fnoio++ # Version 0.0.0.5++ * Rearranged internal modules+++ # Version 0.0.0.4++ * Altered bad haddock markup+++ # Version 0.0.0.3++ * Actually added changelog+++ # Version 0.0.0.2++ * Omit `stdinLn` as likely to be dangerous through misunderstanding.+++ # Version 0.0.0.1++ * Rearrange order of 'Internal' materials.
Pipes/Prelude/Text.hs view
@@ -1,159 +1,156 @@-{-#LANGUAGE RankNTypes#-}-+{-# LANGUAGE RankNTypes #-} module Pipes.Prelude.Text- ( - -- * Simple line-based Text IO- -- $lineio- - fromHandleLn- , toHandleLn- , stdinLn- , stdoutLn- , stdoutLn'- , readFileLn- , writeFileLn- ) where+ ( -- * Simple line-based Text IO+ -- $lineio+ fromHandleLn,+ toHandleLn,+ stdinLn,+ stdoutLn,+ stdoutLn',+ readFileLn,+ writeFileLn,+ )+where -import qualified System.IO as IO import Control.Exception (throwIO, try)-import Foreign.C.Error (Errno(Errno), ePIPE)-import qualified GHC.IO.Exception as G import Data.Text (Text) import qualified Data.Text as T import qualified Data.Text.IO as T+import Foreign.C.Error (Errno (Errno), ePIPE)+import qualified GHC.IO.Exception as G import Pipes+import Pipes.Safe (MonadSafe (..)) import qualified Pipes.Safe.Prelude as Safe-import Pipes.Safe (MonadSafe(..), runSafeT, runSafeP)+import qualified System.IO as IO import Prelude hiding (readFile, writeFile) -{- $lineio- Line-based operations are marked with a final \-@Ln@, like 'stdinLn', 'readFileLn', etc. - They are drop-in 'Text' replacements for the corresponding 'String' operations in - @Pipes.Prelude@ and @Pipes.Safe.Prelude@ - a final \-@Ln@ being added where necessary. - This module can thus be imported unqualified if @Pipes.Prelude@ is imported qualified, as- it must be.-- In using the line-based operations, one is producing and consuming semantically significant individual texts, - understood as lines, just as one would produce or pipe 'Int's or 'Char's or anything else.- The standard materials from @Pipes@ and @Pipes.Prelude@ and- @Data.Text@ are all you need to work with them, and- you can use these operations without using any of the other modules in this package. -- Thus, to take a trivial case, here we upper-case three lines from standard input and write - them to a file. (@runSafeT@ from @Pipes.Safe@ just makes sure to close any handles opened in its scope; - it is only needed for @readFileLn@ and @writeFileLn@.)-->>> import Pipes->>> import qualified Pipes.Prelude as P->>> import qualified Pipes.Prelude.Text as Text->>> import qualified Data.Text as T->>> Text.runSafeT $ runEffect $ Text.stdinLn >-> P.take 3 >-> P.map T.toUpper >-> Text.writeFileLn "threelines.txt"-one<Enter>-two<Enter>-three<Enter>->>> :! cat "threelines.txt"-ONE-TWO-THREE-- The point of view is very much that of @Pipes.Prelude@, substituting @Text@ for @String@. - It would still be the same even if- we did something a bit more sophisticated, like run an ordinary attoparsec 'Text' parser on- each line, as is frequently desirable. Here we use- a minimal attoparsec number parser, @scientific@, on separate lines of standard input, - dropping bad parses with @P.concat@:-->>> import Data.Attoparsec.Text (parseOnly, scientific)->>> P.toListM $ Text.stdinLn >-> P.takeWhile (/= "quit") >-> P.map (parseOnly scientific) >-> P.concat -1<Enter>-2<Enter>-bad<Enter>-3<Enter>-quit<Enter>-[1.0,2.0,3.0]-- The line-based operations are, however, subject to a number of caveats.- - * Where these line-based operations read from a handle, they will - accumulate indefinitely long lines. This makes sense for input - typed in by a user, and for locally produced files of known characteristics, but- otherwise not. See the post on- <http://www.haskellforall.com/2013/09/perfect-streaming-using-pipes-bytestring.html perfect streaming> - to see why @pipes-bytestring@ and this package, outside this module, take a different approach, in which- lines themselves are permitted to stream without accumulation. - - * The line-based operations, - like those in @Data.Text.IO@, use the system encoding (and @T.hGetLine@, @T.hPutLine@ etc.)- and thus are slower than the \'official\' route, which would use the very fast - bytestring IO operations from @Pipes.ByteString@ and the- encoding and decoding functions in @Pipes.Text.Encoding@, which are also quite fast- thanks to the @streaming-commons@ package.-- * The line-based operations (again like those in @Data.Text.IO@) will - generate text exceptions after the fashion of - @Data.Text.Encoding@, rather than returning the undigested bytes in the - style of @Pipes.Text.Encoding@. This is the standard practice in the pipes libraries.---}---{-| Read separate lines of 'Text' from 'IO.stdin' using 'T.getLine', terminating on end of input.+-- $lineio+-- Line-based operations are marked with a final \-@Ln@, like 'stdinLn', 'readFileLn', etc.+-- They are drop-in 'Text' replacements for the corresponding 'String' operations in+-- @Pipes.Prelude@ and @Pipes.Safe.Prelude@ - a final \-@Ln@ being added where necessary.+-- This module can thus be imported unqualified if @Pipes.Prelude@ is imported qualified, as+-- it must be.+--+-- In using the line-based operations, one is producing and consuming semantically significant individual texts,+-- understood as lines, just as one would produce or pipe 'Int's or 'Char's or anything else.+-- The standard materials from @Pipes@ and @Pipes.Prelude@ and+-- @Data.Text@ are all you need to work with them, and+-- you can use these operations without using any of the other modules in this package.+--+-- Thus, to take a trivial case, here we upper-case three lines from standard input and write+-- them to a file. (@runSafeT@ from @Pipes.Safe@ just makes sure to close any handles opened in its scope;+-- it is only needed for @readFileLn@ and @writeFileLn@.)+--+-- >>> import Pipes+-- >>> import qualified Pipes.Prelude as P+-- >>> import qualified Pipes.Prelude.Text as Text+-- >>> import qualified Data.Text as T+-- >>> Text.runSafeT $ runEffect $ Text.stdinLn >-> P.take 3 >-> P.map T.toUpper >-> Text.writeFileLn "threelines.txt"+-- one<Enter>+-- two<Enter>+-- three<Enter>+-- >>> :! cat "threelines.txt"+-- ONE+-- TWO+-- THREE+--+-- The point of view is very much that of @Pipes.Prelude@, substituting @Text@ for @String@.+-- It would still be the same even if+-- we did something a bit more sophisticated, like run an ordinary attoparsec 'Text' parser on+-- each line, as is frequently desirable. Here we use+-- a minimal attoparsec number parser, @scientific@, on separate lines of standard input,+-- dropping bad parses with @P.concat@:+--+-- >>> import Data.Attoparsec.Text (parseOnly, scientific)+-- >>> P.toListM $ Text.stdinLn >-> P.takeWhile (/= "quit") >-> P.map (parseOnly scientific) >-> P.concat+-- 1<Enter>+-- 2<Enter>+-- bad<Enter>+-- 3<Enter>+-- quit<Enter>+-- [1.0,2.0,3.0]+--+-- The line-based operations are, however, subject to a number of caveats.+--+-- * Where these line-based operations read from a handle, they will+-- accumulate indefinitely long lines. This makes sense for input+-- typed in by a user, and for locally produced files of known characteristics, but+-- otherwise not. See the post on+-- <http://www.haskellforall.com/2013/09/perfect-streaming-using-pipes-bytestring.html perfect streaming>+-- to see why @pipes-bytestring@ and this package, outside this module, take a different approach, in which+-- lines themselves are permitted to stream without accumulation.+--+-- * The line-based operations,+-- like those in @Data.Text.IO@, use the system encoding (and @T.hGetLine@, @T.hPutLine@ etc.)+-- and thus are slower than the \'official\' route, which would use the very fast+-- bytestring IO operations from @Pipes.ByteString@ and the+-- encoding and decoding functions in @Pipes.Text.Encoding@, which are also quite fast+-- thanks to the @streaming-commons@ package.+--+-- * The line-based operations (again like those in @Data.Text.IO@) will+-- generate text exceptions after the fashion of+-- @Data.Text.Encoding@, rather than returning the undigested bytes in the+-- style of @Pipes.Text.Encoding@. This is the standard practice in the pipes libraries. - This function will accumulate indefinitely long strict 'Text's. See the caveats above.--}+-- | Read separate lines of 'Text' from 'IO.stdin' using 'T.getLine', terminating on end of input.+--+-- This function will accumulate indefinitely long strict 'Text's. See the caveats above. stdinLn :: MonadIO m => Producer' T.Text m () stdinLn = fromHandleLn IO.stdin-{-# INLINABLE stdinLn #-}-+{-# INLINEABLE stdinLn #-} -{-| Write 'Text' lines to 'IO.stdout' using 'putStrLn', terminating without error on a broken output pipe--}+-- | Write 'Text' lines to 'IO.stdout' using 'putStrLn', terminating without error on a broken output pipe stdoutLn :: MonadIO m => Consumer' T.Text m () stdoutLn = go where go = do- str <- await- x <- liftIO $ try (T.putStrLn str)- case x of- Left (G.IOError { G.ioe_type = G.ResourceVanished- , G.ioe_errno = Just ioe })- | Errno ioe == ePIPE- -> return ()- Left e -> liftIO (throwIO e)- Right () -> go-{-# INLINABLE stdoutLn #-}+ str <- await+ x <- liftIO $ try (T.putStrLn str)+ case x of+ Left+ G.IOError+ { G.ioe_type = G.ResourceVanished,+ G.ioe_errno = Just ioe+ }+ | Errno ioe == ePIPE ->+ return ()+ Left e -> liftIO (throwIO e)+ Right () -> go+{-# INLINEABLE stdoutLn #-} -{-| Write lines of 'Text' to 'IO.stdout'. This does not handle a broken output pipe, - but has a polymorphic return value.--}+-- | Write lines of 'Text' to 'IO.stdout'. This does not handle a broken output pipe,+-- but has a polymorphic return value. stdoutLn' :: MonadIO m => Consumer' T.Text m r-stdoutLn' = for cat (\str -> liftIO (T.putStrLn str))-{-# INLINABLE stdoutLn' #-}+stdoutLn' = for cat (liftIO . T.putStrLn)+{-# INLINE [1] stdoutLn' #-} {-# RULES- "p >-> stdoutLn'" forall p .- p >-> stdoutLn' = for p (\str -> liftIO (T.putStrLn str))+"p >-> stdoutLn'" forall p.+ p >-> stdoutLn' =+ for p (liftIO . T.putStrLn) #-} -{-| Read separate lines of 'Text' from a 'IO.Handle' using 'T.hGetLine', - terminating at the end of input-- This operation will accumulate indefinitely large strict texts. See the caveats above.--}+-- | Read separate lines of 'Text' from a 'IO.Handle' using 'T.hGetLine',+-- terminating at the end of input+--+-- This operation will accumulate indefinitely large strict texts. See the caveats above. fromHandleLn :: MonadIO m => IO.Handle -> Producer' Text m ()-fromHandleLn h = go where- getLine :: IO (Either G.IOException Text)- getLine = try (T.hGetLine h)+fromHandleLn h = go+ where+ getLine :: IO (Either G.IOException Text)+ getLine = try (T.hGetLine h) - go = do txt <- liftIO getLine- case txt of- Left e -> return ()- Right y -> do yield y- go-{-# INLINABLE fromHandleLn #-}+ go = do+ txt <- liftIO getLine+ case txt of+ Left _ -> return ()+ Right y -> do+ yield y+ go+{-# INLINEABLE fromHandleLn #-} --- to do: investigate differences from the above: +-- to do: investigate differences from the above: -- fromHandleLn :: MonadIO m => IO.Handle -> Producer' T.Text m () -- fromHandleLn h = go -- where@@ -165,34 +162,28 @@ -- go -- {-# INLINABLE fromHandleLn #-} - -- | Write separate lines of 'Text' to a 'IO.Handle' using 'T.hPutStrLn' toHandleLn :: MonadIO m => IO.Handle -> Consumer' T.Text m r-toHandleLn handle = for cat (\str -> liftIO (T.hPutStrLn handle str))-{-# INLINABLE toHandleLn #-}+toHandleLn handle = for cat (liftIO . T.hPutStrLn handle)+{-# INLINE [1] toHandleLn #-} {-# RULES- "p >-> toHandleLn handle" forall p handle .- p >-> toHandleLn handle = for p (\str -> liftIO (T.hPutStrLn handle str))+"p >-> toHandleLn handle" forall p handle.+ p >-> toHandleLn handle =+ for p (liftIO . T.hPutStrLn handle) #-} --{-| Stream separate lines of text from a file. Apply @runSafeT@ after running the- pipeline to manage the opening and closing of the handle.- - This operation will accumulate indefinitely long strict text chunks. - See the caveats above.--}+-- | Stream separate lines of text from a file. Apply @runSafeT@ after running the+-- pipeline to manage the opening and closing of the handle.+--+-- This operation will accumulate indefinitely long strict text chunks.+-- See the caveats above. readFileLn :: MonadSafe m => FilePath -> Producer Text m () readFileLn file = Safe.withFile file IO.ReadMode fromHandleLn {-# INLINE readFileLn #-} ---{-| Write lines to a file. Apply @runSafeT@ after running the- pipeline to manage the opening and closing of the handle.--}+-- | Write lines to a file. Apply @runSafeT@ after running the+-- pipeline to manage the opening and closing of the handle. writeFileLn :: (MonadSafe m) => FilePath -> Consumer' Text m r writeFileLn file = Safe.withFile file IO.WriteMode toHandleLn-{-# INLINABLE writeFileLn #-}-+{-# INLINEABLE writeFileLn #-}
Pipes/Text.hs view
@@ -1,109 +1,104 @@-{-# LANGUAGE RankNTypes, TypeFamilies, BangPatterns#-}--{-| The module @Pipes.Text@ closely follows @Pipes.ByteString@ from - the @pipes-bytestring@ package. A draft tutorial can be found in- @Pipes.Text.Tutorial@. --}+{-# LANGUAGE RankNTypes #-}+{-# LANGUAGE TypeFamilies #-} -module Pipes.Text (- -- * Producers- fromLazy+-- | The module @Pipes.Text@ closely follows @Pipes.ByteString@ from+-- the @pipes-bytestring@ package. A draft tutorial can be found in+-- @Pipes.Text.Tutorial@.+module Pipes.Text+ ( -- * Producers+ fromLazy, -- * Pipes- , map- , concatMap- , take- , takeWhile- , filter- , toCaseFold- , toLower- , toUpper- , stripStart- , scan+ map,+ concatMap,+ take,+ takeWhile,+ filter,+ toCaseFold,+ toLower,+ toUpper,+ stripStart,+ scan, -- * Folds- , toLazy- , toLazyM- , foldChars- , head- , last- , null- , length- , any- , all- , maximum- , minimum- , find- , index+ toLazy,+ toLazyM,+ foldChars,+ head,+ last,+ null,+ length,+ any,+ all,+ maximum,+ minimum,+ find,+ index, -- * Primitive Character Parsers- , nextChar- , drawChar- , unDrawChar- , peekChar- , isEndOfChars+ nextChar,+ drawChar,+ unDrawChar,+ peekChar,+ isEndOfChars, -- * Parsing Lenses- , splitAt- , span- , break- , groupBy- , group- , word- , line+ splitAt,+ span,+ break,+ groupBy,+ group,+ word,+ line, -- * Transforming Text and Character Streams- , drop- , dropWhile- , pack- , unpack- , intersperse+ drop,+ dropWhile,+ pack,+ unpack,+ intersperse, -- * FreeT Transformations- , chunksOf- , splitsWith- , splits- , groupsBy- , groups- , lines- , unlines- , words- , unwords- , intercalate+ chunksOf,+ splitsWith,+ splits,+ groupsBy,+ groups,+ lines,+ unlines,+ words,+ unwords,+ intercalate, -- * Re-exports -- $reexports- , module Data.ByteString- , module Data.Text- , module Pipes.Parse- , module Pipes.Group- ) where+ module Data.ByteString,+ module Data.Text,+ module Pipes.Parse,+ module Pipes.Group,+ )+where -import Control.Applicative ((<*))-import Control.Monad (liftM, join)-import Data.Functor.Constant (Constant(..))-import Data.Functor.Identity (Identity)+import Control.Monad (join) import Control.Monad.Trans.State.Strict (modify)--import qualified Data.Text as T-import Data.Text (Text)-import qualified Data.Text.Lazy as TL+import Data.Bits (shiftL) import Data.ByteString (ByteString) import Data.Char (isSpace)+import Data.Foldable (traverse_)+import Data.Functor.Constant (Constant (..))+import Data.Functor.Identity (Identity)+import Data.Text (Text)+import qualified Data.Text as T+import qualified Data.Text.Lazy as TL import Foreign.Storable (sizeOf)-import Data.Bits (shiftL)- import Pipes-import Pipes.Group (folds, maps, concats, intercalates, FreeT(..), FreeF(..))+import Pipes.Group (FreeF (..), FreeT (..), concats, folds, intercalates, maps) import qualified Pipes.Group as PG-import qualified Pipes.Parse as PP import Pipes.Parse (Parser)+import qualified Pipes.Parse as PP import qualified Pipes.Prelude as P----import Prelude hiding (- all,+import Prelude hiding+ ( all, any, break, concat,@@ -114,8 +109,8 @@ filter, head, last,- lines, length,+ lines, map, maximum, minimum,@@ -129,7 +124,8 @@ unlines, unwords, words,- writeFile )+ writeFile,+ ) -- $setup -- >>> :set -XOverloadedStrings@@ -138,11 +134,11 @@ -- >>> import qualified Data.Text.Lazy.IO as TL -- >>> import Data.Char --- | Convert a lazy 'TL.Text' into a 'Producer' of strict 'Text's. Producers in +-- | Convert a lazy 'TL.Text' into a 'Producer' of strict 'Text's. Producers in -- IO can be found in 'Pipes.Text.IO' or in pipes-bytestring, employed with the -- decoding lenses in 'Pipes.Text.Encoding' fromLazy :: (Monad m) => TL.Text -> Producer' Text m ()-fromLazy = TL.foldrChunks (\e a -> yield e >> a) (return ())+fromLazy = TL.foldrChunks (\e a -> yield e >> a) (return ()) {-# INLINE fromLazy #-} (^.) :: a -> ((b -> Constant b b) -> (a -> Constant b a)) -> b@@ -156,70 +152,71 @@ -- OVER GOLDENGROVE UNLEAVING? map :: (Monad m) => (Char -> Char) -> Pipe Text Text m r map f = P.map (T.map f)-{-# INLINABLE map #-}+{-# INLINEABLE map #-} -- | Map a function over the characters of a text stream and concatenate the results--concatMap- :: (Monad m) => (Char -> Text) -> Pipe Text Text m r+concatMap ::+ (Monad m) => (Char -> Text) -> Pipe Text Text m r concatMap f = P.map (T.concatMap f)-{-# INLINABLE concatMap #-}+{-# INLINEABLE concatMap #-} -- | @(take n)@ only allows @n@ individual characters to pass; -- contrast @Pipes.Prelude.take@ which would let @n@ chunks pass. take :: (Monad m, Integral a) => a -> Pipe Text Text m ()-take n0 = go n0 where+take = go+ where go n- | n <= 0 = return ()- | otherwise = do - txt <- await- let len = fromIntegral (T.length txt)- if (len > n)- then yield (T.take (fromIntegral n) txt)- else do- yield txt- go (n - len)-{-# INLINABLE take #-}+ | n <= 0 = return ()+ | otherwise = do+ txt <- await+ let len = fromIntegral (T.length txt)+ if len > n+ then yield (T.take (fromIntegral n) txt)+ else do+ yield txt+ go (n - len)+{-# INLINEABLE take #-} -- | Take characters until they fail the predicate takeWhile :: (Monad m) => (Char -> Bool) -> Pipe Text Text m () takeWhile predicate = go where go = do- txt <- await- let (prefix, suffix) = T.span predicate txt- if (T.null suffix)- then do- yield txt- go- else yield prefix-{-# INLINABLE takeWhile #-}+ txt <- await+ let (prefix, suffix) = T.span predicate txt+ if T.null suffix+ then do+ yield txt+ go+ else yield prefix+{-# INLINEABLE takeWhile #-} -- | Only allows 'Char's to pass if they satisfy the predicate filter :: (Monad m) => (Char -> Bool) -> Pipe Text Text m r filter predicate = P.map (T.filter predicate)-{-# INLINABLE filter #-}+{-# INLINEABLE filter #-} -- | Strict left scan over the characters -- >>> let margaret = ["Margaret, are you grieving\nOver Golde","ngrove unleaving?":: Text] -- >>> let title_caser a x = case a of ' ' -> Data.Char.toUpper x; _ -> x--- >>> toLazy $ each margaret >-> scan title_caser ' ' +-- >>> toLazy $ each margaret >-> scan title_caser ' ' -- " Margaret, Are You Grieving\nOver Goldengrove Unleaving?"--scan- :: (Monad m)- => (Char -> Char -> Char) -> Char -> Pipe Text Text m r+scan ::+ (Monad m) =>+ (Char -> Char -> Char) ->+ Char ->+ Pipe Text Text m r scan step begin = do- yield (T.singleton begin)- go begin+ yield (T.singleton begin)+ go begin where go c = do- txt <- await- let txt' = T.scanl step c txt- c' = T.last txt'- yield (T.tail txt')- go c'-{-# INLINABLE scan #-}+ txt <- await+ let txt' = T.scanl step c txt+ c' = T.last txt'+ yield (T.tail txt')+ go c'+{-# INLINEABLE scan #-} -- | @toCaseFold@, @toLower@, @toUpper@ and @stripStart@ are standard 'Text' utilities, -- here acting as 'Text' pipes, rather as they would on a lazy text@@ -240,557 +237,568 @@ -- | Remove leading white space from an incoming succession of 'Text's stripStart :: Monad m => Pipe Text Text m r stripStart = do- chunk <- await- let text = T.stripStart chunk- if T.null text- then stripStart- else do yield text- cat+ chunk <- await+ let text = T.stripStart chunk+ if T.null text+ then stripStart+ else do+ yield text+ cat {-# INLINEABLE stripStart #-} -{-| Fold a pure 'Producer' of strict 'Text's into a lazy- 'TL.Text'--}+-- | Fold a pure 'Producer' of strict 'Text's into a lazy+-- 'TL.Text' toLazy :: Producer Text Identity () -> TL.Text toLazy = TL.fromChunks . P.toList-{-# INLINABLE toLazy #-}--{-| Fold an effectful 'Producer' of strict 'Text's into a lazy- 'TL.Text'+{-# INLINEABLE toLazy #-} - Note: 'toLazyM' is not an idiomatic use of @pipes@, but I provide it for- simple testing purposes. Idiomatic @pipes@ style consumes the chunks- immediately as they are generated instead of loading them all into memory.--}+-- | Fold an effectful 'Producer' of strict 'Text's into a lazy+-- 'TL.Text'+--+-- Note: 'toLazyM' is not an idiomatic use of @pipes@, but I provide it for+-- simple testing purposes. Idiomatic @pipes@ style consumes the chunks+-- immediately as they are generated instead of loading them all into memory. toLazyM :: (Monad m) => Producer Text m () -> m TL.Text-toLazyM = liftM TL.fromChunks . P.toListM-{-# INLINABLE toLazyM #-}+toLazyM = fmap TL.fromChunks . P.toListM+{-# INLINEABLE toLazyM #-} -- | Reduce the text stream using a strict left fold over characters-foldChars- :: Monad m- => (x -> Char -> x) -> x -> (x -> r) -> Producer Text m () -> m r-foldChars step begin done = P.fold (T.foldl' step) begin done-{-# INLINABLE foldChars #-}-+foldChars ::+ Monad m =>+ (x -> Char -> x) ->+ x ->+ (x -> r) ->+ Producer Text m () ->+ m r+foldChars step = P.fold (T.foldl' step)+{-# INLINEABLE foldChars #-} -- | Retrieve the first 'Char' head :: (Monad m) => Producer Text m () -> m (Maybe Char) head = go where go p = do- x <- nextChar p- case x of- Left _ -> return Nothing- Right (c, _) -> return (Just c)-{-# INLINABLE head #-}+ x <- nextChar p+ case x of+ Left _ -> return Nothing+ Right (c, _) -> return (Just c)+{-# INLINEABLE head #-} -- | Retrieve the last 'Char' last :: (Monad m) => Producer Text m () -> m (Maybe Char) last = go Nothing where go r p = do- x <- next p- case x of- Left () -> return r- Right (txt, p') ->- if (T.null txt)- then go r p'- else go (Just $ T.last txt) p'-{-# INLINABLE last #-}+ x <- next p+ case x of+ Left () -> return r+ Right (txt, p') ->+ if T.null txt+ then go r p'+ else go (Just $ T.last txt) p'+{-# INLINEABLE last #-} -- | Determine if the stream is empty null :: (Monad m) => Producer Text m () -> m Bool null = P.all T.null-{-# INLINABLE null #-}+{-# INLINEABLE null #-} -- | Count the number of characters in the stream length :: (Monad m, Num n) => Producer Text m () -> m n length = P.fold (\n txt -> n + fromIntegral (T.length txt)) 0 id-{-# INLINABLE length #-}+{-# INLINEABLE length #-} -- | Fold that returns whether 'M.Any' received 'Char's satisfy the predicate any :: (Monad m) => (Char -> Bool) -> Producer Text m () -> m Bool any predicate = P.any (T.any predicate)-{-# INLINABLE any #-}+{-# INLINEABLE any #-} -- | Fold that returns whether 'M.All' received 'Char's satisfy the predicate all :: (Monad m) => (Char -> Bool) -> Producer Text m () -> m Bool all predicate = P.all (T.all predicate)-{-# INLINABLE all #-}+{-# INLINEABLE all #-} -- | Return the maximum 'Char' within a text stream maximum :: (Monad m) => Producer Text m () -> m (Maybe Char) maximum = P.fold step Nothing id where step mc txt =- if (T.null txt)+ if T.null txt then mc else Just $ case mc of- Nothing -> T.maximum txt- Just c -> max c (T.maximum txt)-{-# INLINABLE maximum #-}+ Nothing -> T.maximum txt+ Just c -> max c (T.maximum txt)+{-# INLINEABLE maximum #-} -- | Return the minimum 'Char' within a text stream (surely very useful!) minimum :: (Monad m) => Producer Text m () -> m (Maybe Char) minimum = P.fold step Nothing id where step mc txt =- if (T.null txt)+ if T.null txt then mc else case mc of- Nothing -> Just (T.minimum txt)- Just c -> Just (min c (T.minimum txt))-{-# INLINABLE minimum #-}+ Nothing -> Just (T.minimum txt)+ Just c -> Just (min c (T.minimum txt))+{-# INLINEABLE minimum #-} -- | Find the first element in the stream that matches the predicate-find- :: (Monad m)- => (Char -> Bool) -> Producer Text m () -> m (Maybe Char)+find ::+ (Monad m) =>+ (Char -> Bool) ->+ Producer Text m () ->+ m (Maybe Char) find predicate p = head (p >-> filter predicate)-{-# INLINABLE find #-}+{-# INLINEABLE find #-} -- | Index into a text stream-index- :: (Monad m, Integral a)- => a-> Producer Text m () -> m (Maybe Char)+index ::+ (Monad m, Integral a) =>+ a ->+ Producer Text m () ->+ m (Maybe Char) index n p = head (drop n p)-{-# INLINABLE index #-}--+{-# INLINEABLE index #-} -- | Consume the first character from a stream of 'Text' -- -- 'next' either fails with a 'Left' if the 'Producer' has no more characters or -- succeeds with a 'Right' providing the next character and the remainder of the -- 'Producer'.--nextChar- :: (Monad m)- => Producer Text m r- -> m (Either r (Char, Producer Text m r))+nextChar ::+ (Monad m) =>+ Producer Text m r ->+ m (Either r (Char, Producer Text m r)) nextChar = go where go p = do- x <- next p- case x of- Left r -> return (Left r)- Right (txt, p') -> case (T.uncons txt) of- Nothing -> go p'- Just (c, txt') -> return (Right (c, yield txt' >> p'))-{-# INLINABLE nextChar #-}+ x <- next p+ case x of+ Left r -> return (Left r)+ Right (txt, p') -> case T.uncons txt of+ Nothing -> go p'+ Just (c, txt') -> return (Right (c, yield txt' >> p'))+{-# INLINEABLE nextChar #-} -- | Draw one 'Char' from a stream of 'Text', returning 'Left' if the 'Producer' is empty- drawChar :: (Monad m) => Parser Text m (Maybe Char) drawChar = do- x <- PP.draw- case x of- Nothing -> return Nothing- Just txt -> case (T.uncons txt) of- Nothing -> drawChar- Just (c, txt') -> do- PP.unDraw txt'- return (Just c)-{-# INLINABLE drawChar #-}+ x <- PP.draw+ case x of+ Nothing -> return Nothing+ Just txt -> case T.uncons txt of+ Nothing -> drawChar+ Just (c, txt') -> do+ PP.unDraw txt'+ return (Just c)+{-# INLINEABLE drawChar #-} -- | Push back a 'Char' onto the underlying 'Producer' unDrawChar :: (Monad m) => Char -> Parser Text m () unDrawChar c = modify (yield (T.singleton c) >>)-{-# INLINABLE unDrawChar #-}--{-| 'peekChar' checks the first 'Char' in the stream, but uses 'unDrawChar' to- push the 'Char' back--> peekChar = do-> x <- drawChar-> case x of-> Left _ -> return ()-> Right c -> unDrawChar c-> return x---}+{-# INLINEABLE unDrawChar #-} +-- | 'peekChar' checks the first 'Char' in the stream, but uses 'unDrawChar' to+-- push the 'Char' back+--+-- > peekChar = do+-- > x <- drawChar+-- > case x of+-- > Left _ -> return ()+-- > Right c -> unDrawChar c+-- > return x peekChar :: (Monad m) => Parser Text m (Maybe Char) peekChar = do- x <- drawChar- case x of- Nothing -> return ()- Just c -> unDrawChar c- return x-{-# INLINABLE peekChar #-}--{-| Check if the underlying 'Producer' has no more characters-- Note that this will skip over empty 'Text' chunks, unlike- 'PP.isEndOfInput' from @pipes-parse@, which would consider- an empty 'Text' a valid bit of input.+ x <- drawChar+ traverse_ unDrawChar x+ return x+{-# INLINEABLE peekChar #-} -> isEndOfChars = liftM isLeft peekChar--}+-- | Check if the underlying 'Producer' has no more characters+--+-- Note that this will skip over empty 'Text' chunks, unlike+-- 'PP.isEndOfInput' from @pipes-parse@, which would consider+-- an empty 'Text' a valid bit of input.+--+-- > isEndOfChars = liftM isLeft peekChar isEndOfChars :: (Monad m) => Parser Text m Bool isEndOfChars = do- x <- peekChar- return (case x of+ x <- peekChar+ return+ ( case x of Nothing -> True- Just _-> False )-{-# INLINABLE isEndOfChars #-}+ Just _ -> False+ )+{-# INLINEABLE isEndOfChars #-} -- | Splits a 'Producer' after the given number of characters-splitAt- :: (Monad m, Integral n)- => n- -> Lens' (Producer Text m r)- (Producer Text m (Producer Text m r))+splitAt ::+ (Monad m, Integral n) =>+ n ->+ Lens'+ (Producer Text m r)+ (Producer Text m (Producer Text m r)) splitAt n0 k p0 = fmap join (k (go n0 p0)) where go 0 p = return p go n p = do- x <- lift (next p)- case x of- Left r -> return (return r)- Right (txt, p') -> do- let len = fromIntegral (T.length txt)- if (len <= n)- then do- yield txt- go (n - len) p'- else do- let (prefix, suffix) = T.splitAt (fromIntegral n) txt- yield prefix- return (yield suffix >> p')-{-# INLINABLE splitAt #-}-+ x <- lift (next p)+ case x of+ Left r -> return (return r)+ Right (txt, p') -> do+ let len = fromIntegral (T.length txt)+ if len <= n+ then do+ yield txt+ go (n - len) p'+ else do+ let (prefix, suffix) = T.splitAt (fromIntegral n) txt+ yield prefix+ return (yield suffix >> p')+{-# INLINEABLE splitAt #-} -- | Split a text stream in two, producing the longest -- consecutive group of characters that satisfies the predicate -- and returning the rest--span- :: (Monad m)- => (Char -> Bool)- -> Lens' (Producer Text m r)- (Producer Text m (Producer Text m r))+span ::+ (Monad m) =>+ (Char -> Bool) ->+ Lens'+ (Producer Text m r)+ (Producer Text m (Producer Text m r)) span predicate k p0 = fmap join (k (go p0)) where go p = do- x <- lift (next p)- case x of- Left r -> return (return r)- Right (txt, p') -> do- let (prefix, suffix) = T.span predicate txt- if (T.null suffix)- then do- yield txt- go p'- else do- yield prefix- return (yield suffix >> p')-{-# INLINABLE span #-}+ x <- lift (next p)+ case x of+ Left r -> return (return r)+ Right (txt, p') -> do+ let (prefix, suffix) = T.span predicate txt+ if T.null suffix+ then do+ yield txt+ go p'+ else do+ yield prefix+ return (yield suffix >> p')+{-# INLINEABLE span #-} -{-| Split a text stream in two, producing the longest- consecutive group of characters that don't satisfy the predicate--}-break- :: (Monad m)- => (Char -> Bool)- -> Lens' (Producer Text m r)- (Producer Text m (Producer Text m r))+-- | Split a text stream in two, producing the longest+-- consecutive group of characters that don't satisfy the predicate+break ::+ (Monad m) =>+ (Char -> Bool) ->+ Lens'+ (Producer Text m r)+ (Producer Text m (Producer Text m r)) break predicate = span (not . predicate)-{-# INLINABLE break #-}+{-# INLINEABLE break #-} -{-| Improper lens that splits after the first group of equivalent Chars, as- defined by the given equivalence relation--}-groupBy- :: (Monad m)- => (Char -> Char -> Bool)- -> Lens' (Producer Text m r)- (Producer Text m (Producer Text m r))-groupBy equals k p0 = fmap join (k ((go p0))) where+-- | Improper lens that splits after the first group of equivalent Chars, as+-- defined by the given equivalence relation+groupBy ::+ (Monad m) =>+ (Char -> Char -> Bool) ->+ Lens'+ (Producer Text m r)+ (Producer Text m (Producer Text m r))+groupBy equals k p0 = fmap join (k (go p0))+ where go p = do- x <- lift (next p)- case x of- Left r -> return (return r)- Right (txt, p') -> case T.uncons txt of- Nothing -> go p'- Just (c, _) -> (yield txt >> p') ^. span (equals c)-{-# INLINABLE groupBy #-}+ x <- lift (next p)+ case x of+ Left r -> return (return r)+ Right (txt, p') -> case T.uncons txt of+ Nothing -> go p'+ Just (c, _) -> (yield txt >> p') ^. span (equals c)+{-# INLINEABLE groupBy #-} -- | Improper lens that splits after the first succession of identical 'Char' s-group :: Monad m- => Lens' (Producer Text m r)- (Producer Text m (Producer Text m r))+group ::+ Monad m =>+ Lens'+ (Producer Text m r)+ (Producer Text m (Producer Text m r)) group = groupBy (==)-{-# INLINABLE group #-}--{-| Improper lens that splits a 'Producer' after the first word+{-# INLINEABLE group #-} - Unlike 'words', this does not drop leading whitespace--}-word :: (Monad m)- => Lens' (Producer Text m r)- (Producer Text m (Producer Text m r))+-- | Improper lens that splits a 'Producer' after the first word+--+-- Unlike 'words', this does not drop leading whitespace+word ::+ (Monad m) =>+ Lens'+ (Producer Text m r)+ (Producer Text m (Producer Text m r)) word k p0 = fmap join (k (to p0)) where to p = do- p' <- p^.span isSpace- p'^.break isSpace-{-# INLINABLE word #-}+ p' <- p ^. span isSpace+ p' ^. break isSpace+{-# INLINEABLE word #-} -line :: (Monad m)- => Lens' (Producer Text m r)- (Producer Text m (Producer Text m r))+line ::+ (Monad m) =>+ Lens'+ (Producer Text m r)+ (Producer Text m (Producer Text m r)) line = break (== '\n')-{-# INLINABLE line #-}+{-# INLINEABLE line #-} -- | @(drop n)@ drops the first @n@ characters-drop :: (Monad m, Integral n)- => n -> Producer Text m r -> Producer Text m r-drop n p = do- p' <- lift $ runEffect (for (p ^. splitAt n) discard)- p'-{-# INLINABLE drop #-}+drop ::+ (Monad m, Integral n) =>+ n ->+ Producer Text m r ->+ Producer Text m r+drop n p =+ join (lift $ runEffect (for (p ^. splitAt n) discard))+{-# INLINEABLE drop #-} -- | Drop characters until they fail the predicate-dropWhile :: (Monad m)- => (Char -> Bool) -> Producer Text m r -> Producer Text m r-dropWhile predicate p = do- p' <- lift $ runEffect (for (p ^. span predicate) discard)- p'-{-# INLINABLE dropWhile #-}+dropWhile ::+ (Monad m) =>+ (Char -> Bool) ->+ Producer Text m r ->+ Producer Text m r+dropWhile predicate p =+ join (lift $ runEffect (for (p ^. span predicate) discard))+{-# INLINEABLE dropWhile #-} -- | Intersperse a 'Char' in between the characters of stream of 'Text'-intersperse- :: (Monad m) => Char -> Producer Text m r -> Producer Text m r+intersperse ::+ (Monad m) => Char -> Producer Text m r -> Producer Text m r intersperse c = go0 where go0 p = do- x <- lift (next p)- case x of- Left r -> return r- Right (txt, p') -> do- yield (T.intersperse c txt)- go1 p'+ x <- lift (next p)+ case x of+ Left r -> return r+ Right (txt, p') -> do+ yield (T.intersperse c txt)+ go1 p' go1 p = do- x <- lift (next p)- case x of- Left r -> return r- Right (txt, p') -> do- yield (T.singleton c)- yield (T.intersperse c txt)- go1 p'-{-# INLINABLE intersperse #-}-+ x <- lift (next p)+ case x of+ Left r -> return r+ Right (txt, p') -> do+ yield (T.singleton c)+ yield (T.intersperse c txt)+ go1 p'+{-# INLINEABLE intersperse #-} -- | Improper lens from unpacked 'Word8's to packaged 'ByteString's pack :: Monad m => Lens' (Producer Char m r) (Producer Text m r) pack k p = fmap _unpack (k (_pack p))-{-# INLINABLE pack #-}+{-# INLINEABLE pack #-} -- | Improper lens from packed 'ByteString's to unpacked 'Word8's unpack :: Monad m => Lens' (Producer Text m r) (Producer Char m r) unpack k p = fmap _pack (k (_unpack p))-{-# INLINABLE unpack #-}+{-# INLINEABLE unpack #-} _pack :: Monad m => Producer Char m r -> Producer Text m r-_pack p = folds step id done (p^.PG.chunksOf defaultChunkSize)+_pack p = folds step id done (p ^. PG.chunksOf defaultChunkSize) where- step diffAs w8 = diffAs . (w8:)+ step diffAs w8 = diffAs . (w8 :) done diffAs = T.pack (diffAs [])-{-# INLINABLE _pack #-}+{-# INLINEABLE _pack #-} _unpack :: Monad m => Producer Text m r -> Producer Char m r _unpack p = for p (each . T.unpack)-{-# INLINABLE _unpack #-}+{-# INLINEABLE _unpack #-} defaultChunkSize :: Int defaultChunkSize = 16384 - (sizeOf (undefined :: Int) `shiftL` 1) - -- | Split a text stream into 'FreeT'-delimited text streams of fixed size-chunksOf- :: (Monad m, Integral n)- => n -> Lens' (Producer Text m r)- (FreeT (Producer Text m) m r)+chunksOf ::+ (Monad m, Integral n) =>+ n ->+ Lens'+ (Producer Text m r)+ (FreeT (Producer Text m) m r) chunksOf n k p0 = fmap concats (k (FreeT (go p0))) where go p = do- x <- next p- return $ case x of- Left r -> Pure r- Right (txt, p') -> Free $ do- p'' <- (yield txt >> p') ^. splitAt n- return $ FreeT (go p'')-{-# INLINABLE chunksOf #-}-+ x <- next p+ return $ case x of+ Left r -> Pure r+ Right (txt, p') -> Free $ do+ p'' <- (yield txt >> p') ^. splitAt n+ return $ FreeT (go p'')+{-# INLINEABLE chunksOf #-} -{-| Split a text stream into sub-streams delimited by characters that satisfy the- predicate--}-splitsWith- :: (Monad m)- => (Char -> Bool)- -> Producer Text m r -> FreeT (Producer Text m) m r+-- | Split a text stream into sub-streams delimited by characters that satisfy the+-- predicate+splitsWith ::+ (Monad m) =>+ (Char -> Bool) ->+ Producer Text m r ->+ FreeT (Producer Text m) m r splitsWith predicate p0 = FreeT (go0 p0) where go0 p = do- x <- next p- case x of- Left r -> return (Pure r)- Right (txt, p') ->- if (T.null txt)- then go0 p'- else return $ Free $ do- p'' <- (yield txt >> p') ^. span (not . predicate)- return $ FreeT (go1 p'')+ x <- next p+ case x of+ Left r -> return (Pure r)+ Right (txt, p') ->+ if T.null txt+ then go0 p'+ else return $+ Free $ do+ p'' <- (yield txt >> p') ^. span (not . predicate)+ return $ FreeT (go1 p'') go1 p = do- x <- nextChar p- return $ case x of- Left r -> Pure r- Right (_, p') -> Free $ do- p'' <- p' ^. span (not . predicate)- return $ FreeT (go1 p'')-{-# INLINABLE splitsWith #-}+ x <- nextChar p+ return $ case x of+ Left r -> Pure r+ Right (_, p') -> Free $ do+ p'' <- p' ^. span (not . predicate)+ return $ FreeT (go1 p'')+{-# INLINEABLE splitsWith #-} -- | Split a text stream using the given 'Char' as the delimiter-splits :: (Monad m)- => Char- -> Lens' (Producer Text m r)- (FreeT (Producer Text m) m r)+splits ::+ (Monad m) =>+ Char ->+ Lens'+ (Producer Text m r)+ (FreeT (Producer Text m) m r) splits c k p =- fmap (intercalates (yield (T.singleton c))) (k (splitsWith (c ==) p))-{-# INLINABLE splits #-}--{-| Isomorphism between a stream of 'Text' and groups of equivalent 'Char's , using the- given equivalence relation--}-groupsBy- :: Monad m- => (Char -> Char -> Bool)- -> Lens' (Producer Text m x) (FreeT (Producer Text m) m x)-groupsBy equals k p0 = fmap concats (k (FreeT (go p0))) where- go p = do x <- next p- case x of Left r -> return (Pure r)- Right (bs, p') -> case T.uncons bs of- Nothing -> go p'- Just (c, _) -> do return $ Free $ do- p'' <- (yield bs >> p')^.span (equals c)- return $ FreeT (go p'')-{-# INLINABLE groupsBy #-}+ fmap (intercalates (yield (T.singleton c))) (k (splitsWith (c ==) p))+{-# INLINEABLE splits #-} +-- | Isomorphism between a stream of 'Text' and groups of equivalent 'Char's , using the+-- given equivalence relation+groupsBy ::+ Monad m =>+ (Char -> Char -> Bool) ->+ Lens' (Producer Text m x) (FreeT (Producer Text m) m x)+groupsBy equals k p0 = fmap concats (k (FreeT (go p0)))+ where+ go p = do+ x <- next p+ case x of+ Left r -> return (Pure r)+ Right (bs, p') -> case T.uncons bs of+ Nothing -> go p'+ Just (c, _) -> do+ return $+ Free $ do+ p'' <- (yield bs >> p') ^. span (equals c)+ return $ FreeT (go p'')+{-# INLINEABLE groupsBy #-} -- | Like 'groupsBy', where the equality predicate is ('==')-groups- :: Monad m- => Lens' (Producer Text m x) (FreeT (Producer Text m) m x)+groups ::+ Monad m =>+ Lens' (Producer Text m x) (FreeT (Producer Text m) m x) groups = groupsBy (==)-{-# INLINABLE groups #-}--+{-# INLINEABLE groups #-} -{-| Split a text stream into 'FreeT'-delimited lines--}-lines- :: (Monad m) => Lens' (Producer Text m r) (FreeT (Producer Text m) m r)+-- | Split a text stream into 'FreeT'-delimited lines+lines ::+ (Monad m) => Lens' (Producer Text m r) (FreeT (Producer Text m) m r) lines k p = fmap _unlines (k (_lines p))-{-# INLINABLE lines #-}+{-# INLINEABLE lines #-} -unlines- :: Monad m- => Lens' (FreeT (Producer Text m) m r) (Producer Text m r)+unlines ::+ Monad m =>+ Lens' (FreeT (Producer Text m) m r) (Producer Text m r) unlines k p = fmap _lines (k (_unlines p))-{-# INLINABLE unlines #-}+{-# INLINEABLE unlines #-} -_lines :: Monad m- => Producer Text m r -> FreeT (Producer Text m) m r+_lines ::+ Monad m =>+ Producer Text m r ->+ FreeT (Producer Text m) m r _lines p0 = FreeT (go0 p0)- where- go0 p = do- x <- next p- case x of- Left r -> return (Pure r)- Right (txt, p') ->- if (T.null txt)- then go0 p'- else return $ Free $ go1 (yield txt >> p')- go1 p = do- p' <- p ^. break ('\n' ==)- return $ FreeT $ do- x <- nextChar p'- case x of- Left r -> return $ Pure r- Right (_, p'') -> go0 p''-{-# INLINABLE _lines #-}+ where+ go0 p = do+ x <- next p+ case x of+ Left r -> return (Pure r)+ Right (txt, p') ->+ if T.null txt+ then go0 p'+ else return $ Free $ go1 (yield txt >> p')+ go1 p = do+ p' <- p ^. break ('\n' ==)+ return $+ FreeT $ do+ x <- nextChar p'+ case x of+ Left r -> return $ Pure r+ Right (_, p'') -> go0 p''+{-# INLINEABLE _lines #-} -_unlines :: Monad m- => FreeT (Producer Text m) m r -> Producer Text m r+_unlines ::+ Monad m =>+ FreeT (Producer Text m) m r ->+ Producer Text m r _unlines = concats . maps (<* yield (T.singleton '\n'))-{-# INLINABLE _unlines #-}+{-# INLINEABLE _unlines #-} --- | Split a text stream into 'FreeT'-delimited words. Note that +-- | Split a text stream into 'FreeT'-delimited words. Note that -- roundtripping with e.g. @over words id@ eliminates extra space -- characters as with @Prelude.unwords . Prelude.words@-words- :: (Monad m) => Lens' (Producer Text m r) (FreeT (Producer Text m) m r)+words ::+ (Monad m) => Lens' (Producer Text m r) (FreeT (Producer Text m) m r) words k p = fmap _unwords (k (_words p))-{-# INLINABLE words #-}+{-# INLINEABLE words #-} -unwords- :: Monad m- => Lens' (FreeT (Producer Text m) m r) (Producer Text m r)+unwords ::+ Monad m =>+ Lens' (FreeT (Producer Text m) m r) (Producer Text m r) unwords k p = fmap _words (k (_unwords p))-{-# INLINABLE unwords #-}+{-# INLINEABLE unwords #-} _words :: (Monad m) => Producer Text m r -> FreeT (Producer Text m) m r _words p = FreeT $ do- x <- next (dropWhile isSpace p)- return $ case x of- Left r -> Pure r- Right (bs, p') -> Free $ do- p'' <- (yield bs >> p') ^. break isSpace- return (_words p'')-{-# INLINABLE _words #-}+ x <- next (dropWhile isSpace p)+ return $ case x of+ Left r -> Pure r+ Right (bs, p') -> Free $ do+ p'' <- (yield bs >> p') ^. break isSpace+ return (_words p'')+{-# INLINEABLE _words #-} _unwords :: (Monad m) => FreeT (Producer Text m) m r -> Producer Text m r _unwords = intercalates (yield $ T.singleton ' ')-{-# INLINABLE _unwords #-}-+{-# INLINEABLE _unwords #-} -{-| 'intercalate' concatenates the 'FreeT'-delimited text streams after- interspersing a text stream in between them--}-intercalate- :: (Monad m)- => Producer Text m () -> FreeT (Producer Text m) m r -> Producer Text m r+-- | 'intercalate' concatenates the 'FreeT'-delimited text streams after+-- interspersing a text stream in between them+intercalate ::+ (Monad m) =>+ Producer Text m () ->+ FreeT (Producer Text m) m r ->+ Producer Text m r intercalate p0 = go0 where go0 f = do- x <- lift (runFreeT f)- case x of- Pure r -> return r- Free p -> do- f' <- p- go1 f'+ x <- lift (runFreeT f)+ case x of+ Pure r -> return r+ Free p -> do+ f' <- p+ go1 f' go1 f = do- x <- lift (runFreeT f)- case x of- Pure r -> return r- Free p -> do- p0- f' <- p- go1 f'-{-# INLINABLE intercalate #-}----{- $reexports-- @Data.Text@ re-exports the 'Text' type.-- @Pipes.Parse@ re-exports 'input', 'concat', 'FreeT' (the type) and the 'Parse' synonym.--}+ x <- lift (runFreeT f)+ case x of+ Pure r -> return r+ Free p -> do+ p0+ f' <- p+ go1 f'+{-# INLINEABLE intercalate #-} +-- $reexports+--+-- @Data.Text@ re-exports the 'Text' type.+--+-- @Pipes.Parse@ re-exports 'input', 'concat', 'FreeT' (the type) and the 'Parse' synonym. -type Lens' a b = forall f . Functor f => (b -> f b) -> (a -> f a)+type Lens' a b = forall f. Functor f => (b -> f b) -> (a -> f a)
Pipes/Text/Encoding.hs view
@@ -1,276 +1,279 @@-{-# LANGUAGE RankNTypes, BangPatterns #-}+{-# LANGUAGE RankNTypes #-} -- | This module uses the stream decoding functions from--- <http://hackage.haskell.org/package/streaming-commons streaming-commons> +-- <http://hackage.haskell.org/package/streaming-commons streaming-commons> -- package to define decoding functions and lenses. The exported names--- conflict with names in @Data.Text.Encoding@ but not with the @Prelude@ -+-- conflict with names in @Data.Text.Encoding@ but not with the @Prelude@ module Pipes.Text.Encoding- ( - -- * Decoding ByteStrings and Encoding Texts+ ( -- * Decoding ByteStrings and Encoding Texts+ -- ** Simple usage -- $usage- + -- ** Lens usage -- $lenses- - + -- * Basic lens operations- Codec- , decode- , eof+ Codec,+ decode,+ eof,+ -- * Decoding lenses- , utf8- , utf8Pure- , utf16LE- , utf16BE- , utf32LE- , utf32BE- -- * Non-lens decoding functions + utf8,+ utf8Pure,+ utf16LE,+ utf16BE,+ utf32LE,+ utf32BE,++ -- * Non-lens decoding functions -- $decoders- , decodeUtf8- , decodeUtf8Pure- , decodeUtf16LE- , decodeUtf16BE- , decodeUtf32LE- , decodeUtf32BE+ decodeUtf8,+ decodeUtf8Pure,+ decodeUtf16LE,+ decodeUtf16BE,+ decodeUtf32LE,+ decodeUtf32BE,+ -- * Re-encoding functions -- $encoders- , encodeUtf8- , encodeUtf16LE- , encodeUtf16BE- , encodeUtf32LE- , encodeUtf32BE+ encodeUtf8,+ encodeUtf16LE,+ encodeUtf16BE,+ encodeUtf32LE,+ encodeUtf32BE,+ -- * Functions for latin and ascii text -- $ascii- , encodeAscii- , decodeAscii- , encodeIso8859_1- , decodeIso8859_1- ) - where+ encodeAscii,+ decodeAscii,+ encodeIso8859_1,+ decodeIso8859_1,+ )+where -import Data.Functor.Constant (Constant(..))-import Data.Char (ord)-import Data.ByteString as B +import Control.Monad (join)+import Data.ByteString as B import Data.ByteString.Char8 as B8-import Data.Text (Text)-import qualified Data.Text as T -import qualified Data.Text.Encoding as TE +import Data.Char (ord)+import Data.Functor.Constant (Constant (..))+import Data.Streaming.Text (DecodeResult (..)) import qualified Data.Streaming.Text as Stream-import Data.Streaming.Text (DecodeResult(..))-import Control.Monad (join, liftM)+import Data.Text (Text)+import qualified Data.Text as T+import qualified Data.Text.Encoding as TE import Pipes ---{- $usage- Encoding is of course simple. Given --> text :: Producer Text IO ()-- we can encode it with @Data.Text.Encoding.encodeUtf8@ --> TE.encodeUtf8 :: Text -> ByteString-- and ordinary pipe operations:--> text >-> P.map TE.encodeUtf8 :: Producer.ByteString IO ()-- or, equivalently--> for text (yield . TE.encodeUtf8)-- But, using this module, we might use--> encodeUtf8 :: Text -> Producer ByteString m ()-- to write--> for text encodeUtf8 :: Producer.ByteString IO ()-- All of the above come to the same. --- Given--> bytes :: Producer ByteString IO ()-- we can apply a decoding function from this module:--> decodeUtf8 bytes :: Producer Text IO (Producer ByteString IO ())-- The Text producer ends wherever decoding first fails. The un-decoded- material is returned. If we are confident it is of no interest, we can- write: --> void $ decodeUtf8 bytes :: Producer Text IO ()-- Thus we can re-encode- as uft8 as much of our byte stream as is decodeUtf16BE decodable, with, e.g.--> for (decodeUtf16BE bytes) encodeUtf8 :: Producer ByteString IO (Producer ByteString IO ())- - The bytestring producer that is returned begins with where utf16BE decoding- failed; if it didn't fail the producer is empty. ---}--{- $lenses- We get a bit more flexibility, particularly in the use of pipes-style "parsers", - if we use a lens like @utf8@ or @utf16BE@ - that focusses on the text in an appropriately encoded byte stream.--> type Lens' a b = forall f . Functor f => (b -> f b) -> (a -> f a)-- is just an alias for a Prelude type. We abbreviate this further, for our use case, as--> type Codec-> = forall m r . Monad m => Lens' (Producer ByteString m r) (Producer Text m (Producer ByteString m r))-- and call the decoding lenses @utf8@, @utf16BE@ \"codecs\", since they can - re-encode what they have decoded. Thus you use any particular codec with- the @view@ / @(^.)@ , @zoom@ and @over@ functions from the standard lens libraries;- <http://hackage.haskell.org/package/lens lens>,- <http://hackage.haskell.org/package/lens-family lens-family>,- <http://hackage.haskell.org/package/lens-simple lens-simple>, or one of the- and <http://hackage.haskell.org/package/microlens microlens> packages will all work- the same, since we already have access to the types they require. -- Each decoding lens looks into a byte stream that is supposed to contain text.- The particular lenses are named in accordance with the expected - encoding, 'utf8', 'utf16LE' etc. To turn a such a lens or @Codec@ - into an ordinary function, use @view@ / @(^.)@ -- here also called 'decode':--> view utf8 :: Producer ByteString m r -> Producer Text m (Producer ByteString m r)-> decode utf8 Byte.stdin :: Producer Text IO (Producer ByteString IO r)-> Bytes.stdin ^. utf8 :: Producer Text IO (Producer ByteString IO r)-- Of course, we could always do this with the specialized decoding functions, e.g. --> decodeUtf8 :: Producer ByteString m r -> Producer Text m (Producer ByteString m r)-> decodeUtf8 Byte.stdin :: Producer Text IO (Producer ByteString IO r)-- As with these functions, the stream of text that a @Codec@ \'sees\' - in the stream of bytes begins at its head. - At any point of decoding failure, the stream of text ends and reverts to (returns) - the original byte stream. Thus if the first bytes are already- un-decodable, the whole ByteString producer will be returned, i.e.--> view utf8 bad_bytestream -- will just come to the same as --> return bad_bytestream-- Where there is no decoding failure, the return value of the text stream will be- an empty byte stream followed by its own return value. In all cases you must- deal with the fact that it is a /ByteString producer/ that is returned, even if- it can be thrown away with @Control.Monad.void@--> void (Bytes.stdin ^. utf8) :: Producer Text IO ()-- The @eof@ lens permits you to pattern match: if there is a Right value,- it is the leftover bytestring producer, if there is a Right value, it - is the return value of the original bytestring producer:--> Bytes.stdin ^. utf8 . eof :: Producer Text IO (Either (Producer ByteString IO IO) ())- - Thus for the stream of un-decodable bytes mentioned above,--> view (utf8 . eof) bad_bytestream-- will be the same as --> return (Left bad_bytestream)-- @zoom utf8@ converts a Text parser into a ByteString parser:--> zoom utf8 drawChar :: Monad m => StateT (Producer ByteString m r) m (Maybe Char)-- or, using the type synonymn from @Pipes.Parse@:- -> zoom utf8 drawChar :: Monad m => Parser ByteString m (Maybe Char)-- Thus we can define a ByteString parser (in the pipes-parse sense) like this:- -> charPlusByte :: Parser ByteString m (Maybe Char, Maybe Word8))) -> charPlusByte = do char_ <- zoom utf8 Text.drawChar-> byte_ <- Bytes.peekByte-> return (char_, byte_)-- Though @charPlusByte@ is partly defined with a Text parser 'drawChar'; - but it is a ByteString parser; it will return the first valid utf8-encoded - Char in a ByteString, /whatever its byte-length/, - and the first byte following, if both exist. Because - we \'draw\' one and \'peek\' at the other, the parser as a whole only - advances one Char's length along the bytestring, whatever that length may be.- See the slightly more complex example \'decode.hs\' in the - <http://www.haskellforall.com/2014/02/pipes-parse-30-lens-based-parsing.html#batteries-included haskellforall blog> - discussion of this type of byte stream parsing. - -}--type Lens' a b = forall f . Functor f => (b -> f b) -> (a -> f a)--type Codec- = forall m r- . Monad m- => Lens' (Producer ByteString m r)- (Producer Text m (Producer ByteString m r))---{- | @decode@ is just the ordinary @view@ or @(^.)@ of the lens libraries;- exported here under a name appropriate to the material. - Thus given a bytestring producer called @bytes@ we have--> decode utf8 bytes :: Producer Text IO (Producer ByteString IO ())+-- $usage+-- Encoding is of course simple. Given+--+-- > text :: Producer Text IO ()+--+-- we can encode it with @Data.Text.Encoding.encodeUtf8@+--+-- > TE.encodeUtf8 :: Text -> ByteString+--+-- and ordinary pipe operations:+--+-- > text >-> P.map TE.encodeUtf8 :: Producer.ByteString IO ()+--+-- or, equivalently+--+-- > for text (yield . TE.encodeUtf8)+--+-- But, using this module, we might use+--+-- > encodeUtf8 :: Text -> Producer ByteString m ()+--+-- to write+--+-- > for text encodeUtf8 :: Producer.ByteString IO ()+--+-- All of the above come to the same.+--+--+-- Given+--+-- > bytes :: Producer ByteString IO ()+--+-- we can apply a decoding function from this module:+--+-- > decodeUtf8 bytes :: Producer Text IO (Producer ByteString IO ())+--+-- The Text producer ends wherever decoding first fails. The un-decoded+-- material is returned. If we are confident it is of no interest, we can+-- write:+--+-- > void $ decodeUtf8 bytes :: Producer Text IO ()+--+-- Thus we can re-encode+-- as uft8 as much of our byte stream as is decodeUtf16BE decodable, with, e.g.+--+-- > for (decodeUtf16BE bytes) encodeUtf8 :: Producer ByteString IO (Producer ByteString IO ())+--+-- The bytestring producer that is returned begins with where utf16BE decoding+-- failed; if it didn't fail the producer is empty. - All of these are thus the same:+-- $lenses+-- We get a bit more flexibility, particularly in the use of pipes-style "parsers",+-- if we use a lens like @utf8@ or @utf16BE@+-- that focusses on the text in an appropriately encoded byte stream.+--+-- > type Lens' a b = forall f . Functor f => (b -> f b) -> (a -> f a)+--+-- is just an alias for a Prelude type. We abbreviate this further, for our use case, as+--+-- > type Codec+-- > = forall m r . Monad m => Lens' (Producer ByteString m r) (Producer Text m (Producer ByteString m r))+--+-- and call the decoding lenses @utf8@, @utf16BE@ \"codecs\", since they can+-- re-encode what they have decoded. Thus you use any particular codec with+-- the @view@ / @(^.)@ , @zoom@ and @over@ functions from the standard lens libraries;+-- <http://hackage.haskell.org/package/lens lens>,+-- <http://hackage.haskell.org/package/lens-family lens-family>,+-- <http://hackage.haskell.org/package/lens-simple lens-simple>, or one of the+-- and <http://hackage.haskell.org/package/microlens microlens> packages will all work+-- the same, since we already have access to the types they require.+--+-- Each decoding lens looks into a byte stream that is supposed to contain text.+-- The particular lenses are named in accordance with the expected+-- encoding, 'utf8', 'utf16LE' etc. To turn a such a lens or @Codec@+-- into an ordinary function, use @view@ / @(^.)@ -- here also called 'decode':+--+-- > view utf8 :: Producer ByteString m r -> Producer Text m (Producer ByteString m r)+-- > decode utf8 Byte.stdin :: Producer Text IO (Producer ByteString IO r)+-- > Bytes.stdin ^. utf8 :: Producer Text IO (Producer ByteString IO r)+--+-- Of course, we could always do this with the specialized decoding functions, e.g.+--+-- > decodeUtf8 :: Producer ByteString m r -> Producer Text m (Producer ByteString m r)+-- > decodeUtf8 Byte.stdin :: Producer Text IO (Producer ByteString IO r)+--+-- As with these functions, the stream of text that a @Codec@ \'sees\'+-- in the stream of bytes begins at its head.+-- At any point of decoding failure, the stream of text ends and reverts to (returns)+-- the original byte stream. Thus if the first bytes are already+-- un-decodable, the whole ByteString producer will be returned, i.e.+--+-- > view utf8 bad_bytestream+--+-- will just come to the same as+--+-- > return bad_bytestream+--+-- Where there is no decoding failure, the return value of the text stream will be+-- an empty byte stream followed by its own return value. In all cases you must+-- deal with the fact that it is a /ByteString producer/ that is returned, even if+-- it can be thrown away with @Control.Monad.void@+--+-- > void (Bytes.stdin ^. utf8) :: Producer Text IO ()+--+-- The @eof@ lens permits you to pattern match: if there is a Right value,+-- it is the leftover bytestring producer, if there is a Right value, it+-- is the return value of the original bytestring producer:+--+-- > Bytes.stdin ^. utf8 . eof :: Producer Text IO (Either (Producer ByteString IO IO) ())+--+-- Thus for the stream of un-decodable bytes mentioned above,+--+-- > view (utf8 . eof) bad_bytestream+--+-- will be the same as+--+-- > return (Left bad_bytestream)+--+-- @zoom utf8@ converts a Text parser into a ByteString parser:+--+-- > zoom utf8 drawChar :: Monad m => StateT (Producer ByteString m r) m (Maybe Char)+--+-- or, using the type synonymn from @Pipes.Parse@:+--+-- > zoom utf8 drawChar :: Monad m => Parser ByteString m (Maybe Char)+--+-- Thus we can define a ByteString parser (in the pipes-parse sense) like this:+--+-- > charPlusByte :: Parser ByteString m (Maybe Char, Maybe Word8)))+-- > charPlusByte = do char_ <- zoom utf8 Text.drawChar+-- > byte_ <- Bytes.peekByte+-- > return (char_, byte_)+--+-- Though @charPlusByte@ is partly defined with a Text parser 'drawChar';+-- but it is a ByteString parser; it will return the first valid utf8-encoded+-- Char in a ByteString, /whatever its byte-length/,+-- and the first byte following, if both exist. Because+-- we \'draw\' one and \'peek\' at the other, the parser as a whole only+-- advances one Char's length along the bytestring, whatever that length may be.+-- See the slightly more complex example \'decode.hs\' in the+-- <http://www.haskellforall.com/2014/02/pipes-parse-30-lens-based-parsing.html#batteries-included haskellforall blog>+-- discussion of this type of byte stream parsing. -> decode utf8 bytes -> view utf8 bytes-> bytes ^. utf8 -> decodeUtf8 bytes+type Lens' a b = forall f. Functor f => (b -> f b) -> (a -> f a) --}+type Codec =+ forall m r.+ Monad m =>+ Lens'+ (Producer ByteString m r)+ (Producer Text m (Producer ByteString m r)) +-- | @decode@ is just the ordinary @view@ or @(^.)@ of the lens libraries;+-- exported here under a name appropriate to the material.+-- Thus given a bytestring producer called @bytes@ we have+--+-- > decode utf8 bytes :: Producer Text IO (Producer ByteString IO ())+--+-- All of these are thus the same:+--+-- > decode utf8 bytes+-- > view utf8 bytes+-- > bytes ^. utf8+-- > decodeUtf8 bytes decode :: ((b -> Constant b b) -> (a -> Constant b a)) -> a -> b decode codec a = getConstant (codec Constant a) -{- | @eof@ tells you explicitly when decoding stops due to bad bytes or - instead reaches end-of-file happily. (Without it one just makes an explicit - test for emptiness of the resulting bytestring production using next) Thus--> decode (utf8 . eof) bytes :: Producer T.Text IO (Either (Producer B.ByteString IO ()) ())-- If we hit undecodable bytes, the remaining bytestring producer will be - returned as a Left value; in the happy case, a Right value is returned - with the anticipated return value for the original bytestring producer.-- Given a bytestring producer called @bytes@ all of these will be the same:--> decode (utf8 . eof) bytes -> view (utf8 . eof) bytes-> bytes^.utf8.eof---}--eof :: (Monad m, Monad (t m), MonadTrans t) => Lens' (t m (Producer ByteString m r))- (t m (Either (Producer ByteString m r) r))-eof k p0 = fmap fromEither (k (toEither p0)) where-- fromEither = liftM (either id return)+-- | @eof@ tells you explicitly when decoding stops due to bad bytes or+-- instead reaches end-of-file happily. (Without it one just makes an explicit+-- test for emptiness of the resulting bytestring production using next) Thus+--+-- > decode (utf8 . eof) bytes :: Producer T.Text IO (Either (Producer B.ByteString IO ()) ())+--+-- If we hit undecodable bytes, the remaining bytestring producer will be+-- returned as a Left value; in the happy case, a Right value is returned+-- with the anticipated return value for the original bytestring producer.+--+-- Given a bytestring producer called @bytes@ all of these will be the same:+--+-- > decode (utf8 . eof) bytes+-- > view (utf8 . eof) bytes+-- > bytes^.utf8.eof+eof ::+ (Monad m, Monad (t m), MonadTrans t) =>+ Lens'+ (t m (Producer ByteString m r))+ (t m (Either (Producer ByteString m r) r))+eof k p0 = fmap fromEither (k (toEither p0))+ where+ fromEither = fmap (either id return) - toEither pp = do p <- pp- check p+ toEither pp = do+ p <- pp+ check p - check p = do e <- lift (next p)- case e of - Left r -> return (Right r)- Right (bs,pb) -> if B.null bs - then check pb- else return (Left (do yield bs- pb))+ check p = do+ e <- lift (next p)+ case e of+ Left r -> return (Right r)+ Right (bs, pb) ->+ if B.null bs+ then check pb+ else+ return+ ( Left+ ( do+ yield bs+ pb+ )+ ) utf8 :: Codec utf8 = mkCodec decodeUtf8 TE.encodeUtf8@@ -290,38 +293,44 @@ utf32BE :: Codec utf32BE = mkCodec decodeUtf32BE TE.encodeUtf32BE -decodeStream :: Monad m - => (B.ByteString -> DecodeResult) - -> Producer ByteString m r -> Producer Text m (Producer ByteString m r)-decodeStream = loop where- loop dec0 p = - do x <- lift (next p) - case x of - Left r -> return (return r)- Right (chunk, p') -> case dec0 chunk of - DecodeResultSuccess text dec -> do yield text- loop dec p'- DecodeResultFailure text bs -> do yield text - return (do yield bs - p')-{-# INLINABLE decodeStream#-}---{- $decoders- These are functions with the simple type:- -> decodeUtf8 :: Monad m => Producer ByteString m r -> Producer Text m (Producer ByteString m r)-- Thus in general --> decodeUtf8 = view utf8-> decodeUtf16LE = view utf16LE-- and so forth, but these forms- may be more convenient (and give better type errors!) where lenses are- not desired.--}+decodeStream ::+ Monad m =>+ (B.ByteString -> DecodeResult) ->+ Producer ByteString m r ->+ Producer Text m (Producer ByteString m r)+decodeStream = loop+ where+ loop dec0 p =+ do+ x <- lift (next p)+ case x of+ Left r -> return (return r)+ Right (chunk, p') -> case dec0 chunk of+ DecodeResultSuccess text dec -> do+ yield text+ loop dec p'+ DecodeResultFailure text bs -> do+ yield text+ return+ ( do+ yield bs+ p'+ )+{-# INLINEABLE decodeStream #-} +-- $decoders+-- These are functions with the simple type:+--+-- > decodeUtf8 :: Monad m => Producer ByteString m r -> Producer Text m (Producer ByteString m r)+--+-- Thus in general+--+-- > decodeUtf8 = view utf8+-- > decodeUtf16LE = view utf16LE+--+-- and so forth, but these forms+-- may be more convenient (and give better type errors!) where lenses are+-- not desired. decodeUtf8 :: Monad m => Producer ByteString m r -> Producer Text m (Producer ByteString m r) decodeUtf8 = decodeStream Stream.decodeUtf8@@ -347,120 +356,140 @@ decodeUtf32BE = decodeStream Stream.decodeUtf32BE {-# INLINE decodeUtf32BE #-} --{- $encoders- These are simply defined - -> encodeUtf8 = yield . TE.encodeUtf8- - They are intended for use with 'for'- -> for Text.stdin encodeUtf8 :: Producer ByteString IO ()-- which would have the effect of - -> Text.stdin >-> Pipes.Prelude.map (TE.encodeUtf8)-- using the encoding functions from Data.Text.Encoding --}+-- $encoders+-- These are simply defined+--+-- > encodeUtf8 = yield . TE.encodeUtf8+--+-- They are intended for use with 'for'+--+-- > for Text.stdin encodeUtf8 :: Producer ByteString IO ()+--+-- which would have the effect of+--+-- > Text.stdin >-> Pipes.Prelude.map (TE.encodeUtf8)+--+-- using the encoding functions from Data.Text.Encoding encodeUtf8 :: Monad m => Text -> Producer' ByteString m () encodeUtf8 = yield . TE.encodeUtf8+ encodeUtf16LE :: Monad m => Text -> Producer' ByteString m () encodeUtf16LE = yield . TE.encodeUtf16LE+ encodeUtf16BE :: Monad m => Text -> Producer' ByteString m () encodeUtf16BE = yield . TE.encodeUtf16BE+ encodeUtf32LE :: Monad m => Text -> Producer' ByteString m () encodeUtf32LE = yield . TE.encodeUtf32LE+ encodeUtf32BE :: Monad m => Text -> Producer' ByteString m () encodeUtf32BE = yield . TE.encodeUtf32BE -mkCodec :: (forall r m . Monad m => - Producer ByteString m r -> Producer Text m (Producer ByteString m r ))- -> (Text -> ByteString)- -> Codec-mkCodec dec enc = \k p0 -> fmap (\p -> join (for p (yield . enc))) (k (dec p0))----{- $ascii- ascii and latin encodings only use a small number of the characters 'Text'- recognizes; thus we cannot use the pipes @Lens@ style to work with them. - Rather we simply define functions each way. --}+mkCodec ::+ ( forall r m.+ Monad m =>+ Producer ByteString m r ->+ Producer Text m (Producer ByteString m r)+ ) ->+ (Text -> ByteString) ->+ Codec+mkCodec dec enc k p0 = fmap (\p -> join (for p (yield . enc))) (k (dec p0)) +-- $ascii+-- ascii and latin encodings only use a small number of the characters 'Text'+-- recognizes; thus we cannot use the pipes @Lens@ style to work with them.+-- Rather we simply define functions each way. -- | 'encodeAscii' reduces as much of your stream of 'Text' actually is ascii to a byte stream, -- returning the rest of the 'Text' at the first non-ascii 'Char'- encodeAscii :: Monad m => Producer Text m r -> Producer ByteString m (Producer Text m r)-encodeAscii = go where- go p = do e <- lift (next p)- case e of - Left r -> return (return r)- Right (chunk, p') -> - if T.null chunk - then go p'- else let (safe, unsafe) = T.span (\c -> ord c <= 0x7F) chunk- in do yield (B8.pack (T.unpack safe))- if T.null unsafe- then go p'- else return $ do yield unsafe - p'- -{- | Reduce as much of your stream of 'Text' actually is iso8859 or latin1 to a byte stream,- returning the rest of the 'Text' upon hitting any non-latin 'Char'- -}+encodeAscii = go+ where+ go p = do+ e <- lift (next p)+ case e of+ Left r -> return (return r)+ Right (chunk, p') ->+ if T.null chunk+ then go p'+ else+ let (safe, unsafe) = T.span (\c -> ord c <= 0x7F) chunk+ in do+ yield (B8.pack (T.unpack safe))+ if T.null unsafe+ then go p'+ else return $ do+ yield unsafe+ p'++-- | Reduce as much of your stream of 'Text' actually is iso8859 or latin1 to a byte stream,+-- returning the rest of the 'Text' upon hitting any non-latin 'Char' encodeIso8859_1 :: Monad m => Producer Text m r -> Producer ByteString m (Producer Text m r)-encodeIso8859_1 = go where- go p = do e <- lift (next p)- case e of - Left r -> return (return r)- Right (txt, p') -> - if T.null txt - then go p'- else let (safe, unsafe) = T.span (\c -> ord c <= 0xFF) txt- in do yield (B8.pack (T.unpack safe))- if T.null unsafe- then go p'- else return $ do yield unsafe - p'+encodeIso8859_1 = go+ where+ go p = do+ e <- lift (next p)+ case e of+ Left r -> return (return r)+ Right (txt, p') ->+ if T.null txt+ then go p'+ else+ let (safe, unsafe) = T.span (\c -> ord c <= 0xFF) txt+ in do+ yield (B8.pack (T.unpack safe))+ if T.null unsafe+ then go p'+ else return $ do+ yield unsafe+ p' -{- | Reduce a byte stream to a corresponding stream of ascii chars, returning the- unused 'ByteString' upon hitting an un-ascii byte.- -}+-- | Reduce a byte stream to a corresponding stream of ascii chars, returning the+-- unused 'ByteString' upon hitting an un-ascii byte. decodeAscii :: Monad m => Producer ByteString m r -> Producer Text m (Producer ByteString m r)-decodeAscii = go where- go p = do e <- lift (next p)- case e of - Left r -> return (return r)- Right (chunk, p') -> - if B.null chunk - then go p'- else let (safe, unsafe) = B.span (<= 0x7F) chunk- in do yield (T.pack (B8.unpack safe))- if B.null unsafe- then go p'- else return (do yield unsafe - p')+decodeAscii = go+ where+ go p = do+ e <- lift (next p)+ case e of+ Left r -> return (return r)+ Right (chunk, p') ->+ if B.null chunk+ then go p'+ else+ let (safe, unsafe) = B.span (<= 0x7F) chunk+ in do+ yield (T.pack (B8.unpack safe))+ if B.null unsafe+ then go p'+ else+ return+ ( do+ yield unsafe+ p'+ ) -{- | Reduce a byte stream to a corresponding stream of ascii chars, returning the- unused 'ByteString' upon hitting the rare un-latinizable byte.- -}+-- | Reduce a byte stream to a corresponding stream of ascii chars, returning the+-- unused 'ByteString' upon hitting the rare un-latinizable byte. decodeIso8859_1 :: Monad m => Producer ByteString m r -> Producer Text m (Producer ByteString m r)-decodeIso8859_1 = go where- go p = do e <- lift (next p)- case e of - Left r -> return (return r)- Right (chunk, p') -> - if B.null chunk - then go p'- else do let (safe, unsafe) = B.span (<= 0xFF) chunk- yield (T.pack (B8.unpack safe))- if B.null unsafe - then go p'- else return (do yield unsafe - p')---+decodeIso8859_1 = go+ where+ go p = do+ e <- lift (next p)+ case e of+ Left r -> return (return r)+ Right (chunk, p') ->+ if B.null chunk+ then go p'+ else do+ let (safe, unsafe) = B.span (<= 0xFF) chunk+ yield (T.pack (B8.unpack safe))+ if B.null unsafe+ then go p'+ else+ return+ ( do+ yield unsafe+ p'+ )
Pipes/Text/IO.hs view
@@ -1,174 +1,161 @@-{-#LANGUAGE RankNTypes#-}+{-# LANGUAGE RankNTypes #-} +module Pipes.Text.IO+ ( -- * Simple streaming text IO+ -- $textio -module Pipes.Text.IO - ( + -- * Caveats+ -- $caveats - -- * Simple streaming text IO- -- $textio- - -- * Caveats- -- $caveats- - -- * Producers- fromHandle- , stdin- , readFile- - -- * Consumers- , toHandle- , stdout- , writeFile- - -- * Re-exports- , MonadSafe(..)- , runSafeT- , runSafeP- , Safe.withFile- ) where+ -- * Producers+ fromHandle,+ stdin,+ readFile, -import qualified System.IO as IO+ -- * Consumers+ toHandle,+ stdout,+ writeFile,++ -- * Re-exports+ MonadSafe (..),+ runSafeT,+ runSafeP,+ Safe.withFile,+ )+where+ import Control.Exception (throwIO, try)-import Foreign.C.Error (Errno(Errno), ePIPE)-import qualified GHC.IO.Exception as G import Data.Text (Text) import qualified Data.Text as T import qualified Data.Text.IO as T+import Foreign.C.Error (Errno (Errno), ePIPE)+import qualified GHC.IO.Exception as G import Pipes+import Pipes.Safe (MonadSafe (..), runSafeP, runSafeT) import qualified Pipes.Safe.Prelude as Safe-import Pipes.Safe (MonadSafe(..), runSafeT, runSafeP)+import qualified System.IO as IO import Prelude hiding (readFile, writeFile) --{- $textio- Where pipes @IO@ replaces lazy @IO@, @Producer Text IO r@ replaces lazy 'Text'. - The official IO of this package and the pipes ecosystem generally would use the- IO functions in @Pipes.ByteString@ and the encoding and decoding material in - @Pipes.Text.Encoding@.-- The streaming functions exported here, namely, 'readFile', 'writeFile', 'fromHandle', 'toHandle', - 'stdin' and 'stdout' simplify this and use the system encoding on the model of @Data.Text.IO@ - and @Data.Text.Lazy.IO@ Some caveats described below. - - The main points are as in - <https://hackage.haskell.org/package/pipes-bytestring-1.0.0/docs/Pipes-ByteString.html Pipes.ByteString>:- - A 'Handle' can be associated with a 'Producer' or 'Consumer' according - as it is read or written to.- -> import Pipes-> import qualified Pipes.Text as Text-> import qualified Pipes.Text.IO as Text-> import System.IO->-> main =-> withFile "inFile.txt" ReadMode $ \hIn ->-> withFile "outFile.txt" WriteMode $ \hOut ->-> runEffect $ Text.fromHandle hIn >-> Text.toHandle hOut--To stream from files, the following is perhaps more Prelude-like (note that it uses Pipes.Safe):--> import Pipes-> import qualified Pipes.Text as Text-> import qualified Pipes.Text.IO as Text-> import Pipes.Safe->-> main = runSafeT $ runEffect $ Text.readFile "inFile.txt" >-> Text.writeFile "outFile.txt"-- Finally, you can stream to and from 'stdin' and 'stdout' using the predefined 'stdin'- and 'stdout' pipes, as with the following \"echo\" program:--> main = runEffect $ Text.stdin >-> Text.stdout-- These programs, unlike the corresponding programs written with the line-based functions,- will pass along a 1 terabyte line without affecting memory use. ---}---{- $caveats-- The operations exported here are a convenience, like the similar operations in - @Data.Text.IO@ (or rather, @Data.Text.Lazy.IO@, since, again, @Producer Text m r@ is- 'effectful text' and something like the pipes equivalent of lazy Text.)-- * Like the functions in @Data.Text.IO@, they attempt to work with the system encoding. - - * Like the functions in @Data.Text.IO@, they significantly slower than ByteString operations. Where- you know what encoding you are working with, use @Pipes.ByteString@ and @Pipes.Text.Encoding@ instead,- e.g. @view utf8 Bytes.stdin@ instead of @Text.stdin@- - * Like the functions in @Data.Text.IO@ , they use Text exceptions, not the standard Pipes protocols. ---}+-- $textio+-- Where pipes @IO@ replaces lazy @IO@, @Producer Text IO r@ replaces lazy 'Text'.+-- The official IO of this package and the pipes ecosystem generally would use the+-- IO functions in @Pipes.ByteString@ and the encoding and decoding material in+-- @Pipes.Text.Encoding@.+--+-- The streaming functions exported here, namely, 'readFile', 'writeFile', 'fromHandle', 'toHandle',+-- 'stdin' and 'stdout' simplify this and use the system encoding on the model of @Data.Text.IO@+-- and @Data.Text.Lazy.IO@ Some caveats described below.+--+-- The main points are as in+-- <https://hackage.haskell.org/package/pipes-bytestring-1.0.0/docs/Pipes-ByteString.html Pipes.ByteString>:+--+-- A 'Handle' can be associated with a 'Producer' or 'Consumer' according+-- as it is read or written to.+--+-- > import Pipes+-- > import qualified Pipes.Text as Text+-- > import qualified Pipes.Text.IO as Text+-- > import System.IO+-- >+-- > main =+-- > withFile "inFile.txt" ReadMode $ \hIn ->+-- > withFile "outFile.txt" WriteMode $ \hOut ->+-- > runEffect $ Text.fromHandle hIn >-> Text.toHandle hOut+--+-- To stream from files, the following is perhaps more Prelude-like (note that it uses Pipes.Safe):+--+-- > import Pipes+-- > import qualified Pipes.Text as Text+-- > import qualified Pipes.Text.IO as Text+-- > import Pipes.Safe+-- >+-- > main = runSafeT $ runEffect $ Text.readFile "inFile.txt" >-> Text.writeFile "outFile.txt"+--+-- Finally, you can stream to and from 'stdin' and 'stdout' using the predefined 'stdin'+-- and 'stdout' pipes, as with the following \"echo\" program:+--+-- > main = runEffect $ Text.stdin >-> Text.stdout+--+-- These programs, unlike the corresponding programs written with the line-based functions,+-- will pass along a 1 terabyte line without affecting memory use. -{-| Convert a 'IO.Handle' into a text stream using a text size - determined by the good sense of the text library. Note with the remarks - at the head of this module that this- is slower than @view utf8 (Pipes.ByteString.fromHandle h)@- but uses the system encoding and has other nice @Data.Text.IO@ features--}+-- $caveats+--+-- The operations exported here are a convenience, like the similar operations in+-- @Data.Text.IO@ (or rather, @Data.Text.Lazy.IO@, since, again, @Producer Text m r@ is+-- 'effectful text' and something like the pipes equivalent of lazy Text.)+--+-- * Like the functions in @Data.Text.IO@, they attempt to work with the system encoding.+--+-- * Like the functions in @Data.Text.IO@, they significantly slower than ByteString operations. Where+-- you know what encoding you are working with, use @Pipes.ByteString@ and @Pipes.Text.Encoding@ instead,+-- e.g. @view utf8 Bytes.stdin@ instead of @Text.stdin@+--+-- * Like the functions in @Data.Text.IO@ , they use Text exceptions, not the standard Pipes protocols. +-- | Convert a 'IO.Handle' into a text stream using a text size+-- determined by the good sense of the text library. Note with the remarks+-- at the head of this module that this+-- is slower than @view utf8 (Pipes.ByteString.fromHandle h)@+-- but uses the system encoding and has other nice @Data.Text.IO@ features fromHandle :: MonadIO m => IO.Handle -> Producer Text m ()-fromHandle h = go where- go = do txt <- liftIO (T.hGetChunk h)- if T.null txt then return ()- else do yield txt- go -{-# INLINABLE fromHandle#-}-+fromHandle h = go+ where+ go = do+ txt <- liftIO (T.hGetChunk h)+ if T.null txt+ then return ()+ else do+ yield txt+ go+ go+{-# INLINEABLE fromHandle #-} -- | Stream text from 'stdin' stdin :: MonadIO m => Producer Text m () stdin = fromHandle IO.stdin {-# INLINE stdin #-} --{-| Stream text from a file in the simple fashion of @Data.Text.IO@ -->>> runSafeT $ runEffect $ Text.readFile "hello.hs" >-> Text.map toUpper >-> hoist lift Text.stdout-MAIN = PUTSTRLN "HELLO WORLD"--}-+-- | Stream text from a file in the simple fashion of @Data.Text.IO@+--+-- >>> runSafeT $ runEffect $ Text.readFile "hello.hs" >-> Text.map toUpper >-> hoist lift Text.stdout+-- MAIN = PUTSTRLN "HELLO WORLD" readFile :: MonadSafe m => FilePath -> Producer Text m () readFile file = Safe.withFile file IO.ReadMode fromHandle {-# INLINE readFile #-} ---{-| Stream text to 'stdout'-- Unlike 'toHandle', 'stdout' gracefully terminates on a broken output pipe.-- Note: For best performance, it might be best just to use @(for source (liftIO . putStr))@ - instead of @(source >-> stdout)@ .--}+-- | Stream text to 'stdout'+--+-- Unlike 'toHandle', 'stdout' gracefully terminates on a broken output pipe.+--+-- Note: For best performance, it might be best just to use @(for source (liftIO . putStr))@+-- instead of @(source >-> stdout)@ . stdout :: MonadIO m => Consumer' Text m () stdout = go where go = do- txt <- await- x <- liftIO $ try (T.putStr txt)- case x of- Left (G.IOError { G.ioe_type = G.ResourceVanished- , G.ioe_errno = Just ioe })- | Errno ioe == ePIPE- -> return ()- Left e -> liftIO (throwIO e)- Right () -> go-{-# INLINABLE stdout #-}---{-| Convert a text stream into a 'Handle'+ txt <- await+ x <- liftIO $ try (T.putStr txt)+ case x of+ Left+ G.IOError+ { G.ioe_type = G.ResourceVanished,+ G.ioe_errno = Just ioe+ }+ | Errno ioe == ePIPE ->+ return ()+ Left e -> liftIO (throwIO e)+ Right () -> go+{-# INLINEABLE stdout #-} - Note: again, for best performance, where possible use - @(for source (liftIO . hPutStr handle))@ instead of @(source >-> toHandle handle)@.--}+-- | Convert a text stream into a 'Handle'+--+-- Note: again, for best performance, where possible use+-- @(for source (liftIO . hPutStr handle))@ instead of @(source >-> toHandle handle)@. toHandle :: MonadIO m => IO.Handle -> Consumer' Text m r toHandle h = for cat (liftIO . T.hPutStr h)-{-# INLINABLE toHandle #-}--+{-# INLINEABLE toHandle #-} -- | Stream text into a file. Uses @pipes-safe@. writeFile :: (MonadSafe m) => FilePath -> Consumer' Text m ()
Pipes/Text/Tutorial.hs view
@@ -1,21 +1,21 @@ {-# OPTIONS_GHC -fno-warn-unused-imports #-} -module Pipes.Text.Tutorial (- -- * Effectful Text+module Pipes.Text.Tutorial+ ( -- * Effectful Text -- $intro- + -- ** @Pipes.Text@ -- $pipestext- + -- ** @Pipes.Text.IO@ -- $pipestextio- + -- ** @Pipes.Text.Encoding@ -- $pipestextencoding- + -- ** Implicit chunking -- $chunks- + -- * Lenses -- $lenses @@ -27,321 +27,304 @@ -- ** @zoom@ -- $zoom- -- -- * Special types: @Producer Text m (Producer Text m r)@ and @FreeT (Producer Text m) m r@ -- $special- ) where+ )+where import Pipes import Pipes.Text-import Pipes.Text.IO import Pipes.Text.Encoding--{- $intro- This package provides @pipes@ utilities for /character streams/,- realized as streams of 'Text' chunks. The individual chunks are uniformly /strict/,- and thus the @Text@ type we are using is always the one from @Data.Text@, not @Data.Text.Lazy@ - The type @Producer Text m r@, as we are using it, is a sort of /pipes/ equivalent of - the lazy @Text@ type.--}--{- $pipestext- The main @Pipes.Text@ module provides many functions equivalent - in one way or another to the pure functions in- <https://hackage.haskell.org/package/text-1.1.0.0/docs/Data-Text-Lazy.html Data.Text.Lazy> - (and the corresponding @Prelude@ functions for @String@ s): they transform, - divide, group and fold text streams. Though @Producer Text m r@- is the type of \'effectful Text\', the functions in @Pipes.Text@ are \'pure\'- in the sense that they are uniformly monad-independent.--}--{- $pipestextencoding - In the @text@ library, @Data.Text.Lazy.Encoding@ - handles inter-operation with @Data.ByteString.Lazy@. Similarly here, @Pipes.Text.Encoding@ - provides for interoperation with the \'effectful ByteStrings\' of @Pipes.ByteString@.--}--{- $pipestextio- Simple /IO/ operations are defined in @Pipes.Text.IO@ - as lazy IO @Text@- operations are in @Data.Text.Lazy.IO@. There are also some simple line-based operations- in @Pipes.Prelude.Text@. The latter do not depend on the conception of effectful text- implemented elsewhere in this package, but just improve on the @stdinLn@ and @writeFile@ of- @Pipes.Prelude@ and @Pipes.Safe.Prelude@ by replacing 'String' with 'Text'--} ---{- $chunks- Remember that the @Text@ type exported by @Data.Text.Lazy@ is basically - that of a lazy list of strict @Text@: the implementation is arranged so that - the individual strict 'Text' chunks are kept to a reasonable size; the user - is not aware of the divisions between the connected 'Text' chunks, but uses- operations akin to those for strict text.- - So also here: the operations in @Pipes.Text@ are designed to operate on character streams that- in a way that is independent of the boundaries of the underlying @Text@ chunks. - This means that they may freely split text into smaller texts and /discard empty texts/. - The objective, though, is that they should not /concatenate texts/ in order to provide strict upper- bounds on memory usage even for indefinitely complex compositions.-- For example, to stream only the first three lines of 'stdin' to 'stdout' you- might write:--> import Pipes-> import qualified Pipes.Text as Text-> import qualified Pipes.Text.IO as Text-> import Pipes.Group (takes')-> import Lens.Family (view, over) -- or `Lens.Micro.Mtl` or `Control.Lens` or etc.->-> main = runEffect $ takeLines 3 Text.stdin >-> Text.stdout-> where -> takeLines n = view Text.unlines . takes' n . view Text.lines-> -- or equivalently: over Text.unlines (takes' n)-- This program will not bring more into memory than what @Text.stdin@ considers- one chunk of text (~ 32 KB), even if individual lines are split - across many chunks. The division into lines does not join Text fragments.---}---{- $lenses- As the use of @view@ in this example shows, one superficial difference from @Data.Text.Lazy@- is that many of the operations, like 'lines', are \'lensified\'; this has a- number of advantages; in particular it facilitates their use with 'Parser's of Text - (in the general <http://hackage.haskell.org/package/pipes-parse-3.0.1/docs/Pipes-Parse-Tutorial.html pipes-parse>- sense.) The remarks that follow in this section are for non-lens adepts.-- Each lens exported here, e.g. 'lines', 'chunksOf' or 'splitAt', reduces to the- intuitively corresponding function when used with @view@ or @(^.)@. Instead of- writing:-- > splitAt 17 producer-- as we would with the Prelude or Text functions called @splitAt@, we write-- > view (splitAt 17) producer-- or equivalently-- > producer ^. splitAt 17-- This may seem a little indirect, but note that many equivalents of- @Text -> Text@ functions are exported here as 'Pipe's. Here too we recover the intuitively- corresponding functions by prefixing them with @(>->)@. Thus something like--> stripLines = view Text.unlines . Group.maps (>-> Text.stripStart) . view Text.lines-- would drop the leading white space from each line. -- The lenses in this library are marked as /improper/; this just means that- they don't admit all the operations of an ideal lens, but only /getting/ and /focusing/.- Just for this reason, though, the magnificent complexities of the lens libraries- are a distraction. The lens combinators to keep in mind, the ones that make sense for- our lenses, are @view@, @over@, and @zoom@.-- One need only keep in mind that if @l@ is a @Lens' a b@, then the action of the - leading operations, @view@, @over@, and @zoom@ are as follows:---}-{- $view- @view l@ is a function @a -> b@ . Thus @view l a@ (also written @a ^. l@ )- is the corresponding @b@; as was said above, this function will typically be - the pipes equivalent of the function you think it is, given its name. So for example - - > view (Text.splitAt 300) :: Producer Text m r -> Producer Text (Producer Text m r)- > Text.stdin ^. splitAt 300 :: Producer Text IO (Producer Text IO r) - - I.e., it produces the first 300 characters, and returns the rest of the producer. - Thus to uppercase the first n characters- of a Producer, leaving the rest the same, we could write:-- > upper n p = do p' <- p ^. Text.splitAt n >-> Text.toUpper- > p'- - or equivalently:- - > upper n p = join (p ^. Text.splitAt n >-> Text.toUpper)- --}-{- $over- If @l@ is a @Lens a b@, @over l@ is a function @(b -> b) -> a -> a@. - Thus, given a function that modifies- @b@s, the lens lets us modify an @a@ by applying @f :: b -> b@ to- the @b@ that we \"see\" in the @a@ through the lens. - So the type of @over l f@ is @a -> a@ for the concrete type @a@- (it can also be written @l %~ f@).- For any particular @a@, then, @over l f a@ or @(l %~ f) a@ is a revised @a@.- So above we might have written things like these:-- > stripLines = over Text.lines (maps (>-> Text.stripStart))- > stripLines = Text.lines %~ maps (>-> Text.stripStart)- > upper n = Text.splitAt n %~ (>-> Text.toUpper)--}--{- $zoom- @zoom l@, finally, is a function from a @Parser b m r@- to a @Parser a m r@ (or more generally a @StateT (Producer b m x) m r@).- Its use is easiest to see with an decoding lens like 'utf8', which- \"sees\" a Text producer hidden inside a ByteString producer:- @drawChar@ is a Text parser, returning a @Maybe Char@, @zoom utf8 drawChar@ is- a /ByteString/ parser, returning a @Maybe Char@. @drawAll@ is a Parser that returns- a list of everything produced from a Producer, leaving only the return value; it would- usually be unreasonable to use it. But @zoom (splitAt 17) drawAll@- returns a list of Text chunks containing the first seventeen Chars, and returns the rest of- the Text Producer for further parsing. Suppose that we want, inexplicably, to- modify the casing of a Text Producer according to any instruction it might- contain at the start. Then we might write something like this:--> obey :: Monad m => Producer Text m b -> Producer Text m b-> obey p = do (ts, p') <- lift $ runStateT (zoom (Text.splitAt 7) drawAll) p-> let seven = T.concat ts-> case T.toUpper seven of-> "TOUPPER" -> p' >-> Text.toUpper-> "TOLOWER" -> p' >-> Text.toLower-> _ -> do yield seven-> p'---> -- > let doc = each ["toU","pperTh","is document.\n"]-> -- > runEffect $ obey doc >-> Text.stdout-> -- THIS DOCUMENT.-- The purpose of exporting lenses is the mental economy achieved with this three-way- applicability. That one expression, e.g. @lines@ or @splitAt 17@ can have these- three uses is no more surprising than that a pipe can act as a function modifying- the output of a producer, namely by using @>->@ to its left: @producer >-> pipe@- -- but can /also/ modify the inputs to a consumer by using @>->@ to its right:- @pipe >-> consumer@-- The three functions, @view@ \/ @(^.)@, @over@ \/ @(%~)@ and @zoom@ are supplied by- both <http://hackage.haskell.org/package/lens lens> and- <http://hackage.haskell.org/package/lens-family lens-family> The use of 'zoom' is explained- in <http://hackage.haskell.org/package/pipes-parse-3.0.1/docs/Pipes-Parse-Tutorial.html Pipes.Parse.Tutorial>- and to some extent in the @Pipes.Text.Encoding@ module here.---}--{- $special- The simple programs using the 'lines' lens reveal a more important difference from @Data.Text.Lazy@ .- This is in the types that are most closely associated with our central text type,- @Producer Text m r@. In @Data.Text@ and @Data.Text.Lazy@ we find functions like--> splitAt :: Int -> Text -> (Text, Text)-> lines :: Text -> [Text]-> chunksOf :: Int -> Text -> [Text]-- which relate a Text with a pair of Texts or a list of Texts.- The corresponding functions here (taking account of \'lensification\') are--> view . splitAt :: (Monad m, Integral n) => n -> Producer Text m r -> Producer Text m (Producer Text m r)-> view lines :: Monad m => Producer Text m r -> FreeT (Producer Text m) m r-> view . chunksOf :: (Monad m, Integral n) => n -> Producer Text m r -> FreeT (Producer Text m) m r-- Some of the types may be more readable if you imagine that we have introduced- our own type synonyms--> type Text m r = Producer T.Text m r-> type Texts m r = FreeT (Producer T.Text m) m r-- Then we would think of the types above as--> view . splitAt :: (Monad m, Integral n) => n -> Text m r -> Text m (Text m r)-> view lines :: (Monad m) => Text m r -> Texts m r-> view . chunksOf :: (Monad m, Integral n) => n -> Text m r -> Texts m r-- which brings one closer to the types of the similar functions in @Data.Text.Lazy@-- In the type @Producer Text m (Producer Text m r)@ the second- element of the \'pair\' of effectful Texts cannot simply be retrieved- with something like 'snd'. This is an \'effectful\' pair, and one must work- through the effects of the first element to arrive at the second Text stream, even- if you are proposing to throw the Text in the first element away.- Note that we use Control.Monad.join to fuse the pair back together, since it specializes to--> join :: Monad m => Producer Text m (Producer m r) -> Producer m r-- The return type of 'lines', 'words', 'chunksOf' and the other /splitter/ functions,- @FreeT (Producer m Text) m r@ -- our @Texts m r@ -- is the type of (effectful)- lists of (effectful) texts. The type @([Text],r)@ might be seen to gather- together things of the forms:--> r-> (Text,r)-> (Text, (Text, r))-> (Text, (Text, (Text, r)))-> (Text, (Text, (Text, (Text, r))))-> ...-- (We might also have identified the sum of those types with @Free ((,) Text) r@- -- or, more absurdly, @FreeT ((,) Text) Identity r@.)-- Similarly, our type @Texts m r@, or @FreeT (Text m) m r@ -- in fact called- @FreeT (Producer Text m) m r@ here -- encompasses all the members of the sequence:--> m r-> Text m r-> Text m (Text m r)-> Text m (Text m (Text m r))-> Text m (Text m (Text m (Text m r)))-> ...-- We might have used a more specialized type in place of @FreeT (Producer a m) m r@,- or indeed of @FreeT (Producer Text m) m r@, but it is clear that the correct- result type of 'lines' will be isomorphic to @FreeT (Producer Text m) m r@ .-- One might think that--> lines :: Monad m => Lens' (Producer Text m r) (FreeT (Producer Text m) m r)-> view . lines :: Monad m => Producer Text m r -> FreeT (Producer Text m) m r-- should really have the type--> lines :: Monad m => Pipe Text Text m r-- as e.g. 'toUpper' does. But this would spoil the control we are- attempting to maintain over the size of chunks. It is in fact just- as unreasonable to want such a pipe as to want--> Data.Text.Lazy.lines :: Text -> Text+import Pipes.Text.IO - to 'rechunk' the strict Text chunks inside the lazy Text to respect- line boundaries. In fact we have+-- $intro+-- This package provides @pipes@ utilities for /character streams/,+-- realized as streams of 'Text' chunks. The individual chunks are uniformly /strict/,+-- and thus the @Text@ type we are using is always the one from @Data.Text@, not @Data.Text.Lazy@+-- The type @Producer Text m r@, as we are using it, is a sort of /pipes/ equivalent of+-- the lazy @Text@ type. -> Data.Text.Lazy.lines :: Text -> [Text]-> Prelude.lines :: String -> [String]+-- $pipestext+-- The main @Pipes.Text@ module provides many functions equivalent+-- in one way or another to the pure functions in+-- <https://hackage.haskell.org/package/text-1.1.0.0/docs/Data-Text-Lazy.html Data.Text.Lazy>+-- (and the corresponding @Prelude@ functions for @String@ s): they transform,+-- divide, group and fold text streams. Though @Producer Text m r@+-- is the type of \'effectful Text\', the functions in @Pipes.Text@ are \'pure\'+-- in the sense that they are uniformly monad-independent. - where the elements of the list are themselves lazy Texts or Strings; the use- of @FreeT (Producer Text m) m r@ is simply the 'effectful' version of this.+-- $pipestextencoding+-- In the @text@ library, @Data.Text.Lazy.Encoding@+-- handles inter-operation with @Data.ByteString.Lazy@. Similarly here, @Pipes.Text.Encoding@+-- provides for interoperation with the \'effectful ByteStrings\' of @Pipes.ByteString@. - The @Pipes.Group@ module, which can generally be imported without qualification,- provides many functions for working with things of type @FreeT (Producer a m) m r@.- In particular it conveniently exports the constructors for @FreeT@ and the associated- @FreeF@ type -- a fancy form of @Either@, namely+-- $pipestextio+-- Simple /IO/ operations are defined in @Pipes.Text.IO@ - as lazy IO @Text@+-- operations are in @Data.Text.Lazy.IO@. There are also some simple line-based operations+-- in @Pipes.Prelude.Text@. The latter do not depend on the conception of effectful text+-- implemented elsewhere in this package, but just improve on the @stdinLn@ and @writeFile@ of+-- @Pipes.Prelude@ and @Pipes.Safe.Prelude@ by replacing 'String' with 'Text' -> data FreeF f a b = Pure a | Free (f b)+-- $chunks+-- Remember that the @Text@ type exported by @Data.Text.Lazy@ is basically+-- that of a lazy list of strict @Text@: the implementation is arranged so that+-- the individual strict 'Text' chunks are kept to a reasonable size; the user+-- is not aware of the divisions between the connected 'Text' chunks, but uses+-- operations akin to those for strict text.+--+-- So also here: the operations in @Pipes.Text@ are designed to operate on character streams that+-- in a way that is independent of the boundaries of the underlying @Text@ chunks.+-- This means that they may freely split text into smaller texts and /discard empty texts/.+-- The objective, though, is that they should not /concatenate texts/ in order to provide strict upper+-- bounds on memory usage even for indefinitely complex compositions.+--+-- For example, to stream only the first three lines of 'stdin' to 'stdout' you+-- might write:+--+-- > import Pipes+-- > import qualified Pipes.Text as Text+-- > import qualified Pipes.Text.IO as Text+-- > import Pipes.Group (takes')+-- > import Lens.Family (view, over) -- or `Lens.Micro.Mtl` or `Control.Lens` or etc.+-- >+-- > main = runEffect $ takeLines 3 Text.stdin >-> Text.stdout+-- > where+-- > takeLines n = view Text.unlines . takes' n . view Text.lines+-- > -- or equivalently: over Text.unlines (takes' n)+--+-- This program will not bring more into memory than what @Text.stdin@ considers+-- one chunk of text (~ 32 KB), even if individual lines are split+-- across many chunks. The division into lines does not join Text fragments. - for pattern-matching. Consider the implementation of the 'words' function, or- of the part of the lens that takes us to the words; it is compact but exhibits many- of the points under discussion, including explicit handling of the @FreeT@ and @FreeF@- constuctors. Keep in mind that+-- $lenses+-- As the use of @view@ in this example shows, one superficial difference from @Data.Text.Lazy@+-- is that many of the operations, like 'lines', are \'lensified\'; this has a+-- number of advantages; in particular it facilitates their use with 'Parser's of Text+-- (in the general <http://hackage.haskell.org/package/pipes-parse-3.0.1/docs/Pipes-Parse-Tutorial.html pipes-parse>+-- sense.) The remarks that follow in this section are for non-lens adepts.+--+-- Each lens exported here, e.g. 'lines', 'chunksOf' or 'splitAt', reduces to the+-- intuitively corresponding function when used with @view@ or @(^.)@. Instead of+-- writing:+--+-- > splitAt 17 producer+--+-- as we would with the Prelude or Text functions called @splitAt@, we write+--+-- > view (splitAt 17) producer+--+-- or equivalently+--+-- > producer ^. splitAt 17+--+-- This may seem a little indirect, but note that many equivalents of+-- @Text -> Text@ functions are exported here as 'Pipe's. Here too we recover the intuitively+-- corresponding functions by prefixing them with @(>->)@. Thus something like+--+-- > stripLines = view Text.unlines . Group.maps (>-> Text.stripStart) . view Text.lines+--+-- would drop the leading white space from each line.+--+-- The lenses in this library are marked as /improper/; this just means that+-- they don't admit all the operations of an ideal lens, but only /getting/ and /focusing/.+-- Just for this reason, though, the magnificent complexities of the lens libraries+-- are a distraction. The lens combinators to keep in mind, the ones that make sense for+-- our lenses, are @view@, @over@, and @zoom@.+--+-- One need only keep in mind that if @l@ is a @Lens' a b@, then the action of the+-- leading operations, @view@, @over@, and @zoom@ are as follows: -> newtype FreeT f m a = FreeT (m (FreeF f a (FreeT f m a)))-> next :: Monad m => Producer a m r -> m (Either r (a, Producer a m r))+-- $view+-- @view l@ is a function @a -> b@ . Thus @view l a@ (also written @a ^. l@ )+-- is the corresponding @b@; as was said above, this function will typically be+-- the pipes equivalent of the function you think it is, given its name. So for example+--+-- > view (Text.splitAt 300) :: Producer Text m r -> Producer Text (Producer Text m r)+-- > Text.stdin ^. splitAt 300 :: Producer Text IO (Producer Text IO r)+--+-- I.e., it produces the first 300 characters, and returns the rest of the producer.+-- Thus to uppercase the first n characters+-- of a Producer, leaving the rest the same, we could write:+--+-- > upper n p = do p' <- p ^. Text.splitAt n >-> Text.toUpper+-- > p'+--+-- or equivalently:+--+-- > upper n p = join (p ^. Text.splitAt n >-> Text.toUpper) - Thus the @do@ block after the @FreeT@ constructor is in the base monad, e.g. 'IO' or 'Identity';- the later subordinate block, opened by the @Free@ constructor, is in the @Producer@ monad:+-- $over+-- If @l@ is a @Lens a b@, @over l@ is a function @(b -> b) -> a -> a@.+-- Thus, given a function that modifies+-- @b@s, the lens lets us modify an @a@ by applying @f :: b -> b@ to+-- the @b@ that we \"see\" in the @a@ through the lens.+-- So the type of @over l f@ is @a -> a@ for the concrete type @a@+-- (it can also be written @l %~ f@).+-- For any particular @a@, then, @over l f a@ or @(l %~ f) a@ is a revised @a@.+-- So above we might have written things like these:+--+-- > stripLines = over Text.lines (maps (>-> Text.stripStart))+-- > stripLines = Text.lines %~ maps (>-> Text.stripStart)+-- > upper n = Text.splitAt n %~ (>-> Text.toUpper) -> words :: Monad m => Producer Text m r -> FreeT (Producer Text m) m r-> words p = FreeT $ do -- With 'next' we will inspect p's first chunk, excluding spaces;-> x <- next (p >-> dropWhile isSpace) -- note that 'dropWhile isSpace' is a pipe, and is thus *applied* with '>->'.-> return $ case x of -- We use 'return' and so need something of type 'FreeF (Text m) r (Texts m r)'-> Left r -> Pure r -- 'Left' means we got no Text chunk, but only the return value; so we are done.-> Right (txt, p') -> Free $ do -- If we get a chunk and the rest of the producer, p', we enter the 'Producer' monad-> p'' <- view (break isSpace) -- When we apply 'break isSpace', we get a Producer that returns a Producer;-> (yield txt >> p') -- so here we yield everything up to the next space, and get the rest back.-> return (words p'') -- We then carry on with the rest, which is likely to begin with space.+-- $zoom+-- @zoom l@, finally, is a function from a @Parser b m r@+-- to a @Parser a m r@ (or more generally a @StateT (Producer b m x) m r@).+-- Its use is easiest to see with an decoding lens like 'utf8', which+-- \"sees\" a Text producer hidden inside a ByteString producer:+-- @drawChar@ is a Text parser, returning a @Maybe Char@, @zoom utf8 drawChar@ is+-- a /ByteString/ parser, returning a @Maybe Char@. @drawAll@ is a Parser that returns+-- a list of everything produced from a Producer, leaving only the return value; it would+-- usually be unreasonable to use it. But @zoom (splitAt 17) drawAll@+-- returns a list of Text chunks containing the first seventeen Chars, and returns the rest of+-- the Text Producer for further parsing. Suppose that we want, inexplicably, to+-- modify the casing of a Text Producer according to any instruction it might+-- contain at the start. Then we might write something like this:+--+-- > obey :: Monad m => Producer Text m b -> Producer Text m b+-- > obey p = do (ts, p') <- lift $ runStateT (zoom (Text.splitAt 7) drawAll) p+-- > let seven = T.concat ts+-- > case T.toUpper seven of+-- > "TOUPPER" -> p' >-> Text.toUpper+-- > "TOLOWER" -> p' >-> Text.toLower+-- > _ -> do yield seven+-- > p'+--+--+-- > -- > let doc = each ["toU","pperTh","is document.\n"]+-- > -- > runEffect $ obey doc >-> Text.stdout+-- > -- THIS DOCUMENT.+--+-- The purpose of exporting lenses is the mental economy achieved with this three-way+-- applicability. That one expression, e.g. @lines@ or @splitAt 17@ can have these+-- three uses is no more surprising than that a pipe can act as a function modifying+-- the output of a producer, namely by using @>->@ to its left: @producer >-> pipe@+-- -- but can /also/ modify the inputs to a consumer by using @>->@ to its right:+-- @pipe >-> consumer@+--+-- The three functions, @view@ \/ @(^.)@, @over@ \/ @(%~)@ and @zoom@ are supplied by+-- both <http://hackage.haskell.org/package/lens lens> and+-- <http://hackage.haskell.org/package/lens-family lens-family> The use of 'zoom' is explained+-- in <http://hackage.haskell.org/package/pipes-parse-3.0.1/docs/Pipes-Parse-Tutorial.html Pipes.Parse.Tutorial>+-- and to some extent in the @Pipes.Text.Encoding@ module here. --}+-- $special+-- The simple programs using the 'lines' lens reveal a more important difference from @Data.Text.Lazy@ .+-- This is in the types that are most closely associated with our central text type,+-- @Producer Text m r@. In @Data.Text@ and @Data.Text.Lazy@ we find functions like+--+-- > splitAt :: Int -> Text -> (Text, Text)+-- > lines :: Text -> [Text]+-- > chunksOf :: Int -> Text -> [Text]+--+-- which relate a Text with a pair of Texts or a list of Texts.+-- The corresponding functions here (taking account of \'lensification\') are+--+-- > view . splitAt :: (Monad m, Integral n) => n -> Producer Text m r -> Producer Text m (Producer Text m r)+-- > view lines :: Monad m => Producer Text m r -> FreeT (Producer Text m) m r+-- > view . chunksOf :: (Monad m, Integral n) => n -> Producer Text m r -> FreeT (Producer Text m) m r+--+-- Some of the types may be more readable if you imagine that we have introduced+-- our own type synonyms+--+-- > type Text m r = Producer T.Text m r+-- > type Texts m r = FreeT (Producer T.Text m) m r+--+-- Then we would think of the types above as+--+-- > view . splitAt :: (Monad m, Integral n) => n -> Text m r -> Text m (Text m r)+-- > view lines :: (Monad m) => Text m r -> Texts m r+-- > view . chunksOf :: (Monad m, Integral n) => n -> Text m r -> Texts m r+--+-- which brings one closer to the types of the similar functions in @Data.Text.Lazy@+--+-- In the type @Producer Text m (Producer Text m r)@ the second+-- element of the \'pair\' of effectful Texts cannot simply be retrieved+-- with something like 'snd'. This is an \'effectful\' pair, and one must work+-- through the effects of the first element to arrive at the second Text stream, even+-- if you are proposing to throw the Text in the first element away.+-- Note that we use Control.Monad.join to fuse the pair back together, since it specializes to+--+-- > join :: Monad m => Producer Text m (Producer m r) -> Producer m r+--+-- The return type of 'lines', 'words', 'chunksOf' and the other /splitter/ functions,+-- @FreeT (Producer m Text) m r@ -- our @Texts m r@ -- is the type of (effectful)+-- lists of (effectful) texts. The type @([Text],r)@ might be seen to gather+-- together things of the forms:+--+-- > r+-- > (Text,r)+-- > (Text, (Text, r))+-- > (Text, (Text, (Text, r)))+-- > (Text, (Text, (Text, (Text, r))))+-- > ...+--+-- (We might also have identified the sum of those types with @Free ((,) Text) r@+-- -- or, more absurdly, @FreeT ((,) Text) Identity r@.)+--+-- Similarly, our type @Texts m r@, or @FreeT (Text m) m r@ -- in fact called+-- @FreeT (Producer Text m) m r@ here -- encompasses all the members of the sequence:+--+-- > m r+-- > Text m r+-- > Text m (Text m r)+-- > Text m (Text m (Text m r))+-- > Text m (Text m (Text m (Text m r)))+-- > ...+--+-- We might have used a more specialized type in place of @FreeT (Producer a m) m r@,+-- or indeed of @FreeT (Producer Text m) m r@, but it is clear that the correct+-- result type of 'lines' will be isomorphic to @FreeT (Producer Text m) m r@ .+--+-- One might think that+--+-- > lines :: Monad m => Lens' (Producer Text m r) (FreeT (Producer Text m) m r)+-- > view . lines :: Monad m => Producer Text m r -> FreeT (Producer Text m) m r+--+-- should really have the type+--+-- > lines :: Monad m => Pipe Text Text m r+--+-- as e.g. 'toUpper' does. But this would spoil the control we are+-- attempting to maintain over the size of chunks. It is in fact just+-- as unreasonable to want such a pipe as to want+--+-- > Data.Text.Lazy.lines :: Text -> Text+--+-- to 'rechunk' the strict Text chunks inside the lazy Text to respect+-- line boundaries. In fact we have+--+-- > Data.Text.Lazy.lines :: Text -> [Text]+-- > Prelude.lines :: String -> [String]+--+-- where the elements of the list are themselves lazy Texts or Strings; the use+-- of @FreeT (Producer Text m) m r@ is simply the 'effectful' version of this.+--+-- The @Pipes.Group@ module, which can generally be imported without qualification,+-- provides many functions for working with things of type @FreeT (Producer a m) m r@.+-- In particular it conveniently exports the constructors for @FreeT@ and the associated+-- @FreeF@ type -- a fancy form of @Either@, namely+--+-- > data FreeF f a b = Pure a | Free (f b)+--+-- for pattern-matching. Consider the implementation of the 'words' function, or+-- of the part of the lens that takes us to the words; it is compact but exhibits many+-- of the points under discussion, including explicit handling of the @FreeT@ and @FreeF@+-- constuctors. Keep in mind that+--+-- > newtype FreeT f m a = FreeT (m (FreeF f a (FreeT f m a)))+-- > next :: Monad m => Producer a m r -> m (Either r (a, Producer a m r))+--+-- Thus the @do@ block after the @FreeT@ constructor is in the base monad, e.g. 'IO' or 'Identity';+-- the later subordinate block, opened by the @Free@ constructor, is in the @Producer@ monad:+--+-- > words :: Monad m => Producer Text m r -> FreeT (Producer Text m) m r+-- > words p = FreeT $ do -- With 'next' we will inspect p's first chunk, excluding spaces;+-- > x <- next (p >-> dropWhile isSpace) -- note that 'dropWhile isSpace' is a pipe, and is thus *applied* with '>->'.+-- > return $ case x of -- We use 'return' and so need something of type 'FreeF (Text m) r (Texts m r)'+-- > Left r -> Pure r -- 'Left' means we got no Text chunk, but only the return value; so we are done.+-- > Right (txt, p') -> Free $ do -- If we get a chunk and the rest of the producer, p', we enter the 'Producer' monad+-- > p'' <- view (break isSpace) -- When we apply 'break isSpace', we get a Producer that returns a Producer;+-- > (yield txt >> p') -- so here we yield everything up to the next space, and get the rest back.+-- > return (words p'') -- We then carry on with the rest, which is likely to begin with space.
README.md view
@@ -1,10 +1,11 @@-pipes-text-==========+# pipes-text +[](https://github.com/pjones/pipes-text/actions)+[](https://github.com/pjones/pipes-text/releases)+[](https://hackage.haskell.org/package/pipes-text)+ This package follows the rule: pipes-text : pipes-bytestring :: text : bytestring -The division of three modules, `Pipes.Text` , `Pipes.Text.Encoding` and `Pipes.Text.IO` has more or less the significance it has in the `text` library. --Note that the module `Pipes.Text.IO` uses version 0.11.3 or later of the `text` library. (It thus works with the version of `text` that came with the 2013 Haskell Platform. To use an older `text`, install with the flag `-fnoio` +The division of three modules, `Pipes.Text` , `Pipes.Text.Encoding` and `Pipes.Text.IO` has more or less the significance it has in the `text` library.
Setup.hs view
@@ -1,2 +1,3 @@ import Distribution.Simple+ main = defaultMain
− changelog
@@ -1,49 +0,0 @@-# Version 0.0.0.12--* Opposing lenses for `lines` and `unlines` and `words` and `unwords`. - Brought closer in line with `pipes-bytestring` again. Removed `count`, which- was wrong. Scrapped `Iso` and the `profunctors` dependency. --# Version 0.0.0.11--* Updated to use streaming-commons in place of text-stream-decoding.--# Version 0.0.0.10--* Documentation changes.---# Version 0.0.0.9--* Documentation changes.--# Version 0.0.0.7-- * Used the new text-stream-decoding package- * Separated IO and Encoding modules adding flag -fnoio-- # Version 0.0.0.5- - * Rearranged internal modules--- # Version 0.0.0.4-- * Altered bad haddock markup--- # Version 0.0.0.3- - * Actually added changelog--- # Version 0.0.0.2-- * Omit `stdinLn` as likely to be dangerous through misunderstanding.--- # Version 0.0.0.1-- * Rearrange order of 'Internal' materials.--
pipes-text.cabal view
@@ -1,59 +1,101 @@-name: pipes-text-version: 0.0.2.5-synopsis: properly streaming text-description: /New in version 0.0.2.x/: The new module @Pipes.Prelude.Text@ exports line-based @Text@ producers and consumers as a drop-in replacement for the @String@ material in @Pipes.Prelude@ and @Pipes.Safe.Prelude@. They can be used as one uses @Pipes.Prelude@ without reference to the rest of this package. See the caveats in the documentation for that module.- .- The organization of this package follows the rule:- .- * @pipes-text : pipes-bytestring :: text : bytestring@ - .- Familiarity with the other three packages should give one an idea what to expect where. The package has three principal modules, @Pipes.Text@ , @Pipes.Text.Encoding@ and @Pipes.Text.IO@; the division has more or less the significance it has in the @text@ library.- .- The module @Pipes.Text.IO@ is present as a convenience. Official pipes IO uses @Pipes.ByteString@ together with the bytestring decoding functions in @Pipes.Text.Encoding@. In particular, the @Pipes.Text.IO@ functions use Text exceptions, while @Pipes.Text@ uses the standard pipes practice of breaking with a failed parse. Thus, for example, the type of @decodeUtf8@ is- .- * @decodeUtf8 :: Monad m => Producer ByteString m r -> Producer Text m (Producer ByteString m r)@- .- where any unparsed bytes are returned.- .- @Pipes.Text.IO@ and @Pipes.Prelude.Text@ use version 0.11.3 or later of the @text@ library; older versions of @text@ can be used with the flag @-fnoio@--+cabal-version: 2.2+name: pipes-text+version: 1.0.0+synopsis: properly streaming text+description:+ The organization of this package follows the rule:+ .+ * @pipes-text : pipes-bytestring :: text : bytestring@+ .+ Familiarity with the other three packages should give one an idea+ what to expect here. The package has three principal modules,+ @Pipes.Text@ , @Pipes.Text.Encoding@ and @Pipes.Text.IO@; the+ division has more or less the significance it has in the @text@+ library.+ .+ The module @Pipes.Text.IO@ is present as a convenience. Official+ pipes IO uses @Pipes.ByteString@ together with the bytestring+ decoding functions in @Pipes.Text.Encoding@. In particular, the+ @Pipes.Text.IO@ functions use Text exceptions, while @Pipes.Text@+ uses the standard pipes practice of breaking with a failed+ parse. Thus, for example, the type of @decodeUtf8@ is+ .+ * @decodeUtf8 :: Monad m => Producer ByteString m r -> Producer Text m (Producer ByteString m r)@+ .+ where any unparsed bytes are returned. -homepage: https://github.com/michaelt/text-pipes-bug-reports: https://github.com/michaelt/text-pipes/issues-license: BSD3-license-file: LICENSE-author: Michael Thompson-maintainer: what_is_it_to_do_anything@yahoo.com-category: Text, Pipes-build-type: Simple-cabal-version: >=1.10+homepage: https://github.com/pjones/pipes-text+bug-reports: https://github.com/pjones/pipes-text/issues+license: BSD-3-Clause+license-file: LICENSE+author: Michael Thompson+maintainer: Peter Jones <pjones@devalot.com>+category: Text, Pipes+build-type: Simple+extra-source-files:+ CHANGES.md+ README.md -extra-source-files: README.md changelog source-repository head- type: git- location: https://github.com/michaelt/text-pipes+ type: git+ location: https://github.com/pjones/pipes-text flag noio- default: False- Description: Use a version of text earlier than 0.11.3+ default: False+ description: Use a version of text earlier than 0.11.3 -library- exposed-modules: Pipes.Text, Pipes.Text.Encoding- build-depends: base >= 4 && < 5 ,- bytestring >= 0.9.2.1 && < 0.11,- text >= 0.11.2 && < 1.3 ,- streaming-commons >= 0.1 && < 0.2 , - pipes >= 4.0 && < 4.4 ,- pipes-group >= 1.0.0 && < 1.1 ,- pipes-parse >= 3.0.0 && < 3.1 ,- pipes-safe >= 2.1 && < 2.3 , - pipes-bytestring >= 1.0 && < 2.2 ,- transformers >= 0.2.0.0 && < 0.6+flag maintainer+ description: Enable settings for the package maintainer.+ manual: True+ default: False - other-extensions: RankNTypes- default-language: Haskell2010+common dependencies+ build-depends:+ , base >=4 && <5+ , bytestring >=0.9.2.1 && <0.12+ , pipes >=4.0 && <4.4+ , pipes-bytestring >=1.0 && <2.2+ , pipes-group ^>=1.0.0+ , pipes-parse ^>=3.0.0+ , pipes-safe >=2.1 && <2.4+ , streaming-commons >=0.1 && <0.3+ , text >=0.11.2 && <1.3+ , transformers >=0.2.0.0 && <0.6 +common options+ default-language: Haskell2010+ other-extensions: RankNTypes+ ghc-options:+ -Wall -Wno-name-shadowing -Werror=incomplete-record-updates+ -Werror=incomplete-uni-patterns -Werror=missing-home-modules+ -Widentities -Wmissing-export-lists -Wredundant-constraints++ if flag(maintainer)+ ghc-options: -Werror++library+ import: options, dependencies+ exposed-modules:+ Pipes.Text+ Pipes.Text.Encoding+ if !flag(noio)- exposed-modules: Pipes.Text.IO, Pipes.Text.Tutorial, Pipes.Prelude.Text- build-depends: text >=0.11.3 && < 1.3+ exposed-modules:+ Pipes.Prelude.Text+ Pipes.Text.IO+ Pipes.Text.Tutorial++ build-depends: text >=0.11.3 && <1.3++-- test-suite test+-- import: options, dependencies+-- type: exitcode-stdio-1.0+-- hs-source-dirs: test+-- main-is: Test.hs+-- build-depends:+-- , pipes-text+-- , QuickCheck ^>=2.13+-- , test-framework ^>=0.8+-- , test-framework-quickcheck2 ^>=0.3+--+-- other-modules: Utils