# tokenizer-streaming
**Motivation**: You might have stumpled upon the package tokenizer-monad. It
is another project by me, for writing tokenizers that act on pure
text/strings. However, there are situations when you cannot keep all the text
in memory. You might want to tokenize text from network streams or from large
corpus files.
**Main idea**: A monad transformer called `TokenizerT` implements exactly the
same methods as `Tokenizer` from tokenizer-monad, such that all tokenizers can
be ported without code changes (if you used `MonadTokenizer` in the type signatures)
## Supported text types
- streams of Char lists can be tokenized into streams of Char lists
- streams of strict Text can be tokenized into streams of strict Text
- streams of lazy Text can be tokenized into streams of lazy Text
- streams of strict ASCII ByteStrings can be tokenized into streams of strict ASCII ByteStrings
- streams of lazy ASCII ByteStrings can be tokenized into streams of lazy ASCII ByteStrings
- bytestring streams (from streaming-bytestring) with Unicode encodings (UTF-8, UTF-16 LE & BE, UTF-32 LE & BE) can be tokenized into streams of strict Text