# conduit-tokenize-attoparsec
Conduits for tokenizing streams.
`conduit-tokenize-attoparsec` is an extension to the `conduit` library based on
`Data.Conduit.Attoparsec` that allows the tokenization of a stream whilst also
tracking arbitrary state. The library includes in-built support for tracking
such as line-column and stream offset, but can be extended with type-classes to
support alternative state tracking.
For an example, see [`app/Main.hs`](../master/app/Main.hs)
## Building
Prerequisites: `haskell-stack` is installed.
Run the following in the shell:
git clone git@github.com:haskell-works/conduit-tokenize-attoparsec.git
cd conduit-tokenize-attoparsec
stack setup
stack build
stack test
stack ghci --ghc-options -XOverloadedStrings \
--main-is conduit-tokenize-attoparsec:exe:conduit-tokenize-attoparsec-example