xcodec (empty) → 1.0.0.0

raw patch · 7 files changed

+578/−0 lines, 7 filesdep +arraydep +basedep +bytestring

Dependencies added: array, base, bytestring, containers, text

Files

• CHANGELOG.md +76/−0
• LICENSE.txt +28/−0
• README.md +109/−0
• xcodec.cabal +70/−0
• xcodec/Data/XCodec/BinaryTranscoder.hs +238/−0
• xcodec/Data/XCodec/Internal.hs +23/−0
• xcodec/Data/XCodec/StreamTranscoder.hs +34/−0

@@ -0,0 +1,76 @@+[`xcodec`](http://hackage.haskell.org/package/xcodec) change log:+=================================================================++Major release 1.0+-----------------++## Version 1.0.0.0 (05-21-2026)++Introducing this package to hackage.org! This module provides instances of+`BinaryTranscoder`, a type-class for generically working on binary data formats,+and `StreamTranscoder` a type-class interface for designing encoders and+decoders on top of `BinaryTranscoder`.+++### Module `Data.XCodec.BinaryTranscoder`++This module provides the type-class for the `BinaryTranscoder`. It also provides+default instances for `ByteString`, `LazyByteString`, and `ShortByteString` from+the GHC standard library. The minimal definition for a `BinaryTranscoder` is+currently with the following methods (where `bxc` is the type-parameter deriving+`BinaryTranscoder`):++```haskell+-- | Gives back the length in bytes of the Transcoder data.+lengthBytes :: bxc -> Int+-- | Unpacks transcoder data as BitSet, ie, an infinite bitarray.+unpackValue :: bxc -> BitSet+-- | Places the contents of the transcoder into a ByteString Builder.+serializeValue :: bxc -> Builder+-- | Packs Word8 values into transcoder data `bxc`.+packBytes :: [Word8] -> bxc+-- | Unpacks transcoder data into a list of bytes.+unpackBytes :: bxc -> [Word8]+-- | Prepends a byte value to the top of the transcoder data.+pushByte :: bxc -> Word8 -> bxc+-- | Appends the byte value to the end of the transcoder data.+pushByteEnd :: bxc -> Word8 -> bxc+-- | Takes a subsection of bytes from the beginning of the transcoder data.+takeBytes :: Int -> bxc -> bxc+-- | Removes bytes from the beginning of transcoder data.+dropBytes :: Int -> bxc -> bxc+-- | Removes bytes from the end of transcoder data.+takeBytesEnd :: Int -> bxc -> bxc+-- | Removes bytes from the end of transcoder data.+dropBytesEnd :: Int -> bxc -> bxc+-- | Parititions the binary data at the nth byte.+splitAtByte :: Int -> bxc -> (bxc, bxc)+-- | Repeats a byte value in binary data.+replicateByte :: Int -> Word8 -> bxc+-- | Splits byte values based on the first false result from the predicate.+spanBytes :: (Word8 -> Bool) -> bxc -> (bxc, bxc)+-- | Splits the list while the element predicate holds staring from the left+-- side.+spanBytesEnd :: (Word8 -> Bool) -> bxc -> (bxc, bxc)+```++### Module `Data.XCodec.StreamTranscoder`++This module exports the type-class and methods for `StreamTranscoder`. It+provides no default instances, and is a way for the programmer to implement+generic decoders and decoders on streams of data from deriving from+the `BinaryTranscoder` class of types. It defines the follwing method+signatures:++```haskell+-- | Generic Encoder signature+type Encoder xtype ytype = xtype -> ytype++-- | Generic Decoder signature+type Decoder xtype ytype = ytype -> Maybe xtype++-- | StreamTranscoder give a way to make a generic codec methods.+class (BinaryTranscoder xtype) => StreamTranscoder codec xtype ytype where+  streamEncoder :: codec -> Encoder xtype ytype+  streamDecoder :: codec -> Decoder xtype ytype+```

@@ -0,0 +1,28 @@+BSD 3-Clause License++Copyright (c) 2026, Zoey McBride <zoeymcbride@mailbox.org>++Redistribution and use in source and binary forms, with or without+modification, are permitted provided that the following conditions are met:++1. Redistributions of source code must retain the above copyright notice, this+   list of conditions and the following disclaimer.++2. Redistributions in binary form must reproduce the above copyright notice,+   this list of conditions and the following disclaimer in the documentation+   and/or other materials provided with the distribution.++3. Neither the name of the copyright holder nor the names of its+   contributors may be used to endorse or promote products derived from+   this software without specific prior written permission.++THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

@@ -0,0 +1,109 @@+xcodec+------++<!-- TODO: replace header with this to get badges+xcodec [![hackage.haskell.org](+https://img.shields.io/badge/hackage%2Ehaskell%2Eorg-5C5181?logo=haskell)](+https://hackage.haskell.org/package/xcodec)[![builds.sr.ht status](+https://builds.sr.ht/~z0/xcodec/commits/main/xcodec-testing.yml.svg)](+https://builds.sr.ht/~z0/xcodec/commits/main/xcodec-testing.yml?)+-----------------------------------------------------------------+-->++The `XCodec` Haskell library provides a type-class for generic programming on+bit data for writing encoders and decoders for codecs. The `BinaryTranscoder`+class provides a common interface of methods for processing binary data, and+default instances for the `bytestring` types: `ByteString`, `ShortByteString`,+and `LazyByteString`. The `StreamTranscoder` class then provides the abstract+methods for a generic function encoding from and a generic function decoding to+the types implementing `BinaryTranscoder`.++Why?+----++`XCodec` exists to abstract common patterns that arise when writting code with+the `bytestring` library in Haskell:+- Reusing code for `ShortByteString`, `ByteString` and `LazyByteString`.+- Having common interface for transfering into the bytestring `Builder` type.+- Reading to and from numeric values for bitwise manipulation or initializing+from constant values.++Making bytestring code generic, can also make code more memory efficient; for+example, we can write functions for `BinaryTranscoder` and apply them to+`LazyByteString` when reading large files and to `ShortByteString` when working+on smaller internal structures.++Examples+--------++Using `XCodec` we can easily read numeric bit data to binary formats,+programmatically:++```haskell+import Data.ByteString (ByteString)+import Data.ByteString.Short (ShortByteString)+import Data.ByteString.Lazy (LazyByteString)+import Data.Word (Word16)+import Data.XCodec.BinaryTranscoder (packValue)++-- Infers: packValue :: Int -> ByteString+exStrict :: ByteString+exStrict = packValue (0xdeadbeef :: Int)++-- Infers: packValue :: Integer -> LazyByteString+exLazy :: LazyByteString+exLazy = packValue (0xf000000000000000000000000000000000000000000d :: Integer)++-- Infers: packValue :: Word16 -> ShortByteString+exShort :: ShortByteString+exShort = packValue (0xcafe :: Word16)+```++And then serialize them through a common interface:++```haskell+import Data.ByteString.Builder (toLazyByteString, string8)+import Data.XCodec.BinaryTranscoder (serializeValue)++deadbeefThreeTimes :: LazyByteString+deadbeefThreeTimes =+  toLazyByteString . mconcat $+    [ serializeValue exLazy,+      serializeValue exStrict,+      serializeValue exShort,+      string8 "Hello, World!"+    ]+```++It also enables extracting binary data into `Integer` format so that bitwise+transformations can be performed on large sets of binary data in *O(n)* time:++```haskell+import Data.Bits ((.<<.), (.&.))+import Data.Word (Word8)+import Data.XCodec.BinaryTranscoder (unpackValue)++-- Extracts the byte at the bit number (LSB=0).+byteWindowAt :: (BinaryTranscoder bxc) => bxc -> Int -> Word8+byteWindowAt bxc idx = fromIntegral $ (unpackValue bxc .>>. idx) .&. 0xFF+```++<!-- TODO: add unit testing for xcodec++Development+-----------++Unit tests are provided on the main [ipfshs repo](https://git.sr.ht/~z0/ipfshs),+and bugs can be reported on [ipfshs ticket tracker](+https://todo.sr.ht/~z0/ipfshs).++-->++Licensing+---------++The `xcodec` project and its modules are free software and licensed under the+BSD 3-clause license. See [LICENSE.txt](LICENSE.txt).++Copyright © 2026 Zoey McBride | [zoeymcbride@mailbox.org](+mailto:zoeymcbride@mailbox.org)

@@ -0,0 +1,70 @@+cabal-version:   3.0+version:         1.0.0.0+name:            xcodec+build-type:      Simple+author:          Zoey McBride+maintainer:      zoeymcbride@mailbox.org+license:         BSD-3-Clause+license-file:    LICENSE.txt+extra-doc-files: README.md, CHANGELOG.md+category:        Numeric, Data, Serialization+synopsis:        Classes for working with generically typed codecs.+description:+    This module provides a generic interface for working with binary codecs. It+    provides a type-class for manipulating bit data and instances for the+    bytestring package types (ByteString, ShortByteString, and LazyByteString).+    Another type-class is also provided for derivided type-class for importing+    modules to write generic encoders and decoders.+tested-with:+    GHC == 9.6.7++source-repository head {+    type:     git+    location: https://git.sr.ht/~z0/xcodec+}++-- | Sets flag for CI build toggle+flag CI {+    Description: CI Build options+    Default: False+    Manual: True+}++common HaskellConfig {+    hs-source-dirs: xcodec+    default-extensions: StrictData+    default-language: GHC2021+}++common Depends {+    build-depends:+        base >= 4.18 && < 5,+        array ^>= 0.5,+        bytestring >= 0.12 && < 0.13,+        containers >= 0.7 && < 8,+        text >=2.1 && < 2.2+}++common BuildUser {+    import: Depends+    ghc-options: -Wall -Wextra -Wno-unused-top-binds+}++common BuildCI {+    import: Depends+    ghc-options: -Wall -Wextra -Werror -Wno-unused-top-binds+}++library {+    import: HaskellConfig+    if flag(CI) { import: BuildCI }+    else { import: BuildUser }+    exposed-modules:+        -- * Type-class for transcoding binary data.+        Data.XCodec.BinaryTranscoder+        -- * Type-class for transcoding streams derivided from BinaryTranscoder+        Data.XCodec.StreamTranscoder+    other-modules:+        -- * Helper functions+        Data.XCodec.Internal+}

@@ -0,0 +1,238 @@+-- | Module      : Data.XCodec.BinaryTranscoder+--   Description : Type-class for building generic codecs on binary data.+--   Copyright   : Zoey McBride (c) 2026+--   License     : BSD-3-Clause+--   Maintainer  : zoeymcbride@mailbox.org+--   Stability   : experimental+module Data.XCodec.BinaryTranscoder+  ( -- * Interface for generic transcoding of binary data.+    BinaryTranscoder (..),++    -- * Converts streams of bytes into editable bits.+    BitSet,+  )+where++import Data.Bits (Bits, (.&.), (.<<.), (.>>.), (.|.))+import Data.ByteString (ByteString)+import Data.ByteString qualified as Bytes+import Data.ByteString.Builder (Builder)+import Data.ByteString.Builder qualified as Builder+import Data.ByteString.Lazy (LazyByteString)+import Data.ByteString.Lazy qualified as LBS+import Data.ByteString.Short (ShortByteString)+import Data.ByteString.Short qualified as SBS+import Data.Word (Word8)+import Data.XCodec.Internal (iterateInit, replaceNull)++-- | Storage for data with bitwise operations.+type BitSet = Integer++-- | Class of functions for working on binary transcoder streams, that is+-- streams of bytes that we can easily turn into a bitset, and can partition+-- into smaller streams for extracting and parsing data.+class (Monoid bxc) => BinaryTranscoder bxc where+  -- This defines basic methods that all BinaryTranscoders share.+  {-# MINIMAL+    lengthBytes,+    unpackValue,+    serializeValue,+    packBytes,+    unpackBytes,+    pushByte,+    pushByteEnd,+    takeBytes,+    takeBytesEnd,+    dropBytes,+    dropBytesEnd,+    splitAtByte,+    spanBytes,+    spanBytesEnd,+    replicateByte+    #-}++  -- | Gives back the length in bytes of the Transcoder data.+  lengthBytes :: bxc -> Int++  -- | Unpacks transcoder data as BitSet, ie, an infinite bitarray.+  unpackValue :: bxc -> BitSet++  -- | Places the contents of the transcoder into a ByteString Builder.+  serializeValue :: bxc -> Builder++  -- | Packs Word8 values into transcoder data `bxc`.+  packBytes :: [Word8] -> bxc++  -- | Unpacks transcoder data into a list of bytes.+  unpackBytes :: bxc -> [Word8]++  -- | Prepends a byte value to the top of the transcoder data.+  pushByte :: bxc -> Word8 -> bxc++  -- | Appends the byte value to the end of the transcoder data.+  pushByteEnd :: bxc -> Word8 -> bxc++  -- | Takes a subsection of bytes from the beginning of the transcoder data.+  takeBytes :: Int -> bxc -> bxc++  -- | Removes bytes from the beginning of transcoder data.+  dropBytes :: Int -> bxc -> bxc++  -- | Removes bytes from the end of transcoder data.+  takeBytesEnd :: Int -> bxc -> bxc++  -- | Removes bytes from the end of transcoder data.+  dropBytesEnd :: Int -> bxc -> bxc++  -- | Parititions the binary data at the nth byte.+  splitAtByte :: Int -> bxc -> (bxc, bxc)++  -- | Repeats a byte value in binary data.+  replicateByte :: Int -> Word8 -> bxc++  -- | Splits byte values based on the first false result from the predicate.+  spanBytes :: (Word8 -> Bool) -> bxc -> (bxc, bxc)++  -- | Implements `takeWhile` for byte data in binary transcoder.+  takeWhileBytes :: (Word8 -> Bool) -> bxc -> bxc+  takeWhileBytes f = fst . spanBytesEnd f++  -- | Implements `dropWhile` for byte data in binary transcoder.+  dropWhileBytes :: (Word8 -> Bool) -> bxc -> bxc+  dropWhileBytes f = snd . spanBytesEnd f++  -- | Splits the list while the element predicate holds staring from the left+  -- side.+  spanBytesEnd :: (Word8 -> Bool) -> bxc -> (bxc, bxc)++  -- | Implements `takeWhileEnd` for byte data in binary transcoder.+  takeWhileBytesEnd :: (Word8 -> Bool) -> bxc -> bxc+  takeWhileBytesEnd f = snd . spanBytesEnd f++  -- | Implements `dropWhileEnd` for byte data in binary transcoder.+  dropWhileBytesEnd :: (Word8 -> Bool) -> bxc -> bxc+  dropWhileBytesEnd f = fst . spanBytesEnd f++  -- | Represents empty binary transcoder data.+  empty :: bxc+  empty = mempty++  -- | Appends the contents of two binary transcoders.+  append :: bxc -> bxc -> bxc+  append = mappend++  -- | Creates a binary transcoder from a single byte.+  {-# INLINE singleton #-}+  singleton :: Word8 -> bxc+  singleton w8 = packBytes [w8]++  -- | Returns True if the binary transcoder data is empty.+  {-# INLINE null #-}+  null :: bxc -> Bool+  null bxc = lengthBytes bxc == 0++  -- | Returns the first byte removed from the transcoder data, if available.+  {-# INLINE uncons #-}+  uncons :: bxc -> Maybe (Word8, bxc)+  uncons bxc =+    let (start, rest) = splitAtByte 1 bxc+     in case unpackBytes start of+          [top] -> Just (top, rest)+          _ -> Nothing++  -- | Returns the last byte removed from the transcoder data, if available.+  {-# INLINE usnoc #-}+  usnoc :: bxc -> Maybe (bxc, Word8)+  usnoc bxc =+    let offset = lengthBytes bxc - 1+        (rest, taken) = splitAtByte offset bxc+     in case unpackBytes taken of+          [final] -> Just (rest, final)+          _ -> Nothing++  -- | Packs a bit string value into some bytes data `bxc`.+  packValue :: (Integral b, Bits b) => b -> bxc+  packValue intdata =+    -- This code can be optimized; if we precalculate the length of the BitSet+    -- in bits we can mask from the other direction using left shifts, and+    -- remove the call to reverse, maybe we can even use a list comprehension to+    -- build in place for Bytes.packBytes+    packBytes+      . replaceNull [0]+      . reverse+      . map (\(_, byte) -> fromIntegral byte)+      . takeWhile (\(rest, byte) -> rest > 0 || byte > 0)+      $ iterateInit (\(input, _) -> extractByte input) (,0) intdata+    where+      -- Gets the byte from LSB in input and shifts the value 8 bits.+      extractByte input = (input .>>. 8, input .&. 0xFF)++-- | Implements binary decoder for Bytestrings, best for constant values that+-- exist for the lifetime of the program.+-- https://hackage-content.haskell.org/package/bytestring-0.12.2.0/docs/Data-ByteString.html#g:2+instance BinaryTranscoder ByteString where+  lengthBytes = Bytes.length+  packBytes = Bytes.pack+  serializeValue = Builder.byteString+  unpackBytes = Bytes.unpack+  pushByte = flip Bytes.cons+  pushByteEnd = Bytes.snoc+  takeBytes = Bytes.take+  takeBytesEnd = Bytes.takeEnd+  dropBytes = Bytes.drop+  dropBytesEnd = Bytes.dropEnd+  spanBytes = Bytes.span+  spanBytesEnd = Bytes.spanEnd+  splitAtByte = Bytes.splitAt+  replicateByte k = Bytes.replicate $ fromIntegral k+  unpackValue = unpackValue . LBS.fromStrict++-- TODO: we can improve the proformance of unpackBits by examining the current+-- bytestring, and loading a range of 1-8 bytes to OR simulaneously++-- | Implements binary decoder for LazyByteString, best for very large data sets+-- that can be loaded in and out of memory on demand.+-- https://hackage-content.haskell.org/package/bytestring-0.12.2.0/docs/Data-ByteString-Lazy.html+instance BinaryTranscoder LazyByteString where+  lengthBytes = fromIntegral . LBS.length+  packBytes = LBS.pack+  unpackBytes = LBS.unpack+  serializeValue = Builder.lazyByteString+  pushByte = flip LBS.cons+  pushByteEnd = LBS.snoc+  takeBytes k = LBS.take $ fromIntegral k+  takeBytesEnd k = LBS.takeEnd $ fromIntegral k+  dropBytes k = LBS.drop $ fromIntegral k+  dropBytesEnd k = LBS.dropEnd $ fromIntegral k+  spanBytes = LBS.span+  spanBytesEnd = LBS.spanEnd+  splitAtByte k = LBS.splitAt $ fromIntegral k+  replicateByte k = LBS.replicate $ fromIntegral k+  unpackValue =+    -- From LSB to MSB, OR the current byte and shift the total bits.+    LBS.foldl' (\bits byte -> (bits .<<. 8) .|. fromIntegral byte) 0+      . LBS.dropWhile (== 0)++-- | Implements binary decoder for ShortByteString, best for compact data that+-- doesn't exist long in memory. Notably, it packs better in memory than normal+-- ByteString (prone to Heap Fragmentation).+-- https://hackage-content.haskell.org/package/bytestring-0.12.2.0/docs/Data-ByteString-Short.html#g:1+instance BinaryTranscoder ShortByteString where+  lengthBytes = SBS.length+  packBytes = SBS.pack+  unpackBytes = SBS.unpack+  serializeValue = Builder.shortByteString+  pushByte = flip SBS.cons+  pushByteEnd = SBS.snoc+  takeBytes = SBS.take+  takeBytesEnd = SBS.takeEnd+  dropBytes = SBS.drop+  dropBytesEnd = SBS.dropEnd+  spanBytes = SBS.span+  spanBytesEnd = SBS.spanEnd+  splitAtByte = SBS.splitAt+  replicateByte k = SBS.replicate $ fromIntegral k+  unpackValue =+    -- From LSB to MSB, OR the current byte and shift the total bits.+    SBS.foldl' (\bits byte -> (bits .<<. 8) .|. fromIntegral byte) 0+      . SBS.dropWhile (== 0)

@@ -0,0 +1,23 @@+-- | Module      : Data.XCodec.Internal+--   Description : Utility functions for lists.+--   Copyright   : Zoey McBride (c) 2026+--   License     : BSD-3-Clause+--   Maintainer  : zoeymcbride@mailbox.org+--   Stability   : experimental+module Data.XCodec.Internal+  ( -- * List utilities+    replaceNull,+    iterateInit,+  )+where++-- | Return the the first param if the second param is empty.+{-# INLINE replaceNull #-}+replaceNull :: [a] -> [a] -> [a]+replaceNull replace xs+  | null xs = replace+  | otherwise = xs++-- | Wraps iterate to compose with an intermediary type constructor.+iterateInit :: (b -> b) -> (a -> b) -> a -> [b]+iterateInit f iterinit = drop 1 . iterate f . iterinit

@@ -0,0 +1,34 @@+-- | Module      : Data.XCodec.StreamTranscoder+--   Description : Type-class for encoding and decoding binary streams.+--   Copyright   : Zoey McBride (c) 2026+--   License     : BSD-3-Clause+--   Maintainer  : zoeymcbride@mailbox.org+--   Stability   : experimental+module Data.XCodec.StreamTranscoder+  ( -- * Interface for implementing generic digit symbol encoder/decoders.+    StreamTranscoder (..),++    -- * Type-functions for generating generic StreamTranscoder instances.+    Encoder,+    Decoder,+  )+where++import Data.XCodec.BinaryTranscoder (BinaryTranscoder)++-- | Type of function for transforming some `xtype` to some `ytype`.+type Encoder xtype ytype = xtype -> ytype++-- | Type of function for transforming some `ytype` into an `xtype` if the+-- `ytype` is well formed.+type Decoder xtype ytype = ytype -> Maybe xtype++-- | Type-class of `codec`s for which `streamEncoder` produces a representation+-- of an `xtype` through some `ytype` and which `streamDecoder` attempts to form+-- a `xtype` only from well-formed values in `ytype`.+class (BinaryTranscoder xtype) => StreamTranscoder codec xtype ytype where+  -- | Implements a `Encoder` function for the `codec`.+  streamEncoder :: codec -> Encoder xtype ytype++  -- | Implements a `Decoder` function for the `codec`.+  streamDecoder :: codec -> Decoder xtype ytype