diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,76 @@ +[`xcodec`](http://hackage.haskell.org/package/xcodec) change log: +================================================================= + +Major release 1.0 +----------------- + +## Version 1.0.0.0 (05-21-2026) + +Introducing this package to hackage.org! This module provides instances of +`BinaryTranscoder`, a type-class for generically working on binary data formats, +and `StreamTranscoder` a type-class interface for designing encoders and +decoders on top of `BinaryTranscoder`. + + +### Module `Data.XCodec.BinaryTranscoder` + +This module provides the type-class for the `BinaryTranscoder`. It also provides +default instances for `ByteString`, `LazyByteString`, and `ShortByteString` from +the GHC standard library. The minimal definition for a `BinaryTranscoder` is +currently with the following methods (where `bxc` is the type-parameter deriving +`BinaryTranscoder`): + +```haskell +-- | Gives back the length in bytes of the Transcoder data. +lengthBytes :: bxc -> Int +-- | Unpacks transcoder data as BitSet, ie, an infinite bitarray. +unpackValue :: bxc -> BitSet +-- | Places the contents of the transcoder into a ByteString Builder. +serializeValue :: bxc -> Builder +-- | Packs Word8 values into transcoder data `bxc`. +packBytes :: [Word8] -> bxc +-- | Unpacks transcoder data into a list of bytes. +unpackBytes :: bxc -> [Word8] +-- | Prepends a byte value to the top of the transcoder data. +pushByte :: bxc -> Word8 -> bxc +-- | Appends the byte value to the end of the transcoder data. +pushByteEnd :: bxc -> Word8 -> bxc +-- | Takes a subsection of bytes from the beginning of the transcoder data. +takeBytes :: Int -> bxc -> bxc +-- | Removes bytes from the beginning of transcoder data. +dropBytes :: Int -> bxc -> bxc +-- | Removes bytes from the end of transcoder data. +takeBytesEnd :: Int -> bxc -> bxc +-- | Removes bytes from the end of transcoder data. +dropBytesEnd :: Int -> bxc -> bxc +-- | Parititions the binary data at the nth byte. +splitAtByte :: Int -> bxc -> (bxc, bxc) +-- | Repeats a byte value in binary data. +replicateByte :: Int -> Word8 -> bxc +-- | Splits byte values based on the first false result from the predicate. +spanBytes :: (Word8 -> Bool) -> bxc -> (bxc, bxc) +-- | Splits the list while the element predicate holds staring from the left +-- side. +spanBytesEnd :: (Word8 -> Bool) -> bxc -> (bxc, bxc) +``` + +### Module `Data.XCodec.StreamTranscoder` + +This module exports the type-class and methods for `StreamTranscoder`. It +provides no default instances, and is a way for the programmer to implement +generic decoders and decoders on streams of data from deriving from +the `BinaryTranscoder` class of types. It defines the follwing method +signatures: + +```haskell +-- | Generic Encoder signature +type Encoder xtype ytype = xtype -> ytype + +-- | Generic Decoder signature +type Decoder xtype ytype = ytype -> Maybe xtype + +-- | StreamTranscoder give a way to make a generic codec methods. +class (BinaryTranscoder xtype) => StreamTranscoder codec xtype ytype where + streamEncoder :: codec -> Encoder xtype ytype + streamDecoder :: codec -> Decoder xtype ytype +``` diff --git a/LICENSE.txt b/LICENSE.txt new file mode 100644 --- /dev/null +++ b/LICENSE.txt @@ -0,0 +1,28 @@ +BSD 3-Clause License + +Copyright (c) 2026, Zoey McBride + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/README.md b/README.md new file mode 100644 --- /dev/null +++ b/README.md @@ -0,0 +1,109 @@ +xcodec +------ + + + +The `XCodec` Haskell library provides a type-class for generic programming on +bit data for writing encoders and decoders for codecs. The `BinaryTranscoder` +class provides a common interface of methods for processing binary data, and +default instances for the `bytestring` types: `ByteString`, `ShortByteString`, +and `LazyByteString`. The `StreamTranscoder` class then provides the abstract +methods for a generic function encoding from and a generic function decoding to +the types implementing `BinaryTranscoder`. + +Why? +---- + +`XCodec` exists to abstract common patterns that arise when writting code with +the `bytestring` library in Haskell: +- Reusing code for `ShortByteString`, `ByteString` and `LazyByteString`. +- Having common interface for transfering into the bytestring `Builder` type. +- Reading to and from numeric values for bitwise manipulation or initializing +from constant values. + +Making bytestring code generic, can also make code more memory efficient; for +example, we can write functions for `BinaryTranscoder` and apply them to +`LazyByteString` when reading large files and to `ShortByteString` when working +on smaller internal structures. + +Examples +-------- + +Using `XCodec` we can easily read numeric bit data to binary formats, +programmatically: + +```haskell +import Data.ByteString (ByteString) +import Data.ByteString.Short (ShortByteString) +import Data.ByteString.Lazy (LazyByteString) +import Data.Word (Word16) +import Data.XCodec.BinaryTranscoder (packValue) + +-- Infers: packValue :: Int -> ByteString +exStrict :: ByteString +exStrict = packValue (0xdeadbeef :: Int) + +-- Infers: packValue :: Integer -> LazyByteString +exLazy :: LazyByteString +exLazy = packValue (0xf000000000000000000000000000000000000000000d :: Integer) + +-- Infers: packValue :: Word16 -> ShortByteString +exShort :: ShortByteString +exShort = packValue (0xcafe :: Word16) +``` + +And then serialize them through a common interface: + +```haskell +import Data.ByteString.Builder (toLazyByteString, string8) +import Data.XCodec.BinaryTranscoder (serializeValue) + +deadbeefThreeTimes :: LazyByteString +deadbeefThreeTimes = + toLazyByteString . mconcat $ + [ serializeValue exLazy, + serializeValue exStrict, + serializeValue exShort, + string8 "Hello, World!" + ] +``` + +It also enables extracting binary data into `Integer` format so that bitwise +transformations can be performed on large sets of binary data in *O(n)* time: + +```haskell +import Data.Bits ((.<<.), (.&.)) +import Data.Word (Word8) +import Data.XCodec.BinaryTranscoder (unpackValue) + +-- Extracts the byte at the bit number (LSB=0). +byteWindowAt :: (BinaryTranscoder bxc) => bxc -> Int -> Word8 +byteWindowAt bxc idx = fromIntegral $ (unpackValue bxc .>>. idx) .&. 0xFF +``` + + + +Licensing +--------- + +The `xcodec` project and its modules are free software and licensed under the +BSD 3-clause license. See [LICENSE.txt](LICENSE.txt). + +Copyright © 2026 Zoey McBride | [zoeymcbride@mailbox.org]( +mailto:zoeymcbride@mailbox.org) diff --git a/xcodec.cabal b/xcodec.cabal new file mode 100644 --- /dev/null +++ b/xcodec.cabal @@ -0,0 +1,70 @@ +cabal-version: 3.0 +version: 1.0.0.0 +name: xcodec +build-type: Simple +author: Zoey McBride +maintainer: zoeymcbride@mailbox.org +license: BSD-3-Clause +license-file: LICENSE.txt +extra-doc-files: README.md, CHANGELOG.md +category: Numeric, Data, Serialization +synopsis: Classes for working with generically typed codecs. +description: + This module provides a generic interface for working with binary codecs. It + provides a type-class for manipulating bit data and instances for the + bytestring package types (ByteString, ShortByteString, and LazyByteString). + Another type-class is also provided for derivided type-class for importing + modules to write generic encoders and decoders. +tested-with: + GHC == 9.6.7 + +source-repository head { + type: git + location: https://git.sr.ht/~z0/xcodec +} + +-- | Sets flag for CI build toggle +flag CI { + Description: CI Build options + Default: False + Manual: True +} + +common HaskellConfig { + hs-source-dirs: xcodec + default-extensions: StrictData + default-language: GHC2021 +} + +common Depends { + build-depends: + base >= 4.18 && < 5, + array ^>= 0.5, + bytestring >= 0.12 && < 0.13, + containers >= 0.7 && < 8, + text >=2.1 && < 2.2 +} + +common BuildUser { + import: Depends + ghc-options: -Wall -Wextra -Wno-unused-top-binds +} + +common BuildCI { + import: Depends + ghc-options: -Wall -Wextra -Werror -Wno-unused-top-binds +} + +library { + import: HaskellConfig + if flag(CI) { import: BuildCI } + else { import: BuildUser } + exposed-modules: + -- * Type-class for transcoding binary data. + Data.XCodec.BinaryTranscoder + -- * Type-class for transcoding streams derivided from BinaryTranscoder + Data.XCodec.StreamTranscoder + other-modules: + -- * Helper functions + Data.XCodec.Internal +} diff --git a/xcodec/Data/XCodec/BinaryTranscoder.hs b/xcodec/Data/XCodec/BinaryTranscoder.hs new file mode 100644 --- /dev/null +++ b/xcodec/Data/XCodec/BinaryTranscoder.hs @@ -0,0 +1,238 @@ +-- | Module : Data.XCodec.BinaryTranscoder +-- Description : Type-class for building generic codecs on binary data. +-- Copyright : Zoey McBride (c) 2026 +-- License : BSD-3-Clause +-- Maintainer : zoeymcbride@mailbox.org +-- Stability : experimental +module Data.XCodec.BinaryTranscoder + ( -- * Interface for generic transcoding of binary data. + BinaryTranscoder (..), + + -- * Converts streams of bytes into editable bits. + BitSet, + ) +where + +import Data.Bits (Bits, (.&.), (.<<.), (.>>.), (.|.)) +import Data.ByteString (ByteString) +import Data.ByteString qualified as Bytes +import Data.ByteString.Builder (Builder) +import Data.ByteString.Builder qualified as Builder +import Data.ByteString.Lazy (LazyByteString) +import Data.ByteString.Lazy qualified as LBS +import Data.ByteString.Short (ShortByteString) +import Data.ByteString.Short qualified as SBS +import Data.Word (Word8) +import Data.XCodec.Internal (iterateInit, replaceNull) + +-- | Storage for data with bitwise operations. +type BitSet = Integer + +-- | Class of functions for working on binary transcoder streams, that is +-- streams of bytes that we can easily turn into a bitset, and can partition +-- into smaller streams for extracting and parsing data. +class (Monoid bxc) => BinaryTranscoder bxc where + -- This defines basic methods that all BinaryTranscoders share. + {-# MINIMAL + lengthBytes, + unpackValue, + serializeValue, + packBytes, + unpackBytes, + pushByte, + pushByteEnd, + takeBytes, + takeBytesEnd, + dropBytes, + dropBytesEnd, + splitAtByte, + spanBytes, + spanBytesEnd, + replicateByte + #-} + + -- | Gives back the length in bytes of the Transcoder data. + lengthBytes :: bxc -> Int + + -- | Unpacks transcoder data as BitSet, ie, an infinite bitarray. + unpackValue :: bxc -> BitSet + + -- | Places the contents of the transcoder into a ByteString Builder. + serializeValue :: bxc -> Builder + + -- | Packs Word8 values into transcoder data `bxc`. + packBytes :: [Word8] -> bxc + + -- | Unpacks transcoder data into a list of bytes. + unpackBytes :: bxc -> [Word8] + + -- | Prepends a byte value to the top of the transcoder data. + pushByte :: bxc -> Word8 -> bxc + + -- | Appends the byte value to the end of the transcoder data. + pushByteEnd :: bxc -> Word8 -> bxc + + -- | Takes a subsection of bytes from the beginning of the transcoder data. + takeBytes :: Int -> bxc -> bxc + + -- | Removes bytes from the beginning of transcoder data. + dropBytes :: Int -> bxc -> bxc + + -- | Removes bytes from the end of transcoder data. + takeBytesEnd :: Int -> bxc -> bxc + + -- | Removes bytes from the end of transcoder data. + dropBytesEnd :: Int -> bxc -> bxc + + -- | Parititions the binary data at the nth byte. + splitAtByte :: Int -> bxc -> (bxc, bxc) + + -- | Repeats a byte value in binary data. + replicateByte :: Int -> Word8 -> bxc + + -- | Splits byte values based on the first false result from the predicate. + spanBytes :: (Word8 -> Bool) -> bxc -> (bxc, bxc) + + -- | Implements `takeWhile` for byte data in binary transcoder. + takeWhileBytes :: (Word8 -> Bool) -> bxc -> bxc + takeWhileBytes f = fst . spanBytesEnd f + + -- | Implements `dropWhile` for byte data in binary transcoder. + dropWhileBytes :: (Word8 -> Bool) -> bxc -> bxc + dropWhileBytes f = snd . spanBytesEnd f + + -- | Splits the list while the element predicate holds staring from the left + -- side. + spanBytesEnd :: (Word8 -> Bool) -> bxc -> (bxc, bxc) + + -- | Implements `takeWhileEnd` for byte data in binary transcoder. + takeWhileBytesEnd :: (Word8 -> Bool) -> bxc -> bxc + takeWhileBytesEnd f = snd . spanBytesEnd f + + -- | Implements `dropWhileEnd` for byte data in binary transcoder. + dropWhileBytesEnd :: (Word8 -> Bool) -> bxc -> bxc + dropWhileBytesEnd f = fst . spanBytesEnd f + + -- | Represents empty binary transcoder data. + empty :: bxc + empty = mempty + + -- | Appends the contents of two binary transcoders. + append :: bxc -> bxc -> bxc + append = mappend + + -- | Creates a binary transcoder from a single byte. + {-# INLINE singleton #-} + singleton :: Word8 -> bxc + singleton w8 = packBytes [w8] + + -- | Returns True if the binary transcoder data is empty. + {-# INLINE null #-} + null :: bxc -> Bool + null bxc = lengthBytes bxc == 0 + + -- | Returns the first byte removed from the transcoder data, if available. + {-# INLINE uncons #-} + uncons :: bxc -> Maybe (Word8, bxc) + uncons bxc = + let (start, rest) = splitAtByte 1 bxc + in case unpackBytes start of + [top] -> Just (top, rest) + _ -> Nothing + + -- | Returns the last byte removed from the transcoder data, if available. + {-# INLINE usnoc #-} + usnoc :: bxc -> Maybe (bxc, Word8) + usnoc bxc = + let offset = lengthBytes bxc - 1 + (rest, taken) = splitAtByte offset bxc + in case unpackBytes taken of + [final] -> Just (rest, final) + _ -> Nothing + + -- | Packs a bit string value into some bytes data `bxc`. + packValue :: (Integral b, Bits b) => b -> bxc + packValue intdata = + -- This code can be optimized; if we precalculate the length of the BitSet + -- in bits we can mask from the other direction using left shifts, and + -- remove the call to reverse, maybe we can even use a list comprehension to + -- build in place for Bytes.packBytes + packBytes + . replaceNull [0] + . reverse + . map (\(_, byte) -> fromIntegral byte) + . takeWhile (\(rest, byte) -> rest > 0 || byte > 0) + $ iterateInit (\(input, _) -> extractByte input) (,0) intdata + where + -- Gets the byte from LSB in input and shifts the value 8 bits. + extractByte input = (input .>>. 8, input .&. 0xFF) + +-- | Implements binary decoder for Bytestrings, best for constant values that +-- exist for the lifetime of the program. +-- https://hackage-content.haskell.org/package/bytestring-0.12.2.0/docs/Data-ByteString.html#g:2 +instance BinaryTranscoder ByteString where + lengthBytes = Bytes.length + packBytes = Bytes.pack + serializeValue = Builder.byteString + unpackBytes = Bytes.unpack + pushByte = flip Bytes.cons + pushByteEnd = Bytes.snoc + takeBytes = Bytes.take + takeBytesEnd = Bytes.takeEnd + dropBytes = Bytes.drop + dropBytesEnd = Bytes.dropEnd + spanBytes = Bytes.span + spanBytesEnd = Bytes.spanEnd + splitAtByte = Bytes.splitAt + replicateByte k = Bytes.replicate $ fromIntegral k + unpackValue = unpackValue . LBS.fromStrict + +-- TODO: we can improve the proformance of unpackBits by examining the current +-- bytestring, and loading a range of 1-8 bytes to OR simulaneously + +-- | Implements binary decoder for LazyByteString, best for very large data sets +-- that can be loaded in and out of memory on demand. +-- https://hackage-content.haskell.org/package/bytestring-0.12.2.0/docs/Data-ByteString-Lazy.html +instance BinaryTranscoder LazyByteString where + lengthBytes = fromIntegral . LBS.length + packBytes = LBS.pack + unpackBytes = LBS.unpack + serializeValue = Builder.lazyByteString + pushByte = flip LBS.cons + pushByteEnd = LBS.snoc + takeBytes k = LBS.take $ fromIntegral k + takeBytesEnd k = LBS.takeEnd $ fromIntegral k + dropBytes k = LBS.drop $ fromIntegral k + dropBytesEnd k = LBS.dropEnd $ fromIntegral k + spanBytes = LBS.span + spanBytesEnd = LBS.spanEnd + splitAtByte k = LBS.splitAt $ fromIntegral k + replicateByte k = LBS.replicate $ fromIntegral k + unpackValue = + -- From LSB to MSB, OR the current byte and shift the total bits. + LBS.foldl' (\bits byte -> (bits .<<. 8) .|. fromIntegral byte) 0 + . LBS.dropWhile (== 0) + +-- | Implements binary decoder for ShortByteString, best for compact data that +-- doesn't exist long in memory. Notably, it packs better in memory than normal +-- ByteString (prone to Heap Fragmentation). +-- https://hackage-content.haskell.org/package/bytestring-0.12.2.0/docs/Data-ByteString-Short.html#g:1 +instance BinaryTranscoder ShortByteString where + lengthBytes = SBS.length + packBytes = SBS.pack + unpackBytes = SBS.unpack + serializeValue = Builder.shortByteString + pushByte = flip SBS.cons + pushByteEnd = SBS.snoc + takeBytes = SBS.take + takeBytesEnd = SBS.takeEnd + dropBytes = SBS.drop + dropBytesEnd = SBS.dropEnd + spanBytes = SBS.span + spanBytesEnd = SBS.spanEnd + splitAtByte = SBS.splitAt + replicateByte k = SBS.replicate $ fromIntegral k + unpackValue = + -- From LSB to MSB, OR the current byte and shift the total bits. + SBS.foldl' (\bits byte -> (bits .<<. 8) .|. fromIntegral byte) 0 + . SBS.dropWhile (== 0) diff --git a/xcodec/Data/XCodec/Internal.hs b/xcodec/Data/XCodec/Internal.hs new file mode 100644 --- /dev/null +++ b/xcodec/Data/XCodec/Internal.hs @@ -0,0 +1,23 @@ +-- | Module : Data.XCodec.Internal +-- Description : Utility functions for lists. +-- Copyright : Zoey McBride (c) 2026 +-- License : BSD-3-Clause +-- Maintainer : zoeymcbride@mailbox.org +-- Stability : experimental +module Data.XCodec.Internal + ( -- * List utilities + replaceNull, + iterateInit, + ) +where + +-- | Return the the first param if the second param is empty. +{-# INLINE replaceNull #-} +replaceNull :: [a] -> [a] -> [a] +replaceNull replace xs + | null xs = replace + | otherwise = xs + +-- | Wraps iterate to compose with an intermediary type constructor. +iterateInit :: (b -> b) -> (a -> b) -> a -> [b] +iterateInit f iterinit = drop 1 . iterate f . iterinit diff --git a/xcodec/Data/XCodec/StreamTranscoder.hs b/xcodec/Data/XCodec/StreamTranscoder.hs new file mode 100644 --- /dev/null +++ b/xcodec/Data/XCodec/StreamTranscoder.hs @@ -0,0 +1,34 @@ +-- | Module : Data.XCodec.StreamTranscoder +-- Description : Type-class for encoding and decoding binary streams. +-- Copyright : Zoey McBride (c) 2026 +-- License : BSD-3-Clause +-- Maintainer : zoeymcbride@mailbox.org +-- Stability : experimental +module Data.XCodec.StreamTranscoder + ( -- * Interface for implementing generic digit symbol encoder/decoders. + StreamTranscoder (..), + + -- * Type-functions for generating generic StreamTranscoder instances. + Encoder, + Decoder, + ) +where + +import Data.XCodec.BinaryTranscoder (BinaryTranscoder) + +-- | Type of function for transforming some `xtype` to some `ytype`. +type Encoder xtype ytype = xtype -> ytype + +-- | Type of function for transforming some `ytype` into an `xtype` if the +-- `ytype` is well formed. +type Decoder xtype ytype = ytype -> Maybe xtype + +-- | Type-class of `codec`s for which `streamEncoder` produces a representation +-- of an `xtype` through some `ytype` and which `streamDecoder` attempts to form +-- a `xtype` only from well-formed values in `ytype`. +class (BinaryTranscoder xtype) => StreamTranscoder codec xtype ytype where + -- | Implements a `Encoder` function for the `codec`. + streamEncoder :: codec -> Encoder xtype ytype + + -- | Implements a `Decoder` function for the `codec`. + streamDecoder :: codec -> Decoder xtype ytype