diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,45 @@ Major release 1.0 ----------------- +## Version 1.1.0.0 (07-08-2026) + +This version revamps the class interface for `BinaryTranscoder`. Functions now +are named with octets in mind rather than bytes so that functions specific to +the `BinaryTranscoder` are easy to spot. Namely, the following functions got +renamed: + +| Old | New | +|------------------|------------------| +| `lengthBytes` | `totalOctets` | +| `lengthBytes` | `totalOctets` | +| `serializeValue` | `unpackSerial` | +| `packBytes` | `packOctets` | +| `unpackBytes` | `unpackOctets` | +| `pushByte` | `pushOctetTop` | +| `pushByteEnd` | `pushOctetEnd` | +| `takeBytes` | `takeOctetsTop` | +| `takeBytesEnd` | `takeOctetsEnd` | +| `dropBytes` | `dropOctetsTop` | +| `dropBytesEnd` | `dropOctetsEnd` | +| `splitAtByte` | `splitOffset` | +| `spanBytes` | `spanOctetsTop` | +| `spanBytesEnd` | `spanOctetsEnd` | +| `replicateByte` | `replicateOctet` | +| `singleton` | `fromOctet` | + +The `unpackValue` function was replaced with the `unpackValueCPU`, +`unpackValueBE`, and `unpackValueLE` functions, so endianness of the unpacked +value can be specified. + +Also the `packValue` function got the same change. The functions `packValueBE` +and `packValueLE` replace it. + +The function `swapOrder` is introduced so that the internal order of the +transcoder can be flipped without unpacking the value. + +Unit tests for xcodec can now be found as part of [ipfshs]( +https://git.sr.ht/~z0/ipfshs/). + ## Version 1.0.0.0 (05-21-2026) Introducing this package to hackage.org! This module provides instances of diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -1,16 +1,11 @@ -xcodec ------- - - +https://builds.sr.ht/~z0/xcodec/commits/main/ipfshs-ci.yml.svg)]( +https://builds.sr.ht/~z0/xcodec/commits/main/ipfshs-ci.yml?) +------------------------------------------------------------ -The `XCodec` Haskell library provides a type-class for generic programming on +The `xcodec` Haskell library provides a type-class for generic programming on bit data for writing encoders and decoders for codecs. The `BinaryTranscoder` class provides a common interface of methods for processing binary data, and default instances for the `bytestring` types: `ByteString`, `ShortByteString`, @@ -21,10 +16,10 @@ Why? ---- -`XCodec` exists to abstract common patterns that arise when writting code with +`xcodec` exists to abstract common patterns that arise when writing code with the `bytestring` library in Haskell: - Reusing code for `ShortByteString`, `ByteString` and `LazyByteString`. -- Having common interface for transfering into the bytestring `Builder` type. +- Having common interface for transferring into the bytestring `Builder` type. - Reading to and from numeric values for bitwise manipulation or initializing from constant values. @@ -36,7 +31,7 @@ Examples -------- -Using `XCodec` we can easily read numeric bit data to binary formats, +Using `xcodec` we can easily read numeric bit data to binary formats, programmatically: ```haskell @@ -52,43 +47,52 @@ -- Infers: packValue :: Integer -> LazyByteString exLazy :: LazyByteString -exLazy = packValue (0xf000000000000000000000000000000000000000000d :: Integer) +exLazy = packValue (0xf0000000000000000000000d :: Integer) -- Infers: packValue :: Word16 -> ShortByteString exShort :: ShortByteString exShort = packValue (0xcafe :: Word16) ``` -And then serialize them through a common interface: +And then serialize them through a common interface which makes it easy to +intersperse data from different formats: ```haskell -import Data.ByteString.Builder (toLazyByteString, string8) +import Data.ByteString.Builder qualified as Builder +import Data.ByteString.Lazy (LazyByteString) import Data.XCodec.BinaryTranscoder (serializeValue) -deadbeefThreeTimes :: LazyByteString -deadbeefThreeTimes = - toLazyByteString . mconcat $ - [ serializeValue exLazy, - serializeValue exStrict, - serializeValue exShort, - string8 "Hello, World!" +-- We can join several BXCs and Builders into a single unit of data. +exBuilder :: LazyByteString +exBuilder = + Builder.toLazyByteString . mconcat $ + [ serializeValue exLazy, -- serializeValue :: LazyByteString -> Builder + serializeValue exStrict, -- serializeValue :: ByteString -> Builder + serializeValue exShort, -- serializeValue :: ShortByteString -> Builder + Builder.string8 "Hello, World!" -- string8 :: String -> Builder ] ``` It also enables extracting binary data into `Integer` format so that bitwise -transformations can be performed on large sets of binary data in *O(n)* time: +transformations can be performed on large sets of binary data in *O(n)* time +with a specified [byte-order](https://en.wikipedia.org/wiki/Endianness): ```haskell -import Data.Bits ((.<<.), (.&.)) -import Data.Word (Word8) -import Data.XCodec.BinaryTranscoder (unpackValue) +import Data.Bits ((.>>.), (.&.)) +import Data.Word (Word32) +import Data.XCodec.BinaryTranscoder (BinaryTranscoder, unpackValue) --- Extracts the byte at the bit number (LSB=0). -byteWindowAt :: (BinaryTranscoder bxc) => bxc -> Int -> Word8 -byteWindowAt bxc idx = fromIntegral $ (unpackValue bxc .>>. idx) .&. 0xFF -``` +-- We can easily read an entire bit-set representation of transcoder data, +-- directly from LazyByteString because it derives BinaryTranscoder. This +-- function produces its big-endian representation +largeBitSet :: Integer +largeBitSet = unpackValueBE exBuilder - Licensing --------- diff --git a/xcodec.cabal b/xcodec.cabal --- a/xcodec.cabal +++ b/xcodec.cabal @@ -1,5 +1,5 @@ cabal-version: 3.0 -version: 1.0.0.0 +version: 1.1.0.0 name: xcodec build-type: Simple author: Zoey McBride diff --git a/xcodec/Data/XCodec/BinaryTranscoder.hs b/xcodec/Data/XCodec/BinaryTranscoder.hs --- a/xcodec/Data/XCodec/BinaryTranscoder.hs +++ b/xcodec/Data/XCodec/BinaryTranscoder.hs @@ -8,8 +8,32 @@ ( -- * Interface for generic transcoding of binary data. BinaryTranscoder (..), - -- * Converts streams of bytes into editable bits. + -- * BinaryTranscoder data as numeric values BitSet, + Octet, + + -- * Converts BinaryTranscoders into editable streams. + UnpackValue, + UnpackOctets, + UnpackSerial, + + -- * Conversion of Integral values to transcoder data. + packValueBE, + packValueLE, + + -- * BinaryTranscoder utilities + fromOctet, + null, + uncons, + unsnoc, + takeOctetsTop, + dropOctetsTop, + takeOctetsEnd, + dropOctetsEnd, + takeWhileEndOctets, + takeWhileTopOctets, + dropWhileEndOctets, + dropWhileTopOctets, ) where @@ -24,168 +48,105 @@ import Data.ByteString.Short qualified as SBS import Data.Word (Word8) import Data.XCodec.Internal (iterateInit, replaceNull) +import Prelude hiding (null) --- | Storage for data with bitwise operations. +-- | Stores 8-bit values. +type Octet = Word8 + +-- | Stores BinaryTranscoder as numeric values. type BitSet = Integer +-- | Functions used for unpacking transcoder data into BitSets. +type UnpackValue bxc = bxc -> BitSet + +-- | Functions used for unpacking transcoder data into octets/8-bit values. +type UnpackOctets bxc = bxc -> [Octet] + +-- | Functions used for unpacking transcoder data into bytestring Builders. +type UnpackSerial bxc = bxc -> Builder + -- | Class of functions for working on binary transcoder streams, that is -- streams of bytes that we can easily turn into a bitset, and can partition -- into smaller streams for extracting and parsing data. class (Monoid bxc) => BinaryTranscoder bxc where - -- This defines basic methods that all BinaryTranscoders share. {-# MINIMAL - lengthBytes, - unpackValue, - serializeValue, - packBytes, - unpackBytes, - pushByte, - pushByteEnd, - takeBytes, - takeBytesEnd, - dropBytes, - dropBytesEnd, - splitAtByte, - spanBytes, - spanBytesEnd, - replicateByte + totalOctets, + swapOrder, + unpackSerial, + unpackValueCPU, + unpackValueBE, + unpackValueLE, + packOctets, + unpackOctets, + pushOctetTop, + pushOctetEnd, + spanOctetsTop, + spanOctetsEnd, + splitOffset, + replicateOctet #-} - -- | Gives back the length in bytes of the Transcoder data. - lengthBytes :: bxc -> Int + -- | Gives back the length in octets of the Transcoder data. + totalOctets :: bxc -> Int - -- | Unpacks transcoder data as BitSet, ie, an infinite bitarray. - unpackValue :: bxc -> BitSet + -- | Packs Octet values into transcoder data `bxc`. + packOctets :: [Octet] -> bxc -- | Places the contents of the transcoder into a ByteString Builder. - serializeValue :: bxc -> Builder - - -- | Packs Word8 values into transcoder data `bxc`. - packBytes :: [Word8] -> bxc + unpackSerial :: UnpackSerial bxc -- | Unpacks transcoder data into a list of bytes. - unpackBytes :: bxc -> [Word8] - - -- | Prepends a byte value to the top of the transcoder data. - pushByte :: bxc -> Word8 -> bxc - - -- | Appends the byte value to the end of the transcoder data. - pushByteEnd :: bxc -> Word8 -> bxc + unpackOctets :: UnpackOctets bxc - -- | Takes a subsection of bytes from the beginning of the transcoder data. - takeBytes :: Int -> bxc -> bxc + -- | Unpacks transcoder data as BitSet in the host CPU's endianness. + unpackValueCPU :: UnpackValue bxc - -- | Removes bytes from the beginning of transcoder data. - dropBytes :: Int -> bxc -> bxc + -- | Unpacks transcoder data as BitSet in Big-Endian (BE) order. + unpackValueBE :: UnpackValue bxc - -- | Removes bytes from the end of transcoder data. - takeBytesEnd :: Int -> bxc -> bxc + -- | Unpacks transcoder data as BitSet in Little-Endian (LE) order. + unpackValueLE :: UnpackValue bxc - -- | Removes bytes from the end of transcoder data. - dropBytesEnd :: Int -> bxc -> bxc + -- | Reverses the byte order of the transcoder data. + swapOrder :: bxc -> bxc - -- | Parititions the binary data at the nth byte. - splitAtByte :: Int -> bxc -> (bxc, bxc) + -- | Prepends an octet value to the top of the transcoder data. + pushOctetTop :: bxc -> Octet -> bxc - -- | Repeats a byte value in binary data. - replicateByte :: Int -> Word8 -> bxc + -- | Appends the octet value to the end of the transcoder data. + pushOctetEnd :: bxc -> Octet -> bxc - -- | Splits byte values based on the first false result from the predicate. - spanBytes :: (Word8 -> Bool) -> bxc -> (bxc, bxc) + -- | Parititions the binary data at + splitOffset :: Int -> bxc -> (bxc, bxc) - -- | Implements `takeWhile` for byte data in binary transcoder. - takeWhileBytes :: (Word8 -> Bool) -> bxc -> bxc - takeWhileBytes f = fst . spanBytesEnd f + -- | Repeats an octet value in binary data. + replicateOctet :: Int -> Octet -> bxc - -- | Implements `dropWhile` for byte data in binary transcoder. - dropWhileBytes :: (Word8 -> Bool) -> bxc -> bxc - dropWhileBytes f = snd . spanBytesEnd f + -- | Splits octet values based on the first false result from the predicate. + spanOctetsTop :: (Octet -> Bool) -> bxc -> (bxc, bxc) -- | Splits the list while the element predicate holds staring from the left -- side. - spanBytesEnd :: (Word8 -> Bool) -> bxc -> (bxc, bxc) - - -- | Implements `takeWhileEnd` for byte data in binary transcoder. - takeWhileBytesEnd :: (Word8 -> Bool) -> bxc -> bxc - takeWhileBytesEnd f = snd . spanBytesEnd f - - -- | Implements `dropWhileEnd` for byte data in binary transcoder. - dropWhileBytesEnd :: (Word8 -> Bool) -> bxc -> bxc - dropWhileBytesEnd f = fst . spanBytesEnd f - - -- | Represents empty binary transcoder data. - empty :: bxc - empty = mempty - - -- | Appends the contents of two binary transcoders. - append :: bxc -> bxc -> bxc - append = mappend - - -- | Creates a binary transcoder from a single byte. - {-# INLINE singleton #-} - singleton :: Word8 -> bxc - singleton w8 = packBytes [w8] - - -- | Returns True if the binary transcoder data is empty. - {-# INLINE null #-} - null :: bxc -> Bool - null bxc = lengthBytes bxc == 0 - - -- | Returns the first byte removed from the transcoder data, if available. - {-# INLINE uncons #-} - uncons :: bxc -> Maybe (Word8, bxc) - uncons bxc = - let (start, rest) = splitAtByte 1 bxc - in case unpackBytes start of - [top] -> Just (top, rest) - _ -> Nothing - - -- | Returns the last byte removed from the transcoder data, if available. - {-# INLINE usnoc #-} - usnoc :: bxc -> Maybe (bxc, Word8) - usnoc bxc = - let offset = lengthBytes bxc - 1 - (rest, taken) = splitAtByte offset bxc - in case unpackBytes taken of - [final] -> Just (rest, final) - _ -> Nothing - - -- | Packs a bit string value into some bytes data `bxc`. - packValue :: (Integral b, Bits b) => b -> bxc - packValue intdata = - -- This code can be optimized; if we precalculate the length of the BitSet - -- in bits we can mask from the other direction using left shifts, and - -- remove the call to reverse, maybe we can even use a list comprehension to - -- build in place for Bytes.packBytes - packBytes - . replaceNull [0] - . reverse - . map (\(_, byte) -> fromIntegral byte) - . takeWhile (\(rest, byte) -> rest > 0 || byte > 0) - $ iterateInit (\(input, _) -> extractByte input) (,0) intdata - where - -- Gets the byte from LSB in input and shifts the value 8 bits. - extractByte input = (input .>>. 8, input .&. 0xFF) + spanOctetsEnd :: (Octet -> Bool) -> bxc -> (bxc, bxc) -- | Implements binary decoder for Bytestrings, best for constant values that -- exist for the lifetime of the program. -- https://hackage-content.haskell.org/package/bytestring-0.12.2.0/docs/Data-ByteString.html#g:2 instance BinaryTranscoder ByteString where - lengthBytes = Bytes.length - packBytes = Bytes.pack - serializeValue = Builder.byteString - unpackBytes = Bytes.unpack - pushByte = flip Bytes.cons - pushByteEnd = Bytes.snoc - takeBytes = Bytes.take - takeBytesEnd = Bytes.takeEnd - dropBytes = Bytes.drop - dropBytesEnd = Bytes.dropEnd - spanBytes = Bytes.span - spanBytesEnd = Bytes.spanEnd - splitAtByte = Bytes.splitAt - replicateByte k = Bytes.replicate $ fromIntegral k - unpackValue = unpackValue . LBS.fromStrict + totalOctets = Bytes.length + swapOrder = Bytes.reverse + packOctets = Bytes.pack + unpackSerial = Builder.byteString + unpackValueCPU = unpackValueCPU . LBS.fromStrict + unpackValueLE = unpackValueLE . LBS.fromStrict + unpackValueBE = unpackValueBE . LBS.fromStrict + unpackOctets = Bytes.unpack + pushOctetTop = flip Bytes.cons + pushOctetEnd = Bytes.snoc + spanOctetsTop = Bytes.span + spanOctetsEnd = Bytes.spanEnd + splitOffset = Bytes.splitAt + replicateOctet k = Bytes.replicate $ fromIntegral k -- TODO: we can improve the proformance of unpackBits by examining the current -- bytestring, and loading a range of 1-8 bytes to OR simulaneously @@ -194,45 +155,141 @@ -- that can be loaded in and out of memory on demand. -- https://hackage-content.haskell.org/package/bytestring-0.12.2.0/docs/Data-ByteString-Lazy.html instance BinaryTranscoder LazyByteString where - lengthBytes = fromIntegral . LBS.length - packBytes = LBS.pack - unpackBytes = LBS.unpack - serializeValue = Builder.lazyByteString - pushByte = flip LBS.cons - pushByteEnd = LBS.snoc - takeBytes k = LBS.take $ fromIntegral k - takeBytesEnd k = LBS.takeEnd $ fromIntegral k - dropBytes k = LBS.drop $ fromIntegral k - dropBytesEnd k = LBS.dropEnd $ fromIntegral k - spanBytes = LBS.span - spanBytesEnd = LBS.spanEnd - splitAtByte k = LBS.splitAt $ fromIntegral k - replicateByte k = LBS.replicate $ fromIntegral k - unpackValue = + totalOctets = fromIntegral . LBS.length + swapOrder = LBS.reverse + packOctets = LBS.pack + unpackOctets = LBS.unpack + unpackValueLE = + -- From MSB to LSB, OR the current byte and shift the total bits. + LBS.foldr (\byte bits -> (bits .<<. 8) .|. fromIntegral byte) 0 + . LBS.dropWhileEnd (== 0) + unpackValueBE = -- From LSB to MSB, OR the current byte and shift the total bits. LBS.foldl' (\bits byte -> (bits .<<. 8) .|. fromIntegral byte) 0 . LBS.dropWhile (== 0) + unpackSerial = Builder.lazyByteString + pushOctetTop = flip LBS.cons + pushOctetEnd = LBS.snoc + spanOctetsTop = LBS.span + spanOctetsEnd = LBS.spanEnd + splitOffset k = LBS.splitAt $ fromIntegral k + replicateOctet k = LBS.replicate $ fromIntegral k + unpackValueCPU = unpackValueLE -- | Implements binary decoder for ShortByteString, best for compact data that -- doesn't exist long in memory. Notably, it packs better in memory than normal -- ByteString (prone to Heap Fragmentation). -- https://hackage-content.haskell.org/package/bytestring-0.12.2.0/docs/Data-ByteString-Short.html#g:1 instance BinaryTranscoder ShortByteString where - lengthBytes = SBS.length - packBytes = SBS.pack - unpackBytes = SBS.unpack - serializeValue = Builder.shortByteString - pushByte = flip SBS.cons - pushByteEnd = SBS.snoc - takeBytes = SBS.take - takeBytesEnd = SBS.takeEnd - dropBytes = SBS.drop - dropBytesEnd = SBS.dropEnd - spanBytes = SBS.span - spanBytesEnd = SBS.spanEnd - splitAtByte = SBS.splitAt - replicateByte k = SBS.replicate $ fromIntegral k - unpackValue = + totalOctets = SBS.length + swapOrder = SBS.reverse + packOctets = SBS.pack + unpackOctets = SBS.unpack + unpackValueLE = + -- From MSB to LSB , OR the current byte and shift the total bits. + SBS.foldr (\byte bits -> (bits .<<. 8) .|. fromIntegral byte) 0 + . SBS.dropWhileEnd (== 0) + unpackValueBE = -- From LSB to MSB, OR the current byte and shift the total bits. SBS.foldl' (\bits byte -> (bits .<<. 8) .|. fromIntegral byte) 0 . SBS.dropWhile (== 0) + unpackSerial = Builder.shortByteString + pushOctetTop = flip SBS.cons + pushOctetEnd = SBS.snoc + spanOctetsTop = SBS.span + spanOctetsEnd = SBS.spanEnd + splitOffset = SBS.splitAt + replicateOctet k = SBS.replicate $ fromIntegral k + unpackValueCPU = unpackValueLE + +-- | Takes a subsection of octets from the beginning of the transcoder data. +takeOctetsTop :: (BinaryTranscoder bxc) => Int -> bxc -> bxc +takeOctetsTop idx = fst . splitOffset idx + +-- | Removes octets from the beginning of transcoder data. +dropOctetsTop :: (BinaryTranscoder bxc) => Int -> bxc -> bxc +dropOctetsTop idx = snd . splitOffset idx + +-- | Removes octets from the end of transcoder data. +takeOctetsEnd :: (BinaryTranscoder bxc) => Int -> bxc -> bxc +takeOctetsEnd idx bxc = snd $ splitOffset (totalOctets bxc - idx) bxc + +-- | Removes octets from the end of transcoder data. +dropOctetsEnd :: (BinaryTranscoder bxc) => Int -> bxc -> bxc +dropOctetsEnd idx bxc = fst $ splitOffset (totalOctets bxc - idx) bxc + +-- | Implements `takeWhile` for byte data in binary transcoder. +takeWhileTopOctets :: (BinaryTranscoder bxc) => (Octet -> Bool) -> bxc -> bxc +takeWhileTopOctets f = fst . spanOctetsEnd f + +-- | Implements `dropWhile` for byte data in binary transcoder. +dropWhileTopOctets :: (BinaryTranscoder bxc) => (Octet -> Bool) -> bxc -> bxc +dropWhileTopOctets f = snd . spanOctetsEnd f + +-- | Implements `takeWhileEnd` for byte data in binary transcoder. +takeWhileEndOctets :: (BinaryTranscoder bxc) => (Octet -> Bool) -> bxc -> bxc +takeWhileEndOctets f = snd . spanOctetsEnd f + +-- | Implements `dropWhileEnd` for byte data in binary transcoder. +dropWhileEndOctets :: (BinaryTranscoder bxc) => (Octet -> Bool) -> bxc -> bxc +dropWhileEndOctets f = fst . spanOctetsEnd f + +-- | Creates a binary transcoder from a single byte. +{-# INLINE fromOctet #-} +fromOctet :: (BinaryTranscoder bxc) => Octet -> bxc +fromOctet w8 = packOctets [w8] + +-- | Returns True if the binary transcoder data is empty. +{-# INLINE null #-} +null :: (BinaryTranscoder bxc) => bxc -> Bool +null bxc = totalOctets bxc == 0 + +-- | Returns the first byte removed from the transcoder data, if available. +{-# INLINE uncons #-} +uncons :: (BinaryTranscoder bxc) => bxc -> Maybe (Octet, bxc) +uncons bxc = + let (start, rest) = splitOffset 1 bxc + in case unpackOctets start of + [top] -> Just (top, rest) + _ -> Nothing + +-- | Returns the last byte removed from the transcoder data, if available. +{-# INLINE unsnoc #-} +unsnoc :: (BinaryTranscoder bxc) => bxc -> Maybe (bxc, Octet) +unsnoc bxc = + let offset = totalOctets bxc - 1 + (rest, taken) = splitOffset offset bxc + in case unpackOctets taken of + [final] -> Just (rest, final) + _ -> Nothing + +-- | Packs a bit string value into some bytes data `bxc`. +{-# INLINE packValueOrder #-} +packValueOrder :: + (Integral val, Bits val, BinaryTranscoder bxc) => + ([Octet] -> [Octet]) -> + val -> + bxc +packValueOrder setorder intdata = + -- This code can be optimized; if we precalculate the length of the BitSet + -- in bits we can mask from the other direction using left shifts, and + -- remove the call to setorder, maybe we can even use a list comprehension to + -- build in place for packOctets + packOctets + . replaceNull [0] + . setorder + . map (\(_, byte) -> fromIntegral byte) + . takeWhile (\(rest, byte) -> rest > 0 || byte > 0) + $ iterateInit (\(input, _) -> extractByte input) (,0) intdata + where + -- Gets the byte from LSB in input and shifts the value 8 bits. + extractByte input = (input .>>. 8, input .&. 0xFF) + +-- | Packs a bit string value into some bytes data `bxc` in Big-Endian format. +packValueBE :: (Integral val, Bits val, BinaryTranscoder bxc) => val -> bxc +packValueBE = packValueOrder reverse + +-- | Packs a bit string value into some bytes data `bxc` in Little-Endian +-- format. +packValueLE :: (Integral val, Bits val, BinaryTranscoder bxc) => val -> bxc +packValueLE = packValueOrder id