diff --git a/LICENSE b/LICENSE new file mode 100644 --- /dev/null +++ b/LICENSE @@ -0,0 +1,165 @@ + GNU LESSER GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + + This version of the GNU Lesser General Public License incorporates +the terms and conditions of version 3 of the GNU General Public +License, supplemented by the additional permissions listed below. + + 0. Additional Definitions. + + As used herein, "this License" refers to version 3 of the GNU Lesser +General Public License, and the "GNU GPL" refers to version 3 of the GNU +General Public License. + + "The Library" refers to a covered work governed by this License, +other than an Application or a Combined Work as defined below. + + An "Application" is any work that makes use of an interface provided +by the Library, but which is not otherwise based on the Library. +Defining a subclass of a class defined by the Library is deemed a mode +of using an interface provided by the Library. + + A "Combined Work" is a work produced by combining or linking an +Application with the Library. The particular version of the Library +with which the Combined Work was made is also called the "Linked +Version". + + The "Minimal Corresponding Source" for a Combined Work means the +Corresponding Source for the Combined Work, excluding any source code +for portions of the Combined Work that, considered in isolation, are +based on the Application, and not on the Linked Version. + + The "Corresponding Application Code" for a Combined Work means the +object code and/or source code for the Application, including any data +and utility programs needed for reproducing the Combined Work from the +Application, but excluding the System Libraries of the Combined Work. + + 1. Exception to Section 3 of the GNU GPL. + + You may convey a covered work under sections 3 and 4 of this License +without being bound by section 3 of the GNU GPL. + + 2. Conveying Modified Versions. + + If you modify a copy of the Library, and, in your modifications, a +facility refers to a function or data to be supplied by an Application +that uses the facility (other than as an argument passed when the +facility is invoked), then you may convey a copy of the modified +version: + + a) under this License, provided that you make a good faith effort to + ensure that, in the event an Application does not supply the + function or data, the facility still operates, and performs + whatever part of its purpose remains meaningful, or + + b) under the GNU GPL, with none of the additional permissions of + this License applicable to that copy. + + 3. Object Code Incorporating Material from Library Header Files. + + The object code form of an Application may incorporate material from +a header file that is part of the Library. You may convey such object +code under terms of your choice, provided that, if the incorporated +material is not limited to numerical parameters, data structure +layouts and accessors, or small macros, inline functions and templates +(ten or fewer lines in length), you do both of the following: + + a) Give prominent notice with each copy of the object code that the + Library is used in it and that the Library and its use are + covered by this License. + + b) Accompany the object code with a copy of the GNU GPL and this license + document. + + 4. Combined Works. + + You may convey a Combined Work under terms of your choice that, +taken together, effectively do not restrict modification of the +portions of the Library contained in the Combined Work and reverse +engineering for debugging such modifications, if you also do each of +the following: + + a) Give prominent notice with each copy of the Combined Work that + the Library is used in it and that the Library and its use are + covered by this License. + + b) Accompany the Combined Work with a copy of the GNU GPL and this license + document. + + c) For a Combined Work that displays copyright notices during + execution, include the copyright notice for the Library among + these notices, as well as a reference directing the user to the + copies of the GNU GPL and this license document. + + d) Do one of the following: + + 0) Convey the Minimal Corresponding Source under the terms of this + License, and the Corresponding Application Code in a form + suitable for, and under terms that permit, the user to + recombine or relink the Application with a modified version of + the Linked Version to produce a modified Combined Work, in the + manner specified by section 6 of the GNU GPL for conveying + Corresponding Source. + + 1) Use a suitable shared library mechanism for linking with the + Library. A suitable mechanism is one that (a) uses at run time + a copy of the Library already present on the user's computer + system, and (b) will operate properly with a modified version + of the Library that is interface-compatible with the Linked + Version. + + e) Provide Installation Information, but only if you would otherwise + be required to provide such information under section 6 of the + GNU GPL, and only to the extent that such information is + necessary to install and execute a modified version of the + Combined Work produced by recombining or relinking the + Application with a modified version of the Linked Version. (If + you use option 4d0, the Installation Information must accompany + the Minimal Corresponding Source and Corresponding Application + Code. If you use option 4d1, you must provide the Installation + Information in the manner specified by section 6 of the GNU GPL + for conveying Corresponding Source.) + + 5. Combined Libraries. + + You may place library facilities that are a work based on the +Library side by side in a single library together with other library +facilities that are not Applications and are not covered by this +License, and convey such a combined library under terms of your +choice, if you do both of the following: + + a) Accompany the combined library with a copy of the same work based + on the Library, uncombined with any other library facilities, + conveyed under the terms of this License. + + b) Give prominent notice with the combined library that part of it + is a work based on the Library, and explaining where to find the + accompanying uncombined form of the same work. + + 6. Revised Versions of the GNU Lesser General Public License. + + The Free Software Foundation may publish revised and/or new versions +of the GNU Lesser General Public License from time to time. Such new +versions will be similar in spirit to the present version, but may +differ in detail to address new problems or concerns. + + Each version is given a distinguishing version number. If the +Library as you received it specifies that a certain numbered version +of the GNU Lesser General Public License "or any later version" +applies to it, you have the option of following the terms and +conditions either of that published version or of any later version +published by the Free Software Foundation. If the Library as you +received it does not specify a version number of the GNU Lesser +General Public License, you may choose any version of the GNU Lesser +General Public License ever published by the Free Software Foundation. + + If the Library as you received it specifies that a proxy can decide +whether future versions of the GNU Lesser General Public License shall +apply, that proxy's public statement of acceptance of any version is +permanent authorization for you to choose that version for the +Library. diff --git a/README.md b/README.md new file mode 100644 --- /dev/null +++ b/README.md @@ -0,0 +1,5 @@ +Variable-length integer encoding +================================ + +This package provides a collection of functions to encode integers in several +variable-length formats via `ByteString` `Builder`s. diff --git a/Setup.hs b/Setup.hs new file mode 100644 --- /dev/null +++ b/Setup.hs @@ -0,0 +1,2 @@ +import Distribution.Simple +main = defaultMain diff --git a/bytestring-builder-varword.cabal b/bytestring-builder-varword.cabal new file mode 100644 --- /dev/null +++ b/bytestring-builder-varword.cabal @@ -0,0 +1,50 @@ +name: bytestring-builder-varword +version: 0.1.0.0 +synopsis: Variable-length integer encoding +description: + A collection of functions for producing 'Data.ByteString.Builder.Builder's + that encode integer types into compact, variable-length representations. +homepage: https://github.com/concert/hs-varword#readme +bug-reports: https://github.com/concert/hs-varword/issues +stability: Experimental +license: LGPL-3 +license-file: LICENSE +author: Paul Weaver +maintainer: paul@concertdaw.co.uk +copyright: 2018 Concert Audio Technologies Ltd +category: Data +build-type: Simple +extra-source-files: README.md +cabal-version: >=1.10 + +source-repository head + type: git + location: https://github.com/githubuser/attoparsec-varword + +library + hs-source-dirs: src + ghc-options: -Wall + default-extensions: BinaryLiterals + , ScopedTypeVariables + exposed-modules: Data.ByteString.Builder.Prim.VarWord + , Data.ByteString.Builder.VarWord + build-depends: base >= 4.7 && < 5 + , bytestring >= 0.9 && < 0.11 + default-language: Haskell2010 + +test-suite bytestring-builder-varword-test + type: exitcode-stdio-1.0 + main-is: Spec.hs + hs-source-dirs: test + ghc-options: -threaded -rtsopts -with-rtsopts=-N -Wall + default-extensions: BinaryLiterals + , OverloadedStrings + build-depends: base >= 4.7 && < 5 + , bytestring-builder-varword + , attoparsec-varword >= 0.1 && < 0.2 + , attoparsec >= 0.10 && < 0.14 + , bytestring >= 0.9 && < 0.11 + , hspec >= 2.4.3 && < 2.5.0 + , QuickCheck >= 2.7 && < 2.11.4 + other-modules: Data.ByteString.Builder.VarWordSpec + default-language: Haskell2010 diff --git a/src/Data/ByteString/Builder/Prim/VarWord.hs b/src/Data/ByteString/Builder/Prim/VarWord.hs new file mode 100644 --- /dev/null +++ b/src/Data/ByteString/Builder/Prim/VarWord.hs @@ -0,0 +1,71 @@ +{- | Implementations of 'BoundedPrim' for several variable-length integer + encodings. Rather than use these directly, consider the higher level + "Data.ByteString.Builder.VarNum" 'Data.ByteString.Builder.Builder' + interface. +-} +module Data.ByteString.Builder.Prim.VarWord + ( varWordBe, denseVarWordBe + , varWordLe + ) where + +import Data.Bits (Bits, shiftR, setBit, (.&.)) +import Data.ByteString.Builder.Prim.Internal (BoundedPrim, boudedPrim) +import Data.Word +import Foreign.Ptr (Ptr, plusPtr) +import Foreign.Storable (Storable, poke, sizeOf) + + +-- | 'BoundedPrim' for big-endian (most significant chunk first) variable length +-- encoding +varWordBe :: forall a. (Bits a, Integral a, Storable a) => BoundedPrim a +varWordBe = boundedPrim (sizeOf (undefined :: a)) go + where + go :: a -> Ptr Word8 -> IO (Ptr Word8) + go a ptr = write7BitChunks (chunkWordBe a) ptr + +-- | 'BoundedPrim' for big-endian (most significant chunk first) variable length +-- encoding where the continuation bit is also reused to pack information +-- slightly more densely. +denseVarWordBe :: forall a. (Bits a, Integral a, Storable a) => BoundedPrim a +denseVarWordBe = boundedPrim (sizeOf (undefined :: a)) go + where + go :: a -> Ptr Word8 -> IO (Ptr Word8) + go a ptr = write7BitChunks (denseChunkWordBe a) ptr + +-- | 'BoundedPrim' for little-endian (least significant chunk first) variable +-- length encoding +varWordLe :: forall a. (Bits a, Integral a, Storable a) => BoundedPrim a +varWordLe = boundedPrim (sizeOf (undefined :: a)) go + where + go :: a -> Ptr Word8 -> IO (Ptr Word8) + go a ptr = write7BitChunks (chunkWordLe a) ptr + +chunkWordBe :: (Bits a, Integral a) => a -> [Word8] +chunkWordBe = reverse . chunkWordLe' 0 + +denseChunkWordBe :: (Bits a, Integral a) => a -> [Word8] +denseChunkWordBe = reverse . chunkWordLe' 1 + +chunkWordLe :: (Bits a, Integral a) => a -> [Word8] +chunkWordLe = chunkWordLe' 0 + +chunkWordLe' :: (Bits a, Integral a) => a -> a -> [Word8] +chunkWordLe' contDec a = + let + highBits = shiftR a 7 + lowBits = a .&. 0b01111111 + in + fromIntegral lowBits : + if highBits == 0 then [] else chunkWordLe' contDec (highBits - contDec) + +write7BitChunks :: [Word8] -> Ptr Word8 -> IO (Ptr Word8) +write7BitChunks [] ptr = return ptr +write7BitChunks (c:cs) ptr = do + poke ptr $ case cs of + [] -> c + _ -> setBit c 7 + write7BitChunks cs $ plusPtr ptr 1 + +-- Fix a typo from the Builder.Prim.Internal: +boundedPrim :: Int -> (a -> Ptr Word8 -> IO (Ptr Word8)) -> BoundedPrim a +boundedPrim = boudedPrim diff --git a/src/Data/ByteString/Builder/VarWord.hs b/src/Data/ByteString/Builder/VarWord.hs new file mode 100644 --- /dev/null +++ b/src/Data/ByteString/Builder/VarWord.hs @@ -0,0 +1,77 @@ +{- | 'Builder's for several variable-length integer encoding schemes. +-} +module Data.ByteString.Builder.VarWord + ( varWordBe, denseVarWordBe + , varWordLe + ) where + +import Data.ByteString.Builder (Builder) +import Data.ByteString.Builder.Prim (primBounded) +import Data.Bits (Bits) + +import Foreign.Storable (Storable) + +import qualified Data.ByteString.Builder.Prim.VarWord as Prim + + +-- | Produce a 'Builder' to encode an integer with a variable-length (chunked) +-- encoding. 7-bit chunks are produced in big-endian (most significant chunk +-- first) order, and each is prefixed with a continuation bit. +-- +-- > 18464 == 1 0010000 0100000 == +-- > (A) (B) (C) +-- > +-- > +-- Keep reading +-- > | +-- Keep reading +-- > | | +-- Stop +-- > v v v +-- > 10000001 10010000 00100000 +-- > |<--->| |<--->| |<--->| +-- > (A) (B) (C) +-- +-- >>> Builder.toLazyByteString $ varWordBe (18464 :: Word32) +-- "\129\144\32" +varWordBe :: (Bits a, Integral a, Storable a) => a -> Builder +varWordBe = primBounded Prim.varWordBe + +-- | Produce a 'Builder' to encode an integer with a variable-length (chunked) +-- encoding. 7-bit chunks are produced in big-endian (most significant chunk +-- first) order, each is prefixed with a continuation bit, and the +-- continuation bit carries a further bit of information to reduce the size of +-- the resulting encoding. +-- +-- > 34976 == 10 0010001 0100000 == 34976 +-- > (A) (B) (C) +-- > +-- > +-- Keep reading +-- > | +-- Keep reading +-- > | | +-- Stop +-- > v v v +-- > 10000001 10010000 00100000 +-- > |<--->| |<--->| |<--->| +-- > (A-1) (B-1) (C) +-- +-- >>> Builder.toLazyByteString $ denseVarWordBe (34976 :: Word32) +-- "\129\144\32" +denseVarWordBe :: (Bits a, Integral a, Storable a) => a -> Builder +denseVarWordBe = primBounded Prim.denseVarWordBe + +-- | Produce a 'Builder' to encode an integer with a varible-length (chunked) +-- encoding. 7-bit chunks are produced in little-endian (least significant +-- chunk first) order, and each is prefixed with a continuation bit. +-- +-- > 526337 == 100000 0010000 0000001 == +-- > (A) (B) (C) +-- > +-- > +-- Keep reading +-- > | +-- Keep reading +-- > | | +-- Stop +-- > v v v +-- > 10000001 10010000 00100000 +-- > |<--->| |<--->| |<--->| +-- > (C) (B) (A) +-- +-- >>> Builder.toLazyByteString $ varWordLe (526337 :: Word32) +-- "\129\144\32" +varWordLe :: (Bits a, Integral a, Storable a) => a -> Builder +varWordLe = primBounded Prim.varWordLe diff --git a/test/Data/ByteString/Builder/VarWordSpec.hs b/test/Data/ByteString/Builder/VarWordSpec.hs new file mode 100644 --- /dev/null +++ b/test/Data/ByteString/Builder/VarWordSpec.hs @@ -0,0 +1,26 @@ +module Data.ByteString.Builder.VarWordSpec where + +import Test.Hspec + +import Data.ByteString (ByteString) +import Data.ByteString.Builder (Builder) +import qualified Data.ByteString.Builder as Builder +import qualified Data.ByteString.Lazy as LBS +import Data.Word + +import Data.ByteString.Builder.VarWord (varWordBe, denseVarWordBe, varWordLe) + +spec :: Spec +spec = do + varWordSpec "Big-endian chunked builder" + varWordBe "\135\196\64" + varWordSpec "Big-endian dense chunked builder" + denseVarWordBe "\134\195\64" + varWordSpec "Little-endian chunked builder" + varWordLe "\192\196\7" + +varWordSpec :: String -> (Word32 -> Builder) -> ByteString -> Spec +varWordSpec s b expected = describe s $ do + it "should serialise a known number" $ do + LBS.toStrict (Builder.toLazyByteString $ b 123456) + `shouldBe` expected diff --git a/test/Spec.hs b/test/Spec.hs new file mode 100644 --- /dev/null +++ b/test/Spec.hs @@ -0,0 +1,1 @@ +{-# OPTIONS_GHC -F -pgmF hspec-discover #-}