diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -17,21 +17,22 @@ ## Features and non-features -* __Excellent performance__. On microbenchmarks, `flatparse` is around 10 times faster than `attoparsec` or `megaparsec`. On larger examples with heavier use of source positions and spans and/or indentation parsing, the performance difference grows to 20-30 times. Compile times and executable sizes are also significantly better with `flatparse` than with `megaparsec` or `attoparsec`. `flatparse` internals make liberal use of unboxed tuples and GHC primops. As a result, pure validators (parsers returning `()`) in `flatparse` are not difficult to implement with zero heap allocation. +* __Excellent performance__. On microbenchmarks, `flatparse` is around 5-10 times faster than `attoparsec` or `megaparsec`. On larger examples with heavier use of source positions and spans and/or indentation parsing, the performance difference is greater. Compile times and executable sizes are also significantly better with `flatparse` than with `megaparsec` or `attoparsec`. `flatparse` internals make liberal use of unboxed tuples and GHC primops. As a result, pure validators (parsers returning `()`) in `flatparse` are not difficult to implement with zero heap allocation. * __No incremental parsing__, and __only strict `ByteString`__ is supported as input. However, it can be still useful to convert from `Text`, `String` or other types to `ByteString`, and then use `flatparse` for parsing, since `flatparse` performance usually more than makes up for the conversion costs. -* __Only little-endian 64 bit systems are currently supported as the host machine__. This may change in the future. Getting good performance requires architecture-specific optimizations; I've only considered the most common setting at this point. However, `flatparse` does include primitive integer parsers with specific endianness. +* __Only little-endian systems are currently supported as the host machine__. This may change in the future. However, `flatparse` does include primitive integer parsers with specific endianness. * __Support for fast source location handling, indentation parsing and informative error messages__. `flatparse` provides a low-level interface to these. Batteries are _not included_, but it should be possible for users to build custom solutions, which are more sophisticated, but still as fast as possible. In my experience, the included batteries in other libraries often come with major unavoidable overheads, and often we still have to extend existing machinery in order to scale to production features. * The __backtracking model__ of `flatparse` is different to parsec libraries, and is more close to the [nom](https://github.com/Geal/nom) library in Rust. The idea is that _parser failure_ is distinguished from _parsing error_. The former is used for control flow, and we can backtrack from it. The latter is used for unrecoverable errors, and by default it's propagated to the top. `flatparse` does not track whether parsers have consumed inputs. In my experience, what we really care about is the failure/error distinction, and in `parsec` or `megaparsec` the consumed/non-consumed separation is often muddled and discarded in larger parser implementations. By default, basic `flatparse` parsers can fail but can not throw errors, with the exception of the specifically error-throwing operations. Hence, `flatparse` users have to be mindful about grammar, and explicitly insert errors where it is known that the input can't be valid. `flatparse` comes in two flavors: [`FlatParse.Basic`][basic] and [`FlatParse.Stateful`][stateful]. Both support a custom error type. Also, both come in three modes, where we can respectively run `IO` actions, `ST` actions, or no side effects. The modes are selected by a state token type parameter on the parser types. -* [`FlatParse.Basic`][basic] only supports the above features. If you don't need indentation - parsing, this is sufficient. -* [`FlatParse.Stateful`][stateful] additionally supports a built-in `Int` worth of internal state - and an additional custom reader environment. This can support a wide range of indentation parsing - features. There is a slight overhead in performance and code size compared to `Basic`. However, in - small parsers and microbenchmarks the difference between `Basic` and `Stateful` is often reduced - to near zero by GHC and/or LLVM optimization. +* [`FlatParse.Basic`][basic] only supports the above features. If you don't need + indentation parsing, this is sufficient. +* [`FlatParse.Stateful`][stateful] additionally supports a built-in `Int` worth + of internal state and an additional custom reader environment. This can + support a wide range of indentation parsing features. There is a moderate + overhead in performance and code size compared to `Basic`. In microbenchmarks + and small parsers, the performance difference between `Basic` and `Stateful` + is more up to the whims of GHC and LLVM, and is a bit more "random". ## Tutorial @@ -45,36 +46,36 @@ ## Some benchmarks Execution times below. See source code in [bench](bench). Compiled with GHC -9.4.4 `-O2 -fllvm`. Executed on Intel 1165G7 CPU at 28W power draw. Uses -`nightly-2023-02-06` Stackage snapshot for the involved packages. +9.8.3. `-O2 -fllvm` with `flatparse-0.5.2.1`. Executed on Intel 1345U CPU. Uses +`nightly-2024-11-11` Stackage snapshot for the involved packages. | benchmark | runtime | |-----------------------------|------------- -|sexp/fpbasic | 1.93 ms | -|sexp/fpstateful | 2.00 ms | -|sexp/attoparsec | 21.82 ms | -|sexp/megaparsec | 59.60 ms | -|sexp/parsec | 79.81 ms | -|long keyword/fpbasic | 0.1 ms | -|long keyword/fpstateful | 0.1 ms | -|long keyword/attoparsec | 2.43 ms | -|long keyword/megaparsec | 5.2 ms | -|long keyword/parsec | 10.02 ms | -|numeral csv/fpbasic | 0.72 ms | -|numeral csv/fpstateful | 0.56 ms | -|numeral csv/attoparsec | 10.52 ms | -|numeral csv/megaparsec | 19.77 ms | -|numeral csv/parsec | 26.46 ms | +|sexp/fpbasic | 3.262 ms | +|sexp/fpstateful | 2.523 ms | +|sexp/attoparsec | 15.28 ms | +|sexp/megaparsec | 36.03 ms | +|sexp/parsec | 67.16 ms | +|long keyword/fpbasic | 0.079 ms | +|long keyword/fpstateful | 0.078 ms | +|long keyword/attoparsec | 0.705 ms | +|long keyword/megaparsec | 2.014 ms | +|long keyword/parsec | 8.813 ms | +|numeral csv/fpbasic | 0.482 ms | +|numeral csv/fpstateful | 0.580 ms | +|numeral csv/attoparsec | 4.661 ms | +|numeral csv/megaparsec | 7.415 ms | +|numeral csv/parsec | 24.54 ms | Object file sizes for each module containing the `s-exp`, `long keyword` and `numeral csv` benchmarks. | library | object file size (bytes) | | ------- | ------------------------ | -| fpbasic | 20656 | -| fpstateful | 26664 | -| attoparsec | 69384 | -| megaparsec | 226232 | -| parsec | 117696 | +| fpbasic | 26704 | +| fpstateful | 31064 | +| attoparsec | 91216 | +| megaparsec | 219712 | +| parsec | 113152 | [basic]: https://hackage.haskell.org/package/flatparse/docs/FlatParse-Basic.html [stateful]: https://hackage.haskell.org/package/flatparse/docs/FlatParse-Stateful.html diff --git a/bench/Attoparsec.hs b/bench/Attoparsec.hs --- a/bench/Attoparsec.hs +++ b/bench/Attoparsec.hs @@ -7,11 +7,14 @@ isLatinLetter :: Char -> Bool isLatinLetter c = ('A' <= c && c <= 'Z') || ('a' <= c && c <= 'z') +skipWhile1 :: (Char -> Bool) -> Parser () +skipWhile1 pred = satisfy pred *> skipWhile pred + ws, open, close, ident, sexp :: Parser () -ws = skipMany (satisfy \c -> c == ' ' || c == '\n') +ws = skipWhile (\c -> c == ' ' || c == '\n') open = char '(' >> ws close = char ')' >> ws -ident = skipMany1 (satisfy isLatinLetter) <* ws +ident = skipWhile1 isLatinLetter <* ws sexp = (open *> skipMany1 sexp <* close) <|> ident runSexp = parseOnly sexp @@ -21,7 +24,7 @@ runLongws = parseOnly longws numeral, comma, numcsv :: Parser () -numeral = skipMany1 (satisfy \c -> '0' <= c && c <= '9') >> ws +numeral = skipWhile1 (\c -> '0' <= c && c <= '9') >> ws comma = char ',' >> ws numcsv = numeral >> skipMany1 (comma >> numeral) >> endOfInput runNumcsv = parseOnly numcsv diff --git a/bench/Bench.hs b/bench/Bench.hs --- a/bench/Bench.hs +++ b/bench/Bench.hs @@ -3,7 +3,7 @@ module Main where import Data.Primitive.ByteArray -import Gauge +import Test.Tasty.Bench import qualified Data.ByteString.Char8 as B diff --git a/bench/Megaparsec.hs b/bench/Megaparsec.hs --- a/bench/Megaparsec.hs +++ b/bench/Megaparsec.hs @@ -14,14 +14,17 @@ char8 :: Char -> Parser () char8 c = () <$ single (fromIntegral (ord c)) -satisfy8 :: (Char -> Bool) -> Parser () -satisfy8 f = () <$ satisfy (f . chr . fromIntegral) +skipWhile :: (Char -> Bool) -> Parser () +skipWhile f = () <$ takeWhileP Nothing (f . chr . fromIntegral) +skipWhile1 :: (Char -> Bool) -> Parser () +skipWhile1 f = () <$ takeWhile1P Nothing (f . chr . fromIntegral) + ws, open, close, ident, sexp :: Parser () -ws = skipMany (satisfy8 \c -> c == ' ' || c == '\n') +ws = skipWhile (\c -> c == ' ' || c == '\n') open = char8 '(' >> ws close = char8 ')' >> ws -ident = skipSome (satisfy8 isLatinLetter) <* ws +ident = skipWhile1 isLatinLetter <* ws sexp = (open *> skipSome sexp <* close) <|> ident runSexp = runParser sexp "" @@ -31,7 +34,7 @@ runLongws = runParser longws "" numeral, comma, numcsv :: Parser () -numeral = skipSome (satisfy8 \c -> '0' <= c && c <= '9') >> ws +numeral = skipWhile1 (\c -> '0' <= c && c <= '9') >> ws comma = single (fromIntegral (ord ',')) >> ws numcsv = numeral >> skipSome (comma >> numeral) >> eof runNumcsv = runParser numcsv "" diff --git a/flatparse.cabal b/flatparse.cabal --- a/flatparse.cabal +++ b/flatparse.cabal @@ -1,11 +1,11 @@ cabal-version: 1.12 --- This file has been generated from package.yaml by hpack version 0.37.0. +-- This file has been generated from package.yaml by hpack version 0.38.1. -- -- see: https://github.com/sol/hpack name: flatparse -version: 0.5.2.1 +version: 0.5.3.0 synopsis: High-performance parsing from strict bytestrings description: @Flatparse@ is a high-performance parsing library for strict bytestring input. See the README for more information: . @@ -23,10 +23,10 @@ , GHC == 8.8.4 , GHC == 8.10.7 , GHC == 9.0.2 - , GHC == 9.2.5 - , GHC == 9.4.4 - , GHC == 9.6.6 - , GHC == 9.8.3 + , GHC == 9.2.8 + , GHC == 9.4.8 + , GHC == 9.6.7 + , GHC == 9.8.4 extra-source-files: README.md @@ -164,11 +164,11 @@ , base >=4.7 && <5 , bytestring , flatparse - , gauge , integer-gmp , megaparsec , parsec , primitive + , tasty-bench , utf8-string >=1.0.2 && <1.1 default-language: Haskell2010 if flag(dump) diff --git a/src/FlatParse/Basic.hs b/src/FlatParse/Basic.hs --- a/src/FlatParse/Basic.hs +++ b/src/FlatParse/Basic.hs @@ -184,6 +184,7 @@ import qualified Control.Applicative import GHC.IO (IO(..), unsafeIOToST) +import GHC.Int import GHC.Exts import GHC.ForeignPtr import GHC.ST (ST(..)) @@ -506,12 +507,16 @@ _ -> -- >= 8 bytes of input: use efficient 8-byte scanning #if defined(WORDS_BIGENDIAN) -- big-endian ("L->R"): find leftmost null byte - let !x@(I# x#) = Common.zbytel'intermediate (I# (indexIntOffAddr# s 0#)) in + let !x@(I64# x#) = Common.zbytel'intermediate (I64# (indexInt64OffAddr# s 0#)) in #else -- little-endian ("R->L"): find rightmost null byte - let !x@(I# x#) = Common.zbyter'intermediate (I# (indexIntOffAddr# s 0#)) in + let !x@(I64# x#) = Common.zbyter'intermediate (I64# (indexInt64OffAddr# s 0#)) in #endif +#if MIN_VERSION_base(4,17,0) + case eqInt64# x# (intToInt64# 0#) of +#else case x# ==# 0# of +#endif 1# -> go fp eob s0 st sWord -- no 0x00 in next word _ -> -- 0x00 somewhere in next word #if defined(WORDS_BIGENDIAN) @@ -578,7 +583,7 @@ case Common.anyVarintProtobuf# eob s of (# (##) | #) -> Fail# st (# | (# w#, s#, bits# #) #) -> - case bits# ># 63# of + case bits# ># (WORD_SIZE_IN_BITS# -# 1#) of 0# -> OK# st (I# w#) s# _ -> Fail# st -- overflow {-# inline anyVarintProtobuf #-} diff --git a/src/FlatParse/Basic/Bytes.hs b/src/FlatParse/Basic/Bytes.hs --- a/src/FlatParse/Basic/Bytes.hs +++ b/src/FlatParse/Basic/Bytes.hs @@ -14,6 +14,7 @@ import qualified FlatParse.Common.Assorted as Common import Language.Haskell.TH import GHC.Exts +import GHC.Word -- | Read a sequence of bytes. This is a template function, you can use it as @$(bytes [3, 4, 5])@, -- for example, and the splice has type @Parser e ()@. For a non-TH variant see 'FlatParse.Basic.byteString'. @@ -28,7 +29,7 @@ -- The caller must guarantee that the input has enough bytes. bytesUnsafe :: [Word] -> Q Exp bytesUnsafe bytes = do - let !(leading, w8s) = Common.splitBytes bytes + let !(leading, w8s) = Common.splitBytes $ (fromIntegral :: Word -> Word64) <$> bytes !scanw8s = go w8s where go (w8:[] ) = [| word64Unsafe w8 |] go (w8:w8s) = [| word64Unsafe w8 >> $(go w8s) |] @@ -52,12 +53,16 @@ !l = length ws in [| scanPartial64# l w >> $scanw8s |] -scanPartial64# :: Int -> Word -> ParserT st e () -scanPartial64# (I# len) (W# w) = ParserT \fp eob s st -> - case indexWordOffAddr# s 0# of +scanPartial64# :: Int -> Word64 -> ParserT st e () +scanPartial64# (I# len) (W64# w) = ParserT \fp eob s st -> + case indexWord64OffAddr# s 0# of w' -> case uncheckedIShiftL# (8# -# len) 3# of - sh -> case uncheckedShiftL# w' sh of - w' -> case uncheckedShiftRL# w' sh of + sh -> case uncheckedShiftL64# w' sh of + w' -> case uncheckedShiftRL64# w' sh of +#if MIN_VERSION_base(4,17,0) + w' -> case eqWord64# w w' of +#else w' -> case eqWord# w w' of +#endif 1# -> OK# st () (plusAddr# s len) _ -> Fail# st diff --git a/src/FlatParse/Basic/Integers.hs b/src/FlatParse/Basic/Integers.hs --- a/src/FlatParse/Basic/Integers.hs +++ b/src/FlatParse/Basic/Integers.hs @@ -125,22 +125,20 @@ withAnyInt64 = withAnySized# 8# (\a i -> I64# (indexInt64OffAddr# a i)) {-# inline withAnyInt64 #-} --- TODO assumes 64-bit platform -- | Parse any 'Word' (native size) (CPS). withAnyWord :: (Word -> ParserT st e r) -> ParserT st e r withAnyWord p = ParserT \fp eob buf st -> case 8# <=# minusAddr# eob buf of 0# -> Fail# st _ -> let w# = indexWordOffAddr# buf 0# - in runParserT# (p (W# w#)) fp eob (plusAddr# buf 8#) st + in runParserT# (p (W# w#)) fp eob (plusAddr# buf SIZEOF_HSWORD#) st {-# inline withAnyWord #-} --- -- TODO assumes 64-bit platform -- | Parse any 'Int' (native size) (CPS). withAnyInt :: (Int -> ParserT st e r) -> ParserT st e r withAnyInt p = ParserT \fp eob buf st -> case 8# <=# minusAddr# eob buf of 0# -> Fail# st _ -> let i# = indexIntOffAddr# buf 0# - in runParserT# (p (I# i#)) fp eob (plusAddr# buf 8#) st + in runParserT# (p (I# i#)) fp eob (plusAddr# buf SIZEOF_HSWORD#) st {-# inline withAnyInt #-} -------------------------------------------------------------------------------- @@ -185,13 +183,11 @@ anyInt64 = withAnyInt64 pure {-# inline anyInt64 #-} --- TODO 'withAnyWord' assumes 64-bit platform -- | Parse any 'Word' (native size). anyWord :: ParserT st e Word anyWord = withAnyWord pure {-# inline anyWord #-} --- TODO 'withAnyInt' assumes 64-bit platform -- | Parse any 'Int' (native size). anyInt :: ParserT st e Int anyInt = withAnyInt pure diff --git a/src/FlatParse/Common/Assorted.hs b/src/FlatParse/Common/Assorted.hs --- a/src/FlatParse/Common/Assorted.hs +++ b/src/FlatParse/Common/Assorted.hs @@ -82,13 +82,13 @@ -- UTF conversions -------------------------------------------------------------------------------- -packBytes :: [Word] -> Word +packBytes :: [Word64] -> Word64 packBytes = fst . foldl' go (0, 0) where go (acc, shift) w | shift == 64 = error "packBytes: too many bytes" go (acc, shift) w = (unsafeShiftL (fromIntegral w) shift .|. acc, shift+8) -- TODO chunks into 8-bytes for 64-bit performance -splitBytes :: [Word] -> ([Word], [Word]) +splitBytes :: [Word64] -> ([Word64], [Word64]) splitBytes ws = case quotRem (length ws) 8 of (0, _) -> (ws, []) (_, r) -> (as, chunk8s bs) where diff --git a/src/FlatParse/Common/Numbers.hs b/src/FlatParse/Common/Numbers.hs --- a/src/FlatParse/Common/Numbers.hs +++ b/src/FlatParse/Common/Numbers.hs @@ -3,6 +3,7 @@ module FlatParse.Common.Numbers where import FlatParse.Common.Assorted ( shortInteger ) +import Language.Haskell.TH.Syntax (lift) import GHC.Exts import GHC.ForeignPtr @@ -68,12 +69,16 @@ (# n, s' #) | 1# <- eqAddr# s s' -> (# (##) | #) - -- Simple heuristic, 18 digits correspond to somewhere between 2^59 and 2^60, which is - -- well inside the 'IS' constructor. - | 1# <- minusAddr# s' s <=# 18# -> (# | (# shortInteger n, s' #) #) + -- Simple heuristic, using the largest number of digits that can be in an + -- 'Int#', such that we can use the 'IS' constructor. + | 1# <- minusAddr# s' s <=# maxDigitsInt -> (# | (# shortInteger n, s' #) #) | otherwise -> case BC8.readInteger (B.PS (ForeignPtr s fp) 0 (I# (minusAddr# s' s))) of Nothing -> (# (##) | #) Just (i, _) -> (# | (# i, s' #) #) + where + maxDigitsInt :: Int# + !(I# maxDigitsInt) = $(let p e = 10 ^ e - 1 <= toInteger (maxBound :: Int) + in lift $ last $ takeWhile p [0 :: Int ..]) {-# inline anyAsciiDecimalInteger# #-} -- | Parse a non-empty ASCII decimal digit sequence as a positive 'Int'. diff --git a/src/FlatParse/Stateful.hs b/src/FlatParse/Stateful.hs --- a/src/FlatParse/Stateful.hs +++ b/src/FlatParse/Stateful.hs @@ -190,6 +190,7 @@ import qualified Control.Applicative import GHC.IO (IO(..), unsafeIOToST) +import GHC.Int import GHC.Exts import GHC.ForeignPtr import GHC.ST (ST(..)) @@ -483,12 +484,16 @@ _ -> -- >= 8 bytes of input: use efficient 8-byte scanning #if defined(WORDS_BIGENDIAN) -- big-endian ("L->R"): find leftmost null byte - let !x@(I# x#) = Common.zbytel'intermediate (I# (indexIntOffAddr# s 0#)) in + let !x@(I64# x#) = Common.zbytel'intermediate (I64# (indexInt64OffAddr# s 0#)) in #else -- little-endian ("R->L"): find rightmost null byte - let !x@(I# x#) = Common.zbyter'intermediate (I# (indexIntOffAddr# s 0#)) in + let !x@(I64# x#) = Common.zbyter'intermediate (I64# (indexInt64OffAddr# s 0#)) in #endif +#if MIN_VERSION_base(4,17,0) + case eqInt64# x# (intToInt64# 0#) of +#else case x# ==# 0# of +#endif 1# -> go fp r n eob s0 st sWord -- no 0x00 in next word _ -> -- 0x00 somewhere in next word #if defined(WORDS_BIGENDIAN) @@ -555,7 +560,7 @@ case Common.anyVarintProtobuf# eob s of (# (##) | #) -> Fail# st (# | (# w#, s#, bits# #) #) -> - case bits# ># 63# of + case bits# ># (WORD_SIZE_IN_BITS# -# 1#) of 0# -> OK# st (I# w#) s# n _ -> Fail# st -- overflow {-# inline anyVarintProtobuf #-} diff --git a/src/FlatParse/Stateful/Bytes.hs b/src/FlatParse/Stateful/Bytes.hs --- a/src/FlatParse/Stateful/Bytes.hs +++ b/src/FlatParse/Stateful/Bytes.hs @@ -14,6 +14,7 @@ import qualified FlatParse.Common.Assorted as Common import Language.Haskell.TH import GHC.Exts +import GHC.Word -- | Read a sequence of bytes. This is a template function, you can use it as -- @$(bytes [3, 4, 5])@, for example, and the splice has type @Parser e @@ -29,7 +30,7 @@ -- The caller must guarantee that the input has enough bytes. bytesUnsafe :: [Word] -> Q Exp bytesUnsafe bytes = do - let !(leading, w8s) = Common.splitBytes bytes + let !(leading, w8s) = Common.splitBytes $ (fromIntegral :: Word -> Word64) <$> bytes !scanw8s = go w8s where go (w8:[] ) = [| word64Unsafe w8 |] go (w8:w8s) = [| word64Unsafe w8 >> $(go w8s) |] @@ -53,12 +54,16 @@ !l = length ws in [| scanPartial64# l w >> $scanw8s |] -scanPartial64# :: Int -> Word -> ParserT st r e () -scanPartial64# (I# len) (W# w) = ParserT \fp !r eob s n st -> - case indexWordOffAddr# s 0# of +scanPartial64# :: Int -> Word64 -> ParserT st r e () +scanPartial64# (I# len) (W64# w) = ParserT \fp !r eob s n st -> + case indexWord64OffAddr# s 0# of w' -> case uncheckedIShiftL# (8# -# len) 3# of - sh -> case uncheckedShiftL# w' sh of - w' -> case uncheckedShiftRL# w' sh of + sh -> case uncheckedShiftL64# w' sh of + w' -> case uncheckedShiftRL64# w' sh of +#if MIN_VERSION_base(4,17,0) + w' -> case eqWord64# w w' of +#else w' -> case eqWord# w w' of +#endif 1# -> OK# st () (plusAddr# s len) n _ -> Fail# st diff --git a/src/FlatParse/Stateful/Integers.hs b/src/FlatParse/Stateful/Integers.hs --- a/src/FlatParse/Stateful/Integers.hs +++ b/src/FlatParse/Stateful/Integers.hs @@ -127,22 +127,20 @@ withAnyInt64 = withAnySized# 8# (\a i -> I64# (indexInt64OffAddr# a i)) {-# inline withAnyInt64 #-} --- TODO assumes 64-bit platform -- | Parse any 'Word' (native size) (CPS). withAnyWord :: (Word -> ParserT st r e ret) -> ParserT st r e ret withAnyWord p = ParserT \fp !r eob buf n st -> case 8# <=# minusAddr# eob buf of 0# -> Fail# st _ -> let w# = indexWordOffAddr# buf 0# - in runParserT# (p (W# w#)) fp r eob (plusAddr# buf 8#) n st + in runParserT# (p (W# w#)) fp r eob (plusAddr# buf SIZEOF_HSWORD#) n st {-# inline withAnyWord #-} --- TODO assumes 64-bit platform -- | Parse any 'Int' (native size) (CPS). withAnyInt :: (Int -> ParserT st r e ret) -> ParserT st r e ret withAnyInt p = ParserT \fp !r eob buf n st -> case 8# <=# minusAddr# eob buf of 0# -> Fail# st _ -> let i# = indexIntOffAddr# buf 0# - in runParserT# (p (I# i#)) fp r eob (plusAddr# buf 8#) n st + in runParserT# (p (I# i#)) fp r eob (plusAddr# buf SIZEOF_HSWORD#) n st {-# inline withAnyInt #-} -------------------------------------------------------------------------------- @@ -187,13 +185,11 @@ anyInt64 = withAnyInt64 pure {-# inline anyInt64 #-} --- TODO 'withAnyWord' assumes 64-bit platform -- | Parse any 'Word' (native size). anyWord :: ParserT st r e Word anyWord = withAnyWord pure {-# inline anyWord #-} --- TODO 'withAnyInt' assumes 64-bit platform -- | Parse any 'Int' (native size). anyInt :: ParserT st r e Int anyInt = withAnyInt pure diff --git a/test/Test.hs b/test/Test.hs --- a/test/Test.hs +++ b/test/Test.hs @@ -20,6 +20,8 @@ import qualified Data.ByteString.UTF8 as UTF8 +#include "MachDeps.h" + main :: IO () main = hspec $ do basicSpec @@ -467,10 +469,12 @@ FB.anyVarintProtobuf `shouldParseWith` (B.pack [0b10000000, 0b00000001], 128) FB.anyVarintProtobuf `shouldParseWith` (B.pack [0b10010110, 0b00000001], 150) it "fails on overlong varint" $ do - -- 7 bits per byte = max 9 bytes in 64-bit word - let bs n = B.replicate n 0b10101010 - FB.anyVarintProtobuf `shouldParse` B.snoc (bs 8) 0b01010101 - FB.anyVarintProtobuf `shouldParseFail` B.snoc (bs 9) 0b01010101 + -- Variant has 7 bits payload per byte, so 'maxVarintBytesPerWord' gives + -- the max number of bytes we can parse into a (word-size) 'Int'. + let maxVarintBytesPerWord = WORD_SIZE_IN_BITS `div` 7 + bs n = B.replicate n 0b10101010 + FB.anyVarintProtobuf `shouldParse` B.snoc (bs (maxVarintBytesPerWord - 1)) 0b01010101 + FB.anyVarintProtobuf `shouldParseFail` B.snoc (bs maxVarintBytesPerWord) 0b01010101 describe "anyCString" $ do prop "parses arbitrary null-terminated bytestrings" $