nova-cache 0.11.0.0 → 0.11.0.1

raw patch · 5 files changed

+118/−28 lines, 5 filesPVP ok

version bump matches the API change (PVP)

API changes (from Hackage documentation)

Files

• CHANGELOG.md +5/−0
• nova-cache.cabal +1/−1
• src/NovaCache/Bzip2.hs +87/−21
• src/NovaCache/Zstd.hs +3/−3
• test/Bzip2Test.hs +22/−3

@@ -1,5 +1,10 @@ # Changelog +## 0.11.0.1 - 2026-08-21++- **`NovaCache.Bzip2` accepts trailing bytes that do not begin another stream, matching upstream.** The decoder re-initialized at every clean stream end while input remained and fed whatever followed to libbz2's magic check, so a stray NUL, a newline, or any non-stream trailer after the last stream failed the decode. Upstream C++ Nix no longer has a bzip2 sink of its own: `compression.cc` drives every libarchive-supported codec through `ArchiveDecompressionSource`, and libarchive decodes the payload and ignores such a trailer. Measured against libarchive 3.8.2 driven exactly as Nix drives it (`filter_all` + `format_raw` + `format_empty`), four of seven cases diverged, so a historical `.nar.bz2` carrying one stray byte substituted under `nix copy` and failed here, falling through to a source build while the operator was told the cache object was corrupt. Trailing bytes that do begin a stream header are still decoded as a concatenated stream, and a truncated one is still refused: libarchive refuses those too, and silently truncating a real stream is the failure mode a bounded decoder exists to prevent. One divergence is deliberate and documented in place: a trailer that is a well-formed header carrying no block data (`BZh9` alone) is refused here and accepted by libarchive, which buffers past it, and refusing is the safer side of a case that does not arise in practice.+- **The bzip2 decoder is released on every exit instead of at a GC finalizer's leisure.** `decompress` and `withBzip2Source` handed their `bz_stream` to a `ForeignPtr` finalizer and never ended it themselves, so libbz2's block table (up to 3.6 MB at the largest block size, `malloc`'d and therefore invisible to the RTS allocation counter) stayed live on every path, including clean completion. Because that memory creates no GC pressure of its own and is not bounded by `+RTS -M`, sequential decodes accumulated decoder state in proportion to how rarely the collector ran, and a large nursery made the accumulation large. Both entry points now finalize in a bracket, the discipline `NovaCache.Zstd` already had and the one `NovaCache.Xz` documents as unavailable to it because `lzma-static` exposes no live-stream teardown. Concatenated streams within a single decode were never affected.+ ## 0.11.0.0 - 2026-08-21  - **A failed `withXzSource` or `withZstdSource` pull latches any exception, matching `NovaCache.Bzip2`.** Both sources held only their own decoder errors (`XzError`, `ZstdError`) for replay on later pulls; an exception thrown by the compressed source itself - an HTTP failure, a capped body source refusing to read on - left the state untouched, so a consumer that caught the failure and pulled again at exactly a stream or frame boundary of a concatenated payload could read the empty chunk, the clean-end signal, and take a prefix for complete output. Any exception escaping a pull now marks the transfer unfinishable and is rethrown on every later pull, the breadth `NovaCache.Bzip2` shipped with.

@@ -1,6 +1,6 @@ cabal-version:      3.0 name:               nova-cache-version:            0.11.0.0+version:            0.11.0.1 synopsis:           Pure-first Nix binary cache protocol library description:   A pure-first library implementing the Nix binary cache protocol -

@@ -21,10 +21,12 @@ -- about 4 MiB at the format's largest block size (900k) - so decoder -- memory is a constant of the format, not a parameter. ----- Concatenated streams decode as one output: upstream's bzip2--- decompression sink re-initializes the decoder at stream end while--- input remains, and this module does the same.  Trailing bytes that--- do not start a valid stream are refused, as is truncated input.+-- Concatenated streams decode as one output, and trailing bytes that+-- do not begin another stream end the output rather than failing it:+-- upstream C++ Nix decompresses bzip2 through libarchive, which does+-- both.  Truncated input is still refused, including a truncated+-- concatenated stream, since silently truncating output is the+-- failure mode a bounded decoder exists to avoid. -- -- Everything here is IO: the decoder is libbz2, driven over the FFI. -- The binding goes directly over @bzip2-clib@ (nothing but the@@ -44,15 +46,15 @@   ) where -import Control.Exception (Exception, SomeException, throwIO, toException, try)+import Control.Exception (Exception, SomeException, finally, throwIO, toException, try) import Data.ByteString (ByteString) import qualified Data.ByteString as BS import Data.ByteString.Unsafe (unsafeUseAsCStringLen) import Data.IORef (IORef, newIORef, readIORef, writeIORef) import Data.Maybe (fromMaybe)-import Data.Word (Word64)+import Data.Word (Word64, Word8) import Foreign.C.Types (CChar, CInt (..), CUInt (..))-import Foreign.ForeignPtr (FinalizerPtr, ForeignPtr, newForeignPtr, withForeignPtr)+import Foreign.ForeignPtr (FinalizerPtr, ForeignPtr, finalizeForeignPtr, newForeignPtr, withForeignPtr) import Foreign.Marshal.Alloc (alloca, allocaBytes) import Foreign.Ptr (Ptr) import Foreign.Storable (peek)@@ -107,7 +109,14 @@   opened <- newDecoder   case opened of     Left err -> pure (Left err)-    Right decoder -> do+    Right decoder ->+      -- The decoder owns malloc'd libbz2 state (a block table up to+      -- 3.6 MB) that the RTS cannot see, so it creates no GC pressure+      -- and would otherwise be released at a finalizer's leisure.+      -- Release it here on every exit, as the zstd codec does.+      collectFrom decoder `finally` finalizeForeignPtr (decoderStream decoder)+  where+    collectFrom decoder = do       -- The IORef makes the whole input a one-shot chunk source       -- (input, then the empty end marker), so the strict path       -- drives the same engine as the streaming one.@@ -117,7 +126,7 @@             writeIORef remainingRef BS.empty             pure held       collect source decoder []-  where+     collect source decoder acc = do       outcome <- nextDecodedChunk limits source decoder       case outcome of@@ -155,8 +164,19 @@ withBzip2Source :: Bzip2Limits -> IO ByteString -> (IO ByteString -> IO a) -> IO a withBzip2Source limits compressedSource consume = do   opened <- newDecoder-  stateRef <- newIORef (either (Bzip2Failed . toException) Bzip2Streaming opened)-  consume (pullDecompressed limits compressedSource stateRef)+  case opened of+    Left err -> do+      stateRef <- newIORef (Bzip2Failed (toException err))+      consume (pullDecompressed limits compressedSource stateRef)+    Right decoder -> do+      stateRef <- newIORef (Bzip2Streaming decoder)+      -- Deterministic teardown on every exit - clean end, bound+      -- violation, decode failure, or an exception in the consumer.+      -- libbz2's block table is malloc'd and invisible to the RTS, so+      -- leaving it to the ForeignPtr finalizer lets sequential decodes+      -- accumulate decoder state in proportion to how rarely the GC runs.+      consume (pullDecompressed limits compressedSource stateRef)+        `finally` finalizeForeignPtr (decoderStream decoder)  -- | Produce the next decompressed chunk. pullDecompressed :: Bzip2Limits -> IO ByteString -> IORef Bzip2SourceState -> IO ByteString@@ -245,13 +265,7 @@ nextDecodedChunk limits compressedSource = advance   where     advance decoder = case decoderPhase decoder of-      AtStreamBoundary-        | BS.null (decoderLeftover decoder) -> do-            chunk <- compressedSource-            if BS.null chunk-              then pure (Right Nothing)-              else reopen decoder {decoderLeftover = chunk}-        | otherwise -> reopen decoder+      AtStreamBoundary -> continueAfterStream decoder       MidStream         | BS.null (decoderLeftover decoder) -> do             chunk <- compressedSource@@ -260,9 +274,37 @@               else advance decoder {decoderLeftover = chunk}         | otherwise -> decodeStep decoder -    -- Input after a clean stream end: re-initialize and decode it as-    -- the next concatenated stream.  Garbage fails the re-initialized-    -- decoder's magic check, which is the trailing-garbage refusal.+    -- Input after a clean stream end.  Upstream C++ Nix decompresses+    -- bzip2 through libarchive (its own BzipDecompressionSink is gone),+    -- and libarchive decodes the payload and ignores trailing bytes that+    -- do not begin another stream: a single stray NUL or newline after+    -- the last stream substitutes fine under `nix copy` and used to fail+    -- here.  Bytes that DO begin a stream header are decoded as a+    -- concatenated stream, and a truncated one still fails, since+    -- libarchive refuses those too and silently truncating a real stream+    -- is the failure mode worth keeping.  One measured divergence+    -- remains: a trailer that is a well-formed header carrying no block+    -- data (`BZh9` alone) is refused here and accepted by libarchive,+    -- which buffers past it - refusing is the safer side of a case that+    -- does not arise in practice.+    continueAfterStream decoder = do+      trailing <- fillToHeader (decoderLeftover decoder)+      if startsStream trailing+        then reopen decoder {decoderLeftover = trailing}+        else pure (Right Nothing)++    -- Top the held bytes up to a full stream header, so the decision+    -- above is never taken on a short read that more input completes.+    fillToHeader held+      | BS.length held >= streamHeaderLength = pure held+      | otherwise = do+          chunk <- compressedSource+          if BS.null chunk+            then pure held+            else fillToHeader (held <> chunk)++    -- Re-initialize and decode the trailing bytes as the next+    -- concatenated stream.     reopen decoder = do       status <- withForeignPtr (decoderStream decoder) cDecompressReinit       if status == statusOk@@ -290,6 +332,30 @@                 if BS.null outChunk                   then advance continued                   else pure (Right (Just (outChunk, continued)))++-- | Does this begin a bzip2 stream: the @BZh@ magic followed by a+-- block-size digit?  The trailing-bytes decision rests on this, so it+-- reads only the header and never consumes.+startsStream :: ByteString -> Bool+startsStream bytes =+  streamMagic `BS.isPrefixOf` bytes+    && case BS.indexMaybe bytes (BS.length streamMagic) of+      Just level -> level >= minBlockSizeDigit && level <= maxBlockSizeDigit+      Nothing -> False++-- | The bytes every bzip2 stream opens with, before the block-size digit.+streamMagic :: ByteString+streamMagic = "BZh"++-- | A full stream header: the magic and the block-size digit after it.+streamHeaderLength :: Int+streamHeaderLength = BS.length streamMagic + 1++-- | @\'1\'@ and @\'9\'@: the block-size digits bzip2 defines, in+-- hundreds of kilobytes.+minBlockSizeDigit, maxBlockSizeDigit :: Word8+minBlockSizeDigit = 0x31+maxBlockSizeDigit = 0x39  -- | The one place the output bound is enforced: the produced count -- grown by a chunk, refused past the bound.  Inclusive - reaching

@@ -16,7 +16,7 @@ -- properties the driver cannot give: -- -- * The decompression context is created and freed in a bracket---   ('withDecoder'), so its window buffer - sized by the incoming+--   (@withDecoder@), so its window buffer - sized by the incoming --   frame header, i.e. by the peer, up to libzstd's 128 MiB default --   ceiling - is released deterministically on every exit: success, --   bound violation, corrupt frame, or an exception in the@@ -146,7 +146,7 @@ newtype ZstdCompressionLevel = ZstdCompressionLevel Int   deriving (Eq, Ord, Show) --- | Validate a level into 'ZstdCompressionLevel'; 'Nothing' outside+-- | Validate a level into 't:ZstdCompressionLevel'; 'Nothing' outside -- the accepted range. zstdCompressionLevel :: Int -> Maybe ZstdCompressionLevel zstdCompressionLevel level@@ -177,7 +177,7 @@  -- | Compress one payload at the given level.  The produced frame -- records its content size, so consumers with a one-shot decoder can--- allocate exactly.  Total by construction: 'ZstdCompressionLevel'+-- allocate exactly.  Total by construction: 't:ZstdCompressionLevel' -- cannot hold a level the binding's pure one-shot API would reject. compress :: ZstdCompressionLevel -> ByteString -> ByteString compress (ZstdCompressionLevel level) = OneShot.compress level

@@ -163,9 +163,28 @@           -- remains; two streams back-to-back are one valid input.           outcome <- Bzip2.decompress openLimits (textBz2 <> textBz2)           assertEqual "two text streams" (Right (textPlain <> textPlain)) outcome,-        test "trailing garbage after the stream is refused" $ do-          outcome <- Bzip2.decompress openLimits (textBz2 <> "garbage!")-          assertTrue "trailing garbage" (isStreamError outcome),+        -- Upstream Nix decompresses bzip2 through libarchive, which+        -- ignores whatever follows the last stream unless it begins+        -- another one.  Measured against libarchive 3.8.2 driven exactly+        -- as Nix drives it (filter_all + format_raw + format_empty): all+        -- four of these decode to the payload there, and all four were+        -- refused here before.+        test "trailing bytes that are not a stream end the output" $ do+          let trailers = ["garbage!", "\0\0\0\0", "\n", "BZh"]+          results <-+            mapM+              ( \trailer -> do+                  outcome <- Bzip2.decompress openLimits (textBz2 <> trailer)+                  assertEqual ("trailer " <> show trailer) (Right textPlain) outcome+              )+              trailers+          pure (and results),+        test "a truncated concatenated stream is still refused" $ do+          -- The safe half of the rule: bytes that DO begin a stream are+          -- decoded as one, and a truncated one fails rather than+          -- silently truncating the output.+          outcome <- Bzip2.decompress openLimits (textBz2 <> BS.take 40 textBz2)+          assertTrue "truncated second stream" (isStreamError outcome),         test "withBzip2Source decompresses a chunked source" $ do           source <- listSource (chunksOf 7 textBz2)           out <- Bzip2.withBzip2Source openLimits source drainSource