nova-cache 0.9.0.0 → 0.10.0.0
raw patch · 6 files changed
+388/−7 lines, 6 filesdep +zstdPVP ok
version bump matches the API change (PVP)
Dependencies added: zstd
API changes (from Hackage documentation)
Files
- CHANGELOG.md +5/−1
- README.md +1/−1
- nova-cache.cabal +44/−3
- src/NovaCache/Zstd.hs +200/−0
- test/XzTest.hs +2/−2
- test/ZstdTest.hs +136/−0
CHANGELOG.md view
@@ -1,6 +1,10 @@ # Changelog -## Unreleased+## 0.10.0.0 - 2026-08-20++- **Bounded zstd: the new `NovaCache.Zstd`, a public `nova-cache:zstandard` sublibrary.** The modern caches (Cachix, attic, FlakeHub) serve NARs zstd-compressed, and a cache of our own wants the same: near-xz ratio on binaries with decompression an order of magnitude faster, cheap enough to compress at push time (`compress`, frame content size recorded; `defaultCompressionLevel` is libzstd's own 3). `decompress` takes the narinfo's declared NarSize as its output bound and fails past it, and `withZstdSource` decompresses a chunk source into a chunk source under the same limits, pairing with streaming NAR consumption - the xz discipline, ported. Two deliberate divergences from `NovaCache.Xz`, documented in place: decoder-state memory is capped by libzstd's default window limit (128 MiB; the binding exposes no tunable), and a truncated input yields truncated output at this layer - the signed NarSize and NarHash checks above are the arbiter of completeness. Concatenated frames decode as one output, as upstream's sink accepts. The `zstd` dependency bundles libzstd's C sources; no system library on any platform. (`zstandard`, not `zstd`: an in-package component name shadows the like-named dependency, so a sublibrary called `zstd` could never depend on the `zstd` binding.)++## 0.9.0.0 - 2026-08-20 - **`NovaCache.Xz` moves to the public `nova-cache:xz` sublibrary; the `xz` flag is gone.** A dependency's flag cannot be set from a consumer's `.cabal` file, so reaching the decoder forced a mirrored flag plus a matching `constraints: nova-cache +xz` in every downstream - two knobs that had to agree and that the solver could not see. `build-depends: nova-cache:xz` now expresses the need directly, and consumers without it still never build the bundled liblzma, keeping the 0.5.0.0 lesson. Builds that passed `-f xz` drop the flag and add the dependency; the module and its API are unchanged.
README.md view
@@ -116,7 +116,7 @@ cabal test ``` -Optional extras: `--flag server` builds the cache server, and the public `nova-cache:xz` sublibrary carries the bounded xz decoder (liblzma is bundled - no system library needed) - consumers depend on it with `build-depends: nova-cache:xz`. Requires GHC 9.14+ and cabal-install 3.10+.+Optional extras: `--flag server` builds the cache server, and the public `nova-cache:xz` and `nova-cache:zstandard` sublibraries carry the bounded codecs (liblzma and libzstd are bundled - no system libraries needed) - consumers depend on them with `build-depends: nova-cache:xz` and the like. Requires GHC 9.14+ and cabal-install 3.10+. ---
nova-cache.cabal view
@@ -1,13 +1,14 @@ cabal-version: 3.0 name: nova-cache-version: 0.9.0.0+version: 0.10.0.0 synopsis: Pure-first Nix binary cache protocol library description: A pure-first library implementing the Nix binary cache protocol - nix-base32, NAR serialization (whole-tree and streaming), narinfo parsing, Ed25519 signing, store path handling, and content- validation - with an optional WAI server, and bounded xz- decompression as the public @nova-cache:xz@ sublibrary.+ validation - with an optional WAI server, and bounded xz and zstd+ codecs as the public @nova-cache:xz@ and @nova-cache:zstandard@+ sublibraries. license: Apache-2.0 license-file: LICENSE@@ -96,6 +97,32 @@ , bytestring >= 0.11 && < 0.13 , lzma-static >= 5.2.5 && < 5.3 +-- The bounded zstd codec, the same solver-visible opt-in as xz.+-- Named zstandard, not zstd: an in-package component name shadows the+-- like-named external package in every build-depends of this package,+-- so a sublibrary called zstd could never depend on the zstd binding.+-- Consumers substituting from zstd caches (or pushing compressed)+-- depend on nova-cache:zstandard; the zstd package bundles libzstd's+-- C sources, so nobody else builds them.+library zstandard+ visibility: public+ exposed-modules: NovaCache.Zstd+ hs-source-dirs: src+ default-language: Haskell2010+ default-extensions:+ BangPatterns+ OverloadedStrings+ ghc-options:+ -Wall+ -Wcompat+ -Wincomplete-record-updates+ -Wincomplete-uni-patterns++ build-depends:+ base >= 4.22 && < 5+ , bytestring >= 0.11 && < 0.13+ , zstd >= 0.1 && < 0.2+ executable nova-cache-server if !flag(server) buildable: False@@ -153,6 +180,20 @@ base >= 4.22 && < 5 , bytestring >= 0.11 && < 0.13 , nova-cache:xz++test-suite nova-cache-zstd-test+ type: exitcode-stdio-1.0+ main-is: ZstdTest.hs+ hs-source-dirs: test+ default-language: Haskell2010+ default-extensions:+ OverloadedStrings+ ghc-options: -Wall -Wcompat++ build-depends:+ base >= 4.22 && < 5+ , bytestring >= 0.11 && < 0.13+ , nova-cache:zstandard source-repository head type: git
+ src/NovaCache/Zstd.hs view
@@ -0,0 +1,200 @@+-- | Bounded zstd decompression, and compression for the push path.+--+-- The modern caches (Cachix, attic, FlakeHub) serve NARs+-- zstd-compressed, and a cache of our own wants the same: near-xz+-- ratio on binaries with decompression an order of magnitude+-- faster. Substitution decompresses bytes that arrive from the+-- network BEFORE any hash can vouch for them, so the decoder must+-- not be steerable into unbounded allocation. The consumer knows+-- the narinfo's declared NarSize before decompressing:+-- decompression takes that bound and fails past it+-- ('zstdMaxOutputBytes'), so a small compressed input cannot expand+-- to arbitrary memory ahead of the hash check.+--+-- Decoder state is bounded differently from 'NovaCache.Xz': the+-- @zstd@ binding exposes no window-limit parameter, but libzstd+-- itself refuses any frame declaring a window past its default+-- @ZSTD_WINDOWLOG_LIMIT_DEFAULT@ (2^27, 128 MiB), so decoder memory+-- is capped by the library rather than by a caller-chosen number.+-- Take the tunable cap here too if the binding ever exposes+-- @ZSTD_d_windowLogMax@.+--+-- A truncated input yields truncated output at this layer rather+-- than an error: the binding's stream driver cannot observe+-- libzstd's more-input-expected state at end of input. The signed+-- NarSize and NarHash checks above this layer are the arbiter of+-- completeness - the same layering upstream relies on.+--+-- Everything here is IO: the binding's streaming interface is+-- IO-native, unlike lzma's lazy-ST driver under 'NovaCache.Xz'.+--+-- This module lives in the public @nova-cache:zstandard@ sublibrary+-- (a component named @zstd@ would shadow the @zstd@ dependency), the+-- same solver-visible opt-in as @nova-cache:xz@: the @zstd@ package+-- bundles libzstd's C sources (no system library on any platform),+-- and consumers that do not need the codec never build them.+module NovaCache.Zstd+ ( ZstdLimits (..),+ ZstdError (..),+ decompress,+ compress,+ defaultCompressionLevel,+ withZstdSource,+ )+where++import qualified Codec.Compression.Zstd as OneShot+import qualified Codec.Compression.Zstd.Streaming as S+import Control.Exception (Exception, throwIO)+import Data.ByteString (ByteString)+import qualified Data.ByteString as BS+import Data.IORef (IORef, newIORef, readIORef, writeIORef)+import Data.Word (Word64)++-- ---------------------------------------------------------------------------+-- Limits+-- ---------------------------------------------------------------------------++-- | What a decode run may cost. The bound is inclusive: output of+-- exactly 'zstdMaxOutputBytes' passes, one byte more fails - a+-- narinfo's NarSize is exact, so the declared size itself must be+-- reachable. Decoder-state memory is capped by libzstd's default+-- window limit (see the module header), not by a field here.+newtype ZstdLimits = ZstdLimits+ { -- | Maximum decompressed output, in bytes: the narinfo's declared+ -- NarSize.+ zstdMaxOutputBytes :: Word64+ }+ deriving (Eq, Show)++-- ---------------------------------------------------------------------------+-- Errors+-- ---------------------------------------------------------------------------++-- | Everything a bounded decode can refuse. The pure-shaped+-- 'decompress' returns these in 'Left'; the pull source behind+-- 'withZstdSource' throws them (see the 'Exception' instance).+data ZstdError+ = -- | The compressed stream is malformed (libzstd's error name,+ -- rendered with the failing call site).+ ZstdStreamError !String+ | -- | Decompressed output would exceed the bound (carried here).+ ZstdOutputOverBound !Word64+ deriving (Eq, Show)++-- | Thrown by the pull source 'withZstdSource' hands its+-- continuation; a chunk convention has no error channel, and a+-- throwing pull composes with consumers built around one.+instance Exception ZstdError++-- ---------------------------------------------------------------------------+-- Bounded decode+-- ---------------------------------------------------------------------------++-- | Decompress one zstd payload under the given limits. Output+-- stops accumulating the moment it would pass the bound, so a+-- high-expansion input costs at most the bound plus one decoder+-- buffer, never what it claims to hold. Concatenated frames decode+-- as one output, as upstream's decompression sink accepts.+decompress :: ZstdLimits -> ByteString -> IO (Either ZstdError ByteString)+decompress limits input = drive (Just input) 0 [] =<< S.decompress+ where+ bound = zstdMaxOutputBytes limits+ drive pending !produced acc step = case step of+ -- The whole input feeds on the first request; the second+ -- request gets the empty string, the driver's end-of-input+ -- signal.+ S.Consume supply -> case pending of+ Just bytes -> drive Nothing produced acc =<< supply bytes+ Nothing -> drive Nothing produced acc =<< supply BS.empty+ S.Produce out next+ | grown > bound -> pure (Left (ZstdOutputOverBound bound))+ | otherwise -> drive pending grown (out : acc) =<< next+ where+ grown = produced + fromIntegral (BS.length out)+ S.Done out+ | produced + fromIntegral (BS.length out) > bound ->+ pure (Left (ZstdOutputOverBound bound))+ | otherwise -> pure (Right (BS.concat (reverse (out : acc))))+ S.Error site name -> pure (Left (renderError site name))++-- | One libzstd failure in this module's error vocabulary.+renderError :: String -> String -> ZstdError+renderError site name = ZstdStreamError (site <> ": " <> name)++-- ---------------------------------------------------------------------------+-- Compression (push path)+-- ---------------------------------------------------------------------------++-- | Compress one payload at the given level (1 to the library+-- maximum). The produced frame records its content size, so+-- consumers with a one-shot decoder can allocate exactly. The+-- binding's one-shot API is pure and total for in-range levels.+compress :: Int -> ByteString -> ByteString+compress = OneShot.compress++-- | libzstd's own default (level 3): the ratio/speed point the+-- library authors tuned for, and far cheaper than xz at push time.+defaultCompressionLevel :: Int+defaultCompressionLevel = 3++-- ---------------------------------------------------------------------------+-- Streaming bounded decode (IO boundary)+-- ---------------------------------------------------------------------------++-- | What the pull source is doing between calls. The 'IORef'+-- holding this is the module's one piece of mutable state - the+-- same deliberate, documented boundary as the xz source.+data ZstdSourceState+ = ZstdStreaming !S.Result !Word64+ | ZstdDrained++-- | Decompress a chunk source into a chunk source, under the+-- limits. The continuation's pull yields decompressed chunks; the+-- empty chunk means end of output and repeats on further pulls.+-- The compressed source follows the same convention on its side.+-- Pairs with the incremental NAR parser and hashing, so a+-- substituter can fetch, decompress, hash, and unpack in one+-- bounded pass.+--+-- Limit violations and malformed input are thrown as 'ZstdError'+-- from the pull.+withZstdSource :: ZstdLimits -> IO ByteString -> (IO ByteString -> IO a) -> IO a+withZstdSource limits compressedSource consume = do+ start <- S.decompress+ stateRef <- newIORef (ZstdStreaming start 0)+ consume (pullDecompressed limits compressedSource stateRef)++-- | Produce the next decompressed chunk.+pullDecompressed :: ZstdLimits -> IO ByteString -> IORef ZstdSourceState -> IO ByteString+pullDecompressed limits compressedSource stateRef = advance =<< readIORef stateRef+ where+ bound = zstdMaxOutputBytes limits+ advance ZstdDrained = pure BS.empty+ advance (ZstdStreaming step produced) = case step of+ S.Consume supply -> do+ chunk <- compressedSource+ next <- supply chunk+ advance (ZstdStreaming next produced)+ S.Produce out nextAction -> do+ let grown = produced + fromIntegral (BS.length out)+ if grown > bound+ then do+ writeIORef stateRef ZstdDrained+ throwIO (ZstdOutputOverBound bound)+ else do+ next <- nextAction+ writeIORef stateRef (ZstdStreaming next grown)+ -- The driver may hand back an empty buffer at frame+ -- boundaries; returning it would read as end of output.+ if BS.null out+ then advance (ZstdStreaming next grown)+ else pure out+ S.Done out -> do+ writeIORef stateRef ZstdDrained+ if produced + fromIntegral (BS.length out) > bound+ then throwIO (ZstdOutputOverBound bound)+ else pure out+ S.Error site name -> do+ writeIORef stateRef ZstdDrained+ throwIO (renderError site name)
test/XzTest.hs view
@@ -1,5 +1,5 @@--- | Tests for the bounded xz decoder. A separate suite because it--- exists only under the @xz@ flag; the fixtures are real @xz -6@+-- | Tests for the bounded xz decoder. A separate suite because the+-- decoder lives in the nova-cache:xz sublibrary; the fixtures are real @xz -6@ -- output embedded as hex, so no external tool runs at test time. module Main (main) where
+ test/ZstdTest.hs view
@@ -0,0 +1,136 @@+-- | Tests for the bounded zstd codec. A separate suite because the+-- codec lives in the nova-cache:zstandard sublibrary; the compressed+-- fixtures come from the sublibrary's own pure 'Zstd.compress', so+-- no external tool runs at test time.+module Main (main) where++import Control.Exception (try)+import Data.ByteString (ByteString)+import qualified Data.ByteString as BS+import Data.IORef (newIORef, readIORef, writeIORef)+import qualified NovaCache.Zstd as Zstd+import System.Exit (exitFailure, exitSuccess)+import System.IO (hFlush, stdout)++-- ---------------------------------------------------------------------------+-- Harness (mirrors test/XzTest.hs)+-- ---------------------------------------------------------------------------++test :: String -> IO Bool -> IO Bool+test name action = do+ putStr (" " ++ name ++ "... ")+ hFlush stdout+ result <- action+ putStrLn (if result then "OK" else "FAILED")+ pure result++assertEqual :: (Eq a, Show a) => String -> a -> a -> IO Bool+assertEqual label expected actual+ | expected == actual = pure True+ | otherwise = do+ putStrLn ""+ putStrLn (" " ++ label)+ putStrLn (" expected: " ++ show expected)+ putStrLn (" actual: " ++ show actual)+ pure False++assertTrue :: String -> Bool -> IO Bool+assertTrue _ True = pure True+assertTrue label False = do+ putStrLn ""+ putStrLn (" " ++ label ++ ": expected True")+ pure False++-- ---------------------------------------------------------------------------+-- Fixtures+-- ---------------------------------------------------------------------------++-- | Compressible ASCII payload, 2048 bytes.+payload :: ByteString+payload = BS.concat (replicate 64 "nova-cache zstd fixture payload\n")++payloadSize :: Word+payloadSize = fromIntegral (BS.length payload)++limitsOf :: Word -> Zstd.ZstdLimits+limitsOf n = Zstd.ZstdLimits {Zstd.zstdMaxOutputBytes = fromIntegral n}++compressedPayload :: ByteString+compressedPayload = Zstd.compress Zstd.defaultCompressionLevel payload++-- | A pull source yielding the given chunks, then empty forever.+chunkSource :: [ByteString] -> IO (IO ByteString)+chunkSource chunks = do+ ref <- newIORef chunks+ pure $ do+ remaining <- readIORef ref+ case remaining of+ [] -> pure BS.empty+ (c : cs) -> writeIORef ref cs >> pure c++-- | Split a payload into bounded chunks so the streaming path sees+-- many small feeds, as a network body would deliver.+chunksOf :: Int -> ByteString -> [ByteString]+chunksOf n bs+ | BS.null bs = []+ | otherwise = BS.take n bs : chunksOf n (BS.drop n bs)++-- | Drain a decompressed pull source into one strict ByteString.+collectSource :: IO ByteString -> IO ByteString+collectSource pull = go []+ where+ go acc = do+ chunk <- pull+ if BS.null chunk+ then pure (BS.concat (reverse acc))+ else go (chunk : acc)++-- ---------------------------------------------------------------------------+-- Tests+-- ---------------------------------------------------------------------------++main :: IO ()+main = do+ putStrLn "zstd"+ results <-+ sequence+ [ test "roundtrip under the exact bound" $ do+ out <- Zstd.decompress (limitsOf payloadSize) compressedPayload+ assertEqual "roundtrip" (Right payload) out,+ test "one byte under the real size refuses" $ do+ out <- Zstd.decompress (limitsOf (payloadSize - 1)) compressedPayload+ assertEqual "over-bound" (Left (Zstd.ZstdOutputOverBound (fromIntegral (payloadSize - 1)))) out,+ test "garbage refuses" $ do+ out <- Zstd.decompress (limitsOf 64) "not a zstd stream"+ assertTrue "stream error" $ case out of+ Left (Zstd.ZstdStreamError _) -> True+ _ -> False,+ test "concatenated frames decode as one output" $ do+ let second = BS.concat (replicate 8 "second frame\n")+ joined = compressedPayload <> Zstd.compress Zstd.defaultCompressionLevel second+ out <- Zstd.decompress (limitsOf (payloadSize + fromIntegral (BS.length second))) joined+ assertEqual "concatenated" (Right (payload <> second)) out,+ test "trailing garbage after a frame refuses" $ do+ out <- Zstd.decompress (limitsOf (payloadSize + 64)) (compressedPayload <> "trailing garbage")+ assertTrue "trailing" $ case out of+ Left (Zstd.ZstdStreamError _) -> True+ _ -> False,+ test "empty input is empty output" $ do+ out <- Zstd.decompress (limitsOf 0) BS.empty+ assertEqual "empty" (Right BS.empty) out,+ test "source: chunked roundtrip" $ do+ source <- chunkSource (chunksOf 7 compressedPayload)+ out <- Zstd.withZstdSource (limitsOf payloadSize) source collectSource+ assertEqual "source roundtrip" payload out,+ test "source: over-bound throws" $ do+ source <- chunkSource (chunksOf 7 compressedPayload)+ out <- try (Zstd.withZstdSource (limitsOf (payloadSize - 1)) source collectSource)+ assertEqual "source over-bound" (Left (Zstd.ZstdOutputOverBound (fromIntegral (payloadSize - 1)))) out,+ test "source: garbage throws" $ do+ source <- chunkSource ["not a zstd stream"]+ out <- try (Zstd.withZstdSource (limitsOf 64) source collectSource) :: IO (Either Zstd.ZstdError ByteString)+ assertTrue "source garbage" $ case out of+ Left (Zstd.ZstdStreamError _) -> True+ _ -> False+ ]+ if and results then exitSuccess else exitFailure