diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,10 @@ # Changelog -## Unreleased +## 0.10.0.0 - 2026-08-20 + +- **Bounded zstd: the new `NovaCache.Zstd`, a public `nova-cache:zstandard` sublibrary.** The modern caches (Cachix, attic, FlakeHub) serve NARs zstd-compressed, and a cache of our own wants the same: near-xz ratio on binaries with decompression an order of magnitude faster, cheap enough to compress at push time (`compress`, frame content size recorded; `defaultCompressionLevel` is libzstd's own 3). `decompress` takes the narinfo's declared NarSize as its output bound and fails past it, and `withZstdSource` decompresses a chunk source into a chunk source under the same limits, pairing with streaming NAR consumption - the xz discipline, ported. Two deliberate divergences from `NovaCache.Xz`, documented in place: decoder-state memory is capped by libzstd's default window limit (128 MiB; the binding exposes no tunable), and a truncated input yields truncated output at this layer - the signed NarSize and NarHash checks above are the arbiter of completeness. Concatenated frames decode as one output, as upstream's sink accepts. The `zstd` dependency bundles libzstd's C sources; no system library on any platform. (`zstandard`, not `zstd`: an in-package component name shadows the like-named dependency, so a sublibrary called `zstd` could never depend on the `zstd` binding.) + +## 0.9.0.0 - 2026-08-20 - **`NovaCache.Xz` moves to the public `nova-cache:xz` sublibrary; the `xz` flag is gone.** A dependency's flag cannot be set from a consumer's `.cabal` file, so reaching the decoder forced a mirrored flag plus a matching `constraints: nova-cache +xz` in every downstream - two knobs that had to agree and that the solver could not see. `build-depends: nova-cache:xz` now expresses the need directly, and consumers without it still never build the bundled liblzma, keeping the 0.5.0.0 lesson. Builds that passed `-f xz` drop the flag and add the dependency; the module and its API are unchanged. diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -116,7 +116,7 @@ cabal test ``` -Optional extras: `--flag server` builds the cache server, and the public `nova-cache:xz` sublibrary carries the bounded xz decoder (liblzma is bundled - no system library needed) - consumers depend on it with `build-depends: nova-cache:xz`. Requires GHC 9.14+ and cabal-install 3.10+. +Optional extras: `--flag server` builds the cache server, and the public `nova-cache:xz` and `nova-cache:zstandard` sublibraries carry the bounded codecs (liblzma and libzstd are bundled - no system libraries needed) - consumers depend on them with `build-depends: nova-cache:xz` and the like. Requires GHC 9.14+ and cabal-install 3.10+. --- diff --git a/nova-cache.cabal b/nova-cache.cabal --- a/nova-cache.cabal +++ b/nova-cache.cabal @@ -1,13 +1,14 @@ cabal-version: 3.0 name: nova-cache -version: 0.9.0.0 +version: 0.10.0.0 synopsis: Pure-first Nix binary cache protocol library description: A pure-first library implementing the Nix binary cache protocol - nix-base32, NAR serialization (whole-tree and streaming), narinfo parsing, Ed25519 signing, store path handling, and content - validation - with an optional WAI server, and bounded xz - decompression as the public @nova-cache:xz@ sublibrary. + validation - with an optional WAI server, and bounded xz and zstd + codecs as the public @nova-cache:xz@ and @nova-cache:zstandard@ + sublibraries. license: Apache-2.0 license-file: LICENSE @@ -96,6 +97,32 @@ , bytestring >= 0.11 && < 0.13 , lzma-static >= 5.2.5 && < 5.3 +-- The bounded zstd codec, the same solver-visible opt-in as xz. +-- Named zstandard, not zstd: an in-package component name shadows the +-- like-named external package in every build-depends of this package, +-- so a sublibrary called zstd could never depend on the zstd binding. +-- Consumers substituting from zstd caches (or pushing compressed) +-- depend on nova-cache:zstandard; the zstd package bundles libzstd's +-- C sources, so nobody else builds them. +library zstandard + visibility: public + exposed-modules: NovaCache.Zstd + hs-source-dirs: src + default-language: Haskell2010 + default-extensions: + BangPatterns + OverloadedStrings + ghc-options: + -Wall + -Wcompat + -Wincomplete-record-updates + -Wincomplete-uni-patterns + + build-depends: + base >= 4.22 && < 5 + , bytestring >= 0.11 && < 0.13 + , zstd >= 0.1 && < 0.2 + executable nova-cache-server if !flag(server) buildable: False @@ -153,6 +180,20 @@ base >= 4.22 && < 5 , bytestring >= 0.11 && < 0.13 , nova-cache:xz + +test-suite nova-cache-zstd-test + type: exitcode-stdio-1.0 + main-is: ZstdTest.hs + hs-source-dirs: test + default-language: Haskell2010 + default-extensions: + OverloadedStrings + ghc-options: -Wall -Wcompat + + build-depends: + base >= 4.22 && < 5 + , bytestring >= 0.11 && < 0.13 + , nova-cache:zstandard source-repository head type: git diff --git a/src/NovaCache/Zstd.hs b/src/NovaCache/Zstd.hs new file mode 100644 --- /dev/null +++ b/src/NovaCache/Zstd.hs @@ -0,0 +1,200 @@ +-- | Bounded zstd decompression, and compression for the push path. +-- +-- The modern caches (Cachix, attic, FlakeHub) serve NARs +-- zstd-compressed, and a cache of our own wants the same: near-xz +-- ratio on binaries with decompression an order of magnitude +-- faster. Substitution decompresses bytes that arrive from the +-- network BEFORE any hash can vouch for them, so the decoder must +-- not be steerable into unbounded allocation. The consumer knows +-- the narinfo's declared NarSize before decompressing: +-- decompression takes that bound and fails past it +-- ('zstdMaxOutputBytes'), so a small compressed input cannot expand +-- to arbitrary memory ahead of the hash check. +-- +-- Decoder state is bounded differently from 'NovaCache.Xz': the +-- @zstd@ binding exposes no window-limit parameter, but libzstd +-- itself refuses any frame declaring a window past its default +-- @ZSTD_WINDOWLOG_LIMIT_DEFAULT@ (2^27, 128 MiB), so decoder memory +-- is capped by the library rather than by a caller-chosen number. +-- Take the tunable cap here too if the binding ever exposes +-- @ZSTD_d_windowLogMax@. +-- +-- A truncated input yields truncated output at this layer rather +-- than an error: the binding's stream driver cannot observe +-- libzstd's more-input-expected state at end of input. The signed +-- NarSize and NarHash checks above this layer are the arbiter of +-- completeness - the same layering upstream relies on. +-- +-- Everything here is IO: the binding's streaming interface is +-- IO-native, unlike lzma's lazy-ST driver under 'NovaCache.Xz'. +-- +-- This module lives in the public @nova-cache:zstandard@ sublibrary +-- (a component named @zstd@ would shadow the @zstd@ dependency), the +-- same solver-visible opt-in as @nova-cache:xz@: the @zstd@ package +-- bundles libzstd's C sources (no system library on any platform), +-- and consumers that do not need the codec never build them. +module NovaCache.Zstd + ( ZstdLimits (..), + ZstdError (..), + decompress, + compress, + defaultCompressionLevel, + withZstdSource, + ) +where + +import qualified Codec.Compression.Zstd as OneShot +import qualified Codec.Compression.Zstd.Streaming as S +import Control.Exception (Exception, throwIO) +import Data.ByteString (ByteString) +import qualified Data.ByteString as BS +import Data.IORef (IORef, newIORef, readIORef, writeIORef) +import Data.Word (Word64) + +-- --------------------------------------------------------------------------- +-- Limits +-- --------------------------------------------------------------------------- + +-- | What a decode run may cost. The bound is inclusive: output of +-- exactly 'zstdMaxOutputBytes' passes, one byte more fails - a +-- narinfo's NarSize is exact, so the declared size itself must be +-- reachable. Decoder-state memory is capped by libzstd's default +-- window limit (see the module header), not by a field here. +newtype ZstdLimits = ZstdLimits + { -- | Maximum decompressed output, in bytes: the narinfo's declared + -- NarSize. + zstdMaxOutputBytes :: Word64 + } + deriving (Eq, Show) + +-- --------------------------------------------------------------------------- +-- Errors +-- --------------------------------------------------------------------------- + +-- | Everything a bounded decode can refuse. The pure-shaped +-- 'decompress' returns these in 'Left'; the pull source behind +-- 'withZstdSource' throws them (see the 'Exception' instance). +data ZstdError + = -- | The compressed stream is malformed (libzstd's error name, + -- rendered with the failing call site). + ZstdStreamError !String + | -- | Decompressed output would exceed the bound (carried here). + ZstdOutputOverBound !Word64 + deriving (Eq, Show) + +-- | Thrown by the pull source 'withZstdSource' hands its +-- continuation; a chunk convention has no error channel, and a +-- throwing pull composes with consumers built around one. +instance Exception ZstdError + +-- --------------------------------------------------------------------------- +-- Bounded decode +-- --------------------------------------------------------------------------- + +-- | Decompress one zstd payload under the given limits. Output +-- stops accumulating the moment it would pass the bound, so a +-- high-expansion input costs at most the bound plus one decoder +-- buffer, never what it claims to hold. Concatenated frames decode +-- as one output, as upstream's decompression sink accepts. +decompress :: ZstdLimits -> ByteString -> IO (Either ZstdError ByteString) +decompress limits input = drive (Just input) 0 [] =<< S.decompress + where + bound = zstdMaxOutputBytes limits + drive pending !produced acc step = case step of + -- The whole input feeds on the first request; the second + -- request gets the empty string, the driver's end-of-input + -- signal. + S.Consume supply -> case pending of + Just bytes -> drive Nothing produced acc =<< supply bytes + Nothing -> drive Nothing produced acc =<< supply BS.empty + S.Produce out next + | grown > bound -> pure (Left (ZstdOutputOverBound bound)) + | otherwise -> drive pending grown (out : acc) =<< next + where + grown = produced + fromIntegral (BS.length out) + S.Done out + | produced + fromIntegral (BS.length out) > bound -> + pure (Left (ZstdOutputOverBound bound)) + | otherwise -> pure (Right (BS.concat (reverse (out : acc)))) + S.Error site name -> pure (Left (renderError site name)) + +-- | One libzstd failure in this module's error vocabulary. +renderError :: String -> String -> ZstdError +renderError site name = ZstdStreamError (site <> ": " <> name) + +-- --------------------------------------------------------------------------- +-- Compression (push path) +-- --------------------------------------------------------------------------- + +-- | Compress one payload at the given level (1 to the library +-- maximum). The produced frame records its content size, so +-- consumers with a one-shot decoder can allocate exactly. The +-- binding's one-shot API is pure and total for in-range levels. +compress :: Int -> ByteString -> ByteString +compress = OneShot.compress + +-- | libzstd's own default (level 3): the ratio/speed point the +-- library authors tuned for, and far cheaper than xz at push time. +defaultCompressionLevel :: Int +defaultCompressionLevel = 3 + +-- --------------------------------------------------------------------------- +-- Streaming bounded decode (IO boundary) +-- --------------------------------------------------------------------------- + +-- | What the pull source is doing between calls. The 'IORef' +-- holding this is the module's one piece of mutable state - the +-- same deliberate, documented boundary as the xz source. +data ZstdSourceState + = ZstdStreaming !S.Result !Word64 + | ZstdDrained + +-- | Decompress a chunk source into a chunk source, under the +-- limits. The continuation's pull yields decompressed chunks; the +-- empty chunk means end of output and repeats on further pulls. +-- The compressed source follows the same convention on its side. +-- Pairs with the incremental NAR parser and hashing, so a +-- substituter can fetch, decompress, hash, and unpack in one +-- bounded pass. +-- +-- Limit violations and malformed input are thrown as 'ZstdError' +-- from the pull. +withZstdSource :: ZstdLimits -> IO ByteString -> (IO ByteString -> IO a) -> IO a +withZstdSource limits compressedSource consume = do + start <- S.decompress + stateRef <- newIORef (ZstdStreaming start 0) + consume (pullDecompressed limits compressedSource stateRef) + +-- | Produce the next decompressed chunk. +pullDecompressed :: ZstdLimits -> IO ByteString -> IORef ZstdSourceState -> IO ByteString +pullDecompressed limits compressedSource stateRef = advance =<< readIORef stateRef + where + bound = zstdMaxOutputBytes limits + advance ZstdDrained = pure BS.empty + advance (ZstdStreaming step produced) = case step of + S.Consume supply -> do + chunk <- compressedSource + next <- supply chunk + advance (ZstdStreaming next produced) + S.Produce out nextAction -> do + let grown = produced + fromIntegral (BS.length out) + if grown > bound + then do + writeIORef stateRef ZstdDrained + throwIO (ZstdOutputOverBound bound) + else do + next <- nextAction + writeIORef stateRef (ZstdStreaming next grown) + -- The driver may hand back an empty buffer at frame + -- boundaries; returning it would read as end of output. + if BS.null out + then advance (ZstdStreaming next grown) + else pure out + S.Done out -> do + writeIORef stateRef ZstdDrained + if produced + fromIntegral (BS.length out) > bound + then throwIO (ZstdOutputOverBound bound) + else pure out + S.Error site name -> do + writeIORef stateRef ZstdDrained + throwIO (renderError site name) diff --git a/test/XzTest.hs b/test/XzTest.hs --- a/test/XzTest.hs +++ b/test/XzTest.hs @@ -1,5 +1,5 @@ --- | Tests for the bounded xz decoder. A separate suite because it --- exists only under the @xz@ flag; the fixtures are real @xz -6@ +-- | Tests for the bounded xz decoder. A separate suite because the +-- decoder lives in the nova-cache:xz sublibrary; the fixtures are real @xz -6@ -- output embedded as hex, so no external tool runs at test time. module Main (main) where diff --git a/test/ZstdTest.hs b/test/ZstdTest.hs new file mode 100644 --- /dev/null +++ b/test/ZstdTest.hs @@ -0,0 +1,136 @@ +-- | Tests for the bounded zstd codec. A separate suite because the +-- codec lives in the nova-cache:zstandard sublibrary; the compressed +-- fixtures come from the sublibrary's own pure 'Zstd.compress', so +-- no external tool runs at test time. +module Main (main) where + +import Control.Exception (try) +import Data.ByteString (ByteString) +import qualified Data.ByteString as BS +import Data.IORef (newIORef, readIORef, writeIORef) +import qualified NovaCache.Zstd as Zstd +import System.Exit (exitFailure, exitSuccess) +import System.IO (hFlush, stdout) + +-- --------------------------------------------------------------------------- +-- Harness (mirrors test/XzTest.hs) +-- --------------------------------------------------------------------------- + +test :: String -> IO Bool -> IO Bool +test name action = do + putStr (" " ++ name ++ "... ") + hFlush stdout + result <- action + putStrLn (if result then "OK" else "FAILED") + pure result + +assertEqual :: (Eq a, Show a) => String -> a -> a -> IO Bool +assertEqual label expected actual + | expected == actual = pure True + | otherwise = do + putStrLn "" + putStrLn (" " ++ label) + putStrLn (" expected: " ++ show expected) + putStrLn (" actual: " ++ show actual) + pure False + +assertTrue :: String -> Bool -> IO Bool +assertTrue _ True = pure True +assertTrue label False = do + putStrLn "" + putStrLn (" " ++ label ++ ": expected True") + pure False + +-- --------------------------------------------------------------------------- +-- Fixtures +-- --------------------------------------------------------------------------- + +-- | Compressible ASCII payload, 2048 bytes. +payload :: ByteString +payload = BS.concat (replicate 64 "nova-cache zstd fixture payload\n") + +payloadSize :: Word +payloadSize = fromIntegral (BS.length payload) + +limitsOf :: Word -> Zstd.ZstdLimits +limitsOf n = Zstd.ZstdLimits {Zstd.zstdMaxOutputBytes = fromIntegral n} + +compressedPayload :: ByteString +compressedPayload = Zstd.compress Zstd.defaultCompressionLevel payload + +-- | A pull source yielding the given chunks, then empty forever. +chunkSource :: [ByteString] -> IO (IO ByteString) +chunkSource chunks = do + ref <- newIORef chunks + pure $ do + remaining <- readIORef ref + case remaining of + [] -> pure BS.empty + (c : cs) -> writeIORef ref cs >> pure c + +-- | Split a payload into bounded chunks so the streaming path sees +-- many small feeds, as a network body would deliver. +chunksOf :: Int -> ByteString -> [ByteString] +chunksOf n bs + | BS.null bs = [] + | otherwise = BS.take n bs : chunksOf n (BS.drop n bs) + +-- | Drain a decompressed pull source into one strict ByteString. +collectSource :: IO ByteString -> IO ByteString +collectSource pull = go [] + where + go acc = do + chunk <- pull + if BS.null chunk + then pure (BS.concat (reverse acc)) + else go (chunk : acc) + +-- --------------------------------------------------------------------------- +-- Tests +-- --------------------------------------------------------------------------- + +main :: IO () +main = do + putStrLn "zstd" + results <- + sequence + [ test "roundtrip under the exact bound" $ do + out <- Zstd.decompress (limitsOf payloadSize) compressedPayload + assertEqual "roundtrip" (Right payload) out, + test "one byte under the real size refuses" $ do + out <- Zstd.decompress (limitsOf (payloadSize - 1)) compressedPayload + assertEqual "over-bound" (Left (Zstd.ZstdOutputOverBound (fromIntegral (payloadSize - 1)))) out, + test "garbage refuses" $ do + out <- Zstd.decompress (limitsOf 64) "not a zstd stream" + assertTrue "stream error" $ case out of + Left (Zstd.ZstdStreamError _) -> True + _ -> False, + test "concatenated frames decode as one output" $ do + let second = BS.concat (replicate 8 "second frame\n") + joined = compressedPayload <> Zstd.compress Zstd.defaultCompressionLevel second + out <- Zstd.decompress (limitsOf (payloadSize + fromIntegral (BS.length second))) joined + assertEqual "concatenated" (Right (payload <> second)) out, + test "trailing garbage after a frame refuses" $ do + out <- Zstd.decompress (limitsOf (payloadSize + 64)) (compressedPayload <> "trailing garbage") + assertTrue "trailing" $ case out of + Left (Zstd.ZstdStreamError _) -> True + _ -> False, + test "empty input is empty output" $ do + out <- Zstd.decompress (limitsOf 0) BS.empty + assertEqual "empty" (Right BS.empty) out, + test "source: chunked roundtrip" $ do + source <- chunkSource (chunksOf 7 compressedPayload) + out <- Zstd.withZstdSource (limitsOf payloadSize) source collectSource + assertEqual "source roundtrip" payload out, + test "source: over-bound throws" $ do + source <- chunkSource (chunksOf 7 compressedPayload) + out <- try (Zstd.withZstdSource (limitsOf (payloadSize - 1)) source collectSource) + assertEqual "source over-bound" (Left (Zstd.ZstdOutputOverBound (fromIntegral (payloadSize - 1)))) out, + test "source: garbage throws" $ do + source <- chunkSource ["not a zstd stream"] + out <- try (Zstd.withZstdSource (limitsOf 64) source collectSource) :: IO (Either Zstd.ZstdError ByteString) + assertTrue "source garbage" $ case out of + Left (Zstd.ZstdStreamError _) -> True + _ -> False + ] + if and results then exitSuccess else exitFailure