nova-cache 0.2.4.1 → 0.3.0.0
raw patch · 9 files changed
+272/−64 lines, 9 files
Files
- CHANGELOG.md +41/−0
- README.md +22/−10
- exe/Main.hs +102/−29
- nova-cache.cabal +2/−1
- src/NovaCache/Compression.hs +9/−2
- src/NovaCache/NAR.hs +10/−2
- src/NovaCache/Store.hs +20/−8
- test/CompressionTest.hs +29/−7
- test/Main.hs +37/−5
CHANGELOG.md view
@@ -1,5 +1,46 @@ # Changelog +## 0.3.0.0 — 2026-02-28++### Breaking changes++- **`decompressXz`** signature changed from `ByteString -> ByteString` to+ `ByteString -> IO (Either String ByteString)` — catches lzma exceptions+ instead of crashing on malformed input+- **`writeNarInfo`** / **`writeNar`** now return `IO Bool` instead of `IO ()` —+ `False` indicates a rejected path (traversal, empty)++### Security++- **Request body size limit** — PUT endpoints reject bodies over 100 MB with+ 413 Payload Too Large (prevents memory exhaustion)+- **Constant-time auth comparison** — API key check uses `constEq` from+ `Data.ByteArray` instead of `==` (prevents timing side-channel)++### Bug fixes++- **Safe UTF-8 decoding in NAR deserializer** — `decodeUtf8` replaced with+ `decodeUtf8'`; malformed UTF-8 in symlink targets or directory entry names+ now returns a parse error instead of throwing++### New features++- **`GET /narinfo-hashes`** endpoint — returns all cached narinfo hashes as+ newline-delimited text, enabling efficient cache diffing+- **`listNarInfoHashes`** function added to `NovaCache.Store`++### Improvements++- Server logs warnings to stderr when narinfo signing fails (parse error or+ sign error) instead of silently returning unsigned body+- Server returns 400 Bad Request when PUT paths fail sanitization instead of+ silently discarding the upload+- README: license badge and footer corrected from MIT to BSD-3-Clause+- README: `parallel` input documented in seed action table+- README: `CACHE_API_KEY` and `SIGNING_KEY_FILE` env vars documented+- Seed action: replaced per-path HEAD checks with single `GET /narinfo-hashes`+ call and local diff for dramatically faster cache seeding+ ## 0.2.4.1 — 2026-02-26 - License changed from MIT to BSD-3-Clause
README.md view
@@ -8,7 +8,7 @@ [](https://github.com/Novavero-AI/nova-cache/actions/workflows/ci.yml) [](https://hackage.haskell.org/package/nova-cache) -+ </p> </div>@@ -222,11 +222,21 @@ PORT=5000 NIX_CACHE_DIR=./nix-cache nova-cache-server ``` +### Environment Variables++| Variable | Description |+|----------|-------------|+| `PORT` | Server listen port (default: 5000) |+| `NIX_CACHE_DIR` | Store directory (default: `./nix-cache`) |+| `CACHE_API_KEY` | Bearer token required for PUT requests. Omit for open writes. |+| `SIGNING_KEY_FILE` | Path to Ed25519 secret key file for server-side narinfo signing. |+ ### Endpoints | Method | Path | Description | |--------|------|-------------| | `GET` | `/nix-cache-info` | Cache metadata (StoreDir, WantMassQuery, Priority) |+| `GET` | `/narinfo-hashes` | All cached narinfo hashes (newline-delimited) | | `GET` | `/<hash>.narinfo` | Fetch narinfo by store path hash | | `GET` | `/nar/<file>` | Fetch compressed NAR file | | `PUT` | `/<hash>.narinfo` | Upload narinfo |@@ -285,7 +295,7 @@ ``` - **9 modules**, 7 pure + 2 at the IO boundary (Compression optional via flag)-- **54 core tests + 3 compression tests**, hand-rolled harness, no framework dependencies+- **58 core tests + 4 compression tests**, hand-rolled harness, no framework dependencies - **Zero partial functions** — total by construction - **Strict by default** — bang patterns on all data fields @@ -358,18 +368,19 @@ ```haskell compressXz :: ByteString -> ByteString-decompressXz :: ByteString -> ByteString+decompressXz :: ByteString -> IO (Either String ByteString) ``` ### Store ```haskell-newFileStore :: FilePath -> IO FileStore-readNarInfo :: FileStore -> Text -> IO (Maybe ByteString)-writeNarInfo :: FileStore -> Text -> ByteString -> IO ()-readNar :: FileStore -> Text -> IO (Maybe ByteString)-writeNar :: FileStore -> Text -> ByteString -> IO ()-getCacheInfo :: FileStore -> (Text, Bool, Int)+newFileStore :: FilePath -> IO FileStore+readNarInfo :: FileStore -> Text -> IO (Maybe ByteString)+writeNarInfo :: FileStore -> Text -> ByteString -> IO Bool+readNar :: FileStore -> Text -> IO (Maybe ByteString)+writeNar :: FileStore -> Text -> ByteString -> IO Bool+listNarInfoHashes :: FileStore -> IO [Text]+getCacheInfo :: FileStore -> (Text, Bool, Int) ``` Full Haddock documentation is available on [Hackage](https://hackage.haskell.org/package/nova-cache).@@ -392,6 +403,7 @@ | `cache-url` | yes | Base URL of the nova-cache server | | `api-key` | yes | Bearer token for authenticating uploads | | `paths` | no | Explicit store paths (space-separated). Defaults to all paths from `shell.nix` / `default.nix` |+| `parallel` | no | Max concurrent uploads (default: 32) | The action resolves Nix store path closures, exports them to a local binary cache, and uploads all narinfo + NAR files to the server. Works with any CI that has Nix installed. @@ -417,5 +429,5 @@ --- <p align="center">- <sub>MIT License · <a href="https://github.com/Novavero-AI">Novavero AI</a></sub>+ <sub>BSD-3-Clause · <a href="https://github.com/Novavero-AI">Novavero AI</a></sub> </p>
exe/Main.hs view
@@ -1,5 +1,7 @@ module Main (main) where +import Data.Bifunctor (first)+import Data.ByteArray (constEq) import qualified Data.ByteString as BS import qualified Data.ByteString.Lazy as BL import Data.Maybe (fromMaybe)@@ -7,13 +9,26 @@ import qualified Data.Text as T import qualified Data.Text.Encoding as TE import qualified Network.HTTP.Types as HTTP-import Network.Wai (Application, Request, Response, ResponseReceived, pathInfo, requestHeaders, requestMethod, responseLBS, strictRequestBody)+import Network.Wai+ ( Application,+ Request,+ RequestBodyLength (..),+ Response,+ ResponseReceived,+ pathInfo,+ requestBodyLength,+ requestHeaders,+ requestMethod,+ responseLBS,+ strictRequestBody,+ ) import qualified Network.Wai.Handler.Warp as Warp import NovaCache.NarInfo (NarInfo (..), parseNarInfo, renderNarInfo) import NovaCache.Signing (SecretKey, parseSecretKey, sign)-import NovaCache.Store (FileStore, getCacheInfo, newFileStore, readNar, readNarInfo, writeNar, writeNarInfo)+import NovaCache.Store (FileStore, getCacheInfo, listNarInfoHashes, newFileStore, readNar, readNarInfo, writeNar, writeNarInfo) import NovaCache.Validate (validateNarInfo) import System.Environment (getArgs, lookupEnv)+import System.IO (hPutStrLn, stderr) import Text.Read (readMaybe) -- ---------------------------------------------------------------------------@@ -36,6 +51,10 @@ signingKeyEnvVar :: String signingKeyEnvVar = "SIGNING_KEY_FILE" +-- | Maximum allowed request body size (100 MB).+maxBodySize :: Int+maxBodySize = 100 * 1024 * 1024+ -- --------------------------------------------------------------------------- -- Server configuration -- ---------------------------------------------------------------------------@@ -86,8 +105,8 @@ loadSigningKey :: Maybe FilePath -> IO (Maybe SecretKey) loadSigningKey Nothing = pure Nothing loadSigningKey (Just path) = do- contents <- T.strip . TE.decodeUtf8 <$> BS.readFile path- case parseSecretKey contents of+ raw <- BS.readFile path+ case first show (TE.decodeUtf8' raw) >>= parseSecretKey . T.strip of Left err -> do putStrLn ("WARNING: failed to load signing key: " ++ err) pure Nothing@@ -103,6 +122,11 @@ -- GET /nix-cache-info ("GET", ["nix-cache-info"]) -> respond (responseLBS HTTP.status200 textHeaders (BL.fromStrict (renderCacheInfo (cfgStore cfg))))+ -- GET /narinfo-hashes+ ("GET", ["narinfo-hashes"]) -> do+ hashes <- listNarInfoHashes (cfgStore cfg)+ let body = TE.encodeUtf8 (T.unlines hashes)+ respond (responseLBS HTTP.status200 textHeaders (BL.fromStrict body)) -- GET /<hash>.narinfo ("GET", [hashNarinfo]) | Just hashKey <- T.stripSuffix ".narinfo" hashNarinfo -> do@@ -124,28 +148,68 @@ ("PUT", [hashNarinfo]) | Just hashKey <- T.stripSuffix ".narinfo" hashNarinfo -> requireAuth cfg req respond $ do- body <- BL.toStrict <$> strictRequestBody req- case parseNarInfo (TE.decodeUtf8 body) of- Left err ->- respond (badRequest (T.pack err))- Right ni -> case validateNarInfo ni of- Left errs ->- respond (badRequest (T.unlines (map (T.pack . show) errs)))- Right _ -> do- let signed = signNarInfo (cfgSigningKey cfg) body- writeNarInfo (cfgStore cfg) hashKey signed- respond (responseLBS HTTP.status200 textHeaders "ok")+ bodyResult <- readBodyLimited req+ case bodyResult of+ Nothing ->+ respond (responseLBS HTTP.status413 textHeaders "request body too large")+ Just body -> case decodeAndValidate body of+ Left err ->+ respond (badRequest err)+ Right ni -> do+ signed <- signNarInfo (cfgSigningKey cfg) ni+ ok <- writeNarInfo (cfgStore cfg) hashKey signed+ if ok+ then respond (responseLBS HTTP.status200 textHeaders "ok")+ else respond (badRequest "invalid path") -- PUT /nar/<file> (auth required) ("PUT", ["nar", fileName]) -> requireAuth cfg req respond $ do- body <- BL.toStrict <$> strictRequestBody req- writeNar (cfgStore cfg) fileName body- respond (responseLBS HTTP.status200 textHeaders "ok")+ bodyResult <- readBodyLimited req+ case bodyResult of+ Nothing ->+ respond (responseLBS HTTP.status413 textHeaders "request body too large")+ Just body -> do+ ok <- writeNar (cfgStore cfg) fileName body+ if ok+ then respond (responseLBS HTTP.status200 textHeaders "ok")+ else respond (badRequest "invalid path") -- Fallback _ -> respond (responseLBS HTTP.status404 textHeaders "not found") -- ---------------------------------------------------------------------------+-- Validation pipeline+-- ---------------------------------------------------------------------------++-- | Decode a raw narinfo body and validate it in a single pure pipeline.+--+-- Composes UTF-8 decoding, narinfo parsing, and field validation. Returns+-- the validated 'NarInfo' on success, or a user-facing error message.+decodeAndValidate :: BS.ByteString -> Either Text NarInfo+decodeAndValidate body = do+ decoded <- first (const "request body is not valid UTF-8") (TE.decodeUtf8' body)+ ni <- first T.pack (parseNarInfo decoded)+ first (T.unlines . map (T.pack . show)) (validateNarInfo ni)++-- ---------------------------------------------------------------------------+-- Request body limiting+-- ---------------------------------------------------------------------------++-- | Read the request body, rejecting payloads over 'maxBodySize'.+--+-- Returns 'Nothing' if the declared @Content-Length@ exceeds the limit.+-- For chunked transfers the body is read and checked after the fact.+readBodyLimited :: Request -> IO (Maybe BS.ByteString)+readBodyLimited req = case requestBodyLength req of+ KnownLength len+ | len > fromIntegral maxBodySize -> pure Nothing+ _ -> do+ body <- BL.toStrict <$> strictRequestBody req+ if BS.length body > maxBodySize+ then pure Nothing+ else pure (Just body)++-- --------------------------------------------------------------------------- -- Auth -- --------------------------------------------------------------------------- @@ -153,12 +217,14 @@ -- -- If no key is configured, all writes are permitted (open mode). -- Otherwise the request must carry @Authorization: Bearer \<key\>@.+-- Uses constant-time comparison to prevent timing attacks. requireAuth :: Config -> Request -> (Response -> IO ResponseReceived) -> IO ResponseReceived -> IO ResponseReceived requireAuth cfg req respond action = case cfgApiKey cfg of Nothing -> action Just expected -> let provided = lookup HTTP.hAuthorization (requestHeaders req)- in if provided == Just ("Bearer " <> expected)+ expectedHeader = "Bearer " <> expected+ in if maybe False (constEq expectedHeader) provided then action else respond (responseLBS HTTP.status401 textHeaders "unauthorized") @@ -166,16 +232,23 @@ -- Signing -- --------------------------------------------------------------------------- --- | Sign a narinfo body if a signing key is configured.-signNarInfo :: Maybe SecretKey -> BS.ByteString -> BS.ByteString-signNarInfo Nothing body = body-signNarInfo (Just sk) body = case parseNarInfo (TE.decodeUtf8 body) of- Left _ -> body- Right ni -> case sign sk ni of- Left _ -> body- Right sig ->- let signed = ni {niSigs = niSigs ni ++ [sig]}- in TE.encodeUtf8 (renderNarInfo signed)+-- | Sign a validated 'NarInfo' if a signing key is configured.+--+-- Logs a warning to stderr if signing fails, then returns the unsigned+-- rendering so the upload is not rejected.+signNarInfo :: Maybe SecretKey -> NarInfo -> IO BS.ByteString+signNarInfo Nothing ni = pure (renderNarInfoBytes ni)+signNarInfo (Just sk) ni = case sign sk ni of+ Left err -> do+ hPutStrLn stderr ("WARNING: signNarInfo: sign failed: " ++ err)+ pure (renderNarInfoBytes ni)+ Right sig ->+ let signed = ni {niSigs = niSigs ni ++ [sig]}+ in pure (renderNarInfoBytes signed)++-- | Render a 'NarInfo' to its UTF-8 encoded wire format.+renderNarInfoBytes :: NarInfo -> BS.ByteString+renderNarInfoBytes = TE.encodeUtf8 . renderNarInfo -- --------------------------------------------------------------------------- -- Response helpers
nova-cache.cabal view
@@ -1,6 +1,6 @@ cabal-version: 3.0 name: nova-cache-version: 0.2.4.1+version: 0.3.0.0 synopsis: Pure Nix binary cache protocol library description: A focused, minimal, pure-first library implementing the full Nix binary@@ -86,6 +86,7 @@ build-depends: base >= 4.16 && < 5 , bytestring >= 0.11 && < 0.13+ , memory >= 0.18 && < 1 , nova-cache , http-types >= 0.12 && < 0.13 , text >= 2.0 && < 2.2
src/NovaCache/Compression.hs view
@@ -9,6 +9,7 @@ where import qualified Codec.Compression.Lzma as Lzma+import Control.Exception (SomeException, evaluate, try) import Data.ByteString (ByteString) import qualified Data.ByteString.Lazy as BL @@ -17,5 +18,11 @@ compressXz = BL.toStrict . Lzma.compress . BL.fromStrict -- | Decompress an xz-compressed strict 'ByteString'.-decompressXz :: ByteString -> ByteString-decompressXz = BL.toStrict . Lzma.decompress . BL.fromStrict+--+-- Returns 'Left' with an error message if the input is not valid xz data.+decompressXz :: ByteString -> IO (Either String ByteString)+decompressXz bs = do+ result <- try (evaluate (BL.toStrict (Lzma.decompress (BL.fromStrict bs))))+ pure $ case result of+ Left err -> Left ("xz decompression failed: " ++ show (err :: SomeException))+ Right decompressed -> Right decompressed
src/NovaCache/NAR.hs view
@@ -217,7 +217,8 @@ (targetPath, afterPath) <- readStr afterTgt (rp, final) <- readStr afterPath expect tokRParen rp- pure (NarSymlink (TE.decodeUtf8 targetPath), final)+ symTarget <- decodeUtf8Safe targetPath+ pure (NarSymlink symTarget, final) -- | Parse a directory node (zero or more child entries). parseDirectory :: NarParser NarEntry@@ -239,7 +240,8 @@ (entry, afterEntry) <- parseNode afterNodeTok (rp, afterRp) <- readStr afterEntry expect tokRParen rp- go ((TE.decodeUtf8 entryName, entry) : acc) afterRp+ decodedName <- decodeUtf8Safe entryName+ go ((decodedName, entry) : acc) afterRp -- --------------------------------------------------------------------------- -- Wire primitives@@ -287,6 +289,12 @@ expect expected got | got == expected = Right () | otherwise = Left ("expected " ++ show expected ++ ", got " ++ show got)++-- | Decode a UTF-8 bytestring, converting decode failures to parse errors.+decodeUtf8Safe :: ByteString -> Either String Text+decodeUtf8Safe bs = case TE.decodeUtf8' bs of+ Right txt -> Right txt+ Left err -> Left ("invalid UTF-8 in NAR: " ++ show err) -- --------------------------------------------------------------------------- -- Hashing
src/NovaCache/Store.hs view
@@ -13,6 +13,7 @@ writeNarInfo, readNar, writeNar,+ listNarInfoHashes, getCacheInfo, sanitizePath, )@@ -25,6 +26,7 @@ import System.Directory ( createDirectoryIfMissing, doesFileExist,+ listDirectory, ) import System.FilePath ((</>)) @@ -94,11 +96,11 @@ -- | Write a narinfo by its store path hash. ----- Silently rejects path components containing traversal sequences.-writeNarInfo :: FileStore -> Text -> ByteString -> IO ()+-- Returns 'False' for path components containing traversal sequences.+writeNarInfo :: FileStore -> Text -> ByteString -> IO Bool writeNarInfo fs hashKey body = case sanitizePath hashKey of- Nothing -> pure ()- Just safe -> BS.writeFile (fsRoot fs </> narinfoSubdir </> safe) body+ Nothing -> pure False+ Just safe -> BS.writeFile (fsRoot fs </> narinfoSubdir </> safe) body >> pure True -- --------------------------------------------------------------------------- -- NAR operations@@ -114,11 +116,21 @@ -- | Write a NAR file by its filename. ----- Silently rejects path components containing traversal sequences.-writeNar :: FileStore -> Text -> ByteString -> IO ()+-- Returns 'False' for path components containing traversal sequences.+writeNar :: FileStore -> Text -> ByteString -> IO Bool writeNar fs fileName body = case sanitizePath fileName of- Nothing -> pure ()- Just safe -> BS.writeFile (fsRoot fs </> narSubdir </> safe) body+ Nothing -> pure False+ Just safe -> BS.writeFile (fsRoot fs </> narSubdir </> safe) body >> pure True++-- ---------------------------------------------------------------------------+-- Listing+-- ---------------------------------------------------------------------------++-- | List all narinfo hashes currently stored.+--+-- Returns the filenames in the @narinfo/@ subdirectory as 'Text' values.+listNarInfoHashes :: FileStore -> IO [Text]+listNarInfoHashes fs = map T.pack <$> listDirectory (fsRoot fs </> narinfoSubdir) -- --------------------------------------------------------------------------- -- Cache metadata
test/CompressionTest.hs view
@@ -38,6 +38,24 @@ putStrLn (" " ++ label ++ ": expected True") pure False +-- | Assert a Right value matches.+assertRight :: (Eq a, Show a) => String -> a -> Either String a -> IO Bool+assertRight label expected (Right actual) = assertEqual label expected actual+assertRight label _ (Left err) = do+ putStrLn ""+ putStrLn (" " ++ label)+ putStrLn (" expected Right, got Left: " ++ err)+ pure False++-- | Assert a Left (error case).+assertLeft :: (Show a) => String -> Either String a -> IO Bool+assertLeft _ (Left _) = pure True+assertLeft label (Right val) = do+ putStrLn ""+ putStrLn (" " ++ label)+ putStrLn (" expected Left, got Right: " ++ show val)+ pure False+ main :: IO () main = do putStrLn "nova-cache compression tests"@@ -45,22 +63,26 @@ putStrLn "" putStrLn "Compression:" ok1 <-- test "compress/decompress roundtrip" $+ test "compress/decompress roundtrip" $ do let input = BS.pack [72, 101, 108, 108, 111, 32, 87, 111, 114, 108, 100] compressed = Compression.compressXz input- decompressed = Compression.decompressXz compressed- in assertEqual "roundtrip" input decompressed+ result <- Compression.decompressXz compressed+ assertRight "roundtrip" input result ok2 <-- test "compress/decompress roundtrip (empty)" $+ test "compress/decompress roundtrip (empty)" $ do let compressed = Compression.compressXz BS.empty- decompressed = Compression.decompressXz compressed- in assertEqual "roundtrip empty" BS.empty decompressed+ result <- Compression.decompressXz compressed+ assertRight "roundtrip empty" BS.empty result ok3 <- test "compressed is smaller for repetitive data" $ let input = BS.replicate 10000 0x42 compressed = Compression.compressXz input in assertTrue "smaller" (BS.length compressed < BS.length input)+ ok4 <-+ test "decompress invalid data returns Left" $ do+ result <- Compression.decompressXz (BS.pack [0, 1, 2, 3])+ assertLeft "invalid xz" result putStrLn ""- if ok1 && ok2 && ok3+ if ok1 && ok2 && ok3 && ok4 then putStrLn "All compression tests passed." >> exitSuccess else putStrLn "Some compression tests failed." >> exitFailure
test/Main.hs view
@@ -5,6 +5,7 @@ import Data.ByteString (ByteString) import qualified Data.ByteString as BS import qualified Data.ByteString.Base64 as B64+import Data.List (sort) import Data.Text (Text) import qualified Data.Text as T import qualified Data.Text.Encoding as TE@@ -502,19 +503,23 @@ store <- Store.newFileStore tmpDir let hashKey = "testhash123" content = TE.encodeUtf8 ("StorePath: /nix/store/test\n" :: Text)- Store.writeNarInfo store hashKey content+ wOk <- Store.writeNarInfo store hashKey content result <- Store.readNarInfo store hashKey removeDirectoryRecursive tmpDir- assertEqual "narinfo roundtrip" (Just content) result,+ ok1 <- assertTrue "write succeeded" wOk+ ok2 <- assertEqual "narinfo roundtrip" (Just content) result+ pure (ok1 && ok2), test "nar write/read roundtrip" $ do tmpDir <- createTestDir store <- Store.newFileStore tmpDir let fileName = "test.nar.xz" content = BS.pack [1, 2, 3, 4, 5]- Store.writeNar store fileName content+ wOk <- Store.writeNar store fileName content result <- Store.readNar store fileName removeDirectoryRecursive tmpDir- assertEqual "nar roundtrip" (Just content) result,+ ok1 <- assertTrue "write succeeded" wOk+ ok2 <- assertEqual "nar roundtrip" (Just content) result+ pure (ok1 && ok2), test "read nonexistent returns Nothing" $ do tmpDir <- createTestDir store <- Store.newFileStore tmpDir@@ -545,7 +550,34 @@ store <- Store.newFileStore tmpDir result <- Store.readNarInfo store "../../etc/passwd" removeDirectoryRecursive tmpDir- assertEqual "blocked" Nothing result+ assertEqual "blocked" Nothing result,+ test "writeNarInfo rejects traversal" $ do+ tmpDir <- createTestDir+ store <- Store.newFileStore tmpDir+ ok <- Store.writeNarInfo store "../../etc/passwd" "bad"+ removeDirectoryRecursive tmpDir+ assertFalse "write rejected" ok,+ test "writeNar rejects traversal" $ do+ tmpDir <- createTestDir+ store <- Store.newFileStore tmpDir+ ok <- Store.writeNar store "../escape.nar" "bad"+ removeDirectoryRecursive tmpDir+ assertFalse "write rejected" ok,+ test "listNarInfoHashes returns stored hashes" $ do+ tmpDir <- createTestDir+ store <- Store.newFileStore tmpDir+ _ <- Store.writeNarInfo store "hash1" "content1"+ _ <- Store.writeNarInfo store "hash2" "content2"+ hashes <- Store.listNarInfoHashes store+ removeDirectoryRecursive tmpDir+ let sorted = sort hashes+ assertEqual "listed hashes" ["hash1", "hash2"] sorted,+ test "listNarInfoHashes empty store" $ do+ tmpDir <- createTestDir+ store <- Store.newFileStore tmpDir+ hashes <- Store.listNarInfoHashes store+ removeDirectoryRecursive tmpDir+ assertEqual "empty" [] hashes ] -- ---------------------------------------------------------------------------