nvfetcher 0.5.0.0 → 0.6.0.0

raw patch · 25 files changed

+658/−192 lines, 25 filesdep ~binary-instancesdep ~shakePVP ok

version bump matches the API change (PVP)

Dependency ranges changed: binary-instances, shake

API changes (from Hackage documentation)

Files

• CHANGELOG.md +10/−0
• README.md +44/−4
• app/Config.hs +20/−1
• app/Config/PackageFetcher.hs +26/−2
• app/Config/VersionSource.hs +14/−2
• app/Main.hs +1/−1
• nvfetcher.cabal +9/−7
• src/NvFetcher.hs +26/−19
• src/NvFetcher/Config.hs +5/−2
• src/NvFetcher/Core.hs +9/−2
• src/NvFetcher/ExtractSrc.hs +15/−21
• src/NvFetcher/FetchRustGitDeps.hs +3/−1
• src/NvFetcher/GetGitCommitDate.hs +47/−0
• src/NvFetcher/NixExpr.hs +47/−29
• src/NvFetcher/NixFetcher.hs +52/−14
• src/NvFetcher/Nvchecker.hs +34/−18
• src/NvFetcher/Options.hs +11/−1
• src/NvFetcher/PackageSet.hs +31/−4
• src/NvFetcher/Types.hs +148/−27
• src/NvFetcher/Types/ShakeExtras.hs +7/−0
• test/CheckVersionSpec.hs +9/−4
• test/FetchRustGitDepsSpec.hs +1/−1
• test/GetGitCommitDateSpec.hs +30/−0
• test/NixExprSpec.hs +32/−31
• test/PrefetchSpec.hs +27/−1

@@ -1,5 +1,15 @@ # Revision history for nvfetcher +## 0.6.0.0++* Parse error output from nvchecker+* Add source and fetcher for Docker containers (Thanks to @amesgen)+* Add option fetch.force to always rerun the prefetch rule+* Add `fetchGitHubRelease'` to `PackageSet` +* Add `GetGitCommitDate` rule+* Add `--keyfile` to pass [nvchecker keyfile](https://nvchecker.readthedocs.io/en/latest/usage.html#configuration-table)+* Remove IFD from extracing sources+ ## 0.5.0.0  There have been many significant changes since the last release.

@@ -103,6 +103,12 @@ To run nvfetcher as a CLI program, you'll need to provide package sources defined in TOML.  ```+Usage: nvfetcher [--version] [--help] [-o|--build-dir DIR] [--commit-changes]+                 [-l|--changelog FILE] [-j NUM] [-r|--retry NUM] [-t|--timing]+                 [-v|--verbose] [-f|--filter REGEX] [-k|--keyfile FILE] [TARGET]+                 [-c|--config FILE]+  generate nix sources expr for the latest version of packages+ Available options:   --version                Show version   --help                   Show this help text@@ -114,12 +120,13 @@   -j NUM                   Number of threads (0: detected number of processors)                            (default: 0)   -r,--retry NUM           Times to retry of some rules (nvchecker, prefetch,-                           nix-instantiate, etc.) (default: 3)+                           nix-build, etc.) (default: 3)   -t,--timing              Show build time   -v,--verbose             Verbose mode   -f,--filter REGEX        Regex to filter packages to be updated+  -k,--keyfile FILE        Nvchecker keyfile   TARGET                   Two targets are available: 1.build 2.clean-                           (default: "build")+                           (default: build)   -c,--config FILE         Path to nvfetcher TOML config                            (default: "nvfetcher.toml") ```@@ -128,6 +135,11 @@ two required fields and three optional fields in each table. You can find an example of the configuration file, see [`nvfetcher_example.toml`](nvfetcher_example.toml). +### Keyfile++You can specify nvchecker keyfile via command line option.+For the format of this file, please refer to [nvchecker documentation](https://nvchecker.readthedocs.io/en/latest/usage.html#configuration-table).+ #### Nvchecker  Version source -- how do we track upstream version updates?@@ -145,8 +157,9 @@ - `src.openvsx = publisher.ext_name` -- the latest version of a vscode extension from open vsx - `src.vsmarketplace = publisher.ext_name` -- the latest version of a vscode extension from vscode marketplace - `src.cmd = cmd` -- the version from a shell command (e.g. `echo Meow`)+= `src.container = owner/name` - the latest tag of a container from the Docker registry -Optional list options for some version sources (`src.github_tag`, `src.webpage`, and `src.httpheader`),+Optional list options for some version sources (`src.github_tag`, `src.webpage`, and `src.httpheader` and `src.container`), see the corresponding [nvchecker documentation](https://nvchecker.readthedocs.io/en/latest/usage.html#list-options) for details.  - `src.include_regex`@@ -173,6 +186,7 @@ - `fetch.openvsx = publisher.ext_name` - `fetch.vsmarketplace = publisher.ext_name` - `fetch.tarball = tarball_url`+- `fetch.docker = owner/name`  Optional `nix-prefetch fetchgit` config, which make sense only when the fetcher equals to `fetch.github` or `fetch.git`. They can exist simultaneously.@@ -181,6 +195,14 @@ - `git.fetchSubmodules` - `git.leaveDotGit` +Optional config for `fetch.docker`/`dockerTools.pullImage`:++- `docker.os`+- `docker.arch`+- `docker.finalImageName`+- `docker.finalImageTag`+- `docker.tlsVerify`+ #### Extract src  Optional _extract src_ config, files are extracted into build directory, and then read by `readFile` in generated nix expr.@@ -190,7 +212,7 @@ #### Rust support  `rustPlatform.buildRustPackage` now accepts an attribute [`cargoLock`](https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/rust.section.md#importing-a-cargolock-file) to vendor dependencies from `Cargo.lock`,-so we can use this instead TOFU `cargoSha256` for Rust packaging. `nvfetcher` supports automating this process,+so we can use this instead of TOFU `cargoSha256` for Rust packaging. `nvfetcher` supports automating this process, extracting the lock file to build and calculating `cargoLock.outputHashes`, as long as you set the config. There can be many lock files in one source. @@ -208,6 +230,23 @@  - `pinned = true` +#### Git commit date++If the version source of a package is `git`, nvfetcher can finds out the commit date of this revision,+in the format of `%Y-%m-%d` by default. You can provide your own [`strftime`](https://strftime.org/) format:++- `git.date_format = "strftime_format"`++#### Force fetching++Always fetch the package, even if its version (nvchecker output) doesn't change.+This is useful when the file to be downloaded cannot not be determined by the url.++- `fetch.force = true`++> Note: In such case, nvfetcher will not produce a version change log, since the version doesn't change at all.+> Only sha256 in generated files will be updated.+ ### Haskell library  nvfetcher itself is a Haskell library as well, whereas the CLI program is just a trivial wrapper of the library.@@ -230,5 +269,6 @@ ``` $ git clone https://github.com/berberman/nvfetcher $ nix develop+$ cabal update $ cabal build ```

@@ -60,7 +60,9 @@     pcCargoLockFiles :: Maybe PackageCargoLockFiles,     pcNvcheckerOptions :: NvcheckerOptions,     pcPassthru :: PackagePassthru,-    pcUseStale :: UseStaleVersion+    pcUseStale :: UseStaleVersion,+    pcGitDateFormat :: DateFormat,+    pcForceFetch :: ForceFetch   }  toPackage :: PackageKey -> PackageConfig -> Package@@ -73,6 +75,8 @@     pcCargoLockFiles     pcPassthru     pcUseStale+    pcGitDateFormat+    pcForceFetch  packageConfigCodec :: TomlCodec PackageConfig packageConfigCodec =@@ -84,6 +88,8 @@     <*> nvcheckerOptionsCodec .= pcNvcheckerOptions     <*> passthruCodec .= pcPassthru     <*> pinnedCodec .= pcUseStale+    <*> gitDateFormatCodec .= pcGitDateFormat+    <*> forceFetchCodec .= pcForceFetch  -------------------------------------------------------------------------------- @@ -121,3 +127,16 @@     )     (maybe NoStale (\x -> if x then PermanentStale else NoStale))     $ dioptional $ bool "pinned"++gitDateFormatCodec :: TomlCodec DateFormat+gitDateFormatCodec = diwrap $ dioptional $ text "git.date_format"++forceFetchCodec :: TomlCodec ForceFetch+forceFetchCodec =+  dimap+    ( \case+        ForceFetch -> Just True+        NoForceFetch -> Just False+    )+    (maybe NoForceFetch (\x -> if x then ForceFetch else NoForceFetch))+    $ dioptional $ bool "fetch.force"

@@ -1,6 +1,7 @@ {-# LANGUAGE DataKinds #-} {-# LANGUAGE DeriveAnyClass #-} {-# LANGUAGE DeriveGeneric #-}+{-# LANGUAGE FlexibleContexts #-} {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE RecordWildCards #-} {-# LANGUAGE ViewPatterns #-}@@ -36,7 +37,8 @@       vscodeMarketplaceCodec,       gitCodec,       urlCodec,-      tarballCodec+      tarballCodec,+      dockerCodec     ]  fetcherKeys :: [Key]@@ -47,7 +49,8 @@     "fetch.vsmarketplace",     "fetch.git",     "fetch.url",-    "fetch.tarball"+    "fetch.tarball",+    "fetch.docker"   ]  --------------------------------------------------------------------------------@@ -171,3 +174,24 @@     unsupportError     (\t -> Right $ \(coerce -> v) -> tarballFetcher $ T.replace "$ver" v t)     "fetch.tarball"++--------------------------------------------------------------------------------++dockerCodec :: TomlCodec PackageFetcher+dockerCodec =+  dimap unsupportError setImageTag codec+  where+    setImageTag t (Version v) = t & imageTag .~ v++    codec :: TomlCodec (NixFetcher Fresh)+    codec =+      FetchDocker+        <$> Toml.text "fetch.docker" .= _imageName+        <*> pure "" -- will be set in setImageTag+        <*> pure () .= _imageDigest+        <*> pure () .= _sha256+        <*> dioptional (text "docker.os") .= _fos+        <*> dioptional (text "docker.arch") .= _farch+        <*> dioptional (text "docker.finalImageName") .= _finalImageName+        <*> dioptional (text "docker.finalImageTag") .= _finalImageTag+        <*> dioptional (bool "docker.tlsVerify") .= _tlsVerify

@@ -33,7 +33,8 @@       httpHeaderCodec,       openVsxCodec,       vscodeMarketplaceCodec,-      cmdCodec+      cmdCodec,+      containerCodec     ]  versionSourceKeys :: [Key]@@ -49,7 +50,8 @@     "src.httpheader",     "src.openvsx",     "src.vsmarketplace",-    "src.cmd"+    "src.cmd",+    "src.container"   ]  --------------------------------------------------------------------------------@@ -244,3 +246,13 @@  cmdCodec :: TomlCodec VersionSource cmdCodec = dimatch matchCmd Cmd (text "src.cmd")++--------------------------------------------------------------------------------++matchContainer :: VersionSource -> Maybe (Text, ListOptions)+matchContainer x = (,) <$> x ^? vcontainer <*> x ^? listOptions++containerCodec :: TomlCodec VersionSource+containerCodec =+  dimatch matchContainer (uncurry Container) $+    Toml.pair (text "src.container") listOptionsCodec

@@ -33,4 +33,4 @@     Left e -> error $ T.unpack $ Toml.prettyTomlDecodeError $ Toml.ParseError e     Right x -> case parseConfig x of       Left e -> error $ T.unpack $ prettyPackageConfigParseError e-      Right pkgs -> runNvFetcherNoCLI (applyCliOptions def opt) (optTarget opt) $ purePackageSet pkgs+      Right pkgs -> applyCliOptions def opt >>= \o -> runNvFetcherNoCLI o (optTarget opt) $ purePackageSet pkgs

@@ -1,6 +1,6 @@ cabal-version:   2.4 name:            nvfetcher-version:         0.5.0.0+version:         0.6.0.0 synopsis:   Generate nix sources expr for the latest version of packages @@ -13,7 +13,7 @@ license-file:    LICENSE author:          berberman maintainer:      berberman <berberman.yandex.com>-copyright:       2021-2022 berberman+copyright:       2021-2023 berberman category:        Nix build-type:      Simple extra-doc-files:@@ -26,11 +26,11 @@  common common-options   build-depends:-    , aeson                 >=1.5.6   && <2.1+    , aeson                 >=1.5.6    && <2.1     , aeson-pretty-    , base                  >=4.8     && <5+    , base                  >=4.8      && <5     , binary-    , binary-instances      ^>=1.0.1+    , binary-instances      ^>=1.0     , bytestring     , containers     , data-default          ^>=0.7.1@@ -43,7 +43,7 @@     , parsec     , prettyprinter     , regex-tdfa            ^>=1.3.1.1-    , shake                 ^>=0.19.4+    , shake                 ^>=0.19     , text     , tomland               ^>=1.3.2     , transformers@@ -67,6 +67,7 @@     NvFetcher.Core     NvFetcher.ExtractSrc     NvFetcher.FetchRustGitDeps+    NvFetcher.GetGitCommitDate     NvFetcher.NixExpr     NvFetcher.NixFetcher     NvFetcher.Nvchecker@@ -116,6 +117,7 @@   other-modules:     CheckVersionSpec     FetchRustGitDepsSpec+    GetGitCommitDateSpec     NixExprSpec     PrefetchSpec     Utils@@ -129,5 +131,5 @@     , stm     , unliftio -  build-tool-depends: hspec-discover:hspec-discover -any+  build-tool-depends: hspec-discover:hspec-discover   default-language:   Haskell2010

@@ -88,26 +88,33 @@ -- | Similar to 'runNvFetcher', but uses custom @config@ instead of 'def' overridden by 'CLIOptions' runNvFetcher' :: Config -> PackageSet () -> IO () runNvFetcher' config packageSet =-  getCLIOptions cliOptionsParser >>= \cli -> runNvFetcherNoCLI (applyCliOptions config cli) (optTarget cli) packageSet+  getCLIOptions cliOptionsParser >>= \cli ->+    applyCliOptions config cli >>= \o ->+      runNvFetcherNoCLI o (optTarget cli) packageSet  -- | Apply 'CLIOptions' to 'Config'-applyCliOptions :: Config -> CLIOptions -> Config-applyCliOptions config CLIOptions {..} =-  config-    { buildDir = optBuildDir,-      actionAfterBuild = do-        whenJust optLogPath logChangesToFile-        when optCommit commitChanges-        actionAfterBuild config,-      shakeConfig =-        (shakeConfig config)-          { shakeTimings = optTiming,-            shakeVerbosity = if optVerbose then Verbose else Info,-            shakeThreads = optThreads-          },-      filterRegex = optPkgNameFilter,-      retry = optRetry-    }+applyCliOptions :: Config -> CLIOptions -> IO Config+applyCliOptions config CLIOptions {..} = do+  aKeyfile <- case optKeyfile of+    Just k -> Just <$> D.makeAbsolute k+    _ -> pure Nothing+  pure $+    config+      { buildDir = optBuildDir,+        actionAfterBuild = do+          whenJust optLogPath logChangesToFile+          when optCommit commitChanges+          actionAfterBuild config,+        shakeConfig =+          (shakeConfig config)+            { shakeTimings = optTiming,+              shakeVerbosity = if optVerbose then Verbose else Info,+              shakeThreads = optThreads+            },+        filterRegex = optPkgNameFilter,+        retry = optRetry,+        keyfile = aKeyfile+      }  logChangesToFile :: FilePath -> Action () logChangesToFile fp = do@@ -207,7 +214,7 @@ srouces body =   [trimming|     # This file was generated by nvfetcher, please do not modify it manually.-    { fetchgit, fetchurl, fetchFromGitHub }:+    { fetchgit, fetchurl, fetchFromGitHub, dockerTools }:     {       $body     }

@@ -17,7 +17,9 @@     actionAfterClean :: Action (),     retry :: Int,     filterRegex :: Maybe String,-    cacheNvchecker :: Bool+    cacheNvchecker :: Bool,+    -- | Absolute path+    keyfile :: Maybe FilePath   }  instance Default Config where@@ -34,5 +36,6 @@         actionAfterClean = pure (),         retry = 3,         filterRegex = Nothing,-        cacheNvchecker = True+        cacheNvchecker = True,+        keyfile = Nothing       }

@@ -23,6 +23,7 @@ import Development.Shake.Rule import NvFetcher.ExtractSrc import NvFetcher.FetchRustGitDeps+import NvFetcher.GetGitCommitDate import NvFetcher.NixFetcher import NvFetcher.Nvchecker import NvFetcher.Types@@ -36,6 +37,7 @@   prefetchRule   extractSrcRule   fetchRustGitDepsRule+  getGitCommitDateRule   addBuiltinRule noLint noIdentity $ \(WithPackageKey (Core, pkg)) _ _ -> do     -- it's important to always rerun     -- since the package definition is not tracked at all@@ -49,7 +51,7 @@             ..           } -> do           _prversion@(NvcheckerResult version _mOldV _isStale) <- checkVersion versionSource options pkg-          _prfetched <- prefetch $ _pfetcher version+          _prfetched <- prefetch (_pfetcher version) _pforcefetch           buildDir <- getBuildDir           -- extract src           _prextract <-@@ -92,8 +94,13 @@                 pure . Just $ HMap.fromList result               _ -> pure Nothing +          -- Only git version source supports git commit date+          _prgitdate <- case versionSource of+            Git {..} -> Just <$> getGitCommitDate _vurl (coerce version) _pgitdateformat+            _ -> pure Nothing+           -- update changelog-          -- always use on disk verion+          -- always use on disk version           mOldV <- getLastVersionOnDisk pkg           case mOldV of             Nothing ->

@@ -1,8 +1,7 @@ {-# LANGUAGE DataKinds #-} {-# LANGUAGE OverloadedStrings #-}-{-# LANGUAGE QuasiQuotes #-} {-# LANGUAGE ScopedTypeVariables #-}-{-# LANGUAGE ViewPatterns #-}+{-# LANGUAGE TupleSections #-}  -- | Copyright: (c) 2021-2022 berberman -- SPDX-License-Identifier: MIT@@ -20,21 +19,24 @@      -- * Rules     extractSrcRule,++    -- * Functions     extractSrc,     extractSrcs,   ) where  import Control.Monad (void)-import qualified Data.Aeson as A+import Control.Monad.Extra (unlessM) import Data.Binary.Instances () import Data.HashMap.Strict (HashMap)+import qualified Data.HashMap.Strict as HM import qualified Data.List.NonEmpty as NE import Data.Text (Text) import qualified Data.Text as T-import qualified Data.Text.Encoding as T+import qualified Data.Text.IO as T import Development.Shake-import NeatInterpolation (trimming)+import Development.Shake.FilePath ((</>)) import NvFetcher.NixExpr import NvFetcher.Types import NvFetcher.Types.ShakeExtras@@ -43,17 +45,18 @@ -- | Rules of extract source extractSrcRule :: Rules () extractSrcRule = void $-  addOracleCache $ \(q :: ExtractSrcQ) -> withTempFile $ \fp -> withRetry $ do+  addOracle $ \q@(ExtractSrcQ fetcher files) -> withTempFile $ \fp -> withRetry $ do     putInfo . show $ "#" <+> pretty q-    let nixExpr = T.unpack $ wrap $ toNixExpr q+    let nixExpr = T.unpack $ fetcherToDrv fetcher "nvfetcher-extract"     putVerbose $ "Generated nix expr:\n" <> nixExpr     writeFile' fp nixExpr-    -- TODO: Avoid using NIX_PATH-    (CmdTime t, StdoutTrim out, CmdLine c) <- quietly $ cmd Shell $ "nix-instantiate --eval --strict --json --read-write-mode -E 'let pkgs = import <nixpkgs> { }; in ((import " <> fp <> ") pkgs)'"+    (CmdTime t, StdoutTrim out, CmdLine c, Stdouterr err) <- quietly $ cmd $ "nix-build --no-out-link " <> fp     putVerbose $ "Finishing running " <> c <> ", took " <> show t <> "s"-    case A.decodeStrict out of-      Just x -> pure x-      _ -> fail $ "Failed to parse output of nix-instantiate: " <> T.unpack (T.decodeUtf8 out)+    putVerbose $ "Output from stdout: " <> out+    putVerbose $ "Output from stderr: " <> err+    unlessM (doesDirectoryExist out) $+      fail $ "nix-build output is not a directory: " <> out+    HM.fromList <$> sequence [(f,) <$> liftIO (T.readFile $ out </> f) | f <- NE.toList files]  -- | Run extract source with many sources extractSrcs ::@@ -72,12 +75,3 @@   FilePath ->   Action (HashMap FilePath Text) extractSrc fetcher fp = extractSrcs fetcher $ NE.fromList [fp]------------------------------------------------------------------------------------wrap :: NixExpr -> NixExpr-wrap expr =-  [trimming|-    { pkgs, ... }:-    $expr-  |]

@@ -16,6 +16,8 @@      -- * Rules     fetchRustGitDepsRule,++    -- * Functions     fetchRustGitDeps,   ) where@@ -52,7 +54,7 @@       parallel         [ case parse gitSrcParser (T.unpack rname) src of             Right ParsedGitSrc {..} -> do-              (_sha256 -> sha256) <- prefetch $ gitFetcher pgurl pgsha+              (_sha256 -> sha256) <- prefetch (gitFetcher pgurl pgsha) NoForceFetch               -- @${name}-${version}@ -> sha256               pure (rname <> "-" <> coerce rversion, sha256)             Left err -> fail $ "Failed to parse git source in Cargo.lock: " <> show err

@@ -0,0 +1,47 @@+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE ViewPatterns #-}++-- | Copyright: (c) 2021-2022 berberman+-- SPDX-License-Identifier: MIT+-- Maintainer: berberman <berberman@yandex.com>+-- Stability: experimental+-- Portability: portable+--+-- This module provides mechanisms for obtaining the git commit date.+-- The cloned repo will not be preserved.+module NvFetcher.GetGitCommitDate+  ( -- * Types+    DateFormat (..),+    GetGitCommitDate (..),++    -- * Rules+    getGitCommitDateRule,++    -- * Functions+    getGitCommitDate,+  )+where++import Control.Monad (void)+import Data.Coerce (coerce)+import Data.Maybe (fromMaybe)+import Data.Text (Text)+import qualified Data.Text as T+import Development.Shake+import NvFetcher.Types+import Prettyprinter (pretty, (<+>))++getGitCommitDateRule :: Rules ()+getGitCommitDateRule = void $ do+  addOracleCache $ \q@(GetGitCommitDate (T.unpack -> url) (T.unpack -> rev) format) -> withTempDir $ \repo -> do+    putInfo . show $ "#" <+> pretty q+    (StdoutTrim out) <- quietly $ do+      cmd_ [Cwd repo, EchoStderr False, EchoStdout False] ("git init" :: String)+      cmd_ [Cwd repo, EchoStderr False] $ "git remote add origin " <> url+      cmd_ [Cwd repo, EchoStderr False] $ "git fetch --depth 1 origin " <> rev+      cmd_ [Cwd repo, EchoStderr False] ("git checkout FETCH_HEAD" :: String)+      cmd [Cwd repo, Shell] $ "git --no-pager log -1 --format=%cd --date=format:\"" <> T.unpack (fromMaybe "%Y-%m-%d" $ coerce format) <> "\""+    pure $ T.pack out++getGitCommitDate :: Text -> Text -> DateFormat -> Action Text+getGitCommitDate url rev format = askOracle $ GetGitCommitDate url rev format

@@ -18,6 +18,7 @@ module NvFetcher.NixExpr   ( NixExpr,     ToNixExpr (..),+    fetcherToDrv,   ) where @@ -36,11 +37,8 @@ class ToNixExpr a where   toNixExpr :: a -> NixExpr -instance ToNixExpr (NixFetcher Fresh) where-  toNixExpr = nixFetcher "lib.fakeSha256"- instance ToNixExpr (NixFetcher Fetched) where-  toNixExpr f = nixFetcher (quote $ coerce $ _sha256 f) f+  toNixExpr = nixFetcher  instance ToNixExpr Bool where   toNixExpr True = "true"@@ -61,10 +59,10 @@ instance ToNixExpr Version where   toNixExpr = coerce -nixFetcher :: Text -> NixFetcher k -> NixExpr-nixFetcher sha256 = \case+nixFetcher :: NixFetcher Fetched -> NixExpr+nixFetcher = \case   FetchGit-    { _sha256 = _,+    { _sha256 = coerce quote -> sha256,       _rev = quote . toNixExpr -> rev,       _fetchSubmodules = toNixExpr -> fetchSubmodules,       _deepClone = toNixExpr -> deepClone,@@ -83,7 +81,7 @@           }     |]   FetchGitHub-    { _sha256 = _,+    { _sha256 = coerce quote -> sha256,       _rev = quote . toNixExpr -> rev,       _fetchSubmodules = toNixExpr -> fetchSubmodules,       _deepClone = toNixExpr -> deepClone,@@ -117,41 +115,60 @@                  sha256 = $sha256;                })          |]-  (FetchUrl (quote -> url) (nameField -> n) _) ->+  (FetchUrl (quote -> url) (nameField -> n) (coerce quote -> sha256)) ->     [trimming|           fetchurl {             url = $url;$n             sha256 = $sha256;           }     |]-  (FetchTarball (quote -> url) _) ->+  (FetchTarball (quote -> url) (coerce quote -> sha256)) ->     [trimming|           fetchTarball {             url = $url;             sha256 = $sha256;           }     |]+  FetchDocker+    { _imageName = quote . toNixExpr -> imageName,+      _imageTag = quote . toNixExpr -> imageTag,+      _imageDigest = ContainerDigest (quote . toNixExpr -> imageDigest),+      _sha256 = coerce quote -> sha256,+      _fos = optionalStr "os" -> os,+      _farch = optionalStr "arch" -> arch,+      _finalImageName = optionalStr "finalImageName" -> finalImageName,+      _finalImageTag = maybe imageTag (quote . toNixExpr) -> finalImageTag,+      _tlsVerify = optionalField "tlsVerify" -> tlsVerify+    } ->+      [trimming|+            dockerTools.pullImage {+              imageName = $imageName;+              imageDigest = $imageDigest;+              sha256 = $sha256;+              finalImageTag = $finalImageTag;$os$arch$finalImageName$tlsVerify+            }+      |]   where-    nameField = maybe "" (\x -> "\nname = " <> quote x <> ";")--instance ToNixExpr ExtractSrcQ where-  toNixExpr (ExtractSrcQ fetcher files) = extractFiles fetcher files+    optionalField n = maybe "" (\x -> "\n" <> n <> " = " <> toNixExpr x <> ";")+    optionalStr n = optionalField n . fmap quote+    nameField = optionalStr "name" -extractFiles :: NixFetcher Fetched -> NE.NonEmpty FilePath -> NixExpr-extractFiles (toNixExpr -> fetcherExpr) (toNixExpr -> fileNames) =+-- | Create a trivial drv that extracts the source from a fetcher+-- TODO: Avoid using @NIX_PATH@+fetcherToDrv :: NixFetcher Fetched -> Text -> NixExpr+fetcherToDrv (toNixExpr -> fetcherExpr) (quote -> drvName) =   [trimming|-    let-      drv = import (pkgs.writeText "src" ''-        pkgs: {-          src = pkgs.$fetcherExpr;-        }-      '');-      fileNames = $fileNames;-      toFile = f: builtins.readFile ((drv pkgs).src + "/" + f);-    in builtins.listToAttrs (builtins.map (x: {-      name = x;-      value = toFile x;-    }) fileNames)+    with import <nixpkgs> { };+    stdenv.mkDerivation {+      name = $drvName;+      src = $fetcherExpr;+      nativeBuildInputs = [ unzip ];+      dontBuild = true;+      installPhase = ''+        mkdir $$out+        cp -r * $$out+      '';+    }   |]  -- | nix expr snippet like:@@ -228,5 +245,6 @@                 ]           )           _prpassthru-      joined = extract <> cargo <> passthru+      date = maybe "" (\d -> "date = " <> quote d <> ";") _prgitdate+      joined = extract <> cargo <> passthru <> date       appending = if T.null joined then "" else "\n" <> joined

@@ -1,4 +1,6 @@ {-# LANGUAGE DataKinds #-}+{-# LANGUAGE DeriveAnyClass #-}+{-# LANGUAGE DeriveGeneric #-} {-# LANGUAGE FlexibleInstances #-} {-# LANGUAGE LambdaCase #-} {-# LANGUAGE OverloadedStrings #-}@@ -26,6 +28,8 @@ -- a fetcher will be filled with the fetch result (hash) after the prefetch. module NvFetcher.NixFetcher   ( -- * Types+    RunFetch (..),+    ForceFetch (..),     NixFetcher (..),     FetchStatus (..),     FetchResult,@@ -38,6 +42,7 @@     gitHubFetcher,     pypiFetcher,     gitHubReleaseFetcher,+    gitHubReleaseFetcher',     gitFetcher,     urlFetcher,     openVsxFetcher,@@ -46,12 +51,14 @@   ) where -import Control.Monad (void)+import Control.Monad (void, when)+import qualified Data.Aeson as A import Data.Coerce (coerce) import Data.Text (Text) import qualified Data.Text as T import qualified Data.Text.Encoding as T import Development.Shake+import GHC.Generics (Generic) import NeatInterpolation (trimming) import NvFetcher.Types import NvFetcher.Types.ShakeExtras@@ -59,7 +66,7 @@  -------------------------------------------------------------------------------- -runFetcher :: NixFetcher Fresh -> Action Checksum+runFetcher :: NixFetcher Fresh -> Action (NixFetcher Fetched) runFetcher = \case   FetchGit {..} -> do     (CmdTime t, Stdout (T.decodeUtf8 -> out), CmdLine c) <-@@ -73,7 +80,7 @@             <> ["--leaveDotGit" | _leaveDotGit]     putVerbose $ "Finishing running " <> c <> ", took " <> show t <> "s"     case takeWhile (not . T.null) $ reverse $ T.lines out of-      [x] -> pure $ coerce x+      [x] -> pure FetchGit {_sha256 = coerce x, ..}       _ -> fail $ "Failed to parse output from nix-prefetch: " <> T.unpack out   FetchGitHub {..} -> do     (CmdTime t, Stdout (T.decodeUtf8 -> out), CmdLine c) <-@@ -88,7 +95,7 @@             <> ["--leaveDotGit" | _leaveDotGit]     putVerbose $ "Finishing running " <> c <> ", took " <> show t <> "s"     case takeWhile (not . T.null) $ reverse $ T.lines out of-      [x] -> pure $ coerce x+      [x] -> pure FetchGitHub {_sha256 = coerce x, ..}       _ -> fail $ "Failed to parse output from nix-prefetch: " <> T.unpack out   FetchUrl {..} -> do     (CmdTime t, Stdout (T.decodeUtf8 -> out), CmdLine c) <-@@ -96,7 +103,7 @@         command [EchoStderr False] "nix-prefetch" ["fetchurl", "--url", T.unpack _furl]     putVerbose $ "Finishing running " <> c <> ", took " <> show t <> "s"     case takeWhile (not . T.null) $ reverse $ T.lines out of-      [x] -> pure $ coerce x+      [x] -> pure FetchUrl {_sha256 = coerce x, ..}       _ -> fail $ "Failed to parse output from nix-prefetch: " <> T.unpack out   FetchTarball {..} -> do     (CmdTime t, Stdout (T.decodeUtf8 -> out), CmdLine c) <-@@ -104,9 +111,30 @@         command [EchoStderr False] "nix-prefetch" ["fetchTarball", "--url", T.unpack _furl]     putVerbose $ "Finishing running " <> c <> ", took " <> show t <> "s"     case takeWhile (not . T.null) $ reverse $ T.lines out of-      [x] -> pure $ coerce x+      [x] -> pure FetchTarball {_sha256 = coerce x, ..}       _ -> fail $ "Failed to parse output from nix-prefetch: " <> T.unpack out+  FetchDocker {..} -> do+    (CmdTime t, Stdout out, CmdLine c) <-+      quietly $+        command [EchoStderr False] "nix-prefetch-docker" $+          [ "--json",+            T.unpack _imageName,+            T.unpack _imageTag+          ]+            <> concat [["--os", T.unpack os] | Just os <- [_fos]]+            <> concat [["--arch", T.unpack arch] | Just arch <- [_farch]]+    putVerbose $ "Finishing running " <> c <> ",, took " <> show t <> "s"+    case A.eitherDecode out of+      Right FetchedContainer {..} ->+        pure FetchDocker {_sha256 = sha256, _imageDigest = imageDigest, ..}+      Left e -> fail $ "Failed to parse output from nix-prefetch-docker as JSON: " <> e +data FetchedContainer = FetchedContainer+  { imageDigest :: ContainerDigest,+    sha256 :: Checksum+  }+  deriving (Show, Generic, A.FromJSON)+ pypiUrl :: Text -> Version -> Text pypiUrl pypi (coerce -> ver) =   let h = T.cons (T.head pypi) ""@@ -117,14 +145,14 @@ -- | Rules of nix fetcher prefetchRule :: Rules () prefetchRule = void $-  addOracleCache $ \(f :: NixFetcher Fresh) -> do+  addOracleCache $ \(RunFetch force f) -> do+    when (force == ForceFetch) alwaysRerun     putInfo . show $ "#" <+> pretty f-    sha256 <- withRetry $ runFetcher f-    pure $ f {_sha256 = sha256}+    withRetry $ runFetcher f  -- | Run nix fetcher-prefetch :: NixFetcher Fresh -> Action (NixFetcher Fetched)-prefetch = askOracle+prefetch :: NixFetcher Fresh -> ForceFetch -> Action (NixFetcher Fetched)+prefetch f force = askOracle $ RunFetch force f  -------------------------------------------------------------------------------- @@ -150,9 +178,19 @@   -- | file name   Text ->   PackageFetcher-gitHubReleaseFetcher (owner, repo) fp (coerce -> ver) =-  urlFetcher-    [trimming|https://github.com/$owner/$repo/releases/download/$ver/$fp|]+gitHubReleaseFetcher (owner, repo) fp = gitHubReleaseFetcher' (owner, repo) $ const fp++-- | Create a fetcher from github release+gitHubReleaseFetcher' ::+  -- | owner and repo+  (Text, Text) ->+  -- | file name computed from version+  (Version -> Text) ->+  PackageFetcher+gitHubReleaseFetcher' (owner, repo) f (coerce -> ver) =+  let fp = f $ coerce ver+   in urlFetcher+        [trimming|https://github.com/$owner/$repo/releases/download/$ver/$fp|]  -- | Create a fetcher from url urlFetcher :: Text -> NixFetcher Fresh

@@ -26,6 +26,8 @@      -- * Rules     nvcheckerRule,++    -- * Functions     checkVersion,     checkVersion',   )@@ -34,11 +36,11 @@ import Control.Monad (void) import Control.Monad.Extra (fromMaybeM) import qualified Data.Aeson as A+import qualified Data.ByteString.Char8 as BS import Data.Coerce (coerce)-import Data.Maybe (fromJust, mapMaybe)+import Data.Maybe (fromJust) import Data.String (fromString) import qualified Data.Text as T-import qualified Data.Text.Encoding as T import Development.Shake import Development.Shake.Rule import NvFetcher.Types@@ -71,10 +73,10 @@   case useStale of     True       | Just oldVer' <- oldVer -> do-        -- use the stale version if we have-        putInfo $ T.unpack $ "Skip running nvchecker, use stale version " <> coerce oldVer' <> " for " <> coerce pkg-        let result = NvcheckerResult {nvNow = oldVer', nvOld = oldVer, nvStale = True}-        pure $ RunResult ChangedRecomputeSame (encode' result) result+          -- use the stale version if we have+          putInfo $ T.unpack $ "Skip running nvchecker, use stale version " <> coerce oldVer' <> " for " <> coerce pkg+          let result = NvcheckerResult {nvNow = oldVer', nvOld = oldVer, nvStale = True}+          pure $ RunResult ChangedRecomputeSame (encode' result) result      -- run nvchecker     _ -> do@@ -97,26 +99,36 @@ oneShotRule = void $   addOracle $ \key@(CheckVersion versionSource options) -> do     putInfo . show $ pretty key-    NvcheckerRaw now <- runNvchecker (PackageKey "pkg") options versionSource+    now <- runNvchecker (PackageKey "pkg") options versionSource     pure $ NvcheckerResult now Nothing False -runNvchecker :: PackageKey -> NvcheckerOptions -> VersionSource -> Action NvcheckerRaw+runNvchecker :: PackageKey -> NvcheckerOptions -> VersionSource -> Action Version runNvchecker pkg options versionSource = withTempFile $ \config -> withRetry $ do-  let nvcheckerConfig = T.unpack $ Toml.pretty $ mkToml $ genNvConfig pkg options versionSource+  mKeyfile <- getKeyfilePath+  let nvcheckerConfig = T.unpack $ Toml.pretty $ mkToml $ genNvConfig pkg options mKeyfile versionSource   putVerbose $ "Generated nvchecker config for " <> show pkg <> ":" <> nvcheckerConfig   writeFile' config nvcheckerConfig   (CmdTime t, Stdout out, CmdLine c) <- quietly . cmd $ "nvchecker --logger json -c " <> config   putVerbose $ "Finishing running " <> c <> ", took " <> show t <> "s"-  let out' = T.decodeUtf8 out-      result = mapMaybe (A.decodeStrict . T.encodeUtf8) (T.lines out')-  case result of-    [x] -> pure x-    _ -> fail $ "Failed to parse output from nvchecker: " <> T.unpack out'+  case reverse . lines $ out of+    (o : _) | Just raw <- A.decodeStrict' $ BS.pack o -> case raw of+      NvcheckerSuccess x -> pure x+      NvcheckerError err -> fail $ "Failed to run nvchecker: " <> T.unpack err+    _ -> fail $ "Failed to parse output from nvchecker: " <> out -genNvConfig :: PackageKey -> NvcheckerOptions -> VersionSource -> TDSL-genNvConfig pkg options versionSource = table (fromString $ T.unpack $ coerce pkg) $ do-  genVersionSource versionSource-  genOptions options+genNvConfig :: PackageKey -> NvcheckerOptions -> Maybe FilePath -> VersionSource -> TDSL+genNvConfig pkg options mKeyfile versionSource =+  case mKeyfile of+    Just keyfile -> do+      table "__config__" $+        "keyfile" =: Text (T.pack keyfile)+    _ -> pure ()+    >> table+      (fromString $ T.unpack $ coerce pkg)+      ( do+          genVersionSource versionSource+          genOptions options+      )   where     key =:? (Just x) = key =: Text x     _ =:? _ = pure ()@@ -172,6 +184,10 @@       Cmd {..} -> do         "source" =: "cmd"         "cmd" =: Text _vcmd+      Container {..} -> do+        "source" =: "container"+        "container" =: Text _vcontainer+        genListOptions _listOptions     genListOptions ListOptions {..} = do       "include_regex" =:? _includeRegex       "exclude_regex" =:? _excludeRegex

@@ -42,6 +42,7 @@     optTiming :: Bool,     optVerbose :: Bool,     optPkgNameFilter :: Maybe String,+    optKeyfile :: Maybe FilePath,     optTarget :: Target   }   deriving (Show)@@ -84,7 +85,7 @@       ( short 'r'           <> long "retry"           <> metavar "NUM"-          <> help "Times to retry of some rules (nvchecker, prefetch, nix-instantiate, etc.)"+          <> help "Times to retry of some rules (nvchecker, prefetch, nix-build, etc.)"           <> value 3           <> showDefault       )@@ -96,6 +97,15 @@               <> long "filter"               <> metavar "REGEX"               <> help "Regex to filter packages to be updated"+          )+      )+    <*> optional+      ( strOption+          ( short 'k'+              <> long "keyfile"+              <> metavar "FILE"+              <> help "Nvchecker keyfile"+              <> completer (bashCompleter "file")           )       )     <*> argument

@@ -70,6 +70,7 @@     fetchGitHub,     fetchGitHub',     fetchGitHubRelease,+    fetchGitHubRelease',     fetchPypi,     fetchGit,     fetchGit',@@ -84,6 +85,8 @@     tweakVersion,     passthru,     pinned,+    gitDateFormat,+    forceFetch,      -- ** Miscellaneous     Prod,@@ -156,9 +159,11 @@   Maybe PackageCargoLockFiles ->   PackagePassthru ->   UseStaleVersion ->+  DateFormat ->+  ForceFetch ->   PackageSet ()-newPackage name source fetcher extract cargo pasthru useStale =-  liftF $ NewPackage (Package name source fetcher extract cargo pasthru useStale) ()+newPackage name source fetcher extract cargo pasthru useStale format force =+  liftF $ NewPackage (Package name source fetcher extract cargo pasthru useStale format force) ()  -- | Add a list of packages into package set purePackageSet :: [Package] -> PackageSet ()@@ -256,7 +261,9 @@            PackageCargoLockFiles,            NvcheckerOptions,            PackagePassthru,-           UseStaleVersion+           UseStaleVersion,+           DateFormat,+           ForceFetch          ]         r     ) =>@@ -281,6 +288,8 @@       (projMaybe p)       (fromMaybe mempty (projMaybe p))       (fromMaybe NoStale (projMaybe p))+      (fromMaybe (DateFormat Nothing) (projMaybe p))+      (fromMaybe NoForceFetch (projMaybe p))  -- | 'PkgDSL' version of 'newPackage' --@@ -301,7 +310,9 @@          PackageCargoLockFiles,          PackagePassthru,          NvcheckerOptions,-         UseStaleVersion+         UseStaleVersion,+         DateFormat,+         ForceFetch        ]       r   ) =>@@ -473,6 +484,12 @@ fetchGitHubRelease :: Attach PackageFetcher (Text, Text, Text) fetchGitHubRelease e (owner, repo, fp) = fetch e $ gitHubReleaseFetcher (owner, repo) fp +-- | This package is fetched from a file in github release+--+-- Args are owner, repo, and file name computed from version+fetchGitHubRelease' :: Attach PackageFetcher (Text, Text, Version -> Text)+fetchGitHubRelease' e (owner, repo, f) = fetch e $ gitHubReleaseFetcher' (owner, repo) f+ -- | This package is fetched from pypi -- -- Arg is pypi name@@ -543,3 +560,13 @@ -- new version won't be checked if we have a stale version pinned :: PackageSet (Prod r) -> PackageSet (Prod (UseStaleVersion : r)) pinned = flip andThen . pure $ PermanentStale++-- | Specify the date format for getting git commit date+--+-- Available only for git version source+gitDateFormat :: Attach DateFormat (Maybe Text)+gitDateFormat = (. pure . DateFormat) . andThen++-- | Set always fetching regardless of the version changing+forceFetch :: PackageSet (Prod r) -> PackageSet (Prod (ForceFetch : r))+forceFetch = flip andThen . pure $ ForceFetch

@@ -1,3 +1,4 @@+{-# LANGUAGE ConstraintKinds #-} {-# LANGUAGE DataKinds #-} {-# LANGUAGE DeriveAnyClass #-} {-# LANGUAGE DeriveGeneric #-}@@ -6,11 +7,14 @@ {-# LANGUAGE FlexibleInstances #-} {-# LANGUAGE GeneralizedNewtypeDeriving #-} {-# LANGUAGE LambdaCase #-}+{-# LANGUAGE MultiParamTypeClasses #-} {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE RecordWildCards #-} {-# LANGUAGE StandaloneDeriving #-} {-# LANGUAGE TypeFamilies #-}+{-# LANGUAGE TypeOperators #-} {-# LANGUAGE UndecidableInstances #-}+{-# LANGUAGE UndecidableSuperClasses #-}  -- | Copyright: (c) 2021-2022 berberman -- SPDX-License-Identifier: MIT@@ -23,6 +27,7 @@   ( -- * Common types     Version (..),     Checksum (..),+    ContainerDigest (..),     Branch (..),     NixExpr,     VersionChange (..),@@ -39,6 +44,8 @@     UseStaleVersion (..),      -- * Nix fetcher types+    RunFetch (..),+    ForceFetch (..),     NixFetcher (..),     FetchResult,     FetchStatus (..),@@ -49,6 +56,10 @@     -- * FetchRustGitDeps types     FetchRustGitDepsQ (..), +    -- * GetGitCommitDate types+    DateFormat (..),+    GetGitCommitDate (..),+     -- * Core types     Core (..), @@ -92,6 +103,12 @@   deriving stock (Typeable, Generic)   deriving anyclass (Hashable, Binary, NFData) +-- | Digest of a (Docker) container+newtype ContainerDigest = ContainerDigest Text+  deriving newtype (Show, Eq, Ord, A.FromJSON, A.ToJSON, Pretty)+  deriving stock (Typeable, Generic)+  deriving anyclass (Hashable, Binary, NFData)+ -- | Git branch ('Nothing': master) newtype Branch = Branch (Maybe Text)   deriving newtype (Show, Eq, Ord, Default, Pretty)@@ -215,6 +232,7 @@   | OpenVsx {_ovPublisher :: Text, _ovExtName :: Text}   | VscodeMarketplace {_vsmPublisher :: Text, _vsmExtName :: Text}   | Cmd {_vcmd :: Text}+  | Container {_vcontainer :: Text, _listOptions :: ListOptions}   deriving (Show, Typeable, Eq, Ord, Generic, Hashable, Binary, NFData)  instance Pretty VersionSource where@@ -311,6 +329,8 @@         )   pretty Cmd {..} =     "CheckCmd" <> colon <+> pretty _vcmd+  pretty Container {..} =+    "CheckContainer" <> colon <+> pretty _vcontainer  -- | The input of nvchecker data CheckVersion = CheckVersion VersionSource NvcheckerOptions@@ -332,17 +352,39 @@   deriving (Show, Typeable, Eq, Generic, Hashable, Binary, NFData)  -- | Parsed JSON output from nvchecker-newtype NvcheckerRaw = NvcheckerRaw Version+data NvcheckerRaw = NvcheckerSuccess Version | NvcheckerError Text   deriving (Show, Typeable, Eq, Generic)  instance A.FromJSON NvcheckerRaw where-  parseJSON = A.withObject "NvcheckerRaw" $ \o ->-    NvcheckerRaw <$> o A..: "version"+  parseJSON = A.withObject "NvcheckerRaw" $ \o -> do+    mVersion <- o A..:? "version"+    case mVersion of+      Just version -> pure $ NvcheckerSuccess version+      _ -> NvcheckerError <$> o A..: "error"  type instance RuleResult CheckVersion = NvcheckerResult  -------------------------------------------------------------------------------- +-- | Whether to cache the fetched sha256+--+-- @ForceFetch@ indicates @alwaysRerun@ the fetcher rule+data ForceFetch = ForceFetch | NoForceFetch+  deriving (Show, Eq, Ord, Hashable, NFData, Binary, Typeable, Generic)++instance Pretty ForceFetch where+  pretty ForceFetch = "ForceFetch"+  pretty NoForceFetch = "NoForceFetch"++instance Default ForceFetch where+  def = NoForceFetch++-- | The input of prefetch rule+data RunFetch = RunFetch ForceFetch (NixFetcher Fresh)+  deriving (Show, Eq, Ord, Hashable, NFData, Binary, Typeable, Generic)++type instance RuleResult RunFetch = NixFetcher Fetched+ -- | If the package is prefetched, then we can obtain the SHA256 data NixFetcher (k :: FetchStatus)   = FetchGit@@ -352,7 +394,7 @@         _fetchSubmodules :: Bool,         _leaveDotGit :: Bool,         _name :: Maybe Text,-        _sha256 :: FetchResult k+        _sha256 :: FetchResult Checksum k       }   | FetchGitHub       { _fowner :: Text,@@ -362,41 +404,54 @@         _fetchSubmodules :: Bool,         _leaveDotGit :: Bool,         _name :: Maybe Text,-        _sha256 :: FetchResult k+        _sha256 :: FetchResult Checksum k       }   | FetchUrl       { _furl :: Text,         _name :: Maybe Text,-        _sha256 :: FetchResult k+        _sha256 :: FetchResult Checksum k       }   | FetchTarball       { _furl :: Text,-        _sha256 :: FetchResult k+        _sha256 :: FetchResult Checksum k       }+  | FetchDocker+      { _imageName :: Text,+        _imageTag :: Text,+        _imageDigest :: FetchResult ContainerDigest k,+        _sha256 :: FetchResult Checksum k,+        _fos :: Maybe Text,+        _farch :: Maybe Text,+        _finalImageName :: Maybe Text,+        _finalImageTag :: Maybe Text,+        _tlsVerify :: Maybe Bool+      }   deriving (Typeable, Generic) --- | Fetch status-data FetchStatus = Fresh | Fetched+class (c (FetchResult Checksum k), c (FetchResult ContainerDigest k)) => ForFetchResult c k --- | Prefetched fetchers hold hashes-type family FetchResult (k :: FetchStatus) where-  FetchResult Fresh = ()-  FetchResult Fetched = Checksum+instance (c (FetchResult Checksum k), c (FetchResult ContainerDigest k)) => ForFetchResult c k -type instance RuleResult (NixFetcher Fresh) = NixFetcher Fetched+deriving instance Show `ForFetchResult` k => Show (NixFetcher k) -deriving instance Show (FetchResult k) => Show (NixFetcher k)+deriving instance Eq `ForFetchResult` k => Eq (NixFetcher k) -deriving instance Eq (FetchResult k) => Eq (NixFetcher k)+deriving instance Ord `ForFetchResult` k => Ord (NixFetcher k) -deriving instance Ord (FetchResult k) => Ord (NixFetcher k)+deriving instance Hashable `ForFetchResult` k => Hashable (NixFetcher k) -deriving instance Hashable (FetchResult k) => Hashable (NixFetcher k)+deriving instance Binary `ForFetchResult` k => Binary (NixFetcher k) -deriving instance Binary (FetchResult k) => Binary (NixFetcher k)+deriving instance NFData `ForFetchResult` k => NFData (NixFetcher k) -deriving instance NFData (FetchResult k) => NFData (NixFetcher k)+-- | Fetch status+data FetchStatus = Fresh | Fetched +-- | Prefetched fetchers hold hashes+type family FetchResult a (k :: FetchStatus) where+  FetchResult _ Fresh = ()+  FetchResult a Fetched = a+ instance A.ToJSON (NixFetcher Fetched) where   toJSON FetchGit {..} =     A.object@@ -434,6 +489,18 @@         "sha256" A..= _sha256,         "type" A..= A.String "tarball"       ]+  toJSON FetchDocker {..} =+    A.object+      [ "imageName" A..= _imageName,+        "imageTag" A..= _imageTag,+        "imageDigest" A..= _imageDigest,+        "sha256" A..= _sha256,+        "os" A..= _fos,+        "arch" A..= _farch,+        "finalImageName" A..= _finalImageName,+        "finalImageTag" A..= _finalImageTag,+        "tlsVerify" A..= _tlsVerify+      ]  instance Pretty (NixFetcher k) where   pretty FetchGit {..} =@@ -476,6 +543,21 @@         )   pretty FetchTarball {..} =     "FetchTarball" <> colon <+> pretty _furl+  pretty FetchDocker {..} =+    "FetchDocker"+      <> line+      <> indent+        2+        ( vsep $+            [ "imageName" <> colon <+> pretty _imageName,+              "imageTag" <> colon <+> pretty _finalImageTag+            ]+              <> ppField "os" _fos+              <> ppField "arch" _farch+              <> ppField "finalImageName" _finalImageName+              <> ppField "finalImageTag" _finalImageTag+              <> ppField "tlsVerify" _tlsVerify+        )  -------------------------------------------------------------------------------- @@ -488,7 +570,8 @@  instance Pretty ExtractSrcQ where   pretty (ExtractSrcQ f n) =-    "ExtractSrc" <> line+    "ExtractSrc"+      <> line       <> indent         2         ( vsep@@ -510,7 +593,8 @@  instance Pretty FetchRustGitDepsQ where   pretty (FetchRustGitDepsQ f n) =-    "FetchRustGitDeps" <> line+    "FetchRustGitDeps"+      <> line       <> indent         2         ( vsep@@ -521,6 +605,38 @@  -------------------------------------------------------------------------------- +-- | @strftime@ format+--+-- Nothing defaults to @%Y-%m-%d@+newtype DateFormat = DateFormat (Maybe Text)+  deriving newtype (Show, Eq, Ord, Default, Pretty)+  deriving stock (Typeable, Generic)+  deriving anyclass (Hashable, Binary, NFData)++-- | Get the commit date by using shallow clone+--+-- @_gformat@ is in.+-- Note: Requires git >= 2.5+data GetGitCommitDate = GetGitCommitDate {_gurl :: Text, _grev :: Text, _gformat :: DateFormat}+  deriving (Show, Eq, Ord, Hashable, NFData, Binary, Typeable, Generic)++type instance RuleResult GetGitCommitDate = Text++instance Pretty GetGitCommitDate where+  pretty GetGitCommitDate {..} =+    "GetGitCommitDate"+      <> line+      <> indent+        2+        ( vsep+            [ "url" <> colon <+> pretty _gurl,+              "rev" <> colon <+> pretty _grev,+              "format" <> colon <+> pretty _gformat+            ]+        )++--------------------------------------------------------------------------------+ -- | Package name, used in generating nix expr type PackageName = Text @@ -535,7 +651,7 @@   deriving newtype (Semigroup, Monoid)  -- | Using stale value indicates that we will /NOT/ check for new versions if--- there is a known version recoverd from json file or last use of the rule.+-- there is a known version recovered from json file or last use of the rule. -- Normally you don't want a stale version -- unless you need pin a package. data UseStaleVersion@@ -556,7 +672,8 @@ -- 5. optional @Cargo.lock@ path (if it's a rust package) -- 6. an optional pass through map -- 7. if the package version was pinned---+-- 8. optional git date format (if the version source is git)+-- 9. whether to always fetch a package regardless of the version changing -- /INVARIANT: 'Version' passed to 'PackageFetcher' MUST be used textually,/ -- /i.e. can only be concatenated with other strings,/ -- /in case we can't check the equality between fetcher functions./@@ -567,7 +684,9 @@     _pextract :: Maybe PackageExtractSrc,     _pcargo :: Maybe PackageCargoLockFiles,     _ppassthru :: PackagePassthru,-    _ppinned :: UseStaleVersion+    _ppinned :: UseStaleVersion,+    _pgitdateformat :: DateFormat,+    _pforcefetch :: ForceFetch   }  -- | Package key is the name of a package.@@ -604,7 +723,8 @@     _prextract :: Maybe (HashMap FilePath NixExpr),     -- | cargo lock file path in build dir -> (file path in nix, git dependencies)     _prcargolock :: Maybe (HashMap FilePath (NixExpr, HashMap Text Checksum)),-    _prpinned :: UseStaleVersion+    _prpinned :: UseStaleVersion,+    _prgitdate :: Maybe Text   }   deriving (Show, Typeable, Generic, NFData) @@ -619,5 +739,6 @@         "cargoLocks" A..= _prcargolock,         "pinned" A..= case _prpinned of           PermanentStale -> True-          _ -> False+          _ -> False,+        "date" A..= _prgitdate       ]

@@ -30,6 +30,9 @@     -- * Build dir     getBuildDir, +    -- * Keyfile+    getKeyfilePath,+     -- * Last versions     getLastVersionOnDisk,     getRecentLastVersion,@@ -114,6 +117,10 @@ -- | Get build dir getBuildDir :: Action FilePath getBuildDir = buildDir . config <$> getShakeExtras++-- | Get keyfile path+getKeyfilePath :: Action (Maybe FilePath)+getKeyfilePath = keyfile . config <$> getShakeExtras  -- | Get initial version of a package getLastVersionOnDisk :: PackageKey -> Action (Maybe Version)

@@ -30,7 +30,7 @@       runNvcheckerRule (Pypi "example") `shouldReturnJust` Version "0.1.0"      specifyChan "archpkg" $-      runNvcheckerRule (ArchLinux "ipw2100-fw") `shouldReturnJust` Version "1.3"+      runNvcheckerRule (ArchLinux "lib32-libva-vdpau-driver") `shouldReturnJust` Version "0.7.4"      specifyChan "aur" $       runNvcheckerRule (Aur "ssed") `shouldReturnJust` Version "3.62"@@ -64,17 +64,20 @@       runNvcheckerRule (Manual "Meow") `shouldReturnJust` Version "Meow"      specifyChan "openvsx" $-      runNvcheckerRule (OpenVsx "usernamehw" "indent-one-space") `shouldReturnJust` Version "0.2.7"+      runNvcheckerRule (OpenVsx "usernamehw" "indent-one-space") `shouldReturnJust` Version "0.3.0"      specifyChan "repology" $       runNvcheckerRule (Repology "ssed" "aur") `shouldReturnJust` Version "3.62"      specifyChan "vsmarketplace" $-      runNvcheckerRule (VscodeMarketplace "usernamehw" "indent-one-space") `shouldReturnJust` Version "0.2.8"+      runNvcheckerRule (VscodeMarketplace "usernamehw" "indent-one-space") `shouldReturnJust` Version "0.3.0"      specifyChan "cmd" $       runNvcheckerRule (Cmd "echo Meow") `shouldReturnJust` Version "Meow" +    specifyChan "container" $+      runNvcheckerRule (Container "testcontainers/helloworld" def) `shouldReturnJust` Version "1.1.0"+ --------------------------------------------------------------------------------  -- | We need disable nvchecker cache to see if use stale works@@ -132,7 +135,9 @@       _pcargo = undefined,       _pextract = undefined,       _ppassthru = undefined,-      _ppinned = NoStale+      _ppinned = NoStale,+      _pgitdateformat = undefined,+      _pforcefetch = undefined     }  fakePinnedPackage :: Package

@@ -30,7 +30,7 @@           ]  runPrefetchRule :: NixFetcher Fresh -> ReaderT ActionQueue IO (Maybe (NixFetcher Fetched))-runPrefetchRule fetcher = runActionChan $ prefetch fetcher+runPrefetchRule fetcher = runActionChan $ prefetch fetcher NoForceFetch  runFetchRustGitDepsRule :: NixFetcher Fetched -> FilePath -> ReaderT ActionQueue IO (Maybe (HashMap Text Checksum)) runFetchRustGitDepsRule fetcher lockPath = runActionChan $ fetchRustGitDeps fetcher lockPath

@@ -0,0 +1,30 @@+{-# LANGUAGE OverloadedStrings #-}++module GetGitCommitDateSpec where++import Control.Monad.Trans.Reader+import Data.Default (def)+import Data.Text (Text)+import NvFetcher.GetGitCommitDate+import Test.Hspec+import Utils++spec :: Spec+spec =+  aroundShake $+    describe "get git commit date" $ do+      specifyChan "default format" $+        runGetGitCommitDateRule repo rev def+          `shouldReturnJust` "2021-05-31"+      specifyChan "custom format" $+        runGetGitCommitDateRule repo rev (DateFormat $ Just "%Y-%m-%d %H:%M:%S")+          `shouldReturnJust` "2021-05-31 18:43:48"++repo :: Text+repo = "https://gist.github.com/NickCao/6c4dbc4e15db5da107de6cdb89578375"++rev :: Text+rev = "8a5f37a8f80a3b05290707febf57e88661cee442"++runGetGitCommitDateRule :: Text -> Text -> DateFormat -> ReaderT ActionQueue IO (Maybe Text)+runGetGitCommitDateRule url rev format = runActionChan $ getGitCommitDate url rev format

@@ -1,13 +1,16 @@+{-# LANGUAGE DataKinds #-}+{-# LANGUAGE LambdaCase #-} {-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE RecordWildCards #-} {-# LANGUAGE QuasiQuotes #-}  module NixExprSpec where -import qualified Data.List.NonEmpty as NE import NeatInterpolation (trimming) import NvFetcher.NixExpr import NvFetcher.NixFetcher import NvFetcher.Types+import PrefetchSpec import Test.Hspec  spec :: Spec@@ -23,7 +26,7 @@     toNixExpr ["Alice" :: String, "Bob", "Carol"] `shouldBe` [trimming|[ "Alice" "Bob" "Carol" ]|]    it "renders fresh gitFetcher" $-    toNixExpr (gitFetcher "https://example.com" "fake_rev")+    toNixExpr (fakeFetch (gitFetcher "https://example.com" "fake_rev"))       `shouldBe` [trimming|       fetchgit {         url = "https://example.com";@@ -31,61 +34,59 @@         fetchSubmodules = false;         deepClone = false;         leaveDotGit = false;-        sha256 = lib.fakeSha256;+        sha256 = "0000000000000000000000000000000000000000000000000000000000000000";       }     |]    it "renders fresh gitHubFetcher" $-    toNixExpr (gitHubFetcher ("owner", "repo") "fake_rev")+    toNixExpr (fakeFetch (gitHubFetcher ("owner", "repo") "fake_rev"))       `shouldBe` [trimming|       fetchFromGitHub ({         owner = "owner";         repo = "repo";         rev = "fake_rev";         fetchSubmodules = false;-        sha256 = lib.fakeSha256;+        sha256 = "0000000000000000000000000000000000000000000000000000000000000000";       })     |]    it "renders fresh urlFetcher" $-    toNixExpr (urlFetcher "https://example.com")+    toNixExpr (fakeFetch (urlFetcher "https://example.com"))       `shouldBe` [trimming|       fetchurl {         url = "https://example.com";-        sha256 = lib.fakeSha256;+        sha256 = "0000000000000000000000000000000000000000000000000000000000000000";       }     |]    it "renders filename for vsc extension" $-    toNixExpr (openVsxFetcher ("publisher", "extension") "fake_version")+    toNixExpr (fakeFetch (openVsxFetcher ("publisher", "extension") "fake_version"))       `shouldBe` [trimming|       fetchurl {         url = "https://open-vsx.org/api/publisher/extension/fake_version/file/publisher.extension-fake_version.vsix";         name = "extension-fake_version.zip";-        sha256 = lib.fakeSha256;+        sha256 = "0000000000000000000000000000000000000000000000000000000000000000";       }     |] -  it "renders IFD of ExtractSrcQ" $-    toNixExpr-      ( ExtractSrcQ-          (FetchUrl "https://example.com" Nothing (Checksum "calculated sha256"))-          (NE.fromList ["a.txt", "b.txt"])-      )+  it "renders fresh FetchDocker" $ do+    toNixExpr (fakeFetch testDockerFetcher)       `shouldBe` [trimming|-      let-        drv = import (pkgs.writeText "src" ''-          pkgs: {-            src = pkgs.fetchurl {-              url = "https://example.com";-              sha256 = "calculated sha256";-            };-          }-        '');-        fileNames = [ "a.txt" "b.txt" ];-        toFile = f: builtins.readFile ((drv pkgs).src + "/" + f);-      in builtins.listToAttrs (builtins.map (x: {-        name = x;-        value = toFile x;-      }) fileNames)-  |]+      dockerTools.pullImage {+        imageName = "library/alpine";+        imageDigest = "sha256:0000000000000000000000000000000000000000000000000000000000000000";+        sha256 = "0000000000000000000000000000000000000000000000000000000000000000";+        finalImageTag = "3.16.2";+      }+      |]++fakeFetch :: NixFetcher Fresh -> NixFetcher Fetched+fakeFetch = \case+  FetchGit {..} -> FetchGit {_sha256 = fakeSha256, ..}+  FetchGitHub {..} -> FetchGitHub {_sha256 = fakeSha256, ..}+  FetchUrl {..} -> FetchUrl {_sha256 = fakeSha256, ..}+  FetchTarball {..} -> FetchTarball {_sha256 = fakeSha256, ..}+  FetchDocker {..} -> FetchDocker {_sha256 = fakeSha256, _imageDigest = fakeDigest, ..}+  where+    fakeSha256 = Checksum "0000000000000000000000000000000000000000000000000000000000000000"+    fakeDigest = ContainerDigest "sha256:0000000000000000000000000000000000000000000000000000000000000000"

@@ -3,6 +3,7 @@  module PrefetchSpec where +import Control.Arrow ((&&&)) import Control.Monad.Trans.Reader import NvFetcher.NixFetcher import NvFetcher.Types@@ -36,7 +37,32 @@       runPrefetchRule (tarballFetcher "https://github.com/nixos/nixpkgs/archive/3d35529a48d3ad50ad959463755b0b7fe392cfa7.tar.gz")         `shouldReturnJust` Checksum "0la68sv52zz1kjw5s5sn6qslzz9mi9sakhzwi873gp4dhc8df1sg" +    specifyChan "docker" $+      runPrefetchRule' (_sha256 &&& _imageDigest) testDockerFetcher+        `shouldReturnJust`+          ( Checksum "1ly61z3bcs5qvqi2xxp3dd3llh61r9gygphl1ib8pxv64ix738mr",+            ContainerDigest "sha256:65a2763f593ae85fab3b5406dc9e80f744ec5b449f269b699b5efd37a07ad32e"+          )++testDockerFetcher :: NixFetcher Fresh+testDockerFetcher =+  FetchDocker+    { _imageName = "library/alpine",+      _imageTag = "3.16.2",+      _imageDigest = (),+      _sha256 = (),+      _fos = Nothing,+      _farch = Nothing,+      _finalImageName = Nothing,+      _finalImageTag = Nothing,+      _tlsVerify = Nothing+    }+ -------------------------------------------------------------------------------- +-- TODO test force fetch runPrefetchRule :: NixFetcher Fresh -> ReaderT ActionQueue IO (Maybe Checksum)-runPrefetchRule f = runActionChan $ _sha256 <$> prefetch f+runPrefetchRule = runPrefetchRule' _sha256++runPrefetchRule' :: (NixFetcher Fetched -> a) -> NixFetcher Fresh -> ReaderT ActionQueue IO (Maybe a)+runPrefetchRule' g f = runActionChan $ g <$> prefetch f NoForceFetch