nvfetcher 0.3.0.0 → 0.4.0.0
raw patch · 23 files changed
+849/−271 lines, 23 filesdep +asyncdep +hspecdep +stm
Dependencies added: async, hspec, stm, unliftio
Files
- CHANGELOG.md +17/−0
- Main_example.hs +2/−0
- README.md +25/−26
- app/Config.hs +50/−14
- app/Config/PackageFetcher.hs +55/−17
- app/Config/VersionSource.hs +41/−1
- nvfetcher.cabal +24/−1
- src/NvFetcher.hs +24/−18
- src/NvFetcher/Core.hs +37/−12
- src/NvFetcher/ExtractSrc.hs +14/−3
- src/NvFetcher/FetchRustGitDeps.hs +7/−4
- src/NvFetcher/NixExpr.hs +6/−2
- src/NvFetcher/NixFetcher.hs +20/−0
- src/NvFetcher/Nvchecker.hs +7/−1
- src/NvFetcher/Options.hs +13/−13
- src/NvFetcher/PackageSet.hs +182/−153
- src/NvFetcher/Types.hs +15/−6
- test/CheckVersionSpec.hs +79/−0
- test/FetchRustGitDepsSpec.hs +36/−0
- test/NixExprSpec.hs +69/−0
- test/PrefetchSpec.hs +38/−0
- test/Spec.hs +1/−0
- test/Utils.hs +87/−0
CHANGELOG.md view
@@ -1,5 +1,22 @@ # Revision history for nvfetcher +## 0.4.0.0++* Rename `_build` to `_sources`+* Remove the symlink `sources.nix -> _sources/generated.nix`+ * Remove CLI option `--output` (was used to set the symlink source name, `sources.nix` by default)+ * Add CLI option `build-dir` to specify build directory (`_sources` by default)+* Add CLI option `--commit-changes` to commit changes of build directory+* Support openvsx and vsmarketplace version sources (needs new version of nvchecker)+* Support attributes pass through+* Fix the bug that Core rule was cut off even if the configuration has changed+(no longer needs to use `nvfetcher clean` to keep the build system consistency manually)+* Fix the parser of git source in Cargo.lock+* Fix wrong trailing white spances in generated nix expr+* Fix missing semicolon in generated nix expr that reads Cargo.lock file+* Enhance eDSL experience+* Add some unit tests+ ## 0.3.0.0 There are massive enhancements since the last release:
Main_example.hs view
@@ -39,3 +39,5 @@ `sourceManual` "8a5f37a8f80a3b05290707febf57e88661cee442" `fetchGit` "https://gist.github.com/NickCao/6c4dbc4e15db5da107de6cdb89578375" `hasCargoLock` "Cargo.lock"++ define $ package "vscode-LiveServer" `fromOpenVsx` ("ritwickdey", "LiveServer")
README.md view
@@ -4,13 +4,13 @@ [](LICENSE) [](https://github.com/berberman/nvfetcher/actions/workflows/nix.yml) -nvfetcher is a tool to automate packages updates in flakes repos. It's built on top of [shake](https://www.shakebuild.com/),+nvfetcher is a tool to automate nix package updates. It's built on top of [shake](https://www.shakebuild.com/), integrating [nvchecker](https://github.com/lilydjwg/nvchecker). nvfetcher cli program accepts a TOML file as config, which defines a set of package sources to run. ## Overview -For example, given the following configuration file:+For example, feeding the following configuration to`nvfetcher`: ```toml # nvfetcher.toml@@ -24,7 +24,7 @@ git.fetchSubmodules = true ``` -running `nvfetcher build` will create `sources.nix` like:+it can create `sources.nix` like: ```nix # sources.nix@@ -54,21 +54,19 @@ ``` We tell nvfetcher how to get the latest version number of packages and how to fetch their sources given version numbers,-and nvfetcher will help us keep their version and prefetched SHA256 sums up-to-date, stored in `sources.nix`.-Shake will help us handle necessary rebuilds -- we check versions of packages during each run, but only prefetch them when needed.---`sources.nix` is a symlink of `_build/generated.nix`, which means that the "build" results produced by `nvfetcher` are under `_build`,-so you have to keep this directory for further use.+and nvfetcher will help us keep their version and prefetched SHA256 sums up-to-date, stored in `_sources/generated.nix`.+Shake will handle necessary rebuilds as long as you keep `_sources` directory -- we check versions of packages during each run, but only prefetch them when needed. ### Live examples -How to use the generated sources file? Here are some examples:+How to use the generated sources file? Here are several examples: -* My [flakes repo](https://github.com/berberman/flakes)+* [DevOS](https://github.com/divnix/devos/tree/main/pkgs) - Packages are defined in TOML -* Nick Cao's [flakes repo](https://gitlab.com/NickCao/flakes/-/tree/master/pkgs)+* My [flakes repo](https://github.com/berberman/flakes) - Packages are defined in eDSL +* Nick Cao's [flakes repo](https://gitlab.com/NickCao/flakes/-/tree/master/pkgs) - Packages are defined in TOML+ ## Installation `nvfetcher` package is available in [nixpkgs](https://github.com/NixOS/nixpkgs), so you can try it with:@@ -105,9 +103,9 @@ Available options: --version Show version --help Show this help text- -c,--config FILE Path to nvfetcher TOML config- (default: "nvfetcher.toml")- -o,--output FILE Path to output nix file (default: "sources.nix")+ -o,--build-dir FILE Directory that nvfetcher puts artifacts to+ (default: "_sources")+ --commit-changes `git commit` changes in this run (with shake db) -l,--changelog FILE Dump version changes to a file -j NUM Number of threads (0: detected number of processors) (default: 0)@@ -117,6 +115,8 @@ -v,--verbose Verbose mode TARGET Two targets are available: 1.build 2.clean (default: "build")+ -c,--config FILE Path to nvfetcher TOML config+ (default: "nvfetcher.toml") ``` Each *package* corresponds to a TOML table, whose name is encoded as table key, with@@ -126,7 +126,7 @@ #### Nvchecker Version source -- how do we track upstream version updates?-* `src.github = owner/repo` - the latest gituhb release+* `src.github = owner/repo` - the latest github release * `src.github_tag = owner/repo` - the max github tag, usually used with list options (see below) * `src.pypi = pypi_name` - the latest pypi release * `src.git = git_url` (and an optional `src.branch = git_branch`) - **the latest commit** of a repo@@ -136,6 +136,8 @@ * `src.repology = project:repo` -- the latest version from repology * `src.webpage = web_url` and `src.regex` -- a string in webpage that matches with regex * `src.httpheader = request_url` and `src.regex` -- a string in http header that matches with regex+* `src.openvsx = publisher.ext_name` -- the latest version of a vscode extension from open vsx+* `src.vsmarketplace = publisher.ext_name` -- the latest version of a vscode extension from vscode marketplace Optional list options for some version sources (`src.github_tag`, `src.webpage`, and `src.httpheader`),@@ -161,6 +163,8 @@ * `fetch.pypi = pypi_name` * `fetch.git = git_url` * `fetch.url = url`+* `fetch.openvsx = publisher.ext_name`+* `fetch.vsmarketplace = publisher.ext_name` Optional `nix-prefetch-git` config, which make sense only when the fetcher equals to `fetch.github` or `fetch.git`.@@ -182,7 +186,12 @@ extracting the lock file to build and calculating `cargoLock.outputHashes`, as long as you set the config * `cargo_lock = cargo_lock_path` - relative to the source root +#### Passthru +*passthru* config, an additional set of attrs to be generated.++ * `passthru = { k1 = "v1", k2 = "v2", ... }`+ ### Haskell library nvfetcher itsetlf is a Haskell library as well, whereas the CLI program is just a trivial wrapper of the library.@@ -195,16 +204,6 @@ For details of the library, documentation of released versions is available on [Hackage](https://hackage.haskell.org/package/nvfetcher), and of master is on our [github pages](https://nvfetcher.berberman.space).--## Limitations--There is no way to check the equality over version sources and fetchers, so If you change either of them in a package,-you will need to rebuild everything, i.e. run `nvfetcher clean` to remove shake databsae, to make sure that-our build system works correctly. We could automate this process, for example,-calculate the hash of the configuration file and bump `shakeVersion` to trigger the rebuild.-However, this shouldn't happen frequently and we want to minimize the changes, so it's left for you to do manually.--> Adding or removing a package doesn't require such rebuild ## Contributing
app/Config.hs view
@@ -10,10 +10,9 @@ import Config.PackageFetcher import Config.VersionSource+import Data.Coerce (coerce) import Data.List.NonEmpty (NonEmpty ((:|)))-import Data.Maybe (fromMaybe)-import Lens.Micro-import Lens.Micro.Extras (view)+import qualified Data.List.NonEmpty as NE import NvFetcher.Types import Toml import Validation (validationToEither)@@ -25,19 +24,53 @@ go (Right x : xs) se sp = go xs se (x : sp) go [] [] sp = Right sp go [] se _ = Left se- tables = [fmap (toPackage k) $ validationToEither $ Toml.runTomlCodec iCodec v | (Toml.unKey -> (Toml.unPiece -> k) :| _, v) <- Toml.toList $ Toml.tomlTables toml]- toPackage k (versionSource, f, e, c, options) = Package k (NvcheckerQ versionSource options) f e c- iCodec =- (,,,,)- <$> versionSourceCodec .= view _1- <*> fetcherCodec .= view _2- <*> extractFilesCodec .= view _3- <*> cargoLockPathCodec .= view _4- <*> nvcheckerOptionsCodec .= view _5+ tables =+ [ fmap (toPackage (coerce k)) $+ validationToEither $+ Toml.runTomlCodec packageConfigCodec v+ | (Toml.unKey -> (Toml.unPiece -> k) :| _, v) <- Toml.toList $ Toml.tomlTables toml+ ] -extractFilesCodec :: TomlCodec PackageExtractSrc-extractFilesCodec = diwrap $ dimap Just (fromMaybe []) $ dioptional $ arrayOf _String "extract"+-------------------------------------------------------------------------------- +data PackageConfig = PackageConfig+ { pcVersionSource :: VersionSource,+ pcFetcher :: PackageFetcher,+ pcExtractFiles :: Maybe PackageExtractSrc,+ pcCargoLockPath :: Maybe PackageCargoFilePath,+ pcNvcheckerOptions :: NvcheckerOptions,+ pcPassthru :: PackagePassthru+ }++toPackage :: PackageKey -> PackageConfig -> Package+toPackage k PackageConfig {..} =+ Package+ (coerce k)+ (NvcheckerQ pcVersionSource pcNvcheckerOptions)+ pcFetcher+ pcExtractFiles+ pcCargoLockPath+ pcPassthru++packageConfigCodec :: TomlCodec PackageConfig+packageConfigCodec =+ PackageConfig+ <$> versionSourceCodec .= pcVersionSource+ <*> fetcherCodec .= pcFetcher+ <*> extractFilesCodec .= pcExtractFiles+ <*> cargoLockPathCodec .= pcCargoLockPath+ <*> nvcheckerOptionsCodec .= pcNvcheckerOptions+ <*> passthruCodec .= pcPassthru++--------------------------------------------------------------------------------++extractFilesCodec :: TomlCodec (Maybe PackageExtractSrc)+extractFilesCodec =+ dimap+ (fmap (NE.toList . coerce))+ (\mxs -> coerce <$> (mxs >>= NE.nonEmpty))+ $ dioptional $ arrayOf _String "extract"+ cargoLockPathCodec :: TomlCodec (Maybe PackageCargoFilePath) cargoLockPathCodec = dioptional $ diwrap (string "cargo_lock") @@ -47,3 +80,6 @@ <$> dioptional (text "src.prefix") .= _stripPrefix <*> dioptional (text "src.from_pattern") .= _fromPattern <*> dioptional (text "src.to_pattern") .= _toPattern++passthruCodec :: TomlCodec PackagePassthru+passthruCodec = diwrap $ tableHashMap _KeyText text "passthru"
app/Config/PackageFetcher.hs view
@@ -7,9 +7,9 @@ module Config.PackageFetcher (fetcherCodec) where -import Control.Applicative ((<|>)) import Data.Coerce (coerce) import Data.Default (Default, def)+import Data.Foldable (asum) import Data.Maybe (fromMaybe) import qualified Data.Text as T import GHC.Generics (Generic)@@ -23,6 +23,17 @@ unsupportError :: a unsupportError = error "serialization is unsupported" +fetcherCodec :: TomlCodec PackageFetcher+fetcherCodec =+ asum+ [ gitHubCodec,+ pypiCodec,+ openVsxCodec,+ vscodeMarketplaceCodec,+ gitCodec,+ urlCodec+ ]+ -------------------------------------------------------------------------------- data GitOptions = GitOptions@@ -52,7 +63,7 @@ -------------------------------------------------------------------------------- -gitHubICodec :: TomlCodec (Version -> NixFetcher 'Fresh)+gitHubICodec :: TomlCodec PackageFetcher gitHubICodec = textBy unsupportError@@ -62,7 +73,7 @@ ) "fetch.github" -gitHubCodec :: TomlCodec (Version -> NixFetcher Fresh)+gitHubCodec :: TomlCodec PackageFetcher gitHubCodec = dimap ( \f -> let fake = f "$ver" in (f, fromMaybe def $ fake ^? _GitOptions)@@ -71,14 +82,14 @@ $ (,) <$> gitHubICodec .= view _1 <*> gitOptionsCodec .= view _2 ---------------------------------------------------------------------------------gitICodec :: TomlCodec (Version -> NixFetcher 'Fresh)+gitICodec :: TomlCodec PackageFetcher gitICodec = textBy unsupportError (Right . gitFetcher) "fetch.git" -gitCodec :: TomlCodec (Version -> NixFetcher Fresh)+gitCodec :: TomlCodec PackageFetcher gitCodec = dimap ( \f -> let fake = f "$ver" in (f, fromMaybe def $ fake ^? _GitOptions)@@ -87,16 +98,43 @@ $ (,) <$> gitICodec .= view _1 <*> gitOptionsCodec .= view _2 --------------------------------------------------------------------------------+pypiCodec :: TomlCodec PackageFetcher+pypiCodec =+ Toml.textBy+ unsupportError+ (Right . pypiFetcher)+ "fetch.pypi" -fetcherCodec :: TomlCodec PackageFetcher-fetcherCodec =- gitHubCodec- <|> Toml.textBy- unsupportError- (Right . pypiFetcher)- "fetch.pypi"- <|> gitCodec- <|> Toml.textBy- unsupportError- (\t -> Right $ \(coerce -> v) -> urlFetcher $ T.replace "$ver" v t)- "fetch.url"+--------------------------------------------------------------------------------++openVsxCodec :: TomlCodec PackageFetcher+openVsxCodec =+ textBy+ unsupportError+ ( \t -> case T.split (== '.') t of+ -- assume we can't have '.' in extension's name+ [publisher, extName] -> Right $ openVsxFetcher (publisher, extName)+ _ -> Left "unexpected openvsx fetcher: it should be something like [publisher]/[extName]"+ )+ "fetch.openvsx"++--------------------------------------------------------------------------------++vscodeMarketplaceCodec :: TomlCodec PackageFetcher+vscodeMarketplaceCodec =+ textBy+ unsupportError+ ( \t -> case T.split (== '.') t of+ -- assume we can't have '.' in extension's name+ [publisher, extName] -> Right $ vscodeMarketplaceFetcher (publisher, extName)+ _ -> Left "unexpected vscode marketplace fetcher: it should be something like [publisher]/[extName]"+ )+ "fetch.vsmarketplace"++--------------------------------------------------------------------------------+urlCodec :: TomlCodec PackageFetcher+urlCodec =+ Toml.textBy+ unsupportError+ (\t -> Right $ \(coerce -> v) -> urlFetcher $ T.replace "$ver" v t)+ "fetch.url"
app/Config/VersionSource.hs view
@@ -26,7 +26,9 @@ manualCodec, repologyCodec, webpageCodec,- httpHeaderCodec+ httpHeaderCodec,+ openVsxCodec,+ vscodeMarketplaceCodec ] --------------------------------------------------------------------------------@@ -175,3 +177,41 @@ httpHeaderCodec = dimatch matchHttpHeader (uncurry3 HttpHeader) httpHeaderICodec --------------------------------------------------------------------------------++matchOpenVsx :: VersionSource -> Maybe (Text, Text)+matchOpenVsx x = (,) <$> x ^? ovPublisher <*> x ^? ovExtName++openVsxICodec :: TomlCodec (Text, Text)+openVsxICodec =+ textBy+ (\(publisher, extName) -> publisher <> "." <> extName)+ ( \t ->+ case T.split (== '.') t of+ -- assume we can't have '.' in extension's name+ [publisher, extName] -> Right (publisher, extName)+ _ -> Left "unexpected openvsx source format: it should be something like [publisher].[extName]"+ )+ "src.openvsx"++openVsxCodec :: TomlCodec VersionSource+openVsxCodec = dimatch matchOpenVsx (uncurry OpenVsx) openVsxICodec++--------------------------------------------------------------------------------++matchVscodeMarketplace :: VersionSource -> Maybe (Text, Text)+matchVscodeMarketplace x = (,) <$> x ^? vsmPublisher <*> x ^? vsmExtName++vscodeMarketplaceICodec :: TomlCodec (Text, Text)+vscodeMarketplaceICodec =+ textBy+ (\(publisher, extName) -> publisher <> "." <> extName)+ ( \t ->+ case T.split (== '.') t of+ -- assume we can't have '.' in extension's name+ [publisher, extName] -> Right (publisher, extName)+ _ -> Left "unexpected vscode marketplace source format: it should be something like [publisher].[extName]"+ )+ "src.vsmarketplace"++vscodeMarketplaceCodec :: TomlCodec VersionSource+vscodeMarketplaceCodec = dimatch matchVscodeMarketplace (uncurry VscodeMarketplace) vscodeMarketplaceICodec
nvfetcher.cabal view
@@ -1,6 +1,6 @@ cabal-version: 2.4 name: nvfetcher-version: 0.3.0.0+version: 0.4.0.0 synopsis: Generate nix sources expr for the latest version of packages @@ -104,3 +104,26 @@ main-is: Main_example.hs build-depends: nvfetcher ghc-options: -threaded -rtsopts -with-rtsopts=-N++test-suite tests+ import: common-options+ type: exitcode-stdio-1.0+ main-is: Spec.hs+ other-modules:+ CheckVersionSpec+ FetchRustGitDepsSpec+ NixExprSpec+ PrefetchSpec+ Utils++ hs-source-dirs: test+ ghc-options: -threaded -rtsopts -with-rtsopts=-N+ build-depends:+ , async+ , hspec+ , nvfetcher+ , stm+ , unliftio++ build-tool-depends: hspec-discover:hspec-discover -any+ default-language: Haskell2010
src/NvFetcher.hs view
@@ -50,7 +50,7 @@ ) where -import qualified Control.Exception as CE+import Control.Monad.Extra (when, whenJust) import Data.Text (Text) import qualified Data.Text as T import Development.Shake@@ -64,7 +64,6 @@ import NvFetcher.Types import NvFetcher.Types.ShakeExtras import NvFetcher.Utils (getShakeDir)-import System.Directory.Extra (createDirectoryIfMissing, createFileLink, removeFile) -- | Arguments for running nvfetcher data Args = Args@@ -72,8 +71,8 @@ argShakeOptions :: ShakeOptions, -- | Build target argTarget :: String,- -- | Output file path- argOutputFilePath :: FilePath,+ -- | Shake dir+ argBuildDir :: FilePath, -- | Custom rules argRules :: Rules (), -- | Action run after build rule@@ -86,7 +85,7 @@ -- | Default arguments of 'defaultMain' ----- Output file path is @sources.nix@.+-- Build dir is @_sources@. defaultArgs :: Args defaultArgs = Args@@ -96,7 +95,7 @@ } ) "build"- "sources.nix"+ "_sources" (pure ()) (pure ()) (pure ())@@ -114,14 +113,16 @@ cliOptionsToArgs :: CLIOptions -> Args cliOptionsToArgs CLIOptions {..} = defaultArgs- { argOutputFilePath = outputPath,- argActionAfterBuild = maybe (pure ()) logChangesToFile logPath,+ { argActionAfterBuild = do+ whenJust logPath logChangesToFile+ when commit commitChanges, argTarget = target, argShakeOptions = (argShakeOptions defaultArgs) { shakeTimings = timing, shakeVerbosity = if verbose then Verbose else Info,- shakeThreads = threads+ shakeThreads = threads,+ shakeFiles = buildDir } } @@ -130,6 +131,18 @@ changes <- getVersionChanges writeFile' fp $ unlines $ show <$> changes +commitChanges :: Action ()+commitChanges = do+ changes <- getVersionChanges+ let commitMsg = case changes of+ [x] -> Just $ show x+ xs@(_ : _) -> Just $ "Update\n" <> unlines (show <$> xs)+ [] -> Nothing+ whenJust commitMsg $ \msg -> do+ putInfo "Commiting changes"+ getShakeDir >>= \dir -> command_ [] "git" ["add", dir]+ command_ [] "git" ["commit", "-m", msg]+ -- | Entry point of nvfetcher runNvFetcherNoCLI :: Args -> PackageSet () -> IO () runNvFetcherNoCLI args@Args {..} packageSet = do@@ -137,8 +150,7 @@ shakeExtras <- initShakeExtras pkgs argRetries let opts = argShakeOptions- { shakeExtra = addShakeExtra shakeExtras (shakeExtra argShakeOptions),- shakeFiles = "_build"+ { shakeExtra = addShakeExtra shakeExtras (shakeExtra argShakeOptions) } rules = mainRules args shake opts $ want [argTarget] >> rules@@ -148,8 +160,7 @@ mainRules :: Args -> Rules () mainRules Args {..} = do "clean" ~> do- removeFilesAfter "_build" ["//*"]- removeFilesAfter "." [argOutputFilePath]+ getShakeDir >>= flip removeFilesAfter ["//*"] argActionAfterClean "build" ~> do@@ -166,11 +177,6 @@ putVerbose $ "Generating " <> genPath writeFileChanged genPath $ T.unpack $ srouces (T.unlines body) <> "\n" need [genPath]- let outDir = takeDirectory argOutputFilePath- liftIO $ createDirectoryIfMissing True outDir- liftIO $ CE.catch @IOError (removeFile argOutputFilePath) (const $ pure ())- putVerbose $ "Symlinking " <> argOutputFilePath- liftIO $ createFileLink genPath argOutputFilePath argActionAfterBuild argRules
src/NvFetcher/Core.hs view
@@ -43,6 +43,7 @@ prefetchRule extractSrcRule fetchRustGitDepsRule+ -- TODO: maybe an oracle rule is enough addBuiltinRule noLint noIdentity $ \(WithPackageKey (Core, pkg)) mOld mode -> case mode of RunDependenciesSame | Just old <- mOld,@@ -54,24 +55,27 @@ Just Package { _pversion = NvcheckerQ versionSource options,- _pextract = PackageExtractSrc extract,+ _ppassthru = PackagePassthru passthruMap, .. } -> do+ -- it's important to always rerun+ -- since the package definition is not tracked at all+ alwaysRerun (NvcheckerA version mOldV) <- checkVersion versionSource options pkg prefetched <- prefetch $ _pfetcher version shakeDir <- getShakeDir+ -- extract src appending1 <-- if null extract- then pure ""- else do- result <- HMap.toList <$> extractSrc prefetched extract+ case _pextract of+ Just (PackageExtractSrc extract) -> do+ result <- HMap.toList <$> extractSrcs prefetched extract T.unlines <$> sequence [ do -- write extracted files to shake dir -- and read them in nix using 'builtins.readFile' writeFile' (shakeDir </> path) (T.unpack v)- pure $ toNixExpr k <> " = builtins.readFile ./" <> T.pack path+ pure $ toNixExpr k <> " = builtins.readFile ./" <> T.pack path <> ";" | (k, v) <- result, let path = T.unpack _pname@@ -79,10 +83,12 @@ <> T.unpack (coerce version) </> k ]+ _ -> pure ""+ -- cargo lock appending2 <- case _pcargo of Just (PackageCargoFilePath lockPath) -> do- (_, lockData) <- head . HMap.toList <$> extractSrc prefetched [lockPath]+ (_, lockData) <- head . HMap.toList <$> extractSrc prefetched lockPath result <- fetchRustGitDeps prefetched lockPath let body = T.unlines [asString k <> " = " <> coerce (asString $ coerce v) <> ";" | (k, v) <- HMap.toList result] lockPath' =@@ -103,6 +109,10 @@ }; |] _ -> pure ""+ -- passthru+ let appending3 = T.unlines [k <> " = " <> v <> ";" | (k, asString -> v) <- HMap.toList passthruMap]++ -- update changelog case mOldV of Nothing -> recordVersionChange _pname Nothing version@@ -110,15 +120,28 @@ | old /= version -> recordVersionChange _pname (Just old) version _ -> pure ()- let result = gen _pname version prefetched $ appending1 <> appending2++ let result = gen _pname version prefetched $ appending1 <> appending2 <> appending3 pure $ RunResult ChangedRecomputeDiff (encode' (result, version)) result -- | Run the core rule. -- Given a 'PackageKey', run "NvFetcher.Nvchecker", "NvFetcher.NixFetcher"--- (may also run "NvFetcher.ExtractSrc" or "FetchRustGitDeps")--- resulting a nix source snippet like:+-- (may also run "NvFetcher.ExtractSrc" or "NvFetrcher.FetchRustGitDeps") -- -- @+-- Package+-- { _pname = "feeluown-core",+-- _pversion = NvcheckerQ (Pypi "feeluown") def,+-- _pfetcher = pypiFetcher "feeluown",+-- _pextract = Nothing,+-- _pcargo = Nothing,+-- _ppassthru = PackagePassthru (HashMap.fromList [("a", "B")])+-- }+-- @+--+-- resulting a nix exprs snippet like:+--+-- @ -- feeluown-core = { -- pname = "feeluown-core"; -- version = "3.7.7";@@ -126,6 +149,7 @@ -- sha256 = "06d3j39ff9znqxkhp9ly81lcgajkhg30hyqxy2809yn23xixg3x2"; -- url = "https://pypi.io/packages/source/f/feeluown/feeluown-3.7.7.tar.gz"; -- };+-- a = "B"; -- }; -- @ generateNixSourceExpr :: PackageKey -> Action NixExpr@@ -137,7 +161,8 @@ $name = { pname = "$name"; version = "$ver";- src = $srcP;- $appending+ src = $srcP;$appending' }; |]+ where+ appending' = if T.null appending then "" else "\n" <> appending
src/NvFetcher/ExtractSrc.hs view
@@ -21,6 +21,7 @@ -- * Rules extractSrcRule, extractSrc,+ extractSrcs, ) where @@ -28,6 +29,7 @@ import qualified Data.Aeson as A import Data.Binary.Instances () import Data.HashMap.Strict (HashMap)+import qualified Data.List.NonEmpty as NE import Data.Text (Text) import qualified Data.Text as T import qualified Data.Text.Encoding as T@@ -50,14 +52,23 @@ Just x -> pure x _ -> fail $ "Failed to parse output of nix-instantiate: " <> T.unpack (T.decodeUtf8 out) +-- | Run extract source with many sources+extractSrcs ::+ -- | prefetched source+ NixFetcher Fetched ->+ -- | relative file paths to extract+ NE.NonEmpty FilePath ->+ Action (HashMap FilePath Text)+extractSrcs fetcher xs = askOracle (ExtractSrcQ fetcher xs)+ -- | Run extract source extractSrc :: -- | prefetched source NixFetcher Fetched ->- -- | relative file paths to extract- [FilePath] ->+ -- | relative file path to extract+ FilePath -> Action (HashMap FilePath Text)-extractSrc fetcher fp = askOracle (ExtractSrcQ fetcher fp)+extractSrc fetcher fp = extractSrcs fetcher $ NE.fromList [fp] --------------------------------------------------------------------------------
src/NvFetcher/FetchRustGitDeps.hs view
@@ -42,7 +42,7 @@ fetchRustGitDepsRule :: Rules () fetchRustGitDepsRule = void $ addOracleCache $ \(FetchRustGitDepsQ fetcher lockPath) -> do- cargoLock <- head . HMap.elems <$> extractSrc fetcher [lockPath]+ cargoLock <- head . HMap.elems <$> extractSrc fetcher lockPath deps <- case Toml.decode (Toml.list rustDepCodec "package") cargoLock of Right r -> pure $ nubOrdOn rrawSrc r Left err -> fail $ "Failed to parse Cargo.lock: " <> T.unpack (Toml.prettyTomlDecodeErrors err)@@ -78,18 +78,21 @@ } deriving (Show, Eq, Ord) --- | Parse src in lock: git\+([^?]+)(\?rev=(.*))?#(.*)?+-- | Parse git src in cargo lock file -- >>> parse gitSrcParser "test" "git+https://github.com/rust-random/rand.git?rev=0.8.3#6ecbe2626b2cc6110a25c97b1702b347574febc7" -- Right (ParsedGitSrc {pgurl = "https://github.com/rust-random/rand.git", pgsha = "6ecbe2626b2cc6110a25c97b1702b347574febc7"}) -- -- >>> parse gitSrcParser "test" "git+https://github.com/rust-random/rand.git#f0e01ee0a7257753cc51b291f62666f4765923ef" -- Right (ParsedGitSrc {pgurl = "https://github.com/rust-random/rand.git", pgsha = "f0e01ee0a7257753cc51b291f62666f4765923ef"})+--+-- >>> parse gitSrcParser "test" "git+https://github.com/rust-lang/cargo?branch=rust-1.53.0#4369396ce7d270972955d876eaa4954bea56bcd9"+-- Right (ParsedGitSrc {pgurl = "https://github.com/rust-lang/cargo", pgsha = "4369396ce7d270972955d876eaa4954bea56bcd9"}) gitSrcParser :: Parser ParsedGitSrc gitSrcParser = do _ <- string "git+" pgurl <- many1 $ noneOf ['?', '#']- let revParser = string "?rev=" >> many1 (noneOf ['#'])- _rev <- optionMaybe revParser+ -- skip things like ?rev and ?branch+ skipMany (noneOf ['#']) _ <- char '#' pgsha <- manyTill anyChar eof pure $ ParsedGitSrc (T.pack pgurl) (coerce $ T.pack pgsha)
src/NvFetcher/NixExpr.hs view
@@ -21,6 +21,7 @@ where import Data.Coerce (coerce)+import qualified Data.List.NonEmpty as NE import Data.Text (Text) import qualified Data.Text as T import NeatInterpolation (trimming)@@ -44,6 +45,9 @@ instance ToNixExpr a => ToNixExpr [a] where toNixExpr xs = foldl (\acc x -> acc <> " " <> toNixExpr x) "[" xs <> " ]" +instance ToNixExpr a => ToNixExpr (NE.NonEmpty a) where+ toNixExpr = toNixExpr . NE.toList+ instance {-# OVERLAPS #-} ToNixExpr String where toNixExpr = T.pack . show @@ -76,15 +80,15 @@ (FetchUrl (asString -> url) _) -> [trimming| fetchurl {- sha256 = $sha256; url = $url;+ sha256 = $sha256; } |] instance ToNixExpr ExtractSrcQ where toNixExpr (ExtractSrcQ fetcher files) = extractFiles fetcher files -extractFiles :: NixFetcher Fetched -> [FilePath] -> NixExpr+extractFiles :: NixFetcher Fetched -> NE.NonEmpty FilePath -> NixExpr extractFiles (toNixExpr -> fetcherExpr) (toNixExpr -> fileNames) = [trimming| let
src/NvFetcher/NixFetcher.hs view
@@ -39,6 +39,8 @@ gitHubReleaseFetcher, gitFetcher, urlFetcher,+ openVsxFetcher,+ vscodeMarketplaceFetcher, ) where @@ -128,3 +130,21 @@ -- | Create a fetcher from url urlFetcher :: Text -> NixFetcher Fresh urlFetcher = flip FetchUrl ()++-- | Create a fetcher from openvsx+openVsxFetcher ::+ -- | publisher and extension name+ (Text, Text) ->+ PackageFetcher+openVsxFetcher (publisher, extName) (coerce -> ver) =+ urlFetcher+ [trimming|https://open-vsx.org/api/$publisher/$extName/$ver/file/$publisher.$extName-$ver.vsix|]++-- | Create a fetcher from vscode marketplace+vscodeMarketplaceFetcher ::+ -- | publisher and extension name+ (Text, Text) ->+ PackageFetcher+vscodeMarketplaceFetcher (publisher, extName) (coerce -> ver) =+ urlFetcher+ [trimming|https://$publisher.gallery.vsassets.io/_apis/public/gallery/publisher/$publisher/extension/$extName/$ver/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage|]
src/NvFetcher/Nvchecker.hs view
@@ -105,7 +105,7 @@ "strip_release" =: Bool True ArchLinux {..} -> do "source" =: "archpkg"- "aur" =: Text _archpkg+ "archpkg" =: Text _archpkg "strip_release" =: Bool True Pypi {..} -> do "source" =: "pypi"@@ -127,6 +127,12 @@ "url" =: Text _vurl "regex" =: Text _regex genListOptions _listOptions+ OpenVsx {..} -> do+ "source" =: "openvsx"+ "openvsx" =: Text (_ovPublisher <> "." <> _ovExtName)+ VscodeMarketplace {..} -> do+ "source" =: "vsmarketplace"+ "vsmarketplace" =: Text (_vsmPublisher <> "." <> _vsmExtName) genListOptions ListOptions {..} = do "include_regex" =:? _includeRegex "exclude_regex" =:? _excludeRegex
src/NvFetcher/Options.hs view
@@ -19,7 +19,8 @@ -- | Options for nvfetcher CLI data CLIOptions = CLIOptions- { outputPath :: FilePath,+ { buildDir :: FilePath,+ commit :: Bool, logPath :: Maybe FilePath, threads :: Int, retries :: Int,@@ -33,14 +34,18 @@ cliOptionsParser = CLIOptions <$> strOption- ( long "output"+ ( long "build-dir" <> short 'o'- <> metavar "FILE"- <> help "Path to output nix file"+ <> metavar "DIR"+ <> help "Directory that nvfetcher puts artifacts to" <> showDefault- <> value "sources.nix"- <> completer (bashCompleter "file")+ <> value "_sources"+ <> completer (bashCompleter "directory") )+ <*> switch+ ( long "commit-changes"+ <> help "`git commit` changes in this run (with shake db)"+ ) <*> optional ( strOption ( long "changelog"@@ -86,13 +91,8 @@ (opts, ()) <- simpleOptions version- "nvfetcher - generate nix sources expr for the latest version of packages"- ( unlines- [ "It's important to keep _build dir.",- "If you change any field of an existing package, you may have to run target \"clean\" to invalidate the databse,",- "making sure the consistency of our build system."- ]- )+ "nvfetcher"+ "generate nix sources expr for the latest version of packages" parser empty pure opts
src/NvFetcher/PackageSet.hs view
@@ -6,6 +6,8 @@ {-# LANGUAGE KindSignatures #-} {-# LANGUAGE LambdaCase #-} {-# LANGUAGE MultiParamTypeClasses #-}+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE RankNTypes #-} {-# LANGUAGE RecordWildCards #-} {-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TypeFamilies #-}@@ -47,6 +49,8 @@ fromGitHubTag, fromGitHubTag', fromPypi,+ fromOpenVsx,+ fromVscodeMarketplace, -- ** Version sources sourceGitHub,@@ -60,6 +64,8 @@ sourceRepology, sourceWebpage, sourceHttpHeader,+ sourceOpenVsx,+ sourceVscodeMarketplace, -- ** Fetchers fetchGitHub,@@ -69,16 +75,25 @@ fetchGit, fetchGit', fetchUrl,+ fetchOpenVsx,+ fetchVscodeMarketplace, -- * Addons extractSource, hasCargoLock, tweakVersion,+ passthru, -- ** Miscellaneous Prod,+ Append, Member,+ OptionalMember, NotElem,+ Members,+ OptionalMembers,+ Attach,+ AttachMany, coerce, liftIO, @@ -96,14 +111,15 @@ import Control.Monad.IO.Class import Data.Coerce (coerce) import Data.Default (def)+import qualified Data.HashMap.Strict as HMap import Data.Kind (Constraint, Type)-import Data.Map.Strict as HMap+import qualified Data.List.NonEmpty as NE+import Data.Map.Strict as Map import Data.Maybe (fromMaybe, isJust) import Data.Text (Text) import GHC.TypeLits import Lens.Micro import NvFetcher.NixFetcher--- import NvFetcher.PostFetcher import NvFetcher.Types import NvFetcher.Types.Lens @@ -135,10 +151,11 @@ PackageName -> NvcheckerQ -> PackageFetcher ->- PackageExtractSrc ->+ Maybe PackageExtractSrc -> Maybe PackageCargoFilePath ->+ PackagePassthru -> PackageSet ()-newPackage name source fetcher extract cargo = liftF $ NewPackage (Package name source fetcher extract cargo) ()+newPackage name source fetcher extract cargo pasthru = liftF $ NewPackage (Package name source fetcher extract cargo pasthru) () -- | Add a list of packages into package set purePackageSet :: [Package] -> PackageSet ()@@ -152,9 +169,9 @@ runPackageSet = \case Free (NewPackage p g) -> runPackageSet g >>= \m ->- if isJust (HMap.lookup (PackageKey $ _pname p) m)+ if isJust (Map.lookup (PackageKey $ _pname p) m) then fail $ "Duplicate package name: " <> show (_pname p)- else pure $ HMap.insert (PackageKey $ _pname p) p m+ else pure $ Map.insert (PackageKey $ _pname p) p m Free (EmbedIO action g) -> action >>= runPackageSet . g Pure _ -> pure mempty @@ -178,6 +195,7 @@ instance TypeError (ShowType x :<>: 'Text " is undefined") => Member x '[] where proj = undefined +-- | Project optional elements from 'Prod' class OptionalMember (a :: Type) (r :: [Type]) where projMaybe :: Prod r -> Maybe a @@ -196,6 +214,27 @@ NotElem x (_ ': xs) = NotElem x xs NotElem x '[] = () +-- | A list of 'Member'+type family Members xs r :: Constraint where+ Members '[] _ = ()+ Members (x ': xs) r = (Member x r, Members xs r)++-- | A list of 'OptionalMember'+type family OptionalMembers xs r :: Constraint where+ OptionalMembers '[] _ = ()+ OptionalMembers (x ': xs) r = (OptionalMember x r, OptionalMembers xs r)++-- | @xs ++ ys@, at type level+type family Append xs ys where+ Append '[] ys = ys+ Append (x ': xs) ys = x ': Append xs ys++-- | Attach members @xs@, with a function argument @arg@+type AttachMany xs arg = forall r. PackageSet (Prod r) -> arg -> PackageSet (Prod (Append xs r))++-- | Attach member @x@, with a function @arg@+type Attach x arg = AttachMany '[x] arg+ -------------------------------------------------------------------------------- -- | A tagless final style DSL for constructing packages@@ -203,12 +242,19 @@ new :: f PackageName -> f (Prod '[PackageName]) andThen :: f (Prod r) -> f a -> f (Prod (a ': r)) end ::- ( Member PackageName r,- Member VersionSource r,- Member PackageFetcher r,- OptionalMember PackageExtractSrc r,- OptionalMember PackageCargoFilePath r,- OptionalMember NvcheckerOptions r+ ( Members+ '[ PackageName,+ VersionSource,+ PackageFetcher+ ]+ r,+ OptionalMembers+ '[ PackageExtractSrc,+ PackageCargoFilePath,+ NvcheckerOptions,+ PackagePassthru+ ]+ r ) => f (Prod r) -> f ()@@ -227,8 +273,9 @@ (proj p) (NvcheckerQ (proj p) (fromMaybe def (projMaybe p))) (proj p)- (fromMaybe (PackageExtractSrc []) $ projMaybe p) (projMaybe p)+ (projMaybe p)+ (fromMaybe mempty (projMaybe p)) -- | 'PkgDSL' version of 'newPackage' --@@ -238,12 +285,19 @@ -- define $ package "nvfetcher-git" `sourceGit` "https://github.com/berberman/nvfetcher" `fetchGitHub` ("berberman", "nvfetcher") -- @ define ::- ( Member PackageName r,- Member VersionSource r,- Member PackageFetcher r,- OptionalMember PackageExtractSrc r,- OptionalMember PackageCargoFilePath r,- OptionalMember NvcheckerOptions r+ ( Members+ '[ PackageName,+ VersionSource,+ PackageFetcher+ ]+ r,+ OptionalMembers+ '[ PackageExtractSrc,+ PackageCargoFilePath,+ PackagePassthru,+ NvcheckerOptions+ ]+ r ) => PackageSet (Prod r) -> PackageSet ()@@ -254,155 +308,140 @@ package = new . pure -- | Attach version sources-src :: PackageSet (Prod r) -> VersionSource -> PackageSet (Prod (VersionSource ': r))+src :: Attach VersionSource VersionSource src = (. pure) . andThen -- | Attach fetchers-fetch ::- PackageSet (Prod r) ->- PackageFetcher ->- PackageSet (Prod (PackageFetcher ': r))+fetch :: Attach PackageFetcher PackageFetcher fetch = (. pure) . andThen -------------------------------------------------------------------------------- -- | A synonym of 'fetchGitHub' and 'sourceGitHub'-fromGitHub ::- PackageSet (Prod r) ->- (Text, Text) ->- PackageSet- (Prod (PackageFetcher : VersionSource : r))+fromGitHub :: AttachMany '[PackageFetcher, VersionSource] (Text, Text) fromGitHub e (owner, repo) = fromGitHub' e (owner, repo, id) -- | A synonym of 'fetchGitHub'' and 'sourceGitHub'-fromGitHub' ::- PackageSet (Prod r) ->- (Text, Text, NixFetcher Fresh -> NixFetcher Fresh) ->- PackageSet- (Prod (PackageFetcher : VersionSource : r))+fromGitHub' :: AttachMany '[PackageFetcher, VersionSource] (Text, Text, NixFetcher Fresh -> NixFetcher Fresh) fromGitHub' e p@(owner, repo, _) = fetchGitHub' (sourceGitHub e (owner, repo)) p -- | A synonym of 'fetchGitHub' and 'sourceGitHubTag'-fromGitHubTag ::- PackageSet (Prod r) ->- (Text, Text, ListOptions -> ListOptions) ->- PackageSet- (Prod (PackageFetcher : VersionSource : r))+fromGitHubTag :: AttachMany '[PackageFetcher, VersionSource] (Text, Text, ListOptions -> ListOptions) fromGitHubTag e (owner, repo, f) = fromGitHubTag' e (owner, repo, f, id) -- | A synonym of 'fetchGitHub'' and 'sourceGitHubTag' fromGitHubTag' ::- PackageSet (Prod r) ->- (Text, Text, ListOptions -> ListOptions, NixFetcher Fresh -> NixFetcher Fresh) ->- PackageSet- (Prod (PackageFetcher : VersionSource : r))+ AttachMany+ '[PackageFetcher, VersionSource]+ (Text, Text, ListOptions -> ListOptions, NixFetcher Fresh -> NixFetcher Fresh) fromGitHubTag' e (owner, repo, fv, ff) = fetchGitHub' (sourceGitHubTag e (owner, repo, fv)) (owner, repo, ff) -- | A synonym of 'fetchPypi' and 'sourcePypi'-fromPypi ::- PackageSet (Prod r) ->- Text ->- PackageSet- (Prod (PackageFetcher : VersionSource : r))+fromPypi :: AttachMany '[PackageFetcher, VersionSource] Text fromPypi e p = fetchPypi (sourcePypi e p) p +-- | A synonym of 'fetchOpenVsx', 'sourceOpenVsx', and 'passthru' extension's publisher with name+fromOpenVsx :: AttachMany '[PackagePassthru, PackageFetcher, VersionSource] (Text, Text)+fromOpenVsx e x@(publisher, extName) =+ passthru+ (fetchOpenVsx (sourceOpenVsx e x) x)+ [ ("name", extName),+ ("publisher", publisher)+ ]++-- | A synonym of 'fetchVscodeMarketplace', 'sourceVscodeMarketplace', and 'passthru' extension's publisher with name+fromVscodeMarketplace :: AttachMany '[PackagePassthru, PackageFetcher, VersionSource] (Text, Text)+fromVscodeMarketplace e x@(publisher, extName) =+ passthru+ (fetchVscodeMarketplace (sourceVscodeMarketplace e x) x)+ [ ("name", extName),+ ("publisher", publisher)+ ]+ -------------------------------------------------------------------------------- -- | This package follows the latest github release-sourceGitHub ::- PackageSet (Prod r) ->- -- | owner and repo- (Text, Text) ->- PackageSet (Prod (VersionSource : r))+sourceGitHub :: Attach VersionSource (Text, Text) sourceGitHub e (owner, repo) = src e $ GitHubRelease owner repo -- | This package follows the a tag from github-sourceGitHubTag ::- PackageSet (Prod r) ->- -- | owner, repo, and nvchecker list options to find the target tag- (Text, Text, ListOptions -> ListOptions) ->- PackageSet (Prod (VersionSource : r))+--+-- Args are owner, repo, and nvchecker list options to find the target tag+sourceGitHubTag :: Attach VersionSource (Text, Text, ListOptions -> ListOptions) sourceGitHubTag e (owner, repo, f) = src e $ GitHubTag owner repo $ f def -- | This package follows the latest git commit-sourceGit ::- PackageSet (Prod r) ->- -- | git url- Text ->- PackageSet (Prod (VersionSource : r))+--+-- Arg is git url+sourceGit :: Attach VersionSource Text sourceGit e _vurl = src e $ Git _vurl def -- | Similar to 'sourceGit', but allows to specify branch-sourceGit' ::- PackageSet (Prod r) ->- -- | git url and branch- (Text, Text) ->- PackageSet (Prod (VersionSource : r))+--+-- Args are git url and branch+sourceGit' :: Attach VersionSource (Text, Text) sourceGit' e (_vurl, coerce . Just -> _vbranch) = src e $ Git {..} -- | This package follows the latest pypi release-sourcePypi ::- PackageSet (Prod r) ->- -- | pypi name- Text ->- PackageSet (Prod (VersionSource : r))+--+-- Arg is pypi name+sourcePypi :: Attach VersionSource Text sourcePypi e _pypi = src e Pypi {..} -- | This package follows the version of an Arch Linux package-sourceArchLinux ::- PackageSet (Prod r) ->- -- | package name in Arch Linux repo- Text ->- PackageSet (Prod (VersionSource : r))+--+-- Arg is package name in Arch Linux repo+sourceArchLinux :: Attach VersionSource Text sourceArchLinux e _archpkg = src e ArchLinux {..} -- | This package follows the version of an Aur package-sourceAur ::- PackageSet (Prod r) ->- -- | package name in Aur- Text ->- PackageSet (Prod (VersionSource : r))+--+-- Arg is package name in Aur+sourceAur :: Attach VersionSource Text sourceAur e _aur = src e Aur {..} -- | This package follows a pinned version-sourceManual ::- PackageSet (Prod r) ->- Text ->- PackageSet (Prod (VersionSource : r))+--+-- Arg is manual version+sourceManual :: Attach VersionSource Text sourceManual e _manual = src e Manual {..} -- | This package follows the version of a repology package-sourceRepology ::- PackageSet (Prod r) ->- -- | repology project name and repo- (Text, Text) ->- PackageSet (Prod (VersionSource : r))+--+-- Args are repology project name and repo+sourceRepology :: Attach VersionSource (Text, Text) sourceRepology e (project, repo) = src e $ Repology project repo -- | This package follows a version extracted from web page-sourceWebpage ::- PackageSet (Prod r) ->- -- | web page url, regex, and list options- (Text, Text, ListOptions -> ListOptions) ->- PackageSet (Prod (VersionSource : r))+--+-- Args are web page url, regex, and list options+sourceWebpage :: Attach VersionSource (Text, Text, ListOptions -> ListOptions) sourceWebpage e (_vurl, _regex, f) = src e $ Webpage _vurl _regex $ f def -- | This package follows a version extracted from http header-sourceHttpHeader ::- PackageSet (Prod r) ->- -- | url of the http request, regex, and list options- (Text, Text, ListOptions -> ListOptions) ->- PackageSet (Prod (VersionSource : r))+--+-- Args are the url of the http request, regex, and list options+sourceHttpHeader :: Attach VersionSource (Text, Text, ListOptions -> ListOptions) sourceHttpHeader e (_vurl, _regex, f) = src e $ HttpHeader _vurl _regex $ f def +-- | This package follows a version in Open VSX+--+-- Args are publisher and extension name+sourceOpenVsx :: Attach VersionSource (Text, Text)+sourceOpenVsx e (_ovPublisher, _ovExtName) = src e OpenVsx {..}++-- | This package follows a version in Vscode Marketplace+--+-- Args are publisher and extension name+sourceVscodeMarketplace :: Attach VersionSource (Text, Text)+sourceVscodeMarketplace e (_vsmPublisher, _vsmExtName) = src e VscodeMarketplace {..}+ -------------------------------------------------------------------------------- -- | This package is fetched from a github repo-fetchGitHub ::- PackageSet (Prod r) ->- -- | owner and repo- (Text, Text) ->- PackageSet (Prod (PackageFetcher : r))+--+-- Args are owner and repo+fetchGitHub :: Attach PackageFetcher (Text, Text) fetchGitHub e (owner, repo) = fetchGitHub' e (owner, repo, id) -- | This package is fetched from a github repo@@ -413,80 +452,70 @@ -- @ -- define $ package "qliveplayer" `sourceGitHub` ("IsoaSFlus", "QLivePlayer") `fetchGitHub'` ("IsoaSFlus", "QLivePlayer", fetchSubmodules .~ True) -- @-fetchGitHub' ::- PackageSet (Prod r) ->- -- | owner and repo- ( Text,- Text,- NixFetcher Fresh -> NixFetcher Fresh- ) ->- PackageSet (Prod (PackageFetcher : r))+fetchGitHub' :: Attach PackageFetcher (Text, Text, NixFetcher Fresh -> NixFetcher Fresh) fetchGitHub' e (owner, repo, f) = fetch e $ f . gitHubFetcher (owner, repo) -- | This package is fetched from a file in github release-fetchGitHubRelease ::- PackageSet (Prod r) ->- -- | owner, repo, and file name- (Text, Text, Text) ->- PackageSet (Prod (PackageFetcher : r))+--+-- Args are owner, repo, and file name+fetchGitHubRelease :: Attach PackageFetcher (Text, Text, Text) fetchGitHubRelease e (owner, repo, fp) = fetch e $ gitHubReleaseFetcher (owner, repo) fp -- | This package is fetched from pypi-fetchPypi ::- PackageSet (Prod r) ->- -- | pypi name- Text ->- PackageSet (Prod (PackageFetcher : r))+--+-- Arg is pypi name+fetchPypi :: Attach PackageFetcher Text fetchPypi e = fetch e . pypiFetcher -- | This package is fetched from git-fetchGit ::- PackageSet (Prod r) ->- -- | git url- Text ->- PackageSet (Prod (PackageFetcher : r))+--+-- Arg is git url+fetchGit :: Attach PackageFetcher Text fetchGit e u = fetchGit' e (u, id) -- | This package is fetched from git -- -- Similar to 'fetchGit', but allows a modifier to the fetcher. -- See 'fetchGitHub'' for a concret example.-fetchGit' ::- PackageSet (Prod r) ->- -- | git url- (Text, NixFetcher Fresh -> NixFetcher Fresh) ->- PackageSet (Prod (PackageFetcher : r))+fetchGit' :: Attach PackageFetcher (Text, NixFetcher Fresh -> NixFetcher Fresh) fetchGit' e (u, f) = fetch e $ f . gitFetcher u -- | This package is fetched from url-fetchUrl ::- PackageSet (Prod r) ->- -- | url, given a specific version- (Version -> Text) ->- PackageSet (Prod (PackageFetcher : r))+--+-- Arg is a function which constructs the url from a version+fetchUrl :: Attach PackageFetcher (Version -> Text) fetchUrl e f = fetch e (urlFetcher . f) +-- | This package is fetched from Open VSX+--+-- Args are publisher and extension name+fetchOpenVsx :: Attach PackageFetcher (Text, Text)+fetchOpenVsx e = fetch e . vscodeMarketplaceFetcher++-- | This package is fetched from Vscode Marketplace+--+-- Args are publisher and extension name+fetchVscodeMarketplace :: Attach PackageFetcher (Text, Text)+fetchVscodeMarketplace e = fetch e . vscodeMarketplaceFetcher+ -------------------------------------------------------------------------------- -- | Extract files from fetched package source-extractSource ::- PackageSet (Prod r) ->- [FilePath] ->- PackageSet (Prod (PackageExtractSrc : r))-extractSource = (. pure . PackageExtractSrc) . andThen+extractSource :: Attach PackageExtractSrc [FilePath]+extractSource = (. pure . PackageExtractSrc . NE.fromList) . andThen -- | Run 'FetchRustGitDependencies' given the path to @Cargo.lock@ -- -- The lock file will be extracted as well.-hasCargoLock ::- PackageSet (Prod r) ->- FilePath ->- PackageSet (Prod (PackageCargoFilePath : r))+hasCargoLock :: Attach PackageCargoFilePath FilePath hasCargoLock = (. pure . PackageCargoFilePath) . andThen -- | Set 'NvcheckerOptions' for a package, which can tweak the version number we obtain-tweakVersion ::- PackageSet (Prod r) ->- (NvcheckerOptions -> NvcheckerOptions) ->- PackageSet (Prod (NvcheckerOptions : r))+tweakVersion :: Attach NvcheckerOptions (NvcheckerOptions -> NvcheckerOptions) tweakVersion = (. pure . ($ def)) . andThen++-- | An attrs set to pass through+--+-- Arg is a list of kv pairs+passthru :: Attach PackagePassthru [(Text, Text)]+passthru = (. pure . PackagePassthru . HMap.fromList) . andThen
src/NvFetcher/Types.hs view
@@ -56,6 +56,7 @@ PackageFetcher, PackageExtractSrc (..), PackageCargoFilePath (..),+ PackagePassthru (..), Package (..), PackageKey (..), )@@ -65,6 +66,7 @@ import Data.Coerce (coerce) import Data.Default import Data.HashMap.Strict (HashMap)+import qualified Data.List.NonEmpty as NE import Data.Maybe (fromMaybe) import Data.String (IsString) import Data.Text (Text)@@ -158,6 +160,8 @@ | Repology {_repology :: Text, _repo :: Text} | Webpage {_vurl :: Text, _regex :: Text, _listOptions :: ListOptions} | HttpHeader {_vurl :: Text, _regex :: Text, _listOptions :: ListOptions}+ | OpenVsx {_ovPublisher :: Text, _ovExtName :: Text}+ | VscodeMarketplace {_vsmPublisher :: Text, _vsmExtName :: Text} deriving (Show, Typeable, Eq, Ord, Generic, Hashable, Binary, NFData) -- | The input of nvchecker@@ -219,7 +223,7 @@ -- | Extract file contents from package source -- e.g. @Cargo.lock@-data ExtractSrcQ = ExtractSrcQ (NixFetcher Fetched) [FilePath]+data ExtractSrcQ = ExtractSrcQ (NixFetcher Fetched) (NE.NonEmpty FilePath) deriving (Show, Eq, Ord, Hashable, NFData, Binary, Typeable, Generic) type instance RuleResult ExtractSrcQ = HashMap FilePath Text@@ -243,17 +247,21 @@ -- | How to create package source fetcher given a version type PackageFetcher = Version -> NixFetcher Fresh -newtype PackageExtractSrc = PackageExtractSrc [FilePath]+newtype PackageExtractSrc = PackageExtractSrc (NE.NonEmpty FilePath) newtype PackageCargoFilePath = PackageCargoFilePath FilePath +newtype PackagePassthru = PackagePassthru (HashMap Text Text)+ deriving newtype (Semigroup, Monoid)+ -- | A package is defined with: -- -- 1. its name -- 2. how to track its version -- 3. how to fetch it as we have the version--- 4. optional file paths to extract (dump to generated nix expr)--- 5. @Cargo.lock@ path (if it's a rust package)+-- 4. optional file paths to extract (dump to shake dir)+-- 5. optional @Cargo.lock@ path (if it's a rust package)+-- 6. an optional pass through map -- -- /INVARIANT: 'Version' passed to 'PackageFetcher' MUST be used textually,/ -- /i.e. can only be concatenated with other strings,/@@ -262,8 +270,9 @@ { _pname :: PackageName, _pversion :: NvcheckerQ, _pfetcher :: PackageFetcher,- _pextract :: PackageExtractSrc,- _pcargo :: Maybe PackageCargoFilePath+ _pextract :: Maybe PackageExtractSrc,+ _pcargo :: Maybe PackageCargoFilePath,+ _ppassthru :: PackagePassthru } -- | Package key is the name of a package.
+ test/CheckVersionSpec.hs view
@@ -0,0 +1,79 @@+{-# LANGUAGE OverloadedStrings #-}++module CheckVersionSpec where++import Control.Monad.Trans.Reader+import Data.Coerce (coerce)+import Data.Default (def)+import qualified Data.Map.Strict as Map+import Lens.Micro+import NvFetcher.Nvchecker+import NvFetcher.Types+import NvFetcher.Types.Lens+import Test.Hspec+import Utils++-- | We need a fakePackageKey here; otherwise the nvchecker rule would be cutoff+spec :: Spec+spec = aroundShake' (Map.singleton fakePackageKey fakePackage) $+ describe "nvchecker" $ do+ specifyChan "pypi" $+ runNvcheckerRule (Pypi "example") `shouldReturnJust` Version "0.1.0"++ specifyChan "archpkg" $+ runNvcheckerRule (ArchLinux "ipw2100-fw") `shouldReturnJust` Version "1.3"++ specifyChan "aur" $+ runNvcheckerRule (Aur "ssed") `shouldReturnJust` Version "3.62"++ specifyChan "git" $+ runNvcheckerRule+ (Git "https://gitlab.com/gitlab-org/gitlab-test.git" def)+ `shouldReturnJust` Version "ddd0f15ae83993f5cb66a927a28673882e99100b"++ specifyChan "github latest release" $+ runNvcheckerRule (GitHubRelease "harry-sanabria" "ReleaseTestRepo")+ `shouldReturnJust` Version "release3"++ specifyChan "github max tag" $+ runNvcheckerRule (GitHubTag "harry-sanabria" "ReleaseTestRepo" def)+ `shouldReturnJust` "second_release"++ specifyChan "github max tag with ignored" $+ runNvcheckerRule (GitHubTag "harry-sanabria" "ReleaseTestRepo" $ def & ignored ?~ "second_release release3")+ `shouldReturnJust` Version "first_release"++ specifyChan "http header" $ do+ runNvcheckerRule (HttpHeader "https://www.unifiedremote.com/download/linux-x64-deb" "urserver-([\\d.]+).deb" def)+ >>= shouldBeJust++ specifyChan "manual" $+ runNvcheckerRule (Manual "Meow") `shouldReturnJust` Version "Meow"++ specifyChan "openvsx" $+ runNvcheckerRule (OpenVsx "usernamehw" "indent-one-space") `shouldReturnJust` Version "0.2.7"++ specifyChan "repology" $+ runNvcheckerRule (Repology "ssed" "aur") `shouldReturnJust` Version "3.62"++ specifyChan "vsmarketplace" $+ runNvcheckerRule (VscodeMarketplace "usernamehw" "indent-one-space") `shouldReturnJust` Version "0.2.8"++--------------------------------------------------------------------------------++runNvcheckerRule :: VersionSource -> ReaderT ActionQueue IO (Maybe Version)+runNvcheckerRule v = runActionChan $ nvNow <$> checkVersion v def fakePackageKey++fakePackageKey :: PackageKey+fakePackageKey = PackageKey "a-fake-package"++fakePackage :: Package+fakePackage =+ Package+ { _pname = coerce fakePackageKey,+ _pversion = undefined,+ _pfetcher = undefined,+ _pcargo = undefined,+ _pextract = undefined,+ _ppassthru = undefined+ }
+ test/FetchRustGitDepsSpec.hs view
@@ -0,0 +1,36 @@+{-# LANGUAGE DataKinds #-}+{-# LANGUAGE OverloadedStrings #-}++module FetchRustGitDepsSpec where++import Control.Monad.Trans.Reader+import Data.HashMap.Strict (HashMap)+import qualified Data.HashMap.Strict as HMap+import Data.Maybe (fromJust)+import Data.Text (Text)+import NvFetcher.FetchRustGitDeps+import NvFetcher.NixFetcher+import NvFetcher.Types+import Test.Hspec+import Utils++spec :: Spec+spec = aroundShake $+ describe "fetchRustGitDeps" $+ specifyChan "works" $ do+ prefetched <-+ runPrefetchRule $+ gitFetcher+ "https://gist.github.com/NickCao/6c4dbc4e15db5da107de6cdb89578375"+ "8a5f37a8f80a3b05290707febf57e88661cee442"+ shouldBeJust prefetched+ runFetchRustGitDepsRule (fromJust prefetched) "Cargo.lock"+ `shouldReturnJust` HMap.fromList+ [ ("rand-0.8.3", Checksum "1khg0rnz8xxd389cprqmy9vq9sggzz78lb9n7hh2w6xfsl4xfyyc")+ ]++runPrefetchRule :: NixFetcher Fresh -> ReaderT ActionQueue IO (Maybe (NixFetcher Fetched))+runPrefetchRule fetcher = runActionChan $ prefetch fetcher++runFetchRustGitDepsRule :: NixFetcher Fetched -> FilePath -> ReaderT ActionQueue IO (Maybe (HashMap Text Checksum))+runFetchRustGitDepsRule fetcher lockPath = runActionChan $ fetchRustGitDeps fetcher lockPath
+ test/NixExprSpec.hs view
@@ -0,0 +1,69 @@+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE QuasiQuotes #-}++module NixExprSpec where++import qualified Data.List.NonEmpty as NE+import NeatInterpolation (trimming)+import NvFetcher.NixExpr+import NvFetcher.NixFetcher+import NvFetcher.Types+import Test.Hspec++spec :: Spec+spec = describe "toNixExpr" $ do+ it "works on bool" $ do+ toNixExpr True `shouldBe` "true"+ toNixExpr False `shouldBe` "false"++ it "works on string" $+ toNixExpr ("Foo" :: String) `shouldBe` [trimming|"Foo"|]++ it "works on list of strings" $+ toNixExpr ["Alice" :: String, "Bob", "Carol"] `shouldBe` [trimming|[ "Alice" "Bob" "Carol" ]|]++ it "renders fresh gitFetcher" $+ toNixExpr (gitFetcher "https://example.com" "fake_rev")+ `shouldBe` [trimming|+ fetchgit {+ url = "https://example.com";+ rev = "fake_rev";+ fetchSubmodules = false;+ deepClone = false;+ leaveDotGit = false;+ sha256 = lib.fakeSha256;+ }+ |]++ it "renders fresh urlFetcher" $+ toNixExpr (urlFetcher "https://example.com")+ `shouldBe` [trimming|+ fetchurl {+ url = "https://example.com";+ sha256 = lib.fakeSha256;+ }+ |]++ it "renders IFD of ExtractSrcQ" $+ toNixExpr+ ( ExtractSrcQ+ (FetchUrl "https://example.com" (Checksum "calculated sha256"))+ (NE.fromList ["a.txt", "b.txt"])+ )+ `shouldBe` [trimming|+ let+ drv = import (pkgs.writeText "src" ''+ pkgs: {+ src = pkgs.fetchurl {+ url = "https://example.com";+ sha256 = "calculated sha256";+ };+ }+ '');+ fileNames = [ "a.txt" "b.txt" ];+ toFile = f: builtins.readFile ((drv pkgs).src + "/" + f);+ in builtins.listToAttrs (builtins.map (x: {+ name = x;+ value = toFile x;+ }) fileNames)+ |]
+ test/PrefetchSpec.hs view
@@ -0,0 +1,38 @@+{-# LANGUAGE DataKinds #-}+{-# LANGUAGE OverloadedStrings #-}++module PrefetchSpec where++import Control.Monad.Trans.Reader+import NvFetcher.NixFetcher+import NvFetcher.Types+import Test.Hspec+import Utils++spec :: Spec+spec = aroundShake $+ describe "fetchers" $ do+ specifyChan "pypi" $+ runPrefetchRule (pypiFetcher "example" "0.1.0")+ `shouldReturnJust` Checksum "1fy3zylvks372jz7fkgad3x14bscz19m2anlq2bn8xs1r1p3x1zm"++ specifyChan "openvsx" $+ runPrefetchRule (openVsxFetcher ("usernamehw", "indent-one-space") "0.2.6")+ `shouldReturnJust` Checksum "0smb75z10h1aiqmj2irn2vz1z99djfsfpkczwiqk447frx388bd1"++ specifyChan "vsmarketplace" $+ runPrefetchRule (vscodeMarketplaceFetcher ("usernamehw", "indent-one-space") "0.2.6")+ `shouldReturnJust` Checksum "1vmq24hdbv8jwhvy85s7qq9gffcsndvsw8vmphxs95r7bc3439w7"++ specifyChan "git" $+ runPrefetchRule (gitFetcher "https://gitlab.com/gitlab-org/gitlab-test.git" "ddd0f15ae83993f5cb66a927a28673882e99100b")+ `shouldReturnJust` Checksum "10d0yqa0h00pd5a36nwx3hb4apd4f9hki0pi545ffgrpf9nzzg92"++ specifyChan "github" $+ runPrefetchRule (gitHubFetcher ("harry-sanabria", "ReleaseTestRepo") "release3")+ `shouldReturnJust` Checksum "1a8qv2h9ji6w5p4ljwwkgvk49w6hj9j7hgmw0ahw10y1i45s0b3i"++--------------------------------------------------------------------------------++runPrefetchRule :: NixFetcher Fresh -> ReaderT ActionQueue IO (Maybe Checksum)+runPrefetchRule f = runActionChan $ _sha256 <$> prefetch f
+ test/Spec.hs view
@@ -0,0 +1,1 @@+{-# OPTIONS_GHC -F -pgmF hspec-discover #-}
+ test/Utils.hs view
@@ -0,0 +1,87 @@+{-# LANGUAGE DataKinds #-}+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE RankNTypes #-}+{-# LANGUAGE ScopedTypeVariables #-}++module Utils where++import Control.Concurrent.Async+import Control.Concurrent.Extra+import Control.Concurrent.STM+import Control.Exception (Handler (..), SomeException, bracket, catches, throwIO)+import Control.Monad (void)+import Control.Monad.IO.Class+import Control.Monad.Trans.Reader+import Data.Map (Map)+import Data.Maybe (isJust)+import Development.Shake+import Development.Shake.Database+import NvFetcher.Core (coreRules)+import NvFetcher.Types+import NvFetcher.Types.ShakeExtras+import qualified System.IO.Extra as Extra+import System.Time.Extra+import Test.Hspec+import UnliftIO (MonadUnliftIO (withRunInIO))++--------------------------------------------------------------------------------++runAction :: ActionQueue -> Action a -> IO (Maybe a)+runAction chan x = do+ barrier <- newBarrier+ atomically $ writeTQueue chan $ x >>= liftIO . signalBarrier barrier+ -- TODO+ timeout 30 $ waitBarrier barrier++type ActionQueue = TQueue (Action ())++newAsyncActionQueue :: Map PackageKey Package -> IO (ActionQueue, Async ())+newAsyncActionQueue pkgs = Extra.withTempDir $ \dir -> do+ shakeExtras <- liftIO $ initShakeExtras pkgs 3+ chan <- atomically newTQueue+ (getShakeDb, _) <-+ shakeOpenDatabase+ shakeOptions+ { shakeExtra = addShakeExtra shakeExtras (shakeExtra shakeOptions),+ shakeFiles = dir,+ shakeVerbosity = Quiet+ }+ coreRules+ shakeDb <- getShakeDb++ let runner restore = do+ -- sequentially+ act <- liftIO $ atomically $ readTQueue chan+ catches+ (restore $ void $ shakeRunDatabase shakeDb [act])+ [ Handler $ \(e :: AsyncCancelled) -> throwIO e,+ Handler $ \(e :: SomeException) -> putStrLn $ "an exception arose in action runner: " <> show e+ ]+ runner restore++ runnerTask <- asyncWithUnmask $ \r -> runner r+ pure (chan, runnerTask)++--------------------------------------------------------------------------------++aroundShake :: SpecWith ActionQueue -> Spec+aroundShake = aroundShake' mempty++aroundShake' :: Map PackageKey Package -> SpecWith ActionQueue -> Spec+aroundShake' pkgs = aroundAll $ \f ->+ bracket+ (newAsyncActionQueue pkgs)+ (\(_, runnerTask) -> cancel runnerTask)+ (\(chan, _) -> f chan)++shouldReturnJust :: (Show a, Eq a, MonadUnliftIO m) => m (Maybe a) -> a -> m ()+shouldReturnJust f x = withRunInIO $ \run -> run f `shouldReturn` Just x++shouldBeJust :: (MonadIO m, Show a) => Maybe a -> m ()+shouldBeJust x = liftIO $ x `shouldSatisfy` isJust++specifyChan :: HasCallStack => String -> ReaderT ActionQueue IO () -> SpecWith ActionQueue+specifyChan s m = specify s $ \r -> runReaderT m r++runActionChan :: Action a -> ReaderT ActionQueue IO (Maybe a)+runActionChan m = ask >>= \chan -> liftIO $ runAction chan m