nvfetcher-0.8.0.0: README.md
# nvfetcher
[](https://hackage.haskell.org/package/nvfetcher)
[](LICENSE)
[](https://github.com/berberman/nvfetcher/actions/workflows/nix.yml)
`nvfetcher` is a tool to automate nix package updates. It's built on top of [shake](https://www.shakebuild.com/),
integrating [nvchecker](https://github.com/lilydjwg/nvchecker).
`nvfetcher` CLI program accepts a TOML file as config, which defines a set of package sources to run.
## Overview
For example, feeding the following configuration to `nvfetcher`:
```toml
# nvfetcher.toml
[rustdeck]
src.github = "rustdesk/rustdesk"
fetch.github = "rustdesk/rustdesk"
# Compute outputHashes for git dependencies in cargo lock
cargo_lock = ["Cargo.lock"]
```
You would get `_sources/generated.nix`:
```nix
# This file was generated by nvfetcher, please do not modify it manually.
{
fetchgit,
fetchurl,
fetchFromGitHub,
dockerTools,
}:
{
rustdesk = {
pname = "rustdesk";
version = "1.4.2";
src = fetchFromGitHub {
owner = "rustdesk";
repo = "rustdesk";
rev = "1.4.2";
fetchSubmodules = false;
sha256 = "sha256-b/KuWECsmEcz9nPiuPqCcGQ1p+GdbaELScaIXnUTGv8=";
};
cargoLock."Cargo.lock" = {
lockFile = ./. + "/sha256-b_KuWECsmEcz9nPiuPqCcGQ1p+GdbaELScaIXnUTGv8=/Cargo.lock";
outputHashes = {
"filedescriptor-0.8.2" = "sha256-zXTt9eHAVdCPipWAMFeiqk1JteX+7IZcXZ7uvxkzDjQ=";
# ...
};
};
};
}
```
and `_sources/generated.json`:
```json
{
"rustdesk": {
"cargoLock": {
"Cargo.lock": [
"sha256-b_KuWECsmEcz9nPiuPqCcGQ1p+GdbaELScaIXnUTGv8=/Cargo.lock",
{
"android-wakelock-0.1.0": "sha256-09EH/U1BBs3l4galQOrTKmPUYBgryUjfc/rqPZhdYc4=",
// ...
}
]
},
"date": null,
"extract": null,
"name": "rustdesk",
"passthru": null,
"pinned": false,
"src": {
"deepClone": false,
"fetchSubmodules": false,
"leaveDotGit": false,
"name": null,
"owner": "rustdesk",
"repo": "rustdesk",
"rev": "1.4.2",
"sha256": "sha256-b/KuWECsmEcz9nPiuPqCcGQ1p+GdbaELScaIXnUTGv8=",
"sparseCheckout": [],
"type": "github"
},
"version": "1.4.2"
}
}
```
We tell `nvfetcher` how to get the latest version number of packages and how to fetch their sources given version numbers,
and `nvfetcher` will help us keep their version and prefetched SHA256 sums up-to-date, producing ready-to-use nix expressions in `_sources/generated.nix`.
`nvfetcher` reads `generated.json` to produce version change messages, such as `feeluown: 3.8.1 → 3.8.2`.
Versions of packages are always checked during each run, but only do prefetch and further operations when needed.
### Live examples
How to use the generated sources file? Here are several examples:
- My [flakes repo](https://github.com/berberman/flakes)
**Add your own flakes repo here if you want to share it with others!**
## Installation
`nvfetcher` package is available in [nixpkgs](https://github.com/NixOS/nixpkgs), so you can try it with:
```
$ nix-shell -p nvfetcher
```
This repo also has flakes support:
```sh
$ nix run github:berberman/nvfetcher
```
To use it as a Haskell library, the package is available on [Hackage](https://hackage.haskell.org/package/nvfetcher).
If you want to use the Haskell library from flakes, there is also a shell `ghcWithNvfetcher`:
```sh
$ nix develop github:berberman/nvfetcher#ghcWithNvfetcher
$ runghc Main.hs
```
where you can define packages in `Main.hs`. See [Haskell library](#Haskell-library) for details.
## Usage
Basically, there are two ways to use `nvfetcher`, where the difference is how we provide package sources definitions to it.
### CLI
To run `nvfetcher` as a CLI program, you'll need to provide package sources defined in TOML.
```
Usage: nvfetcher [--version] [--help] [-o|--build-dir DIR] [--commit-changes]
[-l|--changelog FILE] [-j NUM] [-r|--retry NUM] [-t|--timing]
[-v|--verbose] [-f|--filter REGEX] [-k|--keyfile FILE]
[--keep-old] [--keep-going] [TARGET] [-c|--config FILE]
generate nix sources expr for the latest version of packages
Available options:
--version Show version
--help Show this help text
-o,--build-dir DIR Directory that nvfetcher puts artifacts to
(default: "_sources")
--commit-changes `git commit` build dir with version changes as commit
message
-l,--changelog FILE Dump version changes to a file
-j NUM Number of threads (0: detected number of processors)
(default: 0)
-r,--retry NUM Times to retry of some rules (nvchecker, prefetch,
nix-build, etc.) (default: 3)
-t,--timing Show build time
-v,--verbose Verbose mode
-f,--filter REGEX Regex to filter packages to be updated
-k,--keyfile FILE Nvchecker keyfile
--keep-old Don't remove old files other than generated json and
nix before build
--keep-going Don't stop if some packages failed to be fetched
TARGET Three targets are available: 1.build 2.clean (remove
all generated files) 3.purge (remove shake db)
(default: build)
-c,--config FILE Path to nvfetcher TOML config
(default: "nvfetcher.toml")
```
Each package corresponds to a TOML table: name is encoded as table key, with
two required fields and three optional fields in each table.
You can find an example of the configuration file, see [`nvfetcher_example.toml`](nvfetcher_example.toml).
### Keyfile
You can specify `nvchecker` keyfile via command line option.
For the format of this file, please refer to [nvchecker documentation](https://nvchecker.readthedocs.io/en/latest/usage.html#configuration-table).
#### Nvchecker
Version source -- how do we track upstream version updates?
- `src.github = owner/repo` - the latest github release
- `src.github_tag = owner/repo` - the max github tag, usually used with list options (see below)
- `src.pypi = pypi_name` - the latest pypi release
- `src.git = git_url` (and an optional `src.branch = git_branch`) - **the latest commit** of a repo
- `src.archpkg = archlinux_pkg_name` -- the latest version of an archlinux package
- `src.aur = aur_pkg_name` -- the latest version of an aur package
- `src.manual = v` -- a fixed version, which never updates
- `src.repology = project:repo` -- the latest version from repology
- `src.webpage = web_url` and `src.regex` -- a string in webpage that matches with regex
- `src.httpheader = request_url` and `src.regex` -- a string in http header that matches with regex
- `src.openvsx = publisher.ext_name` -- the latest version of a vscode extension from open vsx
- `src.vsmarketplace = publisher.ext_name` -- the latest version of a vscode extension from vscode marketplace
- `src.cmd = cmd` -- the version from a shell command (e.g. `echo Meow`)
- `src.container = owner/name` - the latest tag of a container from the Docker registry
Optional list options for some version sources (`src.github_tag`, `src.webpage`, and `src.httpheader` and `src.container`),
see the corresponding [nvchecker documentation](https://nvchecker.readthedocs.io/en/latest/usage.html#list-options) for details.
- `src.include_regex`
- `src.exclude_regex`
- `src.sort_version_key`
- `src.ignored`
Optional global options for all kinds of version sources,
see the corresponding [nvchecker documentation](https://nvchecker.readthedocs.io/en/latest/usage.html#global-options) for details. You can tweak obtained version number using this option, e.g. stripping the prefix `v` or transforming the result by regex.
- `src.prefix`
- `src.from_pattern`
- `src.to_pattern`
#### Nix fetcher
How do we fetch the package source if we have the target version number?
`$ver` is available in string, which will be set to the result of nvchecker.
- `fetch.github = owner/repo`
- `fetch.pypi = pypi_name`
- `fetch.git = git_url`
- `fetch.url = url`
- `fetch.openvsx = publisher.ext_name`
- `fetch.vsmarketplace = publisher.ext_name`
- `fetch.tarball = tarball_url`
- `fetch.docker = owner/name`
Optional config for `nix-prefetch-url`, applies when the fetcher equals to `fetch.url`.
`$ver` is available in string, just like for the fetch config.
- `url.name = file_name`
Optional config for `nix-prefetch-git`, applies when the fetcher equals to `fetch.github` or `fetch.git`.
- `git.deepClone`
- `git.fetchSubmodules`
- `git.leaveDotGit`
Optional config for `fetch.docker`/`dockerTools.pullImage`:
- `docker.os`
- `docker.arch`
- `docker.finalImageName`
- `docker.finalImageTag`
- `docker.tlsVerify`
#### Extract source files
Optional source extracting config, files are extracted into build directory.
- `extract = [ "glob_1", "glob_2", ...]` - globs are relative to the source root
Each glob will be resolved to a list of files matching the pattern.
For each matching file, `nvfetcher` copies it to the build directory, and generates `extract.<file_name> = ./. + "<path>"` entry in the nix expr.
Recursive globs are supported, e.g. one can write `extract = [ "**/*" ]` to pull all files in a package to build directory.
See [Glob](https://hackage.haskell.org/package/Glob-0.10.2/docs/System-FilePath-Glob.html#v:compile) for details.
Note: Directories are ignored.
#### Rust support
`rustPlatform.buildRustPackage` now accepts an attribute [`cargoLock`](https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/rust.section.md#importing-a-cargolock-file) to vendor dependencies from `Cargo.lock`,
so we can use this instead of TOFU `cargoSha256` for Rust packaging. `nvfetcher` supports automating this process,
extracting the lock file to build and calculating `cargoLock.outputHashes`, as long as you set the config.
There can be many lock files in one source.
- `cargo_lock = [ "cargo_lock_glob_1", "cargo_lock_glob_2", ...]` - globs are relative to the source root
Similar to extract, cargo locks will be pulled into build directory.
#### Passthru
_passthru_ config, an additional set of attrs to be generated.
- `passthru = { k1 = "v1", k2 = "v2", ... }`
Note: currently the values can only be strings
#### Pinned
If a package is pinned, we call nvchecker to check the new version iff there's no existing version.
- `pinned = true`
#### Git commit date
If the version source of a package is `git`, nvfetcher can finds out the commit date of this revision,
in the format of `%Y-%m-%d` by default. You can provide your own [`strftime`](https://strftime.org/) format:
- `git.date_format = "strftime_format"`
- `git.date_tz = "timezone"`
You can use the `git.date_tz` option to specify the time zone for the commit date. Accepts IANA time zone names, e.g. `git.date_tz = "America/New_York"`. Plus, you can use `git.date_tz = "local"` to refer to the local time zone.
#### Force fetching
Always fetch the package, even if its version (nvchecker output) doesn't change.
This is useful when the file to be downloaded cannot not be determined by the url.
- `fetch.force = true`
> Note: In such case, nvfetcher will not produce a version change log, since the version doesn't change at all.
> Only sha256 in generated files will be updated.
### Haskell library
nvfetcher itself is a Haskell library as well, whereas the CLI program is just a trivial wrapper of the library.
You can create a Haskell program depending on it directly, by using the `runNvFetcher` entry point.
In this case, we can define packages in Haskell language, getting rid of TOML constraints.
You can find an example of using nvfetcher in the library way, see [`Main_example.hs`](Main_example.hs).
## Documentation
For details of the library, documentation of released versions is available on [Hackage](https://hackage.haskell.org/package/nvfetcher),
and of master is on our [github pages](https://nvfetcher.berberman.space).
## Contributing
Issues and PRs are always welcome. **\_(:з」∠)\_**
Building from source:
```
$ git clone https://github.com/berberman/nvfetcher
$ nix develop
$ cabal update
$ cabal build
```